by saintamh on 4/26/19, 10:35 AM with 685 comments
by reaperducer on 4/26/19, 12:27 PM
It's big enough that when someone complains that a message sent wasn't received, the intended recipient will say, "I never have problems with my Gmail account. It must be you." And the sender has to switch to Gmail to reliably communicate with the outside world.
I wish this was just paranoia, but we've seen multiple discussions on HN about Google programs and policies that alter the internet in ways that only benefit Big G. It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."
by maxaf on 4/26/19, 12:28 PM
How can anyone be sure of this? This is only one of Google’s practices that seems to follow a pervasive pattern of eroding open internet standards while presenting Google’s own proprietary implementation as somehow superior. Eventually, the open standard loses all meaning because the most popular implementation does not actually adhere to it. Meanwhile, Google reaps enormous benefits in the form of additional signals for its advertising business. How can this not be grounded in malicious intent?
> and that there are some very smart people working on spam prevention at Google.
There are some very smart people working on advertising at Google. The rush to forget the primary nature of Google - it’s an adtech firm - is why they’ve been allowed to skate for so long. Gmail’s spam filtering is just a pretext for passing all email through a machine learning system. Sure, one possible signal emitted by that system is whether a message is spam or not. Perhaps this determination is conflated with wether the message is useful for ad targeting: after all, when viewed from Google’s own perspective certain e-mail messages contain no information which can be used for ad targeting, so they must be spam. The user’s interests are clearly secondary to this.
So, back to the “smart people” working on this: at what point do we begin judging engineers for working at Google? There’s a lot of highly vitriolic criticism that emanates from Google’s workforce on a variety of subjects, but how many of them would actually pull the pin and leave their employer? I don’t have any statistics to offer, but it seems to me that we still have a ways to go before Google has become completely drained of engineering mindshare.
by jbergstroem on 4/26/19, 1:20 PM
hotmail-com.olc.protection.outlook.com[104.47.1.33] said: 550 5.7.1
Unfortunately, messages from [<redacted>] weren't sent. Please contact
your Internet service provider since part of their network is on our block
list (S3150). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[VE1EUR01FT028.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)
Reporting-MTA: dns; <redacted>
X-Postfix-Queue-ID: 14A41FEB66
X-Postfix-Sender: rfc822; <redacted>
Arrival-Date: Thu, 14 Mar 2019 14:07:42 +0200 (CEST)by i_dont_know_ on 4/26/19, 12:25 PM
I set up DKIM, SPF, and reverse-DNS records and resented every moment of it. Even after all that, there's some chance that an email from my server will be marked as insecure/spam or otherwise just not be delivered because Google has come up with some new brilliant mail security/auth/permission scheme that the world has to adopt tomorrow or be cut off from all Gmail users.
by jasode on 4/26/19, 12:43 PM
Yep, I know the author's frustration very well. I made a previous comment[0] trying to warn others of personal email servers' outgoing email being spam-holed -- and yet some of the replies still argued I was overstating the difficulties.
Everybody's risk tolerance is different. Personally, I just don't have the bandwidth to administer my own private email server and constantly worry if recipients are receiving my emails.
by reaperducer on 4/26/19, 12:46 PM
Remember for a couple of years, every time you'd send a message to someone at Earthlink, you'd get an automated rely demanding that you verify yourself before the message would go through?
Now I can't remember the last time I saw an @earthlink.* e-mail address.
Google has apparently learned from that and put the "error" in the 550 messages, where they can't be seen by the end user, and lead to non-helpful resolutions for sysadmins.
The result is that the blame for missing messages goes to the sender, not to the recipient's email service.
If Gmail at least notified the sender that there was a problem, then a pattern of responsibility could be established. But this is just another dark pattern.
by LeonM on 4/26/19, 1:28 PM
A couple of things that regularly seems to trigger false positives in spam algorithms:
- no or misconfigured SPF and/or DKIM
- no or misconfigured reverse-DNS
- automatically included footer texts (confidentiality, copyright, safe a tree don't print, etc)
- regular automatic replies from the domain (such as out of office notifications)
- the use of embedded images (logos, human signatures, etc)
We sometimes joke that these triggers were built in by the algorithm developers as a means of punishing those who litter their email with pointless texts and images.
by tambourine_man on 4/26/19, 1:09 PM
So, naturally, as many of you, I went the mail/postfix, DKIM, SPF, etc way. And all is fine until you start receiving random hard bounces with no real debugable answer for Google.
It got me deeply sad and questioning my decisions: since you can’t really ignore Gmail, email isn’t in practice “open” anymore. So I might as well sign up for Facebook, WhatsApp and the likes. It’s been years and I haven’t yet, but it’s getting harder and harder.
by megous on 4/26/19, 3:20 PM
It's like there's an assumption that gmail is perfect, and the problem is with the sender. Even if that was true, a normal mail hosting company would at least tell its customer why the mail is not being delivered, so that the customer can tell the sender what to fix.
The gmail recipient is never exposed to this side of google. So they don't know what a nightmare comapny it is to communicate with.
Why should everyone and their dog be solving gmail users's problems with receiving messages? It's such a demented system. It should be the other way round. Recipients, via their provider should be solving their issues with spam filtering and blocking.
If the gmail user would be blocked by my mail server, I would not tell them to go guess what's wrong, fix gmail, and to have fun. It should not be acceptable the other way round either.
by john37386 on 4/26/19, 6:34 PM
I started to host emails to many friends, small businesses and even a SaaS I developed. The subscription needs an email validation and I'm aware that the activation email ends up in the Spam folder for the new customers using Google emails. This activation email has everything from dkim, spf, dmarc, to unsubscribe link, full physical address of the business, etc and still I can't hit a good enough score.
I was thinking to start using Google service to send the activation link and hosting my personal domains, but seeing that I am not alone, I will continue to improve my little email projects.
Thanks all for cheering me up on this. I'm sure we can come up with a solution and I would be happy to help. When do we start?
by alyandon on 4/26/19, 1:55 PM
Hilariously, Google will flag these emails sent to myself using my own credentials and their infrastructure as spam. I have no faith in them ever getting this right.
by bartimus on 4/26/19, 1:19 PM
https://toolbox.googleapps.com/apps/checkmx/check?domain=tab...
by hazeii on 4/26/19, 1:52 PM
by verisimilitudes on 4/26/19, 5:13 PM
I don't send much mail to gmail, though. Sans that, my only issue has been a mail server that uses Reverse DNS, which I don't have set up, and entirely ignores my email without it.
I suppose I can understand this if some people get a great deal of spam, but requiring so much of this on an unencrypted message seems more like useless reassurances than anything. I'm not criticizing email for being unencrypted, but this seems more like another hoop to jump through than anything.
Also of note, almost all of the spam I receive is from gmail addresses and I wouldn't be surprised if the invalid addresses that send messages demanding bitcoin are also from gmail, but with fake From fields.
by npsimons on 4/26/19, 5:47 PM
I'm in a similar boat; been running email servers since before GMail existed. My personal one I've been running out of a home server closet since 2001. I've also done everything I can to guarantee I'm not running an open relay and not sending email unsolicited. Have been mostly lucky so far, but occasionally I will have people on mailing lists I manage (people I have met IRL and put them on the list to organize group meetings IRL) not get email. Used to be other stupid mail providers (AOL comes to mind), but these days it appears to be Google, sometimes.
I've had this domain nearly twenty years and run email on it for that same amount of time. I'm not going to "just switch", especially to a privacy invading ad-spewing "alternative" that doesn't give me as much control. Fix your damn servers, Google.
by sneak on 4/26/19, 12:31 PM
by pettycashstash2 on 4/26/19, 10:42 AM
by gmailvictim on 4/26/19, 8:58 PM
by petercooper on 4/26/19, 1:15 PM
You could literally have two long standing, legitimately used accounts send an email to each other containing a link to a URL like http://0xANYTHINGHERE.com/ and it would be insta-spammed. I suspect it was a hard coded rule to avoid people using "long IP" URLs to circumvent other filters.. except there are lots of legitimate 0x domains that aren't long IPs.
It was fixed sometime in the past year but I got a lot of use out of it in talks I've given about email deliverability over the years.
by jussij on 4/26/19, 2:30 PM
Without these customisations that forum would be overrun with all sorts of spam.
However, these customisations only stop spam postings but can't stop actual registrations.
Based on the users that I see who are registering I see a great majority of these spammers love using Gmail accounts.
So while it is good that Google Gmail is trying to fix these spam issues, from where I stand Gmail users seem to be a big part of the spamming problem.
Spammers love Gmail only because they can easily create spamming e-mail accounts.
by syntheticnature on 4/26/19, 2:58 PM
Of course, email forwarding turns out to suck, but we're just going to suck it up and move to G Suite for organizational email addresses and let folks forward from there. E-lists, OTOH, I haven't found a good integration to automate membership in the organization vs. G-Suite; perhaps it's time to just move to a forum.
by superpie on 4/26/19, 12:26 PM
The amount of fighting you have to do to stay on everyone's whitelists is absurd.
by feanaro on 4/26/19, 10:01 PM
Google seems to be suffocating the internet bit by bit on all fronts and it needs to be stopped.
by alfiedotwtf on 4/27/19, 12:08 AM
Every new recipient I email, if I don't hear from them within 2 days, I have to contact them out of band to ask them to check their spam folder. The problem is usually Gmail's heavy filtering.
by lucb1e on 4/26/19, 2:06 PM
At work we use sendgrid because of this. Have to trust a centralised third party to send out api keys. It's frustrating.
by crank190426 on 4/26/19, 2:47 PM
This is similar to the +whatevertag trick that gmail pioneered for tagging, however that can be removed by malicious parties (spammers) via a simple regex. So Google have almost all of the infrastructure but should just add a bit more to get the rest of the way there.
What I mean in specific :
1. I want to sign up to and receive your newsletter (you[re Ted) but I don't trust you yet. so I should navigate to gmail.com, click something like "generate another inbox", leave it set it to "For now deliver this mail to my inbox", add the description "for Ted's Possibly Spammy Newsletter", and then click "generate". It should give me inbox3943578423@gmail.com - similar to a phone number but a bit longer and personalized to one recipient - and then I should give that to the recipient to use, in this case the possibly spammy newsletter. It should always be delivered to my inbox, as I've set. Once one of the spammers sells my email address (for example I start getting advance payment scams) I'll be able to disable further spam from there by sending it to the trash but also know that Ted's newsletter is the one that got compromised or sold it. You can do this today by going through the steps of registering a new gmail address and turning on forwarding, but it takes like 10 minutes to do so. it should be like 10 seconds.
This should be possible because people always have easy access to the gmail web interface. There's no reason it can't be a bit more like a social network where you confirm it from the web interface as well.
that's my idea anyway.
by z3t4 on 4/26/19, 8:50 PM
by js2 on 4/26/19, 3:20 PM
e.g. https://pastebin.com/u48DAaLP
That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.
After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.
by cosmin800 on 4/26/19, 1:27 PM
by tssva on 4/26/19, 5:27 PM
Mailman, which most open source projects use for mailing lists, have developed work arounds to address some of the issues. Unfortunately my experience is that many projects run older versions that don't have these work arounds or if running newer versions they have not been enabled. Most likely because no one has revisited the configuration since initial deployment on an older version. After all they didn't start the project to spend their time being mailing list admins.
by surgi on 4/26/19, 1:26 PM
by neilv on 4/26/19, 9:57 PM
I'm not interested in ProtonMail's encryption (and it's potentially a liability, attracting aggressive state action). I'm mainly interested in their apparent respect for the privacy of users' private communications. And also hoping that ProtonMail has a bit more reliable delivery than GMail.
In any case, rising competition lifts all performance boats, or something like that.
by zahllos on 5/1/19, 1:09 PM
I don't think it's unreasonable to be strict regarding DMARC delivery. My MTA has a fairly strict SPF configuration - any email with an invalid spf result is rejected. This can come about because a legitimate company has misconfigured their spf records (happened twice in all the years I have hosted, discussions via postmaster@ helped them configure their dns correctly), but 99.999% of the time it is a spammer. What is worse is that rejecting email for domains without any SPF records can still result in valid email being lost, in 2019.
In this specific case, I don't think Google are "being evil". They're trying to reduce spam in the email ecosystem and they're doing it by using standards they themselves adhere to (Gmail send me reports of dmarc statistics each day google domains receive email from my box).
On the other hand, I do of course support either self hosting, or using another provider so as to ensure we do not end up with a Gmail monopoly. If I did not self host, I would find another provider like (but may not) Fastmail, Posteo etc (I would have to seriously review the options, which I haven't done).
by username223 on 4/26/19, 2:31 PM
by rayiner on 4/26/19, 2:03 PM
by ameliozanchi on 4/26/19, 3:12 PM
by mullingitover on 4/26/19, 7:22 PM
by blunte on 4/27/19, 9:34 AM
One day a colleague and I discovered that he had not received some of my emails (intra-domain - me@example.com to him@example.com).
This is all within the confines of Google. Google had flagged some messages as spam, and by what determination I could not fathom. The content seemed perfectly typical.
I have had really pleasant experiences with G-Suite human support, at least in terms of the quality of interaction. But they could not answer why some intra-domain emails were being flagged as spam. I have suspicions that it would take a whole team of G engineers to maybe identify what bit of logic in their systems (incorrectly) marked some of the emails as spam.
It seems the beast (automation) is just almost not under their control anymore.
by c3534l on 4/26/19, 3:52 PM
by wichert on 4/26/19, 1:07 PM
by specialist on 4/26/19, 3:59 PM
#1 - Does anyone send test emails and measure delivery rates? As in send yourself a bunch of emails and see what happens.
USPS and its major customers and vendors do this with physical mail. They measure stuff like UAA (undeliverable as addressed). FWIW, their Inspector General estimates 4.3% of mail was UAA in 2013. Report Number: MS-AR-14-006 https://www.uspsoig.gov/document/undeliverable-addressed-mai...
#2 - What is the responsibility, liability for email relays to treat everyone equally? For comparison, a US retailer has to accept US currency, but can (sometimes) turn away problematic clients. Is there anything like that for electronic exchanges, transactions?
by johnnyhead on 4/27/19, 2:02 PM
Reminds me of jabber.ccc.de that stopped providing new accounts because they felt they were ruining a federated system.
by beams_of_light on 4/26/19, 1:17 PM
by strimp099 on 4/28/19, 4:07 AM
by duxup on 4/26/19, 1:31 PM
It should be noted that ultimately these efforts to "learn" result in ML, AI whatever pointed AT you, not working for you.
by JohnFen on 4/26/19, 4:30 PM
Like the author, I've been running my own mailserver for over a decade and am very conscientious about ensuring that no attackers use it as a spam relay.
While the vast majority of the people I exchange email with don't use GMail at all, so it can take a while before I notice any issues with it, I did happen to notice that GMail was rejecting my outgoing email a couple of weeks ago.
This week, I finally got around to trying to address the problem (it's not high priority because having GMail reject my emails isn't really a huge deal).
...and I found that it is working again without my changing anything. Weirdness abounds.
by kazinator on 4/26/19, 6:06 PM
Sending SMTP yourself (directly, without an SMTP relay service) sets you up for trouble.
by ddebernardy on 4/26/19, 2:58 PM
by tempestn on 4/27/19, 12:12 AM
Edit: And of course, I do have DKIM and SPF configured.
by dangjc on 4/27/19, 3:27 PM
by Felz on 4/26/19, 5:57 PM
by mtw on 4/26/19, 3:29 PM
> I can't tell other people to go off Gmail
I disagree. There are reasons to switch off gmail. Not just Google eating mail but also for privacy reasons. Google knows all about your banking, eCommerce orders, your media subscriptions, health issues and many other dependencies.
A good alternative is protonmail. It is private, has a mobile app, is a free but you can also pay to support the service. I also consider protonmail much more secure than gmail.
by reedlaw on 4/26/19, 4:55 PM
by aloukissas on 4/26/19, 2:12 PM
Definitely Google has done something to mess up their spam filter algorithms in the last year.
[1] https://news.ycombinator.com/item?id=19536465 [2] https://news.ycombinator.com/item?id=19500357
by jboy55 on 4/26/19, 7:59 PM
by mises on 4/26/19, 3:42 PM
by angry_octet on 4/27/19, 1:32 PM
Spam from small domains might be pretty high as a category, but of course we don't want statistical judgements about categories to outweigh the merits of the individual. Maybe Google's algorithms have been watching too much Fox News.
by no_wizard on 4/26/19, 4:14 PM
I use FastMail for instance and never have this issue, but I know so many people who gave up on running their own mail servers at the small enterprise level because of stuff like this I often wonder how FastMail does not have these issues but others do. Is it a headers thing I wonder?
by cmsimike on 4/26/19, 5:29 PM
by zzo38computer on 4/26/19, 10:25 PM
by SourceParts on 4/29/19, 11:35 AM
by jasonvorhe on 4/26/19, 9:23 PM
People hosting their own servers enabled wide spread abuse due to misconfigurations. Because everyone could do it and because defaults were shit for decades, stuff like open relays were common. People defaulted to the wrong ports. Almost no one bothered to offer STARTTLS/Transport Encryption. Spam would have killed mail by now if it hadn't been for major players like Google, GMX, Hotmail/Outlook/etc.
Back in the day, greylisting was commonly regarded as a best practice, leading to the impression that email is unreliable and prone to latencies.
I'm sorry it's this difficult to host mail by yourself nowadays, but I'm happy to have a spam-free inbox every day and if this is the price for that, I'm sure about 1-2 billion people are willing to pay it.
I'm quite astounded that there have been no updates to mail protocols in the last couple of years to at least mitigate the most common issues, but all I see are band-aids that are complex to setup and horrible to debug in case of issues.
by Tharkun on 4/26/19, 1:37 PM
by ilaksh on 4/29/19, 3:54 AM
by derefnull on 4/27/19, 6:32 PM
I currently self-host for non-mission-critical email, use FastMail for business, and continue to use google apps for personal/mission-critical.
by pera on 4/26/19, 3:19 PM
(tablix.org doesn't have a DMARC record)
by majestik on 4/26/19, 3:18 PM
by vinay_ys on 4/28/19, 10:08 AM
by rootusrootus on 4/26/19, 1:18 PM
by GnarfGnarf on 4/27/19, 10:47 AM
by afarah on 4/26/19, 3:26 PM
by jayalpha on 4/27/19, 12:28 AM
Try fastmail infomaniak.com (can also buy email for external domain) gandi.net (email included in domain)
by jdmoreira on 4/26/19, 1:27 PM
In the end I just gave up and started using mailgun as a relay.
by alanlovestea on 4/26/19, 8:44 PM
by DoctorOetker on 4/26/19, 3:07 PM
Perhaps there really would be a lot more spam without such filtering, but it points to the actual problem being elsewhere. Perhaps we need some kind of cheap and userriendly (uniform but decentralized) email court system, and fine / ban email accounts that misbehave?
by stratosmacker on 4/26/19, 12:37 PM
What can we do about Google's email monopoly?
by ackbar03 on 4/26/19, 3:07 PM
by richardriko on 4/27/19, 6:41 PM
by muppetman on 4/27/19, 5:47 AM
by chrisfinazzo on 4/26/19, 2:35 PM
by rdlecler1 on 4/26/19, 4:46 PM
by vectorEQ on 4/26/19, 3:43 PM
by peterwwillis on 4/27/19, 12:43 AM
Until this happens. Which it does very often, for lots of reasons that are out of your control.
by StreamBright on 4/26/19, 4:08 PM
by TheTruth1234 on 4/26/19, 4:04 PM
I like my emails with pesto sauce, tuna, and cheese ... wicked combination.
by 5874-4b22-a4e0 on 4/26/19, 1:30 PM
by zoom6628 on 4/26/19, 1:34 PM
by imhelpingu on 4/26/19, 4:40 PM