from Hacker News

Gmail having issues

by mangoman on 12/15/20, 10:12 PM with 432 comments

  • by windexh8er on 12/16/20, 1:11 AM

    Just got this from the ProtonMail team:

    > Dear ProtonMail user,

    Starting at around 4:30PM New York (10:30PM Zurich), Gmail suffered a global outage.

    A catastrophic failure at Gmail is causing emails sent to Gmail to permanently fail and bounce back. The error message from Gmail is the following:

    550-5.1.1 The email account that you tried to reach does not exist.

    This is a global issue, and it impacts all email providers trying to send email to Gmail, not just ProtonMail.

    Because Gmail is sending a permanent failure, our mail servers will not automatically retry sending these messages (this is standard practice at all email services for handling permanent failures).

    We are closely monitoring the situation. At this time, little can be done until Google fixes the problem. We recommend attempting to resend the messages to Gmail users when Google has fixed the problem. You can find the latest status from Google's status page:

    https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...

    Best Regards, The ProtonMail Team

  • by jonathanbull on 12/16/20, 12:20 AM

    CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week:

    https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=...

    That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

  • by wenbin on 12/15/20, 10:25 PM

    "Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser y128si147264pfg.177 - gsmtp"

  • by j-wags on 12/15/20, 10:39 PM

    Over the past 24 hours, I've had GitHub request that I re-verify my gmail three times (roughly 22 hours ago, 2 hours ago, and now), each time resetting my primary email's status to "Undeliverable" and "Unverified"

    The triggering event may be an email bounce. I get a lot of github notifications sent to my email, and the failure of just one/a few may trigger the reverification.

  • by firebaze on 12/15/20, 11:06 PM

    As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days.

    Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

  • by gpmcadam on 12/16/20, 9:05 AM

    People will suggest running your own mail server, and if you have the time and energy then definitely do that.

    But the next best thing you can do is simply just use your own domain. That way, you can at least decide to migrate your email elsewhere. Don't use the free domains you get from things like gmail or other providers, because then you have to _change_ your email address, and not just your MX records.

  • by crazypython on 12/16/20, 2:25 AM

    Remember, free and open source software is more reliable: (https://www.gnu.org/software/reliability.en.html)

    "One reason is that free software gets the whole community involved in working together to fix problems. Users not only report bugs, they even fix bugs and send in fixes. Users work together, conversing by email, to get to the bottom of a problem and make the software work trouble-free."

    And Service as a Software Substitute (SaaSS) takes away your freedom: (https://www.gnu.org/philosophy/who-does-that-server-really-s...)

    "The basic point is, you can have control over a program someone else wrote (if it's free), but you can never have control over a service someone else runs, so never use a service where in principle a program would do.

    With free software, we, the users, take back control of our computing. Proprietary software still exists, but we can exclude it from our lives and many of us have done so. However, we are now offered another tempting way to cede control over our computing: Service as a Software Substitute (SaaSS). For our freedom's sake, we have to reject that too.

    With SaaSS, the server operator can change the software in use on the server. He ought to be able to do this, since it's his computer; but the result is the same as using a proprietary application program with a universal back door: someone has the power to silently impose changes in how the user's computing gets done.

    Thus, SaaSS is equivalent to running proprietary software with spyware and a universal back door. It gives the server operator unjust power over the user, and that power is something we must resist."

  • by dannyw on 12/16/20, 1:48 AM

    Just mistakenly disputed a transaction because of this. I have been working with a mediocre freelancer who didn’t reply to my list of remediation items.

    When I followed up today, I got the “this email account does not exist” error from Gmail and proceeded to dispute on PayPal.

    I only found out through hacker news that this was a Gmail bug.

    Google, I expect a follow up retraction of the incorrect error messages. It’s one thing to give a temporary error. It’s another to say “this email account does not exist”.

  • by ttul on 12/16/20, 1:58 AM

    Email sender here (on behalf of about 2.5 million domains). We noticed the issue and mitigated it by transforming the 5xx error into a 4xx error so that messages to Google are re-queues instead of being permfailed. But even with this intervention, the ticket volume was insane...

  • by systemvoltage on 12/15/20, 10:36 PM

    Side note: Love the 2010-era web design. Wtf happened? Everything got bloated, buttons got huge and whitespace took over the screen real-estate.

  • by gramakri on 12/15/20, 11:22 PM

    If this had happened on self hosted email server, people would be claiming "this is why you don't self-host email". My company self hosts email and we have never had an outage of this sort. ever. All support emails to gmail accounts are now bouncing for us. It's also been more than 2 hours since this problem started.

  • by gjanezic on 12/15/20, 10:22 PM

    AWS SES just blocked our account due to high number of bounces :(

  • by cowkingdeluxe on 12/15/20, 10:17 PM

    Any email sent to my gmail is getting permanently bounced. Says my email does not exist at gmail.

  • by Havoc on 12/16/20, 12:37 AM

    Side note - why don't google status updates contain a timezone. You'd think that they'd have some awareness of you know...global business...many users in many timezones.

    > We will provide an update by 12/15/20, 10:30 PM

  • by philmcp on 12/16/20, 12:01 AM

    Anyone else having issues in general with "the internet" over the last few days?

    The images on different websites I've visited don't load (e.g. twitter, bbc etc). When they do they load, they load verrry slowly

  • by nathanyz on 12/15/20, 10:40 PM

    Unless it's a coincidence, I don't think this is just Gmail as I am seeing slow, failed networks connections to services in Google Cloud.

  • by reilly3000 on 12/16/20, 6:01 AM

    I'm glad to have found this thread, but its kept me up cleaning out thousands of "bounced" blocks from the last 2 day's suppression lists. They are both @gmail.com and gSuite/Workplace domains, and incidentally a ton of random blocks from Comcast as well from users that have months of open history.

    This blew away about 10% of our newsletter and marketing subscribers. I can't imagine the time you're having if you send to millions of Google accounts.

    Pardon me for being conspiratorial, but I have to say that the timing of these particular issues is of concern in light of the massive attacks the US government and others have faced this week. It seems adversaries would have plenty of fun if they got a new toy that could mess with the user/resource permission mapping at Google would want to use it to go after inboxes to do email confirmations. Even those this is 99% likely to be a DevOps chore that created an SRE nightmare, I'll allow myself believing 1% this was SecOps locking down huge swaths of accounts to mitigate a mass email verification attack.

  • by throw1234651234 on 12/15/20, 10:18 PM

    We are using a gmail inbox to process some business-critical emails in bulk. I guess those emails will be lost forever?

    Thankfully, we have backups, but we will have to move them to the inbox or elsewhere to have them processed.

    Edit: As an update, we usually have at least 100 emails come in every hour, and I am seeing none since 4:02 pm EST

  • by frlnBorg on 12/15/20, 10:39 PM

    github has un-verified my account and I'm unable to merge PRs, leave comments, etc... until it's verified again.

  • by blibble on 12/16/20, 12:13 AM

    workspace (gsuite) customers: remember to claim under the SLA, cos they don't award it automatically

    under 99.9% is a 3 day credit, and they're currently at 99.4%

  • by mapgrep on 12/16/20, 12:28 AM

    I use Helm for email. It’s a silent little server in my living room routed via Amazon (using a TLS cert that lives in my living room). I’d say about every three or four months it goes down for 5 minutes if I need to reboot my wifi router.

    I use it for privacy (am a fan) but I feel pretty smug knowing I’m getting better reliability too. At least this month :)

    No affiliation with the company.

    https://www.thehelm.com/

  • by dailypeeker on 12/15/20, 11:15 PM

    What is a status page that has the time without timezone

  • by guidoism on 12/16/20, 12:15 AM

    I've noticed references to my.name@google.com instead of my real address my.name@gmail.com — That doesn't seem good.

  • by dpeterson on 12/16/20, 6:02 AM

    LinkedIn telling me I could lose my linkedin account because they can't reach my gmail email address.

  • by cmbailey on 12/16/20, 12:19 PM

    One @gmail.com account shows no email received from 2:17pm ET Dec 15 to 6:05am ET Dec 16. I normally receive ~4 emails per hour, throughout the day and night. After I realized there might be a fault, I sent myself test messages around 6pm ET on Dec 15 and Gmail bounced them with the error:

    > 550-5.1.1 The email account that you tried to reach does not exist. Please try double-checking the recipient's email address for typos or unnecessary spaces. Learn more at https://support.google.com/mail/?p=NoSuchUser x62si100799otb.139 - gsmtp

  • by dimeatree on 12/15/20, 10:57 PM

    It has been fun explaining to customers that G-mail has been having issues.

  • by dkarp on 12/16/20, 10:21 AM

    Google Stadia was also down yesterday for a couple of hours[1]. It appeared to be related to loading user accounts into the game. Some games couldn't load at all, others worked for a short time and others were unaffected.

    [1] https://www.reddit.com/r/Stadia/comments/kdr2ps/its_not_just...

  • by globular-toast on 12/16/20, 8:26 AM

    I've been using Gmail since the invite only beta. In the past year I've had several occasions where important messages get somehow "lost" in the UI. The ones I've noticed are messages I've been waiting for, so I've gone and dug them out of "all mail" in most cases. They just wouldn't show in the normal inbox despite not being flagged as spam.

    This outage has now convinced me I need a new email provider.

  • by izgzhen on 12/16/20, 3:46 AM

    It seems that Gmail is no longer a priority of Google, which is actually reasonable. Email provider is hardly a growing business anymore, and the market is also full of small players. How much gmail user will bother to transfer to other services? Even paying ones? I think minimal. When will Gmail be irrelevant? Probably when email is irrelevant.

  • by noemit on 12/15/20, 11:43 PM

    is there a product you could use, like a cache layer for gmail? An investor I know would love that product. I understand things go down, but it's actually quite a pain to lose access to your existing emails and calendar events. Could he just use a Mail app like Outlook? Yes. But that's too complicated to set up I think.

  • by e15ctr0n on 12/15/20, 10:49 PM

    SMTP seems to be down as well. "Login to server smtp.gmail.com failed." using an email client

    IMAP seems to be working, though.

  • by bluecalm on 12/16/20, 7:59 AM

    It seems it's even worse for gmail to Gmail communication. Some of our company emails disappeared without any kind of bump information which causes a lot of frustration among our customers. I hope they fix it soon, it's a mess of massive proportion.

  • by sorokod on 12/16/20, 12:33 AM

    Gmail service details: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...

    All good now apparently

  • by dengsauve on 12/15/20, 10:16 PM

    We are seeing a lot of email bounced back, but otherwise usable.

  • by ssn on 12/16/20, 9:18 AM

    Still having error trying to fetch email from one Gmail account to another via POP3. Seem different from the error described in the post link.

  • by layoutIfNeeded on 12/16/20, 2:09 AM

    This wouldn’t have happened with Google Wave.

  • by chevman on 12/15/20, 10:26 PM

    Started sometime earlier today - we got alerted by Salesforce deliverability team around 11am CST.

  • by lostmsu on 12/16/20, 12:34 AM

    Imagine surprise reading this after a whole day of productive freelance-related email exchanges that went seemingly without issues, and wondering if all important stuff was received.

    By the end of the thread I wondered why it was not affected, until I remembered my small business email is actually on Exchange 365 :-D

  • by xianwen on 12/16/20, 12:29 AM

    Is Microsoft's Live Mail (Outlook.com) more reliable than Gmail?

  • by anuraj on 12/16/20, 11:39 AM

    COVID seems to have possessed Google - Services are down, Android Apps that used to take 24 hours for approval are now taking ages - and as usual no support forthcoming. Is Google unravelling?

  • by OpuRahman on 12/16/20, 8:54 AM

    In Bangladesh we faced serious problems last night.

  • by rangoon626 on 12/16/20, 3:57 AM

    The less google services we use, the better

  • by paulie_a on 12/16/20, 12:17 AM

    Google assistant is also having troubles

  • by haint_ on 12/16/20, 6:04 AM

    Will there be a public post mortem?

  • by tsg on 12/16/20, 7:56 AM

    Now showing as fixed fwiw.

  • by notabul on 12/16/20, 4:16 AM

    What should I use instead?

  • by aerosmile on 12/15/20, 11:58 PM

    Resolved as of 3:51 PST.

  • by gregjotau on 12/15/20, 10:36 PM

    Same issue here

  • by kalvisk on 12/15/20, 11:11 PM

    They are getting ready to be acquired by Microsoft :D