by vorador on 5/4/22, 12:36 AM with 168 comments
"As part of our efforts to enhance our security and in response to an incident published on status.heroku.com, we wanted to inform you that we will begin resetting user account passwords on May 4, 2022. We recommend that you reset your user account password in advance here and follow the best practices below:
Minimum of 16 characters Minimum complexity of 3 out of 4: Uppercase, Lowercase, Numeric, Symbol Don't just add a letter or a 1 digit number to the existing password while changing Passwords may not be duplicated across accounts If you do not reset your password and your user account password is reset by Heroku on May 4, 2022, your existing password will no longer work. To log in to Heroku, you must reset your password by accessing the Heroku login page and clicking the "Forgot your password?" link . Please be aware that you may be required to reset your passwords again in the future. "
by jarcoal on 5/4/22, 12:53 AM
For those of you that haven't been following, Heroku has been adding non-update updates to this security thread over the last couple of weeks, which began with the announcement that some (or maybe all) of their GitHub granted access tokens had been compromised: https://status.heroku.com/incidents/2413
Now, weeks later, we're hearing that all account passwords are being reset, and for some reason if you have been using an HTTPS-style log drain that you should reset any secrets related to it as well.
Heroku needs to come out and clearly state what they know about this situation, and more importantly what they don't know -- which is starting to sound like the answer is "a lot". It's not even clear they know how this all happened -- whatever door was left open might still be open. So if you've gone and rotated all of your application secrets (which you probably should do), be prepared to rotate them again when this is all over.
by ryanSrich on 5/4/22, 3:02 AM
The initial issue was supposedly a breach of customer repositories. Which sounds bad, but if you’re not storing credentials in code, then the worst case is that a potential hacker had access to download your code. Not great, but certainly not as catastrophic as some breaches have been.
Since then, Heroku has been acting beyond strange. Everyday they update the incident with essentially the same non-update, but written differently, with vague references to the same information they’ve sent 14 days in a row.
Now, they send another very ominous and strange update about “some” customers having their passwords reset. However, based on this thread and my own experience, it seems like every customer is getting this message.
What does this have to do with the initial issue? Were actual Heroku accounts compromised?
This behavior is either extreme incompetence wrt customer communication, or they’re preparing to announce a truly insane breach that may include everyone that has ever used Heroku.
They need to get their shit together and quickly.
by eastern on 5/4/22, 3:41 AM
I received this email. The reset password link in it is NOT https. If I manually change the http to https it turns out that the server, click.msg.salesforce.com, is returning a certificate that is only valid for click.s10.exacttarget.com
by jmbpiano on 5/4/22, 12:45 AM
This concerns me. How are they checking that no other account has the same password? Wouldn't that imply a strong possibility they're either hashing with the same salt across all accounts or not hashing at all?
Hopefully some of the smarter folks here can tell me why I'm way off base...
by dml2135 on 5/4/22, 3:16 AM
I'm at a startup that's been on Heroku since its inception and this breach has got my team thinking about moving to AWS for the first time, even if our scale doesn't necessarily demand it yet.
Does anyone that has done this transition in the past have any advice?
by ezekg on 5/4/22, 2:11 AM
by mgomez on 5/4/22, 12:53 AM
by collectedparts on 5/4/22, 2:20 AM
by janejeon on 5/4/22, 3:49 AM
I've went ahead and just deleted the whole account - it's at the point where I don't want to do business with this company, full stop.
by stevenicr on 5/4/22, 3:30 AM
I sent heroku an email about a week or bit more ago - complaining that I had gotten a very well crafted phishing email - which was seriously very clever in it's timing (or just blind lucky for them, as I was changing servers and dns and things within 48 hours of this) - that made it look I needed to login and do something with server credentials or something similar.. and the link to click to fill info went to a heroku app with a name attached to it..
So I emailed them and asked them to look into it to see if it was overly obvious that user XY was getting clicks and if they had set up some sort of official looking form to phish server accounts or something like that - memory a bit foggy in the midst of so many changes past couple weeks.
Never heard back from them, but maybe my email helped them realize some things were not being used as they should of been authorized to be used for. Maybe not.
by vr46 on 5/4/22, 12:55 AM
by ab-dm on 5/4/22, 2:58 AM
This whole things has been an absolute s*tshow. Now I'm worried that there is a much bigger issue at play, and we might actually be in a bit of trouble.
by urthor on 5/4/22, 5:29 AM
As a general thing, I like the idea of a "swift, well written" email to your entire userbase saying "please reset your password, now."
That seems like a perfectly reasonably response, and request of your user-base.
by 8organicbits on 5/4/22, 8:34 AM
But at password generation time, it's much easier. You don't need to tell the user all these rules about length, complexity, reuse, etc. The server generates the password in a way that's sufficient, and the user just saves it. It also avoids the mess of password manager generated passwords not meeting some criteria, or passwords meeting client side checks but failing server side checks.
When you've got so many sites asking for passwords >=10, >=16 characters in length, is anyone actually able to memorize a truely unique password per-site any more?
Besides, a server generated password has known entropy, so you can model the security much more accurately. This push for really long passwords is to guard against low entropy passwords that users create, although users seem to respond predictably: [password] too short? I'll try [password]123 or [password][password].
by ChrisArchitect on 5/4/22, 1:55 AM
(2) just saw that the password reset will invalidate all API tokens. Arrrrggh. Annoying!
by ergocoder on 5/4/22, 1:09 AM
And heroku doesn't seem to acknowledge that at all.
by Mandatum on 5/4/22, 3:37 AM
by tialaramex on 5/4/22, 8:07 AM
Because passwords are shared secrets, the relying party (here Heroku) can lose them and then will predictably act like somehow this is your problem when it happens, see also "Identity theft" aka "We're incompetent and our money got stolen but it's convenient to pretend that's actually your money and ask when you'll fix it".
Use WebAuthn. If (when) outfits like Heroku incompetently lose the credentials nothing interesting happens.
by bpeebles on 5/5/22, 3:31 AM
> On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.
...
> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.
by alephnan on 5/4/22, 2:49 AM
Hobbyists and Developers who are looking such a platform should expect they are least technical to solving harder software problems then they are, and that’s worth paying the premium for.
May the 4th be with you.
by antoniuschan99 on 5/4/22, 5:22 AM
NOTE: A password reset will also invalidate your API access tokens. As a result, any automations you’ve built to integrate with the Heroku Platform API that use these tokens may result in 403 forbidden errors . To avoid downtime you will need to re-enable direct authorizations by following the instructions here and update your integrations to use your newly generated token.
by onphonenow on 5/4/22, 3:26 AM
Do they not have any fraud screening or rate limiting on these things, 2FA on unknown devices or something?
I've had my same password forever (I'm old) on google without issue) that is scary short. However, when I login from a new device or every 30 days or go to an admin / security page it asks me to stick my yubico key into my computer. Works fine.
Is this same password used for high volume API stuff without anything else? Normally that's with an access token type system (at least on AWS). Those, yes, should be highly complex. The AWS access / secret key combo is ridiculous frankly in length (mixed case + numbers + symbols) * 40.
by MobileVet on 5/4/22, 2:21 AM
Was this a system wide event?
We have been on Heroku for ~5 years and the last 18 months have been far less reliable. Sad.
by NoXero on 5/4/22, 12:47 PM
by User123987 on 5/4/22, 4:32 PM
hmm, how do they know if those passwords are same across accounts?
Either they save them in plaintext or they can't check if I have same password as some other account.
by thomgo on 5/4/22, 1:21 AM
by tosh on 5/4/22, 6:13 PM
by teaearlgraycold on 5/4/22, 2:11 AM
by dz0ny on 5/4/22, 2:29 PM