from Hacker News

Heroku CI and Review App Secrets Compromised

by himeexcelanta on 5/18/22, 12:48 AM with 95 comments

• by bradleybuda on 5/18/22, 1:16 AM

  Text of the email:

  At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an issue affecting your account. Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved.

  As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022.

  As a result, any secrets you set in Review Apps and Heroku CI config vars may have been compromised and should be rotated. In addition, any Heroku tokens stored in these pipeline config vars would potentially have allowed access to your Heroku account between April 7, 2022 and May 5, 2022, when your passwords were reset, invalidating all Heroku tokens as a result.

  Please note, these pipeline-level config vars are different from standard app config vars. App config vars were not stored in this database and we have no evidence to suggest app config vars were compromised.

• by hkhanna on 5/18/22, 2:16 AM

  I spent the last two days migrating my company to Render from Heroku, and now I'm glad I did. Render is a little rough around the edges; Heroku is far more polished.

  But it's probably to Render's credit that, in my opinion, the most annoying thing about Render is that it's impossible to google about Render because "render" is such a common word in the tech world!

  Their support is good and responsive, and the developer experience was good enough. It has some warts, and there were definitely times I missed Heroku, but their speed of improvement gives me confidence in their future.

  Sad to leave Heroku after almost a decade with them. They were far ahead of their time.

• by hthrowaway5 on 5/18/22, 1:11 AM

  Yep, they outright lied about env vars. Incredible.

  It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now.

  They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situation.

  If you're still on it, make your plans to move away now. Time is ticking until a major outage or another security incident like this one. See my comment history and related threads for more. Specifically this summary: https://news.ycombinator.com/item?id=31374048

• by mepiethree on 5/18/22, 12:53 AM

  11 days ago they said "While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets."

  I guess that was a lie?!

• by zevir on 5/30/22, 2:47 PM

  For those looking for a great alternative to Review Apps - Livecycle is great (https://livecycle.io/). It offers the automated per-PR ephemeral environment and much more. It also includes a rich layer of built-in collaboration and annotation tools that allow all collaborators to join the PR review and leave their comments visually, on top of the product UI. The comments are maintained in Livecycle and also synced back to Git as review comments so that developers can see the issues faster, understand them better and address them sooner. There are setup templates that make it easy to simply copy over your docker file and get started within a few minutes. And the team is eager to help if you have any questions or issues.

• by itsmeste on 5/18/22, 1:42 PM

  When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software.

  Why? Commercialism.

  Founders sell to the highest bidder to make their exit worthwhile for themselves, not caring about the future of the product (and customers).

  It's a no-brainer that a commercial company like Salesforce (it's in their name!) doesn't have what it takes to build AAA software, but focuses on maximizing their profit. They drove away their best staff, focused on the wrong features, and are seemingly overwhelmed by maintaining their purchased software, all while probably not even realizing their demise.

  We should all come to the agreement that takeovers of fundamental software by incompetent companies should be seen as a hostility towards every current user of said software.

• by nameless912 on 5/18/22, 1:48 AM

  Yup, that's game, set, and match. I really feel bad for all the herokai still left holding the line, but damn am I glad I got out when I did.

• by jamespetercook on 5/18/22, 6:55 AM

  Slightly off-topic, but can anyone tell me how you’d know that your database has been accessed by a threat actor? Should I be periodically reviewing all my logs for something unusual?

• by Mandatum on 5/18/22, 1:50 AM

  Sounds like a customer's canary token triggered this based on the current reporting.

  "Trust is our Number 1 value."

• by gault8121 on 5/18/22, 2:36 AM

  Has anyone done a load test comparison for Heroku vs. Render.com? The "Pro Ultra" on Render is $450/month for 32 GB RAM + 8 CPU. The Heroku Performance L Dyno is $500 a month for 14 GB RAM. The Render server seems like a much better offering.

• by kaycebasques on 5/18/22, 3:08 AM

  Any guesses as to how the "threat actor" got access to the databases? I understand most guesses would be conjecture (unless someone here has an inside scoop). Just curious about how stuff like this usually gets compromised.

• by daudmalik06 on 5/24/22, 10:47 PM

  This is why we at vulert.com never access the customer's codebase or any installation, who doesn't know vulert, it's a service that notifies you for security issues in your software dependencies.

• by heartbreak on 5/18/22, 2:23 AM

  I completely stopped getting their update emails when I deleted all of my running apps. This is the second one I've seen on HN that I haven't received (though I received the others).

  Is the impact limited to specific customer accounts, or are they just not updating me anymore?

• by boesboes on 5/18/22, 7:02 AM

  Still havent had any notification on this. A+ for Heroku

• by vmception on 5/18/22, 1:39 AM

  oh shit. I'm surprised we haven't heard about major services getting hacked to oblivion right now, so much is stored in environment variables

  are there any mystery hacks occurring yet?

  is this database known to have been spread anywhere?

• by oxff on 5/18/22, 3:00 AM

  My sides, what a shitshow.