from Hacker News

AWS injected bad code into customer run Java apps

by prashantrajan on 12/22/21, 1:46 AM with 2 comments

• by prashantrajan on 12/22/21, 1:46 AM

  AWS decided it was a good idea to push out a "security" patch late on a Friday (Dec 17) without advance warning that monkey patched running customer owned Java code.

  The patch is auto applied on Amazon Linux AMIs at boot time since it's marked as a critical update causing Java web apps to fail. This caused all our auto scaling processes to fail. Note that the code is injected even in customer re-bundled AMIs of Amazon Linux because it attaches itself as a hard dependency of the JDK and gets applied as a JDK upgrade if you opted into "critical" OS security updates.

  In their recklessness to rush out this change thinking they know all the ways Java apps have been built over the last 30 years they've likely caused users to now opt out of their automatic security updates (https://aws.amazon.com/amazon-linux-ami/faqs/#:~:text=Q%3A%2...).

  Their first and only announcement of this kind was done via https://alas.aws.amazon.com/announcements/2021-001.html (no email or anything) and fails to mention the critical fact that it gets applied to previously baked AMIs.

  AWS has long left the customer to manage their own environment within AWS and this approach to security patching in a non standard way (monkey patching user written code) is a betrayal of that trust and policy.

• by prashantrajan on 12/22/21, 3:51 AM

  It's infuriating that they pushed out this breaking change late on a Friday, screwing over their customers and all our enterprise account managers are now conveniently out on their holiday break.

  5 days later and they are still "investigating" instead of rolling back the change.

  Lesson: Don't use Amazon Linux. Pick an OS with mature stewardship like Ubuntu/Debian/RedHat