from Hacker News

Backdoor in upstream xz/liblzma leading to SSH server compromise

by rkta on 3/29/24, 4:16 PM with 1849 comments

  • by rwmj on 3/29/24, 4:55 PM

    Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.

    He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.

  • by move-on-by on 3/29/24, 5:18 PM

    Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.

    > If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released.

    Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.

  • by Aissen on 3/29/24, 6:17 PM

    Looks like one of the backdoor authors even went and disabled the feature the exploit relied on directly on oss-fuzz to prevent accidental discovery: https://social.treehouse.systems/@Aissen/112180302735030319 https://github.com/google/oss-fuzz/pull/10667

    But luckily there was some serendipity: "I accidentally found a security issue while benchmarking postgres changes." https://mastodon.social/@AndresFreundTec/112180083704606941

  • by arp242 on 3/29/24, 4:58 PM

    I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

    Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor.

    Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.

  • by bonyt on 3/29/24, 4:58 PM

    For those panicking, here are some key things to look for, based on the writeup:

    - A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.

    - A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux.

    - Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.

    Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

    It is possible there are other flaws or backdoors in liblzma5, though.

  • by Epa095 on 3/30/24, 8:16 AM

    I hope Lasse Collin is doing OK! Here is a older message from him [1]

    "I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

    It's also good to keep in mind that this is an unpaid hobby project. "

    Github (Microsoft) are in a unique position to figure out if his account is hacked or not, and find a way to reach him. I hope they reach out and offer him some proper support! Economic support (if that's needed), or just help clearing his name.

    This is another tale of how we are building multi trillion dollar industries on the back of unpaid volunteers. It's not github 'job', and many other organisations have benefited even more from Lasses work, but they are in a unique position, and would be literally pocket change for them.

    1:https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

  • by returningfory2 on 3/29/24, 5:22 PM

    A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz

    About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.

  • by cf100clunk on 3/29/24, 4:40 PM

      I am *not* a security researcher, nor a reverse engineer.  There's lots of
      stuff I have not analyzed and most of what I observed is purely from
      observation rather than exhaustively analyzing the backdoor code.
    I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.
  • by dang on 3/29/24, 10:58 PM

    Related ongoing threads:

    Xz: Disable ifunc to fix Issue 60259 - https://news.ycombinator.com/item?id=39869718

    FAQ on the xz-utils backdoor - https://news.ycombinator.com/item?id=39869068

    Everything I Know About the XZ Backdoor - https://news.ycombinator.com/item?id=39868673

  • by 0xthr0w4 on 3/29/24, 8:31 PM

    Out of curiosity I looked at the list of followers of the account who committed the backdoor.

    Randomly picked https://github.com/Neustradamus and looked at all their contributions.

    Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago.

    [0] https://github.com/microsoft/vcpkg/issues/37197

    [1] https://github.com/microsoft/vcpkg/pull/37199

  • by perihelions on 3/29/24, 5:29 PM

    Imagine a more competent backdoor attempt on xz(1)—one that wouldn't have been noticed this quickly. xz is everywhere. They could pull off a "reflections on trusting trust": an xz which selectively modifies a tiny subset of the files it sees, like .tar.xz software tarballs underlying certain build processes. Not source code tarballs (someone might notice)—tarballs distributing pre-compiled binaries.

    edit to add: Arch Linux' entire package system used to run on .tar.xz binaries (they switched to Zstd a few years ago [0]).

    [0] https://news.ycombinator.com/item?id=19478171 ("Arch Linux propose changing compression method from xz to zstd (archlinux.org)")

  • by pfortuny on 3/29/24, 6:02 PM

    Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.
  • by the_errorist on 3/30/24, 1:57 PM

    Looks like Lasse Collin has commented on LKML: https://lkml.org/lkml/2024/3/30/188

    Also, some info here: https://tukaani.org/xz-backdoor/

  • by bawolff on 3/29/24, 4:54 PM

    The terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems.

    Makes you wonder what more competent actors can do.

  • by gmnon on 3/29/24, 6:50 PM

    Funny how Lasse Collin started to ccing himself and Jia Tan from 2024-03-20 (that was a day of tons of xz kernel patches), he never did that before. :)

    https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...

  • by vhiremath4 on 3/29/24, 8:50 PM

    My favorite part was the analysis of "I'm not really a security researcher or reverse engineer but here's a complete breakdown of exactly how the behavior changes."

    You only get this kind of humility when you're working with absolute wizards on a consistent basis.

  • by asveikau on 3/29/24, 4:42 PM

    That's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.
  • by Decabytes on 3/30/24, 12:34 PM

    So when are we going to stop pretending that OSS maintainers/projects are reaping what they sow when they "work for free" and give away their source code away using OSS licensed software, while large companies profit off of them? If they were paid more (or in some cases even actually paid), then they could afford to quit their day jobs, reducing burn out, they could actually hire a team of trusted vetted devs instead of relying on the goodwill of strangers who step up "just to help them out" and they could pay security researchers to vet their code.

    Turns out burned out maintainers are a great attack vector and if you are willing to play the long game you can ingratiate yourself with the community with your seemingly innocuous contributions.

  • by thesnide on 3/29/24, 4:57 PM

    The discussion to upload it to Debian is interesting on its own https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
  • by rpigab on 3/29/24, 8:52 PM

    I'd love to be at Microsoft right now and have the power to review this user's connection history to Github, even though VPN exists, many things can be learned from connection habits, links to ISPs, maybe even guess if VPNs were used, roundtrip time on connections can give hints.

    I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

  • by alright2565 on 3/29/24, 5:33 PM

    https://github.com/tukaani-project/tukaani-project.github.io...

    > Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored.

    The author was thinking ahead! Latest commit hash for this repo: 8a3b5f28d00ebc2c1619c87a8c8975718f12e271

  • by kzrdude on 3/30/24, 8:49 AM

    Jia Tan "cleaned up" in all their ZSTD branches some hours ago, probably hiding something https://github.com/JiaT75/zstd/branches/all
  • by zh3 on 3/30/24, 8:43 AM

    Comment from Andres Freund on how and why he found it [0] and more information on the LWN story about the backdoor. Recommend people read this to see how close we came (and think about what this is going to mean for the future).

    [0] https://lwn.net/Articles/967194/

  • by dhx on 3/30/24, 9:18 AM

    A mirror of the offending repository created by someone else is available at [1]. GitHub should be keeping the evidence in the open (even if just renamed or archived in a safer format) instead of deleting it/hiding it away.

    The offending tarball for v5.6.1 is easier to find, an example being.[2]

    m4/.gitignore was updated 2 weeks ago to hide build-to-host.m4 that is only present in the release tarball and is used to inject the backdoor at build time.[3]

    [1] https://git.phial.org/d6/xz-analysis-mirror

    [2] https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz

    [3] https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0...

  • by xyzzy_plugh on 3/30/24, 1:49 AM

    This gist summarizes the current situation very well: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...

    Definitely looking like they were most likely some sort of state actor. This is very well done and all in plain sight. It's reassuring that it was discovered but given a simple audit of the release build artifacts would have raised alarms, how prevalent is this behavior in other projects? Terrifying stuff.

  • by wood_spirit on 3/29/24, 5:43 PM

    A lot of eyes will be dissecting this specific exploit, and investigating this specific account, but how can we find the same kind of attack in a general way if it’s being used in other projects and using other contributor names?
  • by q3k on 3/29/24, 4:43 PM

    NixOS/Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd/liblzma, and the backdoor doesn't get configured in (only happens on .deb/.rpm systems).
  • by bhaak on 3/29/24, 8:00 PM

    I looked at the differences between the GitHub repository and released packages. About 60 files are in a release package that are not in the repo (most are generated files for building) but also some of the .po files have changes.

    That's devastating.

    If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.

  • by colanderman on 3/29/24, 4:58 PM

    The latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail:

    https://github.com/tukaani-project/xz/commit/af071ef7702debe...

    Not sure what to make of this.

  • by 20after4 on 3/29/24, 5:12 PM

    > "Docs: Simplify SECURITY.md."

    https://github.com/tukaani-project/xz/commit/af071ef7702debe...

    Removes instructions about details relevant to security reports. Heh, nice one.

  • by Tenobrus on 3/29/24, 5:51 PM

    It looks like the person who added the backdoor is in fact the current co-maintainer of the project (and the more active of the two): https://tukaani.org/about.html
  • by CGamesPlay on 3/30/24, 1:24 AM

    Why has Github disabled the (apparently official) xz repository, but left the implicated account open to the world? It makes getting caught up on the issue pretty difficult, when GitHub has revoked everyone's access to see the affected source code.

    https://github.com/tukaani-project/xz vs https://github.com/JiaT75

  • by 5p4n911 on 3/29/24, 8:11 PM

    The author (Jia Tan) also changed the xz.tukaani.org (actually the github.io, where the main contributor is, surprise, also them) release description to state all new releases are signed by their OpenPGP key. I'd guess that was one of the first steps to a complete project takeover.

    I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.

  • by weinzierl on 3/29/24, 6:23 PM

    The backdoor is not in the C source directly, but a build script uses data from files in the test dir to only create the backdoor in the release tars. Did I summarize that correctly?
  • by elchief on 3/29/24, 7:48 PM

    "Amazon Linux customers are not affected by this issue, and no action is required. AWS infrastructure and services do not utilize the affected software and are not impacted. Users of Bottlerocket are not affected."

    https://aws.amazon.com/security/security-bulletins/AWS-2024-...

  • by liveoneggs on 3/29/24, 7:41 PM

    The best part is everyone disabling security tests that started failing
  • by ozgune on 3/29/24, 9:13 PM

    I read through the entire report and it gradually got more interesting. Then, I got to the very end, saw Andres Freund's name, and it put a smile on my face. :)

    Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?

  • by jaromilrojo on 3/30/24, 1:40 PM

    This is another proof that systemd is an anti-pattern for security: with its crawling and ever growing web of dependencies, it extends the surface of vulnerability to orders of magnitude, and once embraced not even large distro communities can defend you from that.

    A malware code injection in upstream xz-tools is a vector for remote exploitation of the ssh daemon due to a dependency on systemd for notifications and due to systemd's call to dlopen() liblzma library (CVE-2024-3094). The resulting build interferes with authentication in sshd via systemd.

  • by sn on 3/30/24, 1:07 AM

    For bad-3-corrupt_lzma2.xz, the claim was that "the original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files." with no indication of what the seed was.

    I got curious and decided to run 'ent' https://www.fourmilab.ch/random/ to see how likely the data in the bad stream was to be random. I used some python to split the data into 3 streams, since it's supposed to be the middle one that's "bad":

    I used this regex to split in python, and wrote to "tmp":

        re.split(b'\xfd7zXZ', x)
    
    I manually used dd and truncate to strip out the remaining header and footer according to the specification, which left 48 bytes:

        $ ent tmp2 # bad file payload
        Entropy = 4.157806 bits per byte.
        
        Optimum compression would reduce the size
        of this 48 byte file by 48 percent.
        
        Chi square distribution for 48 samples is 1114.67, and randomly
        would exceed this value less than 0.01 percent of the times.
        
        Arithmetic mean value of data bytes is 51.4167 (127.5 = random).
        Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
        Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0).
        
        $ ent tmp3 # urandom
        Entropy = 5.376629 bits per byte.
        
        Optimum compression would reduce the size
        of this 48 byte file by 32 percent.
        
        Chi square distribution for 48 samples is 261.33, and randomly
        would exceed this value 37.92 percent of the times.
        
        Arithmetic mean value of data bytes is 127.8125 (127.5 = random).
        Monte Carlo value for Pi is 3.500000000 (error 11.41 percent).
        Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0).
        
    
    The data does not look random. From https://www.fourmilab.ch/random/ for the Chi-square Test, "We interpret the percentage as the degree to which the sequence tested is suspected of being non-random. If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random. If the percentage is between 99% and 95% or between 1% and 5%, the sequence is suspect. Percentages between 90% and 95% and 5% and 10% indicate the sequence is “almost suspect”."
  • by 0x0 on 3/29/24, 11:24 PM

    All these older (4.x, 5.0.x etc) releases that were suddenly uploaded a few months ago should probably also be considered suspect: https://github.com/tukaani-project/tukaani-project.github.io...
  • by kn100 on 3/30/24, 5:47 AM

    Here's a handy bash script I threw together to audit any docker containers you might be running on your machine. It's hacky, but will quickly let you know what version, if any, of xz, is running in your docker containers.

    ``` #!/bin/bash

    # Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}")

    # Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container")

        # Execute xz --version inside the container
        version=$(docker exec "$container" xz --version)
    
        # Write container name, image, and command output to a text file
        echo "Container: $container" >> docker_container_versions.txt
        echo "Image: $image" >> docker_container_versions.txt
        echo "xz Version:" >> docker_container_versions.txt
        echo "$version" >> docker_container_versions.txt
        echo "" >> docker_container_versions.txt
    
    done

    echo "Output written to docker_container_versions.txt" ```

  • by Roark66 on 3/30/24, 9:35 AM

    Sadly this is exactly one of the cases where open source is much more vulnerable to a state actor sponsored attack than proprietary software. (it is also easier to find such backdoors in OS software but that's BTW)

    Why? Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he. Also they have to be hiring in the right team etc. Your operative has to be in a different country, needs a CV that checks out, passports/ids are checked etc.

    But to contribute to an OS project? You just need an email address. Your operative sends good contributions until they build trust, then they start introducing backdoors in the part of the code "no one, but them understands".

    The cost of such attack is a lot lower for a state actor so we have to assume every single OS project that has a potential to get back doored had many attempts of doing so. (proprietary software too, but as mentioned, this is much more expensive)

    So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.

  • by Scaevolus on 3/29/24, 5:55 PM

    It's wild that this could have laid dormant for far longer if the exploit was better written-- if it didn't spike slow down logins or disturb valgrind.
  • by lpapez on 3/29/24, 4:47 PM

    So many security companies publishing daily generic blog posts about "serious supply chain compromises" in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise.

    I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

  • by gouggoug on 3/29/24, 8:39 PM

    List of pull request requesting the updating to liblzma 5.6.0 [0]

    I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under.

    [0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests

  • by snickerer on 3/30/24, 5:14 PM

    When I search for "digital masquerade" on Google, the first result is a book with this title from the author Jia Tan. I assume that is how the attackers got their fake name. Or they think using this author's name is a joke.
  • by dlenski on 3/29/24, 11:14 PM

    A lot of software (including https://gitlab.com/openconnect/openconnect of which I'm a maintainer) uses libxml2, which in turn transitively links to libzma, using it to load and store compressed XML.

    I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML…

    But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.

  • by afh1 on 3/30/24, 12:25 AM

    Potentially malicious commit by same author on libarchive: https://github.com/libarchive/libarchive/pull/1609
  • by youainti on 3/29/24, 4:42 PM

    Summary: "The upstream xz repository and the xz tarballs have been backdoored."

    It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.

  • by Randalthorro on 3/30/24, 11:09 AM

    Since GitHub disabled the repos.. I uploaded all GitHub Events from the two suspected users and from their shared project repo as easy to consume CSV files:

    https://github.com/emirkmo/xz-backdoor-github

    For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)

  • by yogorenapan on 3/29/24, 4:40 PM

    Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries
  • by ParetoOptimal on 3/29/24, 5:08 PM

    If you have a recently updated NixOS unstable it has the affected version:

        $ xz --version
        xz (XZ Utils) 5.6.1
        liblzma 5.6.1
    
    EDIT: I've been informed on the NixOS matrix that they are 99% sure NixOS isn't affected, based on conversations in #security:nixos.org
  • by mik1998 on 3/29/24, 9:17 PM

    Personally, I use lzip ever since I read https://www.nongnu.org/lzip/xz_inadequate.html Seems like the complexity of XZ has backfired severely, as expected.
  • by buildbot on 3/29/24, 4:43 PM

    This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?

    Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)

  • by pdimitar on 3/30/24, 8:06 AM

    This was only a matter of time. Open source projects are under-staffed, maintainers are overworked and burned out, and everyone relies on the goodwill of all actors.

    Obviously a bad actor will make use of these conditions and the assumption of good will.

    We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.

  • by devttyeu on 3/30/24, 12:55 AM

    Wouldn’t be surprised that the ssh auth being made slower was deliberate - that makes it fairly easy to index all open ssh servers on the internet, then to see which ones get slower to fail preauth as they install the backdoor
  • by bananapub on 3/29/24, 8:58 PM

    people are mis-reading the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

    it wasn't the apparently newly-created identity "Hans Jansen" just asking for a new version to be uploaded, it was "Hans Jansen" providing a new version to be uploaded as a non-maintainer-upload - Debian-speak for "the maintainer is AWOL, someone else is uploading their package". if "Hans Jansen" is another attacker then they did this cleverly, providing the new - compromised - upstream tarballs in an innocent-looking way and avoiding anyone examining the upstream diff.

  • by userbinator on 3/29/24, 9:44 PM

    Looking at how many requests to update to the backdoored version have been made, I wonder if the fact that many people (including developers) have been conditioned to essentially accept updates as "always-good" is a huge contributing factor in how easy it is to spread something like this.

    The known unknowns can be better than the unknown unknowns.

  • by A1kmm on 3/30/24, 1:06 AM

    Looks like GitHub has suspended access to the repository, which while it protects against people accidentally compiling and using the code, but certainly complicates forensic analysis for anyone who doesn't have a clone or access to history (which is what I think a lot of people will be doing now to understand their exposure).
  • by multimoon on 3/29/24, 6:31 PM

    It seems like based on the (very well written) analysis that this is a way to bypass ssh auth, not something that phones out which would've been even scarier.

    My server runs arch w/ a LTS kernel (which sounds dumb on the surface, but was by far the easiest way to do ZFS on Linux that wasn't Ubuntu) and it seems that since I don't have SSH exposed to the outside internet for good reason, and my understanding is Arch never patched shhd to begin with that I and most people who would be in similar situations to me are unaffected.

    Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.

  • by 0x0 on 3/29/24, 11:18 PM

    Interesting commit in January where the actual OpenPGP key was changed: https://github.com/tukaani-project/tukaani-project.github.io...
  • by mrbluecoat on 3/30/24, 12:21 AM

    > I am *not* a security researcher, nor a reverse engineer.

    Could have fooled me - impressive write-up!

  • by secondary_op on 3/30/24, 5:18 AM

    Github making suspect repository private and hiding recent account activity is wrong move and is interfering with citizens investigation efforts.
  • by londons_explore on 3/29/24, 5:42 PM

    I think the lesson here for packagers is that binary testdata should not be present while doing the build.

    It is too easy to hide things in testdata.

  • by pushedx on 3/29/24, 7:08 PM

    Mirror of the report, since the Openwall servers appear to be down.

    https://web.archive.org/web/20240329182300/https://www.openw...

  • by nolist_policy on 3/30/24, 3:43 PM

    Debian is considering that their infrastructure may be compromised[1].

    [1] https://fulda.social/@Ganneff/112184975950858403

  • by fourfour3 on 3/29/24, 4:47 PM

    Looks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.
  • by rossant on 3/30/24, 7:52 AM

    Incredible. It's like discovering your colleague for 2 years at the secret nuclear weapon facility is a spy for another country, covering his tracks until the very last minute. Feels like a Hollywood movie is coming up.

    Should we start doing background checks on all committers to such critical IT infrastructure?

  • by Luker88 on 3/29/24, 6:01 PM

    @people who write github scanners for updates and security issues (dependabot and the like)

    Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)?

    I can't track them and I don't want them in my projects.

    Might not be very helpful as it is easy to create new identities, but I see no reason to make it easier for them. Also, I might approach differently someone with lots of contributions to known projects than a new account, so it still helps.

  • by ikekkdcjkfke on 3/29/24, 7:48 PM

    Github should probably remove the dopamine hits of green checkmarks etc. like in serious stock broker apps
  • by 8organicbits on 3/30/24, 9:25 AM

    There's good discussion of the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...
  • by dlachausse on 3/29/24, 4:36 PM

    > openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

    It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?

  • by notyoutube on 3/29/24, 5:25 PM

    Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?
  • by rasengan on 3/29/24, 4:37 PM

    > One portion of the backdoor is solely in the distributed tarballs. For easier reference, here's a link to debian's import of the tarball, but it is also present in the tarballs for 5.6.0 and 5.6.1:

    Ubuntu 22.04 version:

    dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library

    Whew!

  • by bagels on 3/29/24, 5:15 PM

    Is this a crime? Has anyone been prosecuted for adding a backdoor like this?
  • by dmarto on 3/29/24, 10:00 PM

    Kinda relevant, as I saw few comments about how safer languages are the solution.

    Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1].

    [0] - https://github.com/c-skills/rust1

    [1] - https://checkmarx.com/blog/over-170k-users-affected-by-attac...

  • by markus_zhang on 3/29/24, 7:24 PM

    Keeps one wonder how many similar backdoors are there in the wild. What is the best way to execute such a move? This is sophisticated enough, but not good enough to stay unnoticed for a long while. If I were a state actor I'd think about at least 6-12 months.
  • by kapouer on 3/29/24, 9:45 PM

    Both https://github.com/tukaani-project members accounts have been suspended. (to see that, you can list the followers of each account).
  • by oxymoron290 on 3/29/24, 8:01 PM

    Jai Tan's commit history on his github profile suggests he took off for Christmas, new years, and spring break. I smell an American.
  • by formerly_proven on 3/29/24, 4:58 PM

    Quite ironic: The most recent commit in the git repo is "Simplify SECURITY.md", committed by the same Github account which added the backdoor.

    https://github.com/tukaani-project/xz/commit/af071ef7702debe...

  • by bheadmaster on 3/31/24, 12:11 PM

    This is exactly why I fight the windmills so hard when it comes automatic updates in Linux software.

    So much damage is caused just by adding a single maintainer to a project - imagine how much power you would have to wield the remote execution systems put in place by naive developers for "automatic updates".

    All it takes is a single malicious maintainer given access to the new version update of some popular user software, and they have a new botnet of thousands of devices at their disposal. Better yet, after the backdoor installation, they can just release the real update and cover their tracks forever.

    Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.

  • by byearthithatius on 3/30/24, 2:41 AM

    I hope mainstream news cover this so the general population can understand the issue with our software ecoysystems reliance on unpaid open-source maintainers
  • by AdmiralAsshat on 3/29/24, 4:46 PM

    > Red Hat assigned this issue CVE-2024-3094.

    Does that mean this affects RHEL and Fedora?

  • by PedroBatista on 3/29/24, 5:12 PM

    Given the recent ( not so recent ) attacks/"bugs" I feel there is a need to do more than the already hard task of investigating and detecting attacks but also to bring IRL consequences to these people.

    My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both )

    Getting punched in the face is actually a necessary human condition for a healthy civilization.

  • by agwa on 3/29/24, 4:40 PM

    > openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

    The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.

  • by korginator on 3/30/24, 3:08 AM

    xz is so pervasive, I just discovered on my Mac that the (affected?) version 5.6.1 made it into homebrew. The post in the linked article says that only Linux x86-64 systems are affected, but now I'm left scratching my head whether my Mac is also in trouble, just that we don't know it yet.
  • by jchoksi on 3/29/24, 10:26 PM

    The two active maintainers seem to be: Lasse Collin <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>

    Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03

    Blog post: "Kuso拍賣.有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301>

    Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/b...>

    The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank.

    The blog entry is basically a QA between jiat0218 and various other people about these chopsticks.

    If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.).

    Example question 38:

        Question 38
        accounta066 (3): Are these chopsticks really that good? I kind of want to buy
        them! But I recently sent money for online shopping but didn’t receive anything.
        It’s very risky; currently jiat0218 you don’t have any reviews, you can
        interview me. Do you want to hand it over?! … A sincere buyer will keep it.
        
        Reply to
        jiat0218 (4): First of all, I would like to express my condolences to you for
        your unfortunate experience! What can I say about this kind of thing...My little
        sister has always been trustworthy. What’s more, this is a pair of spiritual 
        chopsticks, so I hope to have a good one. It’s the beginning! As you can see,
        my little sister is very careful and takes her time when answering your
        questions. Except for the two messages that were accidentally deleted by her,
        she always answers your questions. If this still doesn’t reassure you, then I
        can only say that I still have room to work hard. You are still welcome
        to bid... ^_^
    
    Note however, it could all just be what it purports to be which is a prank auction of spiritual chopsticks.
  • by dboreham on 3/29/24, 10:31 PM

    Something about this I found surprising is that Linux distros are pulling and packaging pre-built binaries from upstream projects. I'd have expected them to build from source.
  • by 0x0 on 3/29/24, 4:41 PM

    Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?
  • by CanaryLayout on 3/30/24, 6:18 AM

    Well isn't this an interesting commit. He finished his inject macro to compose the payload at build, so now he can start clearing up the repo so none of that shit gets seen when cruising through it.

    https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...

  • by c_rrodriguez on 3/29/24, 7:42 PM

    Everybody here In jumping into the pure malice bandwagon, I have a better hypothesis.

    Abandonment and inaction, the actual developers of these tools are elsewhere, oblivious to this drama, trying to make living because most of the time you are not compensated nor any corporation cares about making things sustainable at all. This is the default status of everything your fancy cloud depends on underneath.

    An attacker took over of the project slowly and stayed dormant until recently.

  • by autoexecbat on 3/29/24, 8:47 PM

    I'm really curious about if the act of injecting a backdoor into OSS software is legal/illegal ?

    Are they somehow in the clear unless we can show they actively exploited it?

  • by jcalvinowens on 3/29/24, 7:41 PM

    Oof, this is on my Sid laptop:

      {0}[calvinow@mozart ~] dpkg-query -W liblzma5
      liblzma5:amd64  5.6.0-0.2
      {0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
      1
    
    Glad I stopped running sshd on my laptop a long time ago... still probably going to reinstall :/
  • by costco on 3/30/24, 4:06 PM

    Anyone have any idea what the code in the malicious liblzma_la-crc64-fast.o is actually doing? It's difficult to follow statically.
  • by Retr0id on 3/30/24, 7:23 AM

    The `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so)

    It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account).

    I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this.

    [0] https://pack.ac/

    [1] https://news.ycombinator.com/item?id=39793805

    [2] https://github.com/PackOrganization/Pack/issues/10

  • by MaximilianEmel on 3/29/24, 11:23 PM

    We need to get these complex & bloated build-systems under control.
  • by haolez on 3/29/24, 10:34 PM

    I'm not trying to troll, but I'm wondering if a distro like Gentoo is less susceptible to such attacks, since the source code feels more transparent with their approach. But then again, it seems that upstream was infected in this case, so I'm not sure if a culture of compiling from source locally would help.
  • by crispyambulance on 3/29/24, 6:29 PM

    I am not embarrassed to say... is there anything in there that someone who runs a server with ssh needs to know?

    I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.

    If I am keeping stuff up to date, is there anything at all to worry about?

  • by joshhansen on 3/31/24, 7:02 AM

    Is it time to deprecate the ability for code to implement linker symbols in other libraries? Shouldn't there be a strict namespace separation between binaries/libraries? liblzma being to implement openssh symbols seems like a symptom of a much larger problem.
  • by jeffbee on 3/29/24, 4:36 PM

    Safety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.
  • by BobbyTables2 on 3/30/24, 3:54 AM

    Why doesn’t GitHub force “releases” to be a simple repo tarball for sources and with binaries from GitHub actions or such…

    I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!)

    The real villain is not the perpetrator, it is Microsoft, and it is all of us.

  • by wannacboatmovie on 3/29/24, 7:32 PM

    Really disappointed in the number of posters here who are playing down rushing to judgement and suggesting perhaps a legitimate developer was compromised, when it's very clear this is sophisticated and not the work of a single person.

    I'm recalling bad memories of the Juniper backdoor years ago.

    Whoever did this, was playing the long game. As the top post pointed out, there was an effort to get this into Fedora.... which eventually makes its way into RHEL (read: high value targets). This was not for short term payoffs by some rogue developer trying to mine crypto or other such nonsense. What you are seeing here is the planting of seeds for something months or a year down the road.

  • by Brian_K_White on 3/30/24, 2:20 AM

    It doesn't really relate to this issue other than that both issues share a common source, but I wish we'd never fallen for xz.

    I agree with the lzip guy

    https://www.nongnu.org/lzip/xz_inadequate.html

  • by qxfys on 3/30/24, 3:16 PM

    So, it's been almost 24 hours since I read this yesterday. Is it confirmed that Jia Tan is the perpetrator? do we know who he/she really is? Or are we going to live for the rest of our lives only knowing the pseudo name? just like Satoshi Nakamoto did to us. ;)
  • by n3uman on 3/29/24, 6:10 PM

    https://github.com/tukaani-project/tukaani-project.github.io... Does this mean anything that it changed to a parameter??
  • by sschueller on 3/29/24, 6:31 PM

    So much for a quiet Easter holiday. Fuck
  • by BarbaryCoast on 3/30/24, 5:36 AM

    There's a bug in the detection script. The line:

    if [ "$path" == "" ]

    should be

    if [ "$path" = "" ]

  • by vasili111 on 3/29/24, 8:59 PM

    Could anyone please tell me if current stable version of Debian has that backdoor or not?
  • by ptx on 3/30/24, 8:30 PM

    Python for Windows bundles liblzma from this project, but it appears to be version 5.2.5 [0] vendored into the Python project's repo on 2022-04-18 [1], so that should be fine, right?

    [0] https://github.com/python/cpython/blob/main/PCbuild/get_exte...

    [1] https://github.com/python/cpython-source-deps/tree/xz

  • by 17e55aab on 3/30/24, 4:59 AM

    a user offered 5.6.0 and 5.4.5 in an issue to microsoft/vcpkg

    5.4.5 can be compromised

    https://github.com/microsoft/vcpkg/issues/37197

  • by croemer on 3/31/24, 1:02 AM

    Which nation state (if any) is most likely behind this? China based on name, or is this a red herring?

    The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia.

    See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

  • by lacoolj on 3/29/24, 9:20 PM

    What a disappointment.

    It's something always in the back of our minds as developers using public libraries, but when something like this happens, non-developers that hear about it start to associate it with the rest of the open-source community.

    It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.

  • by kazinator on 3/29/24, 8:59 PM

    Doesn't this call for criminal charges?
  • by Dribble4633 on 3/30/24, 9:45 AM

    Hello,

    Github just disabled the repo : https://github.com/tukaani-project/xz

    Do someone have an up to date fork to see the project history ?

  • by _zephyrus_ on 3/30/24, 10:07 AM

    Is there any news concerning the payload analysis? Just curious to see if it can be correlated with something I have in my sshd logs (e.g. login attempt with specific RSA keys).
  • by sirsinsalot on 3/30/24, 9:41 AM

    I think we have to assume that all community software is a target. The payoff for bad actors is too great.

    For every one of these we spot, assume there are two we have not.

  • by frankjr on 3/29/24, 6:01 PM

    Now consider that your average Linux distribution pulls in tens of thousands of packages, each of which can be similarly compromised. Pretty scary to think about.
  • by 3v1n0 on 4/2/24, 12:02 PM

    Also the attacker included in the 5.6.0 release the support for the long-awaited multi-threading decompression (and - broken - sandbox) making it very attractive to upgrade to...

    It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.

  • by 65a on 3/30/24, 4:51 AM

    Is there a proper reverse engineering of the payload yet?
  • by mdip on 3/29/24, 6:20 PM

    Anyone keeping current with OpenSUSE Tumbleweed got a update...downgrade. Prior to `zypper dup --no-allow-vendor-change` I had 5.6.0, now I'm at 5.4.6.
  • by hcks on 3/30/24, 9:12 AM

    It was caught out of luck due to performance degradation. So nobody reads the code - not even once- prior to merging into upstream supply chain?
  • by sylware on 3/29/24, 6:27 PM

    This is why the less the better... even if it means less comfortable... to a certain point obviously. And that includes SDKs...
  • by zeehio on 3/30/24, 6:24 AM

    On Ubuntu there is a bug report asking to sync the 5.6 version from Debian experimental https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055...
  • by Rucadi on 3/29/24, 4:33 PM

    Saw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.
  • by squarefoot on 3/30/24, 9:34 AM

    State actor or not, let's not ignore that the backdoor has been discovered thanks to the open nature of the projects involved that allowed digging into the code. Just another example like the infamous Borland InterBase backdoor in the early 2K that remained dormant for years and was discovered months after the source code has been released. If the xz malware authors worked for any corp that produced closed source drivers or blobs that can't be properly audited, we would be fucked; I just hope this is not already happening, because the attack surface in all those devices and appliances out there running closed code is huge.
  • by perryizgr8 on 3/31/24, 1:51 AM

    Why are projects like xz and sshd still active? Just freeze it, it works fine. Only changes should be fixes for vulnerabilities. None of this complicated new functionality. If you want something like that make a new project. If it is truly better people will use it.
  • by dfgdfg34545456 on 3/30/24, 9:02 AM

    chmod u+x running detect_sh script just runs with no output on my arch linux box?

    https://www.openwall.com/lists/oss-security/2024/03/29/4

  • by notmysql_ on 3/29/24, 11:00 PM

    Interestingly on of the accounts that the GitHub account who introduced the backdoor follows was suspended very recently [1] who is also part of the org who runs XZ

    [1] https://github.com/JiaT75?tab=following

  • by west0n on 3/30/24, 1:41 AM

    It seems that to counter this type of supply chain attack, the best practices for managing software dependencies are to pin the version numbers of dependencies instead of using `latest`, and to use static linking instead of dynamic linking.
  • by hypnagogic on 3/30/24, 1:59 PM

    In the future: automated `diff` or any other A/B check to see whether or not the tarball matches the source repo (if not, auto-flag with a mismatch warning attribute), is that feasible to implement?
  • by bicepjai on 4/2/24, 3:48 AM

    For someone who does not understand the packages used, could you please summarize in layman non technical terms. Thanks I did read the main post.
  • by itsTyrion on 3/30/24, 8:48 AM

    that's... creative. and patient. 11/10 concerning - now I'm wondering how many other projects could have shit like this in them or added right as I'm writing this shudder
  • by wowserszzzzz on 3/30/24, 8:08 PM

  • by fwungy on 3/30/24, 2:33 AM

    Brain fart: would it be possible to attach passwords to a crypto based micro transaction such that every time you attempted a password entry your crypto account was charged a small fee for the login attempt?

    This would thwart brute force attacks, but not be a significant cost for users. If you could attach your login to the crypto account it would mean the account would have to be funded to allow the attempt. The token wouldn't store passwords it would just be a gatekeeper to the login attempt.

    The fees would be paid to the service providers as mining fees.

    E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.

  • by neoneye2 on 3/30/24, 12:22 AM

    Damn. I'm on macOS and use homebrew. To my surprise I had "xz" version 6.5.1 installed on my computer!

    I ran "brew upgrade" and that downgraded to version 5.4.6.

  • by LeoPanthera on 3/29/24, 6:34 PM

    xz is just a horribly designed format, and always has been. If you use it, please switch to Lzip. Same compression level, but designed by someone competent.

    https://www.nongnu.org/lzip/

    https://www.nongnu.org/lzip/xz_inadequate.html

  • by bitwize on 3/30/24, 2:26 AM

    Looks like Jonathan Blow was right about open source.
  • by user20180120 on 3/30/24, 9:27 AM

    Why is the Long Range Zip lrzip compression format not used? It gives better compression than xz when using the correct switches.
  • by zingelshuher on 3/30/24, 12:28 PM

    Why isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.
  • by jum4 on 3/30/24, 7:29 AM

    Maybe @JiaT75 got forced to do it. Maybe someone has more personal contact with him and can check how he is doing.
  • by andix on 3/30/24, 12:32 AM

    Is there already a list of distributions that included the affected versions in non-prereelase channels?
  • by inevitable112 on 3/30/24, 11:44 AM

    Surely the real target of this was Tor (which links liblzma) not random SSH servers.
  • by MaximilianEmel on 3/29/24, 8:08 PM

    Has this affected OpenBSD at all?
  • by imanhodjaev on 3/30/24, 9:05 AM

    I wonder which browsers link liblzma and can this lead to https eavesdropping?
  • by nateskulic on 3/29/24, 9:30 PM

    Fairly deep bugs for a Bazaar.
  • by shp0ngle on 3/30/24, 5:13 AM

    we should take this diagram and change "random person in nebraska" to "possibly a state-level attacker"

    https://xkcd.com/2347/

    nice

  • by elintknower on 3/29/24, 7:50 PM

    Candidly how would someone protect against a vulnerability like this?
  • by evilmonkey19 on 3/30/24, 3:53 PM

    Which OS are affected by this compromise?? Is Ubuntu affected?
  • by betaby on 3/30/24, 1:31 AM

    How that backdoor is triggered and what exactly it does?
  • by xvilka on 3/30/24, 12:20 AM

    Maybe it's finally time to start sunsetting LZMA and xz all together in favor of newer algorithms like Zstandard that also offer better performance but compression rates on par with LZMA.
  • by llmblockchain on 3/29/24, 11:31 PM

    Was Debian 12/stable unaffected? Only sid?
  • by hypnagogic on 3/30/24, 3:43 PM

    - * _ring ring_ * - "Hello?" - "It's Lasse Collin." - "Why are you collin me? Why not just use the backdoor?"
  • by pinley on 3/30/24, 12:40 PM

  • by KOLANICH on 3/30/24, 11:52 AM

    Please note: the changes have been made after GitHub has enforced 2FA (certainly not for "better security", but for promotion of FIDO2 and Windows Hello biometric impl of FIDO2, see https://codeberg.org/KOLANICH/Fuck-GuanTEEnomo for more info. Until recent times (for now access via git protocol is blocked for my acc, I guess based on lack of 2FA set up) it was even possible to push into all repos one has access by just using single-factor SSH key even without enabling 2FA in the account). As I have warned, nothing will protect when a backdoor is introduced by a malicious maintainer, or a "smart entrepreneur" who sold his project to a ad-company, or a loyal "patriot" living and earning money within reach of some state, or just a powerless man who got an offer he can't refuse. In general supply chain attacks by "legitimate" maintainers cannot be prevented. "Jia Tan" is just a sockpuppet to mitigate consequences to maintainers to make it look like they are not involved into it. They surely are. At least according to the current info it were they who have given the malicious account the permission to publish releases on behalf of the project and access to the repo.

    IMHO all maintainers of the backdooored projects anyhow related to accepting the malicious changes should be considered as accomplices and boycotted. We don't need evidence of their liability, it is they who need to maintain their reputation. We are just free to take our decisions based on their reputation. Even if they were hacked themselves, it is not our problem, it is their problem. Our problem is to keep ourselves safe. It may feel "unjust" to ruin reputation of a person based on the fact he may be cheated or hacked… But if a person can be cheated or hacked, why should he/she have such a good reputation as everyone else?! So, it makes a lot of sense to just exclude and replace everyone, for whome there exists evidence of comprometation, no matter due to unconcern or malice. But FOSS is a doocracy serving products at dumpling prices ($0, free of charge), and for majority backdoored software is completely acceptable given that they get them free of charge. And powerful actors who can afford to pay for software will just hire devs to develop their private versions, while allowing the public to pay $0 for their free versions and use the backdoors placed into them themselves. In other words a complete market failure.

    I think that 1. xz project must be shut down completely. I mean projects should stop using it as a dependency, exclude from distros, boycott it. LZMA algo was developed by Igor Pavlov in 7z project, but somehow it has happenned that liblzma was developed and maintained by unrelated folks. liblzma should be developed as a part of 7z project taking no code other than the trivial one for API compatibility adapter from xz. 2. Projects created by compromised authkrs should be boycotted. 3. Other projects touched by the compromised devs/maintainers should be audited. 4. All the projects using autotools should be audited and must replace autotools with cmake/meson. Autotools is a piece of shit, completely uncomprehensible. There is no surprise it was used to hude a backdoor - according to my experience in FOSS noone likes to touch its scripts anyhow. 5. No project should be built from releases. Project should be built from git directly. Implementing full support of SHA256 in git and git forges (GitHub, GitLab, Codeberg, sr.ht) should be accelerated to mitigate attacks using collisions to replace approved commits (I guess the randomness can be concealed from reviewer's eye in binary resource files, like pictures).

  • by Rhea_Karty on 3/30/24, 6:08 AM

    TLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

    These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out.

    The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

    Observation 1: Time zone basic analysis

    Here is the data on Jia’s time zone and the number of times he was recorded in that time zone:

    3: + 0200 (in winter: February and November)

    6: +0300 (in summer: in Jun, Jul, early October)

    440: +0800

    1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert.

    2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March).

    Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

    Observation 2: Time zone inconsistencies

    Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.

    Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but flight from China to anywhere in Eastern Europe is at a min 10 hours

    Jia Tan Thu, 5 May 2022 20:53:42 +0800

    Jia Tan Sat, 19 Nov 2022 23:18:04 +0800

    Jia Tan Mon, 7 Nov 2022 16:24:14 +0200

    Jia Tan Sun, 23 Oct 2022 21:01:08 +0800

    Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1)

    Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1)

    Jia Tan Wed, 5 Oct 2022 23:54:12 +0800

    Jia Tan Wed, 5 Oct 2022 20:57:16 +0800

    —> again, given the flight time, this is even more impossible

    Jia Tan Fri, 2 Sep 2022 20:18:55 +0800

    Jia Tan Thu, 8 Sep 2022 15:07:00 +0300

    Jia Tan Mon, 25 Jul 2022 18:30:05 +0300

    Jia Tan Mon, 25 Jul 2022 18:20:01 +0300

    Jia Tan Fri, 1 Jul 2022 21:19:26 +0800

    Jia Tan Thu, 16 Jun 2022 17:32:19 +0300

    Jia Tan Mon, 13 Jun 2022 20:27:03 +0800

    —> the ordering of these time stamps, and the switching back and forth looks strange.

    Jia Tan Thu, 15 Feb 2024 22:26:43 +0800

    Jia Tan Thu, 15 Feb 2024 01:53:40 +0800

    Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

    Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

    Jia Tan Tue, 13 Feb 2024 22:38:58 +0800

    —> this travel time is possible, but the duration of stay is unlikely

    Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

    Observation 4: Bank holiday inconsistencies

    We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person.

    Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

    Chinese bank holidays (just looking at 2023):

    - Working on 2023, 29 September: Mid Autumn Festival

    - Working on 2023, 05 April: Tomb Sweeping Day

    - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

    Eastern European holidays:

    - Never working on Dec 25: Christmas (for many EET countries)

    - Never working Dec 31 or Jan 1: New Years

    Observation 5: No weekend work —> salary job?

    The most common working days for Jia was Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to be EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

    These times also line up well with Hans Jansen and Lasse Collin.

    I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

  • by alathers on 3/29/24, 9:35 PM

    Thank the gods I didn't plan on having a life this weekend
  • by 7ero on 3/30/24, 1:30 PM

    Is this sev0?
  • by krascovict on 3/31/24, 11:51 AM

    Hello everybody.

    I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor.

    The purpose of this commentary is focused on open source information (OSINT).

    I am not a judge of anyone or any action that may occur, the objective of this comment is to help through accurate and quick information to help the core developers of the affected packages and consequently the Linux kernel (which may have been indirectly or directly affected) take action necessary in relation to the fact that occurred.

    NOTE: This comment will always have "edit" so always review it for information.

    Information I have so far.

    Summary: 1. GitHub Account Suspension: - The accounts of @JiaT75 and @Larhzu were suspended by GitHub. - All Tukaani repositories, including downloads, were disabled. - Investigate the cause of the account suspensions and whether there is any correlation with suspicious activities.

    2. Possible Backdoor in xz/liblzma: - There are concerns about the presence of a backdoor in xz/liblzma. - Investigate whether there is evidence of compromise in the source code and recent updates. - Examine potential impacts, especially if the software is used in critical systems.

    3. Updates and Patches in Packages: - Note recent updates in packages such as MinGW w64, pacman-static, Alpine, and OpenSUSE. - Review changelogs to understand if these updates are related to security fixes.

    4. Jia's Activities on Platforms and Projects: - Investigate Jia's contributions to different projects and platforms, such as Arch Linux, Alpine Linux, and OpenSUSE. - Check for correlations between Jia's activities and reported security issues.

    5. Libera Registration Information: - Analyze Jia's registration details on Libera to determine the timeline of their online activities. - Consider correlating this information with other online activities of Jia.

    6. VPN Usage: - Confirm Jia's use of VPN and assess its impact on security investigations. - Explore possible reasons for using a VPN and how it may affect the identification and tracking of online activities.

    Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760

    Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...

  • by xyst on 3/29/24, 7:00 PM

    Time for another OS wipe. Glad I keep bleeding edge versions VMd
  • by 7ero on 3/30/24, 1:29 PM

    is this sev0?
  • by kosolam on 3/29/24, 7:59 PM

    Jesus! Does anyone know if Debian stable is affected?
  • by imanhodjaev on 3/30/24, 9:04 AM

    now I wonder which browsers link liblzma?
  • by Rhea_Karty on 3/30/24, 6:05 AM

    Notes on time stamps and time zones.

    A few interesting bits that I haven't fully fleshed out. TLDR: Some people have been throwing around that Jia is from “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

    The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

    Observation 1: Time zone basic analysis

    Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800

    1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). 1. Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

    Observation 2: Time zone inconsistencies

    Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.

    Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but a flight from China to anywhere in Eastern Europe is at a min 10 hours

    Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible

    Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps and the switching back and forth between time zones looks strange.

    Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely

    Observation 3: Strange record of time stamps

    It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

    Observation 4: Bank holiday inconsistencies

    We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person.

    Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bank holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

    Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

    Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years

    Observation 5: Little weekend work —> salary job?

    The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

    These times also line up well with Hans Jansen and Lasse Collin.

    I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

  • by returningfory2 on 3/29/24, 8:03 PM

    Another interesting data point: about 2 years ago there was a clear pressure campaign to name a new maintainer: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...

    At the time I thought it was just rude, but maybe this is when it all started.

  • by k8svet on 3/29/24, 8:10 PM

    Wait, I'm on mobile. Did this partially slip by because of the ABSURD PRACTICE of publishing release.tarballs that do not 1:1 correspond with source?

    Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first.

    EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.

  • by port443 on 3/29/24, 8:09 PM

    I think its much more likely this was not a bad actor, given their long history of commits.

    It's a known fact that China will "recruit" people to operate them. A quote:

    > They talk to them, say my friend, I see you like our special menu. Are you from China? Are you here on a VISA? Do you have family back there? Would you like your family to stay alive? Is your loyalty to this temporary employer or is your loyalty to your motherland? You know, a whole bunch of stuff like that. That’s how Chinese intelligence operations acts...

    This just gives feelings of less "compromised account" and more "Your account is now our account"

  • by AdmiralAsshat on 3/29/24, 5:00 PM

    Yikes! Do you have any info on the individual's background or possible motivations?
  • by mikolajw on 3/29/24, 8:41 PM

    Tukaani website states "jiatan" as the nickname of the malicious code committer on Libera Chat.

    WHOWAS jiatan provided me the following information:

    jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024

    WHOIS yields nothing, the user is not present on the network at the moment.

    Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.

  • by mrcoffee4u on 3/29/24, 10:09 PM

    can someone ELI5 ?
  • by fullstackchris on 3/29/24, 11:12 PM

    pRoBaBlY a StaTe AcToR

    zero definition of what that means...

    egos of people who just like to say cool words they don't understand

    lol

    this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad

    state actor? who? what motive? what country? all comments involving "state actor" are very broad and strange... i would like people to stop using words that have no meaning, as it really takes away from the overall conversation of what is going on.

    i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid

  • by throwaway67743 on 3/30/24, 12:10 PM

    It's always Debian, like last time when they removed RNG randomness from ssh because of a warning.
  • by mise_en_place on 3/30/24, 3:25 PM

    This is why we never upgrade software versions. I’ve been asked by our customers why we use such an old AMI version. This is why.
  • by circusfly on 3/29/24, 8:45 PM

    Waiting for the new YouTube videos on this. "Woah! Linux has a back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022 isn't affected.
  • by stephc_int13 on 3/29/24, 10:41 PM

    I guess that rewriting liblzma in Rust would not have prevented this backdoor. But would have likely increased the confidence in its safety.

    Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.

  • by shortsunblack on 3/29/24, 11:26 PM

    Pretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/
  • by Zigurd on 3/29/24, 9:20 PM

    "Lasse Collin," as other posters here have found, does not seem to exist as an experienced coder. Oddly, there is a Swedish jazz musician named Lasse Collin, which would otherwise be one of those names, especially the last name, that would stick out. Instead it is buried under a lot of mentions of a musician.