from Hacker News

Encrypted traffic interception on Hetzner and Linode targeting Jabber service

by f311a on 10/20/23, 12:40 PM with 306 comments

  • by costco on 10/20/23, 4:49 PM

    This is almost certainly related to some sort of Russian cybercrime investigation. If you ever read Krebs or peruse some of the seedier Russian forums, xmpp.ru will sound familiar to you, because it is. Not imputing anything to the operator, that's just the nature of operating an anonymous service.

    https://ddanchev.blogspot.com/2021/03/exposing-currently-act...

    https://ddanchev.blogspot.com/2019/07/profiling-currently-ac...

    https://blog.talosintelligence.com/picking-apart-remcos/

    https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-...

    Really interesting writeup though. I guess it's a practical example of why everyone should get a CT monitoring service!

  • by rodlette on 10/20/23, 4:34 PM

    I agree with most of the mitigation suggestions.

    For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard.

    > All issued SSL/TLS certificates are subject to certificate transparency.

    +1. crt.sh has RSS feeds for this.

    > Limit validation methods and set exact account identifiers

    Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?

    > Monitor SSL/TLS certificate changes on all your services using external service

    +1. High-risk targets should be aware of what certs are valid at any time, and be checking for those.

    > Monitor MAC address of default gateway for changes

    A more sophisticated attack could preserve the MAC address.

    > "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate.

    TIL.

  • by lima on 10/20/23, 5:21 PM

    Perfectly pulling off an actual MitM attack and then forgetting to renew the certificate is certainly a very German thing :-)

  • by LinuxBender on 10/20/23, 2:12 PM

    Archive [1]

    Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof.

        openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
    SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16
    [1] - https://archive.ph/C0jYJ [updated]

  • by Libre___ on 10/20/23, 6:42 PM

    I was a bit curious about the MAC address since it isn't locally administered which indicates to me that someone likely deliberately chose this range of addresses.

    https://www.google.com/search?q=%2290%253Ade%253A01%22+linod...

    Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure

  • by throwaway77384 on 10/20/23, 2:45 PM

    This is baseless speculation, but I'm assuming Jabber is being targeted as it's famously used on darknet markets for drug trades (or other illicit activity). Goes to show that you should never just trust "it's encrypted, bro". You need to PGP your messages at the very least. Is PGP crackable by quantum computers? Will there be hardening against those kinds of attacks in the future? Since, if the messages have been hoovered up in encrypted form, it's just a matter of time until they get decrypted. And this appears to be done for just about all web traffic they can get their hands on... see https://en.wikipedia.org/wiki/Utah_Data_Center

  • by upofadown on 10/20/23, 4:14 PM

    >End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords.

    The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it.

    Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.

  • by mkup on 10/21/23, 9:02 AM

    There's one thing I can't understand in this story: if that's lawful interception, why Hetzner and Linode bothered to set up MitM interception with different LE certificate and key, rather than extract the TLS private key directly from the RAM and/or storage device of the VPS? Even if this is a physically dedicated server, they can extract the private key from the RAM by dumping the RAM contents after unscheduled reboot. Extraction of the private key isn't visible in CT logs, much more stealthier, practically undetectable.

  • by harrymit907 on 10/21/23, 3:00 PM

    A user reported this on Reddit 3 days ago: https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzn...

  • by Andrew_nenakhov on 10/20/23, 4:21 PM

    I wonder if there is a legal way to demand Hetzner/Linode comments on this situation. Likely, the entity behind the interception is some government agency or police.

  • by kristjank on 10/20/23, 3:43 PM

    This really wants me to re-evaluate all the times I have gracefuly TOFU'd while SSHing into a machine or even logging onto a website.

  • by hhsectech on 10/21/23, 11:58 AM

    I've always thought a solution like LetsEncrypt would be an ideal backdoor for LEAs wanting to circumvent SSL/TLS.

    The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?

  • by codedokode on 10/20/23, 10:48 PM

    Wouldn't be surprised if this happened in Russia; but not in Europe. Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability - doesn't this mean that large national ISPs can easily issue ceritificates for any site hosted within a country?

  • by whytevuhuni on 10/20/23, 11:16 PM

    Could this be a blue pill attack? A vulnerability in the xmpp server exploited to inject a rootkit, which then hides itself inside the kernel?

    Or creates network/pid namespaces and puts you in them, while leaving the mitm server in the original one?

    If so, the mitm could be on the same host, and wouldn't need the cooperation of the hosting provider.

    I'm not sure how to check for either of these without restarting (which the admin does not seem to want to do, as it is a live service).

    https://en.wikipedia.org/wiki/Blue_Pill_(software)

  • by zhovner on 10/20/23, 2:48 PM

    >The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023

    Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?

  • by beeboobaa on 10/20/23, 3:37 PM

    Are there any services that monitor the certificate transparency logs and send an email when a new certificate is issued for your domain? That would alert you to this kind of MITM

  • by wutwutwat on 10/20/23, 4:45 PM

    As an end user, how can I monitor if I’m being MITM’d? If everything I do is proxied through an attacker’s network, nothing I do online could be trusted to properly test if I were being proxied, right? I know apps can do cert pinning, but as an end user how can I validate that when connect to anything that it’s the real one and not a request from being middled?

  • by chgo1 on 10/21/23, 8:01 AM

    @dang: .org.ru should probably be treated as TLD

  • by mermaidsalad on 10/21/23, 8:15 AM

    Next time someone in Matrix discussion will ask "why we need Matrix when there is jabber/xmpp" show them this:

    > All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.

  • by vb-8448 on 10/20/23, 3:57 PM

    > The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023

    > We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.

    > Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.

    And what if the attacker tricked somehow the letsencrypt challenges?

    Or this is supposed to be impossible?

  • by olegarch on 10/21/23, 2:59 PM

    Using DANE or so (RRs CAA/TLSA) can help in this case (when provider route traffic to specific port), but does not solve the problem completely. If DNS record will be compromised, then hacker can setup "correct" TLSA/CAA record to trust his fake certificate. As result, problem can be solved, if we will have reliable DNS subsystem, where is impossible to compromise DNS provider or infrastructure.

    I think, emerDNS can be used in such critical application: https://emercoin.com/en/documentation/blockchain-services/em...

  • by SergeAx on 10/20/23, 10:09 PM

    And this is why, kids, you need to enforce certificate pinning on your critical infrastructure!

  • by Traubenfuchs on 10/20/23, 10:21 PM

    We are using self developed application layer security. It requires specialized client code and is terminated at a gateway that proxies requests to our normal applications.

    I really, really hate the idea of this kind of eavesdropping.

  • by withinboredom on 10/22/23, 8:01 PM

  • by psvz on 10/21/23, 2:58 PM

    let's encrypt is a public non-profit, who would definitely assist in finding out how domain control verification (DCV) happened - somehow that obvious line of pursuit was missing from the investigation. The mystery of the MAC 90:de:01:49:76:ae is interesting though, but CIA-custom-Made NIC might be a bit superfluous as an idea. Linode (acquired by Akamai) might use something in-house designed and OEM-manufactured, so I wouldn't treat it as strong evidence. Talk to Let's Encrypt - that will likely unveil what happened.

    Interestingly, search "90:de:01" in this page: https://www.cyberciti.biz/faq/howto-linux-configuring-defaul...

    ^^ looks like another victim VM of the alleged mysterious interceptor :-)

  • by zxwrt on 10/20/23, 4:45 PM

    What a nefarious move by Hetzner and Linode. How to trust them after this?

  • by mrbonner on 10/21/23, 12:27 AM

    I’m wondering if mTLS (aka zero-trust) is used, would take prevent this kind of MITM attack? My understanding is that it should since the cert is self-signed to be used from both ends.

  • by snvzz on 10/21/23, 5:27 AM

    A reminder e2ee is a must.

  • by mak234 on 10/20/23, 5:35 PM

    so is jabber.ru still secure now? what needs to be done on the user side to make it so? can one reissue/request new server certificate from the server and not from let's encrypt? thanks

  • by AtNightWeCode on 10/20/23, 6:50 PM

    The way that certbot works with let's encrypt the only surprise is that this does not occur more often. We have monitoring on several certs used for TLS. We should probably add an alert if an A-record changes as well.

    I assume the root cause is DNS tricks.

  • by wizzard0 on 10/20/23, 10:57 PM

    well i must admit monitoring certificate issuance with LetsEncrypt is quite boring even if you HAVE alerts set up (I do)

    so… not surprised. Still cool. What a time to be alive

  • by riku_iki on 10/20/23, 11:15 PM

    So, if I rent server in those services, it all potentially can be compromised and there is no way to make sure things are secure?..

  • by hnarn on 10/20/23, 8:05 PM

  • by gigatexal on 10/20/23, 10:27 PM

    This is a deal breaker for me. Which is a shame because Hetzner has such great pricing. Oh well.

  • by dathinab on 10/20/23, 5:45 PM

    > We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.

    This seems to be somewhat jumping the gun I think.

    Given the certs and the target assuming it's an lawful interception seems reasonable.

    But there is nothing there which requires Hetzner or Lindoe complying or knowing about this.

    Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.

  • by 0xbadcafebee on 10/20/23, 6:47 PM

    Getting a valid cert for a domain you don't own is getting close to trivial. All of the mitigation strategies are defeated a number of ways, have been for years. I've commented on incidents like these for years on HN. I'm not some internet big wig, and any blog I write isn't going to trend, so this never gets traction. But anyone who understands the tech can find a half dozen ways to get a cert. If you really need to trust your connection, don't rely on internet PKI / public certs. It was secure enough for general purpose in the aughts, it's not anymore.