from Hacker News

Hacking millions of modems and investigating who hacked my modem

by albinowax_ on 6/3/24, 6:51 AM with 272 comments

by phs318u on 6/4/24, 6:07 AM
What a great article. Very easy to follow. The best part was that instead of attacking the messenger and denying any problem, Cox seem to have acted like the very model of responsible security response in this kind of situation. I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs. It's the kind of error that could easily be missed by superficial testing or depending on the reason behind the bug, perhaps not even experienced in the test environment.
by Bluecobra on 6/4/24, 1:00 PM
What sucks about this situation is when your ISP forces you to use their modem or router. For example, I have AT&T fiber and it does some kind of 802.1X authentication with certificates to connect to their network. If they didn't do this, I could just plug any arbitrary device into the ONT. There are/were workarounds to this but I don't want to go through all those hoops to get online. Instead, I ended up disabling everything on the AT&T router and have my own router that I keep up to date plugged into that. Unbeknownst to me, the AT&T router could be hacked and I would never notice unless it was adversely affects my service.
Thank god most things use HTTPS these days.
by kn100 on 6/4/24, 1:20 PM
Great read, and fantastic investigation. Also nice to see a story of some big corp not going nuclear on a security researcher.
I can't say for certain, and the OP if they're here I'd love for you to validate this - but I'm not convinced requests to the local admin interface on these Nokia routers is properly authenticated. I know this because I recently was provisioned with one and found there were certain settings I could not change as a regular admin, and I was refused the super admin account by the ISP. turns out you could just inspector hack the page to undisable the fields and change the fields yourself, and the API would happily accept them.
if this is the case, and an application can be running inside your network, it wouldn't be hard to compromise the router that way, but seems awfully specific!
by rwmj on 6/4/24, 9:17 AM
> After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse
Would you trust a thing they say? It seems their whole network is swiss cheese.
by stonks on 6/4/24, 3:27 PM
Many routers require manual firmware updates. GL.iNet routers had several RCE (Remote Code Execution) vulnerabilities within the last 6 months. I advise you to have a quick look in your own router to ensure its not hacked, and possibly upgrade firmware.
As a typical user the noticeable symptoms for me were: - internet speed noticeably slows down - WiFi signal drops and personal devices either don't see it, or struggle to connect. At the same time the router is still connected to the internet - router's internal admin page (192.168.8.1) stopped responding
I imagine many users haven't updated their routers and thus may be hacked. In my case the hacker installed Pawns app from IPRoyal, which makes the router a proxy server and lets hacker and IPRoyal make money. The hacker also stole system logs containing information about who and when they use the device, whether any NAS is attached. They also had a reverse shell.
Solution: 1. Upgrade firmware to ensure these vulnerabilities are patched. 2. Then wipe the router to remove the actual malware. 3. Then disable SSH access, e.g. for GL.iNet routers that's possible within the Luci dashboard. 4. Afterwards disable remote access to the router, e.g. by turning Dynamic DNS off in GL.iNet. If remote access is needed, consider Cloudflare Tunnel or Zero Trust or similar. There is also GoodCloud, ZeroTier, Tailscale, etc. I am not too sure what they all do and which one would be suitable for protected access remotely. If anyone has advice, I would appreciate a comment.
Consider avoiding GL.iNet routers. They do not follow principle of least privilege (PoLP) - router runs processes using root user by default. SSH is also enabled by default (with root access), anyone can try to bruteforce in (10 symbol password consisting of [0-9A-Z] and possibly might be more predictable). I set mine to only allow ssh keys rather than a password to prevent that. Despite running OpenWrt they are actually running their own flavor of OpenWrt. So upgrading from OpenWrt 21.02 to 23.05 is not possible at the moment.
by daneel_w on 6/4/24, 10:07 AM
> "...and found no history of abuse..."
Because they didn't have enough logging or auditing to start with, or no logs or audit data left since the hack.
by mavamaarten on 6/4/24, 7:41 AM
What sort of authentication system just lets calls through randomly sometimes... The incompetence!
by underlogic on 6/4/24, 6:36 PM
Did they *pay* him? He kind of saved them, tipped them off to a complete compromise of their security infrastructure which was not trivial to discover. Looks like he got nothing in return for "doing the right thing". How insulting is that? What is their perception of someone walking in to their offices with this essential information? I guarantee his self image and their perception are very different. They see an overly caffeinated attention seeking "nerd" just handed them a 300k exploit in exchange for a gold star and then they ran like smeg to cover their asses and take all the credit internally. He feels like superman, goes home to his basement apt, microwaves some noodles and writes a blogpost. This is a perfect example why you never, never report a 0day.
by mannyv on 6/4/24, 3:04 PM
An open question is still: how were the attackers able to grab his HTTP traffic?
Some CPEs have a cloud Wireshark-like capability for debugging. I'm not sure if those are even on the Cox production firmware images. Usually there's a set of firmware for production and a set for test (which obviously makes it hard to test for problems in production).
I suppose Cox could do a check to see what firmware versions are out there. ISPs can auto-upgrade firmware that doesn't match a specific firmware revision, and this was a Cox modem so they probably have firmware for it. So if it was a debug firmware how did it get there and survive?
by megous on 6/4/24, 7:43 AM
One of the reasons to not be excited about ISP provided cable modems with WiFi functionality and to have good endpoint/service security on your LAN. (TLS, DNS over TLS at least accross the modem/ISP)
I just put it in bridge mode, disable wifi, and all network functionality is served by my own devices.
The last modem I rented from ISP, the ISP didn't bother with any firmware updates for ~10 years. It was rock stable because of that, though. :)
by cdaringe on 6/4/24, 3:44 PM
Nightmare fuel. Giant tech company, giant vuln. There’s so much to say, but more than anything Im just upset. The article and this dude are amazing. The exploit is not excusable.
by lanrat on 6/5/24, 3:48 PM
I observed very similar behavior a few years back when transferring files between two servers under my control on different parts of a large university network.
We also initially thought we were the subject of a breach, but after the investigation we determined that the network's IDS was monitoring all traffic, and upon certain triggers, would make identical requests from external networks.
We found a way to identify all other similar IDSs across the internet and even "weaponize" this behavior. We ended up writing a paper on it: https://ian.ucsd.edu/papers/cset2023_fireye.pdf
by peter_d_sherman on 6/4/24, 1:04 PM
Observation: The root of this problem is NOT because Cox's engineering practices lacked a comprehensive enough security review process to find and fix security vulnerabilities prior to them being discovered post deployment ("hindsight is always 20/20" as they say), but rather because there was (and still is) an Information Asymmetry between Cox and Cox's customers, i.e., in terms of complete knowledge of how Cox's devices actually work under the hood...
Although, in fairness to Cox, this Information Asymmetry -- also exists between most companies that produce tech consumer goods and most tech consumers (i.e., is it really a big deal if most other big tech companies engage in the same practices?), with the occasional exception of the truly rare, completely transparent, 100% Open Source Hardware, 100% Open Source Software company...
https://en.wikipedia.org/wiki/Information_asymmetry
Anyway, a very interesting article!
by mrbluecoat on 6/4/24, 2:19 PM
Great article, but unfortunately a determined threat actor would just go to the source and get a remote job as a Cox technician to gain access to millions of routers to add to their botnet. A real solution by the ISP would be to implement a software (or, preferably, hardware) setting that prevents remote access by default unless explicitly enabled by the customer. That approach would slow a social engineering campaign and limit the scope of a hack like this.
by fellerts on 6/4/24, 7:31 AM
Great read, I loved following your thought process as you kept digging.
At what point did you inform Cox about your findings? It doesn't sound like you were ever given the green light to poke at their management platform. Isn't work like this legally dubious, even if it is done purely in white-hat fashion?
by zaptrem on 6/4/24, 8:51 AM
Why do y’all think the attacker was replying all of his requests? Could they be probing for unintentionally exposed endpoints themselves?
by goshx on 6/4/24, 4:18 PM
It's unbelievable that Cox offers no compensation or reward for incredible work like this.
by biosboiii on 6/4/24, 9:38 AM
Holy hell, but how are your laws in the US aligned so doing something like this is okay?
In Germany you would get minimum 3 years in jail for this, people got in front of court for way way way way less.
by TeMPOraL on 6/4/24, 8:46 PM
Great writeup. There's just one thing I don't get: the auth part. It seems the author managed to access protected endpoints without any auth, by just repeating the same request over and over until the endpoint randomly accepted it. The part that confuses me is, how could that possibly happen? What possible architecture could this system have to enable this specific failure mode?
I struggle to think of anything, short of auth handling being a separate service injected between a load balancer and the API servers, and someone somehow forgot to include that in autoscaling config; but surely this is not how you do things, is it?
by mannyv on 6/4/24, 2:45 PM
The intermittent auth thing in /profilesearch is a sign that they're round-robinning the servers and misconfigured one.
Also, it looks like he hit a front-end API that drives the TR-069 backend. Changing the WiFi SSID is a long way from being able to "...execute commands on the device"
by longsword on 6/4/24, 12:21 PM
i'm really glad that i can use my own modem. In germany every ISP is by law required to accept self brought modems. They can't force you to use their often shitty hardware. My current modem/router is up for 3 months without a single interruption to my connection.
by psd1 on 6/4/24, 10:57 AM
I see arguments in favour of tr069, but it's the mechanism that BT used to reboot my modem every night at 3am. I hate ISPs.
by wouldbecouldbe on 6/4/24, 8:49 AM
This is seems like a huge vulnerability, are there any legal repercussion that happens in those situations?
by taink on 6/4/24, 12:44 PM
```
  One of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd.
```
I was thinking about this while reading. My guess is that the vulnerability was limited to reading incoming requests (to the modem) or something along those lines, not full control of the network. Replaying the requests is a good way to get both ends of the traffic if you can only access one. For instance, a login + password being authenticated. Just a thought!
EDIT: I'd be hard-pressed to know how one could exploit this, given TLS would encrypt the requests. Maybe they're counting on using badly encrypted requests, encrypted with e.g. TLSv1.0?
by thecodemonkey on 6/4/24, 7:29 AM
It's easy to hate on big companies. But can we just applaud Cox for having patched this within a day? That's incredible.
by arrty88 on 6/4/24, 12:30 PM
> One of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd.
Did you determine if POSTs were replayed? As in, logging into accounts and sending payment info and account info?
by Namidairo on 6/5/24, 2:27 PM
> Somehow, someone was intercepting and replaying the web traffic from likely every single device on my home network.
Normally I'd laugh and assume device compromise but...
The largest ISP in Australia (Telstra) got caught doing exactly this over a decade ago. People got extra paranoid when they noticed the originating IP was from Rackspace as opposed to within Telstra. Turned out to be a filter vendor scraping with dubious acceptance from customers. The ToS was quietly and promptly updated.
by EligibleDecoy on 6/4/24, 2:27 PM
My bet on the replays was that the attacker misconfigured their payload or something and it was meant to replay command and control requests to obfuscate where the C2 server was
by gianpaj on 6/5/24, 9:39 AM
This reminded me to turn off "privacy settings" to "keep your vehicle in good condition and observe the vehicle's health" on my Volvo XC40 after the mechanics asked me to turn it on yesterday during the yearly maintenance. I don't know if they can change some settings remotely, but I prefer to be cautious
by jokoon on 6/4/24, 8:46 AM
I remember creating some webserver at work years ago, and I saw a router querying it. I warned the company admin.
Also, my wifi firmware occasionally crashes and needs to be restarted.
I don't work in cyber security or on anything sensitive, but if I was told I'm under surveillance by some government or some criminal, I would not be surprised.
by __turbobrew__ on 6/4/24, 8:41 PM
Another reason to not use ISP provided hardware. I have never had issues using my own OpenBSD box as a router.
by Heidaradar on 6/4/24, 11:07 AM
I love how well he explains it, even to someone like me who knows p much nothing about cybersecurity.
by metadat on 6/5/24, 1:13 AM
> there were about 700 different API calls..
That's more API endpoints than some first tier public clouds, wow. For a modem.
Somebody wanted (and sorta deserves) promo..
But also not, because the whole platform turned out to be incredibly insecure! Egregious!!!
by _benj on 6/4/24, 7:52 PM
Wow! I just what a high a security researcher would feel while performing this research and keep finding open doors!
I wonder if it’s a mix of exhilaration and being terrified!
by webninja on 6/5/24, 1:54 AM
Are there any creation of new laws or removal of hindering laws that would facilitate the fixing of these devastating security vulnerabilities?
by worewood on 6/4/24, 5:00 PM
Not trusting the modems we're given is a damn good reason to use a VPN, as opposed to the market bulsshsait the VPN companies usually propagate
by amluto on 6/4/24, 2:11 PM
Some CPE exposes an API on the LAN side, and some of these APIs aren’t protected against CSRF. I wonder whether the modem in question is vulnerable.
by codedokode on 6/4/24, 6:42 PM
Replaying requests might be not a malicious attacker, but simply an ISP wishing to know and sell customer's interests.
by system2 on 6/5/24, 12:37 AM
It was so fun to read this. I am also surprised COX hot patched it within a day.
by kernal on 6/4/24, 3:37 PM
Moral of the story - never turn on remote access on your modem.
by jokoon on 6/4/24, 8:47 AM
What were those fbi redacted things? Were those backdoors?
by andrewstuart on 6/4/24, 12:31 PM
>> Authenticate your access patterns.
What does this mean?
by sammy2255 on 6/4/24, 2:41 PM
No payout?
by syngrog66 on 6/4/24, 2:44 PM
WARNING: nerd sniping. lol
by wiz21c on 6/4/24, 9:21 AM
page is now 404 :-/