by gnyman on 9/16/23, 6:27 AM
You cannot hide anything on the internet anymore, the full IPv4 range is scanned regularly by multiple entities. If you open a port on a public IP it will get found.
If it's a obscure non-standard port it might take longer, but if it's on any of the standard ports it will get probed very quickly and included tools like shodan.io
The reason why I'm repeating this, is that not everyone knows this. People still (albeit less) put up elastic and mongodb instances with no authentication on public IP's.
The second thing which isn't well known is the Certificate Transparency logs. This is the reason why you can't (without a wildcard cert) hide any HTTPS service. When you ask Let's Encrypt (or any CA actually) to generate veryobscure.domain.tld they will send that to the Certificate Transparency logs. You can find every certificate which was minted for a domain on a tool like https://crt.sh
There are many tools like subdomain.center, https://hackertarget.com/find-dns-host-records/ comes to mind. The most impressive one I've seen, which found more much more than expected, is Detectify (which is a paid service, no affiliation), they seem to combine the passive data collection (like subdomain.center) with active brute to find even more subdomains.
But you can probably get 95% there by using CT and a brute-force tool like https://github.com/aboul3la/Sublist3r
by banana_giraffe on 9/16/23, 4:39 AM
by Brananarchy on 9/16/23, 5:44 AM
As others have said, certificate transparency seems to be doing some heavy lifting here. It reports subdomains for me that have never had a public CNAME or A record, but have had let's encrypt certs issued for internal use.
It's also missing some that have not had certs issued, but that are in public DNS
by TheHappyOddish on 9/16/23, 11:48 AM
Hardly "all subdomains". Unless it's doing an AXFR of my zone file (unlikely), this isn't possible.
It's a scraper/guesser, using cert transparency, common names, etc. Cute toy, but false claims.
by hankchinaski on 9/16/23, 3:45 AM
I would be keen to know what techniques are used. Usually subdomain discovery is done with dns axfr transfer request which leaks the entire dns zone (but this only works on ancient and unpatched nameservers) or with dictionary attacks. There are some other techniques you can check if you look at the source code of amass (open source Golang reconnaissance/security tool), or CT logs. Dns dumpster is one of the tools I used alongside pentest tools (commercial) and amass (oss)
by derefr on 9/16/23, 4:08 AM
Interesting. Our domain has some subdomains with a numeric suffix; and the API response here has entries in that pattern for not only the particular subdomains that exist or ever existed, but also for subdomains of the same pattern that go
beyond any suffix number we've ever actually used.
You'd think they'd at least be filtering their response by checking which subdomains actually have an A/AAAA/CNAME record on them...
by blueflow on 9/16/23, 10:16 AM
I entered my own domains and i got so many garbage entries. It feels like an AI reading letsencrypt logs and then adding made up shit to it.
by internet2000 on 9/16/23, 4:27 AM
For my personal domain: it got the ones I have on the SSL cert alternative subject names, made up three, returned one I deleted more than a year ago, and didn't find two. Very curious.
by donatj on 9/16/23, 7:08 AM
Interesting. It only found less that a quarter of the subdomains of the site I work on, and everything it did find is public facing. I wonder if that’s maybe something to do with how we set up certificates for public vs internal subdomains? It even missed “staging.” which should be nearly identical in configuration to www
by SushiHippie on 9/16/23, 10:18 AM
Note, if you looked up a domain and it had no results, you should check back again after some minutes. I looked my domain up and had zero results, which was weird as it should at least find some in the ct logs, but a few minutes later it showed some subdomains.
by RockRobotRock on 9/16/23, 3:45 AM
This is certificate transparency doing most of the work, right?
by Arubis on 9/16/23, 4:49 PM
If this were able to determine which wildcard subdomains were active for a given domain, you could use it to figure out a lot of B2B companies’ client/customer list.
by Xorakios on 9/16/23, 10:45 PM
Just for giggles, does anyone else remember when "subdomains" were called "machine names" because physical devices were limited to one service?
www.
ftp.
mail.
... weren't theoretical or merely mnemonic.
Felt like an old coot when using "machine name" to a 40 year old IT professional and she was perplexed!
by p4bl0 on 9/16/23, 6:44 AM
It gave me empty results for some of my domains that have multiple subdomains that have TLS certificate associated with them so that must appear in the certificate transparency log.
I guess it should be "discover some subdomains for some domains".
by pabs3 on 9/16/23, 5:12 AM
by ohuf on 9/16/23, 6:15 AM
by keepamovin on 9/16/23, 5:10 AM
This is fantastic!!!
What kind of security considerations are there to having multi-tenant user applications on subdomains and then having them exposed like this?
I'm building a SaaS right now, and I guess one thing is that a given username can then be discovered as a valid login for the system...but obviously that's only part of the login credential.
Maintaining a list of mappings to opaque subdomains seems to reduce targeting, and conceal login partial credentials, but doesn't seem to offer much besides.
Analysis?
by cm2187 on 9/16/23, 4:36 AM
One thing I noticed looking at my logs is that there is almost no unsolicited traffic (i.e. failed authentication attempts, exploits of various worldpress bugs, etc) through ipv6. I think it's a function of 1) those coming from networks (compromised home devices, etc) that don't support v6, 2) the v6 address space being too large to scan (the size of an encryption key), so good security by obscurity. This would nullify 2).
by weird-eye-issue on 9/16/23, 6:01 AM
I got back an empty list for my domain on Cloudflare with several subdomains (non wildcard)
edit: I retried on my computer (was on my phone earlier) and now it returns all of our subdomains, even picking up our test R2 bucket. In guessing I was rate limited because I accidentally loaded the example file a few times
by hbcondo714 on 9/16/23, 5:17 AM
by franky47 on 9/16/23, 5:02 AM
Sublist3r [1] does a similar job, as long as you have the authorisation to use it on a particular domain, as it uses more aggressive discovery techniques.
[1] https://github.com/aboul3la/Sublist3r
by asmor on 9/16/23, 6:19 AM
by johntiger1 on 9/16/23, 4:19 AM
by perryizgr8 on 9/16/23, 4:26 AM
It detects only some of mine. To be precise, it does not detect subdomains being served by a service behind a CloudFlare tunnel.
by xg15 on 9/16/23, 2:53 PM
I think as soon as cert transparency was introduced, it was pretty clear we would eventually get something like this.
by judge2020 on 9/16/23, 4:50 AM
by TechBro8615 on 9/16/23, 6:03 AM
I get a rate limit error when I click the text input (I'm on a VPN).
by mmarquezs on 9/16/23, 5:55 AM
Nice, last time I used Wolframalpha for this.
by webprofusion on 9/16/23, 5:04 AM
This is a CT log search right?
by zX41ZdbW on 9/16/23, 4:55 AM
How can I download the entire dataset from this service?
by maul666 on 9/20/23, 8:19 AM
dpd.co.uk
by Ocha on 9/16/23, 4:25 AM
Missed some for me
by tobinfekkes on 9/16/23, 4:32 AM
This is crazy, I was just looking for this exact thing a couple days ago. Thank you for sharing. Brilliant work.