from Hacker News

Curl-Impersonate

by jakeogh on 12/30/24, 9:18 AM with 111 comments

  • by cle on 12/30/24, 1:34 PM

    The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi

    I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyone is only using software approved by them, and to ensure that they can identify you. From multiple angles too (security, bots, DDoS, etc.), and it's not just limited to browsers either.

    End goal seems to be: prove your identity to the megacorps so they can track everything you do and also ensure you are only doing things they approve of. I think the security arguments are just convenient rationalizations in service of this goal.

  • by oefrha on 12/30/24, 1:51 PM

    What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-lived token/cookie with a headless browser on a heavily defended site, then performing requests using said tokens with this lightweight client for a while?

  • by jandrese on 12/30/24, 5:42 PM

    The build scripts on this repo seem a bit cursed. It uses autotools but has you build them in a subdirectory. The default built target is a help text instead of just building the project. When you do use the listed build target it doesn't have the dependencies set up correctly so you have to run it like 6 times to get to the point where it is building the application.

    Ultimately I was not able to get it to build because the BoringSSL disto it downloaded failed to build even though I made sure all of the dependencies the INSTALL.md listed are installed. This might be because the machine I was trying to build it on is an older Ubuntu 20 release.

    Edit: Tried it on Ubuntu 22, but BoringSSL again failed to build. The make script did work better however, only requiring a single invocation of make chrome-build before blowing up.

    Looks like a classic case of "don't ship -Werror because compiler warnings are unpredictable".

    Died on:

    /extensions.cc:3416:16: error: ‘ext_index’ may be used uninitialized in this function [-Werror=maybe-uninitialized]

    The good news is that removing -Werror from the CMakeLists.txt in BoringSSL got around that issue. Bad news is that the dependency list is incomplete. You will also need libc++-XX-dev and libc++abi-XX-dev where the XX is the major version number of GCC on your machine. Once you fix that it will successfully build, but the install process is slightly incomplete. It doesn't run ldconfig for you, you have to do it yourself.

    On a final note, despite the name BoringSSL is huge library that takes a surprisingly long time to build. I thought it would be like LibreSSL where they trim it down to the core to keep the attack surface samll, but apparently Google went in the opposite direction.

  • by zlagen on 12/30/24, 12:59 PM

    In case anyone is interested, I created something similar but for python(using chromium's network stack) https://github.com/lagenar/python-cronet I'm looking for help to create the build for windows.

  • by Sytten on 12/30/24, 3:30 PM

    Thankfully only a small fraction of website does JA3/JA4 fingerprinting. Some do more advanced stuff like correlating headers to the fingerprint. We have been able to get away without doing much in Caido for a long time but I am working on an OSS rust based equivalent. Neat trick, you can use the fingerprint of our competitor (Burp Suite) since it is whitelisted for the security folks to do their job. Only time you will not hear me complain about checkbox security.

  • by Retr0id on 12/30/24, 1:07 PM

    I recently used ja3proxy, which uses utls for the impersonation. It exposes an HTTP proxy that you can use with any regular HTTP client (unmodified curl, python, etc.) and wraps it in a TLS client fingerprint of your choice. Although I don't think it does anything special for http/2, which curl-impersonate does advertise support for.

    https://github.com/LyleMi/ja3proxy

    https://github.com/refraction-networking/utls

  • by peetistaken on 12/30/24, 12:41 PM

    https://github.com/bogdanfinn/tls-client is the go-to package for the go world, it does the same thing

  • by kerblang on 12/30/24, 7:46 PM

    Interesting in light of another much-discussed story about AI scraper farms swamping/DDOSing sites https://news.ycombinator.com/item?id=42549624

  • by TekMol on 12/30/24, 1:23 PM

    What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software?

    When I have to do HTTP requests these days, I default to a headless browser right away, because that seems to be the best bet. Even then, some website are not readable because they use captchas and whatnot.

  • by jollyllama on 12/30/24, 1:27 PM

    >The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser.

    Why is this?

  • by 0x676e67 on 1/2/25, 2:30 PM

    I think someone should need this. It is based on boring tls and makes some fake extensions similar to utls to support Firefox TLS fingerprint imitation

    repo: https://github.com/penumbra-x/rquest

  • by jakeogh on 12/31/24, 8:51 AM

  • by userbinator on 12/30/24, 11:47 PM

    I can't help but think that projects like these shouldn't be posted here, since the enemy is among us. Prodding the bear even more might lead to an acceleration towards the dystopia that others here have already prophesised.

    The following browsers can be impersonated.

    ...unfortunately no Firefox to be seen.

    I've had to fight this too, since I use a filtering proxy. User-agent discrimination should be illegal. One may think the EU could have some power to change things, but then again, they're also hugely into the whole "digital identity" thing.

  • by aninteger on 12/30/24, 2:32 PM

    I think we should list the sites where this fingerprinting is done. I have a suspicion that Microsoft does it for conditional access policies but I am not sure of other services.

  • by londons_explore on 12/30/24, 12:34 PM

    > The resulting curl looks, from a network perspective, identical to a real browser.

    How close is it? If I ran wireshark, would the bytes be exactly the same in the exact same packets?

  • by ape4 on 12/30/24, 1:58 PM

    I like this project!

    Is there a way to request impersonization of the current version of Chrome (or whatever)?