from Hacker News

Cleaning up bad bots

by m1guelpf on 9/23/19, 7:07 PM with 111 comments

• by thegeomaster on 9/23/19, 9:52 PM

  Great! Let's add it to the list of obstacles to accessing content for us savages in small, poor, out-of-sight-out-of-mind countries:

  - Harrassment with an extra verification step every god damn time you log in into a website you've been a paying customer at for years because some rando risk model thinks you're a bad guy

  - Harrassment by reCAPTCHA to respond to US-centric image challenges when it decides it's up for some sadistic fun (Mark all images with a store front or street light? Sure, let me just Google how they are "supposed" to look, because I sure as hell ain't seeing any I'm familiar with here.)

  - Blanket IP range bans which serve you a default 503 error page and call it a day, with ripple effects throghout tons of unrelated websites. I always know when my ISP's IP range is added to some new blacklist.

  - Harrassment by Cloudflare "verifying my browser", presumably burning my CPU cycles so they can be sure that my browser, which has been hitting their IP ranges for years with not as much as a cookie wipe, has not suddenly turned into a bot

  - "Your name is invalid": no, your regex is

  - "Our fetishism for credit prevents us from accepting your debit card payment unless you submit a scan of your passport signed in blood"

  - (NEW) CloudFlare's anti-bot measures which will surely not misfire because no one thought to test their shiny new model on the traffic patterns of some culture with < 50 million Internet users

  Apologize for the off-topic rant, but you wouldn't believe how using the Internet for basic things has gotten difficult in the past few years over here (and from what I hear, in many other "forgotten" geographies). No one cares. Long live colonialism!

• by comex on 9/23/19, 8:19 PM

  > Our bot detection breaks down into four large components:

  > - Identification of well known legitimate bots;

  What about non-well-known legitimate bots? If I run my own web crawler, am I at risk of falling into the tarpit (and having my IP address reported)?

• by ikeboy on 9/23/19, 8:35 PM

  >Another trend we have seen is the increase of the combination of bots with botnets, particularly in the world of inventory hoarding bots. The motivation and willingness to spend for these bot operators is quite high.

  >The targets are goods of generally of limited supply and high in demand and in value. Think sneakers, concert tickets, airline seats, and popular short run Broadway musicals. Bot operators who are able to purchase those items at retail can charge massive premiums in aftermarket sales. When the operator identifies a target site, such as an ecommerce retailer, and a specific item, such as a new pair of sneakers going on sale, they can purchase time on the new Residential Proxy as a Service market to gain access to end user machines and (relatively) clean IPs from which to launch their attack.

  They then go on to spout some economic nonsense about how such bots are harmful. Actually, resellers make the market more efficient, and cloudflare is doing a disservice by lumping legitimate bots in with malicious ones like their credential stuffing example.

• by mfontani on 9/23/19, 8:28 PM

  I've toyed with the idea of a tarpit service for badly behaved crawlers which just don't get other hints.

  I'm glad to see that if my services are behind cloudflare, I could just turn something on and let _them_ deal with it.

• by kylehotchkiss on 9/23/19, 7:49 PM

  If any cloudflare staff happen to see this... is Cloudflare Warp coming soon? I was pretty excited to give that a try (and it seemed like something I would be willing to pay for)

• by acolytic on 9/23/19, 8:14 PM

  I'm not very familiar with the concept of tarpitting. How do they get the bot to run CPU intensive code? By passing in extra Javascript? Can this affect a bot that doesn't run any JS?

• by andrerm on 9/23/19, 10:27 PM

  This is Claudfare saying that now that they arbitrated X and everybody said "OK" they are moving from infrastructure into infra and arbitration.

  My question is, what if you are starting an alternative search engine or something legit?

  Edit: my point is, the rules than make a legit or not bot, crawler, scrapper etc. are not clear at all.

• by tony on 9/23/19, 8:46 PM

  Cloudflare, as a service, has it all.

  Nice UX, fast, free. Nice domain service if you transfer to them. Fast DNS management. DOS/bot mitigation. Caching. Quick SSL, and affordable upgrade options. 2FA via TOTP.

  With all the networking/domain stuff momentum build, it would be nice to be able to spin up servers for apps/db.

  Main language at cloudflare is golang? Rust? Any python over there?

  Request: Allow changing the super administrator for cloudflare account more easily. At least for early-stage accounts.

• by bonerman69 on 9/23/19, 8:05 PM

  > for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline;

  What's that mean, 'kicked offline'?

  Isn't scraping 'legal'?