from Hacker News

We need an evolved robots.txt and regulations to enforce it

by andreagrandi on 6/22/24, 3:16 PM with 4 comments

• by Bluestein on 6/22/24, 4:04 PM

  We do. Much in the same way private property is protected, we need regulation enabling the technical means to keep bad actors off private machines.-

  This, back in the quaint, good, ol' days, was sufficiently implemented through the voluntary, good will, communal, neighborly agreement that robot.txt embodies.-

  Unfortunately, sadly, that is no longer enough.-

• by astine on 6/22/24, 3:35 PM

  I agree. Robots.txt is a suitable means of preventing crawlers from accidentally DOSing your site, but it doesn't really give you any protections as to how your content is used by automated services. The current anything-goes approach is just too exploitable.

• by verdverm on 6/22/24, 3:34 PM

  After ranting about AI, the disclaimer is rich

• by nuc1e0n on 6/26/24, 4:53 AM

  There's always range banning.