by xena on 5/9/25, 2:11 PM with 177 comments
by Zdh4DYsGvdjJ on 5/14/25, 6:01 PM
by TheNewsIsHere on 5/14/25, 1:41 PM
From a long-term, clean network I have been consistently seeing these “whoa there!” secondary rate limit errors for over a month when browsing more than 2-3 files in a repo.
My experience has been that once they’ve throttled your IP under this policy, you cannot even reach a login page to authenticate. The docs direct you to file a ticket (if you’re a paying customer, which I am) if you consistently get that error.
I was never able to file a ticket when this happened because their rate limiter also applies to one of the required backend services that the ticketing system calls from the browser. Clearly they don’t test that experience end to end.
by gnabgib on 5/9/25, 4:12 PM
5000 req/hour for authenticated - personal
15000 req/hour for authenticated - enterprise org
According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi...
I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.
by PaulDavisThe1st on 5/14/25, 4:53 PM
Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work.
I'm not arguing that the specific steps Github have taken are the right ones. They might be, they might not, but they do help to address the problem. Our choice for now has been based on noticing that the trawlers are always fetching commits, so we tweaked things such that the overall http-facing git repo works, but you cannot access commit-based URLs. If you want that, you need to use our github mirror :)
by whitehexagon on 5/15/25, 7:29 AM
But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and close my account. Being on the outside, it feels like those walls get taller every month. I often see an interesting project mentioned on HN and clone the repo, but more and more times that is failing. Trying to browse online is now limited, and they recently disabled search without an account.
For such a critical piece of worldwide technology infrastructure, maybe it would be better run by a not-for-profit independent foundation. I guess, since it is just git, anyone could start this, and migration would be easy.
by jorams on 5/14/25, 3:58 PM
Or randomly when clicking through a repository file tree. The first time I hit a rate limit was when I was skimming through a repository on my phone, and about the 5th file I clicked I was denied and locked out. Not for a few seconds either, it lasted long enough that I gave up on waiting then refreshing every ~10 seconds.
by hardwaresofton on 5/15/25, 2:00 AM
At this point knowledge seems to be gathered and replicated to great effect and sites that either want to monetize their content OR prevent bot traffic wasting resources seem to have one easy option.
by jrochkind1 on 5/15/25, 1:00 PM
I forget because I don't use them, but weren't there some products meant as dependency package repositories that github had introduced at some point, for some platforms? Does this apply to them? (I would hope not unless they want to kill them?)
This rather enormously changes github's potential place in ecosystems.
What with the poor announcement/rollout -- also unusual for what we expect of github, if they had realized how much this effects -- I wonder if this was an "emergency" thing not fully thought out in response to the crazy decentralized bot deluge we've all been dealing with. I wonder if they will reconsider and come up with another solution -- this one and the way it was rolled out do not really match the ingenuity and competence we usually count on from github.
I think it will hurt github's reputation more than they realize if they don't provide a lot more context, with suggested workarounds for various use cases, and/or a rollback. This is actually quite an impactful change, in a way that the subtle rollout seems to suggest they didn't realize?
by Animats on 5/15/25, 7:29 AM
I put this on a web site once, and didn't notice for a month that someone was making queries at a frantic rate. It had zero impact on other traffic.
by jrochkind1 on 5/14/25, 9:01 PM
This rather significantly changes the place of github hosted code in the ecosystem.
I understand it is probably a response to the ill-behaved decentralized bot-nets doing mass scraping with cloaked user-agents (that everyone assumes is AI-related, but I think it's all just speculation and it's quite mysterious) -- which is affecting most of us.
The mystery bot net(s) are kind of destroying the open web, by the counter-measures being chosen.
by thih9 on 5/14/25, 7:23 PM
> You have exceeded a secondary rate limit.
Edit and self-answer:
> In addition to primary rate limits, GitHub enforces secondary rate limits
(…)
> These secondary rate limits are subject to change without notice. You may also encounter a secondary rate limit for undisclosed reasons.
https://docs.github.com/en/rest/using-the-rest-api/rate-limi...
by pogue on 5/14/25, 6:34 AM
Or maybe your IP/browser is questionable.
by jhgg on 5/14/25, 9:49 PM
by croemer on 5/15/25, 12:52 AM
Also, neither the new nor the old rate limits are mentioned.
by pdimitar on 5/15/25, 7:19 PM
Why can't people harden their software with guards? Proper DDoS protection? Better caching? Rewrite the hot paths in C, Rust, Zig, Go, Haskell etc.?
It strikes me as very odd, the atmosphere of these threads. So much doom and gloom. If my site was hit by an LLM scraper I'd be like "oh, it's on!", a big smile, and I'll get to work right away. And I'll have that work approved because I'll use the occasion to convince the executives of the need. And I'll have tons of fun.
Can somebody offer a take on why are we, the forefront of the tech sector, just surrendering almost without a single shot?
by Zdh4DYsGvdjJ on 5/14/25, 7:11 AM
by londons_explore on 5/15/25, 4:03 AM
All of public github is only 21TB. Can't they just host that on a dumb cache and let the bots crawl to their heart's content?
by jarofgreen on 5/14/25, 6:51 AM
I encountered this too once, but thought it was a glitch. Worrying if they can't sort it.
by Euphorbium on 5/14/25, 7:09 AM
by trallnag on 5/14/25, 5:11 PM
by jrochkind1 on 5/14/25, 8:57 PM
by mmsc on 5/14/25, 10:25 PM
by spacephysics on 5/14/25, 9:14 PM
by InfiniteLoup on 5/14/25, 3:59 PM
by watermelon0 on 5/14/25, 6:53 AM
by stevekemp on 5/14/25, 5:14 PM
by knowitnone on 5/15/25, 12:13 AM
by micw on 5/14/25, 1:20 PM
by xnx on 5/14/25, 2:07 PM
by jarofgreen on 5/14/25, 7:10 AM
by radicality on 5/14/25, 7:03 AM