- From: Peter Linss via GitHub <sysbot+gh@w3.org>
- Date: Sat, 05 Apr 2025 02:44:35 +0000
- To: public-css-archive@w3.org
The "root cause" is that the server is constantly getting hit by hundreds of crawlers that ignore robots.txt. The traffic also spikes when multi-billion dollar companies, who don't contribute to the draft servers at all, post links to drafts in press and social media releases touting their latest features, or even crawl the server themselves to feed their LLMs. While the drafts are currently served by two fairly robust servers, they're not able to handle the load without multiple layers of bot defenses. Rate limiting, followed by IP blocks, has been an effective tool to that end and is not going to get turned off any time soon. The rate limits for regular drafts are currently 1 request per second with bursts of up to 50 requests, images allow bursts of up to 500. Multiple rate limit violations are required before IP blocks take effect. If Ladybird contributors are hitting that limit, then they're either behind a single NAT or proxy, or they need to rethink their draft access. If you want to provide specific IPs, I can look at the logs and see what's going on. I can also add a few IPs, or a small range to the blocking safelist (provided they're not abusing the server themselves). As far as "addressing the root cause", you can either get everyone to stop crawling the site, get someone to start sponsoring the hosting so I can scale up the servers, or find someone to take over hosting it entirely. I'm more than happy for someone else to take over, but anyone offering to do so, or replace the server with a different solution, would be best served by starting out asking what the server is really hosting and what kinds of loads it's under, before proposing half-assed solutions. I'm currently paying about $150/month out of my own pocket for the servers, for no personal gain, and have been doing so for years. And that's not taking my time into account, which has been considerable. So maybe being a bit less demanding here would also be a good place to start. -- GitHub Notification of comment by plinss Please view or discuss this issue at https://github.com/w3c/csswg-drafts/issues/11354#issuecomment-2780154879 using your GitHub account -- Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config
Received on Saturday, 5 April 2025 02:44:36 UTC