- From: Tab Atkins Jr. via GitHub <sysbot+gh@w3.org>
- Date: Wed, 07 May 2025 17:06:19 +0000
- To: public-css-archive@w3.org
> Speaking from a Reffy/Webref perspective, we throttle network requests and only fetch the resources that we need to get to the contents of the specs (for example, the crawler already avoids fetching stylesheets, images and additional scripts that specs may embed). If there are further improvements to consider, I'm happy to look into them! The W3C spider isn't the concern, there's just a jillion spammy, badly-behaved spiders that crawl everything (likely for LLM-training purposes, since there's been a dramatic uptick since their popularity increased) and that's what's causing the issues. -- GitHub Notification of comment by tabatkins Please view or discuss this issue at https://github.com/w3c/csswg-drafts/issues/12054#issuecomment-2859381629 using your GitHub account -- Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config
Received on Wednesday, 7 May 2025 17:06:20 UTC