Ok. I'll do the manual thing I'm doing. 1. Can I have the proper robots.txt config that should be at the root. I read from Ryan that this might help wit the load generated by crawlers because it is indexing unneeded and heavy database hits pages. 2. My current status is to finalize full import into git of the web application to have it run locally and then tweak and test them prior to make such big changes. Regards Renoir Boulanger | Developer operations engineer W3C | webplatform.org http://w3.org/people/#renoirb ✪ https://renoirboulanger.com ✪ @renoirb ~Received on Monday, 19 August 2013 17:09:34 UTC
This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:13:52 UTC