W3C home > Mailing lists > Public > www-html@w3.org > June 2005

Question about web spiders...

From: Peter Kupfer <peter.kupfer@sbcglobal.net>
Date: Sat, 25 Jun 2005 12:21:50 -0500
Message-ID: <42BD92AE.8050403@sbcglobal.net>
To: www-html@w3.org

I am not sure if this is the right list to be posting to, but out of the 
dozens of possible lists, this seems like the best one. I also didn't 
subscribe to the list, because I didn't see the link on the page I ended 
up at, <http://lists.w3.org/Archives/Public/www-html/>, so if you could 
please cc me any responses I would appreciate it. :)

My questions is about web spiders. I use a service that spiders my site 
and creates a site map for me. I have links to my subdomains on my main 
page that I do not want the spider to follow. Their help told me to use 
the <nofollow></nofollow> tags so that it wouldn't follow those links. 
This works fine with the spider, but it is not W3 compliant.

The only information I have found anywhere is that I can add a meta tag 
that says nofollow and the spider won't follow any link on that page. 
However, this is not what I want. I just want the spider to not follow 
specific links on the page, but I want it to follow the rest.

Questions:

1) Is there another way to accomplish what I am trying to accomplish?
2) Does the W3C plan to implement the <nofollow> tag or anything like it 
in the near future?

I want to be standards compliant, but I also want to be able to tell a 
spider where it can and can not go.

Thank you.
-- 
Peter Kupfer
peschtra@yahoo.com
Received on Sunday, 26 June 2005 00:48:41 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 5 February 2014 07:19:03 UTC