Question about web spiders...

I am not sure if this is the right list to be posting to, but out of the 
dozens of possible lists, this seems like the best one. I also didn't 
subscribe to the list, because I didn't see the link on the page I ended 
up at, <http://lists.w3.org/Archives/Public/www-html/>, so if you could 
please cc me any responses I would appreciate it. :)

My questions is about web spiders. I use a service that spiders my site 
and creates a site map for me. I have links to my subdomains on my main 
page that I do not want the spider to follow. Their help told me to use 
the <nofollow></nofollow> tags so that it wouldn't follow those links. 
This works fine with the spider, but it is not W3 compliant.

The only information I have found anywhere is that I can add a meta tag 
that says nofollow and the spider won't follow any link on that page. 
However, this is not what I want. I just want the spider to not follow 
specific links on the page, but I want it to follow the rest.

Questions:

1) Is there another way to accomplish what I am trying to accomplish?
2) Does the W3C plan to implement the <nofollow> tag or anything like it 
in the near future?

I want to be standards compliant, but I also want to be able to tell a 
spider where it can and can not go.

Thank you.
-- 
Peter Kupfer
peschtra@yahoo.com

Received on Sunday, 26 June 2005 00:48:41 UTC