Committed new feature from John Punin for handling redirections in webbot

John Punin has provided this new cool feature: Added support for handling
redirection in the robot - that is, now it can follow redirection links
instead of just recording the redirection. This can be enabled using the
-redir command line option

For diffs, see

	http://dev.w3.org/cgi-bin/cvsweb/libwww/Robot/src/#HTRobMan.html

Instructions for how to get it is at

	http://www.w3.org/Library/cvs.html#update

and instructions on how to compile at

	http://www.w3.org/Library/cvs.html#Easy

Henrik

--
Henrik Frystyk Nielsen,
World Wide Web Consortium
http://www.w3.org/People/Frystyk