checklink: 3.9.3 beta, feedback requests

I have committed the stuff I had on my mind for link checker 3.9.3. 
http://qa-dev.w3.org/wlc/checklink is up to date.

Regarding the beta announcement, I would like to request feedback in
particular of the following (feel free to rephrase, comment, and add
items):

- Robots exclusion standard support in the link checker.  Good or bad?

- Should the "Check linked documents recursively" checkbox be removed?
  Recursion is implicit if there's anything in the "depth" field, so
  the checkbox is sort of redundant.

Changelog since 3.9.2 is attached.  The corresponding in-a-nutshell, or
NEWS-like list (excluding command line only changes) would be something
like:

- Support for the robots exclusion standard has been added.
- Sleep 1 second between requests to servers.
- Cookie handling logic is now more robust and predictable.
- Harmonized layout with the Markup Validator.
- Do not send an Accept-Language header at all if the client did not
  send us one.  Previously, "Accept-Language: *" was sent by default.
- Miscellaneous markup, layout, and documentation improvements, minor
  bug fixes and warning cleanups.