list of tokens?

Is it sensible to define a simple type "list of token".  
Part 2 states clearly that "The lexical space of token is the set of strings that do not contain the line feed (#xA) nor tab (#x9) characters, that have no leading or trailing spaces (#x20) and that have no internal sequences of two or more spaces."
However, the definition of list states that it is "a datatype whose ... ·lexical space· is composed of white space separated lists of literals of the ·itemType·".

I guess the question is, when processing a list, how does a processor work:  does it process characters until it finds the next item separator, or does it process until it finds the first non-itemType (token) character?  

If the latter, then the following is useful, but if the former, then clearly not:

  <simpleType name="tokenListType">
    <annotation>
      <documentation>Token is the set of strings that do not contain the line feed (#xA) nor tab (#x9) characters, that have no leading or trailing spaces (#x20) and that have no internal sequences of two or more spaces.
Tokens may be represented in a list using line feed, tab, or two-or-more consecutive spaces as separators.  
</documentation>
    </annotation>
    <list itemType="token"/>
  </simpleType>


______
Simon.Cox@csiro.au  CSIRO Exploration & Mining
26 Dick Perry Avenue, Kensington WA 6151
PO Box 1130, Bentley WA 6102  AUSTRALIA
T: +61(8) 6436 8639  F: +61(8) 6436 8555  C: +61(4) 0330 2672
http://www.em.csiro.au

Received on Sunday, 27 July 2003 06:51:54 UTC