[Bug 2766] Word or Token (need clarification)

http://www.w3.org/Bugs/Public/show_bug.cgi?id=2766





------- Additional Comments From pcase@crs.loc.gov  2006-01-25 17:44 -------
This comment amends my previous one which said:

>If we decide we need to keep the word "token" in Section 4, I agree it should 
>be defined, defined as a word returned by a tokenizer used as a search operand.

Remembering that Full Text is part of XQuery and XPath and may someday fold 
into those specs, and knowing the XQuery uses the word "token" for items other 
than words (without defining it), we probably should not use tokens in a more 
restrictive way within Full text, so we probably shouldn't define tokens in 
terms of words within Full Text. 

I recommend always using "word" instead of "token" because it is more specific 
to full-text querying.

Received on Wednesday, 25 January 2006 17:44:22 UTC