[Bug 2766] Word or Token (need clarification)

http://www.w3.org/Bugs/Public/show_bug.cgi?id=2766





------- Additional Comments From pcase@crs.loc.gov  2006-01-25 14:51 -------
Thanks for raising this discrepancy Joaquin.

Early on, in consultation with the I18n, we decided to use "words" 
not "tokens". It holds more meaning and is less obscure. I still feel strongly 
that we were right in that decision.

We struggled, again with assistance from the I18n, to produce what I think is 
an excellent definition of the word "word". 

In Section 4 we use "TokenInfo" (and define it) and variables such 
as "$searchToken". I would prefer they be "WordInfo" and $searchWord", but can 
live with these. 

In Section 4 I see occurrences of token and search token. I would like to see 
these changed to word, $searchToken, or another variable name where appropriate.

If we decide we need to keep the word "token" in Section 4, I agree it should 
be defined, defined as a word returned by a tokenizer used as a search operand.

I think I am correct that a token is always a word? When we treat phrases, 
sentences, and paragraphs as single units we call them intervals, right?

Received on Wednesday, 25 January 2006 14:51:54 UTC