W3C home > Mailing lists > Public > www-international@w3.org > October to December 2004

Re: What is a language detection algorithm?

From: KUROSAKA Teruhiko <kuro@bhlab.com>
Date: Thu, 11 Nov 2004 14:39:59 -0800
Message-ID: <4193EA3F.3010306@bhlab.com>
To: Frank Yung-Fong Tang <ytang0648@aol.com>
CC: "smj (by way of Martin Duerst <duerst@w3.org>)" <smj1@sndi.net>, www-international@w3.org

Frank,

> Not really. A 4-gram in byte space will be ~ half a pair of DBCS 
> characters and ~ half of one DBCS characters with one trail byte of the 
> previous character and a lead byte of the next character. This won'b be 
> a big deal in the case of Shift-JIS or Big5 since their lead byte and 
> trail byte use quite different range. However for GB2312, EUC-KR, 
> EUC-TW, EUC-JP, it is big problem since the lead byte and trail byte for 
> the most common characters are using the same range (0xa1-0xfe).

The byte-based N-gram algorithm is based on the statistical
byte patters, and it does not need to understand the character
boundaries at all.  For the purpose of N-gram, the Japanese
text in EUC-JP and Japanese text in Shift_JIS are treated
as though they were different languages.  In other words,
the algorithm detects a language and the character encoding
combination.

-- 
KUROSAKA ("Kuro") Teruhiko, San Francisco, California, USA
Internationalization Consultant
http://www.bhlab.com/
Received on Thursday, 11 November 2004 22:40:30 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 2 June 2009 19:17:04 GMT