Goals and Scope

The goal of the Character Model for the World Wide Web is to facilitate use of the Web by all people, regardless of their language, script, writing system, and cultural conventions, in accordance with the W3C goal of universal access. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.

This document builds on Character Model for the World Wide Web: Fundamentals [[!CHARMOD]]. Understanding the concepts in that document are important to being able to understand and apply this document successfully.

This part of the Character Model for the World Wide Web covers string matching—the process by which a specification or implementation defines whether two string values are the same or different from one another. It describes the ways in which texts that are semantically equivalent can be encoded differently and the impact this has on matching operations important to formal languages (such as those used in the formats and protocols that make up the Web). Finally, it discusses the problem of substring searching within documents.

The main target audience of this specification is W3C specification developers. This specification and parts of it can be referenced from other W3C specifications and it defines conformance criteria for W3C specifications, as well as other specifications.

Other audiences of this specification include software developers, content developers, and authors of specifications outside the W3C. Software developers and content developers implement and use W3C specifications. This specification defines some conformance criteria for implementations (software) and content that implement and use W3C specifications. It also helps software developers and content developers to understand the character-related provisions in W3C specifications.

The character model described in this specification provides authors of specifications, software developers, and content developers with a common reference for consistent, interoperable text manipulation on the World Wide Web. Working together, these three groups can build a globally accessible Web.

Structure of this Document

This document defines two basic building blocks for the Web related to this problem. First, it defines rules and processes for String Identity Matching in document formats. These rules are designed for the identifiers and structural markup (markup) used in document formats to ensure consistent processing of each and are targeted to Specification writers. Second, it defines broader guidelines for handling user visible text (the "Shakespeare"), such as natural language text that forms most of the content of the Web. This section is targeted to implementers.

This document is divided into three main sections.

The first section lays out the problems involved in string matching; the effects of Unicode and case folding on these problems; and outlines the various issues and normalization mechanisms that might be used to address these issues.

The second section provides requirements and recommendations for string identity matching for use in formal languages, such as many of the document formats defined in W3C Specifications. This primarily is concerned with making the Web functional and providing document authors with consistent results.

The third section discusses considerations for the handling of content by implementations, such as browsers or text editors on the Web. This mainly is related to how and why to preserve the author's original sequences and how to search or find content in natural language text.

Background

This section provides some historical background on the topics addressed in this specification.

At the core of the character model is the Universal Character Set (UCS), defined jointly by the Unicode Standard [[!UNICODE]] and ISO/IEC 10646 [[!ISO10646]]. In this document, Unicode is used as a synonym for the Universal Character Set. A successful character model allows Web documents authored in the world's writing systems, scripts, and languages (and on different platforms) to be exchanged, read, and searched by the Web's users around the world.

The first few chapters of the Unicode Standard [[!UNICODE]] provide useful background reading.

For information about the requirements that informed the development of important parts of this specification, see Requirements for String Identity Matching and String Indexing [[CHARREQ]].

Terminology and Notation

This section contains terminology and notation specific to this document.

The Web is built on text-based formats and protocols. In order to describe string matching or searching effectively, it is necessary to establish terminology that allows us to talk about the different kinds of text within a given format or protocol, as the requirements and details vary significantly.

Unicode code points are denoted as U+hhhh, where hhhh is a sequence of at least four, and at most six hexadecimal digits. For example, the character € EURO SIGN has the code point U+20AC.

Some characters that are used in the various examples might not appear as intended unless you have the appropriate font. Care has been taken to ensure that the examples nevertheless remain understandable.

A legacy character encoding is a character encoding not based on the Unicode character set.

A grapheme cluster is a sequence of one or more Unicode characters that form a single user-perceived "character".A grapheme is a sequence of one or more characters that form a single user-perceived "character". Unicode Standard Annex #29: Text Segmentation defines a unit called the grapheme cluster which is intended to approximately match user-perceived graphemes. A grapheme cluster roughly corresponds to the user's perception of where the character boundaries occur in a visually rendered text, rather than to the Unicode code points used to encode the text in a string. A discussion of grapheme clusters is given at the end of Section 2.10 of the Unicode Standard, [[!UNICODE]]; a formal definition is given in Unicode Standard Annex #29 [[!UTR29]]. What the Unicode Standard actually defines is default grapheme clustering. Some languages require tailoring to this default. For example, a Slovak user might wish to treat the default pair of grapheme clusters "ch" as a single grapheme cluster. Note that the interaction between the language of string content and the end-user's preferences might be complex. We should standardise the way we create ids for terms. Either camelCase or with hyphens. With or without def- at start.

This document uses the word grapheme as a shorter synonym for a grapheme cluster. References to a grapheme or graphemes refer to user-perceived text units as described above.

Natural language content refers to the language-bearing content in a document and not to any of the surrounding markup or identifiers that form part of the document structure. You can think of it as the actual "content" of the document or the "message" in a given protocol. Note that the natural language content can include items such as the document title ("Much Ado About Nothing") as well as prose content within the document.

Markup is any text in a document format or protocol that belongs to the structure or language of the format or protocol. This definition can include values that are not typically thought of as "markup", such as the name of a field in an HTTP header, as well as all of the characters that form the structure of a format or protocol. For example, < or > are part of the markup in an HTML document.

Markup usually is defined by a specification or specifications and includes both the defined, reserved keywords for the given protocol or format as well as string tokens and, identifiers and enumerated values that are defined by document authors to form the structure of the document (rather than the "content" of the document).

A resource is a given document, file, or protocol "message" which includes both the natural language content) as well as the markupmarkup such as identifiers surrounding or containing it. For example, in an HTML document that also has some CSS and a few script tags with embedded JavaScript, the entire HTML document, considered as a file, is the resource.

A vocabulary provides the list of reserved names as well as the set of rules and specifications controlling how user values (such as identifiers) can be assigned in a format or protocol. This can include restrictions on range, order, or type of characters that can appear in different places. For example, HTML defines the names of its elements and attributes, which defines the "vocabulary" of HTML markup. ECMAScript restricts the range of characters that can appear at the start or in the body of an identifier or variable name (while different rules apply to the values of, say, string literals).

Following example has accessibility issues and is hard to use in hard copy.

Conformance

This specification places conformance criteria on specifications, on software (implementations) and on Web content. To aid the reader, all conformance criteria are preceded by '[X]' where 'X' is one of 'S' for specifications, 'I' for software implementations, and 'C' for Web content. These markers indicate the relevance of the conformance criteria and allow the reader to quickly locate relevant conformance criteria by searching through this document.

Specifications conform to this document if they:

1. do not violate any conformance criteria preceded by [S] where the imperative is MUST or MUST NOT,

2. document the reason for any deviation from criteria where the imperative is SHOULD, SHOULD NOT, or RECOMMENDED,

3. make it a conformance requirement for implementations to conform to this document,

4. make it a conformance requirement for content to conform to this document.

Software conforms to this document if it does not violate any conformance criteria preceded by [I].

Content conforms to this document if it does not violate any conformance criteria preceded by [C].

Where this specification contains a procedural description, it is to be understood as a way to specify the desired external behavior. Implementations can use other means of achieving the same results, as long as observable behavior is not affected.

Case Folding

Some scripts and writing systems havemake a distinction between UPPER, lower, and Title case characters. Examples of such scripts include the Latin script used in the majority of this document, as well as scripts such as Greek, Armenian or Cyrillic. Most scripts, such as the Brahmic scripts of India, the Arabic script, and the non-Latin scripts used to write Chinese, Japanese, or Korean do not have a case distinction. you could also say: most don't but a number of important ones do, such as...

Some document formats or protocols seek to aid interoperability or provide an aid to content authors by ignoring case variations in the vocabulary they define or in user-defined values permitted by the format or protocol. For example, this occurs when matching class names betweenin an HTML document and selectors in its associated style sheet. Consider this HTML fragment:

<style type="text/css">

  SPAN.h\e9llo {
     color: red;
  }
</style>

<span class="h&#xe9;llo">Hello World!</span>

The SPAN in the stylesheet matches the span we should mark up things like span as keywords, rather than just code, and put translate=no attributes on them; see examples in our articles element in the document, even though one is uppercase and the other is not.

The process of making two texts identical which differ in case but are otherwise "the same" identical is called "case folding". Case folding make this a term definition! might, at first, appear simple. However there are variations that need to be considered when treating the full range of Unicode in diverse languages. For more information, Unicode [[!UNICODE]] Section 5.18 discusses case folding in detail.

Case folding in Unicode has a number of side-effects or potential side-effects on the processing of a resource. One is that case folding may not preserve the length of the original text: some mappings increase or decrease the total number of characters needed. In addition, case folding removes information from a string which cannot be recovered later.

Another aspect of case folding is that it can be language sensitive. Unicode defines default case mappings for each encoded character, but these are only defaults and are not appropriate in all cases. Some languages need case-folding to be tailored to meet specific linguistic needs. One common example of this are Turkic languages written in the Latin script.

Case-sensitive matching: code points are compared directly with no case folding. Case-sensitive matching is RECOMMENDED as the default for any new protocol or format.this needs mustardising

Case-sensitive matching is the easiest to implement and introduces the least potential for confusion, since it generally consists of a comparison of the underlying Unicode code point sequence. Because it is not affected by considerations such as language-specific case mappings, it produces the least surprise for document authors that have included words (such as the Turkish example above) in their markup.

The different forms of case-insensitive matching it's very easy to read "different forms of case-sensitive matching", which doesn't make sense - perhaps we should make more of the fact that we're now moving on to something different are useful in contexts where case may vary in a way that is not semantically meaningful or in which case distinctions cannot be controlled by the user. This is particularly true when searching a document, but also applies when defining rules for matching user- or content-generated values, such as identifierssounds like we're encouraging use for ids. i think the next sentence belongs with the next para. When defining a vocabulary, one important consideration is whether the values are restricted to the ASCII subset of Unicode or if they permit characters (such as accents on Latin letters or a broad range of Unicode including non-Latin scripts) which potentially have a more complex case folding.

ASCII case-insensitive matching compares a sequence of code points as if all ASCII code points in the range 0x41 to 0x5A (A to Z) were mapped to the corresponding code points in the range 0x61 to 0x7A (a to z). When a vocabulary is itself constrained to ASCII, ASCII case-insensitive matching can be required. However, if the vocabulary is not restricted to ASCII or permits user-defined values that use a broader range of Unicode, ASCII case-insensitive matching must not be required.more mustard

Unicode case-insensitive matching compares a sequence of code points as-if one of the Unicode-defined language-independent default case folding formscrikey! let me read that again... maybe 3 or 4 times... (see [[!UNICODE]], Section 5.18) had been applied to both input sequences. These forms are:

1. Unicode Full Casefold (Unicode C+F). This case-fold form is RECOMMENDED for Web specifications and implementations.<-mustardise me! This case-fold uses the "common" plus the "full" case-fold.whatever that means...
2. Unicode Simple Casefold (Unicode C+S). Unicode also defines a "common" plus "simple" case-fold. This form is not appropriate for string identity matching on the Web.<-mustardise me!

The above needs significant expansion - it's some of the stuff that we need to explain most clearly to spec developers and implementers. We should also add that the case folding algorithms are somewhat different from the simple upper and lower case conversions.

Language-sensitive case-sensitive matching is useful in the rare case where a document format or protocol contains information about the language of the markup and where language-sensitive case folding might sensibly be applied. In these cases, tailoring of the Unicode case-fold mappings above to match the expectations of that language SHOULD be specified and applied.<-mustardise me! These case-fold mappings are defined in the Common Locale Data Repository [[UAX35]] project of the Unicode Consortium.but we may need to help people to find them! :(

However, language-sensitive case-sensitive matching in document formats and protocols is NOT RECOMMENDED because language information can be hard to obtain, verify, or manage and the resulting operations can produce results that frustrate users.<-mustardise me!

Unicode Normalization

Other kinds of variations can occur in Unicode text: some "characters" or<-not clear whether this 'or' is an opposition or elaboration graphemes <-link to the new definition i provided can be represented by several different Unicode code point sequences. Consider the character Ǻ LATIN LETTER CAPITAL A WITH RING ABOVE AND ACUTE. Here are some of the different ways that an HTML document could represent this character:

We should add a fully decomposed example, at least.

As in the first examplewhich?, each of the above strings contains the same apparent semantic meaning (Ǻactually the name of the codepoint may be better here), but each one is encoded slightly differently. More variations are possible, but are omitted for brevity: for example, any of the characters could be replaced with a character escape i don't think escapes are relevant here.

Because applications need to find the semantic equivalence in texts that use different code point sequences, Unicode defines a means of making two semantically equivalent texts identical: the Unicode Normalization Forms [[!UAX15]].

Document formats or protocols are sensitive to these variations because their specifications and implementations on the Web generally do not supply Unicode Normalization of the content being exchanged or in the string matching algorithms used when processing the markup and content later. Users and resourcescontent authors need to ensure that they have provided a consistent representation in order to avoid problems later. It can be difficult for users to assure that a given resource or set of resources uses a consistent textual representation because the differences are usually not visible when viewing a resource as text. Tools and implementations thus need to consider the difficulties experienced by users when visually or logically equivalent strings that "ought to" match (in the user's mind) are considered to be distinct values. Providing a means for users to see these differences and/or normalize them as appropriate makes it possible for end users to avoid failures that spring from invisible differences in their source documents. For example, the W3C Validator warns when an HTML document is not fully in Unicode Normalization Form C.some of these sentences are complicated and the para makes more than one point - i think we need to rewrite it a bit. Here's a quick suggestion below...

Document formats or protocols are often susceptible to the effects of these variations because their specifications and implementations on the Web do not require Unicode normalization of the text, nor do they take into consideration the string matching algorithms used when processing the markup and content later. For this reason, content developers need to ensure that they have provided a consistent representation in order to avoid problems later.

However, it can be difficult for content developers to assure that a given resource or set of resources uses a consistent textual representation because the differences are usually not visible when viewed as text. Tools and implementations need to consider the difficulties experienced by users when visually or logically equivalent strings that "ought to" match (in the user's mind) are considered to be distinct values. Providing a means for users to see these differences and/or normalize them as appropriate makes it possible for end users to avoid failures that spring from invisible differences in their source documents. For example, the W3C Validator warns when an HTML document is not fully in Unicode Normalization Form C. There's some mustard missing here wrt tools.

Character Escapes

Document formats or protocols also generallyoften provide escaping mechanisms to permit the inclusion of characters that are otherwise difficult to input, process, or encode. These escape mechanisms provide an additional equivalent means of representing characters inside a given resource. TheseThey also allow for the encoding of Unicode characters not represented in the character encoding scheme used by the document or for convenience of the editor.

For example, € (U+20AC EURO SIGN) can also be encoded in HTML as the hexadecimal entity € or as the decimal entity €. In a JavaScript or JSON file, it can appear as \u20ac while in a CSS stylesheet it can appear as \20ac. All of these representations encode the same literal character value: "€".better use nbsp, which has some justification for being escaped - the euro sign typically does not

Character escapes are normally interpreted before a document is processed and strings within the format or protocol are matched. Returning to an example we used above:

<style type="text/css">

  span.h\e9llo {
     color: red;
  }
</style>

<span class="h&#xe9;llo">Hello World!</span>

You would expect that text to display like the following: Hello world!this is lost when printed in b&w- use different styling

In order for this to work, the user-agent (browser) had to match two strings representing the class name héllo, even though the CSS and HTML each used a different escaping mechanism. The above fragment demonstrates one way that text can vary and still be considered "the same" according to a specification: the class name h\e9llo matched the class name in the HTML mark-up héllo (and would also match the literal value héllo using the code point U+00E9).

Unicode Controls and Invisible Markers

Unicode provides a number of special purpose control characters or invisible markers that help document authors control the appearance or performance of text. TIn poorly implemented applications, these characters interfere with string matching when they are not semantically part of the text but do form part of the encoded character sequence.

A special case is for ZWJ and ZWNJ. These invisible controls sometimes affect meaning.

Examples of these include:

This section was added and needs further fleshing out. The requirement probably wants to live in the requirements section. 2015-02-07AP

Legacy Character Encodings

add something along the lines of 'you really should use Unicode, and then you can forget about this section, but ... Finally, resources can use different character encoding schemes, including legacy character encodings, to serialize document formats on the Web. Each character encoding scheme uses different syntax?, byte values, and sequences to represent a given subset of the Universal Character Set.

For example, € (U+20AC EURO SIGN) is encoded as 0x80 in the windows-1252 encoding, but as the byte sequence 0xE2.82.AC in UTF-8.

Specifications mainly address these resulting variations by considering each document to be a sequence of Unicode characters after converting from the document's character encoding (be it a legacy character encoding or a Unicode encoding such as UTF-8) and then unescaping any character escapes before proceeding to process the document.

The following paragraphs about normalization transcoders are "at risk". The WG feels that this requirement is difficult for content authors or implementers to verify. Needed action: verify if all of [[Encoding]] spec's transcoders are normalizing.

Even within a single legacy character encoding there can be variations in implementation. One famous example is the legacy Japanese encoding Shift_JIS. Different transcoder implementations faced choices about how to map specific byte sequences to Unicode. So the byte sequence 0x80.60 (0x2141 in the JIS X 0208 character set) was mapped by some implementations to U+301C WAVE DASH while others chose U+FF5E FULL WIDTH TILDE. This means that two reasonable, self-consistent, transcoders could produce different Unicode character sequences from the same input. The [[Encoding]] specification exists, in part, to ensure that Web implementations use interoperable and identical mappings. However, extant transcoders might be applied to documents found on the Web.

For content authors and implementations, it is RECOMMENDED that conversions from legacy character encodings use a "normalizing transcoder".mustardise me!

A normalizing transcoder is a transcoder that converts from a legacy character encoding to a Unicode encoding form and ensures that the result is in Unicode Normalization Form C. For most legacy character encodings, it is possible to construct a normalizing transcoder (by using any transcoder followed by a normalizer); it is not possible to do so if the encoding's repertoire contains characters not represented in Unicode.

The Matching Algorithm

This section defines the algorithm for matching strings. String identity matching MUST be performed as if the following steps were followed:

1. Conversion to a common Unicode encoding form of the strings to be compared [[Encoding]].

2. Expansion of all character escapes and includes.

   The expansion of character escapes and includes is dependent on context, that is, on which markup or programming language is considered to apply when the string matching operation is performed. Consider a search for the string 'suçon' in an XML document containing suçon but not suçon. If the search is performed in a plain text editor, the context is plain text (no markup or programming language applies), the &#xE7; character escape is not recognized, hence not expanded and the search fails. If the search is performed in an XML browser, the context is XML, the character escape (defined by XML) is expanded and the search succeeds.

   An intermediate case would be an XML editor that purposefully provides a view of an XML document with entity references left unexpanded. In that case, a search over that pseudo-XML view will deliberately not expand entities: in that particular context, entity references are not considered includes and need not be expanded

3. Perform the appropriate case foldingone of the following case foldings, as appropriate:
   1. Case sensitive: Go to step 4.
   2. ASCII case folding: map all code points in the range 0x41 to 0x5A (A to Z) to the corresponding code points in the range 0x61 to 0x7A (a to z).
   3. Unicode case folding: map all code points to their Unicode C+F case fold equivalents. Note that this can change the length of the string.
4. Remove Unicode control characters.
5. Test the resulting sequences of code points bit-by-bit for identity.

Requirements for String Identity Matching

In the Web environment, where multiple character encodings are used to represent strings, including some character encodings which allow multiple representations for the same thing, it's important to establish a consistent process for evaluating string identity.sounds like unnecessary fluff

One main consideration in string identity matching is whether the comparison is case sensitive or case insensitive.

However, cases exist in which case-insensitivity is desirable.

Where case-insensitive matching is desired, there are several implementation choices that a formal language needs to consider. If the vocabulary of strings to be compared is limited to the Basic Latin (ASCII) subset of Unicode, ASCII-case-insensitive matching MAY be used.because?

If the vocabulary of strings to be compared is not limited, then ASCII case-insensitive matching MUST NOT be used. Unicode case-insensitive matching MUST be applied, even if the vocabulary does not allow the full range of Unicode.

Unicode case-insensitive matching can take several forms. Unicode defines the "common" (C) casefoldings for characters that always have 1:1 mappings of the character to its case folded form and this covers the majority of characters that have a case folding. A few characters in Unicode have a 1:many case folding. This 1:many mapping is called the "full" (F) case fold mapping. For compatibility with certain types of implementation, Unicode also defines a "simple" (S) case fold that is always 1:1.this para belongs in the explanatory section about case-folding?

A vocabulary is considered to be "ASCII-only" if and only if all tokens and identifiers are defined by the specification directly and these identifiers or tokens use only the Basic Latin subset of Unicode. If user-defined identifiers are permitted, the full range of Unicode characters (limited, as appropriate, for security or interchange concerns, see [[UTR36]]) SHOULD be allowed and Unicode case insensitivity used for identity matching.

Note that an ASCII-only vocabulary can exist inside a document format or protocol that allows a larger range of Unicode in identifiers or values.

Insert example from CSS here.

Considerations for Matching Natural Language Content

This section was identified as a new area needing document as part of the overall rearchitecting of the document. The text here is incomplete and needs further development. Contributions from the community are invited.

Searching content (one example is using the "find" command in your browser) generates different user expectations and thus has different requirements from the need for absolute identity matching needed by document formats and protocols. Searching text has different contextual needs and often provides different features.

One description of Unicode string searching can be found in Section 8 (Searching and Matching) of [[UTS10]].

One of the primary considerations for string searching is that, quite often, the user's input is not identical to the way that the text is encoded in the text being searched. Users generally expect matching to be more "promiscuous", particularly when they don't add additional effort to their input. For example, they expect a term entered in lowercase to match uppercase equivalents. Conversely, when the user expends more effort on the input—by using the shift key to produce uppercase or by entering a letter with diacritics instead of just the base letter—they expect their search results to match (only) their more-specific input.

This effect might vary depending on context as well. For example, a person using a physical keyboard may have direct access to accented letters, while a virtual or on-screen keyboard may require extra effort to access and select the same letters.

Consider a document containing these strings: "re-resume", "RE-RESUME", "re-résumé", and "RE-RÉSUMÉ".

In the table below, the user's input (on the left) might be considered a match for the above items as follows:

In addition to variations of case or the use of accents, Unicode also has an array of canonical equivalents or compatibility characters (as described in the sections above) that might impact string searching.

For example, consider the letter "K". Characters with a compatibility mapping to U+004B LATIN CAPITAL LETTER K include:

1. Ķ U+0136
2. Ǩ U+01E8
3. ᴷ U+1D37
4. Ḱ U+1E30
5. Ḳ U+1E32
6. Ḵ U+1E34
7. K U+212A
8. Ⓚ U+24C0
9. ㎅ U+3385
10. ㏍ U+33CD
11. ㏎ U+33CE
12. Ｋ U+FF2B
13. (a variety of mathematical symbols such as U+1D40A,U+1D43E,U+1D472,U+1D4A6,U+1D4DA)
14. 🄚 U+1F11A
15. 🄺 U+1F13A.

Other differences include Unicode Normalization forms (or lack thereof). There are also ignorable characters (such as the variation selectors), whitespace differences, bidirectional controls, and other code points that can interfere with a match.

Users might also expect certain kinds of equivalence to be applied to matching. For example, a Japanese user might expect that hiragana, katakana, and half-width compatibility katakana equivalents all match each other (regardless of which is used to perform the selection or encoded in the text).

When searching text, the concept of "grapheme boundaries" and "user-perceived characters" can be important. See Section 3 of Character Model for the World Wide Web: Fundamentals [[!CHARMOD]] for a description. For example, if the user has entered a capital "A" into a search box, should the software find the character À (U+00C0 LATIN CAPITAL LETTER A WITH ACCENT GRAVE)? What about the character "A" followed by U+0300 (a combining accent grave)? What about writing systems, such as Devanagari, which use combining marks to suppress or express certain vowels?