Indian language complexities

India has large linguistic diversity with 22 constitutionally recognized languages and 12 scripts.This document is currently focused on the Devanagari script. The expectation is that over time its scope will widen to cover additional major scripts from the list below.

The mapping between languages and scripts is complex. Multiple languages may have common scripts, while a language can be written in multiple scripts. Each language and script is unique in nature and cannot be easily replicated, even if they share common characteristics. The orthographic changes may also occur in some languages and adoption of new orthography is a gradual process, thus posing additional challenges.

The scripts of South Asia share so many common features that a side-by-side comparison of a few will often reveal structural similarities even in the modern letter forms. They are all abugidas in which most symbols stand for a consonant plus an inherent vowel (usually the sound /a/).The North Indian branch of scripts was, like Brahmi itself, mainly used to write Indo-European languages such as Pali and Sanskrit, and eventually the Hindi, Bengali, and Gujarati languages, though it was also the source for scripts for non-Indo-European languages such as Tibetan, Mongolian, and Lepcha. The South Indian scripts are also derived from Brahmi and, therefore, share many similarities in structural characteristics. For more details visit [[!South-Asian-Scripts]].

The following figure shows the evolution of Indian scripts over a period of times from Brahmi script.

Basic components of Indian languages

Unicode Normalization

Unicode normalization[[!UAX15]] is a form of text normalization that transforms equivalent sequences of characters into the same representation. Unicode normalization is important in Unicode text processing applications, because it affects the semantics of comparing, searching, and sorting Unicode sequences

When a unique representation is required , a normalized form of Unicode text can be used to eliminate unwanted distinctions. The key part of normalization is to provide a unique canonical order for visually non distinct sequences of combining characters.

Canonical & Compatible Equivalence

Unicode contains numerous characters to maintain compatibility with existing standards, some of which are functionally equivalent to other characters or sequences of characters. Because of this, Unicode defines some code point sequences as equivalent. Unicode provides two notions of equivalence: canonical and compatible.

Canonical equivalence is a form of equivalence that preserves visually and functionally equivalent characters.

The following figure shows the canonical equivalence:

Unicode Code charts- Devanagari & Devanagari Extended

The following Unicode Character Code chart as per The Unicode Standard, Version 7.0 :

The Unicode code charts for other Indic scripts are available at [[!Code-Charts]]

Need for ABNF valid segmentation

ABNF Valid Segmentation based Indic orthographic syllable definition is provided here for correct and standardized representation of Indian languages layout. This will address various issues mentioned in the following sections.

This definition will be useful in order to get the uniform display of Indic layout in the browsers, applications, Digital publishing etc.

ABNF based definition of Indic orthographic syllable

Augmented Backus–Naur Form (ABNF) is a meta-language based on Backus–Naur Form (BNF), but consisting of its own syntax and derivation rules. The motive principle for ABNF is to describe a formal system of a language to be used as a bidirectional communications protocol.

V[m] |{CH}C[v][m]|CH


The linguistic definition of Indic orthographic syllable has been mapped to ABNF(Augmented Backus–Naur Form) for the purpose of text segmentation, line breaking , drop letter, letter spacing in horizontal text and vertical text representation. The definition has been elaborated , taking Hindi as an example.

The definition is a combination of 3 rules :

Rule 1 : V[m]

Rule 2 : {CH}C[v][m]

Rule 3 : CH (This rule is applicable only at the end of the word)

Various Use cases of ABNF based Indic orthographic syllable definition for Hindi language as example

Rule 1 : V[m]

Hyphenation

There are different cases of hyphenation, some of the cases are given below :

Case 1 : Hyphens are commonly used in Copulative compounds words in Hindi language. Hindi has both prefixes and suffixes which are joined to words with a hyphen.

नर- नारी, लाभ- हानि, माता-पिता, ऊंच - नीच

Case 2: Single word can breaks at the end of the line at Indic orthographic syllable level using hyphen

In the below screenshot, words आकर्षण and विज्ञापन not follow Indic orthographic syllable definition in some of the browsers.

Guiding principles of Line breaking for Indian languages

In Indic writing system , it is preferred that line breaks at word boundaries ,if required following principles may be adhered :

Rule 1: New line cannot begin with following symbols/Punctuation marks. Also these should be retain with the associated text

• Closing brackets
• Devanagari Danda /Purnaviram
• Commas
• Visarga
• Decimal symbols
• Semicolon
• Repetition of punctuation marks such as semicolon with closing brackets, Semicolon with single/ Double quotes , Closing brackets with commas/Semicolon etc
• Mathematical operators

Rule 2: The definition of Indic orthographic syllable may be used to break the line and a hyphen should be at the breaking point so that word can be read intuitively

Rule 3: The hyphenated words can be broken at the hyphen e.g.:

• नर-नारी should be treated as:
• नर- on the first line and नारी on the next line

Rule 4: Expression with mathematical symbol should be treated as single unit so that at the end of the line expression should not breaks at operator level

Rule 5: Breaking should not be allowed at numerical values such as currency values, year etc. e.g.

“100.00” or “10,000”, nor in “12:59”

Initial letter styling

Drop initial is a typographic effect emphasizing the initial letter(s) of a block element with a presentation similar to a 'floated' element.

Typical drop initial usage in Indic scripts

Most of the Indic drop initial letters in magazines and newspapers use 2 to 4 line drops. Some examples are shown below.

In examples of this kind, reference points on the drop cap must align precisely with reference points in the text. In Indic scripts the top reference point is the hanging base line for those scripts that have one, and the bottom alignment point is the text after -edge.

Alignment of the top line of the non-highlighted characters is at the top of the thicker top line of the initial letter is commonly used in India.In some examples top lines of the initial letter and the following letters don't touch. This is due to variable technology/formats used by the publishers. It is preferred that both the top lines of Initial letter and neighbouring text should touch.

Here are some additional examples of initial highlighted letter and drop letter based on the Indic syllable definition. In Devanagari the hanging baseline may be preferred as an alignment point. In that case the primary connection point connects the text-after-edge of the initial letter with the text-after-edge of the nth line, but the secondary connection point connects the hanging baselines of the initial letter and the initial line. The drop initial effect may also be used for writing systems which use different alignment strategies.

The remainder of this section describes the detailed rules for placement and alignment of hindi characters with initial letter styling relative to the adjacent text.

Indic scripts with hanging baseline

Indian languages which use hanging baseline such as Hindi, Bengali, Gujarati, Marathi, Punjabi etc. The part from the hanging baseline and the ascent of the Initial letter may follow the following mechanism :

Scripts that have hanging baseline , the hanging baseline of both Initial letter and first line of text should be same aligned.

Scripts that don't have hanging baseline such as Kannada, Tamil, Telugu, Malayalam , Odia etc

The publishers in India commonly used following rules for such scripts :

• The height of the Initial letter is based on the Indic orthographic syllable described in section 2.
• Ascent of the first non highlighted line is equal t o the median/mean line of the Initial letter as shown below :

Based on above observations the general rule for South Indian languages Indian languages scripts will be :

• ¼ height of the total Drop Cap Height projected or ascended above the ascent of the first-line
• ¾ of the total Drop Cap Height occupied or descended from ascent of first line or X1 to descent of the last line or line XN.

Initial Letter box formatting in Indian languages

The Indian publishers commonly used different height of the boxes and sizes of the characters. But it is proposed that the syllable with in the box is centre-aligned with reference to box parameters as shown in the figure below :

Letter Spacing

In styling issues like horizontal spacing, the spacing between characters like C E R T I F I C A T E, the space is given between the every character in case of English. But in case of Indian language, the space needs to be introduced after each syllable for correct representation.

For letter spacing in Indian languages it is recommended that spacing should follow Indic orthographic syllable definition.

Here is the some examples of letter spacing that based on definition :

Vertical arrangements of characters

In vertical arrangement of characters writing each character on a new line may not be suitable in Indian languages. The vertical arrangements of characters are sometimes used in Indian texts. In order to form correct arrangements, it is preferred to follow tailored grapheme cluster approach. Variations of vertical arrangement of the characters in Hindi is represent below :

Variations in vertical arrangements

Vertical representation of the word 'स्वागतम्' based on Indic orthographic syllable definition:

Collation

Collation is one of the most important features for Indic languages . It determines the order in which a given culture indexes its characters. This is best seen in a dictionary sorting order where for easy search words are sorted and arranged in a specific order. Within a given script, each allo-script may have a different sort-order. Thus in Hindi the conjunct glyph क्ष is sorted along with क , since the first letter of that conjunct is क and on a similar principle ज्ञ is sorted along with ज . The same is not the case with Marathi and Nepali which admit a different sort order.

Different scripts admit different sort orders and for all high end NLP applications. Sorting is a crucial feature to ensure that the applications index data as per the cultural perception of that community. In quite a few States, sort order is clearly defined by the statutory bodies of that state and hence it is crucial that such sort order be ascertained and introduced in the document .

The order(left to right) as given below is pertinent to sorting by a computer program and is compliant with CLDR as laid down by Unicode.

Following is the sort order of Consonant 'क'