An Introduction to Writing Systems & Unicode

Arabic and Hebrew scripts usually do not represent short vowel sounds. The languages are so heavily pattern based that readers can adequately guess at the pronunciation of the words.

In circumstances where ambiguity appears, such as the name of the German town Mainz in the example on the slide, short vowels are represented as diacritics attached to the base consonants.

Here, for example, the slide shows the Arabic word for engineer, pronounced ‘muhandis’.

It is actually written, ‘mhnds’.

If needed, the short vowels (there are only 3 in Arabic) are represented as shown on the lower line of the slide. Note that the small circle diacritic indicates NO intervening vowel. (Sequences of code points in Arabic and Hebrew on this and following slides will be shown in left to right order, to emphasise that the underlying order is logical.)

These short vowels are separate combining characters in the text stream that are displayed in the same two-dimensional visual space with the base character. Combining characters do not generally appear without a base character.

 go to top of page

When displaying combining characters, care has to be given to appropriate positioning. In the Thai example on the slide, the same character code is used to represent both of the tone mark glyphs that are circled. There are not two different characters based on the desired visual position. The font has to work out the best position for the glyph according to the run-time visual context.

This slide provides another example of context-sensitive positioning of combining characters.

The short vowel ‘i’ in Arabic is usually drawn below the base character. This is normally the only way of distinguishing it from the short vowel ‘a’, which is displayed above the base character.

In this example, however, an additional shadda diacritic is introduced. The shadda is used to lengthen the consonant it is attached to. In that context it is common (though not mandatory) for the ‘i’ vowel diacritic to appear above the base character, but below the shadda so you can still tell it apart from ‘a’.

Note also that this example introduces the idea that you can have more than one combining character associated with a base character.

 go to top of page

Here is another example of context-sensitive placement, this time in the indic script, Devanagari. The long U vowel sign, pointed to by the arrows, usually appears below the consonant (as shown on the right). However, after a RA (shown on the left), it may appear to the side of the consonant.

 go to top of page

In Indic scripts and scripts derived from them a consonant character carries with it an inherent vowel. The character on the top line on the slide is transcribed ‘ka’, not just ‘k’.

If you want to follow the ‘k’ sound with a different vowel, you append a vowel sign to the consonant character. This vowel sign overrides the inherent vowel with a different sound.

Thai, being derived from Indic scripts, also has vowel signs, although they are used in a slightly more complex way.

In the example on this slide, three vowel signs surround the consonant พ to produce the desired effect.

Whereas in the Indic scripts all vowel signs are combining characters, only one of the vowel signs in this example is combining. The other two (indicated by arrows) are normal spacing characters. This is a distinction introduced to Unicode at the request of the Thai national standards body.

This means that Thai follows a visual, rather than logical, model for positioning of some characters.

 go to top of page

There are many precomposed characters in Unicode that have an accent or diacritic already combined with a base character (such as a-acute above). It is however also possible to represent this character using a simple ‘a’ followed by a combining acute accent. This is referred to as a decomposed character sequence.

 go to top of page

When it comes to implementing combining characters, an important question to ask is what order should be applied to them and the base character. Unless you have agreement on this, you can have serious problems when passing data between systems.

The Unicode Standard requires that all combining characters follow the base consonant in a Unicode string. (So the example to the left on the slide is correct.)

Each combining character has a combining class property expressed as a numeric value. Combining characters that appear in the same location relative to the base character when displayed will typically share the same combining class. For example acute, grave and circumflex accents all appear above the base character and all share the same combining class.

Multiple combining characters do not have to be in any particular order unless they are in one of the Unicode normalisation forms. The standard requires that sequences of combining characters should be treated as equivalent if they all have different combining classes.

Unicode normalisation, however, applies a canonical ordering to multiple combining characters.

If characters have the same combining class they are likely to interact typographically to produce different possible results, as in the case above. In this case the ‘inside-out’ rule is applied. This rule states that the proximity of the combining character in the text stream must match the visual proximity.

 go to top of page

To facilitate the process of string comparison for operations such as searching, sorting and comparison it is helpful to adopt a standard policy with regard to precomposed versus decomposed variants of a character sequence, and the order in which multiple combining characters appear. This can be achieved by applying an appropriate normalization form. The Unicode Standard provides a normalization form called NFD that represents all character sequences in maximally decomposed form. In addition to decomposition, NFD applies a standard order to multiple composing characters attached to a base character. As an alternative, the Unicode Standard offers NFC. NFC is achieved by applying NFD to the text, then re-composing characters for which precomposed forms exist in version 3.0 of the standard.

Note that there are actually some precomposed forms in the Unicode character set that are not generated by NFC, for reasons we will not go into here. In addition, where there is no precomposed form, a character sequence is left decomposed, but canonical ordering is still applied to all combining characters.

The Unicode Standard also offers two more normalization forms, NFKD and NFKC, where K stands for ‘kompatibility’. These forms are provided because the Unicode character set includes many characters merely to provide round-trip compatibility with other character sets. Such characters represent such things as glyph variants, shaped forms, alternative compositions, and so on, but can be represented by other ‘canonical’ versions of the character or characters already in Unicode. Ideally, such compatibility variants should not be used. The NFKD and NFKC normalization forms replace them with more appropriate characters or character sequences. (This, of course, can cause a problem if you intend to convert data back into its original encoding, because you lose the original information.)

 go to top of page

In Hebrew and Greek there are certain characters (only a small number) that look different in the middle of a word and at the end. Two examples are shown on the slide. In each example, the same consonant appears in the middle of a word and at the end of a word in the sample text, and has a different appearance.

Due to traditional approaches, these shapes are encoded separately and are typed in using distinct keys on the keyboard. This is manageable because there are so few such characters.

In other scripts a very different approach has to be taken.

 go to top of page

Arabic is often referred to as a cursive script with the meaning that letters in a word are usually joined to each other – whether handwritten or printed.

The slide shows the unjoined form of the letter AIN at the top right, and, on the left, three joined examples of of the same letter. As you can see, the shape changes quite dramatically.

This slide shows some more examples of un-joined Arabic letters (right column) and their various joining forms (to the left).

It is important to understand that there is only ONE code point here for each letter. The various different visual forms are only font-based glyphs chosen to suit the run-time visual context.

(There are compatibility characters encoded in Unicode for specific joining forms, but these should not be used for storing Arabic text edited in Unicode. They are only provided to allow round-trip conversions between Unicode and legacy character encodings. In Unicode normalized text these are all mapped to the main Unicode Arabic block.)

The shapes on the slide can be referred to (from right to left) as independent, initial, medial and final.

 go to top of page

On previous slides we mentioned the ‘run-time’ context. This is quite important. If you type in the Arabic letter HEH shown at the top of the slide it will initially be in an independent glyph form. If you press exactly the same key on the keyboard and insert exactly the same character alongside it in memory, however, the original letter HEH will be expected to join with the second HEH. The shape of the first HEH will therefore change to ‘initial’, and the second HEH will be in ‘final’ shape. Type another HEH and the second will become ‘medial’, and so on.

In this way Arabic text is constantly changing as you type. The editing application also has to adapt these glyphs as you do things such as backspace, insert or delete text.

 go to top of page

When two Indic consonants appear together without any intervening vowel sound they may form a conjunct, ie. the consonant cluster is rendered as a composite shape. This composite shape may show a vertical or horizontal mixture of the base shapes. In some cases the original constituents of a conjunct may not be recognizable.

One approach that is very common is the use of a half-form to represent the initial consonant in the cluster. An example of this is shown on the bottom line of the slide.

It is important to bear in mind, once again, that this is all glyph magic. The individual consonants are all still represented using the regular code points in memory, it is only the visual appearance that changes. There are no special code points for half-form glyphs. The alternate glyph appropriate for the context is simply applied at display time according to the rendering rules of the script.

In fact, there is a vital ingredient to a conjunct form that we have not yet discussed. It has various script-specific names, but here we will use the generic term 'virama'. The virama is often also called ‘vowel killer’.

If you simply put two consonants side by side in Unicode, as in the top line on the slide, you will get two separate consonants displayed (with the assumption on the part of the reader that there is an inherent vowel between them).

It is only when you put a virama character between them that they combine to form a conjunct. So the conjunct glyph shown middle right actually represents three underlying characters.

The number of conjunct forms can vary from font to font. Some fonts will be capable of rendering more than others. So what happens if the font you are using doesn’t have a conjunct glyph for the combination you want to create? In such a case the virama is shown visually as a combining mark – see the last line on the slide. (In fact, in modern Tamil this is the default approach.)

 go to top of page

The concepts we have discussed so far in this section on combining characters and glyph shaping have shown that there is no one-to-one correspondence, as there usually is in English, between the characters in memory and the glyphs displayed on screen. Indeed, sometimes complex rules are needed to determine the displayed result.

We have seen some of the more basic transformation cases, but over the next few slides we will take a quick look at some additional possibilities. This is by no means intended to give you all the information you need to implement these scripts – merely expose you to some slightly more advanced behavior.

Half-form glyphs in Indic scripts express the absence of a following vowel. This is a systematic and functional use of special shaping forms. The variation in shape carries a meaning to the reader.

 go to top of page

In complex scripts it is common to find variations in glyph shape for the same underlying character, depending on the context in which the character appears. In this slide, the difference occurs for practical reasons: the underlying base character is wider in one case than the other, and the glyph that reaches over it has to therefore be slightly wider also.

 go to top of page

This slide shows similar changes in the Myanmar (Burmese) script, as the vowel sign changes shape to fit with the glyphs in the remainder of the syllable.

 go to top of page

Still with the Myanmar script, we see in this slide how the Burmese 'asat' combining character changes shape when associated with a tall base character.

The important thing to remember, in all of these cases, is that there is no change to the underlying characters in use. There is only one asat character, and the shape used to display it is picked from the font's repertoire of glyphs at run time according to rules encoded in the font. The font, therefore, has many more glyphs than there are characters in the script.

 go to top of page

This slide shows a word with and without some additional ligated forms in Arabic.