r12a >> blog

>> Use it

>> Read it

1. paulmcj Says:
   May 12th, 2010 at 10:46 am e

   Hi Guys,

   I like your pickers and what is appealing even more is that you’ve added transliteration, which is a very nice feature for any picker.

   Paul

>> Use it

>> Use it

I just received email from Derek Reid in the XMetal team at JustSystems to say that they have significantly improved the way the XMetal XML editor uses xml:lang attributes in the source code in conjunction with its spell-checker.

Basically, XMetal will switch spell-checking dictionaries based on the xml:lang settings in the markup. It also supports xml:lang=”” and xml:lang=”zxx” for places you don’t want to spell-check. It even does this when using interactive red squiggles to highlight potential misspellings.

I wrote a blog post about this in 2007, when the capability was only partially developed. Derek says:

I read this post when you first wrote it, and after getting feedback from a large number of our clients I was finally able to convince our development and management teams to properly support language auto-switching for spell checking in conjunction with xml:lang attribute values in our product.

We made a big effort to deal with these limitations during the past year and our XMetaL Author Enterprise 6.0 release addresses most or all of them.

If you are interested, I have posted instructions on how to configure XMetaL Author Enterprise 6.0 to properly support this feature:
http://forums.xmetal.com/index.php/topic,539.msg1701

I haven’t had a chance to try it out yet, but it sounds exciting. Now how about DreamWeaver…

>> Use it

About the tool: BCP 47 language tags are built from subtags in the IANA Subtag Registry. This tool helps you find or look up subtags and check for errors in language tags. It also provides information to guide your choices.

Latest changes: I reworked the informational text that accompanies macrolanguages, their encompassed languages, and extlang subtags. As part of that, I changed the code to allow for highlighting of specific cases. For example, where legacy may dictate that the macrolanguage subtag (zh) is more useful for Mandarin Chinese than the more specific tags (cmn or zh-cmn).

I simplified the intro to the page, but added a link to the new article Choosing a Language Tag, which provides useful step-by-step guidelines on creating language tags.

I also changed the user interface somewhat. The input fields are easier to work with and take up less vertical space. Also, you can now submit a query by simply hitting return after typing into a field. I had originally required you to click on a submit button so that all values in other fields would be retained when the answer is shown – this was so that while checking various subtags you could build up a language tag in the Check field for later checking. I just found that the annoyance of continually having to resubmit after forgetting to click on the submit button wasn’t worth the extra functionality (and I was also encouraged to do so by feedback from Bert Bos).

>> See what it can do

>> Use it

It took me a while to find the time, but I have finally upgraded UniView to suport the final 5.2 release of Unicode, plus a few extra features.

The order of blocks listed in the top left pulldown menu was changed to ressemble the order in the Unicode Charts page. Several sub-block selections were also added to the list (as in the Unicode page), and are displayed in italics.

When you display details of a character in the right panel, the heading Script group has now been used to indicate the sub-block-level headings in the block listings of the Unicode Standard. The link to the Unicode block now follows the heading Unicode block. These sub-block-level headings are also shown when you display a range as a list (as opposed to a matrix).

When you mouse over characters displayed in a matrix, the codepoint and name information for that character now appear just above the matrix. This makes it much easier to locate characters you are looking for.

Finally, but by no means least, small and large graphics are now available for all 1071 Egyptian Hieroglyph characters. This was the last block for which graphics were completely unavailable.

>> Read the notes

Today I put the finishing touches to and uploaded my first draft notes about the long lost ishidic script. See what you think of it.

Here’s a small section of the sample text shown at the bottom of the post. Click on it to see the whole transcript.

1. Liam R E Quin Says:
   November 29th, 2009 at 2:24 am e

   At first glance it looks like a mix of Arabic and Cree, with the artificiality and non-calligraphic nature of Cree combined with the long extenders and curves of Arabic. If you tried to write with it by hand for a long time, maybe it would become more condensed, like Pitman short-hand. Or if you wrote it with a calligraphic pen — maybe I should try that!

2. David Clarke Says:
   December 2nd, 2009 at 3:59 pm e

   Do you have a translation, or is it to be treated as a linqua-cryptographic puzzle?

3. r12a Says:
   December 2nd, 2009 at 4:27 pm e

   Where would the fun be in providing a translation? 😉

   Here’s a head start:
   “The decipherment of a newly discovered or perennially mysterious text is the most glamorous aspect of the study of writing systems. …”

Removed the ‘beta’ from the version number and replaced with .0.1. New version converts u+… (ie. lowercase u) as well as U+… now.

See https://r12a.github.io/tools/conversion/

Thanks to Martin Dürst for the suggestion.

>> Use it

I have added a bunch of additional new features to my lookup tool to help with choosing language tags. There is additional information available when you look up subtags (such as what to use if the subtag is deprecated, and what subtags macrolanguages enclose, etc.), and more tests on well-formedness with clearer explanations of the problem. Example.

This should make it a lot more useful to people who haven’t read BCP 47 and want to create language tags. Hopefully, in a short while, I’ll also write and link to an article that describes how to use subtags from the ground up in a procedural way, that will complement the tool.

For further assistance, you can now link from a language subtag result to the SIL Ethnologue, to make it easier to check whether that subtag really does refer to the language you were thinking of.

In addition, script subtag results link to Unicode blocks in UniView.

>> Use it

The IANA Subtag Registry has been recently updated to contain 220 extlang subtags and the ISO 639-3 language subtags, taking the total number of subtags to almost 8,000.

I have produced a new version of my lookup tool to help with language tagging. In addition to helping you find subtags and lookup the meaning of subtags, it now helps check the well-formedness of a language tag.

The tool provides access to all currently defined subtags, including the new extlang subtags.

Parsing language tags. In addition to trying to make the user interface more friendly, I also added the ability to parse hyphenated tags and discover their structure and check for errors. I’m not claiming with this release that the new parser field tests all the corner cases, but it should provide reports for most of the typical errors.

It reports errors for the following:

– subtags that are not in the registry (by type)
– incorrectly ordered subtags
– duplicate variant tags and multiple tags of other types
– overlong private use subtags

Try this example.

It doesn’t yet handle extensions, but then there aren’t any valid ones to handle yet anyway.

I hope that’s useful.

>> See what it can do

>> Use it

Following hot on the heels of the last release come some further significant changes to UniView aimed at making it easier to use as Unicode grows.

The big change is that UniView now starts up in graphics mode by default. This means that pages load more slowly, but (especially with the continuing growth of Unicode) also means that you are more likely to be able to see the characters you are looking for. It’s easy to switch between modes at any point, using the “Use graphics” checkbox. (And if you preferred font glyphs as a default, you just need to change the URI in your bookmarked link slightly, and you can continue to work that way.)

To facilitate this change, I created my own graphics for a number of blocks which are not yet covered by decodeunicode, or which are no longer fully covered by decodeunicode. The blocks for which I provided graphics are Latin Extended-C, Latin Extended-D, Latin Extended Additional, Cyrillic Supplement, Cyrillic Extended-B, Modifier Tone Letters, Tibetan, Malayalam, Saurashtra, Ol Chiki, Myanmar, Kayah Li, Cham, Rejang, Vai, Supplemental Punctuation, and Miscellaneous Symbols and Arrows.

There are still many characters for which there are no graphics (especially the new characters in Unicode 5.2), but coverage is much better than it was. As I find more fonts, I will be able to create graphics for the remaining characters.

I also put a grey box around the characters in tables. This is particularly useful if there are no graphics or font glyphs for a block or range of characters, as it makes it easier to locate the character you are looking for.

I also fixed a bug that was preventing Chrome and Safari and IE from displaying the first two Latin blocks. I think the bug was actually in the Unicode data file.

>> See what it can do

>> Use it

With the family now in Japan, I had some extra time to spare this weekend, so I upgraded UniView to handle all the proposed characters for Unicode 5.2.

While the properties for new and modified characters are still in beta they are not officially stable, however the character allocations should be stable at this point. UniView therefore alerts you if you are looking at a new character.

If the Unicode database information has changed for a given character you are also warned, and provided with a link that points to the previous information for that character. These warnings will be removed from UniView when Unicode 5.2 is released.

Of course, you are unlikely to be able to actually see the new characters themselves, unless you are lucky enough to have a very new font to hand. The graphic alternatives are not available yet for these characters. I’m wondering whether it’s possible for me to do something about that, but that will take a little longer. In the meantime, you might find it more useful to view blocks in list view. (Click on ‘Show range as list’).

This release also fixes a few small bugs in the HTML and JavaScript code.

A new version of this very popular tool is now available, in a new location. Although it is currently labeled ‘beta’, I recommend that you use that instead, and change any links and bookmarks to the new location. There are a number of new features.

There is also a vastly improved code base. If you are one of the many people who have contacted me to ask how I coded the conversions, please take a look at the new javascript code. It is much cleaner and more compact.

New features include:

* New mixed input field and position of some fields changed.
* New field for conversion of 0x… notation hex escapes.
* Enabled invisible and ambiguous characters to be made visible in the XML output.
* Added support for all HTML entities in HTML/XML input.
* All code rewritten to use characters as the internal representation, rather than code points. Also, code is much smaller and cleaner, partly through use of regular expression matching.
* Various filters available for conversion, such as allowing ASCII or Latin1 characters to remain unconverted in NCR output.
* New icon to quickly select all contents of a field.

There is also a new demonstration feature.

If there are no issues raised/remaining in a couple of months, I’ll remove the beta tag.

>> Use it

This is a new tool that helps you to locate a country or territory on a map of the world. Ever wondered where Khazakhstan is? This will show you.

The map is in SVG and expands to fill the window. Territories are coloured red. Very small territories are marked by a red dot.

The map comes from Wikipedia. The list of territories comes from the regions listed in the IANA Language Subtag Registry. I can’t guarrantee that all the territories in the pulldown list are viewable, but nearly all are.

It’s quite a big SVG file, so it takes a little while to draw. I’ll try to speed that up in the future. It seems to draw much faster on Chrome or Opera than on Firefox or IE.

For the future I have some other ideas, such as displaying the country name natively, and linking to Wikipedia articles, CLDR data, etc. But that’s for later.

Update: Almost every time I located a country, I found myself wondering what the countries alongside are. So now as you move your mouse over a country, the name of that country pops up.

Enjoy.

>> See what it can do !

>> Use it !

The major changes in this version include a new feature to normalise text as NFC or NFD, the ability to accept decimal code point values, and an overhaul of top part of the user interface.

Added buttons to the Text area to allow conversion of the text to NFC or NFD normalization forms. (You may not notice the change until you list the characters.)

The control panel was also substantially rearranged again to hopefully make it easier for newcomers to see what they can do.

The Code point conversion feature was upgraded to handle decimal code point values.

A single character in the codepoints area or text area is now listed in the lower left panel when you click on , rather than in the right-hand properties panel. This is to improve consistency and avoid surprises.

Added a link to the CLDR property demo from the right panel to give access to additional properties.

Improved the parsing of codepoints when surrounded by text in the Code point input field, so that it now works with &#x…; and \u… and \U… escapes.

Jettisoned some unneeded code to reduce download by around 40-50K bytes. Implemented the NFC/NFD feature using AJAX, to avoid putting the download size back up.

When you delete the contents of the text area or the code point area, the associated input field is given focus, so you are ready for input.

A couple more minor bug fixes.

I was asked to make available the code for my normalization functions in JavaScript and PHP. The links are below. I’m making the code available under a Creative Commons Attribution-Noncommercial-Share Alike licence.

Disclaimers Note that I make no claim to have produced polished, compact or well-optimised code! The code does what I need, and I’m happy with that. You are welcome to suggest improvements, and I’m sure there are many that could be made.

As they say, this code is made available in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.

The code is a little more convoluted that it ought to be, to get around the fact that JavaScript doesn’t understand supplementary characters, and PHP just doesn’t naturally understand Unicode. (How I long for PHP6.)

Update: [[I meant to mention that there is a way of doing normalization in PHP already. I made this code available just because I had it. I created it as a learning exercise. It may be useful, however, if you are unable to load the ICU and intl packages onto your server.]]

To use the code, simply call nfc('your-text-string') or nfd('your-text-string') from your code and capture the result.

For PHP you’ll need these routines and this data.

For JavaScript look at these routines and this data. There is also a lite version of the data file that doesn’t include Han characters. I use this sometimes for bandwidth savings (about 14K less).

Test files I also created some test files for PHP and for JavaScript.
Both of these expect to find a copy of http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt in the local directory. These files run 71,076 tests.

Cautions Be careful about the editor you use for the data files. I spent several hours fruitlessly debugging the routines, only to find that Notepad++ was displaying certain supplementary characters ok, but corrupting them on save. I switched to Notepad and the problem evaporated. And I probably don’t need to add that editing the data files in something like DreamWeaver is a bad idea because it will probably normalize the data before saving.

Another point: you may see Unicode replacement characters at a couple of points in the PHP source. These represent the first and last characters in the high surrogate range.

Experimenting If you want to play with something that uses this you could try my Tłįchǫ (Dogrib) character picker, or my Normalizer tool. I will slowly fit this to all the pickers and to UniView. I have a local version of UniView waiting in the wings that uses the PHP files via AJAX, to reduce download size. For that you need a file that returns the result as plain text across the wire, such as this.

Well, I hope that that may be of use to someone, somewhere. I hope I haven’t forgotten anything.

>> Try it !

This tool allows you to normalise short pieces of text to Unicode forms NFC or NFD. You can paste the relevant text into a text area, or append it to the uri that calls the page, eg. Vietnamese example.

Note that, although I spell normalisation in the British way in this post, the uri uses the American spelling, since I suspect most users of the tool will expect it to be spelt that way.

Wondering what normalisation is? In Unicode a letter like á can be represented by a (precomposed) single character or by an a followed by an acute accent (a decomposed sequence). Unicode regards these two representations as formally equivalent. If you are comparing strings, therefore, you need to know which representations are equivalent. Usually you would want to normalise your text prior to comparison to a given normalisation form, so that the comparison process can be efficient. Unicode defines four normalization forms, two of which, NFD and NFC, are handled by this tool.

Basically NFD reduces all precomposed characters to their decomposed equivalents, whereas NFC uses precomposed characters for most common situations.