Input Method Editors for Latin based character set

Fonts and More General Font Discussions
1

I would like to propose a discussion group on transliteration Input Method Editors for entering alternative styles of Latin characters into a Latin based character set. It is often desirable to have alternative styles of some characters and be able to enter them into an application as easily as typing the normal characters. This technique of transliteration is well known and established as the routine input method for languages in Central and Far East Asia, as well in Arabic languages. My interest though is to adapt such transliteration Input Method Editor for use specifically with Latin based charcter fonts.
Is anyone interested in joining a new group to promote a solution to this topic?

Known transliteration Input Editors currently available:
1- Microsoft IME available for Windows XP and 7.
2- Atok IME Pad (this is a commercial software for entering Kanji characters)
3- Google IME pad. Google currently offers transliteration for 20 different languages. Their program looks adaptable for Latin texts.

2

I have used the Microsoft Keyboard Layout Creator to create my own customised Windows keyboard with which I can type a wide range of accented characters for European languages, Pāḷi, and Sanskrit.

Full List of Shortcuts

I suppose you could create a similar keyboard to type alternate characters in the Private Use Area, which is the right place to map alternate contextual ligatures. Such a keyboard would be hard to use, and there is no agreed standard mapping method to follow.

I am not familiar with any input method used outside of Windows on a desktop PC.

3

I actually have an MSKLC layout that I can use to type in any of a number of scripts: Greek, Cyrillic, Hiragana, Katakana, Runic, Ogham, Cherokee, Arabic, Hebrew, etc. The problem with any Windows IME is that they do not recognize capital letters, and I’m not even sure exactly how it works with the non-alphabetic keys. Whether these same limitations exist for the Google and Atok IMEs - and whether custom IMEs can be defined for those tools - is a technical hurdle that needs to be investigated before you can even start on this kind of project.

4

Could you explain how this would be used please?

For example, would it be like the following, or would it be different?

Suppose that a font contains a glyph for an alternate lowercase e where the glyph is of a lowercase e with a swash flourish such as a calligrapher might use at the end of a line of text.

Suppose there is a line of poetry ending a poem.

For example, as follows.

And beyond, an apple tree

If someone wanted to key that line of poetry so that the alternate lowercase e where the glyph is of a lowercase e with a swash flourish is used at the end of the line, would that person key the text as follows?

And beyond, an apple tre%e

Is what you are wanting to develop a method so that keying that sequence would automatically produce a display using the alternate lowercase e?

Is that what you want to achieve, or is what you want to achieve something different?

On what platforms and with what software are you wanting the system you seek to work please?

An approach that I have used is to produce a pdf (portable document format) document containing one of each of the alternate character glyphs; glyphs that are mapped into the plane 0 Private Use Area. A user could then copy from that document onto the clipboard, paste into WordPad and then reformat using the font. That technique allows a person using a program such as WordPad that does not have, in at least the version on the computer that I am using running Windows xp, an Input Symbol facility.

An example of such a pdf document, for my Sonnet Calligraphic font, is available in the following place.

http://forum.high-logic.com:9080/t/sonnet-calligraphic/2698/33

That technique is a little awkward to use, yet does work well without needing to add special software into the operating system. A pdf can be made to have internal hyperlinks, so an opening page with the letters of the alphabet each set up as a hyperlink to another page and a collection of pages, one for each letter, with the alternate glyphs on the pages could be produced within the pdf.

However, that method is not automatic, so an automated method would be of interest.

William Overington

10 July 2012

5

Thanks for your inputs chaps. I try to be a little clearer with the attached example in pdf.
You will notice that I dleiberately mispelt the word “most” to read “morst” so that I could show the absence of ligature when I use the normal “r” which, in most cases, requires a long leading stroke and the case when the “r” follows the letter “o” or “v” or “w” in which case a short trailing stroke is available on such letters, and the letter “r” requires also a short leading stroke. In this case I created an alternative small letter “r” which I placed in the Private User Area, hex E028. Using the Alt+ decimal code I can then call up this alternative letter as you can see in the second part of the example.

The problem is that if the font has to be made available to a general public then this method, or any of the known methods, are really too technical hence unsuitable. For us it is relatively easy to use one of the methods you referred to and others that come to mind, but for the general public this would not be acceptable. Furthermore, if a friendly transliteration IME is available one can expands the alternative characters to any number. For example in my character set I am providing over 52 alternative characters even without bothering with Spencerians loops. If I were to add those as well then the added characters would be substantially more. I trust this makes clear the objective behind my request.

I believe that the only way to go is to use an IME pad, such as the Google Android IME for Greek which works on a Windows PC as well. I looked at this download and it works very well plus it seems easily customizable. Unfortunately I do not have sufficient programming skills to undertake a dissecting of this program. But I understand that modifications are allowed by Google. After all it means creating a link to a different character set database.

Finally, I am not aware why the IME cannot handle capital letters. The Google IME can handle both small and capital letters. I am using the MS IME pad to enter kana and kanji characters and I do not see this problem as those charcaters are essentially capitals as they are defined within essentially a bilinear height for horizontal, left to right, scripting and a fixed column width is writing from right to left vertically.

William I live in Stourbridge which used to be part of Worcestershire!

Cheers
Example Alternative letter small R.pdf (13 KB)

6

There are several reasons why OpenType glyph substitutions are the best solution to this problem:

  1. Users don’t have to learn any special shortcuts or install any special keyboards. They just type as usual.
  2. The text string in the application is not modified, so words that contain ligatures do not break spell-checking.
  3. Search and replace operations work normally.
  4. If the user changes the font to one lacking OpenType features, the text is still correct.
  5. Even in applications that don’t support OpenType features, the fonts work, though no ligatures are substituted.

The downside, of course, is that there are not yet many applications that support OpenType features.

The way to go for the future is surely to promote and support application support for OpenType or other Smart Font technologies.

7

Your suggestions are noted but since I am not familiar with Open Type fonts I am not able to comment further. However, I spent some time yesterday reading about OTF on the fontshop.com website. It seems to me that the literature on this subject assumes that the reader knows what OTF can do for him or her and dwell more on technical issues which are of no help to people who are still wondering if OTF meets their objectives. Based on what I could understand it seems to me that OTF will not be sufficiently “smart” to make some smart decisions by itself as the built-in rules would be too many for the font to make the best choices. For example. if the letter “o” has been typed then all the following letters need to be chosen from another set of letters having short lead-strokes. Likewise if the letters “v” or “w” are chosen during typing then the following vowels or consonants have to be chosen from another set having an even shorter leading stroke. And I can think of further additional rules for Spencerian and non Spencerian loops. A simple dialog pop-up window would leave the choice to the user. However, I do accept that my limited knowledge on this subject might be misleading and I will let the matter rest. Please note that my comments refer to cursive scripts, otherwise called Copperplate, in which both the leading tail and the trailing tail are connected to the previous and following letters. Unlike all the commercially available cursive fonts which break these rules.
However, I do see the commercial opportunity to have such small program available in the market and perhaps Erwin should take a look at it and come out with an innovative software product specific to European Latin and Greek/Cyrillic languages. I believe the market for the Asian languages is well covered by the Atok program sold by Just Systems.

I also noted that the second line in the Example I posted yesterday does not show the letter “r” linked to the previous letter “o”. This was my mistake which went unnoticed. I forgot to load the latest version of my font in which I had already adjusted the kerning pair “o-r”.

Best regards

8

The beauty of OpenType glyph substitutions is that the user doesn’t have to select which glyph to insert. A simple lookup table is used. So if a user types f, then i, the ligature fi is used. If the user backspaces, then types another f, the ff ligature is used. If they type f, f, then l, the ff ligature is changed for the ffl ligature, etc. Its fully automatic.

lookup ligaSub { 
    sub f f i -> ffi; 
    sub f f l -> ffl; 
    sub f f -> ff; 
    sub f i -> fi; 
}
9

Does anyone know the answer to the following question please?

Can the right-hand side of a glyph substitution statement (I do not know if that is the correct parlance) in an OpenType GSUB table have more than one glyph listed within that place?

William Overington

12 July 2012

10

You are welcome to post another pdf in a later post.

I studied the pdf that you posted yesterday with great interest and found the two versions of the lowercase r.

If you do try another pdf a sentence that shows both types of r would be of interest.

For example, something like the following.

She writes about the flowers in a forest.

Can anyone think of any words that contain the sequence vr that could be used in an example please?

Thus far I have thought of avril, the French word for April.

William Overington

12 July 2012

11

No. Graphite does, however, support one-to-many and many-to-many substitutions. I believe that AAT also supports multiple substitutions as well. OpenType only allows one-to-one (single substitution) and many-to-one (ligature) substitutions.

12

Thank you.

That seems to imply to me that an OpenType solution will not provide the solution to the problem.

Thinking about the original problem, I ask the following three questions please.

  1. If a transliteration Input Method Editor method were implemented and were in use with the font, how would the following phrase be entered?

The three great forests

The letter r is used three times. As I understand it at the present time, the r in three and the r in great would be the one design of r and the r in forest would be the other design of r, because that r follows a letter o. Is that correct?

  1. If a transliteration Input Method Editor method were implemented and were in use with the font, if the font were to have an alternate design of a letter t with a swash ascender design, such that that t could be used wherever the person using the font felt that it would look good, how would that person indicate that the alternate t were to be used?

Would it be by using %%t or in some other way?

  1. If a transliteration Input Method Editor method were implemented and someone obtained a copy, how would he or she use the facility on a computer?

Would it be some system level method of which one is unaware unless one uses it? Would it be usable with any application program that one is using?

William Overington

13 July 2012

13

Your comments and inputs were much appreciated. I am enclosing another pdf example sheet in which I try to answer some of the questions. Judge for yourself if I grasped the issue well enough and do not forget to bear in mind that I am a “dilettante” in typography.

More specifically I think that I understood Pesala’s lookup substitution list. And it seems very easy to me to implement. If it is so, it really begs the question: why isn’t OT the font by default?
With reference to William’s question I believe I understood it and rephrased it in example 8 in my pdf file. The reply is meaningful and I look forward to learn what Pesala’s answer will be.

If I may be allowed to use a more mathematical expression the question is: If a character set has, say, 365 glyphs in all of which some are standard and some are modified versions, in the lookup ligaSub list, does the right hand side of the equality sign need to be another of those 365 glyphs or a combination of two gliphs kerned together? The same question can be extended to those few cases where three letters require some form of common ligatures.

I have some other questions as whatever the answer will be I can see that OT is the way to go.
1- How do I go about converting my ttf font to OT?
2- Does Microsoft Word support OT fonts fully?
3- What other software supports OT fonts?
4- Do Apple’s programs support OTF?

Regards
Example2.pdf (32.6 KB)

14

Application support is limited, adding OpenType Features involves more work, most fonts don’t really need them.

The ligature features use many → one substitutions, so e.g. f + f → ff ligature.
Stylistic alternates (salt) can use one → one substitutions, but they require manual input, which is inefficient, but very flexible as the user can decide which letter form to insert. One glyph has a lookup table for several other forms to substitute for that glyph — the user has to choose from a dialogue presented by the application, which Stylistic Set to use.

}  lookup StylisticAlternates {
    sub asterisk -> [asterisk asteriskmath uni2051 uni2042 uni203B uni273B];
    sub plus -> [plus uni2795 uni271A-uni271C uni2720];
    sub at -> [at uni2121 uni213B uni260E uni260F uni2709];
    sub copyright -> [copyright uni2117 uniF000 estimated uni2139 uni2638];
    sub multiply -> [multiply uni2715-uni2718 uni274E];
    sub dagger -> [dagger uni2620 uni2622 uni26A1 uni26B0 uni2690];
    sub daggerdbl -> [daggerdbl uni2623 uni26A0 uni2621 uni26B1 uni2691];
    sub bullet -> [bullet uni204D uni2023  uni2767 uni2712 uni261B];
}

Stylistic Alternates.png

Study the tutorial on Adding OpenType Features. The free program MS VOLT can also be used, but I prefer the OpenType Compiler method. Once a script has been written, its very easy to add the same features to similar fonts, and sharing or editing scripts is straightforward.

Word 2010 supports OpenType Features. Levels of OpenType support in some well-known applications.

Word 2010 InDesign, Serif™ PagePlus X5, PagePlus X6, or DrawPlus X5.

I am sure they do, but I am not sure if they support mine (see my sig).

15

Easiest way is to define the different glyphs as ligatures of the base character and a variation selector, and then have the keyboard set up to input variation selectors. Variation selectors are default ignorable code points, so you would be able to do proper search and if the text were copied to a program where the font weren’t available, it would still be displayed as legible text.

16

Bhikkhu wrote:

The ligature features use many → one substitutions, so e.g. f + f → ff ligature.
Stylistic alternates (salt) can use one → one substitutions, but they require manual input, which is inefficient, but very flexible as the user can decide which letter form to insert. One glyph has a lookup table for several other forms to substitute for that glyph — the user has to choose from a dialogue presented by the application, which Stylistic Set to use.

Thanks again for your inputs.
Your answer many ->one substitution is clear and that would still be ok.
The second answer is limited to one → one substitution but if it requires manual input then I think it may not be as useful if such input is something like Alt+ 5 digits.

To answer WIlliam and Vanisaac question let me explain how the Atok IME pad works. The Japanese language uses a syllabic “alphabet” consisting of 228 charcacters plus some 2100 other ideograms derived from the much larger set of the Chinese ideograms. In all therefore to write in Japanese a user needs to select characters out of some 2350 glyphs. The way they have solved the problem is to use the latin alphabet, which they call Romaji, to enter the sound of the Japanese word spelt using the Romaji letters. The Atok IME pad then transliterates transparently and pops up with a dialog box which lists the corrsponding words that use that phonetic entered input. The look up glyphs that come up on the dialogue pop-up window can be few or many depending on the word that was entered. The user chooses the right one with the cursor keys and then proceeds to type the next text. There is no need to enter special code such as %%, etc. You just look up on the dialogue box and choose one of the optional glyphs that are shown, with a mouse or cursur keys or shortcuts.
The process is very user friendly and effective. It does slow down the typing speed but it works positively and is amenable to all kind of users, whether they are PC literate or not.

In the case of adopting such method for the Latin character fonts the job is much simplified as here we do not need transliteration as such but the opportunity to flag, on a pop-up dialogue box, alternative styles of letters whenever one of the basic letters is typed. For example, if I was typing the word “corrugated” then upon typing the letter “o” after the first “c” the IME pad would pop-up the dialogue box expecting that the next word to be entered could well be an r, or a v, or a w, as in such cases -provided that the font type was cursive- those letters would need to have a short leading stroke. So the user can choose the most appropriate style on the fly and continue typing undisturbed. I have been using the Atok bar for a while and I can assure you that it is painless and very user friendly.

I estimate that to make a cursive script, such as the one I am completing, truly cursive using the rule that the pen is never lifted from the paper, it essentially requires another set of 52 glyphs. Then obviously one can add some alternative styles, for example a minuscule “z” without descender if the main one was designed with a descender; and so on. Clearly if such system was available one would take advantage of this IME to add several alternative styles as well.

according to my understanding, I believe that this Romaji transliteration method is also used for the Chinese language where they have more than 50000 glyphs. I believe it may also apply to Koreaan and other similar languages.

I trust this clarifies how the Atok pad works.

17

If it’s truly contextual, you can actually hard code the rules into open type/graphite, and it will automatically select the right contextual glyphs. I think you are becoming the embodiment of the old axiom of every problem looking like a nail. There are lots of technical answers to alternate glyph selection already, and Unicode and smart font technologies were designed to accommodate a lot of different options. So let’s actually clearly define the problem, and then we can actually move forward on a solution.

PS, I know that I am intimately familiar with the vagueries of oriental orthography and computer input, and my understanding is that the others involved are also quite familiar with those technologies and their possibilities as well.

18

Actually, there are several Chinese input methods, some of which use Pinyin - the Chinese equivalent of Romaji - some use Bopomofo - the Chinese equivalent of kana - and some of which, like Wubi, ZhengMa, and Canjie, are radical/shape/stroke based. See http://en.wikipedia.org/wiki/Chinese_input_methods_for_computers. Atok is really just a single IME in a stand-alone, platform-independent app, rather than a full IME format. It actually contains a bunch of additional features like date formats, handwriting recognition, kanji search, etc. Korean usually uses regular keyboards, as two of the three Unicode Korean text models (don’t ask) allows for syllables to be composed, so Hangul encoding and input is simple alphabetic. Amharic (Ethiopian) is actually more commonly input with an IME than Korean.

19

Vanisaac wrote:

If it’s truly contextual, you can actually hard code the rules into open type/graphite, and it will automatically select the right contextual glyphs. I think you are becoming the embodiment of the old axiom of every problem looking like a nail. There are lots of technical answers to alternate glyph selection already, and Unicode and smart font technologies were designed to accommodate a lot of different options. So let’s actually clearly define the problem, and then we can actually move forward on a solution.

Well, I thought the question had been well understood by now: how to enter a Latin based extended character set inclusive of alternate styles of glyphs in a user friendly way. You say it can be done with an OTF/ graphite. I do not understand the term graphite. As a scientist I am familiar with graphite for nuclear reactors, now obsolete, or the new material graphene but graphite for typo is kind of double dutch to me. The quest for Higg’s boson is more familiar to me and I don’t believe that it has yet been found despite the recent announcements. Yet that would not preclude me from talking to someone at a basic level even if he or she were not skilled in the art or science of Physics.
Going back to the IME if such a method exists I would be grateful if someone can let me know so I can buy it and start using my Latin based character set. But my understanding is that such a method does not exist as a product and that is why this discussion was started.
Cheers all the same.

20

I thought the answer was well understood by now — there isn’t a user-friendly way if the user has to stop and choose the alternate glyphs, or stop and think which shortcut to use. Users should just be able to type normally and the smart font technology should decide which letter form is best depending on the preceding letter(s).

Graphite (SIL)