MS OFFICE XP and UNICODE

[mda mike]

This is really an MS issue, but I can't seem to get help from them, so I am hoping someone out there might have some help for me.

I have created Unicode fonts, with special characters in the Private Use area of the Unicode mapping.

The fonts work well, and can be used in MS by inserting symbol, and selecting the characters.

But, when I create a keyboard, and try to type the characters from my keyboard, MS will decide I am using Japanese and change the font. I turned off the automatic language detection as per the help files, but this did not change anything.

Has anyone else run into this after creating a font?

mike

[Erwin Denissen]

I assume you are using Microsoft Keyboard Layout Creator to create the keyboard layout? Although I haven't used it, maybe other people have tried this tool.

Can you send the font file to me, so I can investigate the problem?

[Joop Jagers]

I'm quite sure there's nothing wrong with the font. I'm afraid it's a Microsoft Word issue. I have the same problem after using Microsoft's program to create a keyboard. Apparently the programmers of Word usurpate the Private Use Area for their own purposes (they'd better check the meaning of 'private'). The way they do this is completely undocumented. If you use your new keyboard in WordPad you'll see it functions correctly.

[Erwin Denissen]

I have just received the font and put it through Microsoft Font Validator. I did find one thing related to your problem:

From Font Validator Help, Message: W0307
Microsoft recommends that use of the Private Use Area (PUA) in Unicode fonts is avoided. For legacy reasons, the 'fi' and 'fl' ligatures are the only glyphs that should be mapped in this area, as follows:

fi ligature [Unicode: U+FB01 and U+F001]
fl ligature [Unicode: U+FB02 and U+F002]

Hope this helps

[mda mike]

Erwin and Joop,

Thank you for the replies and assistance with this problem. I have heard from another programmer/font designer who also said that MS has taken the private out of private use area for their products (this is not just a Word issue).

I have tried the following: I have gone back to my font and under the settings, unicode range - selected Private Use Area (something I forgot to do). This helped with some of the characters. But others still wanted to change to Japanese characters. So I went back and selected the Japanese Code page. This allowed my characters to be displayed, however, they appeared on their side... still not quite the result I want.

I am continuing to work on this, and will get back with what I find, if I find anything.

[Joop Jagers]

I was partly wrong: Microsoft apparently *did* document its use of PUA. I found this site
http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&item_id=VendorUseOfPUA
which provides a quick overview of the ares used by Microsoft and links to Microsoft pages.

[Erwin Denissen]

Great source!

Thanks for sharing the link.

[mda mike]

Joop, Thanks for the link. Another friend directed me there right before you did. That is good information, and has many other good links as well.

Since i last wrote:
I removed the Japanese code page because I didn't like my characters on their sides. I also found that any characters produced with normal key strokes would initially change to Japanese each time I started the document, and sometimes when I started new paragraphs, but generally worked after changing them to English one time. But the reveal codes did reveal that the text was still viewed as Japanese. MS Support is looking into this...

But deadkeys always failed, with both Keyman keyboards, and Microsoft Keyboard Creator layouts.

I have downloaded the Unicode charts and found places other than the PUA to put my characters, and remapped all 636 characters to other locations.

see: http://www.unicode.org/charts/

I have only tested 2 characters since the remapping with my Keyman keyboard layout program, but now they both work with a deadkey. This is good news.

I believe that MS has divided the PUA into sections, and any character from those sections is determined to be a different language, which makes the program automatically (with no choice of the user) switch to a different font and symbol. But using other parts of the Unicode seem to work. I used the IPA, Phonetic, Diacritic, and Superscript areas.

[Joop Jagers]

Mike,

Could you keep me posted on this subject and MS Support's reaction?

Microsoft Keyboard Layout Creator has severe limitations in creating dead key sequences. The trouble is that these limitations *don't* show when testing the keyboard within KLC; only after compiling and installing the keyboard some dead key combinations don't work. It turns out that the dead key feature only works properly when the key which is made dead keeps its original value. Reaction from Microsoft:

The test dialog attempts to simulate the real-world behavior, but without real knowledge of the limitations (and this is knowledge that no longer exists in Microsoft!), we rely on bug reports to help.

You said:

I have downloaded the Unicode charts and found places other than the PUA to put my characters, and remapped all 636 characters to other locations.

I take this to mean that you remapped your 636 to locations freely available *within* some subsets of Unicode. If this is correct I must advise strongly against this method. There's no way of knowing if the open spaces in a Unicode subset will be used by Unicode somewhere in the future. If this happens your font will be incompatible with Unicode. Characters not (yet) recognized by Unicode should be mapped to the PUA. The best one can do is look for an anused part of PUA rather than using reserved code points.

[mda mike]

Joop,

I have been corresponding with a Microsoft Professional Help support person. I explained the problem to them a number of times, and sent him a copy of the font, the keyboard and another detailed explanation of the problem.

The response I received is that since I used a 3rd part keyboard program, it is a problem with that software, not MS.

I did write back and pass on to them that this is a documented problem in MS, but they did not respond back.

I don't know if you read it, but here is another site that documents the problem specifically in MS products.

http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&item_id=PUACharsInMSSotware

I will keep trying to get some kind of solution. But I think it will come down to writing program in VBA for keyboard substitutions specifically for Word where the font and language can be defined programmatically.

mike

mike

[Joop Jagers]

Mike, thanks for keeping me posted. Well, apparently that's the way Microsoft treats her customers. First, they deny there's a problem. Then they say it's someone else's fault. When proven wrong they maintain an eardeafening silence.

In my view the most sensible thing to do is finding another location in PUA for your glyphs which is safer. You might have a look at http://www.evertype.com/standards/csur/ to avoid conflicts with other users. But I know I'd *hate* to reassign 636 characters.