Introduction

What is Unicode?top

This much reduced table illustrates the extent of Unicode coverage:-

Why was Unicode needed?top

In the case of video displays, this was organised by a chip called a video controller which would know how to draw a particular character on the screen. The program would send the video controller two bytes at a time. The first would tell the controller which character to draw, and the second would provide the character's "attribute" (its colour, background and whether it was flashing, bold, or underlined).
Since the character's value was contained in one byte, the number of different characters which could be displayed was limited. In practice the first 32 characters were used as control characters (a carriage return value 13, and linefeed value 10 being good examples), and value 255 was reserved, so that only 222 characters were available.
In serial communication it was even more restricted since one bit was used for parity checking.
The actual characters capable of being drawn with these arrangements were called a character set. The ASCII character set, in the first 7-bits of the byte, could represent the usual Roman characters (a to z, A to Z), Arabic numbers (0 to 9), and some additional characters and punctuation marks commonly used. IBM established a character set which included line draw characters using the 8^th bit.
To draw other characters on a screen meant the video controller had to be told to switch character sets and draw a different character when sent a particular value. So different character sets were established to represent those. But here was the first problem. Since text was drawn on a byte-by-byte basis, languages which needed more than the number of characters available in a single-byte character set [for example Chinese, Japanese and Korean (Hangeul)] could not be represented fully. So Double and Multi-Byte Character Sets (DBCS and MBCS) were developed. These required special parsing and drawing techniques.

One difficulty with this arrangement arises partly from inconsistency in the individual character sets. For example the Windows character set appears to be usable by Hungarian speakers, but in fact there are four characters missing (Ő,ő,Ű and ű). So Hungarians have to use the Central European single-byte codepage iso-8859-2 which does have those characters (a codepage being a version within a character set). But then there is another problem. The iso-8859-2 codepage does not have another character which may be needed (Õ)! If this is a problem for those using Hungarian, imagine the difficulties faced by those using Asian scripts.
A second problem arises from the fact that the default character set in any particular computer is set on installation to the "locale" of the computer. On Windows 9x machines, this can only be altered using the Control Panel and can not be altered at run-time. Now some system displays which are very useful to the programmer (such as MessageBox), can only draw using this default character set. This means that such displays will be wrong if the locale is not set properly by the user.
In other types of controls, considerable care is needed by the programmer to ensure that the correct character set is used for the language to be represented.

As the requirement has grown for more and more languages to be represented, this could have been achieved by introducing more and more character sets. But instead the decision was made to go down the Unicode route. The advantages are:-

• Since every character can be represented in the Unicode character set, two characters which look the same can have the same Unicode value even if from different languages. This keeps the representation of Japanese, Chinese and Korean characters and punctuation more compact than if individual character sets were used for each of these languages.
• It is much easier to identify a particular character in Unicode since each character has a unique value. You don't need to know the character set as well. This makes it much easier to transfer international character data between machines. However, to display characters properly in complex scripts you may well need to know the language in which the characters are being used, to make sure that they are drawn to the proportions required by that language.
• Since characters are identified by value irrespective of character set, the display functions can draw characters from different languages in one string.
• Whole characters sets were needed for each language to provide the sort order, that is to say the alphabetical order of the characters. In Unicode this is handled by the operating system using the National Language Support ("NLS") APIs.
• Unicode has plenty of space to accommodate new characters as they are required. For example new codepages had to be created to enable the Euro character (€) to be drawn, yet it was easy to add to Unicode (value 20ACh).

Drawing non-Roman characters - it's font-basedtop

So the trick is to find the correct font and character set to draw the characters which need to be drawn.

Not all fonts and character sets will be installed on all machines. What fonts and character sets are loaded on installation will depend on which version of Windows is being used, whether it is an English or other language version, and what applications are loaded (they can come with their own fonts).
Every Windows machine will have installed Courier New, Arial, Times New Roman, Symbol, Wingdings, MS Serif, MS Sans Serif and (on later versions) Tahoma. Depending on the Windows version, between them these fonts can handle at least Western and Central European, Hebrew, Arabic, Greek, Turkish, Baltic, Cyrillic and Vietnamese scripts.
You can see what fonts are loaded on your machine by looking in the Windows\fonts folder or at "Control Panel, fonts". You will also find that installed fonts are listed in the registry in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft and sub-keys (which depend on the Windows version). However, there is not much information about the fonts. What you need to know is what languages are supported by what fonts. The easiest way to see this is to go into "Internet Options, General" and clicking "Fonts". Then in the drop down combo box select the language you wish to portray. A list of fonts then appears which have character sets capable of displaying the language specified. The "ChooseFont" common dialog which is used by various applications does a similar thing, except that you choose the font first and then the "script" drop down combo box shows what languages the font is capable of displaying.
The capability of a particular font can be expected to vary from machine to machine. For example, the Times New Roman font supplied with Windows 98 is limited to Western script and Greek, but on Windows XP can also draw Hebrew, Arabic, Turkish, Baltic, Central European, Cyrillic and Vietnamese.

If you have Word installed you can see the characters available for each font and character set by using "Insert, symbol" and then clicking the subset of the range you are interested in.

As developers we are more interested with what can be done at run-time to check that the correct font is installed. You can enumerate the loaded fonts which suit specified character sets using EnumFontFamiliesEx. This is a callback routine capable of reporting the language and character set in two ways. Either by using the CHARSET value (a fixed byte value of between 0 and 255 which you find at +17h in the LOGFONT structure), or by using the name of a language in words (reported in the ENUMLOGFONTEX structure) which is a more flexible arrangement. The languages in words reported by EnumFontFamiliesEx seem to be those which appear in the "Internet Options, General", "Fonts" combo box which probably uses that function anyway to fill its contents.

Generally fonts are capable of representing only a limited selection of values in the Unicode character ranges, so under Windows 2000/XP EnumFontFamiliesEx API is extended so that you can also receive the Unicode range of a particular font through the FONTSIGNATURE structure. The API GetUnicodeRanges can also be used to discover the Unicode range of a particular font.

Microsoft advises that the user of your application should also be able to find the correct font to display the language required using the ChooseFont common control, if necessary.

Windows 2000/XP uses fonts in some clever ways to maximise their versitility. For example, if a particular font does not support a particular character which needs to be drawn, Windows looks at "linked fonts" to see if there is another font which will do so. The fonts which are linked are contained in the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\NT\CurrentVersion\FontLink\SystemLink. These are added to each time a new Language Group is added to the machine by the user, using Control Panel, Regional Options.

When you have found the font suitable for the characters you wish to draw, you would get a handle to the font using CreateFont or CreateFontIndirect and then select the font into the device context ready to draw.

If your application runs on both W9x/ME and XP/2000 platforms there is a very handy pseudo-font you can use: MS Shell Dlg. You can specify this font either in the resource file or in the LOGFONT structure and it will automatically map to the correct font to suit the platform. On W9x/ME machines it maps to MS Sans Serif, and on XP/2000 machines it maps to Microsoft Sans Serif (or to Tahoma if you specify that you want a fixed pitch font). Both Tahoma and Microsoft Sans Serif (as found on XP/2000 machines) support Greek, Hebrew, Arabic, Turkish, Baltic, Central European, Cyrillic, Vietnamese and Thai scripts. Note that Microsoft Sans Serif is not the same as MS Sans Serif. What MS Shell Dlg switches to can be found in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\NT\CurrentVersion\FontSubstitutes.

What can be done with the ANSI APIs?top

What can be done with the Unicode APIs?top

Windows NT/2000 and XP provide Unicode versions of the other string APIs such as DrawText, and Unicode common controls, menus, and dialogs. These are capable of drawing Unicode text irrespective of character sets and codepages provided they are given the correct font.

There is also the Uniscribe set of APIs which are highly sophisticated and designed to analyse complex Unicode characters and their individual components (glyphs) and display them correctly.

Getting the strings to give to the APIstop

• keeping the Unicode text in a separate file to be loaded at run-time so that the strings could be used straight from data; or
• keeping the Unicode text in a resource file either as a STRINGTABLE or RCDATA resource, or within DIALOG or MENU resources; then at compile time the resources would then be bound into the executable, or into special-language DLLs which would be selected for loading at run-time.

PUSH 100h                ;size of the buffer to receive the string
PUSH ADDR BUFFER         ;pointer to buffer to receive the string
PUSH 23                  ;identifier of the string to be obtained
PUSH [hInst]             ;handle to executable containing the resource
CALL LoadStringW

PUSH 40h                 ;information + ok button
PUSH ADDR TITLE_TEXT     ;pointer to text for the title
PUSH ADDR BUFFER         ;pointer to string 23 just retrieved
PUSH [hWnd]              ;to be child of main window
CALL MessageBoxW

What is new in GoAsm is that you can keep your strings in your assembler source script and declare them in data, just as you would with an ANSI string. This is because GoAsm reads Unicode source scripts and include files. See part 2 for details of how strings can be kept in data using GoAsm and GoRC.

Part 1: Using the "Go" tools to make Unicode programstop

Types of Unicode source files readable by the "Go" toolstop

Making a Unicode source file with a Unicode editortop

Personally I have found NoteXPad to be very reliable. It comes in two versions, one in English and one in Chinese. Look either for "EN" or "SC" in the downloaded filename.

When you use the Unicode editor to create a source file, you must ensure that the file is saved in the correct format for the "Go" tools, that is either UTF-8 with BOM or UTF-16LE with BOM (sometimes referred to just as "Unicode").

Unicode command linetop

In this way you can define a word (using the /d switch) using Unicode characters.

You can also use Unicode characters for the names of your input and output files.

Note that extensions to filenames must be in Roman characters if the extension is important to the "Go" tools. For example GoRC creates an obj file if given a .res file. Here "res" cannot be in anything other than in this form. Similarly the extension of GoAsm and GoRC's output files will always be in Roman characters, for example if you assemble the file Процесса.asm this will produce a file called Процесса.obj.

GoLink looks for specified files with ".obj" and ".res" extensions for its input files, and specified ".dll", ".exe", ".drv", and "ocx" files when looking for imports. These extensions must be in this form, but the filename itself can use any characters permitted by the operating system.

Unicode output filestop

Unicode filenames in your source scripttop

Unicode comments in your source scripttop

Code and data labels in assembler source scripts (ASM files)top

Resource IDs from resource scripts (RC files)top

Exporting Unicode named functions from your executable or DLL and calling them from other executablestop

Unicode defined words in your source scripttop

функция equ invoke
пересылка equ mov
ПолучитьХендлМодуля equ GetModuleHandle
МодальноеДиалоговоеОкно equ DialogBoxParam
ИнициализацияДиалога equ WM_INITDIALOG
СообщКоманда equ WM_COMMAND

How Unicode labels and IDs appear in the debuggertop

Conversion methods used by GoAsmtop

Part 2:
Using Unicode strings in DATAtop

Controlling the string formattop

PUSH 40h                 ;information + ok button
PUSH ADDR TITLE_TEXT     ;pointer to text for the title
PUSH ADDR STRING_23      ;pointer to string 23 in data
PUSH [hWnd]              ;to be child of main window
CALL MessageBoxW         ;call Unicode version of MessageBox

STRING_23 DB 'Μπορώ να φάω σπασμένα γυαλιά χωρίς να πάθω τίποτα'
          DW 0                ;null terminator

Using DUS (declare Unicode sequence)top

STRING4 DUS "The strings, new line chars",0Dh,0Ah,"and null are in UTF-16",0

L', A' and 8' overridestop

The overrides work like this:-
L' override - string will always be put in the object file in Unicode UTF-16 format
A' override - string will always be put in the object file in ANSI format
8' override - string will always be put in the object file in Unicode UTF-8 format

Overriding using the STRINGS directivetop

STRINGS UNICODE  ;from this point all strings are converted to UTF-16
STRINGS ANSI     ;from this point all strings are converted to ANSI

Note that as a directive to GoAsm, STRINGS is only recognised in the source script and in "a" extension include files. It will not be recognised in non "a" include files because they are not treated as assembler source scripts, but merely lists of defined word (equates, macros and structs).

Declaring data using the overridestop

STRING_23 DB L'Μπορώ να φάω σπασμένα γυαλιά χωρίς να πάθω τίποτα'
          DW 0                ;null terminator

STRING_28 DW L'Convert me to UTF-16',0

STRINGS UNICODE            ;all strings from now on to be UTF-16
STRING_30 DW 'nem lesz tőle bajom',0    ;null-terminated UTF-16 string

STRING_36 DB A"This string will always be in ANSI"

STRING_40 DB 8"The strings, new line chars",0Dh,0Ah,8"and null are in UTF-8",0

Pushing null terminated Unicode stringstop

STRINGS UNICODE
PUSH 40h                 ;information + ok button
PUSH 'На берегу пустынных волн'   ;pointer to title
PUSH 'Стоял он, дум великих полн' ;pointer to text
PUSH [hWnd]              ;to be child of main window
CALL MessageBoxW

INVOKE MessageBoxW, [hWnd], L'Meg tudom enni az üveget', \
                            L'nem lesz tőle bajom', \
                            40h

Using the correct string in quoted immediatestop

STRINGS UNICODE
CMP AX,'л'

However,

CMP AX,8'л'

And like ordinary quoted strings, a quoted immediate in a Unicode source script saved in UTF-8 format would be converted to UTF-16 if neither the STRINGS directive nor an override is used.

Using Unicode strings in structurestop

MyStruct STRUCT
   DB 0
   DW L"I am a null terminated Unicode string",0
   DB 0
ENDS
;
Label64 MyStruct              ;apply the structure template

If the initialisation of the structure is itself overriden, that takes priority for example:-

UP STRUCT
   DB L'I would prefer to forget this'
ENDS
DontGive UP <8"I won't let you">

When the overrides applytop

The STRINGS directive applies to macros, equates and structures when they are applied in the source script, and not when they are declared. Strings in data are different because they are applied and established in data when they are declared. So for example:-

STRINGS ANSI
MyString1="I could be either"      ;an equate
MyString2 DB "I couldn't"          ;an ANSI data declaration
STRINGS UNICODE
PUSH MyString1                     ;push pointer to Unicode string
PUSH ADDR MyString2                ;use the ANSI string

STRINGS ANSI
MyString1=A"I know what I am"      ;an ANSI string equate
MyString2 DB L"So do I"            ;a Unicode data declaration
STRINGS UNICODE
PUSH MyString1                     ;push pointer to ANSI string
PUSH ADDR MyString2                ;use the Unicode string

Using SIZEOF on Unicode stringstop

MyString DW 'Здравствуй Мир (Hello World in Russian)',0
MyStringSize DD SIZEOF MyString

If you use structures or equates with strings which change size depending on whether you are making a Unicode or ANSI version of your program using a Unicode/ANSI switch, ensure that the correct switch is set when SIZEOF is used. This is because the size in bytes of the structure, structure member, or data declaration is measured at the time when SIZEOF is used, rather than at any other time.

Part 3: Unicode/ANSI switching

What is Unicode/ANSI switching and why is it needed?top

Switching using run-time loadingtop

• On start-up the application would check which platform it was running on by calling GetVersionExA (note that GetVersionExW should not be called since it is not implemented in Windows 9x/ME).
• The application would not call any APIs directly which were not available under Windows 9x/ME. The reason for this is that when an application is loaded by the system, the loader writes into the application itself as held in memory, the addresses of the APIs which the application uses. If the system loader cannot find a particular API (or DLL which is supposed to contain that API) the application will not start at all.
• Because of the above fact, if running under NT/2000/XP, the application must obtain the addresses of the Unicode APIs at run-time using run-time loading. In fact, this is a fairly simple and well-established procedure, using the APIs LoadLibrary and GetProcAddress. Note that if you were sure a particular DLL would be loaded by the system anyway because you have called an API directly then you would use GetModuleHandle instead of LoadLibrary for that DLL). An example is given in "Hello Unicode 1". In a large application the switching could be dealt with straightforwardly using tables and strings in data. An example of this is given in "Run Time Loading".
• At run time the application would call the ANSI APIs in the usual way if running under Windows 9x/ME and the Unicode APIs using the addresses which it found at start-up if running under NT/2000/XP. Note that not every API needs to be dealt with in this way - many of them are available under both platforms.

Switching using the same sourcetop

A developer may still want to make two versions of the application, one in ANSI and one in Unicode. This might be to avoid the system's Unicode/ANSI conversions at run-time, to make full use of the added functionality of NT/2000 and XP, or to permit different language versions with non-Roman characters on the NT/2000 and XP platform. This is where Unicode/ANSI switching comes in. Using Unicode/ANSI switching it is possible to have just one version of the source script, but at compile-time you tell the assembler whether to make an ANSI version of the application or a Unicode version, using conditional assembly.

This is the switching which may be needed in such a source script or in include files:-

• Switching of APIs which have both Unicode and ANSI versions
• Switching of constants
• Switching of quoted strings and immediates
• Switching of string sequences with control characters
• Switching of the size of a write-to or read-from memory instruction by switching the type indicator
• Switching of the size of a character to get structure members and data buffers to the correct size
• Switching of initialisation of data

How switching is achievedtop

In GoAsm you can define UNICODE in one of two ways, either by adding /d to the command line for example:-

GoAsm Myfile /d UNICODE

#define UNICODE

You can then use this switch like this:-

#ifdef UNICODE
                ;assemble these lines
                ;if UNICODE is defined
                ;then jump to #endif
#else
                ;assemble these lines
                ;if UNICODE is not defined
#endif

#ifdef UNICODE
AW=W             ;switch for APIs more
STRINGS UNICODE  ;switch for quoted strings more
DSS=DUS          ;switch for string sequences more
S=2              ;type indicator more and character size switch more
#else
AW=A
STRINGS ANSI
DSS=DB
S=1
#endif

#undef UNICODE

Unicode/ANSI switching of APIstop

CALL DefWindowProc##AW

#ifdef UNICODE
AW=W
#else
AW=A
#endif

You don't have to use AW in the switch - you could use anything you like, for example

CALL DefWindowProc##SWITCH

CALL WriteFile

Switching of constantstop

#define TB_ADDBUTTONSA          (WM_USER + 20)
#define TB_INSERTBUTTONA        (WM_USER + 21)
#define TB_INSERTBUTTONW        (WM_USER + 67)
#define TB_ADDBUTTONSW          (WM_USER + 68)

#ifdef UNICODE
#define TB_INSERTBUTTON         TB_INSERTBUTTONW
#define TB_ADDBUTTONS           TB_ADDBUTTONSW
#else
#define TB_INSERTBUTTON         TB_INSERTBUTTONA
#define TB_ADDBUTTONS           TB_ADDBUTTONSA
#endif

TB_ADDBUTTONSA   = WM_USER+20
TB_INSERTBUTTONA = WM_USER+21
TB_INSERTBUTTONW = WM_USER+67
TB_ADDBUTTONSW   = WM_USER+68

TB_INSERTBUTTON  = TB_INSERTBUTTON##AW
TB_ADDBUTTONS    = TB_ADDBUTTONS##AW

Switching quoted strings and immediatestop

#ifdef UNICODE
STRINGS UNICODE
#else
STRINGS ANSI
#endif

ST36 DB "This string can be in Unicode or ANSI"
INVOKE MessageBox##AW,[hWnd],'Hello switched user',ADDR ST36,40h

CMP SWREG,'a'

more about the STRINGS directive.
other ways of switching strings.

Switching string sequences with control characterstop

#ifdef UNICODE
DSS=DUS
#else
DSS=DB
#endif

MyLabel DSS 'I am a string of varying character',0Dh,0Ah,0

Using the switched type indicatortop

#ifdef UNICODE
S=2
#else
S=1
#endif

MOV S[EDI],0         ;insert a single zero if ANSI, double if Unicode
INC S[COUNT]         ;increment byte at COUNT if ANSI, word if Unicode
LOCAL CharUnder:S    ;make local byte if ANSI, word if Unicode
LOCAL BUFFER[256]:S  ;make 256 byte local buffer if ANSI, 512 if Unicode

#ifdef UNICODE
S=W
#else
S=B
#endif

Using a character size switchtop

#ifdef UNICODE
S=2
#else
S=1
#endif

ADD EDI,S               ;increment EDI by 1 if ANSI, 2 if Unicode
HLabel DB 256*S DUP 0   ;make 256 byte buffer if ANSI, 512 if Unicode

NMTTDISPINFO STRUCT
  hdr               NMHDR
  lpszText          DD
  szText            DB S*80 DUP 0   ;80 bytes if ANSI, 160 if Unicode
  hinst             DD
  uFlags            DD
  lParam            DD
NMTTDISPINFO ENDS

Other ways to achieve switchingtop

GoAsm Myfile /d UNICODE=1

#define UNICODE 1

UNICODE=1

You can then use this switch:-

#if UNICODE=1
                ;assemble these lines
                ;if UNICODE is "on"
                ;then jump to #endif
#else
                ;assemble these lines
                ;if UNICODE is "off"
#endif

#define UNICODE 0

UNICODE=0

Another way to switch stringstop

#ifdef UNICODE
#define TEXT(x) L##x
#else
#define TEXT(x) x
#endif

MyLabel DB TEXT("The string")

Other ways to switch APIstop

#ifdef UNICODE
#define DefWindowProc DefWindowProcW
#define MessageBox    MessageBoxW
#else
#define DefWindowProc DefWindowProcA
#define MessageBox    MessageBoxA
#endif
;
CALL DefWindowProc         ;switched depending on whether UNICODE defined
CALL MessageBox            ;switched depending on whether UNICODE defined

#ifdef UNICODE
AW=W
#else
AW=A
#endif
;
DefWindowProc = DefWindowProc##AW
MessageBox    = MessageBox##AW
;
CALL DefWindowProc         ;switched depending on whether UNICODE defined
CALL MessageBox            ;switched depending on whether UNICODE defined

#ifdef UNICODE
AW(a)=a##W
#else
AW(a)=a##A
#endif
;
CALL AW(DefWindowProc)

#ifdef UNICODE
CALLAW(a)=CALL a##W
#else
CALLAW(a)=CALL a##A
#endif
;
CALLAW (DefWindowProc)

Testing the STRINGS directivetop

#if STRINGS UNICODE          ;or#if STRINGS ANSI

Part 4:
Summary of Unicode support in the "Go" toolstop

More information, references and links top

The Unicode Consortium official site for Unicode news, updates, information and links
Alan Wood's Unicode resources a must-visit site kept up to date with information and resources
Dr International Microsoft's international site for developers - clear and understandable
SIL (Summer Institute for Liguistics) computer page various tools and resources
Institute of Estonian language letter database view Unicode values, character sets, and requirements of the different languages

Newsgroups:-
MSDN international newsgroup

Various articles:-
Design a Single Unicode App that Runs on Both Windows 98 and Windows 2000 by F Avery Bishop (April 1999).
Supporting Multilanguage Text Layout and Complex Scripts with Windows NT 5.0 by F Avery Bishop, David C Brown and Davis M Meltzer (November 1998).
Get World-Ready - Fonts Global Development and Computing Portal.
Writing Win32 Multilingual User Interface Applications Global Development and Computing Portal.
Configurable Language and Cultural Settings Global Development and Computing Portal..
Character sets by Ken Fowles, Microsoft Typography site.
Unicode Fonts for Windows Computers by Alan Wood.

Other References:-
"The Unicode Standard Version 3.0", published by The Unicode Consortium.
"Unicode - A Primer" by Tony Graham, M & T Books.
Various articles in the Microsoft Development Library, Knowledge Base and Windows Software Development Kit ("SDK").
"Hello World" by Rob Pike, Ken Thompson AT & T Bell Laboratories January 1992.
"An Essay in Endian Order" 1996 by Dr William T Verts.
"Alternate formats", February 2000, Network Working Group.
"Under the Hood" (Unicode support in Windows NT) 1998 by Matt Pietrek.
"Forms of Unicode" 1999 by Mark Davis (IBM developerWorks).

Acknowledgements top