CodePage简介(转)
CodePage简介(转)
1. Codepage的定义和历史
字符内码(charcter code)指的是用来代表字符的内码.读者在输入和存储文档时都要使用内码,内码分为
- 单字节内码 -- Single-Byte character sets (SBCS),可以支持256个字符编码.
- 双字节内码 -- Double-Byte character sets)(DBCS),可以支持65000个字符编码.主要用来对大字符集的东方文字进行编码.
codepage 指的是一个经过挑选的以特定顺序排列的字符内码列表,对于早期的单字节内码的语种,codepage中的内码顺序使得系统
可以按照此列表来根据键盘的输入值给出一个对应的内码.对于双字节内码,则给出的是MultiByte到Unicode的对应表,这样就可以把
以Unicode形式存放的字符转化为相应的字符内码,或者反之,在Linux核心中对应的函数就是utf8_mbtowc和utf8_wctomb. ......1. Codepage的定义和历史
字符内码(charcter code)指的是用来代表字符的内码.读者在输入和存储文档时都要使用内码,内码分为
- 单字节内码 -- Single-Byte character sets (SBCS),可以支持256个字符编码.
- 双字节内码 -- Double-Byte character sets)(DBCS),可以支持65000个字符编码.主要用来对大字符集的东方文字进行编码.
codepage 指的是一个经过挑选的以特定顺序排列的字符内码列表,对于早期的单字节内码的语种,codepage中的内码顺序使得系统
可以按照此列表来根据键盘的输入值给出一个对应的内码.对于双字节内码,则给出的是MultiByte到Unicode的对应表,这样就可以把
以Unicode形式存放的字符转化为相应的字符内码,或者反之,在Linux核心中对应的函数就是utf8_mbtowc和utf8_wctomb.
在1980年前,仍然没有任何国际标准如ISO-8859或Unicode来定义如何扩展US-ASCII编码以便非英语国家的用户使用.很多IT 厂商发明了他们自己的编码,并且使用了难以记忆的数目来标识:
例如936代表简体中文. 950代表繁体中文.
1.1 CJK Codepage
同 Extended Unix Coding ( EUC )编码大不一样的是,下面所有的远东 codepage 都利用了C1控制码 { =80..=9F } 做为首字节, 使用ASCII值 { =40..=7E { 做为第二字节,这样才能包含多达数万个双字节字符,这表明在这种编码之中小于3F的ASCII值不一定代表ASCII字符.
CP932
Shift-JIS包含日本语 charset JIS X 0201 (每个字符一个字节) 和 JIS X 0208 (每个字符两个字节),所以 JIS X 0201平假名包含一个字节半宽的字符,其剩馀的60个字节被用做7076个汉字以及648个其他全宽字符的首字节.同EUC-JP编码区别的是, Shift-JIS没有包含JIS X 202中定义的5802个汉字.
CP936
GBK 扩展了 EUC-CN 编码( GB 2312-80编码,包含 6763 个汉字)到Unicode (GB13000.1-93)中定义的20902个汉字,中国大陆使用的是简体中文zh_CN.
CP949
UnifiedHangul (UHC) 是韩文 EUC-KR 编码(KS C 5601-1992 编码,包括2350 韩文音节和 4888 个汉字a)的超集,包含 8822个附加的韩文音节( 在C1中 )
CP950
是代替EUC-TW (CNS 11643-1992)的 Big5 编码(13072 繁体 zh_TW 中文字) 繁体中文,这些定义都在Ken Lunde的 CJK.INF中或者 Unicode 编码表中找到.
注意: Microsoft采用以上四种Codepage,因此要访问Microsoft的文件系统时必需采用上面的Codepage .
1.2 IBM的远东语言Codepage
IBM的Codepage分为SBCS和DBCS两种:
IBM SBCS Codepage
- 37 (英文) *
- 290 (日文) *
- 833 (韩文) *
- 836 (简体中文) *
- 891 (韩文)
- 897 (日文)
- 903 (简体中文)
- 904 (繁体中文)
IBM DBCS Codepage
- 300 (日文) *
- 301 (日文)
- 834 (韩文) *
- 835 (繁体中文) *
- 837 (简体中文) *
- 926 (韩文)
- 927 (繁体中文)
- 928 (简体中文)
将SBCS的Codepage和DBCS的Codepage混合起来就成为: IBM MBCS Codepage
- 930 (日文) (Codepage 300 加 290) *
- 932 (日文) (Codepage 301 加 897)
- 933 (韩文) (Codepage 834 加 833) *
- 934 (韩文) (Codepage 926 加 891)
- 938 (繁体中文) (Codepage 927 加 904)
- 936 (简体中文) (Codepage 928 加 903)
- 5031 (简体中文) (Codepage 837 加 836) *
- 5033 (繁体中文) (Codepage 835 加 37) *
*代表采用EBCDIC编码格式
由此可见,Mircosoft的CJK Codepage来源于IBM的Codepage.
2. Linux下Codepage的作用
在Linux下引入对Codepage的支持主要是为了访问FAT/VFAT/FAT32/NTFS/NCPFS等文件系统下的多语种文件名的问题,目前在NTFS和FAT32/VFAT下的文件系统上都使用了Unicode,这就需要系统在读取这些文件名时动态将其转换为相应的语言编码.因此引入了NLS支持.其相应的程序文件在/usr/src/linux/fs/nls下:
- Config.in
- Makefile
- nls_base.c
- nls_cp437.c
- nls_cp737.c
- nls_cp775.c
- nls_cp850.c
- nls_cp852.c
- nls_cp855.c
- nls_cp857.c
- nls_cp860.c
- nls_cp861.c
- nls_cp862.c
- nls_cp863.c
- nls_cp864.c
- nls_cp865.c
- nls_cp866.c
- nls_cp869.c
- nls_cp874.c
- nls_cp936.c
- nls_cp950.c
- nls_iso8859-1.c
- nls_iso8859-15.c
- nls_iso8859-2.c
- nls_iso8859-3.c
- nls_iso8859-4.c
- nls_iso8859-5.c
- nls_iso8859-6.c
- nls_iso8859-7.c
- nls_iso8859-8.c
- nls_iso8859-9.c
- nls_koi8-r.c
实现了下列函数:
- extern int utf8_mbtowc(__u16 *, const __u8 *, int);
- extern int utf8_mbstowcs(__u16 *, const __u8 *, int);
- extern int utf8_wctomb(__u8 *, __u16, int);
- extern int utf8_wcstombs(__u8 *, const __u16 *, int);
这样在加载相应的文件系统时就可以用下面的参数来设置Codepage:
对于Codepage 437 来说
mount -t vfat /dev/hda1 /mnt/1 -o codepage=437,iocharset=cp437
这样在Linux下就可以正常访问不同语种的长文件名了.
3. Linux下支持的Codepage
- nls codepage 437 -- 美国/加拿大英语
- nls codepage 737 -- 希腊语
- nls codepage 775 -- 波罗的海语
- nls codepage 850 -- 包括西欧语种(德语,西班牙语,意大利语)中的一些字符
- nls codepage 852 -- Latin 2 包括中东欧语种(阿尔巴尼亚语,克罗地亚语,捷克语,英语,芬兰语,匈牙利语,爱尔兰语,德语,波兰语,罗马利亚语,塞尔维亚语,斯洛伐克语,斯洛文尼亚语,Sorbian语)
- nls codepage 855 -- 斯拉夫语
- nls codepage 857 -- 土耳其语
- nls codepage 860 -- 葡萄牙语
- nls codepage 861 -- 冰岛语
- nls codepage 862 -- 希伯来语
- nls codepage 863 -- 加拿大语
- nls codepage 864 -- 阿拉伯语
- nls codepage 865 -- 日尔曼语系
- nls codepage 866 -- 斯拉夫语/俄语
- nls codepage 869 -- 希腊语(2)
- nls codepage 874 -- 泰语
- nls codepage 936 -- 简体中文GBK
- nls codepage 950 -- 繁体中文Big5
- nls iso8859-1 -- 西欧语系(阿尔巴尼亚语,西班牙加泰罗尼亚语,丹麦语,荷兰语,英语,Faeroese语,芬兰语,法语,德语,加里西亚语,爱尔兰语,冰岛语,意大利语,挪威语,葡萄牙语,瑞士语.)这同时适用于美国英语.
- nls iso8859-2 -- Latin 2 字符集,斯拉夫/中欧语系(捷克语,德语,匈牙利语,波兰语,罗马尼亚语,克罗地亚语,斯洛伐克语,斯洛文尼亚语)
- nls iso8859-3 -- Latin 3 字符集, (世界语,加里西亚语,马耳他语,土耳其语)
- nls iso8859-4 -- Latin 4 字符集, (爱莎尼亚语,拉脱维亚语,立陶宛语),是Latin 6 字符集的前序标准
- nls iso8859-5 -- 斯拉夫语系(保加利亚语,Byelorussian语,马其顿语,俄语,塞尔维亚语,乌克兰语) 一般推荐使用 KOI8-R codepage
- nls iso8859-6 -- 阿拉伯语.
- nls iso8859-7 -- 现代希腊语
- nls iso8859-8 -- 希伯来语
- nls iso8859-9 -- Latin 5 字符集, (去掉了 Latin 1中不经常使用的一些冰岛语字符而代以土耳其语字符)
- nls iso8859-10 -- Latin 6 字符集, (因纽特(格陵兰)语,萨摩斯岛语等Latin 4 中没有包括的北欧语种)
- nls iso8859-15 -- Latin 9 字符集, 是Latin 1字符集的更新版本,去掉一些不常用的字符,增加了对爱莎尼亚语的支持,修正了法语和芬兰语部份,增加了欧元字符)
- nls koi8-r -- 俄语的缺省支持
4. 简体中文GBK/繁体中文Big5的Codepage
如何制作简体中文GBK/繁体中文Big5的Codepage?
- 从 Unicode 组织取得GBK/Big5的Unicode的定义.
由于GBK是基于ISO 10646-1:1993标准的,而相应的日文是JIS X 0221-1995,韩文是KS C 5700-1995,他们被提交到Unicode标准的时间表为:
Unicode Version 1.0
Unicode Version 1.1 <-> ISO 10646-1:1993, JIS X 0221-1995, GB 13000.1-93
Unicode Version 2.0 <-> KS C 5700-1995从Windows 95开始均采用GBK编码. 您需要的是 CP936.TXT和 BIG5.TXT
- 然后使用下面的程序就可以将其转化为Linux核心需要的Unicode<->GBK码表
./genmap BIG5.txt | perl uni2big5.pl
./genmap CP936.txt | perl uni2gbk.pl - 再修改fat/vfat/ntfs的相关函数就可以完成对核心的修改工作. 具体使用时可以使用下面的命令:
- 从 Unicode 组织取得GBK/Big5的Unicode的定义.
- 简体中文: mount -t vfat /dev/hda1 /mnt/1 -o codepage=936,iocharset=cp936
- 繁体中文: mount -t vfat /dev/hda1 /mnt/1 -o codepage=950,iocharset=cp936
有趣的是,由于GBK包含了全部的GB2312/Big5/JIS的内码,所以使用936的Codepage也可以显示Big5的文件名.
5. 附录
5.1 作者和相关文档
制作codepage950支持的是台湾的 cosmos先生, 主页为 http://www.cis.nctu.edu.tw:8080/~is84086/Project/kernel_cp950/
制作GBK的cp936支持的是TurboLinux的中文研发小组的 方汉和 陈向阳
5.2 genmap
#!/bin/sh
cat $1 | awk '{if(index($1,"#")==0)print $0}' | awk 'BEGIN{FS="0x"}{print $2 $3}' |
awk '{if(length($1)==length($2))print $1,$2}'5.3 uni2big5.pl
1 # !/usr/bin/perl
2
3 @code = (
4 " 00 " , " 01 " , " 02 " , " 03 " , " 04 " , " 05 " , " 06 " , " 07 " ,
5 " 08 " , " 09 " , " 0A " , " 0B " , " 0C " , " 0D " , " 0E " , " 0F " ,
6 " 10 " , " 11 " , " 12 " , " 13 " , " 14 " , " 15 " , " 16 " , " 17 " ,
7 " 18 " , " 19 " , " 1A " , " 1B " , " 1C " , " 1D " , " 1E " , " 1F " ,
8 " 20 " , " 21 " , " 22 " , " 23 " , " 24 " , " 25 " , " 26 " , " 27 " ,
9 " 28 " , " 29 " , " 2A " , " 2B " , " 2C " , " 2D " , " 2E " , " 2F " ,
10 " 30 " , " 31 " , " 32 " , " 33 " , " 34 " , " 35 " , " 36 " , " 37 " ,
11 " 38 " , " 39 " , " 3A " , " 3B " , " 3C " , " 3D " , " 3E " , " 3F " ,
12 " 40 " , " 41 " , " 42 " , " 43 " , " 44 " , " 45 " , " 46 " , " 47 " ,
13 " 48 " , " 49 " , " 4A " , " 4B " , " 4C " , " 4D " , " 4E " , " 4F " ,
14 " 50 " , " 51 " , " 52 " , " 53 " , " 54 " , " 55 " , " 56 " , " 57 " ,
15 " 58 " , " 59 " , " 5A " , " 5B " , " 5C " , " 5D " , " 5E " , " 5F " ,
16 " 60 " , " 61 " , " 62 " , " 63 " , " 64 " , " 65 " , " 66 " , " 67 " ,
17 " 68 " , " 69 " , " 6A " , " 6B " , " 6C " , " 6D " , " 6E " , " 6F " ,
18 " 70 " , " 71 " , " 72 " , " 73 " , " 74 " , " 75 " , " 76 " , " 77 " ,
19 " 78 " , " 79 " , " 7A " , " 7B " , " 7C " , " 7D " , " 7E " , " 7F " ,
20 " 80 " , " 81 " , " 82 " , " 83 " , " 84 " , " 85 " , " 86 " , " 87 " ,
21 " 88 " , " 89 " , " 8A " , " 8B " , " 8C " , " 8D " , " 8E " , " 8F " ,
22 " 90 " , " 91 " , " 92 " , " 93 " , " 94 " , " 95 " , " 96 " , " 97 " ,
23 " 98 " , " 99 " , " 9A " , " 9B " , " 9C " , " 9D " , " 9E " , " 9F " ,
24 " A0 " , " A1 " , " A2 " , " A3 " , " A4 " , " A5 " , " A6 " , " A7 " ,
25 " A8 " , " A9 " , " AA " , " AB " , " AC " , " AD " , " AE " , " AF " ,
26 " B0 " , " B1 " , " B2 " , " B3 " , " B4 " , " B5 " , " B6 " , " B7 " ,
27 " B8 " , " B9 " , " BA " , " BB " , " BC " , " BD " , " BE " , " BF " ,
28 " C0 " , " C1 " , " C2 " , " C3 " , " C4 " , " C5 " , " C6 " , " C7 " ,
29 " C8 " , " C9 " , " CA " , " CB " , " CC " , " CD " , " CE " , " CF " ,
30 " D0 " , " D1 " , " D2 " , " D3 " , " D4 " , " D5 " , " D6 " , " D7 " ,
31 " D8 " , " D9 " , " DA " , " DB " , " DC " , " DD " , " DE " , " DF " ,
32 " E0 " , " E1 " , " E2 " , " E3 " , " E4 " , " E5 " , " E6 " , " E7 " ,
33 " E8 " , " E9 " , " EA " , " EB " , " EC " , " ED " , " EE " , " EF " ,
34 " F0 " , " F1 " , " F2 " , " F3 " , " F4 " , " F5 " , " F6 " , " F7 " ,
35 " F8 " , " F9 " , " FA " , " FB " , " FC " , " FD " , " FE " , " FF " );
36
37 while ( < STDIN > ){
38 ( $unicode , $big5 ) = split ;
39 ( $high , $low ) = $unicode =~ / ( .. )( .. ) / ;
40 $table2 { $high }{ $low } = $big5 ;
41 ( $high , $low ) = $big5 =~ / ( .. )( .. ) / ;
42 $table { $high }{ $low } = $unicode ;
43 }
44
45 print << EOF ;
46 /*
47 * linux / fs / nls_cp874 . c
48 *
49 * Charset cp874 translation tables .
50 * Generated automatically from the Unicode and charset
51 * tables from the Unicode Organization (www . unicode . org) .
52 * The Unicode to charset table has only exact mappings .
53 */
54
55 # include
59
60 /* A1 - F9 */
61 static struct nls_unicode charset2uni[( 0xF9 - 0xA1 + 1 ) * ( 0x100 - 0x60 )] = {
62 EOF
63
64 for ( $high = 0xA1 ; $high <= 0xF9 ; $high ++ ){
65 for ( $low = 0x40 ; $low <= 0x7F ; $low ++ ){
66 $unicode = $table2 { $code [ $high ]}{ $code [ $low ]};
67 $unicode = " 0000 " if ( ! ( defined $unicode ));
68 print " \n\t " if ( $low % 4 == 0 );
69 print " /* $code[$high]$code[$low]*/\n\t " if ( $low % 0x10 == 0 );
70 ( $uhigh , $ulow ) = $unicode =~ / ( .. )( .. ) / ;
71 printf ( " {0x%2s, 0x%2s}, " , $ulow , $uhigh );
72 }
73 for ( $low = 0xA0 ; $low <= 0xFF ; $low ++ ){
74 $unicode = $table2 { $code [ $high ]}{ $code [ $low ]};
75 $unicode = " 0000 " if ( ! ( defined $unicode ));
76 print " \n\t " if ( $low % 4 == 0 );
77 print " /* $code[$high]$code[$low]*/\n\t " if ( $low % 0x10 == 0 );
78 ( $uhigh , $ulow ) = $unicode =~ / ( .. )( .. ) / ;
79 printf ( " {0x%2s, 0x%2s}, " , $ulow , $uhigh );
80 }
81 }
82
83 print " \n};\n\n " ;
84 for ( $high = 1 ; $high <= 255 ; $high ++ ){
85 if ( defined $table { $code [ $high ]}){
86 print " static unsigned char page$code[$high]\[512\] = {\n\t " ;
87 for ( $low = 0 ; $low <= 255 ; $low ++ ){
88 $big5 = $table { $code [ $high ]}{ $code [ $low ]};
89 $big5 = " 3F3F " if ( ! ( defined $big5 ));
90 if ( $low > 0 && $low % 4 == 0 ){
91 printf ( " /* 0x%02X-0x%02X */\n\t " , $low - 4 , $low - 1 );
92 }
93 print " \n\t " if ( $low == 0x80 );
94 ( $bhigh , $blow ) = $big5 =~ / ( .. )( .. ) / ;
95 printf ( " 0x%2s, 0x%2s, " , $bhigh , $blow );
96 }
97 print " /* 0xFC-0xFF */\n};\n\n " ;
98 }
99 }
100
101 print " static unsigned char *page_uni2charset[256] = { " ;
102 for ( $high = 0 ; $high <= 255 ; $high ++ ){
103 print " \n\t " if ( $high % 8 == 0 );
104 if ( $high > 0 && defined $table { $code [ $high ]}){
105 print " page$code[$high], " ;
106 }
107 else {
108 print " NULL, " ;
109 }
110 }
111 print << EOF ;
112
113 };
114
115 static unsigned char charset2upper[ 256 ] = {
116 0x00 , 0x01 , 0x02 , 0x03 , 0x04 , 0x05 , 0x06 , 0x07 , /* 0x00 - 0x07 */
117 0x08 , 0x09 , 0x0a , 0x0b , 0x0c , 0x0d , 0x0e , 0x0f , /* 0x08 - 0x0f */
118 0x10 , 0x11 , 0x12 , 0x13 , 0x14 , 0x15 , 0x16 , 0x17 , /* 0x10 - 0x17 */
119 0x18 , 0x19 , 0x1a , 0x1b , 0x1c , 0x1d , 0x1e , 0x1f , /* 0x18 - 0x1f */
120 0x20 , 0x21 , 0x22 , 0x23 , 0x24 , 0x25 , 0x26 , 0x27 , /* 0x20 - 0x27 */
121 0x28 , 0x29 , 0x2a , 0x2b , 0x2c , 0x2d , 0x2e , 0x2f , /* 0x28 - 0x2f */
122 0x30 , 0x31 , 0x32 , 0x33 , 0x34 , 0x35 , 0x36 , 0x37 , /* 0x30 - 0x37 */
123 0x38 , 0x39 , 0x3a , 0x3b , 0x3c , 0x3d , 0x3e , 0x3f , /* 0x38 - 0x3f */
124 0x40 , 0x41 , 0x42 , 0x43 , 0x44 , 0x45 , 0x46 , 0x47 , /* 0x40 - 0x47 */
125 0x48 , 0x49 , 0x4a , 0x4b , 0x4c , 0x4d , 0x4e , 0x4f , /* 0x48 - 0x4f */
126 0x50 , 0x51 , 0x52 , 0x53 , 0x54 , 0x55 , 0x56 , 0x57 , /* 0x50 - 0x57 */
127 0x58 , 0x59 , 0x5a , 0x5b , 0x5c , 0x5d , 0x5e , 0x5f , /* 0x58 - 0x5f */
128 0x60 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , /* 0x60 - 0x67 */
129 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , /* 0x68 - 0x6f */
130 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , /* 0x70 - 0x77 */
131 0x00 , 0x00 , 0x00 , 0x7b , 0x7c , 0x7d , 0x7e , 0x7f , /* 0x78 - 0x7f */
132 0x80 , 0x81 , 0x82 , 0x83 , 0x84 , 0x85 , 0x86 , 0x87 , /* 0x80 - 0x87 */
133 0x88 , 0x89 , 0x8a , 0x8b , 0x8c , 0x8d , 0x8e , 0x8f , /* 0x88 - 0x8f */
134 0x90 , 0x91 , 0x92 , 0x93 , 0x94 , 0x95 , 0x96 , 0x97 , /* 0x90 - 0x97 */
135 0x98 , 0x99 , 0x9a , 0x00 , 0x9c , 0x00 , 0x00 , 0x00 , /* 0x98 - 0x9f */
136 0x00 , 0x00 , 0x00 , 0x00 , 0xa4 , 0xa5 , 0xa6 , 0xa7 , /* 0xa0 - 0xa7 */
137 0xa8 , 0xa9 , 0xaa , 0xab , 0xac , 0xad , 0xae , 0xaf , /* 0xa8 - 0xaf */
138 0xb0 , 0xb1 , 0xb2 , 0xb3 , 0xb4 , 0xb5 , 0xb6 , 0xb7 , /* 0xb0 - 0xb7 */
139 0xb8 , 0xb9 , 0xba , 0xbb , 0xbc , 0xbd , 0xbe , 0xbf , /* 0xb8 - 0xbf */
140 0xc0 , 0xc1 , 0xc2 , 0xc3 , 0xc4 , 0xc5 , 0xc6 , 0xc7 , /* 0xc0 - 0xc7 */
141 0xc8 , 0xc9 , 0xca , 0xcb , 0xcc , 0xcd , 0xce , 0xcf , /* 0xc8 - 0xcf */
142 0xd0 , 0xd1 , 0xd2 , 0xd3 , 0xd4 , 0xd5 , 0x00 , 0x00 , /* 0xd0 - 0xd7 */
143 0x00 , 0xd9 , 0xda , 0xdb , 0xdc , 0x00 , 0x00 , 0xdf , /* 0xd8 - 0xdf */
144 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , /* 0xe0 - 0xe7 */
145 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0x00 , 0xef , /* 0xe8 - 0xef */
146 0xf0 , 0xf1 , 0x00 , 0x00 , 0x00 , 0xf5 , 0x00 , 0xf7 , /* 0xf0 - 0xf7 */
147 0xf8 , 0xf9 , 0x00 , 0x00 , 0x00 , 0x00 , 0xfe , 0xff , /* 0xf8 - 0xff */
148 };
149
150
151 static void inc_use_count(void)
152 {
153 MOD_INC_USE_COUNT;
154 }
155
156 static void dec_use_count(void)
157 {
158 MOD_DEC_USE_COUNT;
159 }
160
161 static struct nls_table table = {
162 " cp950 " ,
163 page_uni2charset ,
164 charset2uni ,
165 inc_use_count ,
166 dec_use_count ,
167 NULL
168 };
169
170 int init_nls_cp950(void)
171 {
172 return register_nls();
173 }
174
175 # ifdef MODULE
176 int init_module(void)
177 {
178 return init_nls_cp950();
179 }
180
181
182 void cleanup_module(void)
183 {
184 unregister_nls();
185 return ;
186 }
187 # endif
188
189 /*
190 * Overrides for Emacs so that we follow Linus ' s tabbing style.
191 * Emacs will notice this stuff at the end of the file and automatically
192 * adjust the settings for this buffer only. This must remain at the end
193 * of the file.
194 *
195 ---------------------------------------------------------------------------
196 * Local variables:
197 * c-indent-level: 8
198 * c-brace-imaginary-offset: 0
199 * c-brace-offset: -8
200 * c-argdecl-indent: 8
201 * c-label-offset: -8
202 * c-continued-statement-offset: 8
203 * c-continued-brace-offset: 0
204 * End:
205 */
206 EOF
207
208 5.4 uni2gbk.pl
209
210 #!/usr/bin/perl
211
212 @code = (
213 "00", "01", "02", "03", "04", "05", "06", "07",
214 "08", "09", "0A", "0B", "0C", "0D", "0E", "0F",
215 "10", "11", "12", "13", "14", "15", "16", "17",
216 "18", "19", "1A", "1B", "1C", "1D", "1E", "1F",
217 "20", "21", "22", "23", "24", "25", "26", "27",
218 "28", "29", "2A", "2B", "2C", "2D", "2E", "2F",
219 "30", "31", "32", "33", "34", "35", "36", "37",
220 "38", "39", "3A", "3B", "3C", "3D", "3E", "3F",
221 "40", "41", "42", "43", "44", "45", "46", "47",
222 "48", "49", "4A", "4B", "4C", "4D", "4E", "4F",
223 "50", "51", "52", "53", "54", "55", "56", "57",
224 "58", "59", "5A", "5B", "5C", "5D", "5E", "5F",
225 "60", "61", "62", "63", "64", "65", "66", "67",
226 "68", "69", "6A", "6B", "6C", "6D", "6E", "6F",
227 "70", "71", "72", "73", "74", "75", "76", "77",
228 "78", "79", "7A", "7B", "7C", "7D", "7E", "7F",
229 "80", "81", "82", "83", "84", "85", "86", "87",
230 "88", "89", "8A", "8B", "8C", "8D", "8E", "8F",
231 "90", "91", "92", "93", "94", "95", "96", "97",
232 "98", "99", "9A", "9B", "9C", "9D", "9E", "9F",
233 "A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7",
234 "A8", "A9", "AA", "AB", "AC", "AD", "AE", "AF",
235 "B0", "B1", "B2", "B3", "B4", "B5", "B6", "B7",
236 "B8", "B9", "BA", "BB", "BC", "BD", "BE", "BF",
237 "C0", "C1", "C2", "C3", "C4", "C5", "C6", "C7",
238 "C8", "C9", "CA", "CB", "CC", "CD", "CE", "CF",
239 "D0", "D1", "D2", "D3", "D4", "D5", "D6", "D7",
240 "D8", "D9", "DA", "DB", "DC", "DD", "DE", "DF",
241 "E0", "E1", "E2", "E3", "E4", "E5", "E6", "E7",
242 "E8", "E9", "EA", "EB", "EC", "ED", "EE", "EF",
243 "F0", "F1", "F2", "F3", "F4", "F5", "F6", "F7",
244 "F8", "F9", "FA", "FB", "FC", "FD", "FE", "FF");
245
246 while (){ 247 ($unicode, $big5) = split;
248 ($high, $low) = $unicode =~ /(..)(..)/;
249 $table2{$high}{$low} = $big5;
250 ($high, $low) = $big5 =~ /(..)(..)/;
251 $table{$high}{$low} = $unicode;
252 }
253
254 print <
256 * linux/fs/nls_cp936.c
257 *
258 * Charset cp936 translation tables.
259 * Generated automatically from the Unicode and charset
260 * tables from the Unicode Organization (www.unicode.org).
261 * The Unicode to charset table has only exact mappings.
262 */
263
264 #include
269 /* 81 - FE*/
270 static struct nls_unicode charset2uni[(0xFE-0x81+1)*(0x100-0x40)] = {
271 EOF
272
273 for ($high=0x81; $high <= 0xFE; $high++){
274 for ($low=0x40; $low <= 0x7F; $low++){
275 $unicode = $table2{$code[$high]}{$code[$low]};
276 $unicode = "0000" if (!(defined $unicode));
277 print "\n\t" if ($low%4 == 0);
278 print "/* $code[$high]$code[$low]*/\n\t" if ($low%0x10 == 0);
279 ($uhigh, $ulow) = $unicode =~ /(..)(..)/;
280 printf("{0x%2s, 0x%2s}, ", $ulow, $uhigh);
281 }
282 for ($low=0x80; $low <= 0xFF; $low++){
283 $unicode = $table2{$code[$high]}{$code[$low]};
284 $unicode = "0000" if (!(defined $unicode));
285 print "\n\t" if ($low%4 == 0);
286 print "/* $code[$high]$code[$low]*/\n\t" if ($low%0x10 == 0);
287 ($uhigh, $ulow) = $unicode =~ /(..)(..)/;
288 printf("{0x%2s, 0x%2s}, ", $ulow, $uhigh);
289 }
290 }
291
292 print "\n};\n\n";
293 for ($high=1; $high <= 255;$high++){
294 if (defined $table{$code[$high]}){
295 print "static unsigned char page$code[$high]\[512\] = {\n\t";
296 for ($low=0; $low<=255;$low++){
297 $big5 = $table{$code[$high]}{$code[$low]};
298 $big5 = "3F3F" if (!(defined $big5));
299 if ($low > 0 && $low%4 == 0){
300 printf("/* 0x%02X-0x%02X */\n\t", $low-4, $low-1);
301 }
302 print "\n\t" if ($low == 0x80);
303 ($bhigh, $blow) = $big5 =~ /(..)(..)/;
304 printf("0x%2s, 0x%2s, ", $bhigh, $blow);
305 }
306 print "/* 0xFC-0xFF */\n};\n\n";
307 }
308 }
309
310 print "static unsigned char *page_uni2charset[256] = {";
311 for ($high=0; $high<=255;$high++){
312 print "\n\t" if ($high%8 == 0);
313 if ($high>0 && defined $table{$code[$high]}){
314 print "page$code[$high], ";
315 }
316 else{
317 print "NULL, ";
318 }
319 }
320 print <
322 };
323
324 static unsigned char charset2upper[256] = {
325 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, /* 0x00-0x07 */
326 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, /* 0x08-0x0f */
327 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, /* 0x10-0x17 */
328 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, /* 0x18-0x1f */
329 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, /* 0x20-0x27 */
330 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, /* 0x28-0x2f */
331 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, /* 0x30-0x37 */
332 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, /* 0x38-0x3f */
333 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, /* 0x40-0x47 */
334 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, /* 0x48-0x4f */
335 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, /* 0x50-0x57 */
336 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, /* 0x58-0x5f */
337 0x60, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x60-0x67 */
338 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x68-0x6f */
339 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x70-0x77 */
340 0x00, 0x00, 0x00, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, /* 0x78-0x7f */
341 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, /* 0x80-0x87 */
342 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, /* 0x88-0x8f */
343 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, /* 0x90-0x97 */
344 0x98, 0x99, 0x9a, 0x00, 0x9c, 0x00, 0x00, 0x00, /* 0x98-0x9f */
345 0x00, 0x00, 0x00, 0x00, 0xa4, 0xa5, 0xa6, 0xa7, /* 0xa0-0xa7 */
346 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, /* 0xa8-0xaf */
347 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, /* 0xb0-0xb7 */
348 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, /* 0xb8-0xbf */
349 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, /* 0xc0-0xc7 */
350 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, /* 0xc8-0xcf */
351 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0x00, 0x00, /* 0xd0-0xd7 */
352 0x00, 0xd9, 0xda, 0xdb, 0xdc, 0x00, 0x00, 0xdf, /* 0xd8-0xdf */
353 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xe0-0xe7 */
354 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xef, /* 0xe8-0xef */
355 0xf0, 0xf1, 0x00, 0x00, 0x00, 0xf5, 0x00, 0xf7, /* 0xf0-0xf7 */
356 0xf8, 0xf9, 0x00, 0x00, 0x00, 0x00, 0xfe, 0xff, /* 0xf8-0xff */
357 };
358
359
360 static void inc_use_count(void)
361 {
362 MOD_INC_USE_COUNT;
363 }
364
365 static void dec_use_count(void)
366 {
367 MOD_DEC_USE_COUNT;
368 }
369
370 static struct nls_table table = {
371 "cp936",
372 page_uni2charset,
373 charset2uni,
374 inc_use_count,
375 dec_use_count,
376 NULL
377 };
378
379 int init_nls_cp936(void)
380 {
381 return register_nls();
382 }
383
384 #ifdef MODULE
385 int init_module(void)
386 {
387 return init_nls_cp936();
388 }
389
390
391 void cleanup_module(void)
392 {
393 unregister_nls();
394 return;
395 }
396 #endif
397
398 /*
399 * Overrides for Emacs so that we follow Linus ' s tabbing style .
400 * Emacs will notice this stuff at the end of the file and automatically
401 * adjust the settings for this buffer only . This must remain at the end
402 * of the file .
403 *
404 ---------------------------------------------------------------------------
405 * Local variables :
406 * c - indent - level : 8
407 * c - brace - imaginary - offset : 0
408 * c - brace - offset : - 8
409 * c - argdecl - indent : 8
410 * c - label - offset : - 8
411 * c - continued - statement - offset : 8
412 * c - continued - brace - offset : 0
413 * End :
414 */
415 EOF
416
417 5.5 转换CODEPAGE的工具
418
419 /*
420 * CPI . C : A program to examine MSDOS codepage files ( *. cpi)
421 * and extract specific codepages .
422 * Compiles under Linux & DOS (using BC ++ 3.1 ) .
423 *
424 * Compile : gcc - o cpi cpi . c
425 * Call : codepage file . cpi [ - a |- l | nnn]
426 *
427 * Author : Ahmed M . Naas (ahmed @oea . xs4all . nl)
428 * Many changes : aeb @cwi . nl [changed until it would handle all
429 * *. cpi files people have sent me; I have no documentation ,
430 * so all this is experimental]
431 * Remains to do : DRDOS fonts .
432 *
433 * Copyright : Public domain .
434 */
435
436 # include
440
441 int handle_codepage( int );
442 void handle_fontfile(void);
443
444 # define PACKED __attribute__ ((packed))
445 /* Use this (instead of the above) to compile under MSDOS */
446 /* # define PACKED */
447
448 struct {
449 unsigned char id[ 8 ] PACKED;
450 unsigned char res[ 8 ] PACKED;
451 unsigned short num_pointers PACKED;
452 unsigned char p_type PACKED;
453 unsigned long offset PACKED;
454 } FontFileHeader;
455
456 struct {
457 unsigned short num_codepages PACKED;
458 } FontInfoHeader;
459
460 struct {
461 unsigned short size PACKED;
462 unsigned long off_nexthdr PACKED;
463 unsigned short device_type PACKED; /* screen = 1 ; printer = 2 */
464 unsigned char device_name[ 8 ] PACKED;
465 unsigned short codepage PACKED;
466 unsigned char res[ 6 ] PACKED;
467 unsigned long off_font PACKED;
468 } CPEntryHeader;
469
470 struct {
471 unsigned short reserved PACKED;
472 unsigned short num_fonts PACKED;
473 unsigned short size PACKED;
474 } CPInfoHeader;
475
476 struct {
477 unsigned char height PACKED;
478 unsigned char width PACKED;
479 unsigned short reserved PACKED;
480 unsigned short num_chard PACKED;
481 } ScreenFontHeader;
482
483 struct {
484 unsigned short p1 PACKED;
485 unsigned short p2 PACKED;
486 } PrinterFontHeader;
487
488 FILE * in , * out;
489 void usage(void);
490
491 int opta , optc , optl , optL , optx;
492 extern int optind;
493 extern char * optarg;
494
495 unsigned short codepage;
496
497 int main ( int argc , char * argv[])
498 {
499 if (argc < 2 )
500 usage();
501
502 if ((in = fopen(argv[ 1 ] , " r " )) == NULL) {
503 printf ( " \nUnable to open file %s.\n " , argv[ 1 ]);
504 exit ( 0 );
505 }
506
507 opta = optc = optl = optL = optx = 0 ;
508 optind = 2 ;
509 if (argc == 2 )
510 optl = 1 ;
511 else
512 while ( 1 ) {
513 switch(getopt(argc , argv , " alLc " )) {
514 case ' a ' :
515 opta = 1 ;
516 continue ;
517 case ' c ' :
518 optc = 1 ;
519 continue ;
520 case ' L ' :
521 optL = 1 ;
522 continue ;
523 case ' l ' :
524 optl = 1 ;
525 continue ;
526 case ' ? ' :
527 default :
528 usage();
529 case - 1 :
530 break;
531 }
532 break;
533 }
534 if (optind != argc) {
535 if (optind != argc - 1 || opta)
536 usage();
537 codepage = atoi(argv[optind]);
538 optx = 1 ;
539 }
540
541 if (optc)
542 handle_codepage( 0 );
543 else
544 handle_fontfile();
545
546 if (optx) {
547 printf ( " no page %d found\n " , codepage);
548 exit ( 1 );
549 }
550
551 fclose(in);
552 return ( 0 );
553 }
554
555 void
556 handle_fontfile(){
557 int i , j;
558
559 j = fread( , 1 , sizeof(FontFileHeader) , in);
560 if (j != sizeof(FontFileHeader)) {
561 printf ( " error reading FontFileHeader - got %d chars\n " , j);
562 exit ( 1 );
563 }
564 if ( ! strcmp(FontFileHeader . id + 1 , " DRFONT " )) {
565 printf ( " this program cannot handle DRDOS font files\n " );
566 exit ( 1 );
567 }
568 if (optL)
569 printf ( " FontFileHeader: id=%8.8s res=%8.8s num=%d typ=%c offset=%ld\n\n " ,
570 FontFileHeader . id , FontFileHeader . res ,
571 FontFileHeader . num_pointers ,
572 FontFileHeader . p_type ,
573 FontFileHeader . offset);
574
575 j = fread( , 1 , sizeof(FontInfoHeader) , in);
576 if (j != sizeof(FontInfoHeader)) {
577 printf ( " error reading FontInfoHeader - got %d chars\n " , j);
578 exit ( 1 );
579 }
580 if (optL)
581 printf ( " FontInfoHeader: num_codepages=%d\n\n " ,
582 FontInfoHeader . num_codepages);
583
584 for (i = FontInfoHeader . num_codepages; i; i -- )
585 if (handle_codepage(i - 1 ))
586 break;
587 }
588
589 int
590 handle_codepage( int more_to_come) {
591 int j;
592 char outfile[ 20 ];
593 unsigned char * fonts;
594 long inpos , nexthdr;
595
596 j = fread( , 1 , sizeof(CPEntryHeader) , in);
597 if (j != sizeof(CPEntryHeader)) {
598 printf ( " error reading CPEntryHeader - got %d chars\n " , j);
599 exit ( 1 );
600 }
601 if (optL) {
602 int t = CPEntryHeader . device_type;
603 printf ( " CPEntryHeader: size=%d dev=%d [%s] name=%8.8s \
604 codepage=%d\n\t\tres=%6.6s nxt=%ld off_font=%ld\n\n " ,
605 CPEntryHeader . size ,
606 t , (t == 1 ) ? " screen " : (t == 2 ) ? " printer " : " ? " ,
607 CPEntryHeader . device_name ,
608 CPEntryHeader . codepage ,
609 CPEntryHeader . res ,
610 CPEntryHeader . off_nexthdr , CPEntryHeader . off_font);
611 } else if (optl) {
612 printf ( " \nCodepage = %d\n " , CPEntryHeader . codepage);
613 printf ( " Device = %.8s\n " , CPEntryHeader . device_name);
614 }
615 # if 0
616 if (CPEntryHeader . size != sizeof(CPEntryHeader)) {
617 /* seen 26 and 28 , so that the difference below is - 2 or 0 */
618 if (optl)
619 printf ( " Skipping %d bytes of garbage\n " ,
620 CPEntryHeader . size - sizeof(CPEntryHeader));
621 fseek(in , CPEntryHeader . size - sizeof(CPEntryHeader) ,
622 SEEK_CUR);
623 }
624 # endif
625 if ( ! opta && ( ! optx || CPEntryHeader . codepage != codepage) && ! optc)
626 goto next ;
627
628 inpos = ftell(in);
629 if (inpos != CPEntryHeader . off_font && ! optc) {
630 if (optL)
631 printf ( " pos=%ld font at %ld\n " , inpos , CPEntryHeader . off_font);
632 fseek(in , CPEntryHeader . off_font , SEEK_SET);
633 }
634
635 j = fread( , 1 , sizeof(CPInfoHeader) , in);
636 if (j != sizeof(CPInfoHeader)) {
637 printf ( " error reading CPInfoHeader - got %d chars\n " , j);
638 exit ( 1 );
639 }
640 if (optl) {
641 printf ( " Number of Fonts = %d\n " , CPInfoHeader . num_fonts);
642 printf ( " Size of Bitmap = %d\n " , CPInfoHeader . size);
643 }
644 if (CPInfoHeader . num_fonts == 0 )
645 goto next ;
646 if (optc)
647 return 0 ;
648
649 sprintf (outfile , " %d.cp " , CPEntryHeader . codepage);
650 if ((out = fopen(outfile , " w " )) == NULL) {
651 printf ( " \nUnable to open file %s.\n " , outfile);
652 exit ( 1 );
653 } else printf ( " \nWriting %s\n " , outfile);
654
655 fonts = (unsigned char * ) malloc(CPInfoHeader . size);
656
657 fread(fonts , CPInfoHeader . size , 1 , in);
658 fwrite( , sizeof(CPEntryHeader) , 1 , out);
659 fwrite( , sizeof(CPInfoHeader) , 1 , out);
660 j = fwrite(fonts , 1 , CPInfoHeader . size , out);
661 if (j != CPInfoHeader . size) {
662 printf ( " error writing %s - wrote %d chars\n " , outfile , j);
663 exit ( 1 );
664 }
665 fclose(out);
666 free(fonts);
667 if (optx) exit ( 0 );
668 next :
669 /*
670 * It seems that if entry headers and fonts are interspersed ,
671 * then nexthdr will point past the font , regardless of
672 * whether more entries follow .
673 * Otherwise , first all entry headers are given , and then
674 * all fonts; in this case nexthdr will be 0 in the last entry .
675 */
676 nexthdr = CPEntryHeader . off_nexthdr;
677 if (nexthdr == 0 || nexthdr == - 1 ) {
678 if (more_to_come) {
679 printf ( " mode codepages expected, but nexthdr=%ld\n " ,
680 nexthdr);
681 exit ( 1 );
682 } else
683 return 1 ;
684 }
685
686 inpos = ftell(in);
687 if (inpos != CPEntryHeader . off_nexthdr) {
688 if (optL)
689 printf ( " pos=%ld nexthdr at %ld\n " , inpos , nexthdr);
690 if (opta && ! more_to_come) {
691 printf ( " no more code pages, but nexthdr != 0\n " );
692 return 1 ;
693 }
694
695 fseek(in , CPEntryHeader . off_nexthdr , SEEK_SET);
696 }
697
698 return 0 ;
699 }
700
701 void usage(void)
702 {
703 printf ( " \nUsage: cpi code_page_file [-c] [-L] [-l] [-a|nnn]\n " );
704 printf ( " -c: input file is a single codepage\n " );
705 printf ( " -L: print header info (you don't want to see this)\n " );
706 printf ( " -l or no option: list all codepages contained in the file\n " );
707 printf ( " -a: extract all codepages from the file\n " );
708 printf ( " nnn (3 digits): extract codepage nnn from the file\n " );
709 printf ( " Example: cpi ega.cpi 850 \n " );
710 printf ( " will create a file 850.cp containing the requested codepage.\n\n " );
711 exit ( 1 );
712 }