Dear Users,
In v2019.05.19.0 we’ve added an option to convert the character encoding of text files.
To achieve this we’ve added the following Pascal Script: psChangeTxtEncodingExt
Function psChangeTxtEncodingExt(Source, Destination: String; SrcEncoding, DstEncoding: Integer; WriteBOM: Boolean): Boolean;
- Source: Source File
- Destination: Destination File
- SrcEncoding: Encoding ID Source (please check list below)
- DstEncoding: Encoding ID Destination (please check list below)
- WriteBOM: Write BOM Destination File (Normally True)
The function is also able to read the encoding tag of xml source files and automatically set the source encoding ID.
In the following example we’re going to convert EBCDIC Codepage 273 (German) text files to UTF8.
> Be sure to add and set a file filter so it will only handle text files (txt, xml, …)
>Add a Pascal Script as Destination
Const
ctOutputPath = 'C:\Test\In_Patrick\Out\';
Begin
psExitCode:= 0;
// Source File: EBCDIC in EBCDIC Codepage 273 (German) or Codepage 1141 (German incl. EUR-Sign)
// Destination File: UTF8
// https://limagito.com/text-character-encoding-conversion/
If psChangeTxtEncodingExt(psFilePath + psFileName, ctOutputpath + psFileName, 20273, 65001, True) Then
Begin
psLogWrite(1, '', 'Conversion to ' + ctOutputpath + psFileName + ' Successful');
psExitCode:= 1;
End;
End.
> RunTime Log Result
The function is able to convert quite some encodings:
We have our own Encoding ID’s:
EncodingID Additional information
0 UTF8
1 UTF7
2 Unicode
3 Default
4 Big Endian Unicode
5 ASCII
6 ANSI
7 Reserved
8 Reserved
9 XML Auto Encoding detection (Source Only)
But also all Windows Encoding ID’s can be used:
Code Page Identifiers
EncodingID Additional information
37 IBM037, IBM EBCDIC US-Canada
437 IBM437, OEM United States
500 IBM500, IBM EBCDIC International
708 ASMO-708, Arabic (ASMO 708)
709 ASMO-709, Arabic ASMO-449+, BCON V4
710 ASMO-710, Arabic Transparent Arabic
720 DOS-720, Arabic (Transparent ASMO); Arabic (DOS)
737 IBM737, OEM Greek (formerly 437G); Greek (DOS)
775 IBM775, OEM Baltic; Baltic (DOS)
850 IBM850, OEM Multilingual Latin 1; Western European (DOS)
852 IBM852, OEM Latin 2; Central European (DOS)
855 IBM855, OEM Cyrillic (primarily Russian)
857 IBM857, OEM Turkish; Turkish (DOS)
858 IBM858, OEM Multilingual Latin 1 + Euro symbol
860 IBM860, OEM Portuguese; Portuguese (DOS)
861 IBM861, OEM Icelandic; Icelandic (DOS)
862 DOS-862, OEM Hebrew; Hebrew (DOS)
863 IBM863, OEM French Canadian; French Canadian (DOS)
864 IBM864, OEM Arabic; Arabic (864)
865 IBM865, OEM Nordic; Nordic (DOS)
866 cp866, OEM Russian; Cyrillic (DOS)
869 IBM869, OEM Modern Greek; Greek, Modern (DOS)
870 IBM870, IBM EBCDIC Multilingual/ROECE (Latin 2); IBM EBCDIC Multilingual Latin 2
874 windows-874, ANSI/OEM Thai (ISO 8859-11); Thai (Windows)
875 cp875, IBM EBCDIC Greek Modern
932 shift_jis, ANSI/OEM Japanese; Japanese (Shift-JIS)
936 gb2312, ANSI/OEM Simplified Chinese (PRC, Singapore); Chinese Simplified (GB2312)
949 ks_c_5601-1987, ANSI/OEM Korean (Unified Hangul Code)
950 big5, ANSI/OEM Traditional Chinese (Taiwan; Hong Kong SAR, PRC); Chinese Traditional (Big5)
1026 IBM1026, IBM EBCDIC Turkish (Latin 5)
1047 IBM01047, IBM EBCDIC Latin 1/Open System
1140 IBM01140, IBM EBCDIC US-Canada (037 + Euro symbol); IBM EBCDIC (US-Canada-Euro)
1141 IBM01141, IBM EBCDIC Germany (20273 + Euro symbol); IBM EBCDIC (Germany-Euro)
1142 IBM01142, IBM EBCDIC Denmark-Norway (20277 + Euro symbol); IBM EBCDIC (Denmark-Norway-Euro)
1143 IBM01143, IBM EBCDIC Finland-Sweden (20278 + Euro symbol); IBM EBCDIC (Finland-Sweden-Euro)
1144 IBM01144, IBM EBCDIC Italy (20280 + Euro symbol); IBM EBCDIC (Italy-Euro)
1145 IBM01145, IBM EBCDIC Latin America-Spain (20284 + Euro symbol); IBM EBCDIC (Spain-Euro)
1146 IBM01146, IBM EBCDIC United Kingdom (20285 + Euro symbol); IBM EBCDIC (UK-Euro)
1147 IBM01147, IBM EBCDIC France (20297 + Euro symbol); IBM EBCDIC (France-Euro)
1148 IBM01148, IBM EBCDIC International (500 + Euro symbol); IBM EBCDIC (International-Euro)
1149 IBM01149, IBM EBCDIC Icelandic (20871 + Euro symbol); IBM EBCDIC (Icelandic-Euro)
1200 utf-16, Unicode UTF-16, little endian byte order (BMP of ISO 10646); available only to managed applications
1201 unicodeFFFE, Unicode UTF-16, big endian byte order; available only to managed applications
1250 windows-1250, ANSI Central European; Central European (Windows)
1251 windows-1251, ANSI Cyrillic; Cyrillic (Windows)
1252 windows-1252, ANSI Latin 1; Western European (Windows)
1253 windows-1253, ANSI Greek; Greek (Windows)
1254 windows-1254, ANSI Turkish; Turkish (Windows)
1255 windows-1255, ANSI Hebrew; Hebrew (Windows)
1256 windows-1256, ANSI Arabic; Arabic (Windows)
1257 windows-1257, ANSI Baltic; Baltic (Windows)
1258 windows-1258, ANSI/OEM Vietnamese; Vietnamese (Windows)
1361 Johab, Korean (Johab)
10000 macintosh, MAC Roman; Western European (Mac)
10001 x-mac-japanese, Japanese (Mac)
10002 x-mac-chinesetrad, MAC Traditional Chinese (Big5); Chinese Traditional (Mac)
10003 x-mac-korean, Korean (Mac)
10004 x-mac-arabic, Arabic (Mac)
10005 x-mac-hebrew, Hebrew (Mac)
10006 x-mac-greek, Greek (Mac)
10007 x-mac-cyrillic, Cyrillic (Mac)
10008 x-mac-chinesesimp, MAC Simplified Chinese (GB 2312); Chinese Simplified (Mac)
10010 x-mac-romanian, Romanian (Mac)
10017 x-mac-ukrainian, Ukrainian (Mac)
10021 x-mac-thai, Thai (Mac)
10029 x-mac-ce, MAC Latin 2; Central European (Mac)
10079 x-mac-icelandic, Icelandic (Mac)
10081 x-mac-turkish, Turkish (Mac)
10082 x-mac-croatian, Croatian (Mac)
12000 utf-32, Unicode UTF-32, little endian byte order; available only to managed applications
12001 utf-32BE, Unicode UTF-32, big endian byte order; available only to managed applications
20000 x-Chinese_CNS, CNS Taiwan; Chinese Traditional (CNS)
20001 x-cp20001, TCA Taiwan
20002 x_Chinese-Eten, Eten Taiwan; Chinese Traditional (Eten)
20003 x-cp20003, IBM5550 Taiwan
20004 x-cp20004, TeleText Taiwan
20005 x-cp20005, Wang Taiwan
20105 x-IA5, IA5 (IRV International Alphabet No. 5, 7-bit); Western European (IA5)
20106 x-IA5-German, IA5 German (7-bit)
20107 x-IA5-Swedish, IA5 Swedish (7-bit)
20108 x-IA5-Norwegian, IA5 Norwegian (7-bit)
20127 us-ascii, US-ASCII (7-bit)
20261 x-cp20261, T.61
20269 x-cp20269, ISO 6937 Non-Spacing Accent
20273 IBM273, IBM EBCDIC Germany
20277 IBM277, IBM EBCDIC Denmark-Norway
20278 IBM278, IBM EBCDIC Finland-Sweden
20280 IBM280, IBM EBCDIC Italy
20284 IBM284, IBM EBCDIC Latin America-Spain
20285 IBM285, IBM EBCDIC United Kingdom
20290 IBM290, IBM EBCDIC Japanese Katakana Extended
20297 IBM297, IBM EBCDIC France
20420 IBM420, IBM EBCDIC Arabic
20423 IBM423, IBM EBCDIC Greek
20424 IBM424, IBM EBCDIC Hebrew
20833 x-EBCDIC-KoreanExtended, IBM EBCDIC Korean Extended
20838 IBM-Thai, IBM EBCDIC Thai
20866 koi8-r, Russian (KOI8-R); Cyrillic (KOI8-R)
20871 IBM871, IBM EBCDIC Icelandic
20880 IBM880, IBM EBCDIC Cyrillic Russian
20905 IBM905, IBM EBCDIC Turkish
20924 IBM00924, IBM EBCDIC Latin 1/Open System (1047 + Euro symbol)
20932 EUC-JP, Japanese (JIS 0208-1990 and 0212-1990)
20936 x-cp20936, Simplified Chinese (GB2312); Chinese Simplified (GB2312-80)
20949 x-cp20949, Korean Wansung
21025 cp1025, IBM EBCDIC Cyrillic Serbian-Bulgarian
21027 (deprecated)
21866 koi8-u, Ukrainian (KOI8-U); Cyrillic (KOI8-U)
28591 iso-8859-1, ISO 8859-1 Latin 1; Western European (ISO)
28592 iso-8859-2, ISO 8859-2 Central European; Central European (ISO)
28593 iso-8859-3, ISO 8859-3 Latin 3
28594 iso-8859-4, ISO 8859-4 Baltic
28595 iso-8859-5, ISO 8859-5 Cyrillic
28596 iso-8859-6, ISO 8859-6 Arabic
28597 iso-8859-7, ISO 8859-7 Greek
28598 iso-8859-8, ISO 8859-8 Hebrew; Hebrew (ISO-Visual)
28599 iso-8859-9, ISO 8859-9 Turkish
28603 iso-8859-13, ISO 8859-13 Estonian
28605 iso-8859-15, ISO 8859-15 Latin 9
29001 x-Europa, Europa 3
38598 iso-8859-8-i, ISO 8859-8 Hebrew; Hebrew (ISO-Logical)
50220 iso-2022-jp, ISO 2022 Japanese with no halfwidth Katakana; Japanese (JIS)
50221 csISO2022JP, ISO 2022 Japanese with halfwidth Katakana; Japanese (JIS-Allow 1 byte Kana)
50222 iso-2022-jp, ISO 2022 Japanese JIS X 0201-1989; Japanese (JIS-Allow 1 byte Kana - SO/SI)
50225 iso-2022-kr, ISO 2022 Korean
50227 x-cp50227, ISO 2022 Simplified Chinese; Chinese Simplified (ISO 2022)
50229 ISO 2022 Traditional Chinese
50930 EBCDIC Japanese (Katakana) Extended
50931 EBCDIC US-Canada and Japanese
50933 EBCDIC Korean Extended and Korean
50935 EBCDIC Simplified Chinese Extended and Simplified Chinese
50936 EBCDIC Simplified Chinese
50937 EBCDIC US-Canada and Traditional Chinese
50939 EBCDIC Japanese (Latin) Extended and Japanese
51932 euc-jp, EUC Japanese
51936 EUC-CN, EUC Simplified Chinese; Chinese Simplified (EUC)
51949 euc-kr, EUC Korean
51950 EUC, Traditional Chinese
52936 hz-gb-2312, HZ-GB2312 Simplified Chinese; Chinese Simplified (HZ)
54936 GB18030, Windows XP and later: GB18030 Simplified Chinese (4 byte); Chinese Simplified (GB18030)
57002 x-iscii-de, ISCII Devanagari
57003 x-iscii-be, ISCII Bangla
57004 x-iscii-ta, ISCII Tamil
57005 x-iscii-te, ISCII Telugu
57006 x-iscii-as, ISCII Assamese
57007 x-iscii-or, ISCII Odia
57008 x-iscii-ka, ISCII Kannada
57009 x-iscii-ma, ISCII Malayalam
57010 x-iscii-gu, ISCII Gujarati
57011 x-iscii-pa, ISCII Punjabi
65000 utf-7, Unicode (UTF-7)
65001 utf-8, Unicode (UTF-8)
Please let us know if you need any help.
Regards,
Limagito Team
