Convert a string to and from various encodings.

The basic supported encodings are roughly as specified in the WHATWG Encoding Standard, but more are also supported unless restriction to web encodings is explicitly specified.

Most encodings supported by Python are implemented, but not currently idna or punycode. Note however that Python makes x-mac-japanese an alias of shift_jis; this has not been done here. Also note that the behaviour in regards to association of encoding names with variants is somewhat different to Python's, partly due to following WHATWG: this affects most CJK codecs (e.g. Python treats shift_jis and ms-kanji differently, while this package does not), but also e.g. "ISO-8859-1".

Main entry points for the package are codecs.infrastructure.encode, codecs.infrastructure.decode and codecs.infrastructure.lookup, all three of which are also available as e.g. codecs.encode for convenience.

The list of codecs (not an exhaustive list of labels, nor close to one) is as follows.

Single-byte extended ASCII encodings:

Major label(s) Meaning
cp437 8-bit United States (DOS)
cp720 8-bit Arabic Letters and Box Drawing (DOS)
cp737 8-bit Greek and Box Drawing (DOS)
cp775 8-bit Baltic Rim (DOS)
cp850 8-bit Western Europe and Canada (DOS)
cp852 8-bit Central European (DOS)
cp855 8-bit Balkan Cyrillic (DOS)
cp856 8-bit Hebrew (DOS)
cp857 8-bit Turkish (DOS)
cp858 8-bit Western Europe and Canada with Euro (DOS)
cp860 8-bit European Portugese (DOS)
cp861 8-bit Icelandic (DOS)
cp862 8-bit Hebrew and Box Drawing (DOS)
cp863 8-bit Quebecois French (DOS)
cp864 8-bit Arabic Positional Forms (DOS)
cp865 8-bit Continental Nordic (DOS)
cp866, ibm866 8-bit Russian Cyrillic (DOS)
cp869 8-bit Greek (DOS)
cp1006 8-bit Urdu
cp1125 8-bit Ukrainian Cyrillic (DOS)
ecma-43-dv, cp367, csascii "8-bit Plain ASCII", i.e. ASCII without backspace composition, and with high bit unused. Note: most ASCII labels are mapped to Windows-1252, per WHATWG.
hp-roman8 8-bit Roman (HP)
iso-8859-2 8-bit Central European (ISO)
iso-8859-3 8-bit South European (Maltese/Esperanto)
iso-8859-4 8-bit North European
iso-8859-5 8-bit Cyrillic (ISO)
iso-8859-6 8-bit Arabic (ASMO/ISO)
iso-8859-7 8-bit Greek (ISO)
iso-8859-8, iso-8859-8-i 8-bit Hebrew (without vowel points). Although some, but not all, of the labels using this mapping request legacy visual-order behaviour (e.g. iso-8859-8, iso-8859-8-e or even visual, but not e.g. iso-8859-8-i), bidirectional conversion for any given markup format is beyond the scope of this package: determining from the label whether legacy visual-order behaviour should be used, and responding if so, should be implemented separately if needed.
iso-8859-10 8-bit Nordic
iso-8859-13 8-bit Baltic Rim (ISO)
iso-8859-14 8-bit Celtic
iso-8859-15 8-bit New Western European
iso-8859-16 8-bit South-Eastern European (ISO)
koi8-r 8-bit Russian Cyrillic (KOI8)
koi8-u, koi8-ru 8-bit Ruthenian/Ukrainian/Belarusian Cyrillic (KOI8)
koi8-t 8-bit Tajik Cyrillic
kz1048 8-bit Kazakh Cyrillic
macintosh 8-bit Roman (Macintosh)
palmos PalmOS code page
ptcp154 8-bit Asian Cyrillic (Paratype)
windows-874, iso-8859-11, tis-620, cp874 8-bit Thai
windows-1250 8-bit Central European (Windows)
windows-1251 8-bit Cyrillic (Windows)
windows-1252, ascii, iso-8859-1, latin1 8-bit Western European. This is in accordance with WHATWG specification in re which mappings to associate with which labels. Note: Python's latin1 is sometimes used to round-trip arbitrary sensu stricto extended ASCII data; in Kuroko, it is better to use x-user-defined for that.
windows-1253 8-bit Greek (Windows)
windows-1254, iso-8859-9 8-bit Turkish
windows-1255 8-bit Hebrew (logical with vowel points)
windows-1256 8-bit Arabic (Windows)
windows-1257 8-bit Baltic Rim (Windows)
windows-1258 8-bit Vietnamese (Windows). Basic codec: encoder will accept text in the form generated by the decoder, but neither NFC nor NFD normalised forms. This follows both Python and WHATWG behaviour. Conversion of text in NFC or NFD forms to encodable form may need to be done in a separate step before using the encoder.
x-mac-arabic 8-bit Arabic (Macintosh)
x-mac-ce 8-bit Central European (Macintosh)
x-mac-croatian 8-bit Gajica
x-mac-cyrillic 8-bit Cyrillic (Macintosh)
x-mac-farsi 8-bit Persian (Macintosh)
x-mac-greek 8-bit Greek (Macintosh)
x-mac-icelandic 8-bit Icelandic (Macintosh)
x-mac-romanian 8-bit Romanian (Macintosh)
x-mac-turkish 8-bit Turkish (Macintosh)
x-user-defined 8-bit User Defined (ASCII based variant: using U+0000–007F, U+F780–F7FF)

Single-byte symbol or dingbat font encodings:

Major label(s) Meaning
cp042 8-bit User Defined (variant using U+0000–001F, U+F020–F0FF). Windows uses that mapping for symbol fonts in some contexts.

8-bit multi-byte Unicode codecs:

Major label(s) Meaning
cesu-8, utf8mb3, utf8-ucs2 CESU-8 (to UTF-16 as UTF-8 is to UTF-32). Mostly for interoperability with existing systems that use it.
gb18030 Chinese GB18030, WHATWG version. Not technically a full UTF in this implementation, since one PUA character is changed to an ideographic space per WHATWG.
utf-8, utf8mb4, utf8-ucs4 UTF-8 without a byte order mark
utf-8-sig UTF-8 with a byte order mark
utf-16 UTF-16 with byte order mark, little endian if missing
utf-16be UTF-16, big endian, no byte order mark
utf-16le UTF-16, little endian, no byte order mark
utf-32 UTF-32 with byte order mark (though byte order can usually also be detected in its absence)
utf-32be UTF-32, big endian, no byte order mark
utf-32le UTF-32, little endian, no byte order mark

8-bit multi-byte legacy CJK codecs:

Major label(s) Meaning
big5, big5-eten Traditional Chinese Big-5, ETen version, condoning HKSCS extensions when decoding.
big5-hkscs Traditional Chinese Big-5 with HKSCS extensions in both directions.
big5-nonetenkana, big5-tw Traditional Chinese Big-5, with BIG5.TXT (non-ETen) layout for kana and Cyrillic.
euc-jp, x-euc-jp Japanese EUC-JP, with Microsoft extensions, permitting JIS X 0212 only when decoding.
euc-jp-full Japanese EUC-JP, with Microsoft extensions, permitting JIS X 0212 in both directions.
euc-jisx0213, euc-jis-2004 Japanese EUC-JP, with JIS X 0213 mappings and extensions.
euc-kr, uhc, windows-949 Korean Unified Hangul Code (superset of EUC-KR, encodes KS C 5601).
gbk, gb2312 Chinese GBK (GB2312 extension), condoning GB18030 when decoding.
johab, johab-ascii Korean Johab (ASCII-compatible stateless standard version)
shift_jis, ms-kanji, windows-31j Japanese Shift JIS (Windows compatible version)
shift-jisx0213, shift-jis-2004 Japanese Shift JIS (JIS X 0213 version)
x-mac-chinesesimp Simplified Chinese GB2312, Macintosh version
x-mac-chinesetrad Traditional Chinese Big5, Macintosh version
x-mac-korean Korean HangulTalk (Macintosh encoding, another superset of EUC-KR)

7-bit stateful codecs:

Major label(s) Meaning
hz-gb-2312 HZ (Usenet Simplified Chinese) encoding
iso-2022-cn 7-bit stateful Chinese (Simplified and Traditional)
iso-2022-jp 7-bit stateful Japanese, web version
iso-2022-jp-ext 7-bit stateful Japanese, including JIS X 0212 and preserving katakana width
iso-2022-jp-1 7-bit stateful Japanese, including JIS X 0212
iso-2022-jp-2 7-bit stateful Multilingual (Japanese, Korean, Greek, Simplified Chinese, Western European)
iso-2022-jp-3 7-bit stateful Japanese, including JIS X 0213 (2000 edition format)
iso-2022-jp-2004 7-bit stateful Japanese, including JIS X 0213 (2004 edition format)
iso-2022-kr 7-bit stateful Korean
jis_encoding 7-bit stateful Japanese, comprehensive version
utf-7 A largely obsolete scheme for mixing ASCII and Base64'd UTF-16BE in e-mail. Included mostly for Python parity.

EBCDIC codecs:

Major label(s) Meaning
cp037 EBCDIC Default (United States, Netherlands, Portugal, Brazil, Australia, New Zealand, Canadian ESA/390)
cp273 EBCDIC German
cp424 EBCDIC Hebrew
cp500 EBCDIC "International" (Belgium, Switzerland, Canadian AS/400)
cp875 EBCDIC Greek
cp933, ibm-933, ibm-1364, johab-ebcdic EBCDIC Korean (Johab, IBM stateful version for EBCDIC)
cp1026 EBCDIC Turkish
cp1140 EBCDIC with Euro Sign

Codecs with unusual behaviour:

Major label(s) Meaning
inverse-base64 Base64 with inverse semantics to preserve type correctness (encoder reads, decoder creates). Error handler is ignored.
inverse-base64hqx Same, but using the BinHex4 alphabet (note: does not in and of itself create the BinHex4 format)
inverse-base64uu Same, but using the uuencode alphabet (note: does not in and of itself create the uuencode format)
inverse-quopri Quoted-Printable, with inverse semantics (encoder reads, decoder creates). Error handler is ignored.
japanese Attempts to detect the encoding of a Japanese document (like the unified "Japanese" option now offered by some browsers' encoding override menus), and raises ValueError if it cannot. Not intended to be used in the encode direction, but will behave as utf-8-sig in that case.
undefined, replacement Represents data for which encoding/decoding must not be attempted. Following WHATWG (and differing from Python), error handlers are accepted, though only by the decoder: the encoder will ignore them.

Notes on error conditions in the ISO-2022-JP family

Like most codecs, the ISO-2022-JP family will generate errors in place of sequences which it cannot interpret. However, they will also generate errors over certain sequences which have no immediate effect on the stream being outputted, so as to prevent them being used for masking syntax that would otherwise be sanitised or escaped (with errors="replace", this will cause a U+FFFD to be inserted, thus preserving the interruption in any syntax). For this to happen is per WHATWG; the specific circumstances in which this happens, however, deliberately vary somewhat from the WHATWG specification.

The iso-2022-jp codec is somewhat more pedantic than the WHATWG approach, and is intended to accept a strict subset of what the WHATWG approach accepts (but a superset of what the WHATWG and Python codecs generate for a single non-concatenated stream), excluding cases that are unlikely to occur in reality except as masking sequences. The WHATWG approach follows UTR #36 in forbidding exactly those cases which RFC 1468 does not permit, meaning that it forbids some cases that often occur in reality as a result of concatenation and are usually benign, and also permits some cases that are less likely to occur in reality and less likely to be benign. The jis_encoding codec (which is also used for decoding, but not for encoding, the iso-2022-jp* labels except for iso-2022-jp itself) permits the cases that result from concatenation but otherwise behaves the same as the iso-2022-jp codec in this regard; this, however, means that Shift Out and Shift In, which are not interpreted by the iso-2022-jp codec but are by the jis_encoding codec, are not currently checked for zero-effect use.

In reaction to the WHATWG approach, UTC L2/20-202 defines two "end states", the first of which forbids no such cases, and the second of which forbids certain additional cases while also being more lenient on the WHATWG-forbidden cases that actually occur in practice—however, it forbids the ordinary output of Python's iso-2022-jp codec under certain circumstances (see bold in table below), and still misses some obvious zero-effect switching sequences (notably, it permits ASCII→ASCII switches anywhere). Accordingly, end state 2 is not followed exactly by either codec either.

All the which having been said, this is purely for the sake of a theoretical consistency, and absolutely should not be used as an excuse to sanitise or escape text while it is encoded as ISO-2022-JP (nor any other stateful encoding). Accomodating established and plausible variation in encoders means that some masking sequences may still be possible; furthermore, the jis_encoding decoder does not penalise zero-effect Shift In characters. All sanitisation or escaping of data received in ISO-2022-JP must be carried out over the Unicode stream. Furthermore, using the ISO-2022-JP family or other stateful encodings inside an ASCII-delimited structure such as JSON should be avoided if possible; even if it cannot be avoided, one must not substitute an untrusted ISO-2022-JP* or JIS_Encoding sequence straight into an ASCII structure without verifying that it in fact returns to Shift In state (if applicable), with ASCII designated, at the end of the sequence, with no trailing single-shifts.

A summary of the differing behaviours is listed in the table below:

iso-2022-jp No Good No Good Only at start/end or next to 5C/7E or C0 control code; generated before 5C/7E Only at start/end or next to 5C/7E or C0 control code; generated before 5C/7E Only at start No Good Okay
jis_encoding Okay No Good Only at start/end or next to 5C/7E or C0 control code; generated before 5C/7E Only at start/end or next to 5C/7E or C0 control code; generated after 5C/7E Only at start No Good Okay
Python Okay Okay Okay; generated before 5C/7E Okay; generated after 5C/7E Okay Okay Okay
WHATWG No Good No Good Okay; generated before 5C/7E Okay; generated before 5C/7E Okay Okay Okay
End State 1 Okay Okay Okay Okay Okay Okay Okay
End State 2 Okay No Good Only at end or before 5C/7E Only at end or before 5C/7E Okay Only before 5C/7E Only before 5C/7E

Package contents