codecs.dbextra

← codecs

This module includes some additional variable-width or wide encodings not specified by WHATWG.

As such, none of the codecs in this module should be used in HTML.

Classes

class AsciiJohabIncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for the PC Johab encoding (code page 1361).

property AsciiJohabIncrementalDecoder.decoding_map

class AsciiJohabIncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for the PC Johab encoding (code page 1361).

property AsciiJohabIncrementalEncoder.encoding_map

class Big5NonEtenKanaIncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for Big5 with non-ETEN layout of kana, Cyrillic, list markers.

The other ETEN extension section (the one retained by Microsoft's version) is still included.

Although this is the kana/Cyrillic/list marker layout included in the UTC's BIG5.TXT, it is the less common of the two (most extension schemes for Big5 use the ETEN layout), and has several problems (katakana lacks the vowel extender, and Cyrillic lacks several capitals) which the ETEN layout does not have. However, this codec corresponds roughly to Python's big5, and more closely to its (built-in, as opposed to if/when Python aliases it to mbcs) cp950.

property Big5NonEtenKanaIncrementalDecoder.decoding_map

class Big5NonEtenKanaIncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for Big5 with non-ETEN layout of kana, Cyrillic, list markers.

The other ETEN extension section (the one retained by Microsoft's version) is still included.

Although this is the kana/Cyrillic/list marker layout included in the UTC's BIG5.TXT, it is the less common of the two (most extension schemes for Big5 use the ETEN layout), and has several problems (katakana lacks the vowel extender, and Cyrillic lacks several capitals) which the ETEN layout does not have. However, this codec corresponds roughly to Python's big5, and more closely to its (built-in, as opposed to if/when Python aliases it to mbcs) cp950.

property Big5NonEtenKanaIncrementalEncoder.encoding_map

class Cesu8IncrementalDecoder(Utf8IncrementalDecoder)

IncrementalDecoder implementation for CESU-8, a deprecated UTF-8-like encoding still used by some systems, such as TCL, and still mis-called "utf8" in some places for legacy reasons.

Cesu8IncrementalDecoder._error_handler(error)

class Cesu8IncrementalEncoder(IncrementalEncoder)

IncrementalEncoder implementation for CESU-8, a deprecated UTF-8-like encoding still used by some systems, such as TCL, and still mis-called "utf8" in some places for legacy reasons.

Cesu8IncrementalEncoder.encode(string,final)

Implements IncrementalEncoder.encode

Cesu8IncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

Cesu8IncrementalEncoder.reset()

Implements IncrementalEncoder.reset

Cesu8IncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

class EbcdicJohabIncrementalDecoder(BaseEbcdicIncrementalDecoder)

IncrementalDecoder implementation for code page 1364, a stateful EBCDIC variant of Johab.

property EbcdicJohabIncrementalDecoder.dbcshost_decode

property EbcdicJohabIncrementalDecoder.sbcs_decode

class EbcdicJohabIncrementalEncoder(BaseEbcdicIncrementalEncoder)

IncrementalEncoder implementation for code page 1364, a stateful EBCDIC variant of Johab.

property EbcdicJohabIncrementalEncoder.dbcshost_encode

property EbcdicJohabIncrementalEncoder.sbcs_encode

class EucJis2004IncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for the JIS X 0213 version of EUC-JP.

property EucJis2004IncrementalDecoder.decoding_map

class EucJis2004IncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for the JIS X 0213 version of EUC-JP.

property EucJis2004IncrementalEncoder.encoding_map

class EucJpFullIncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for EUC-JP, including JIS X 0212.

property EucJpFullIncrementalEncoder.encoding_map

class HzIncrementalDecoder(IncrementalDecoder)

IncrementalDecoder implementation for HZ-GB-2312 (Usenet simplified Chinese).

This is an old scheme for embedding GB 2312 data into a pure ASCII stream.

HzIncrementalDecoder.decode(data_in,final)

Implements IncrementalDecoder.decode

HzIncrementalDecoder.getstate()

Implements IncrementalDecoder.getstate

HzIncrementalDecoder.reset()

Implements IncrementalDecoder.reset

HzIncrementalDecoder.setstate(state)

Implements IncrementalDecoder.setstate

class HzIncrementalEncoder(IncrementalEncoder)

IncrementalEncoder implementation for HZ-GB-2312 (Usenet simplified Chinese).

This is an old scheme for embedding GB 2312 data into a pure ASCII stream.

HzIncrementalEncoder.encode(string,final)

Implements IncrementalEncoder.encode

HzIncrementalEncoder.ensure_state_number(state,out)

HzIncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

HzIncrementalEncoder.reset()

Implements IncrementalEncoder.reset

HzIncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

class Iso2022CnIncrementalDecoder(Iso2022NonJpIncrementalDecoder)

IncrementalDecoder implementation for ISO-2022-CN (7-bit stateful Chinese).

ISO-2022-CN-Ext is not included (it requires a much larger set of tables and is very rare).

property Iso2022CnIncrementalDecoder.decodes

class Iso2022CnIncrementalEncoder(Iso2022NonJpIncrementalEncoder)

IncrementalEncoder implementation for ISO-2022-CN (7-bit stateful Chinese).

ISO-2022-CN-Ext is not included (it requires a much larger set of tables and is very rare).

property Iso2022CnIncrementalEncoder.encodes

class Iso2022Jp1IncrementalEncoder(Iso2022JpIncrementalEncoder)

IncrementalEncoder implementation for 7-bit stateful Japanese with JIS X 0212.

This differs from the ISO-2022-JP encoder in that it will encode to JIS X 0212, and does so whenever possible (i.e. it will favour it over any web extensions to JIS X 0208).

property Iso2022Jp1IncrementalEncoder.encodes_dbcs

property Iso2022Jp1IncrementalEncoder.encodes_sbcs

class Iso2022Jp2004IncrementalEncoder(Iso2022JpIncrementalEncoder)

IncrementalEncoder implementation for 7-bit stateful Japanese with JIS X 0213-2004.

property Iso2022Jp2004IncrementalEncoder.encodes_dbcs

property Iso2022Jp2004IncrementalEncoder.encodes_sbcs

class Iso2022Jp2IncrementalEncoder(Iso2022JpIncrementalEncoder)

IncrementalEncoder implementation for 7-bit stateful Japanese with multilingual extensions.

property Iso2022Jp2IncrementalEncoder.encode_supershift_greek

property Iso2022Jp2IncrementalEncoder.encode_supershift_latin

property Iso2022Jp2IncrementalEncoder.encodes_dbcs

property Iso2022Jp2IncrementalEncoder.encodes_sbcs

class Iso2022Jp3IncrementalEncoder(Iso2022JpIncrementalEncoder)

IncrementalEncoder implementation for 7-bit stateful Japanese with JIS X 0213-2000.

property Iso2022Jp3IncrementalEncoder.encodes_dbcs

property Iso2022Jp3IncrementalEncoder.encodes_sbcs

class Iso2022JpExtIncrementalEncoder(Iso2022JpIncrementalEncoder)

IncrementalEncoder implementation for 7-bit stateful Japanese.

This differs from the ISO-2022-JP-1 encoder in that it preserves katakana width.

property Iso2022JpExtIncrementalEncoder.encodes_dbcs

property Iso2022JpExtIncrementalEncoder.encodes_sbcs

class Iso2022KrIncrementalDecoder(Iso2022NonJpIncrementalDecoder)

IncrementalDecoder implementation for ISO-2022-KR (7-bit stateful Korean, South).

property Iso2022KrIncrementalDecoder.decodes

class Iso2022KrIncrementalEncoder(Iso2022NonJpIncrementalEncoder)

IncrementalEncoder implementation for ISO-2022-KR (7-bit stateful Korean, South).

Iso2022KrIncrementalEncoder.run_prelude(out)

property Iso2022KrIncrementalEncoder.encodes

class Iso2022NonJpIncrementalDecoder(IncrementalDecoder)

IncrementalDecoder subclass, base class for ISO-2022-KR and ISO-2022-CN. Not used directly.

Iso2022NonJpIncrementalDecoder.decode(data_in,final)

Implements IncrementalDecoder.decode

Iso2022NonJpIncrementalDecoder.getstate()

Implements IncrementalDecoder.getstate

Iso2022NonJpIncrementalDecoder.reset()

Implements IncrementalDecoder.reset

Iso2022NonJpIncrementalDecoder.setstate(state)

Implements IncrementalDecoder.setstate

class Iso2022NonJpIncrementalEncoder(IncrementalEncoder)

IncrementalEncoder subclass, base class for ISO-2022-KR and ISO-2022-CN. Not used directly.

Iso2022NonJpIncrementalEncoder.encode(string,final)

Implements IncrementalEncoder.encode

Iso2022NonJpIncrementalEncoder.ensure_shift_designation(state,out)

Iso2022NonJpIncrementalEncoder.ensure_shift_state(state,out)

Iso2022NonJpIncrementalEncoder.ensure_super3_designation(state,out)

Iso2022NonJpIncrementalEncoder.ensure_super_designation(state,out)

Iso2022NonJpIncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

Iso2022NonJpIncrementalEncoder.reset()

Implements IncrementalEncoder.reset

Iso2022NonJpIncrementalEncoder.run_prelude(out)

Iso2022NonJpIncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

class JapaneseAutodetectIncrementalDecoder(IncrementalDecoder)

IncrementalDecoder implementation for the automatic "Japanese" character encoding option.

This will attempt to interpret the stream as the web versions of ISO-2022-JP, Shift_JIS and EUC-JP, as well as UTF-8, at once, and start returning the data once it has narrowed it down to one. If it fails to narrow it down conclusively, it will wait until the final call before making an educated guess. If it doesn't seem to be any of them, it will raise ValueError.

let x = JapaneseAutodetectIncrementalDecoder(errors)

JapaneseAutodetectIncrementalDecoder.__init__(errors)

JapaneseAutodetectIncrementalDecoder.decode(data,final)

Implements IncrementalDecoder.decode

JapaneseAutodetectIncrementalDecoder.getstate()

Implements IncrementalDecoder.getstate

JapaneseAutodetectIncrementalDecoder.reset()

Implements IncrementalDecoder.reset

JapaneseAutodetectIncrementalDecoder.setstate(state)

Implements IncrementalDecoder.setstate

class JisEncodingIncrementalDecoder(Iso2022JpIncrementalDecoder)

IncrementalDecoder implementation for 7-bit stateful Japanese.

This is differs from the ISO-2022-JP decoder in that it will:

• Decode 1978 JIS with a separate table, including 1978 JIS, NEC extensions and IBM backports.
• Accept and decode extensions from ISO-2022-JP-2 (and -1), ISO-2022-JP-3 and ISO-2022-JP-2004.
• Not generate an error for immediately concatenated JIS-Kanji→ASCII→JIS-Kanji designations.
• Accept katakana via Shift Out / Shift In.

This is used as the decoder for all other ISO-2022-JP variants besides plain ISO-2022-JP.

property JisEncodingIncrementalDecoder.decode_shiftout

property JisEncodingIncrementalDecoder.decode_supershift_greek

property JisEncodingIncrementalDecoder.decode_supershift_latin

property JisEncodingIncrementalDecoder.decodes_dbcs

property JisEncodingIncrementalDecoder.decodes_sbcs

class JisEncodingIncrementalEncoder(Iso2022JpIncrementalEncoder)

IncrementalEncoder implementation for 7-bit stateful Japanese with all features.

This differs from the ISO-2022-JP encoder in that it will:

• Encode forms present in 1978 JIS but simplified by (and absent in) 1983 JIS to 1978 JIS.
• For characters not present in either table, try JIS X 0212, 2000 JIS and 2004 JIS in that order.
• For characters not present in any JIS set, try GB 2312 and Wansung.
• Preserve width of katakana.

property JisEncodingIncrementalEncoder.encode_supershift_greek

property JisEncodingIncrementalEncoder.encode_supershift_latin

property JisEncodingIncrementalEncoder.encodes_dbcs

property JisEncodingIncrementalEncoder.encodes_sbcs

class ShiftJis2004IncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for the JIS X 0213 version of Shift_JIS.

property ShiftJis2004IncrementalDecoder.decoding_map

class ShiftJis2004IncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for the JIS X 0213 version of Shift_JIS.

property ShiftJis2004IncrementalEncoder.encoding_map

class Utf32BeIncrementalDecoder(Utf32IncrementalDecoder)

IncrementalDecoder implementation for UTF-32, big endian, without a byte order mark.

class Utf32BeIncrementalEncoder(Utf32IncrementalEncoder)

IncrementalEncoder implementation for UTF-32, big endian, without a byte order mark.

class Utf32IncrementalDecoder(IncrementalDecoder)

IncrementalDecoder implementation for UTF-32, detected byte order, removing any byte order mark.

Utf32IncrementalDecoder.decode(data_in,final)

Implements IncrementalDecoder.decode

Utf32IncrementalDecoder.getstate()

Implements IncrementalDecoder.getstate

Utf32IncrementalDecoder.reset()

Implements IncrementalDecoder.reset

Utf32IncrementalDecoder.setstate(state)

Implements IncrementalDecoder.setstate

class Utf32IncrementalEncoder(IncrementalEncoder)

IncrementalEncoder implementation for UTF-32 with byte order mark.

Utf32IncrementalEncoder.encode(string,final)

Implements IncrementalEncoder.encode

Utf32IncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

Utf32IncrementalEncoder.push_word(word,out)

Utf32IncrementalEncoder.reset()

Implements IncrementalEncoder.reset

Utf32IncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

property Utf32IncrementalEncoder.encoding_map

class Utf32LeIncrementalDecoder(Utf32IncrementalDecoder)

IncrementalDecoder implementation for UTF-32, little endian, without a byte order mark.

class Utf32LeIncrementalEncoder(Utf32IncrementalEncoder)

IncrementalEncoder implementation for UTF-32, little endian, without a byte order mark.

class Utf7IncrementalDecoder(IncrementalDecoder)

IncrementalDecoder implementation for UTF-7, a largely obsolete (and forbidden in HTML5) scheme for mixing ASCII with Base64'd UTF-16BE in e-mail.

let x = Utf7IncrementalDecoder(errors)

Utf7IncrementalDecoder.__init__(errors)

Utf7IncrementalDecoder.decode(data_in,final)

Implements IncrementalDecoder.decode

Utf7IncrementalDecoder.getstate()

Implements IncrementalDecoder.getstate

Utf7IncrementalDecoder.reset()

Implements IncrementalDecoder.reset

Utf7IncrementalDecoder.setstate(state)

Implements IncrementalDecoder.setstate

class Utf7IncrementalEncoder(IncrementalEncoder)

IncrementalEncoder implementation for UTF-7, a largely obsolete (and forbidden in HTML5) scheme for mixing ASCII with Base64'd UTF-16BE in e-mail.

let x = Utf7IncrementalEncoder(errors)

Utf7IncrementalEncoder.__init__(errors)

Utf7IncrementalEncoder.encode(data,final)

Implements IncrementalEncoder.encode

Utf7IncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

Utf7IncrementalEncoder.reset()

Implements IncrementalEncoder.reset

Utf7IncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

class XMacChineseSimpIncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for EUC-CN, Apple version (hence slightly reduced lead byte range).

Mappings to more-recently added characters are used for the vertical forms, rather than Apple transcoding hints (or GB18030 private use codes).

property XMacChineseSimpIncrementalDecoder.decoding_map

class XMacChineseSimpIncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for EUC-CN, Apple version (hence slightly reduced lead byte range).

Mappings to more-recently added characters are used for the vertical forms, rather than Apple transcoding hints (or GB18030 private use codes).

property XMacChineseSimpIncrementalEncoder.encoding_map

class XMacChineseTradIncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for Big5 with Apple's additions and reduced lead byte range.

The Unicode mappings are partly changed to be closer to Apple's (as opposed to Microsoft's) correspondences; however, Microsoft's are retained where following Apple's would have required PUA transcoding hints to round-trip.

property XMacChineseTradIncrementalDecoder.decoding_map

class XMacChineseTradIncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for Big5 with Apple's additions and reduced lead byte range.

The Unicode mappings are partly changed to be closer to Apple's (as opposed to Microsoft's) correspondences; however, Microsoft's are retained where following Apple's would have required PUA transcoding hints to round-trip.

property XMacChineseTradIncrementalEncoder.encoding_map

class XMacKoreanIncrementalDecoder(AsciiIncrementalDecoder)

IncrementalDecoder implementation for the HangulTalk (MacKorean) encoding.

HangulTalk is notorious for frequently not corresponding one-to-one to Unicode; the mappings used here are somewhat updated and improved compared to all versions of Apple's mappings and especially the Adobe CID mappings. However, bear in mind that content will not necessarily be decoded to the same Unicode sequences as by other implementations. In particular, decoding to the Apple's Corporate Private Use Area has been avoided for the most part, even where this results in poorly matched and/or convergent decoded forms, since preserving legibility has been afforded greater priority than round tripping.

property XMacKoreanIncrementalDecoder.decoding_map

class XMacKoreanIncrementalEncoder(AsciiIncrementalEncoder)

IncrementalEncoder implementation for the HangulTalk (MacKorean) encoding.

HangulTalk is notorious for frequently not corresponding one-to-one to Unicode. In places, multiple Unicode representations will be accepted for a given HangulTalk representation.

property XMacKoreanIncrementalEncoder.encoding_map

Other Members

let data_7bit = _DBExtraData7Bit

let data_8bit = _DBExtraData8Bit

let more_dbdata = _MoreDBData