codecs.infrastructure

← codecs

Underpinning infrastructure for the codecs module.

Classes

class AsciiIncrementalDecoder(IncrementalDecoder)

Decoder for ISO/IEC 4873-DV, and base class for simple sensu lato extended ASCII decoders. Decoders for more complex cases, such as ISO-2022-JP, do not inherit from this class.

ISO/IEC 4873-DV is, as of the current (third) edition of ISO/IEC 4873, the same as what people usually mean when they say "ASCII" (i.e. an eighth bit exists but is never used, and backspace composition is not a thing which exists for encoding characters).

AsciiIncrementalDecoder.decode(data_in,final)

Implements IncrementalDecoder.decode

class AsciiIncrementalEncoder(IncrementalEncoder)

Encoder for ISO/IEC 4873-DV, and base class for simple sensu lato extended ASCII encoders. Encoders for more complex cases, such as ISO-2022-JP, do not inherit from this class.

ISO/IEC 4873-DV is, as of the current (third) edition of ISO/IEC 4873, the same as what people usually mean when they say "ASCII" (i.e. an eighth bit exists but is never used, and backspace composition is not a thing which exists for encoding characters).

let x = AsciiIncrementalEncoder(errors)

AsciiIncrementalEncoder.__init__(errors)

AsciiIncrementalEncoder.encode(string_in,final)

Implements IncrementalEncoder.encode

AsciiIncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

AsciiIncrementalEncoder.reset()

Implements IncrementalEncoder.reset

AsciiIncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

class BaseEbcdicIncrementalDecoder(IncrementalDecoder)

Base class for EBCDIC decoders.

On its own, it is only capable of decoding U+3000 (from x'0E', x'40', x'40', x'0F'); hence, it should not, generally speaking, be used directly.

BaseEbcdicIncrementalDecoder.decode(data_in,final)

Implements IncrementalDecoder.decode

BaseEbcdicIncrementalDecoder.getstate()

Implements IncrementalDecoder.getstate

BaseEbcdicIncrementalDecoder.reset()

Implements IncrementalDecoder.reset

BaseEbcdicIncrementalDecoder.setstate(state)

Implements IncrementalDecoder.setstate

class BaseEbcdicIncrementalEncoder(IncrementalEncoder)

Base class for EBCDIC encoders.

On its own, it is only capable of encoding U+3000 (as x'0E', x'40', x'40', x'0F'); hence, it should not, generally speaking, be used directly.

BaseEbcdicIncrementalEncoder.encode(string,final)

Implements IncrementalEncoder.encode

BaseEbcdicIncrementalEncoder.getstate()

Implements IncrementalEncoder.getstate

BaseEbcdicIncrementalEncoder.reset()

Implements IncrementalEncoder.reset

BaseEbcdicIncrementalEncoder.setstate(state)

Implements IncrementalEncoder.setstate

class ByteCatenator(object)

Helper class for maintaining a stream to which bytes objects will be repeatedly catenated in place.

let x = ByteCatenator()

ByteCatenator.__init__()

ByteCatenator.add(data)

ByteCatenator.getvalue()

class IncrementalDecoder(object)

Incremental decoder, allowing more Unicode data to be generated as more encoded data is obtained. Note that the return values from decode are not guaranteed to encompass all data which has been passed in, until it is called with final=True.

This is the base class and should not be instantiated directly.

let x = IncrementalDecoder(errors)

IncrementalDecoder.__init__(errors)

IncrementalDecoder._handle_truncation(out,unused,final,data,offset,leader)

Helper function used by subclasses to handle any pending data when returning from decode.

IncrementalDecoder.decode(data_in,final)

Passes the given bytes in to the encoder, and returns a Unicode string. When final=False, the return value might not represent the entire input (some of which may become represented at the start of the value returned by the next call). When final=True, all of the input will be represented, and an error will be generated if it is truncated.

IncrementalDecoder.getstate()

Returns an arbitrary object encapsulating decoder state.

IncrementalDecoder.reset()

Reset decoder to initial state, without outputting, discarding any pending data.

IncrementalDecoder.setstate(state)

Sets decoder state to one previously returned by getstate().

class IncrementalEncoder(object)

Incremental encoder, allowing more encoded data to be generated as more Unicode data is obtained. Note that the return values from encode are not guaranteed to encompass all data which has been passed in, until it is called with final=True.

This is the base class and should not be instantiated directly.

let x = IncrementalEncoder(errors)

IncrementalEncoder.__init__(errors)

IncrementalEncoder.encode(string,final)

Passes the given string in to the encoder, and returns a sequence of bytes. When final=False, the return value might not represent the entire input (some of which may become represented at the start of the value returned by the next call). When final=True, all of the input will be represented, and any final state change sequence required by the encoding will be outputted.

IncrementalEncoder.getstate()

Returns an arbitrary object encapsulating encoder state.

IncrementalEncoder.reset()

Reset encoder to initial state, without outputting, discarding any pending data.

IncrementalEncoder.setstate(state)

Sets encoder state to one previously returned by getstate().

class KurokoCodecInfo(object)

Descriptor for the registered encoder and decoder for a given label. Has five members:

• name: the label covered by this descriptor.
• encode: encode a complete Unicode sequence.
• decode: decode a complete byte sequence.
• incrementalencoder: IncrementalEncoder subclass.
• incrementaldecoder: IncrementalDecoder subclass.

let x = KurokoCodecInfo(label,encoder,decoder)

KurokoCodecInfo.__init__(label,encoder,decoder)

KurokoCodecInfo.decode(data,errors)

Decode a complete byte sequence to a complete Unicode stream. Semantic of name passed to errors= is as documented for lookup_error().

KurokoCodecInfo.encode(string,errors)

Encode a complete Unicode sequence to a complete byte string. Semantic of name passed to errors= is as documented for lookup_error().

class StringCatenator(object)

Helper class for maintaining a stream to which str objects will be repeatedly catenated in place.

let x = StringCatenator()

StringCatenator.__init__()

StringCatenator.add(string)

StringCatenator.getvalue()

class UndefinedIncrementalDecoder(IncrementalDecoder)

Decoder which errors out on all input. For use on input for which decoding should not be attempted. Error handler is honoured, and called once per non-empty decode method call.

UndefinedIncrementalDecoder.decode(data,final)

class UndefinedIncrementalEncoder(IncrementalEncoder)

Encoder which errors out on all input. For use on input for which encoding should not be attempted. Error handler is ignored.

let x = UndefinedIncrementalEncoder(errors)

UndefinedIncrementalEncoder.__init__(errors)

UndefinedIncrementalEncoder.encode(string,final)

class decodesto7bit(object)

Decoding map for a 7-bit set, wrapping an decoding map for an 8-bit EUC or EUC-superset encoding.

let x = decodesto7bit(base)

decodesto7bit.__init__(base)

key in decodesto7bit

decodesto7bit.__contains__(key)

decodesto7bit[key]

decodesto7bit.__getitem__(key)

for x in decodesto7bit:

decodesto7bit.__iter__()

decodesto7bit.keys()

class encodesto7bit(object)

Encoding map for a 7-bit set, wrapping an encoding map for an 8-bit EUC or EUC-superset encoding.

let x = encodesto7bit(base)

encodesto7bit.__init__(base)

key in encodesto7bit

encodesto7bit.__contains__(key)

encodesto7bit[key]

encodesto7bit.__getitem__(key)

for x in encodesto7bit:

encodesto7bit.__iter__()

encodesto7bit.keys()

Functions

backslashreplace_errors(exc)

Handler for backslashreplace errors: replace unencodable character with Python/Kuroko style escape sequence. For Basic Multilingual Plane characters, this also matches JavaScript; beyond that, they differ.

decode(data,label,web,errors)

Decode a complete byte sequence in the given encoding to a complete Unicode stream. Semantic of the web= argument is the same as with lookup(). Semantic of name passed to errors= is as documented for lookup_error().

Can be simply accessed as codecs.decode.

encode(string,label,web,errors)

Encode a complete Unicode sequence to a complete byte string in the given encoding. Semantic of the web= argument is the same as with lookup(). Semantic of name passed to errors= is as documented for lookup_error().

Can be simply accessed as codecs.encode.

ignore_errors(exc)

Handler for ignore errors: skip invalid sequences.

lazy_property(method)

Like property(…), but memoises the value returned. The return value is assumed to be constant at the class level, i.e. the same for all instances.

lookup(label,web)

Obtain a KurokoCodecInfo for a given label. If web=False (the default), will always succeed, but the resulting KurokoCodecInfo might be unable to encode and/or unable to decode if the label is not recognised in that direction. If web=True, will raise KeyError if the label is not a WHATWG-permitted label, and will map certain labels to undefined per the WHATWG spec.

Can be simply accessed as codecs.lookup.

lookup_error(name)

Look up an error handler function registered with a certain name. By default, the following are registered. It is important to note that nothing obligates a codec to actually use the error handler if it is not deemed possible or appropriate, and so specifying a non-strict error handler will not guarantee an exception will not be raised, especially when working with a codec which is not a "normal" text encoding (e.g. undefined or inverse-base64).

• strict: raise an exception.
• ignore: skip invalid substrings. Not always recommended: can facilitate masked injection.
• replace: insert a replacement character (decoding) or question mark (encoding).
• warnreplace: like replace but prints a message to stderr; good for debugging.
• backslashreplace: replace with Python/Kuroko style Unicode escapes. Note that this only matches JavaScript escape syntax for Basic Multilingual Plane characters. Encoding only.
• xmlcharrefreplace: replace with HTML/XML numerical entities. Note that this will, per WHATWG, never generate entities for Shift Out, Shift In and Escape (i.e. when encoding to a stateful encoding which uses them, e.g. ISO-2022-JP), instead generating an entity for the replacement character. Encoding only.

register_error(name,handler)

Reister a new error handler. The handler should be a function taking a UnicodeError and either raising an exception or returning a tuple of (substitute, resume_index). The substitute should be bytes (usually expected to be in ASCII) for a UnicodeEncodeError, str otherwise.

register_kuroko_codec(labels,incremental_encoder_class,incremental_decoder_class)

Register a given IncrementalEncoder subclass and a given IncrementalDecoder subclass with a given list of labels. Usually, this is expected to include the encoding name, along with a list labels for aliases and/or subsets of the encoding. Either coder class may be None, if the encoder/decoder labels are being registered asymmetrically.

replace_errors(exc)

Handler for replace errors: insert replacement character (if decoding) or question mark (if encoding).

strict_errors(exc)

Handler for strict errors: raise the exception.

warnreplace_errors(exc)

Handler for warnreplace errors: insert replacement character (if decoding) or question mark (if encoding) and print a warning to stderr.

xmlcharrefreplace_errors(exc)

Handler for xmlcharrefreplace errors: replace unencodable character with XML numeric entity for the character unless it is Shift Out, Shift In or Escape, in which case insert the XML numeric entity for the replacement character (as stipulated by WHATWG for ISO-2022-JP).

Exceptions

class UnicodeDecodeError(UnicodeError)

UnicodeError subclass raised when an error is encountered in the process of decoding.

class UnicodeEncodeError(UnicodeError)

UnicodeError subclass raised when an error is encountered in the process of encoding.

class UnicodeError(ValueError)

Exception raised when an error is encountered or detected in the process of encoding or decoding. May instead be passed to a handler when not in strict mode. Contains machine-readable information about the error encountered, allowing approaches to respond to it.

let x = UnicodeError(encoding,object,start,end,reason)

UnicodeError.__init__(encoding,object,start,end,reason)