:mod:`codecs` --- Codec registry and base classes

=================================================

.. module:: codecs

   :synopsis: Encode and decode data and streams.

.. moduleauthor:: Marc-Andre Lemburg <mal@lemburg.com>

.. sectionauthor:: Marc-Andre Lemburg <mal@lemburg.com>

.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de>

.. index::

   single: Unicode

   single: Codecs

   pair: Codecs; encode

   pair: Codecs; decode

   single: streams

   pair: stackable; streams

This module defines base classes for standard Python codecs (encoders and

decoders) and provides access to the internal Python codec registry which

manages the codec and error handling lookup process.

It defines the following functions:

.. function:: register(search_function)

   Register a codec search function. Search functions are expected to take one

   argument, the encoding name in all lower case letters, and return a

   :class:`CodecInfo` object having the following attributes:

   * ``name`` The name of the encoding;

   * ``encode`` The stateless encoding function;

   * ``decode`` The stateless decoding function;

   * ``incrementalencoder`` An incremental encoder class or factory function;

   * ``incrementaldecoder`` An incremental decoder class or factory function;

   * ``streamwriter`` A stream writer class or factory function;

   * ``streamreader`` A stream reader class or factory function.

   The various functions or classes take the following arguments:

   *encode* and *decode*: These must be functions or methods which have the same

   interface as the :meth:`encode`/:meth:`decode` methods of Codec instances (see

   Codec Interface). The functions/methods are expected to work in a stateless

   mode.

   *incrementalencoder* and *incrementaldecoder*: These have to be factory

   functions providing the following interface:

   ``factory(errors='strict')``

   The factory functions must return objects providing the interfaces defined by

   the base classes :class:`IncrementalEncoder` and :class:`IncrementalDecoder`,

   respectively. Incremental codecs can maintain state.

   *streamreader* and *streamwriter*: These have to be factory functions providing

   the following interface:

   ``factory(stream, errors='strict')``

   The factory functions must return objects providing the interfaces defined by

   the base classes :class:`StreamWriter` and :class:`StreamReader`, respectively.

   Stream codecs can maintain state.

   Possible values for errors are ``'strict'`` (raise an exception in case of an

   encoding error), ``'replace'`` (replace malformed data with a suitable

   replacement marker, such as ``'?'``), ``'ignore'`` (ignore malformed data and

   continue without further notice), ``'xmlcharrefreplace'`` (replace with the

   appropriate XML character reference (for encoding only)) and

   ``'backslashreplace'`` (replace with backslashed escape sequences (for encoding

   only)) as well as any other error handling name defined via

   :func:`register_error`.

   In case a search function cannot find a given encoding, it should return

   ``None``.

.. function:: lookup(encoding)

   Looks up the codec info in the Python codec registry and returns a

   :class:`CodecInfo` object as defined above.

   Encodings are first looked up in the registry's cache. If not found, the list of

   registered search functions is scanned. If no :class:`CodecInfo` object is

   found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object

   is stored in the cache and returned to the caller.

To simplify access to the various codecs, the module provides these additional

functions which use :func:`lookup` for the codec lookup:

.. function:: getencoder(encoding)

   Look up the codec for the given encoding and return its encoder function.

   Raises a :exc:`LookupError` in case the encoding cannot be found.

.. function:: getdecoder(encoding)

   Look up the codec for the given encoding and return its decoder function.

   Raises a :exc:`LookupError` in case the encoding cannot be found.

.. function:: getincrementalencoder(encoding)

   Look up the codec for the given encoding and return its incremental encoder

   class or factory function.

   Raises a :exc:`LookupError` in case the encoding cannot be found or the codec

   doesn't support an incremental encoder.

   .. versionadded:: 2.5

.. function:: getincrementaldecoder(encoding)

   Look up the codec for the given encoding and return its incremental decoder

   class or factory function.

   Raises a :exc:`LookupError` in case the encoding cannot be found or the codec

   doesn't support an incremental decoder.

   .. versionadded:: 2.5

.. function:: getreader(encoding)

   Look up the codec for the given encoding and return its StreamReader class or

   factory function.

   Raises a :exc:`LookupError` in case the encoding cannot be found.

.. function:: getwriter(encoding)

   Look up the codec for the given encoding and return its StreamWriter class or

   factory function.

   Raises a :exc:`LookupError` in case the encoding cannot be found.

.. function:: register_error(name, error_handler)

   Register the error handling function *error_handler* under the name *name*.

   *error_handler* will be called during encoding and decoding in case of an error,

   when *name* is specified as the errors parameter.

   For encoding *error_handler* will be called with a :exc:`UnicodeEncodeError`

   instance, which contains information about the location of the error. The error

   handler must either raise this or a different exception or return a tuple with a

   replacement for the unencodable part of the input and a position where encoding

   should continue. The encoder will encode the replacement and continue encoding

   the original input at the specified position. Negative position values will be

   treated as being relative to the end of the input string. If the resulting

   position is out of bound an :exc:`IndexError` will be raised.

   Decoding and translating works similar, except :exc:`UnicodeDecodeError` or

   :exc:`UnicodeTranslateError` will be passed to the handler and that the

   replacement from the error handler will be put into the output directly.

.. function:: lookup_error(name)

   Return the error handler previously registered under the name *name*.

   Raises a :exc:`LookupError` in case the handler cannot be found.

.. function:: strict_errors(exception)

   Implements the ``strict`` error handling.

.. function:: replace_errors(exception)

   Implements the ``replace`` error handling.

.. function:: ignore_errors(exception)

   Implements the ``ignore`` error handling.

.. function:: xmlcharrefreplace_errors(exception)

   Implements the ``xmlcharrefreplace`` error handling.

.. function:: backslashreplace_errors(exception)

   Implements the ``backslashreplace`` error handling.

To simplify working with encoded files or stream, the module also defines these

utility functions:

.. function:: open(filename, mode[, encoding[, errors[, buffering]]])

   Open an encoded file using the given *mode* and return a wrapped version

   providing transparent encoding/decoding.  The default file mode is ``'r'``

   meaning to open the file in read mode.

   .. note::

      The wrapped version will only accept the object format defined by the codecs,

      i.e. Unicode objects for most built-in codecs.  Output is also codec-dependent

      and will usually be Unicode as well.

   .. note::

      Files are always opened in binary mode, even if no binary mode was

      specified.  This is done to avoid data loss due to encodings using 8-bit

      values.  This means that no automatic conversion of ``'\n'`` is done

      on reading and writing.

   *encoding* specifies the encoding which is to be used for the file.

   *errors* may be given to define the error handling. It defaults to ``'strict'``

   which causes a :exc:`ValueError` to be raised in case an encoding error occurs.

   *buffering* has the same meaning as for the built-in :func:`open` function.  It

   defaults to line buffered.

.. function:: EncodedFile(file, input[, output[, errors]])

   Return a wrapped version of file which provides transparent encoding

   translation.

   Strings written to the wrapped file are interpreted according to the given

   *input* encoding and then written to the original file as strings using the

   *output* encoding. The intermediate encoding will usually be Unicode but depends

   on the specified codecs.

   If *output* is not given, it defaults to *input*.

   *errors* may be given to define the error handling. It defaults to ``'strict'``,

   which causes :exc:`ValueError` to be raised in case an encoding error occurs.

.. function:: iterencode(iterable, encoding[, errors])

   Uses an incremental encoder to iteratively encode the input provided by

   *iterable*. This function is a :term:`generator`.  *errors* (as well as any

   other keyword argument) is passed through to the incremental encoder.

   .. versionadded:: 2.5

.. function:: iterdecode(iterable, encoding[, errors])

   Uses an incremental decoder to iteratively decode the input provided by

   *iterable*. This function is a :term:`generator`.  *errors* (as well as any

   other keyword argument) is passed through to the incremental decoder.

   .. versionadded:: 2.5

The module also provides the following constants which are useful for reading

and writing to platform dependent files:

.. data:: BOM

          BOM_BE

          BOM_LE

          BOM_UTF8

          BOM_UTF16

          BOM_UTF16_BE

          BOM_UTF16_LE

          BOM_UTF32

          BOM_UTF32_BE

          BOM_UTF32_LE

   These constants define various encodings of the Unicode byte order mark (BOM)

   used in UTF-16 and UTF-32 data streams to indicate the byte order used in the

   stream or file and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either

   :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's

   native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`,

   :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for

   :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32

   encodings.

.. _codec-base-classes:

Codec Base Classes

------------------

The :mod:`codecs` module defines a set of base classes which define the

interface and can also be used to easily write your own codecs for use in

Python.

Each codec has to define four interfaces to make it usable as codec in Python:

stateless encoder, stateless decoder, stream reader and stream writer. The

stream reader and writers typically reuse the stateless encoder/decoder to

implement the file protocols.

The :class:`Codec` class defines the interface for stateless encoders/decoders.

To simplify and standardize error handling, the :meth:`encode` and

:meth:`decode` methods may implement different error handling schemes by

providing the *errors* string argument.  The following string values are defined

and implemented by all standard Python codecs:

+-------------------------+-----------------------------------------------+

| Value                   | Meaning                                       |

+=========================+===============================================+

| ``'strict'``            | Raise :exc:`UnicodeError` (or a subclass);    |

|                         | this is the default.                          |

+-------------------------+-----------------------------------------------+

| ``'ignore'``            | Ignore the character and continue with the    |

|                         | next.                                         |

+-------------------------+-----------------------------------------------+

| ``'replace'``           | Replace with a suitable replacement           |

|                         | character; Python will use the official       |

|                         | U+FFFD REPLACEMENT CHARACTER for the built-in |

|                         | Unicode codecs on decoding and '?' on         |

|                         | encoding.                                     |

+-------------------------+-----------------------------------------------+

| ``'xmlcharrefreplace'`` | Replace with the appropriate XML character    |

|                         | reference (only for encoding).                |

+-------------------------+-----------------------------------------------+

| ``'backslashreplace'``  | Replace with backslashed escape sequences     |

|                         | (only for encoding).                          |

+-------------------------+-----------------------------------------------+

The set of allowed values can be extended via :meth:`register_error`.

.. _codec-objects:

Codec Objects

^^^^^^^^^^^^^

The :class:`Codec` class defines these methods which also define the function

interfaces of the stateless encoder and decoder:

.. method:: Codec.encode(input[, errors])

   Encodes the object *input* and returns a tuple (output object, length consumed).

   While codecs are not restricted to use with Unicode, in a Unicode context,

   encoding converts a Unicode object to a plain string using a particular

   character set encoding (e.g., ``cp1252`` or ``iso-8859-1``).

   *errors* defines the error handling to apply. It defaults to ``'strict'``

   handling.

   The method may not store state in the :class:`Codec` instance. Use

   :class:`StreamCodec` for codecs which have to keep state in order to make

   encoding/decoding efficient.

   The encoder must be able to handle zero length input and return an empty object

   of the output object type in this situation.

.. method:: Codec.decode(input[, errors])

   Decodes the object *input* and returns a tuple (output object, length consumed).

   In a Unicode context, decoding converts a plain string encoded using a

   particular character set encoding to a Unicode object.

   *input* must be an object which provides the ``bf_getreadbuf`` buffer slot.

   Python strings, buffer objects and memory mapped files are examples of objects

   providing this slot.

   *errors* defines the error handling to apply. It defaults to ``'strict'``

   handling.

   The method may not store state in the :class:`Codec` instance. Use

   :class:`StreamCodec` for codecs which have to keep state in order to make

   encoding/decoding efficient.

   The decoder must be able to handle zero length input and return an empty object

   of the output object type in this situation.

The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide

the basic interface for incremental encoding and decoding. Encoding/decoding the

input isn't done with one call to the stateless encoder/decoder function, but

with multiple calls to the :meth:`encode`/:meth:`decode` method of the

incremental encoder/decoder. The incremental encoder/decoder keeps track of the

encoding/decoding process during method calls.

The joined output of calls to the :meth:`encode`/:meth:`decode` method is the

same as if all the single inputs were joined into one, and this input was

encoded/decoded with the stateless encoder/decoder.

.. _incremental-encoder-objects:

IncrementalEncoder Objects

^^^^^^^^^^^^^^^^^^^^^^^^^^

.. versionadded:: 2.5

The :class:`IncrementalEncoder` class is used for encoding an input in multiple

steps. It defines the following methods which every incremental encoder must

define in order to be compatible with the Python codec registry.

.. class:: IncrementalEncoder([errors])

   Constructor for an :class:`IncrementalEncoder` instance.

   All incremental encoders must provide this constructor interface. They are free

   to add additional keyword arguments, but only the ones defined here are used by

   the Python codec registry.

   The :class:`IncrementalEncoder` may implement different error handling schemes

   by providing the *errors* keyword argument. These parameters are predefined:

   * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.

   * ``'ignore'`` Ignore the character and continue with the next.

   * ``'replace'`` Replace with a suitable replacement character

   * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference

   * ``'backslashreplace'`` Replace with backslashed escape sequences.

   The *errors* argument will be assigned to an attribute of the same name.

   Assigning to this attribute makes it possible to switch between different error

   handling strategies during the lifetime of the :class:`IncrementalEncoder`

   object.

   The set of allowed values for the *errors* argument can be extended with

   :func:`register_error`.

   .. method:: encode(object[, final])

      Encodes *object* (taking the current state of the encoder into account)

      and returns the resulting encoded object. If this is the last call to

      :meth:`encode` *final* must be true (the default is false).

   .. method:: reset()

      Reset the encoder to the initial state.

.. _incremental-decoder-objects:

IncrementalDecoder Objects

^^^^^^^^^^^^^^^^^^^^^^^^^^

The :class:`IncrementalDecoder` class is used for decoding an input in multiple

steps. It defines the following methods which every incremental decoder must

define in order to be compatible with the Python codec registry.

.. class:: IncrementalDecoder([errors])

   Constructor for an :class:`IncrementalDecoder` instance.

   All incremental decoders must provide this constructor interface. They are free

   to add additional keyword arguments, but only the ones defined here are used by

   the Python codec registry.

   The :class:`IncrementalDecoder` may implement different error handling schemes

   by providing the *errors* keyword argument. These parameters are predefined:

   * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.

   * ``'ignore'`` Ignore the character and continue with the next.

   * ``'replace'`` Replace with a suitable replacement character.

   The *errors* argument will be assigned to an attribute of the same name.

   Assigning to this attribute makes it possible to switch between different error

   handling strategies during the lifetime of the :class:`IncrementalDecoder`

   object.

   The set of allowed values for the *errors* argument can be extended with

   :func:`register_error`.

   .. method:: decode(object[, final])

      Decodes *object* (taking the current state of the decoder into account)

      and returns the resulting decoded object. If this is the last call to

      :meth:`decode` *final* must be true (the default is false). If *final* is

      true the decoder must decode the input completely and must flush all

      buffers. If this isn't possible (e.g. because of incomplete byte sequences

      at the end of the input) it must initiate error handling just like in the

      stateless case (which might raise an exception).

   .. method:: reset()

      Reset the decoder to the initial state.

The :class:`StreamWriter` and :class:`StreamReader` classes provide generic

working interfaces which can be used to implement new encoding submodules very

easily. See :mod:`encodings.utf_8` for an example of how this is done.

.. _stream-writer-objects:

StreamWriter Objects

^^^^^^^^^^^^^^^^^^^^

The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the

following methods which every stream writer must define in order to be

compatible with the Python codec registry.

.. class:: StreamWriter(stream[, errors])

   Constructor for a :class:`StreamWriter` instance.

   All stream writers must provide this constructor interface. They are free to add

   additional keyword arguments, but only the ones defined here are used by the

   Python codec registry.

   *stream* must be a file-like object open for writing binary data.

   The :class:`StreamWriter` may implement different error handling schemes by