:mod:`pickle` --- Python object serialization

=============================================

.. index::

   single: persistence

   pair: persistent; objects

   pair: serializing; objects

   pair: marshalling; objects

   pair: flattening; objects

   pair: pickling; objects

.. module:: pickle

   :synopsis: Convert Python objects to streams of bytes and back.

.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>.

.. sectionauthor:: Barry Warsaw <barry@zope.com>

The :mod:`pickle` module implements a fundamental, but powerful algorithm for

serializing and de-serializing a Python object structure.  "Pickling" is the

process whereby a Python object hierarchy is converted into a byte stream, and

"unpickling" is the inverse operation, whereby a byte stream is converted back

into an object hierarchy.  Pickling (and unpickling) is alternatively known as

"serialization", "marshalling," [#]_ or "flattening", however, to avoid

confusion, the terms used here are "pickling" and "unpickling".

This documentation describes both the :mod:`pickle` module and the

:mod:`cPickle` module.

Relationship to other Python modules

------------------------------------

The :mod:`pickle` module has an optimized cousin called the :mod:`cPickle`

module.  As its name implies, :mod:`cPickle` is written in C, so it can be up to

1000 times faster than :mod:`pickle`.  However it does not support subclassing

of the :func:`Pickler` and :func:`Unpickler` classes, because in :mod:`cPickle`

these are functions, not classes.  Most applications have no need for this

functionality, and can benefit from the improved performance of :mod:`cPickle`.

Other than that, the interfaces of the two modules are nearly identical; the

common interface is described in this manual and differences are pointed out

where necessary.  In the following discussions, we use the term "pickle" to

collectively describe the :mod:`pickle` and :mod:`cPickle` modules.

The data streams the two modules produce are guaranteed to be interchangeable.

Python has a more primitive serialization module called :mod:`marshal`, but in

general :mod:`pickle` should always be the preferred way to serialize Python

objects.  :mod:`marshal` exists primarily to support Python's :file:`.pyc`

files.

The :mod:`pickle` module differs from :mod:`marshal` several significant ways:

* The :mod:`pickle` module keeps track of the objects it has already serialized,

  so that later references to the same object won't be serialized again.

  :mod:`marshal` doesn't do this.

  This has implications both for recursive objects and object sharing.  Recursive

  objects are objects that contain references to themselves.  These are not

  handled by marshal, and in fact, attempting to marshal recursive objects will

  crash your Python interpreter.  Object sharing happens when there are multiple

  references to the same object in different places in the object hierarchy being

  serialized.  :mod:`pickle` stores such objects only once, and ensures that all

  other references point to the master copy.  Shared objects remain shared, which

  can be very important for mutable objects.

* :mod:`marshal` cannot be used to serialize user-defined classes and their

  instances.  :mod:`pickle` can save and restore class instances transparently,

  however the class definition must be importable and live in the same module as

  when the object was stored.

* The :mod:`marshal` serialization format is not guaranteed to be portable

  across Python versions.  Because its primary job in life is to support

  :file:`.pyc` files, the Python implementers reserve the right to change the

  serialization format in non-backwards compatible ways should the need arise.

  The :mod:`pickle` serialization format is guaranteed to be backwards compatible

  across Python releases.

.. warning::

   The :mod:`pickle` module is not intended to be secure against erroneous or

   maliciously constructed data.  Never unpickle data received from an untrusted or

   unauthenticated source.

Note that serialization is a more primitive notion than persistence; although

:mod:`pickle` reads and writes file objects, it does not handle the issue of

naming persistent objects, nor the (even more complicated) issue of concurrent

access to persistent objects.  The :mod:`pickle` module can transform a complex

object into a byte stream and it can transform the byte stream into an object

with the same internal structure.  Perhaps the most obvious thing to do with

these byte streams is to write them onto a file, but it is also conceivable to

send them across a network or store them in a database.  The module

:mod:`shelve` provides a simple interface to pickle and unpickle objects on

DBM-style database files.

Data stream format

------------------

.. index::

   single: XDR

   single: External Data Representation

The data format used by :mod:`pickle` is Python-specific.  This has the

advantage that there are no restrictions imposed by external standards such as

XDR (which can't represent pointer sharing); however it means that non-Python

programs may not be able to reconstruct pickled Python objects.

By default, the :mod:`pickle` data format uses a printable ASCII representation.

This is slightly more voluminous than a binary representation.  The big

advantage of using printable ASCII (and of some other characteristics of

:mod:`pickle`'s representation) is that for debugging or recovery purposes it is

possible for a human to read the pickled file with a standard text editor.

There are currently 3 different protocols which can be used for pickling.

* Protocol version 0 is the original ASCII protocol and is backwards compatible

  with earlier versions of Python.

* Protocol version 1 is the old binary format which is also compatible with

  earlier versions of Python.

* Protocol version 2 was introduced in Python 2.3.  It provides much more

  efficient pickling of :term:`new-style class`\es.

Refer to :pep:`307` for more information.

If a *protocol* is not specified, protocol 0 is used. If *protocol* is specified

as a negative value or :const:`HIGHEST_PROTOCOL`, the highest protocol version

available will be used.

.. versionchanged:: 2.3

   Introduced the *protocol* parameter.

A binary format, which is slightly more efficient, can be chosen by specifying a

*protocol* version >= 1.

Usage

-----

To serialize an object hierarchy, you first create a pickler, then you call the

pickler's :meth:`dump` method.  To de-serialize a data stream, you first create

an unpickler, then you call the unpickler's :meth:`load` method.  The

:mod:`pickle` module provides the following constant:

.. data:: HIGHEST_PROTOCOL

   The highest protocol version available.  This value can be passed as a

   *protocol* value.

   .. versionadded:: 2.3

.. note::

   Be sure to always open pickle files created with protocols >= 1 in binary mode.

   For the old ASCII-based pickle protocol 0 you can use either text mode or binary

   mode as long as you stay consistent.

   A pickle file written with protocol 0 in binary mode will contain lone linefeeds

   as line terminators and therefore will look "funny" when viewed in Notepad or

   other editors which do not support this format.

The :mod:`pickle` module provides the following functions to make the pickling

process more convenient:

.. function:: dump(obj, file[, protocol])

   Write a pickled representation of *obj* to the open file object *file*.  This is

   equivalent to ``Pickler(file, protocol).dump(obj)``.

   If the *protocol* parameter is omitted, protocol 0 is used. If *protocol* is

   specified as a negative value or :const:`HIGHEST_PROTOCOL`, the highest protocol

   version will be used.

   .. versionchanged:: 2.3

      Introduced the *protocol* parameter.

   *file* must have a :meth:`write` method that accepts a single string argument.

   It can thus be a file object opened for writing, a :mod:`StringIO` object, or

   any other custom object that meets this interface.

.. function:: load(file)

   Read a string from the open file object *file* and interpret it as a pickle data

   stream, reconstructing and returning the original object hierarchy.  This is

   equivalent to ``Unpickler(file).load()``.

   *file* must have two methods, a :meth:`read` method that takes an integer

   argument, and a :meth:`readline` method that requires no arguments.  Both

   methods should return a string.  Thus *file* can be a file object opened for

   reading, a :mod:`StringIO` object, or any other custom object that meets this

   interface.

   This function automatically determines whether the data stream was written in

   binary mode or not.

.. function:: dumps(obj[, protocol])

   Return the pickled representation of the object as a string, instead of writing

   it to a file.

   If the *protocol* parameter is omitted, protocol 0 is used. If *protocol* is

   specified as a negative value or :const:`HIGHEST_PROTOCOL`, the highest protocol

   version will be used.

   .. versionchanged:: 2.3

      The *protocol* parameter was added.

.. function:: loads(string)

   Read a pickled object hierarchy from a string.  Characters in the string past

   the pickled object's representation are ignored.

The :mod:`pickle` module also defines three exceptions:

.. exception:: PickleError

   A common base class for the other exceptions defined below.  This inherits from

   :exc:`Exception`.

.. exception:: PicklingError

   This exception is raised when an unpicklable object is passed to the

   :meth:`dump` method.

.. exception:: UnpicklingError

   This exception is raised when there is a problem unpickling an object. Note that

   other exceptions may also be raised during unpickling, including (but not

   necessarily limited to) :exc:`AttributeError`, :exc:`EOFError`,

   :exc:`ImportError`, and :exc:`IndexError`.

The :mod:`pickle` module also exports two callables [#]_, :class:`Pickler` and

:class:`Unpickler`:

.. class:: Pickler(file[, protocol])

   This takes a file-like object to which it will write a pickle data stream.

   If the *protocol* parameter is omitted, protocol 0 is used. If *protocol* is

   specified as a negative value or :const:`HIGHEST_PROTOCOL`, the highest

   protocol version will be used.

   .. versionchanged:: 2.3

      Introduced the *protocol* parameter.

   *file* must have a :meth:`write` method that accepts a single string argument.

   It can thus be an open file object, a :mod:`StringIO` object, or any other

   custom object that meets this interface.

   :class:`Pickler` objects define one (or two) public methods:

   .. method:: dump(obj)

      Write a pickled representation of *obj* to the open file object given in the

      constructor.  Either the binary or ASCII format will be used, depending on the

      value of the *protocol* argument passed to the constructor.

   .. method:: clear_memo()

      Clears the pickler's "memo".  The memo is the data structure that remembers

      which objects the pickler has already seen, so that shared or recursive objects

      pickled by reference and not by value.  This method is useful when re-using

      picklers.

      .. note::

         Prior to Python 2.3, :meth:`clear_memo` was only available on the picklers

         created by :mod:`cPickle`.  In the :mod:`pickle` module, picklers have an

         instance variable called :attr:`memo` which is a Python dictionary.  So to clear

         the memo for a :mod:`pickle` module pickler, you could do the following::

            mypickler.memo.clear()

         Code that does not need to support older versions of Python should simply use

         :meth:`clear_memo`.

It is possible to make multiple calls to the :meth:`dump` method of the same

:class:`Pickler` instance.  These must then be matched to the same number of

calls to the :meth:`load` method of the corresponding :class:`Unpickler`

instance.  If the same object is pickled by multiple :meth:`dump` calls, the

:meth:`load` will all yield references to the same object. [#]_

:class:`Unpickler` objects are defined as:

.. class:: Unpickler(file)

   This takes a file-like object from which it will read a pickle data stream.

   This class automatically determines whether the data stream was written in

   binary mode or not, so it does not need a flag as in the :class:`Pickler`

   factory.

   *file* must have two methods, a :meth:`read` method that takes an integer

   argument, and a :meth:`readline` method that requires no arguments.  Both

   methods should return a string.  Thus *file* can be a file object opened for

   reading, a :mod:`StringIO` object, or any other custom object that meets this

   interface.

   :class:`Unpickler` objects have one (or two) public methods:

   .. method:: load()

      Read a pickled object representation from the open file object given in

      the constructor, and return the reconstituted object hierarchy specified

      therein.

      This method automatically determines whether the data stream was written

      in binary mode or not.

   .. method:: noload()

      This is just like :meth:`load` except that it doesn't actually create any

      objects.  This is useful primarily for finding what's called "persistent

      ids" that may be referenced in a pickle data stream.  See section

      :ref:`pickle-protocol` below for more details.

      **Note:** the :meth:`noload` method is currently only available on

      :class:`Unpickler` objects created with the :mod:`cPickle` module.

      :mod:`pickle` module :class:`Unpickler`\ s do not have the :meth:`noload`

      method.

What can be pickled and unpickled?

----------------------------------

The following types can be pickled:

* ``None``, ``True``, and ``False``

* integers, long integers, floating point numbers, complex numbers

* normal and Unicode strings

* tuples, lists, sets, and dictionaries containing only picklable objects

* functions defined at the top level of a module

* built-in functions defined at the top level of a module

* classes that are defined at the top level of a module

* instances of such classes whose :attr:`__dict__` or :meth:`__setstate__` is

  picklable  (see section :ref:`pickle-protocol` for details)

Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`

exception; when this happens, an unspecified number of bytes may have already

been written to the underlying file. Trying to pickle a highly recursive data

structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be

raised in this case. You can carefully raise this limit with

:func:`sys.setrecursionlimit`.

Note that functions (built-in and user-defined) are pickled by "fully qualified"

name reference, not by value.  This means that only the function name is

pickled, along with the name of module the function is defined in.  Neither the

function's code, nor any of its function attributes are pickled.  Thus the

defining module must be importable in the unpickling environment, and the module

must contain the named object, otherwise an exception will be raised. [#]_

Similarly, classes are pickled by named reference, so the same restrictions in

the unpickling environment apply.  Note that none of the class's code or data is

pickled, so in the following example the class attribute ``attr`` is not

restored in the unpickling environment::

   class Foo:

       attr = 'a class attr'

   picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined in

the top level of a module.

Similarly, when class instances are pickled, their class's code and data are not

pickled along with them.  Only the instance data are pickled.  This is done on

purpose, so you can fix bugs in a class or add methods to the class and still

load objects that were created with an earlier version of the class.  If you

plan to have long-lived objects that will see many versions of a class, it may

be worthwhile to put a version number in the objects so that suitable

conversions can be made by the class's :meth:`__setstate__` method.

.. _pickle-protocol:

The pickle protocol

-------------------

.. currentmodule:: None

This section describes the "pickling protocol" that defines the interface

between the pickler/unpickler and the objects that are being serialized.  This

protocol provides a standard way for you to define, customize, and control how

your objects are serialized and de-serialized.  The description in this section

doesn't cover specific customizations that you can employ to make the unpickling

environment slightly safer from untrusted pickle data streams; see section

:ref:`pickle-sub` for more details.

.. _pickle-inst:

Pickling and unpickling normal class instances

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. method:: object.__getinitargs__()

   When a pickled class instance is unpickled, its :meth:`__init__` method is

   normally *not* invoked.  If it is desirable that the :meth:`__init__` method

   be called on unpickling, an old-style class can define a method

   :meth:`__getinitargs__`, which should return a *tuple* containing the

   arguments to be passed to the class constructor (:meth:`__init__` for

   example).  The :meth:`__getinitargs__` method is called at pickle time; the

   tuple it returns is incorporated in the pickle for the instance.

.. method:: object.__getnewargs__()

   New-style types can provide a :meth:`__getnewargs__` method that is used for

   protocol 2.  Implementing this method is needed if the type establishes some

   internal invariants when the instance is created, or if the memory allocation

   is affected by the values passed to the :meth:`__new__` method for the type

   (as it is for tuples and strings).  Instances of a :term:`new-style class`

   ``C`` are created using ::

      obj = C.__new__(C, *args)

   where *args* is the result of calling :meth:`__getnewargs__` on the original

   object; if there is no :meth:`__getnewargs__`, an empty tuple is assumed.

.. method:: object.__getstate__()

   Classes can further influence how their instances are pickled; if the class

   defines the method :meth:`__getstate__`, it is called and the return state is

   pickled as the contents for the instance, instead of the contents of the

   instance's dictionary.  If there is no :meth:`__getstate__` method, the

   instance's :attr:`__dict__` is pickled.

.. method:: object.__setstate__() 

   Upon unpickling, if the class also defines the method :meth:`__setstate__`,

   it is called with the unpickled state. [#]_ If there is no

   :meth:`__setstate__` method, the pickled state must be a dictionary and its

   items are assigned to the new instance's dictionary.  If a class defines both

   :meth:`__getstate__` and :meth:`__setstate__`, the state object needn't be a

   dictionary and these methods can do what they want. [#]_

   .. warning::

      For :term:`new-style class`\es, if :meth:`__getstate__` returns a false

      value, the :meth:`__setstate__` method will not be called.

Pickling and unpickling extension types

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. method:: object.__reduce__()

   When the :class:`Pickler` encounters an object of a type it knows nothing

   about --- such as an extension type --- it looks in two places for a hint of

   how to pickle it.  One alternative is for the object to implement a

   :meth:`__reduce__` method.  If provided, at pickling time :meth:`__reduce__`

   will be called with no arguments, and it must return either a string or a

   tuple.

   If a string is returned, it names a global variable whose contents are

   pickled as normal.  The string returned by :meth:`__reduce__` should be the

   object's local name relative to its module; the pickle module searches the

   module namespace to determine the object's module.

   When a tuple is returned, it must be between two and five elements long.

   Optional elements can either be omitted, or ``None`` can be provided as their

   value.  The contents of this tuple are pickled as normal and used to

   reconstruct the object at unpickling time.  The semantics of each element

   are:

   * A callable object that will be called to create the initial version of the

     object.  The next element of the tuple will provide arguments for this

     callable, and later elements provide additional state information that will

     subsequently be used to fully reconstruct the pickled data.

     In the unpickling environment this object must be either a class, a

     callable registered as a "safe constructor" (see below), or it must have an

     attribute :attr:`__safe_for_unpickling__` with a true value. Otherwise, an

     :exc:`UnpicklingError` will be raised in the unpickling environment.  Note

     that as usual, the callable itself is pickled by name.

   * A tuple of arguments for the callable object.

     .. versionchanged:: 2.5

        Formerly, this argument could also be ``None``.

   * Optionally, the object's state, which will be passed to the object's

     :meth:`__setstate__` method as described in section :ref:`pickle-inst`.  If

     the object has no :meth:`__setstate__` method, then, as above, the value

     must be a dictionary and it will be added to the object's :attr:`__dict__`.

   * Optionally, an iterator (and not a sequence) yielding successive list

     items.  These list items will be pickled, and appended to the object using

     either ``obj.append(item)`` or ``obj.extend(list_of_items)``.  This is

     primarily used for list subclasses, but may be used by other classes as

     long as they have :meth:`append` and :meth:`extend` methods with the

     appropriate signature.  (Whether :meth:`append` or :meth:`extend` is used

     depends on which pickle protocol version is used as well as the number of

     items to append, so both must be supported.)

   * Optionally, an iterator (not a sequence) yielding successive dictionary

     items, which should be tuples of the form ``(key, value)``.  These items

     will be pickled and stored to the object using ``obj[key] = value``. This

     is primarily used for dictionary subclasses, but may be used by other

     classes as long as they implement :meth:`__setitem__`.

.. method:: object.__reduce_ex__(protocol) 

   It is sometimes useful to know the protocol version when implementing

   :meth:`__reduce__`.  This can be done by implementing a method named

   :meth:`__reduce_ex__` instead of :meth:`__reduce__`. :meth:`__reduce_ex__`,

   when it exists, is called in preference over :meth:`__reduce__` (you may

   still provide :meth:`__reduce__` for backwards compatibility).  The

   :meth:`__reduce_ex__` method will be called with a single integer argument,