:mod:`multiprocessing` --- Process-based "threading" interface

==============================================================

.. module:: multiprocessing

   :synopsis: Process-based "threading" interface.

.. versionadded:: 2.6

Introduction

----------------------

:mod:`multiprocessing` is a package that supports spawning processes using an

API similar to the :mod:`threading` module.  The :mod:`multiprocessing` package

offers both local and remote concurrency, effectively side-stepping the

:term:`Global Interpreter Lock` by using subprocesses instead of threads.  Due

to this, the :mod:`multiprocessing` module allows the programmer to fully

leverage multiple processors on a given machine.  It runs on both Unix and

Windows.

.. warning::

    Some of this package's functionality requires a functioning shared semaphore

    implementation on the host operating system. Without one, the 

    :mod:`multiprocessing.synchronize` module will be disabled, and attempts to 

    import it will result in an :exc:`ImportError`. See 

    :issue:`3770` for additional information.

.. note::

    Functionality within this package requires that the ``__main__`` method be

    importable by the children. This is covered in :ref:`multiprocessing-programming`

    however it is worth pointing out here. This means that some examples, such

    as the :class:`multiprocessing.Pool` examples will not work in the

    interactive interpreter. For example::

        >>> from multiprocessing import Pool

        >>> p = Pool(5)

        >>> def f(x):

        ... 	return x*x

        ... 

        >>> p.map(f, [1,2,3])

        Process PoolWorker-1:

        Process PoolWorker-2:

        Traceback (most recent call last):

        Traceback (most recent call last):

        AttributeError: 'module' object has no attribute 'f'

        AttributeError: 'module' object has no attribute 'f'

        AttributeError: 'module' object has no attribute 'f'

The :class:`Process` class

~~~~~~~~~~~~~~~~~~~~~~~~~~

In :mod:`multiprocessing`, processes are spawned by creating a :class:`Process`

object and then calling its :meth:`~Process.start` method.  :class:`Process`

follows the API of :class:`threading.Thread`.  A trivial example of a

multiprocess program is ::

    from multiprocessing import Process

    def f(name):

        print 'hello', name

    if __name__ == '__main__':

        p = Process(target=f, args=('bob',))

        p.start()

        p.join()

To show the individual process IDs involved, here is an expanded example::

    from multiprocessing import Process

    import os

    def info(title):

        print title

        print 'module name:', __name__

        print 'parent process:', os.getppid()

        print 'process id:', os.getpid()

    def f(name):

        info('function f')

        print 'hello', name

    if __name__ == '__main__':

        info('main line')

        p = Process(target=f, args=('bob',))

        p.start()

        p.join()

For an explanation of why (on Windows) the ``if __name__ == '__main__'`` part is

necessary, see :ref:`multiprocessing-programming`.

Exchanging objects between processes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:mod:`multiprocessing` supports two types of communication channel between

processes:

**Queues**

   The :class:`Queue` class is a near clone of :class:`Queue.Queue`.  For

   example::

      from multiprocessing import Process, Queue

      def f(q):

          q.put([42, None, 'hello'])

       if __name__ == '__main__':

           q = Queue()

           p = Process(target=f, args=(q,))

           p.start()

           print q.get()    # prints "[42, None, 'hello']"

           p.join()

   Queues are thread and process safe.

**Pipes**

   The :func:`Pipe` function returns a pair of connection objects connected by a

   pipe which by default is duplex (two-way).  For example::

      from multiprocessing import Process, Pipe

      def f(conn):

          conn.send([42, None, 'hello'])

          conn.close()

      if __name__ == '__main__':

          parent_conn, child_conn = Pipe()

          p = Process(target=f, args=(child_conn,))

          p.start()

          print parent_conn.recv()   # prints "[42, None, 'hello']"

          p.join()

   The two connection objects returned by :func:`Pipe` represent the two ends of

   the pipe.  Each connection object has :meth:`~Connection.send` and

   :meth:`~Connection.recv` methods (among others).  Note that data in a pipe

   may become corrupted if two processes (or threads) try to read from or write

   to the *same* end of the pipe at the same time.  Of course there is no risk

   of corruption from processes using different ends of the pipe at the same

   time.

Synchronization between processes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:mod:`multiprocessing` contains equivalents of all the synchronization

primitives from :mod:`threading`.  For instance one can use a lock to ensure

that only one process prints to standard output at a time::

   from multiprocessing import Process, Lock

   def f(l, i):

       l.acquire()

       print 'hello world', i

       l.release()

   if __name__ == '__main__':

       lock = Lock()

       for num in range(10):

           Process(target=f, args=(lock, num)).start()

Without using the lock output from the different processes is liable to get all

mixed up.

Sharing state between processes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As mentioned above, when doing concurrent programming it is usually best to

avoid using shared state as far as possible.  This is particularly true when

using multiple processes.

However, if you really do need to use some shared data then

:mod:`multiprocessing` provides a couple of ways of doing so.

**Shared memory**

   Data can be stored in a shared memory map using :class:`Value` or

   :class:`Array`.  For example, the following code ::

      from multiprocessing import Process, Value, Array

      def f(n, a):

          n.value = 3.1415927

          for i in range(len(a)):

              a[i] = -a[i]

      if __name__ == '__main__':

          num = Value('d', 0.0)

          arr = Array('i', range(10))

          p = Process(target=f, args=(num, arr))

          p.start()

          p.join()

          print num.value

          print arr[:]

   will print ::

      3.1415927

      [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]

   The ``'d'`` and ``'i'`` arguments used when creating ``num`` and ``arr`` are

   typecodes of the kind used by the :mod:`array` module: ``'d'`` indicates a

   double precision float and ``'i'`` indicates a signed integer.  These shared

   objects will be process and thread safe.

   For more flexibility in using shared memory one can use the

   :mod:`multiprocessing.sharedctypes` module which supports the creation of

   arbitrary ctypes objects allocated from shared memory.

**Server process**

   A manager object returned by :func:`Manager` controls a server process which

   holds Python objects and allows other processes to manipulate them using

   proxies.

   A manager returned by :func:`Manager` will support types :class:`list`,

   :class:`dict`, :class:`Namespace`, :class:`Lock`, :class:`RLock`,

   :class:`Semaphore`, :class:`BoundedSemaphore`, :class:`Condition`,

   :class:`Event`, :class:`Queue`, :class:`Value` and :class:`Array`.  For

   example, ::

      from multiprocessing import Process, Manager

      def f(d, l):

          d[1] = '1'

          d['2'] = 2

          d[0.25] = None

          l.reverse()

      if __name__ == '__main__':

          manager = Manager()

          d = manager.dict()

          l = manager.list(range(10))

          p = Process(target=f, args=(d, l))

          p.start()

          p.join()

          print d

          print l

   will print ::

       {0.25: None, 1: '1', '2': 2}

       [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

   Server process managers are more flexible than using shared memory objects

   because they can be made to support arbitrary object types.  Also, a single

   manager can be shared by processes on different computers over a network.

   They are, however, slower than using shared memory.

Using a pool of workers

~~~~~~~~~~~~~~~~~~~~~~~

The :class:`~multiprocessing.pool.Pool` class represents a pool of worker

processes.  It has methods which allows tasks to be offloaded to the worker

processes in a few different ways.

For example::

   from multiprocessing import Pool

   def f(x):

       return x*x

   if __name__ == '__main__':

       pool = Pool(processes=4)              # start 4 worker processes

       result = pool.apply_async(f, [10])     # evaluate "f(10)" asynchronously

       print result.get(timeout=1)           # prints "100" unless your computer is *very* slow

       print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

Reference

---------

The :mod:`multiprocessing` package mostly replicates the API of the

:mod:`threading` module.

:class:`Process` and exceptions

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. class:: Process([group[, target[, name[, args[, kwargs]]]]])

   Process objects represent activity that is run in a separate process. The

   :class:`Process` class has equivalents of all the methods of

   :class:`threading.Thread`.

   The constructor should always be called with keyword arguments. *group*

   should always be ``None``; it exists solely for compatibility with

   :class:`threading.Thread`.  *target* is the callable object to be invoked by

   the :meth:`run()` method.  It defaults to ``None``, meaning nothing is

   called. *name* is the process name.  By default, a unique name is constructed

   of the form 'Process-N\ :sub:`1`:N\ :sub:`2`:...:N\ :sub:`k`' where N\

   :sub:`1`,N\ :sub:`2`,...,N\ :sub:`k` is a sequence of integers whose length

   is determined by the *generation* of the process.  *args* is the argument

   tuple for the target invocation.  *kwargs* is a dictionary of keyword

   arguments for the target invocation.  By default, no arguments are passed to

   *target*.

   If a subclass overrides the constructor, it must make sure it invokes the

   base class constructor (:meth:`Process.__init__`) before doing anything else

   to the process.

   .. method:: run()

      Method representing the process's activity.

      You may override this method in a subclass.  The standard :meth:`run`

      method invokes the callable object passed to the object's constructor as

      the target argument, if any, with sequential and keyword arguments taken

      from the *args* and *kwargs* arguments, respectively.

   .. method:: start()

      Start the process's activity.

      This must be called at most once per process object.  It arranges for the

      object's :meth:`run` method to be invoked in a separate process.

   .. method:: join([timeout])

      Block the calling thread until the process whose :meth:`join` method is

      called terminates or until the optional timeout occurs.

      If *timeout* is ``None`` then there is no timeout.

      A process can be joined many times.

      A process cannot join itself because this would cause a deadlock.  It is

      an error to attempt to join a process before it has been started.

   .. attribute:: name

      The process's name.

      The name is a string used for identification purposes only.  It has no

      semantics.  Multiple processes may be given the same name.  The initial

      name is set by the constructor.

   .. method:: is_alive

      Return whether the process is alive.

      Roughly, a process object is alive from the moment the :meth:`start`

      method returns until the child process terminates.

   .. attribute:: daemon

      The process's daemon flag, a Boolean value.  This must be called before

      :meth:`start` is called.

      The initial value is inherited from the creating process.

      When a process exits, it attempts to terminate all of its daemonic child

      processes.

      Note that a daemonic process is not allowed to create child processes.

      Otherwise a daemonic process would leave its children orphaned if it gets

      terminated when its parent process exits.

   In addition to the  :class:`Threading.Thread` API, :class:`Process` objects

   also support the following attributes and methods:

   .. attribute:: pid

      Return the process ID.  Before the process is spawned, this will be

      ``None``.

   .. attribute:: exitcode

      The child's exit code.  This will be ``None`` if the process has not yet

      terminated.  A negative value *-N* indicates that the child was terminated

      by signal *N*.

   .. attribute:: authkey

      The process's authentication key (a byte string).

      When :mod:`multiprocessing` is initialized the main process is assigned a

      random string using :func:`os.random`.

      When a :class:`Process` object is created, it will inherit the

      authentication key of its parent process, although this may be changed by

      setting :attr:`authkey` to another byte string.

      See :ref:`multiprocessing-auth-keys`.

   .. method:: terminate()

      Terminate the process.  On Unix this is done using the ``SIGTERM`` signal;

      on Windows :cfunc:`TerminateProcess` is used.  Note that exit handlers and

      finally clauses, etc., will not be executed.

      Note that descendant processes of the process will *not* be terminated --

      they will simply become orphaned.

      .. warning::

         If this method is used when the associated process is using a pipe or

         queue then the pipe or queue is liable to become corrupted and may

         become unusable by other process.  Similarly, if the process has

         acquired a lock or semaphore etc. then terminating it is liable to

         cause other processes to deadlock.

   Note that the :meth:`start`, :meth:`join`, :meth:`is_alive` and

   :attr:`exit_code` methods should only be called by the process that created

   the process object.

   Example usage of some of the methods of :class:`Process`::

       >>> import multiprocessing, time, signal

       >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))

       >>> print p, p.is_alive()

       <Process(Process-1, initial)> False

       >>> p.start()

       >>> print p, p.is_alive()

       <Process(Process-1, started)> True

       >>> p.terminate()

       >>> print p, p.is_alive()

       <Process(Process-1, stopped[SIGTERM])> False

       >>> p.exitcode == -signal.SIGTERM

       True

.. exception:: BufferTooShort

   Exception raised by :meth:`Connection.recv_bytes_into()` when the supplied

   buffer object is too small for the message read.

   If ``e`` is an instance of :exc:`BufferTooShort` then ``e.args[0]`` will give

   the message as a byte string.

Pipes and Queues

~~~~~~~~~~~~~~~~

When using multiple processes, one generally uses message passing for

communication between processes and avoids having to use any synchronization

primitives like locks.

For passing messages one can use :func:`Pipe` (for a connection between two

processes) or a queue (which allows multiple producers and consumers).

The :class:`Queue` and :class:`JoinableQueue` types are multi-producer,

multi-consumer FIFO queues modelled on the :class:`Queue.Queue` class in the

standard library.  They differ in that :class:`Queue` lacks the

:meth:`~Queue.Queue.task_done` and :meth:`~Queue.Queue.join` methods introduced

into Python 2.5's :class:`Queue.Queue` class.

If you use :class:`JoinableQueue` then you **must** call

:meth:`JoinableQueue.task_done` for each task removed from the queue or else the

semaphore used to count the number of unfinished tasks may eventually overflow

raising an exception.

Note that one can also create a shared queue by using a manager object -- see

:ref:`multiprocessing-managers`.

.. note::

   :mod:`multiprocessing` uses the usual :exc:`Queue.Empty` and

   :exc:`Queue.Full` exceptions to signal a timeout.  They are not available in

   the :mod:`multiprocessing` namespace so you need to import them from

   :mod:`Queue`.

.. warning::

   If a process is killed using :meth:`Process.terminate` or :func:`os.kill`

   while it is trying to use a :class:`Queue`, then the data in the queue is

   likely to become corrupted.  This may cause any other processes to get an

   exception when it tries to use the queue later on.

.. warning::

   As mentioned above, if a child process has put items on a queue (and it has

   not used :meth:`JoinableQueue.cancel_join_thread`), then that process will

   not terminate until all buffered items have been flushed to the pipe.

   This means that if you try joining that process you may get a deadlock unless

   you are sure that all items which have been put on the queue have been

   consumed.  Similarly, if the child process is non-daemonic then the parent

   process may hang on exit when it tries to join all its non-daemonic children.

   Note that a queue created using a manager does not have this issue.  See

   :ref:`multiprocessing-programming`.

For an example of the usage of queues for interprocess communication see

:ref:`multiprocessing-examples`.

.. function:: Pipe([duplex])

   Returns a pair ``(conn1, conn2)`` of :class:`Connection` objects representing

   the ends of a pipe.

   If *duplex* is ``True`` (the default) then the pipe is bidirectional.  If

   *duplex* is ``False`` then the pipe is unidirectional: ``conn1`` can only be

   used for receiving messages and ``conn2`` can only be used for sending

   messages.

.. class:: Queue([maxsize])

   Returns a process shared queue implemented using a pipe and a few

   locks/semaphores.  When a process first puts an item on the queue a feeder

   thread is started which transfers objects from a buffer into the pipe.

   The usual :exc:`Queue.Empty` and :exc:`Queue.Full` exceptions from the

   standard library's :mod:`Queue` module are raised to signal timeouts.

   :class:`Queue` implements all the methods of :class:`Queue.Queue` except for

   :meth:`~Queue.Queue.task_done` and :meth:`~Queue.Queue.join`.

   .. method:: qsize()

      Return the approximate size of the queue.  Because of