Skip to content
Snippets Groups Projects
multiprocessing.rst 109 KiB
Newer Older
  • Learn to ignore specific revisions
  • :mod:`multiprocessing` --- Process-based parallelism
    ====================================================
    
    
    .. module:: multiprocessing
    
       :synopsis: Process-based parallelism.
    
    **Source code:** :source:`Lib/multiprocessing/`
    
    --------------
    
    ------------
    
    :mod:`multiprocessing` is a package that supports spawning processes using an
    API similar to the :mod:`threading` module.  The :mod:`multiprocessing` package
    offers both local and remote concurrency, effectively side-stepping the
    
    :term:`Global Interpreter Lock <global interpreter lock>` by using
    subprocesses instead of threads.  Due
    
    to this, the :mod:`multiprocessing` module allows the programmer to fully
    leverage multiple processors on a given machine.  It runs on both Unix and
    Windows.
    
    The :mod:`multiprocessing` module also introduces APIs which do not have
    analogs in the :mod:`threading` module.  A prime example of this is the
    :class:`~multiprocessing.pool.Pool` object which offers a convenient means of
    parallelizing the execution of a function across multiple input values,
    distributing the input data across processes (data parallelism).  The following
    example demonstrates the common practice of defining such functions in a module
    so that child processes can successfully import that module.  This basic example
    of data parallelism using :class:`~multiprocessing.pool.Pool`, ::
    
       def f(x):
           return x*x
    
       if __name__ == '__main__':
           with Pool(5) as p:
               print(p.map(f, [1, 2, 3]))
    
    The :class:`Process` class
    ~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    In :mod:`multiprocessing`, processes are spawned by creating a :class:`Process`
    
    object and then calling its :meth:`~Process.start` method.  :class:`Process`
    
    follows the API of :class:`threading.Thread`.  A trivial example of a
    multiprocess program is ::
    
    
    Georg Brandl's avatar
    Georg Brandl committed
       from multiprocessing import Process
    
           print('hello', name)
    
    Georg Brandl's avatar
    Georg Brandl committed
       if __name__ == '__main__':
           p = Process(target=f, args=('bob',))
           p.start()
           p.join()
    
    
    To show the individual process IDs involved, here is an expanded example::
    
        from multiprocessing import Process
        import os
    
        def info(title):
    
            print('parent process:', os.getppid())
    
        def f(name):
            info('function f')
    
        if __name__ == '__main__':
            info('main line')
            p = Process(target=f, args=('bob',))
            p.start()
            p.join()
    
    For an explanation of why the ``if __name__ == '__main__'`` part is
    
    necessary, see :ref:`multiprocessing-programming`.
    
    
    
    
    Contexts and start methods
    ~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    Depending on the platform, :mod:`multiprocessing` supports three ways
    to start a process.  These *start methods* are
    
      *spawn*
    
        The parent process starts a fresh Python interpreter process.  The
    
        child process will only inherit those resources necessary to run
    
        the process object's :meth:`~Process.run` method.  In particular,
    
        unnecessary file descriptors and handles from the parent process
        will not be inherited.  Starting a process using this method is
        rather slow compared to using *fork* or *forkserver*.
    
    
        Available on Unix and Windows.  The default on Windows and macOS.
    
    
      *fork*
        The parent process uses :func:`os.fork` to fork the Python
        interpreter.  The child process, when it begins, is effectively
        identical to the parent process.  All resources of the parent are
        inherited by the child process.  Note that safely forking a
        multithreaded process is problematic.
    
        Available on Unix only.  The default on Unix.
    
      *forkserver*
        When the program starts and selects the *forkserver* start method,
        a server process is started.  From then on, whenever a new process
    
        is needed, the parent process connects to the server and requests
    
        that it fork a new process.  The fork server process is single
        threaded so it is safe for it to use :func:`os.fork`.  No
        unnecessary resources are inherited.
    
        Available on Unix platforms which support passing file descriptors
    
       On macOS, the *spawn* start method is now the default.  The *fork* start
       method should be considered unsafe as it can lead to crashes of the
       subprocess. See :issue:`33725`.
    
    .. versionchanged:: 3.4
       *spawn* added on all unix platforms, and *forkserver* added for
    
       Child processes no longer inherit all of the parents inheritable
    
    
    On Unix using the *spawn* or *forkserver* start methods will also
    
    start a *resource tracker* process which tracks the unlinked named
    system resources (such as named semaphores or
    :class:`~multiprocessing.shared_memory.SharedMemory` objects) created
    by processes of the program.  When all processes
    have exited the resource tracker unlinks any remaining tracked object.
    
    Usually there should be none, but if a process was killed by a signal
    
    there may be some "leaked" resources.  (Neither leaked semaphores nor shared
    memory segments will be automatically unlinked until the next reboot. This is
    problematic for both objects because the system allows only a limited number of
    named semaphores, and shared memory segments occupy some space in the main
    memory.)
    
    To select a start method you use the :func:`set_start_method` in
    
    the ``if __name__ == '__main__'`` clause of the main module.  For
    example::
    
           import multiprocessing as mp
    
    
    
           if __name__ == '__main__':
               mp.set_start_method('spawn')
    
               q = mp.Queue()
               p = mp.Process(target=foo, args=(q,))
    
               p.join()
    
    :func:`set_start_method` should not be used more than once in the
    program.
    
    
    Alternatively, you can use :func:`get_context` to obtain a context
    object.  Context objects have the same API as the multiprocessing
    module, and allow one to use multiple start methods in the same
    program. ::
    
           import multiprocessing as mp
    
           def foo(q):
               q.put('hello')
    
           if __name__ == '__main__':
               ctx = mp.get_context('spawn')
               q = ctx.Queue()
               p = ctx.Process(target=foo, args=(q,))
               p.start()
               print(q.get())
               p.join()
    
    Note that objects related to one context may not be compatible with
    processes for a different context.  In particular, locks created using
    
    the *fork* context cannot be passed to processes started using the
    
    *spawn* or *forkserver* start methods.
    
    A library which wants to use a particular start method should probably
    use :func:`get_context` to avoid interfering with the choice of the
    library user.
    
    .. warning::
    
       The ``'spawn'`` and ``'forkserver'`` start methods cannot currently
       be used with "frozen" executables (i.e., binaries produced by
       packages like **PyInstaller** and **cx_Freeze**) on Unix.
       The ``'fork'`` start method does work.
    
    
    Exchanging objects between processes
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    :mod:`multiprocessing` supports two types of communication channel between
    processes:
    
    **Queues**
    
    
       The :class:`Queue` class is a near clone of :class:`queue.Queue`.  For
    
       example::
    
          from multiprocessing import Process, Queue
    
          def f(q):
              q.put([42, None, 'hello'])
    
    
          if __name__ == '__main__':
              q = Queue()
              p = Process(target=f, args=(q,))
              p.start()
              print(q.get())    # prints "[42, None, 'hello']"
              p.join()
    
       Queues are thread and process safe.
    
    
    **Pipes**
    
       The :func:`Pipe` function returns a pair of connection objects connected by a
       pipe which by default is duplex (two-way).  For example::
    
          from multiprocessing import Process, Pipe
    
          def f(conn):
              conn.send([42, None, 'hello'])
              conn.close()
    
          if __name__ == '__main__':
              parent_conn, child_conn = Pipe()
              p = Process(target=f, args=(child_conn,))
              p.start()
    
              print(parent_conn.recv())   # prints "[42, None, 'hello']"
    
              p.join()
    
       The two connection objects returned by :func:`Pipe` represent the two ends of
    
       the pipe.  Each connection object has :meth:`~Connection.send` and
       :meth:`~Connection.recv` methods (among others).  Note that data in a pipe
       may become corrupted if two processes (or threads) try to read from or write
       to the *same* end of the pipe at the same time.  Of course there is no risk
       of corruption from processes using different ends of the pipe at the same
       time.
    
    
    
    Synchronization between processes
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    :mod:`multiprocessing` contains equivalents of all the synchronization
    primitives from :mod:`threading`.  For instance one can use a lock to ensure
    that only one process prints to standard output at a time::
    
       from multiprocessing import Process, Lock
    
       def f(l, i):
           l.acquire()
    
           try:
               print('hello world', i)
           finally:
               l.release()
    
    
       if __name__ == '__main__':
           lock = Lock()
    
           for num in range(10):
               Process(target=f, args=(lock, num)).start()
    
    Without using the lock output from the different processes is liable to get all
    mixed up.
    
    
    Sharing state between processes
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    As mentioned above, when doing concurrent programming it is usually best to
    avoid using shared state as far as possible.  This is particularly true when
    using multiple processes.
    
    However, if you really do need to use some shared data then
    :mod:`multiprocessing` provides a couple of ways of doing so.
    
    **Shared memory**
    
       Data can be stored in a shared memory map using :class:`Value` or
       :class:`Array`.  For example, the following code ::
    
          from multiprocessing import Process, Value, Array
    
          def f(n, a):
              n.value = 3.1415927
              for i in range(len(a)):
                  a[i] = -a[i]
    
          if __name__ == '__main__':
              num = Value('d', 0.0)
              arr = Array('i', range(10))
    
              p = Process(target=f, args=(num, arr))
              p.start()
              p.join()
    
    
              print(num.value)
              print(arr[:])
    
    
       will print ::
    
          3.1415927
          [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
    
       The ``'d'`` and ``'i'`` arguments used when creating ``num`` and ``arr`` are
       typecodes of the kind used by the :mod:`array` module: ``'d'`` indicates a
    
    Georg Brandl's avatar
    Georg Brandl committed
       double precision float and ``'i'`` indicates a signed integer.  These shared
    
       objects will be process and thread-safe.
    
    
       For more flexibility in using shared memory one can use the
       :mod:`multiprocessing.sharedctypes` module which supports the creation of
       arbitrary ctypes objects allocated from shared memory.
    
    **Server process**
    
       A manager object returned by :func:`Manager` controls a server process which
    
    Georg Brandl's avatar
    Georg Brandl committed
       holds Python objects and allows other processes to manipulate them using
    
       A manager returned by :func:`Manager` will support types
    
       :class:`list`, :class:`dict`, :class:`~managers.Namespace`, :class:`Lock`,
    
       :class:`RLock`, :class:`Semaphore`, :class:`BoundedSemaphore`,
       :class:`Condition`, :class:`Event`, :class:`Barrier`,
       :class:`Queue`, :class:`Value` and :class:`Array`.  For example, ::
    
    
          from multiprocessing import Process, Manager
    
          def f(d, l):
              d[1] = '1'
              d['2'] = 2
              d[0.25] = None
              l.reverse()
    
          if __name__ == '__main__':
    
              with Manager() as manager:
                  d = manager.dict()
                  l = manager.list(range(10))
    
                  p = Process(target=f, args=(d, l))
                  p.start()
                  p.join()
    
    
       will print ::
    
           {0.25: None, 1: '1', '2': 2}
           [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
    
       Server process managers are more flexible than using shared memory objects
       because they can be made to support arbitrary object types.  Also, a single
       manager can be shared by processes on different computers over a network.
       They are, however, slower than using shared memory.
    
    
    Using a pool of workers
    ~~~~~~~~~~~~~~~~~~~~~~~
    
    
    The :class:`~multiprocessing.pool.Pool` class represents a pool of worker
    
    processes.  It has methods which allows tasks to be offloaded to the worker
    processes in a few different ways.
    
    For example::
    
    
       from multiprocessing import Pool, TimeoutError
       import time
       import os
    
    
       def f(x):
           return x*x
    
       if __name__ == '__main__':
    
           # start 4 worker processes
           with Pool(processes=4) as pool:
    
               # print "[0, 1, 4,..., 81]"
               print(pool.map(f, range(10)))
    
               # print same numbers in arbitrary order
               for i in pool.imap_unordered(f, range(10)):
                   print(i)
    
    
               # evaluate "f(20)" asynchronously
               res = pool.apply_async(f, (20,))      # runs in *only* one process
               print(res.get(timeout=1))             # prints "400"
    
               # evaluate "os.getpid()" asynchronously
               res = pool.apply_async(os.getpid, ()) # runs in *only* one process
               print(res.get(timeout=1))             # prints the PID of that process
    
               # launching multiple evaluations asynchronously *may* use more processes
               multiple_results = [pool.apply_async(os.getpid, ()) for i in range(4)]
               print([res.get(timeout=1) for res in multiple_results])
    
    
               # make a single worker sleep for 10 seconds
    
               res = pool.apply_async(time.sleep, (10,))
               try:
                   print(res.get(timeout=1))
               except TimeoutError:
                   print("We lacked patience and got a multiprocessing.TimeoutError")
    
               print("For the moment, the pool remains available for more work")
    
    
           # exiting the 'with'-block has stopped the pool
    
           print("Now the pool is closed and no longer available")
    
    Note that the methods of a pool should only ever be used by the
    process which created it.
    
    
    .. note::
    
       Functionality within this package requires that the ``__main__`` module be
       importable by the children. This is covered in :ref:`multiprocessing-programming`
       however it is worth pointing out here. This means that some examples, such
       as the :class:`multiprocessing.pool.Pool` examples will not work in the
       interactive interpreter. For example::
    
          >>> from multiprocessing import Pool
          >>> p = Pool(5)
          >>> def f(x):
          ...     return x*x
          ...
    
          Process PoolWorker-1:
          Process PoolWorker-2:
          Process PoolWorker-3:
          Traceback (most recent call last):
          Traceback (most recent call last):
          Traceback (most recent call last):
          AttributeError: 'module' object has no attribute 'f'
          AttributeError: 'module' object has no attribute 'f'
          AttributeError: 'module' object has no attribute 'f'
    
       (If you try this it will actually output three full tracebacks
       interleaved in a semi-random fashion, and then you may have to
    
       stop the parent process somehow.)
    
    
    Reference
    ---------
    
    The :mod:`multiprocessing` package mostly replicates the API of the
    :mod:`threading` module.
    
    
    :class:`Process` and exceptions
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    
    Ezio Melotti's avatar
    Ezio Melotti committed
    .. class:: Process(group=None, target=None, name=None, args=(), kwargs={}, \
                       *, daemon=None)
    
    
       Process objects represent activity that is run in a separate process. The
       :class:`Process` class has equivalents of all the methods of
       :class:`threading.Thread`.
    
       The constructor should always be called with keyword arguments. *group*
    
       should always be ``None``; it exists solely for compatibility with
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       :class:`threading.Thread`.  *target* is the callable object to be invoked by
    
       the :meth:`run()` method.  It defaults to ``None``, meaning nothing is
    
       called. *name* is the process name (see :attr:`name` for more details).
       *args* is the argument tuple for the target invocation.  *kwargs* is a
       dictionary of keyword arguments for the target invocation.  If provided,
       the keyword-only *daemon* argument sets the process :attr:`daemon` flag
       to ``True`` or ``False``.  If ``None`` (the default), this flag will be
       inherited from the creating process.
    
       By default, no arguments are passed to *target*. The *args* argument,
       which defaults to ``()``, can be used to specify a list or tuple of the arguments
       to pass to *target*.
    
    
       If a subclass overrides the constructor, it must make sure it invokes the
       base class constructor (:meth:`Process.__init__`) before doing anything else
       to the process.
    
    
       .. versionchanged:: 3.3
          Added the *daemon* argument.
    
    
       .. method:: run()
    
          Method representing the process's activity.
    
          You may override this method in a subclass.  The standard :meth:`run`
          method invokes the callable object passed to the object's constructor as
          the target argument, if any, with sequential and keyword arguments taken
          from the *args* and *kwargs* arguments, respectively.
    
    
          Using a list or tuple as the *args* argument passed to :class:`Process`
          achieves the same effect.
    
          Example::
    
             >>> from multiprocessing import Process
             >>> p = Process(target=print, args=[1])
             >>> p.run()
             1
             >>> p = Process(target=print, args=(1,))
             >>> p.run()
             1
    
    
       .. method:: start()
    
          Start the process's activity.
    
          This must be called at most once per process object.  It arranges for the
          object's :meth:`run` method to be invoked in a separate process.
    
       .. method:: join([timeout])
    
    
          If the optional argument *timeout* is ``None`` (the default), the method
          blocks until the process whose :meth:`join` method is called terminates.
          If *timeout* is a positive number, it blocks at most *timeout* seconds.
    
          Note that the method returns ``None`` if its process terminates or if the
          method times out.  Check the process's :attr:`exitcode` to determine if
          it terminated.
    
    
          A process can be joined many times.
    
          A process cannot join itself because this would cause a deadlock.  It is
          an error to attempt to join a process before it has been started.
    
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       .. attribute:: name
    
          The process's name.  The name is a string used for identification purposes
          only.  It has no semantics.  Multiple processes may be given the same
          name.
    
          The initial name is set by the constructor.  If no explicit name is
          provided to the constructor, a name of the form
          'Process-N\ :sub:`1`:N\ :sub:`2`:...:N\ :sub:`k`' is constructed, where
          each N\ :sub:`k` is the N-th child of its parent.
    
       .. method:: is_alive
    
    
          Return whether the process is alive.
    
          Roughly, a process object is alive from the moment the :meth:`start`
          method returns until the child process terminates.
    
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       .. attribute:: daemon
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
          The process's daemon flag, a Boolean value.  This must be set before
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
          :meth:`start` is called.
    
    
          The initial value is inherited from the creating process.
    
          When a process exits, it attempts to terminate all of its daemonic child
          processes.
    
          Note that a daemonic process is not allowed to create child processes.
          Otherwise a daemonic process would leave its children orphaned if it gets
    
          terminated when its parent process exits. Additionally, these are **not**
          Unix daemons or services, they are normal processes that will be
    
    Georg Brandl's avatar
    Georg Brandl committed
          terminated (and not joined) if non-daemonic processes have exited.
    
       In addition to the  :class:`threading.Thread` API, :class:`Process` objects
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       also support the following attributes and methods:
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       .. attribute:: pid
    
    
          Return the process ID.  Before the process is spawned, this will be
          ``None``.
    
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       .. attribute:: exitcode
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
          The child's exit code.  This will be ``None`` if the process has not yet
    
          terminated.
    
          If the child's :meth:`run` method returned normally, the exit code
          will be 0.  If it terminated via :func:`sys.exit` with an integer
          argument *N*, the exit code will be *N*.
    
          If the child terminated due to an exception not caught within
          :meth:`run`, the exit code will be 1.  If it was terminated by
          signal *N*, the exit code will be the negative value *-N*.
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
       .. attribute:: authkey
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
          The process's authentication key (a byte string).
    
    
          When :mod:`multiprocessing` is initialized the main process is assigned a
    
          random string using :func:`os.urandom`.
    
    
          When a :class:`Process` object is created, it will inherit the
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
          authentication key of its parent process, although this may be changed by
          setting :attr:`authkey` to another byte string.
    
    
          See :ref:`multiprocessing-auth-keys`.
    
    
       .. attribute:: sentinel
    
          A numeric handle of a system object which will become "ready" when
          the process ends.
    
    
          You can use this value if you want to wait on several events at
          once using :func:`multiprocessing.connection.wait`.  Otherwise
          calling :meth:`join()` is simpler.
    
    
          On Windows, this is an OS handle usable with the ``WaitForSingleObject``
          and ``WaitForMultipleObjects`` family of API calls.  On Unix, this is
          a file descriptor usable with primitives from the :mod:`select` module.
    
          .. versionadded:: 3.3
    
    
       .. method:: terminate()
    
          Terminate the process.  On Unix this is done using the ``SIGTERM`` signal;
    
          on Windows :c:func:`TerminateProcess` is used.  Note that exit handlers and
    
          finally clauses, etc., will not be executed.
    
    
          Note that descendant processes of the process will *not* be terminated --
          they will simply become orphaned.
    
          .. warning::
    
             If this method is used when the associated process is using a pipe or
             queue then the pipe or queue is liable to become corrupted and may
             become unusable by other process.  Similarly, if the process has
             acquired a lock or semaphore etc. then terminating it is liable to
             cause other processes to deadlock.
    
    
       .. method:: kill()
    
          Same as :meth:`terminate()` but using the ``SIGKILL`` signal on Unix.
    
          .. versionadded:: 3.7
    
    
       .. method:: close()
    
          Close the :class:`Process` object, releasing all resources associated
          with it.  :exc:`ValueError` is raised if the underlying process
          is still running.  Once :meth:`close` returns successfully, most
          other methods and attributes of the :class:`Process` object will
          raise :exc:`ValueError`.
    
          .. versionadded:: 3.7
    
    
       Note that the :meth:`start`, :meth:`join`, :meth:`is_alive`,
    
       :meth:`terminate` and :attr:`exitcode` methods should only be called by
    
       the process that created the process object.
    
       Example usage of some of the methods of :class:`Process`:
    
       .. doctest::
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
           >>> import multiprocessing, time, signal
           >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
    
           >>> print(p, p.is_alive())
    
           >>> print(p, p.is_alive())
    
           >>> time.sleep(0.1)
    
           >>> print(p, p.is_alive())
    
           <Process ... stopped exitcode=-SIGTERM> False
    
    Benjamin Peterson's avatar
    Benjamin Peterson committed
           >>> p.exitcode == -signal.SIGTERM
    
    .. exception:: ProcessError
    
       The base class of all :mod:`multiprocessing` exceptions.
    
    
    .. exception:: BufferTooShort
    
       Exception raised by :meth:`Connection.recv_bytes_into()` when the supplied
       buffer object is too small for the message read.
    
       If ``e`` is an instance of :exc:`BufferTooShort` then ``e.args[0]`` will give
       the message as a byte string.
    
    
    .. exception:: AuthenticationError
    
       Raised when there is an authentication error.
    
    .. exception:: TimeoutError
    
       Raised by methods with a timeout when the timeout expires.
    
    
    Pipes and Queues
    ~~~~~~~~~~~~~~~~
    
    When using multiple processes, one generally uses message passing for
    communication between processes and avoids having to use any synchronization
    primitives like locks.
    
    For passing messages one can use :func:`Pipe` (for a connection between two
    processes) or a queue (which allows multiple producers and consumers).
    
    
    The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types
    are multi-producer, multi-consumer :abbr:`FIFO (first-in, first-out)`
    queues modelled on the :class:`queue.Queue` class in the
    
    standard library.  They differ in that :class:`Queue` lacks the
    
    :meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced
    into Python 2.5's :class:`queue.Queue` class.
    
    
    If you use :class:`JoinableQueue` then you **must** call
    :meth:`JoinableQueue.task_done` for each task removed from the queue or else the
    
    semaphore used to count the number of unfinished tasks may eventually overflow,
    
    Note that one can also create a shared queue by using a manager object -- see
    :ref:`multiprocessing-managers`.
    
    
       :mod:`multiprocessing` uses the usual :exc:`queue.Empty` and
       :exc:`queue.Full` exceptions to signal a timeout.  They are not available in
    
       the :mod:`multiprocessing` namespace so you need to import them from
    
    .. note::
    
       When an object is put on a queue, the object is pickled and a
       background thread later flushes the pickled data to an underlying
       pipe.  This has some consequences which are a little surprising,
    
       but should not cause any practical difficulties -- if they really
       bother you then you can instead use a queue created with a
       :ref:`manager <multiprocessing-managers>`.
    
    
       (1) After putting an object on an empty queue there may be an
    
    Richard Oudkerk's avatar
    Richard Oudkerk committed
           infinitesimal delay before the queue's :meth:`~Queue.empty`
    
           method returns :const:`False` and :meth:`~Queue.get_nowait` can
    
           return without raising :exc:`queue.Empty`.
    
    
       (2) If multiple processes are enqueuing objects, it is possible for
           the objects to be received at the other end out-of-order.
           However, objects enqueued by the same process will always be in
           the expected order with respect to each other.
    
    
    .. warning::
    
       If a process is killed using :meth:`Process.terminate` or :func:`os.kill`
       while it is trying to use a :class:`Queue`, then the data in the queue is
    
       likely to become corrupted.  This may cause any other process to get an
    
       exception when it tries to use the queue later on.
    
    .. warning::
    
       As mentioned above, if a child process has put items on a queue (and it has
    
       not used :meth:`JoinableQueue.cancel_join_thread
       <multiprocessing.Queue.cancel_join_thread>`), then that process will
    
       not terminate until all buffered items have been flushed to the pipe.
    
       This means that if you try joining that process you may get a deadlock unless
       you are sure that all items which have been put on the queue have been
       consumed.  Similarly, if the child process is non-daemonic then the parent
    
    Georg Brandl's avatar
    Georg Brandl committed
       process may hang on exit when it tries to join all its non-daemonic children.
    
    
       Note that a queue created using a manager does not have this issue.  See
       :ref:`multiprocessing-programming`.
    
    For an example of the usage of queues for interprocess communication see
    :ref:`multiprocessing-examples`.
    
    
    .. function:: Pipe([duplex])
    
    
       Returns a pair ``(conn1, conn2)`` of
       :class:`~multiprocessing.connection.Connection` objects representing the
       ends of a pipe.
    
    
       If *duplex* is ``True`` (the default) then the pipe is bidirectional.  If
       *duplex* is ``False`` then the pipe is unidirectional: ``conn1`` can only be
       used for receiving messages and ``conn2`` can only be used for sending
       messages.
    
    
    .. class:: Queue([maxsize])
    
       Returns a process shared queue implemented using a pipe and a few
       locks/semaphores.  When a process first puts an item on the queue a feeder
       thread is started which transfers objects from a buffer into the pipe.
    
    
       The usual :exc:`queue.Empty` and :exc:`queue.Full` exceptions from the
    
       standard library's :mod:`queue` module are raised to signal timeouts.
    
       :class:`Queue` implements all the methods of :class:`queue.Queue` except for
       :meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join`.
    
    
       .. method:: qsize()
    
          Return the approximate size of the queue.  Because of
          multithreading/multiprocessing semantics, this number is not reliable.
    
          Note that this may raise :exc:`NotImplementedError` on Unix platforms like
    
          macOS where ``sem_getvalue()`` is not implemented.
    
    
       .. method:: empty()
    
          Return ``True`` if the queue is empty, ``False`` otherwise.  Because of
          multithreading/multiprocessing semantics, this is not reliable.
    
       .. method:: full()
    
          Return ``True`` if the queue is full, ``False`` otherwise.  Because of
          multithreading/multiprocessing semantics, this is not reliable.
    
    
          Put obj into the queue.  If the optional argument *block* is ``True``
    
          (the default) and *timeout* is ``None`` (the default), block if necessary until
    
          a free slot is available.  If *timeout* is a positive number, it blocks at
    
          most *timeout* seconds and raises the :exc:`queue.Full` exception if no
    
          free slot was available within that time.  Otherwise (*block* is
          ``False``), put an item on the queue if a free slot is immediately
    
          available, else raise the :exc:`queue.Full` exception (*timeout* is
    
          .. versionchanged:: 3.8
             If the queue is closed, :exc:`ValueError` is raised instead of
             :exc:`AssertionError`.
    
    
    
       .. method:: get([block[, timeout]])
    
          Remove and return an item from the queue.  If optional args *block* is
          ``True`` (the default) and *timeout* is ``None`` (the default), block if
          necessary until an item is available.  If *timeout* is a positive number,
    
          it blocks at most *timeout* seconds and raises the :exc:`queue.Empty`
    
          exception if no item was available within that time.  Otherwise (block is
          ``False``), return an item if one is immediately available, else raise the
    
          :exc:`queue.Empty` exception (*timeout* is ignored in that case).
    
          .. versionchanged:: 3.8
             If the queue is closed, :exc:`ValueError` is raised instead of
             :exc:`OSError`.
    
    
       .. method:: get_nowait()
    
          Equivalent to ``get(False)``.
    
       :class:`multiprocessing.Queue` has a few additional methods not found in
    
    Georg Brandl's avatar
    Georg Brandl committed
       :class:`queue.Queue`.  These methods are usually unnecessary for most
       code:
    
    
       .. method:: close()
    
          Indicate that no more data will be put on this queue by the current
          process.  The background thread will quit once it has flushed all buffered
          data to the pipe.  This is called automatically when the queue is garbage
          collected.
    
       .. method:: join_thread()
    
          Join the background thread.  This can only be used after :meth:`close` has
          been called.  It blocks until the background thread exits, ensuring that
          all data in the buffer has been flushed to the pipe.
    
          By default if a process is not the creator of the queue then on exit it
          will attempt to join the queue's background thread.  The process can call
    
          :meth:`cancel_join_thread` to make :meth:`join_thread` do nothing.
    
    
       .. method:: cancel_join_thread()
    
          Prevent :meth:`join_thread` from blocking.  In particular, this prevents
          the background thread from being joined automatically when the process
    
          exits -- see :meth:`join_thread`.
    
          A better name for this method might be
          ``allow_exit_without_flush()``.  It is likely to cause enqueued
    
    Mason's avatar
    Mason committed
          data to be lost, and you almost certainly will not need to use it.
    
          It is really only there if you need the current process to exit
          immediately without waiting to flush enqueued data to the
          underlying pipe, and you don't care about lost data.
    
    
       .. note::
    
          This class's functionality requires a functioning shared semaphore
          implementation on the host operating system. Without one, the
          functionality in this class will be disabled, and attempts to
          instantiate a :class:`Queue` will result in an :exc:`ImportError`. See
          :issue:`3770` for additional information.  The same holds true for any
          of the specialized queue types listed below.
    
    
       It is a simplified :class:`Queue` type, very close to a locked :class:`Pipe`.
    
    
       .. method:: close()
    
          Close the queue: release internal resources.
    
          A queue must not be used anymore after it is closed. For example,
          :meth:`get`, :meth:`put` and :meth:`empty` methods must no longer be
          called.
    
          .. versionadded:: 3.9
    
    
       .. method:: empty()
    
          Return ``True`` if the queue is empty, ``False`` otherwise.
    
       .. method:: get()
    
          Remove and return an item from the queue.
    
       .. method:: put(item)
    
          Put *item* into the queue.
    
    
    
    .. class:: JoinableQueue([maxsize])
    
       :class:`JoinableQueue`, a :class:`Queue` subclass, is a queue which
       additionally has :meth:`task_done` and :meth:`join` methods.
    
       .. method:: task_done()
    
    
          Indicate that a formerly enqueued task is complete. Used by queue
          consumers.  For each :meth:`~Queue.get` used to fetch a task, a subsequent
    
          call to :meth:`task_done` tells the queue that the processing on the task
          is complete.
    
          If a :meth:`~queue.Queue.join` is currently blocking, it will resume when all
    
          items have been processed (meaning that a :meth:`task_done` call was
          received for every item that had been :meth:`~Queue.put` into the queue).
    
    
          Raises a :exc:`ValueError` if called more times than there were items
          placed in the queue.
    
    
       .. method:: join()
    
          Block until all items in the queue have been gotten and processed.
    
          The count of unfinished tasks goes up whenever an item is added to the
    
          queue.  The count goes down whenever a consumer calls
    
          :meth:`task_done` to indicate that the item was retrieved and all work on
          it is complete.  When the count of unfinished tasks drops to zero,
    
          :meth:`~queue.Queue.join` unblocks.
    
    
    
    Miscellaneous
    ~~~~~~~~~~~~~
    
    .. function:: active_children()
    
       Return list of all live children of the current process.
    
    
       Calling this has the side effect of "joining" any processes which have
    
       already finished.
    
    .. function:: cpu_count()
    
    
       Return the number of CPUs in the system.
    
       This number is not equivalent to the number of CPUs the current process can
       use.  The number of usable CPUs can be obtained with
       ``len(os.sched_getaffinity(0))``
    
    
       When the number of CPUs cannot be determined a :exc:`NotImplementedError`
       is raised.
    
    .. function:: current_process()
    
       Return the :class:`Process` object corresponding to the current process.
    
       An analogue of :func:`threading.current_thread`.
    
    
    .. function:: parent_process()
    
       Return the :class:`Process` object corresponding to the parent process of
       the :func:`current_process`. For the main process, ``parent_process`` will
       be ``None``.
    
       .. versionadded:: 3.8
    
    
    .. function:: freeze_support()
    
       Add support for when a program which uses :mod:`multiprocessing` has been
       frozen to produce a Windows executable.  (Has been tested with **py2exe**,
       **PyInstaller** and **cx_Freeze**.)