Newer
Older
:mod:`multiprocessing` --- Process-based parallelism
====================================================
.. module:: multiprocessing
:synopsis: Process-based parallelism.
Terry Jan Reedy
committed
**Source code:** :source:`Lib/multiprocessing/`
--------------
:mod:`multiprocessing` is a package that supports spawning processes using an
API similar to the :mod:`threading` module. The :mod:`multiprocessing` package
offers both local and remote concurrency, effectively side-stepping the
:term:`Global Interpreter Lock <global interpreter lock>` by using
subprocesses instead of threads. Due
to this, the :mod:`multiprocessing` module allows the programmer to fully
leverage multiple processors on a given machine. It runs on both Unix and
Windows.
Antoine Pitrou
committed
The :mod:`multiprocessing` module also introduces APIs which do not have
analogs in the :mod:`threading` module. A prime example of this is the
:class:`~multiprocessing.pool.Pool` object which offers a convenient means of
parallelizing the execution of a function across multiple input values,
distributing the input data across processes (data parallelism). The following
example demonstrates the common practice of defining such functions in a module
so that child processes can successfully import that module. This basic example
of data parallelism using :class:`~multiprocessing.pool.Pool`, ::
Antoine Pitrou
committed
from multiprocessing import Pool
Antoine Pitrou
committed
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
Antoine Pitrou
committed
will print to standard output ::
[1, 4, 9]
The :class:`Process` class
~~~~~~~~~~~~~~~~~~~~~~~~~~
In :mod:`multiprocessing`, processes are spawned by creating a :class:`Process`
object and then calling its :meth:`~Process.start` method. :class:`Process`
follows the API of :class:`threading.Thread`. A trivial example of a
multiprocess program is ::
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
To show the individual process IDs involved, here is an expanded example::
from multiprocessing import Process
import os
def info(title):
Ezio Melotti
committed
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
Ezio Melotti
committed
print('process id:', os.getpid())
Ezio Melotti
committed
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
Richard Oudkerk
committed
For an explanation of why the ``if __name__ == '__main__'`` part is
necessary, see :ref:`multiprocessing-programming`.
Contexts and start methods
~~~~~~~~~~~~~~~~~~~~~~~~~~
Richard Oudkerk
committed
.. _multiprocessing-start-methods:
Richard Oudkerk
committed
Depending on the platform, :mod:`multiprocessing` supports three ways
to start a process. These *start methods* are
*spawn*
The parent process starts a fresh Python interpreter process. The
Richard Oudkerk
committed
child process will only inherit those resources necessary to run
the process object's :meth:`~Process.run` method. In particular,
Richard Oudkerk
committed
unnecessary file descriptors and handles from the parent process
will not be inherited. Starting a process using this method is
rather slow compared to using *fork* or *forkserver*.
Available on Unix and Windows. The default on Windows and macOS.
Richard Oudkerk
committed
*fork*
The parent process uses :func:`os.fork` to fork the Python
interpreter. The child process, when it begins, is effectively
identical to the parent process. All resources of the parent are
inherited by the child process. Note that safely forking a
multithreaded process is problematic.
Available on Unix only. The default on Unix.
*forkserver*
When the program starts and selects the *forkserver* start method,
a server process is started. From then on, whenever a new process
Georg Brandl
committed
is needed, the parent process connects to the server and requests
Richard Oudkerk
committed
that it fork a new process. The fork server process is single
threaded so it is safe for it to use :func:`os.fork`. No
unnecessary resources are inherited.
Available on Unix platforms which support passing file descriptors
over Unix pipes.
Richard Oudkerk
committed
.. versionchanged:: 3.8
Victor Stinner
committed
On macOS, the *spawn* start method is now the default. The *fork* start
method should be considered unsafe as it can lead to crashes of the
subprocess. See :issue:`33725`.
.. versionchanged:: 3.4
*spawn* added on all unix platforms, and *forkserver* added for
some unix platforms.
Child processes no longer inherit all of the parents inheritable
handles on Windows.
Richard Oudkerk
committed
On Unix using the *spawn* or *forkserver* start methods will also
start a *resource tracker* process which tracks the unlinked named
system resources (such as named semaphores or
:class:`~multiprocessing.shared_memory.SharedMemory` objects) created
by processes of the program. When all processes
have exited the resource tracker unlinks any remaining tracked object.
Richard Oudkerk
committed
Usually there should be none, but if a process was killed by a signal
there may be some "leaked" resources. (Neither leaked semaphores nor shared
memory segments will be automatically unlinked until the next reboot. This is
problematic for both objects because the system allows only a limited number of
named semaphores, and shared memory segments occupy some space in the main
memory.)
Richard Oudkerk
committed
To select a start method you use the :func:`set_start_method` in
Richard Oudkerk
committed
the ``if __name__ == '__main__'`` clause of the main module. For
example::
import multiprocessing as mp
def foo(q):
q.put('hello')
Richard Oudkerk
committed
if __name__ == '__main__':
mp.set_start_method('spawn')
q = mp.Queue()
p = mp.Process(target=foo, args=(q,))
Richard Oudkerk
committed
p.start()
print(q.get())
Richard Oudkerk
committed
p.join()
:func:`set_start_method` should not be used more than once in the
program.
Alternatively, you can use :func:`get_context` to obtain a context
object. Context objects have the same API as the multiprocessing
module, and allow one to use multiple start methods in the same
program. ::
import multiprocessing as mp
def foo(q):
q.put('hello')
if __name__ == '__main__':
ctx = mp.get_context('spawn')
q = ctx.Queue()
p = ctx.Process(target=foo, args=(q,))
p.start()
print(q.get())
p.join()
Note that objects related to one context may not be compatible with
processes for a different context. In particular, locks created using
the *fork* context cannot be passed to processes started using the
*spawn* or *forkserver* start methods.
A library which wants to use a particular start method should probably
use :func:`get_context` to avoid interfering with the choice of the
library user.
Richard Oudkerk
committed
.. warning::
The ``'spawn'`` and ``'forkserver'`` start methods cannot currently
be used with "frozen" executables (i.e., binaries produced by
packages like **PyInstaller** and **cx_Freeze**) on Unix.
The ``'fork'`` start method does work.
Richard Oudkerk
committed
Exchanging objects between processes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:mod:`multiprocessing` supports two types of communication channel between
processes:
**Queues**
The :class:`Queue` class is a near clone of :class:`queue.Queue`. For
example::
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
Georg Brandl
committed
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print(q.get()) # prints "[42, None, 'hello']"
p.join()
Queues are thread and process safe.
**Pipes**
The :func:`Pipe` function returns a pair of connection objects connected by a
pipe which by default is duplex (two-way). For example::
from multiprocessing import Process, Pipe
def f(conn):
conn.send([42, None, 'hello'])
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
print(parent_conn.recv()) # prints "[42, None, 'hello']"
p.join()
The two connection objects returned by :func:`Pipe` represent the two ends of
the pipe. Each connection object has :meth:`~Connection.send` and
:meth:`~Connection.recv` methods (among others). Note that data in a pipe
may become corrupted if two processes (or threads) try to read from or write
to the *same* end of the pipe at the same time. Of course there is no risk
of corruption from processes using different ends of the pipe at the same
time.
Synchronization between processes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:mod:`multiprocessing` contains equivalents of all the synchronization
primitives from :mod:`threading`. For instance one can use a lock to ensure
that only one process prints to standard output at a time::
from multiprocessing import Process, Lock
def f(l, i):
l.acquire()
try:
print('hello world', i)
finally:
l.release()
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
if __name__ == '__main__':
lock = Lock()
for num in range(10):
Process(target=f, args=(lock, num)).start()
Without using the lock output from the different processes is liable to get all
mixed up.
Sharing state between processes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As mentioned above, when doing concurrent programming it is usually best to
avoid using shared state as far as possible. This is particularly true when
using multiple processes.
However, if you really do need to use some shared data then
:mod:`multiprocessing` provides a couple of ways of doing so.
**Shared memory**
Data can be stored in a shared memory map using :class:`Value` or
:class:`Array`. For example, the following code ::
from multiprocessing import Process, Value, Array
def f(n, a):
n.value = 3.1415927
for i in range(len(a)):
a[i] = -a[i]
if __name__ == '__main__':
num = Value('d', 0.0)
arr = Array('i', range(10))
p = Process(target=f, args=(num, arr))
p.start()
p.join()
print(num.value)
print(arr[:])
will print ::
3.1415927
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
The ``'d'`` and ``'i'`` arguments used when creating ``num`` and ``arr`` are
typecodes of the kind used by the :mod:`array` module: ``'d'`` indicates a
double precision float and ``'i'`` indicates a signed integer. These shared
objects will be process and thread-safe.
For more flexibility in using shared memory one can use the
:mod:`multiprocessing.sharedctypes` module which supports the creation of
arbitrary ctypes objects allocated from shared memory.
**Server process**
A manager object returned by :func:`Manager` controls a server process which
holds Python objects and allows other processes to manipulate them using
A manager returned by :func:`Manager` will support types
Senthil Kumaran
committed
:class:`list`, :class:`dict`, :class:`~managers.Namespace`, :class:`Lock`,
:class:`RLock`, :class:`Semaphore`, :class:`BoundedSemaphore`,
:class:`Condition`, :class:`Event`, :class:`Barrier`,
:class:`Queue`, :class:`Value` and :class:`Array`. For example, ::
from multiprocessing import Process, Manager
def f(d, l):
d[1] = '1'
d['2'] = 2
d[0.25] = None
l.reverse()
if __name__ == '__main__':
with Manager() as manager:
d = manager.dict()
l = manager.list(range(10))
p = Process(target=f, args=(d, l))
p.start()
p.join()
print(d)
print(l)
will print ::
{0.25: None, 1: '1', '2': 2}
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
Server process managers are more flexible than using shared memory objects
because they can be made to support arbitrary object types. Also, a single
manager can be shared by processes on different computers over a network.
They are, however, slower than using shared memory.
Using a pool of workers
~~~~~~~~~~~~~~~~~~~~~~~
The :class:`~multiprocessing.pool.Pool` class represents a pool of worker
processes. It has methods which allows tasks to be offloaded to the worker
processes in a few different ways.
For example::
from multiprocessing import Pool, TimeoutError
import time
import os
def f(x):
return x*x
if __name__ == '__main__':
Richard Oudkerk
committed
# start 4 worker processes
with Pool(processes=4) as pool:
# print "[0, 1, 4,..., 81]"
print(pool.map(f, range(10)))
# print same numbers in arbitrary order
for i in pool.imap_unordered(f, range(10)):
print(i)
# evaluate "f(20)" asynchronously
res = pool.apply_async(f, (20,)) # runs in *only* one process
print(res.get(timeout=1)) # prints "400"
# evaluate "os.getpid()" asynchronously
res = pool.apply_async(os.getpid, ()) # runs in *only* one process
print(res.get(timeout=1)) # prints the PID of that process
# launching multiple evaluations asynchronously *may* use more processes
multiple_results = [pool.apply_async(os.getpid, ()) for i in range(4)]
print([res.get(timeout=1) for res in multiple_results])
# make a single worker sleep for 10 seconds
res = pool.apply_async(time.sleep, (10,))
try:
print(res.get(timeout=1))
except TimeoutError:
print("We lacked patience and got a multiprocessing.TimeoutError")
Richard Oudkerk
committed
print("For the moment, the pool remains available for more work")
Richard Oudkerk
committed
# exiting the 'with'-block has stopped the pool
print("Now the pool is closed and no longer available")
Richard Oudkerk
committed
Note that the methods of a pool should only ever be used by the
process which created it.
Antoine Pitrou
committed
.. note::
Functionality within this package requires that the ``__main__`` module be
importable by the children. This is covered in :ref:`multiprocessing-programming`
however it is worth pointing out here. This means that some examples, such
as the :class:`multiprocessing.pool.Pool` examples will not work in the
interactive interpreter. For example::
>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
... return x*x
...
Pablo Galindo
committed
>>> with p:
... p.map(f, [1,2,3])
Antoine Pitrou
committed
Process PoolWorker-1:
Process PoolWorker-2:
Process PoolWorker-3:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
AttributeError: 'module' object has no attribute 'f'
AttributeError: 'module' object has no attribute 'f'
AttributeError: 'module' object has no attribute 'f'
(If you try this it will actually output three full tracebacks
interleaved in a semi-random fashion, and then you may have to
stop the parent process somehow.)
Antoine Pitrou
committed
Reference
---------
The :mod:`multiprocessing` package mostly replicates the API of the
:mod:`threading` module.
:class:`Process` and exceptions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. class:: Process(group=None, target=None, name=None, args=(), kwargs={}, \
*, daemon=None)
Process objects represent activity that is run in a separate process. The
:class:`Process` class has equivalents of all the methods of
:class:`threading.Thread`.
The constructor should always be called with keyword arguments. *group*
should always be ``None``; it exists solely for compatibility with
:class:`threading.Thread`. *target* is the callable object to be invoked by
the :meth:`run()` method. It defaults to ``None``, meaning nothing is
called. *name* is the process name (see :attr:`name` for more details).
*args* is the argument tuple for the target invocation. *kwargs* is a
dictionary of keyword arguments for the target invocation. If provided,
the keyword-only *daemon* argument sets the process :attr:`daemon` flag
to ``True`` or ``False``. If ``None`` (the default), this flag will be
inherited from the creating process.
By default, no arguments are passed to *target*. The *args* argument,
which defaults to ``()``, can be used to specify a list or tuple of the arguments
to pass to *target*.
If a subclass overrides the constructor, it must make sure it invokes the
base class constructor (:meth:`Process.__init__`) before doing anything else
to the process.
.. versionchanged:: 3.3
Added the *daemon* argument.
.. method:: run()
Method representing the process's activity.
You may override this method in a subclass. The standard :meth:`run`
method invokes the callable object passed to the object's constructor as
the target argument, if any, with sequential and keyword arguments taken
from the *args* and *kwargs* arguments, respectively.
Using a list or tuple as the *args* argument passed to :class:`Process`
achieves the same effect.
Example::
>>> from multiprocessing import Process
>>> p = Process(target=print, args=[1])
>>> p.run()
1
>>> p = Process(target=print, args=(1,))
>>> p.run()
1
.. method:: start()
Start the process's activity.
This must be called at most once per process object. It arranges for the
object's :meth:`run` method to be invoked in a separate process.
.. method:: join([timeout])
Charles-François Natali
committed
If the optional argument *timeout* is ``None`` (the default), the method
blocks until the process whose :meth:`join` method is called terminates.
If *timeout* is a positive number, it blocks at most *timeout* seconds.
Berker Peksag
committed
Note that the method returns ``None`` if its process terminates or if the
method times out. Check the process's :attr:`exitcode` to determine if
it terminated.
A process can be joined many times.
A process cannot join itself because this would cause a deadlock. It is
an error to attempt to join a process before it has been started.
The process's name. The name is a string used for identification purposes
only. It has no semantics. Multiple processes may be given the same
name.
The initial name is set by the constructor. If no explicit name is
provided to the constructor, a name of the form
'Process-N\ :sub:`1`:N\ :sub:`2`:...:N\ :sub:`k`' is constructed, where
each N\ :sub:`k` is the N-th child of its parent.
Return whether the process is alive.
Roughly, a process object is alive from the moment the :meth:`start`
method returns until the child process terminates.
The process's daemon flag, a Boolean value. This must be set before
The initial value is inherited from the creating process.
When a process exits, it attempts to terminate all of its daemonic child
processes.
Note that a daemonic process is not allowed to create child processes.
Otherwise a daemonic process would leave its children orphaned if it gets
Alexandre Vassalotti
committed
terminated when its parent process exits. Additionally, these are **not**
Unix daemons or services, they are normal processes that will be
terminated (and not joined) if non-daemonic processes have exited.
In addition to the :class:`threading.Thread` API, :class:`Process` objects
also support the following attributes and methods:
Return the process ID. Before the process is spawned, this will be
``None``.
The child's exit code. This will be ``None`` if the process has not yet
terminated.
If the child's :meth:`run` method returned normally, the exit code
will be 0. If it terminated via :func:`sys.exit` with an integer
argument *N*, the exit code will be *N*.
If the child terminated due to an exception not caught within
:meth:`run`, the exit code will be 1. If it was terminated by
signal *N*, the exit code will be the negative value *-N*.
The process's authentication key (a byte string).
When :mod:`multiprocessing` is initialized the main process is assigned a
random string using :func:`os.urandom`.
When a :class:`Process` object is created, it will inherit the
authentication key of its parent process, although this may be changed by
setting :attr:`authkey` to another byte string.
See :ref:`multiprocessing-auth-keys`.
.. attribute:: sentinel
A numeric handle of a system object which will become "ready" when
the process ends.
You can use this value if you want to wait on several events at
once using :func:`multiprocessing.connection.wait`. Otherwise
calling :meth:`join()` is simpler.
On Windows, this is an OS handle usable with the ``WaitForSingleObject``
and ``WaitForMultipleObjects`` family of API calls. On Unix, this is
a file descriptor usable with primitives from the :mod:`select` module.
.. versionadded:: 3.3
.. method:: terminate()
Terminate the process. On Unix this is done using the ``SIGTERM`` signal;
on Windows :c:func:`TerminateProcess` is used. Note that exit handlers and
finally clauses, etc., will not be executed.
Note that descendant processes of the process will *not* be terminated --
they will simply become orphaned.
.. warning::
If this method is used when the associated process is using a pipe or
queue then the pipe or queue is liable to become corrupted and may
become unusable by other process. Similarly, if the process has
acquired a lock or semaphore etc. then terminating it is liable to
cause other processes to deadlock.
.. method:: kill()
Same as :meth:`terminate()` but using the ``SIGKILL`` signal on Unix.
.. versionadded:: 3.7
.. method:: close()
Close the :class:`Process` object, releasing all resources associated
with it. :exc:`ValueError` is raised if the underlying process
is still running. Once :meth:`close` returns successfully, most
other methods and attributes of the :class:`Process` object will
raise :exc:`ValueError`.
.. versionadded:: 3.7
Note that the :meth:`start`, :meth:`join`, :meth:`is_alive`,
:meth:`terminate` and :attr:`exitcode` methods should only be called by
the process that created the process object.
Example usage of some of the methods of :class:`Process`:
.. doctest::
Stéphane Wirtel
committed
:options: +ELLIPSIS
>>> import multiprocessing, time, signal
>>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
<Process ... initial> False
<Process ... started> True
<Process ... stopped exitcode=-SIGTERM> False
.. exception:: ProcessError
The base class of all :mod:`multiprocessing` exceptions.
.. exception:: BufferTooShort
Exception raised by :meth:`Connection.recv_bytes_into()` when the supplied
buffer object is too small for the message read.
If ``e`` is an instance of :exc:`BufferTooShort` then ``e.args[0]`` will give
the message as a byte string.
.. exception:: AuthenticationError
Raised when there is an authentication error.
.. exception:: TimeoutError
Raised by methods with a timeout when the timeout expires.
Pipes and Queues
~~~~~~~~~~~~~~~~
When using multiple processes, one generally uses message passing for
communication between processes and avoids having to use any synchronization
primitives like locks.
For passing messages one can use :func:`Pipe` (for a connection between two
processes) or a queue (which allows multiple producers and consumers).
The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types
are multi-producer, multi-consumer :abbr:`FIFO (first-in, first-out)`
queues modelled on the :class:`queue.Queue` class in the
standard library. They differ in that :class:`Queue` lacks the
:meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced
into Python 2.5's :class:`queue.Queue` class.
If you use :class:`JoinableQueue` then you **must** call
:meth:`JoinableQueue.task_done` for each task removed from the queue or else the
semaphore used to count the number of unfinished tasks may eventually overflow,
raising an exception.
Note that one can also create a shared queue by using a manager object -- see
:ref:`multiprocessing-managers`.
:mod:`multiprocessing` uses the usual :exc:`queue.Empty` and
:exc:`queue.Full` exceptions to signal a timeout. They are not available in
the :mod:`multiprocessing` namespace so you need to import them from
:mod:`queue`.
.. note::
When an object is put on a queue, the object is pickled and a
background thread later flushes the pickled data to an underlying
pipe. This has some consequences which are a little surprising,
but should not cause any practical difficulties -- if they really
bother you then you can instead use a queue created with a
:ref:`manager <multiprocessing-managers>`.
(1) After putting an object on an empty queue there may be an
method returns :const:`False` and :meth:`~Queue.get_nowait` can
return without raising :exc:`queue.Empty`.
(2) If multiple processes are enqueuing objects, it is possible for
the objects to be received at the other end out-of-order.
However, objects enqueued by the same process will always be in
the expected order with respect to each other.
.. warning::
If a process is killed using :meth:`Process.terminate` or :func:`os.kill`
while it is trying to use a :class:`Queue`, then the data in the queue is
likely to become corrupted. This may cause any other process to get an
exception when it tries to use the queue later on.
.. warning::
As mentioned above, if a child process has put items on a queue (and it has
not used :meth:`JoinableQueue.cancel_join_thread
<multiprocessing.Queue.cancel_join_thread>`), then that process will
not terminate until all buffered items have been flushed to the pipe.
This means that if you try joining that process you may get a deadlock unless
you are sure that all items which have been put on the queue have been
consumed. Similarly, if the child process is non-daemonic then the parent
process may hang on exit when it tries to join all its non-daemonic children.
Note that a queue created using a manager does not have this issue. See
:ref:`multiprocessing-programming`.
For an example of the usage of queues for interprocess communication see
:ref:`multiprocessing-examples`.
.. function:: Pipe([duplex])
Bo Bayles
committed
Returns a pair ``(conn1, conn2)`` of
:class:`~multiprocessing.connection.Connection` objects representing the
ends of a pipe.
If *duplex* is ``True`` (the default) then the pipe is bidirectional. If
*duplex* is ``False`` then the pipe is unidirectional: ``conn1`` can only be
used for receiving messages and ``conn2`` can only be used for sending
messages.
.. class:: Queue([maxsize])
Returns a process shared queue implemented using a pipe and a few
locks/semaphores. When a process first puts an item on the queue a feeder
thread is started which transfers objects from a buffer into the pipe.
The usual :exc:`queue.Empty` and :exc:`queue.Full` exceptions from the
standard library's :mod:`queue` module are raised to signal timeouts.
:class:`Queue` implements all the methods of :class:`queue.Queue` except for
:meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join`.
.. method:: qsize()
Return the approximate size of the queue. Because of
multithreading/multiprocessing semantics, this number is not reliable.
Note that this may raise :exc:`NotImplementedError` on Unix platforms like
macOS where ``sem_getvalue()`` is not implemented.
.. method:: empty()
Return ``True`` if the queue is empty, ``False`` otherwise. Because of
multithreading/multiprocessing semantics, this is not reliable.
.. method:: full()
Return ``True`` if the queue is full, ``False`` otherwise. Because of
multithreading/multiprocessing semantics, this is not reliable.
Senthil Kumaran
committed
.. method:: put(obj[, block[, timeout]])
Senthil Kumaran
committed
Put obj into the queue. If the optional argument *block* is ``True``
(the default) and *timeout* is ``None`` (the default), block if necessary until
a free slot is available. If *timeout* is a positive number, it blocks at
most *timeout* seconds and raises the :exc:`queue.Full` exception if no
free slot was available within that time. Otherwise (*block* is
``False``), put an item on the queue if a free slot is immediately
available, else raise the :exc:`queue.Full` exception (*timeout* is
ignored in that case).
Zackery Spytz
committed
.. versionchanged:: 3.8
If the queue is closed, :exc:`ValueError` is raised instead of
:exc:`AssertionError`.
Senthil Kumaran
committed
.. method:: put_nowait(obj)
Senthil Kumaran
committed
Equivalent to ``put(obj, False)``.
.. method:: get([block[, timeout]])
Remove and return an item from the queue. If optional args *block* is
``True`` (the default) and *timeout* is ``None`` (the default), block if
necessary until an item is available. If *timeout* is a positive number,
it blocks at most *timeout* seconds and raises the :exc:`queue.Empty`
exception if no item was available within that time. Otherwise (block is
``False``), return an item if one is immediately available, else raise the
:exc:`queue.Empty` exception (*timeout* is ignored in that case).
Zackery Spytz
committed
.. versionchanged:: 3.8
If the queue is closed, :exc:`ValueError` is raised instead of
:exc:`OSError`.
.. method:: get_nowait()
Equivalent to ``get(False)``.
:class:`multiprocessing.Queue` has a few additional methods not found in
:class:`queue.Queue`. These methods are usually unnecessary for most
code:
.. method:: close()
Indicate that no more data will be put on this queue by the current
process. The background thread will quit once it has flushed all buffered
data to the pipe. This is called automatically when the queue is garbage
collected.
.. method:: join_thread()
Join the background thread. This can only be used after :meth:`close` has
been called. It blocks until the background thread exits, ensuring that
all data in the buffer has been flushed to the pipe.
By default if a process is not the creator of the queue then on exit it
will attempt to join the queue's background thread. The process can call
:meth:`cancel_join_thread` to make :meth:`join_thread` do nothing.
.. method:: cancel_join_thread()
Prevent :meth:`join_thread` from blocking. In particular, this prevents
the background thread from being joined automatically when the process
exits -- see :meth:`join_thread`.
A better name for this method might be
``allow_exit_without_flush()``. It is likely to cause enqueued
data to be lost, and you almost certainly will not need to use it.
It is really only there if you need the current process to exit
immediately without waiting to flush enqueued data to the
underlying pipe, and you don't care about lost data.
Berker Peksag
committed
.. note::
This class's functionality requires a functioning shared semaphore
implementation on the host operating system. Without one, the
functionality in this class will be disabled, and attempts to
instantiate a :class:`Queue` will result in an :exc:`ImportError`. See
:issue:`3770` for additional information. The same holds true for any
of the specialized queue types listed below.
.. class:: SimpleQueue()
It is a simplified :class:`Queue` type, very close to a locked :class:`Pipe`.
.. method:: close()
Close the queue: release internal resources.
A queue must not be used anymore after it is closed. For example,
:meth:`get`, :meth:`put` and :meth:`empty` methods must no longer be
called.
.. versionadded:: 3.9
.. method:: empty()
Return ``True`` if the queue is empty, ``False`` otherwise.
.. method:: get()
Remove and return an item from the queue.
.. method:: put(item)
Put *item* into the queue.
.. class:: JoinableQueue([maxsize])
:class:`JoinableQueue`, a :class:`Queue` subclass, is a queue which
additionally has :meth:`task_done` and :meth:`join` methods.
.. method:: task_done()
Indicate that a formerly enqueued task is complete. Used by queue
consumers. For each :meth:`~Queue.get` used to fetch a task, a subsequent
call to :meth:`task_done` tells the queue that the processing on the task
is complete.
If a :meth:`~queue.Queue.join` is currently blocking, it will resume when all
items have been processed (meaning that a :meth:`task_done` call was
received for every item that had been :meth:`~Queue.put` into the queue).
Raises a :exc:`ValueError` if called more times than there were items
placed in the queue.
.. method:: join()
Block until all items in the queue have been gotten and processed.
The count of unfinished tasks goes up whenever an item is added to the
queue. The count goes down whenever a consumer calls
:meth:`task_done` to indicate that the item was retrieved and all work on
it is complete. When the count of unfinished tasks drops to zero,
:meth:`~queue.Queue.join` unblocks.
Miscellaneous
~~~~~~~~~~~~~
.. function:: active_children()
Return list of all live children of the current process.
Calling this has the side effect of "joining" any processes which have
already finished.
.. function:: cpu_count()
Return the number of CPUs in the system.
This number is not equivalent to the number of CPUs the current process can
use. The number of usable CPUs can be obtained with
``len(os.sched_getaffinity(0))``
When the number of CPUs cannot be determined a :exc:`NotImplementedError`
is raised.
Charles-Francois Natali
committed
.. seealso::
:func:`os.cpu_count`
.. function:: current_process()
Return the :class:`Process` object corresponding to the current process.
An analogue of :func:`threading.current_thread`.
.. function:: parent_process()
Return the :class:`Process` object corresponding to the parent process of
the :func:`current_process`. For the main process, ``parent_process`` will
be ``None``.
.. versionadded:: 3.8
.. function:: freeze_support()
Add support for when a program which uses :mod:`multiprocessing` has been
frozen to produce a Windows executable. (Has been tested with **py2exe**,
**PyInstaller** and **cx_Freeze**.)