1 

     2 :mod:`subprocess` --- Subprocess management

     3 ===========================================

     4 

     5 .. module:: subprocess

     6    :synopsis: Subprocess management.

     7 .. moduleauthor:: Peter Åstrand <astrand@lysator.liu.se>

     8 .. sectionauthor:: Peter Åstrand <astrand@lysator.liu.se>

     9 

    10 

    11 .. versionadded:: 2.4

    12 

    13 The :mod:`subprocess` module allows you to spawn new processes, connect to their

    14 input/output/error pipes, and obtain their return codes.  This module intends to

    15 replace several other, older modules and functions, such as::

    16 

    17    os.system

    18    os.spawn*

    19    os.popen*

    20    popen2.*

    21    commands.*

    22 

    23 Information about how the :mod:`subprocess` module can be used to replace these

    24 modules and functions can be found in the following sections.

    25 

    26 .. seealso::

    27 

    28    :pep:`324` -- PEP proposing the subprocess module

    29 

    30 

    31 Using the subprocess Module

    32 ---------------------------

    33 

    34 This module defines one class called :class:`Popen`:

    35 

    36 

    37 .. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)

    38 

    39    Arguments are:

    40 

    41    *args* should be a string, or a sequence of program arguments.  The program

    42    to execute is normally the first item in the args sequence or the string if a

    43    string is given, but can be explicitly set by using the *executable*

    44    argument.

    45 

    46    On Unix, with *shell=False* (default): In this case, the Popen class uses

    47    :meth:`os.execvp` to execute the child program. *args* should normally be a

    48    sequence.  A string will be treated as a sequence with the string as the only

    49    item (the program to execute).

    50 

    51    On Unix, with *shell=True*: If args is a string, it specifies the command string

    52    to execute through the shell.  If *args* is a sequence, the first item specifies

    53    the command string, and any additional items will be treated as additional shell

    54    arguments.

    55 

    56    On Windows: the :class:`Popen` class uses CreateProcess() to execute the child

    57    program, which operates on strings.  If *args* is a sequence, it will be

    58    converted to a string using the :meth:`list2cmdline` method.  Please note that

    59    not all MS Windows applications interpret the command line the same way:

    60    :meth:`list2cmdline` is designed for applications using the same rules as the MS

    61    C runtime.

    62 

    63    *bufsize*, if given, has the same meaning as the corresponding argument to the

    64    built-in open() function: :const:`0` means unbuffered, :const:`1` means line

    65    buffered, any other positive value means use a buffer of (approximately) that

    66    size.  A negative *bufsize* means to use the system default, which usually means

    67    fully buffered.  The default value for *bufsize* is :const:`0` (unbuffered).

    68 

    69    The *executable* argument specifies the program to execute. It is very seldom

    70    needed: Usually, the program to execute is defined by the *args* argument. If

    71    ``shell=True``, the *executable* argument specifies which shell to use. On Unix,

    72    the default shell is :file:`/bin/sh`.  On Windows, the default shell is

    73    specified by the :envvar:`COMSPEC` environment variable.

    74 

    75    *stdin*, *stdout* and *stderr* specify the executed programs' standard input,

    76    standard output and standard error file handles, respectively.  Valid values are

    77    ``PIPE``, an existing file descriptor (a positive integer), an existing file

    78    object, and ``None``.  ``PIPE`` indicates that a new pipe to the child should be

    79    created.  With ``None``, no redirection will occur; the child's file handles

    80    will be inherited from the parent.  Additionally, *stderr* can be ``STDOUT``,

    81    which indicates that the stderr data from the applications should be captured

    82    into the same file handle as for stdout.

    83 

    84    If *preexec_fn* is set to a callable object, this object will be called in the

    85    child process just before the child is executed. (Unix only)

    86 

    87    If *close_fds* is true, all file descriptors except :const:`0`, :const:`1` and

    88    :const:`2` will be closed before the child process is executed. (Unix only).

    89    Or, on Windows, if *close_fds* is true then no handles will be inherited by the

    90    child process.  Note that on Windows, you cannot set *close_fds* to true and

    91    also redirect the standard handles by setting *stdin*, *stdout* or *stderr*.

    92 

    93    If *shell* is :const:`True`, the specified command will be executed through the

    94    shell.

    95 

    96    If *cwd* is not ``None``, the child's current directory will be changed to *cwd*

    97    before it is executed.  Note that this directory is not considered when

    98    searching the executable, so you can't specify the program's path relative to

    99    *cwd*.

   100 

   101    If *env* is not ``None``, it must be a mapping that defines the environment

   102    variables for the new process; these are used instead of inheriting the current

   103    process' environment, which is the default behavior.

   104 

   105    If *universal_newlines* is :const:`True`, the file objects stdout and stderr are

   106    opened as text files, but lines may be terminated by any of ``'\n'``, the Unix

   107    end-of-line convention, ``'\r'``, the old Macintosh convention or ``'\r\n'``, the

   108    Windows convention. All of these external representations are seen as ``'\n'``

   109    by the Python program.

   110 

   111    .. note::

   112 

   113       This feature is only available if Python is built with universal newline support

   114       (the default).  Also, the newlines attribute of the file objects :attr:`stdout`,

   115       :attr:`stdin` and :attr:`stderr` are not updated by the communicate() method.

   116 

   117    The *startupinfo* and *creationflags*, if given, will be passed to the

   118    underlying CreateProcess() function.  They can specify things such as appearance

   119    of the main window and priority for the new process.  (Windows only)

   120 

   121 

   122 Convenience Functions

   123 ^^^^^^^^^^^^^^^^^^^^^

   124 

   125 This module also defines two shortcut functions:

   126 

   127 

   128 .. function:: call(*popenargs, **kwargs)

   129 

   130    Run command with arguments.  Wait for command to complete, then return the

   131    :attr:`returncode` attribute.

   132 

   133    The arguments are the same as for the Popen constructor.  Example::

   134 

   135       retcode = call(["ls", "-l"])

   136 

   137 

   138 .. function:: check_call(*popenargs, **kwargs)

   139 

   140    Run command with arguments.  Wait for command to complete. If the exit code was

   141    zero then return, otherwise raise :exc:`CalledProcessError`. The

   142    :exc:`CalledProcessError` object will have the return code in the

   143    :attr:`returncode` attribute.

   144 

   145    The arguments are the same as for the Popen constructor.  Example::

   146 

   147       check_call(["ls", "-l"])

   148 

   149    .. versionadded:: 2.5

   150 

   151 

   152 Exceptions

   153 ^^^^^^^^^^

   154 

   155 Exceptions raised in the child process, before the new program has started to

   156 execute, will be re-raised in the parent.  Additionally, the exception object

   157 will have one extra attribute called :attr:`child_traceback`, which is a string

   158 containing traceback information from the childs point of view.

   159 

   160 The most common exception raised is :exc:`OSError`.  This occurs, for example,

   161 when trying to execute a non-existent file.  Applications should prepare for

   162 :exc:`OSError` exceptions.

   163 

   164 A :exc:`ValueError` will be raised if :class:`Popen` is called with invalid

   165 arguments.

   166 

   167 check_call() will raise :exc:`CalledProcessError`, if the called process returns

   168 a non-zero return code.

   169 

   170 

   171 Security

   172 ^^^^^^^^

   173 

   174 Unlike some other popen functions, this implementation will never call /bin/sh

   175 implicitly.  This means that all characters, including shell metacharacters, can

   176 safely be passed to child processes.

   177 

   178 

   179 Popen Objects

   180 -------------

   181 

   182 Instances of the :class:`Popen` class have the following methods:

   183 

   184 

   185 .. method:: Popen.poll()

   186 

   187    Check if child process has terminated.  Set and return :attr:`returncode`

   188    attribute.

   189 

   190 

   191 .. method:: Popen.wait()

   192 

   193    Wait for child process to terminate.  Set and return :attr:`returncode`

   194    attribute.

   195 

   196    .. warning::

   197 

   198       This will deadlock if the child process generates enough output to a

   199       stdout or stderr pipe such that it blocks waiting for the OS pipe buffer

   200       to accept more data.  Use :meth:`communicate` to avoid that.

   201 

   202 

   203 .. method:: Popen.communicate(input=None)

   204 

   205    Interact with process: Send data to stdin.  Read data from stdout and stderr,

   206    until end-of-file is reached.  Wait for process to terminate. The optional

   207    *input* argument should be a string to be sent to the child process, or

   208    ``None``, if no data should be sent to the child.

   209 

   210    :meth:`communicate` returns a tuple ``(stdout, stderr)``.

   211 

   212    Note that if you want to send data to the process's stdin, you need to create

   213    the Popen object with ``stdin=PIPE``.  Similarly, to get anything other than

   214    ``None`` in the result tuple, you need to give ``stdout=PIPE`` and/or

   215    ``stderr=PIPE`` too.

   216 

   217    .. note::

   218 

   219       The data read is buffered in memory, so do not use this method if the data

   220       size is large or unlimited.

   221 

   222 

   223 .. method:: Popen.send_signal(signal)

   224 

   225    Sends the signal *signal* to the child.

   226 

   227    .. note::

   228 

   229       On Windows only SIGTERM is supported so far. It's an alias for

   230       :meth:`terminate`.

   231 

   232    .. versionadded:: 2.6

   233 

   234 

   235 .. method:: Popen.terminate()

   236 

   237    Stop the child. On Posix OSs the method sends SIGTERM to the

   238    child. On Windows the Win32 API function :cfunc:`TerminateProcess` is called

   239    to stop the child.

   240 

   241    .. versionadded:: 2.6

   242 

   243 

   244 .. method:: Popen.kill()

   245 

   246    Kills the child. On Posix OSs the function sends SIGKILL to the child.

   247    On Windows :meth:`kill` is an alias for :meth:`terminate`.

   248 

   249    .. versionadded:: 2.6

   250 

   251 

   252 The following attributes are also available:

   253 

   254 .. warning::

   255 

   256    Use :meth:`communicate` rather than :meth:`.stdin.write`,

   257    :meth:`.stdout.read` or :meth:`.stderr.read` to avoid deadlocks due

   258    to any of the other OS pipe buffers filling up and blocking the child

   259    process.

   260 

   261 

   262 .. attribute:: Popen.stdin

   263 

   264    If the *stdin* argument is ``PIPE``, this attribute is a file object that

   265    provides input to the child process.  Otherwise, it is ``None``.

   266 

   267 

   268 .. attribute:: Popen.stdout

   269 

   270    If the *stdout* argument is ``PIPE``, this attribute is a file object that

   271    provides output from the child process.  Otherwise, it is ``None``.

   272 

   273 

   274 .. attribute:: Popen.stderr

   275 

   276    If the *stderr* argument is ``PIPE``, this attribute is file object that

   277    provides error output from the child process.  Otherwise, it is ``None``.

   278 

   279 

   280 .. attribute:: Popen.pid

   281 

   282    The process ID of the child process.

   283 

   284 

   285 .. attribute:: Popen.returncode

   286 

   287    The child return code, set by :meth:`poll` and :meth:`wait` (and indirectly

   288    by :meth:`communicate`).  A ``None`` value indicates that the process

   289    hasn't terminated yet.

   290    

   291    A negative value ``-N`` indicates that the child was terminated by signal

   292    ``N`` (Unix only).

   293 

   294 

   295 .. _subprocess-replacements:

   296 

   297 Replacing Older Functions with the subprocess Module

   298 ----------------------------------------------------

   299 

   300 In this section, "a ==> b" means that b can be used as a replacement for a.

   301 

   302 .. note::

   303 

   304    All functions in this section fail (more or less) silently if the executed

   305    program cannot be found; this module raises an :exc:`OSError` exception.

   306 

   307 In the following examples, we assume that the subprocess module is imported with

   308 "from subprocess import \*".

   309 

   310 

   311 Replacing /bin/sh shell backquote

   312 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   313 

   314 ::

   315 

   316    output=`mycmd myarg`

   317    ==>

   318    output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

   319 

   320 

   321 Replacing shell pipeline

   322 ^^^^^^^^^^^^^^^^^^^^^^^^

   323 

   324 ::

   325 

   326    output=`dmesg | grep hda`

   327    ==>

   328    p1 = Popen(["dmesg"], stdout=PIPE)

   329    p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)

   330    output = p2.communicate()[0]

   331 

   332 

   333 Replacing os.system()

   334 ^^^^^^^^^^^^^^^^^^^^^

   335 

   336 ::

   337 

   338    sts = os.system("mycmd" + " myarg")

   339    ==>

   340    p = Popen("mycmd" + " myarg", shell=True)

   341    sts = os.waitpid(p.pid, 0)

   342 

   343 Notes:

   344 

   345 * Calling the program through the shell is usually not required.

   346 

   347 * It's easier to look at the :attr:`returncode` attribute than the exit status.

   348 

   349 A more realistic example would look like this::

   350 

   351    try:

   352        retcode = call("mycmd" + " myarg", shell=True)

   353        if retcode < 0:

   354            print >>sys.stderr, "Child was terminated by signal", -retcode

   355        else:

   356            print >>sys.stderr, "Child returned", retcode

   357    except OSError, e:

   358        print >>sys.stderr, "Execution failed:", e

   359 

   360 

   361 Replacing os.spawn\*

   362 ^^^^^^^^^^^^^^^^^^^^

   363 

   364 P_NOWAIT example::

   365 

   366    pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")

   367    ==>

   368    pid = Popen(["/bin/mycmd", "myarg"]).pid

   369 

   370 P_WAIT example::

   371 

   372    retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")

   373    ==>

   374    retcode = call(["/bin/mycmd", "myarg"])

   375 

   376 Vector example::

   377 

   378    os.spawnvp(os.P_NOWAIT, path, args)

   379    ==>

   380    Popen([path] + args[1:])

   381 

   382 Environment example::

   383 

   384    os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)

   385    ==>

   386    Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})

   387 

   388 

   389 Replacing os.popen\*

   390 ^^^^^^^^^^^^^^^^^^^^

   391 

   392 ::

   393 

   394    pipe = os.popen(cmd, 'r', bufsize)

   395    ==>

   396    pipe = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE).stdout

   397 

   398 ::

   399 

   400    pipe = os.popen(cmd, 'w', bufsize)

   401    ==>

   402    pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin

   403 

   404 ::

   405 

   406    (child_stdin, child_stdout) = os.popen2(cmd, mode, bufsize)

   407    ==>

   408    p = Popen(cmd, shell=True, bufsize=bufsize,

   409              stdin=PIPE, stdout=PIPE, close_fds=True)

   410    (child_stdin, child_stdout) = (p.stdin, p.stdout)

   411 

   412 ::

   413 

   414    (child_stdin,

   415     child_stdout,

   416     child_stderr) = os.popen3(cmd, mode, bufsize)

   417    ==>

   418    p = Popen(cmd, shell=True, bufsize=bufsize,

   419              stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)

   420    (child_stdin,

   421     child_stdout,

   422     child_stderr) = (p.stdin, p.stdout, p.stderr)

   423 

   424 ::

   425 

   426    (child_stdin, child_stdout_and_stderr) = os.popen4(cmd, mode, bufsize)

   427    ==>

   428    p = Popen(cmd, shell=True, bufsize=bufsize,

   429              stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)

   430    (child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)

   431 

   432 

   433 Replacing popen2.\*

   434 ^^^^^^^^^^^^^^^^^^^

   435 

   436 .. note::

   437 

   438    If the cmd argument to popen2 functions is a string, the command is executed

   439    through /bin/sh.  If it is a list, the command is directly executed.

   440 

   441 ::

   442 

   443    (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)

   444    ==>

   445    p = Popen(["somestring"], shell=True, bufsize=bufsize,

   446              stdin=PIPE, stdout=PIPE, close_fds=True)

   447    (child_stdout, child_stdin) = (p.stdout, p.stdin)

   448 

   449 ::

   450 

   451    (child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize, mode)

   452    ==>

   453    p = Popen(["mycmd", "myarg"], bufsize=bufsize,

   454              stdin=PIPE, stdout=PIPE, close_fds=True)

   455    (child_stdout, child_stdin) = (p.stdout, p.stdin)

   456 

   457 The popen2.Popen3 and popen2.Popen4 basically works as subprocess.Popen, except

   458 that:

   459 

   460 * subprocess.Popen raises an exception if the execution fails

   461 

   462 * the *capturestderr* argument is replaced with the *stderr* argument.

   463 

   464 * stdin=PIPE and stdout=PIPE must be specified.

   465 

   466 * popen2 closes all file descriptors by default, but you have to specify

   467   close_fds=True with subprocess.Popen.

   468