:mod:`doctest` --- Test interactive Python examples

===================================================

.. module:: doctest

   :synopsis: Test pieces of code within docstrings.

.. moduleauthor:: Tim Peters <tim@python.org>

.. sectionauthor:: Tim Peters <tim@python.org>

.. sectionauthor:: Moshe Zadka <moshez@debian.org>

.. sectionauthor:: Edward Loper <edloper@users.sourceforge.net>

The :mod:`doctest` module searches for pieces of text that look like interactive

Python sessions, and then executes those sessions to verify that they work

exactly as shown.  There are several common ways to use doctest:

* To check that a module's docstrings are up-to-date by verifying that all

  interactive examples still work as documented.

* To perform regression testing by verifying that interactive examples from a

  test file or a test object work as expected.

* To write tutorial documentation for a package, liberally illustrated with

  input-output examples.  Depending on whether the examples or the expository text

  are emphasized, this has the flavor of "literate testing" or "executable

  documentation".

Here's a complete but small example module::

   """

   This is the "example" module.

   The example module supplies one function, factorial().  For example,

   >>> factorial(5)

   120

   """

   def factorial(n):

       """Return the factorial of n, an exact integer >= 0.

       If the result is small enough to fit in an int, return an int.

       Else return a long.

       >>> [factorial(n) for n in range(6)]

       [1, 1, 2, 6, 24, 120]

       >>> [factorial(long(n)) for n in range(6)]

       [1, 1, 2, 6, 24, 120]

       >>> factorial(30)

       265252859812191058636308480000000L

       >>> factorial(30L)

       265252859812191058636308480000000L

       >>> factorial(-1)

       Traceback (most recent call last):

           ...

       ValueError: n must be >= 0

       Factorials of floats are OK, but the float must be an exact integer:

       >>> factorial(30.1)

       Traceback (most recent call last):

           ...

       ValueError: n must be exact integer

       >>> factorial(30.0)

       265252859812191058636308480000000L

       It must also not be ridiculously large:

       >>> factorial(1e100)

       Traceback (most recent call last):

           ...

       OverflowError: n too large

       """

       import math

       if not n >= 0:

           raise ValueError("n must be >= 0")

       if math.floor(n) != n:

           raise ValueError("n must be exact integer")

       if n+1 == n:  # catch a value like 1e300

           raise OverflowError("n too large")

       result = 1

       factor = 2

       while factor <= n:

           result *= factor

           factor += 1

       return result

   if __name__ == "__main__":

       import doctest

       doctest.testmod()

If you run :file:`example.py` directly from the command line, :mod:`doctest`

works its magic::

   $ python example.py

   $

There's no output!  That's normal, and it means all the examples worked.  Pass

:option:`-v` to the script, and :mod:`doctest` prints a detailed log of what

it's trying, and prints a summary at the end::

   $ python example.py -v

   Trying:

       factorial(5)

   Expecting:

       120

   ok

   Trying:

       [factorial(n) for n in range(6)]

   Expecting:

       [1, 1, 2, 6, 24, 120]

   ok

   Trying:

       [factorial(long(n)) for n in range(6)]

   Expecting:

       [1, 1, 2, 6, 24, 120]

   ok

And so on, eventually ending with::

   Trying:

       factorial(1e100)

   Expecting:

       Traceback (most recent call last):

           ...

       OverflowError: n too large

   ok

   2 items passed all tests:

      1 tests in __main__

      8 tests in __main__.factorial

   9 tests in 2 items.

   9 passed and 0 failed.

   Test passed.

   $

That's all you need to know to start making productive use of :mod:`doctest`!

Jump in.  The following sections provide full details.  Note that there are many

examples of doctests in the standard Python test suite and libraries.

Especially useful examples can be found in the standard test file

:file:`Lib/test/test_doctest.py`.

.. _doctest-simple-testmod:

Simple Usage: Checking Examples in Docstrings

---------------------------------------------

The simplest way to start using doctest (but not necessarily the way you'll

continue to do it) is to end each module :mod:`M` with::

   if __name__ == "__main__":

       import doctest

       doctest.testmod()

:mod:`doctest` then examines docstrings in module :mod:`M`.

Running the module as a script causes the examples in the docstrings to get

executed and verified::

   python M.py

This won't display anything unless an example fails, in which case the failing

example(s) and the cause(s) of the failure(s) are printed to stdout, and the

final line of output is ``***Test Failed*** N failures.``, where *N* is the

number of examples that failed.

Run it with the :option:`-v` switch instead::

   python M.py -v

and a detailed report of all examples tried is printed to standard output, along

with assorted summaries at the end.

You can force verbose mode by passing ``verbose=True`` to :func:`testmod`, or

prohibit it by passing ``verbose=False``.  In either of those cases,

``sys.argv`` is not examined by :func:`testmod` (so passing :option:`-v` or not

has no effect).

Since Python 2.6, there is also a command line shortcut for running

:func:`testmod`.  You can instruct the Python interpreter to run the doctest

module directly from the standard library and pass the module name(s) on the

command line::

   python -m doctest -v example.py

This will import :file:`example.py` as a standalone module and run

:func:`testmod` on it.  Note that this may not work correctly if the file is

part of a package and imports other submodules from that package.

For more information on :func:`testmod`, see section :ref:`doctest-basic-api`.

.. _doctest-simple-testfile:

Simple Usage: Checking Examples in a Text File

----------------------------------------------

Another simple application of doctest is testing interactive examples in a text

file.  This can be done with the :func:`testfile` function::

   import doctest

   doctest.testfile("example.txt")

That short script executes and verifies any interactive Python examples

contained in the file :file:`example.txt`.  The file content is treated as if it

were a single giant docstring; the file doesn't need to contain a Python

program!   For example, perhaps :file:`example.txt` contains this::

   The ``example`` module

   ======================

   Using ``factorial``

   -------------------

   This is an example text file in reStructuredText format.  First import

   ``factorial`` from the ``example`` module:

       >>> from example import factorial

   Now use it:

       >>> factorial(6)

       120

Running ``doctest.testfile("example.txt")`` then finds the error in this

documentation::

   File "./example.txt", line 14, in example.txt

   Failed example:

       factorial(6)

   Expected:

       120

   Got:

       720

As with :func:`testmod`, :func:`testfile` won't display anything unless an

example fails.  If an example does fail, then the failing example(s) and the

cause(s) of the failure(s) are printed to stdout, using the same format as

:func:`testmod`.

By default, :func:`testfile` looks for files in the calling module's directory.

See section :ref:`doctest-basic-api` for a description of the optional arguments

that can be used to tell it to look for files in other locations.

Like :func:`testmod`, :func:`testfile`'s verbosity can be set with the

:option:`-v` command-line switch or with the optional keyword argument

*verbose*.

Since Python 2.6, there is also a command line shortcut for running

:func:`testfile`.  You can instruct the Python interpreter to run the doctest

module directly from the standard library and pass the file name(s) on the

command line::

   python -m doctest -v example.txt

Because the file name does not end with :file:`.py`, :mod:`doctest` infers that

it must be run with :func:`testfile`, not :func:`testmod`.

For more information on :func:`testfile`, see section :ref:`doctest-basic-api`.

.. _doctest-how-it-works:

How It Works

------------

This section examines in detail how doctest works: which docstrings it looks at,

how it finds interactive examples, what execution context it uses, how it

handles exceptions, and how option flags can be used to control its behavior.

This is the information that you need to know to write doctest examples; for

information about actually running doctest on these examples, see the following

sections.

.. _doctest-which-docstrings:

Which Docstrings Are Examined?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The module docstring, and all function, class and method docstrings are

searched.  Objects imported into the module are not searched.

In addition, if ``M.__test__`` exists and "is true", it must be a dict, and each

entry maps a (string) name to a function object, class object, or string.

Function and class object docstrings found from ``M.__test__`` are searched, and

strings are treated as if they were docstrings.  In output, a key ``K`` in

``M.__test__`` appears with name ::

   <name of M>.__test__.K

Any classes found are recursively searched similarly, to test docstrings in

their contained methods and nested classes.

.. versionchanged:: 2.4

   A "private name" concept is deprecated and no longer documented.

.. _doctest-finding-examples:

How are Docstring Examples Recognized?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In most cases a copy-and-paste of an interactive console session works fine, but

doctest isn't trying to do an exact emulation of any specific Python shell.  All

hard tab characters are expanded to spaces, using 8-column tab stops.  If you

don't believe tabs should mean that, too bad:  don't use hard tabs, or write

your own :class:`DocTestParser` class.

.. versionchanged:: 2.4

   Expanding tabs to spaces is new; previous versions tried to preserve hard tabs,

   with confusing results.

::

   >>> # comments are ignored

   >>> x = 12

   >>> x

   12

   >>> if x == 13:

   ...     print "yes"

   ... else:

   ...     print "no"

   ...     print "NO"

   ...     print "NO!!!"

   ...

   no

   NO

   NO!!!

   >>>

Any expected output must immediately follow the final ``'>>> '`` or ``'... '``

line containing the code, and the expected output (if any) extends to the next

``'>>> '`` or all-whitespace line.

The fine print:

* Expected output cannot contain an all-whitespace line, since such a line is

  taken to signal the end of expected output.  If expected output does contain a

  blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line

  is expected.

  .. versionchanged:: 2.4

     ``<BLANKLINE>`` was added; there was no way to use expected output containing

     empty lines in previous versions.

* Output to stdout is captured, but not output to stderr (exception tracebacks

  are captured via a different means).

* If you continue a line via backslashing in an interactive session, or for any

  other reason use a backslash, you should use a raw docstring, which will

  preserve your backslashes exactly as you type them::

     >>> def f(x):

     ...     r'''Backslashes in a raw docstring: m\n'''

     >>> print f.__doc__

     Backslashes in a raw docstring: m\n

  Otherwise, the backslash will be interpreted as part of the string. For example,

  the "\\" above would be interpreted as a newline character.  Alternatively, you

  can double each backslash in the doctest version (and not use a raw string)::

     >>> def f(x):

     ...     '''Backslashes in a raw docstring: m\\n'''

     >>> print f.__doc__

     Backslashes in a raw docstring: m\n

* The starting column doesn't matter::

     >>> assert "Easy!"

           >>> import math

               >>> math.floor(1.9)

               1.0

  and as many leading whitespace characters are stripped from the expected output

  as appeared in the initial ``'>>> '`` line that started the example.

.. _doctest-execution-context:

What's the Execution Context?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By default, each time :mod:`doctest` finds a docstring to test, it uses a

*shallow copy* of :mod:`M`'s globals, so that running tests doesn't change the

module's real globals, and so that one test in :mod:`M` can't leave behind

crumbs that accidentally allow another test to work.  This means examples can

freely use any names defined at top-level in :mod:`M`, and names defined earlier

in the docstring being run. Examples cannot see names defined in other

docstrings.

You can force use of your own dict as the execution context by passing

``globs=your_dict`` to :func:`testmod` or :func:`testfile` instead.

.. _doctest-exceptions:

What About Exceptions?

^^^^^^^^^^^^^^^^^^^^^^

No problem, provided that the traceback is the only output produced by the

example:  just paste in the traceback. [#]_ Since tracebacks contain details

that are likely to change rapidly (for example, exact file paths and line

numbers), this is one case where doctest works hard to be flexible in what it

accepts.

Simple example::

   >>> [1, 2, 3].remove(42)

   Traceback (most recent call last):

     File "<stdin>", line 1, in ?

   ValueError: list.remove(x): x not in list

That doctest succeeds if :exc:`ValueError` is raised, with the ``list.remove(x):

x not in list`` detail as shown.

The expected output for an exception must start with a traceback header, which

may be either of the following two lines, indented the same as the first line of

the example::

   Traceback (most recent call last):

   Traceback (innermost last):

The traceback header is followed by an optional traceback stack, whose contents

are ignored by doctest.  The traceback stack is typically omitted, or copied

verbatim from an interactive session.

The traceback stack is followed by the most interesting part: the line(s)

containing the exception type and detail.  This is usually the last line of a

traceback, but can extend across multiple lines if the exception has a

multi-line detail::

   >>> raise ValueError('multi\n    line\ndetail')

   Traceback (most recent call last):

     File "<stdin>", line 1, in ?

   ValueError: multi

       line

   detail

The last three lines (starting with :exc:`ValueError`) are compared against the

exception's type and detail, and the rest are ignored.

Best practice is to omit the traceback stack, unless it adds significant

documentation value to the example.  So the last example is probably better as::

   >>> raise ValueError('multi\n    line\ndetail')

   Traceback (most recent call last):

       ...

   ValueError: multi

       line

   detail

Note that tracebacks are treated very specially.  In particular, in the

rewritten example, the use of ``...`` is independent of doctest's

:const:`ELLIPSIS` option.  The ellipsis in that example could be left out, or

could just as well be three (or three hundred) commas or digits, or an indented

transcript of a Monty Python skit.

Some details you should read once, but won't need to remember:

* Doctest can't guess whether your expected output came from an exception

  traceback or from ordinary printing.  So, e.g., an example that expects

  ``ValueError: 42 is prime`` will pass whether :exc:`ValueError` is actually

  raised or if the example merely prints that traceback text.  In practice,

  ordinary output rarely begins with a traceback header line, so this doesn't

  create real problems.

* Each line of the traceback stack (if present) must be indented further than

  the first line of the example, *or* start with a non-alphanumeric character.

  The first line following the traceback header indented the same and starting

  with an alphanumeric is taken to be the start of the exception detail.  Of

  course this does the right thing for genuine tracebacks.

* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is is specified,

  everything following the leftmost colon is ignored.

* The interactive shell omits the traceback header line for some

  :exc:`SyntaxError`\ s.  But doctest uses the traceback header line to

  distinguish exceptions from non-exceptions.  So in the rare case where you need

  to test a :exc:`SyntaxError` that omits the traceback header, you will need to

  manually add the traceback header line to your test example.

* For some :exc:`SyntaxError`\ s, Python displays the character position of the

  syntax error, using a ``^`` marker::

     >>> 1 1

       File "<stdin>", line 1

         1 1

           ^

     SyntaxError: invalid syntax

  Since the lines showing the position of the error come before the exception type

  and detail, they are not checked by doctest.  For example, the following test

  would pass, even though it puts the ``^`` marker in the wrong location::

     >>> 1 1

     Traceback (most recent call last):

       File "<stdin>", line 1

         1 1

         ^

     SyntaxError: invalid syntax

.. versionchanged:: 2.4

   The ability to handle a multi-line exception detail, and the

   :const:`IGNORE_EXCEPTION_DETAIL` doctest option, were added.

.. _doctest-options:

Option Flags and Directives

^^^^^^^^^^^^^^^^^^^^^^^^^^^

A number of option flags control various aspects of doctest's behavior.

Symbolic names for the flags are supplied as module constants, which can be

or'ed together and passed to various functions.  The names can also be used in

doctest directives (see below).

The first group of options define test semantics, controlling aspects of how

doctest decides whether actual output matches an example's expected output:

.. data:: DONT_ACCEPT_TRUE_FOR_1

   By default, if an expected output block contains just ``1``, an actual output

   block containing just ``1`` or just ``True`` is considered to be a match, and

   similarly for ``0`` versus ``False``.  When :const:`DONT_ACCEPT_TRUE_FOR_1` is

   specified, neither substitution is allowed.  The default behavior caters to that

   Python changed the return type of many functions from integer to boolean;

   doctests expecting "little integer" output still work in these cases.  This

   option will probably go away, but not for several years.