1 :mod:`doctest` --- Test interactive Python examples

     2 ===================================================

     3 

     4 .. module:: doctest

     5    :synopsis: Test pieces of code within docstrings.

     6 .. moduleauthor:: Tim Peters <tim@python.org>

     7 .. sectionauthor:: Tim Peters <tim@python.org>

     8 .. sectionauthor:: Moshe Zadka <moshez@debian.org>

     9 .. sectionauthor:: Edward Loper <edloper@users.sourceforge.net>

    10 

    11 

    12 The :mod:`doctest` module searches for pieces of text that look like interactive

    13 Python sessions, and then executes those sessions to verify that they work

    14 exactly as shown.  There are several common ways to use doctest:

    15 

    16 * To check that a module's docstrings are up-to-date by verifying that all

    17   interactive examples still work as documented.

    18 

    19 * To perform regression testing by verifying that interactive examples from a

    20   test file or a test object work as expected.

    21 

    22 * To write tutorial documentation for a package, liberally illustrated with

    23   input-output examples.  Depending on whether the examples or the expository text

    24   are emphasized, this has the flavor of "literate testing" or "executable

    25   documentation".

    26 

    27 Here's a complete but small example module::

    28 

    29    """

    30    This is the "example" module.

    31 

    32    The example module supplies one function, factorial().  For example,

    33 

    34    >>> factorial(5)

    35    120

    36    """

    37 

    38    def factorial(n):

    39        """Return the factorial of n, an exact integer >= 0.

    40 

    41        If the result is small enough to fit in an int, return an int.

    42        Else return a long.

    43 

    44        >>> [factorial(n) for n in range(6)]

    45        [1, 1, 2, 6, 24, 120]

    46        >>> [factorial(long(n)) for n in range(6)]

    47        [1, 1, 2, 6, 24, 120]

    48        >>> factorial(30)

    49        265252859812191058636308480000000L

    50        >>> factorial(30L)

    51        265252859812191058636308480000000L

    52        >>> factorial(-1)

    53        Traceback (most recent call last):

    54            ...

    55        ValueError: n must be >= 0

    56 

    57        Factorials of floats are OK, but the float must be an exact integer:

    58        >>> factorial(30.1)

    59        Traceback (most recent call last):

    60            ...

    61        ValueError: n must be exact integer

    62        >>> factorial(30.0)

    63        265252859812191058636308480000000L

    64 

    65        It must also not be ridiculously large:

    66        >>> factorial(1e100)

    67        Traceback (most recent call last):

    68            ...

    69        OverflowError: n too large

    70        """

    71 

    72        import math

    73        if not n >= 0:

    74            raise ValueError("n must be >= 0")

    75        if math.floor(n) != n:

    76            raise ValueError("n must be exact integer")

    77        if n+1 == n:  # catch a value like 1e300

    78            raise OverflowError("n too large")

    79        result = 1

    80        factor = 2

    81        while factor <= n:

    82            result *= factor

    83            factor += 1

    84        return result

    85 

    86 

    87    if __name__ == "__main__":

    88        import doctest

    89        doctest.testmod()

    90 

    91 If you run :file:`example.py` directly from the command line, :mod:`doctest`

    92 works its magic::

    93 

    94    $ python example.py

    95    $

    96 

    97 There's no output!  That's normal, and it means all the examples worked.  Pass

    98 :option:`-v` to the script, and :mod:`doctest` prints a detailed log of what

    99 it's trying, and prints a summary at the end::

   100 

   101    $ python example.py -v

   102    Trying:

   103        factorial(5)

   104    Expecting:

   105        120

   106    ok

   107    Trying:

   108        [factorial(n) for n in range(6)]

   109    Expecting:

   110        [1, 1, 2, 6, 24, 120]

   111    ok

   112    Trying:

   113        [factorial(long(n)) for n in range(6)]

   114    Expecting:

   115        [1, 1, 2, 6, 24, 120]

   116    ok

   117 

   118 And so on, eventually ending with::

   119 

   120    Trying:

   121        factorial(1e100)

   122    Expecting:

   123        Traceback (most recent call last):

   124            ...

   125        OverflowError: n too large

   126    ok

   127    2 items passed all tests:

   128       1 tests in __main__

   129       8 tests in __main__.factorial

   130    9 tests in 2 items.

   131    9 passed and 0 failed.

   132    Test passed.

   133    $

   134 

   135 That's all you need to know to start making productive use of :mod:`doctest`!

   136 Jump in.  The following sections provide full details.  Note that there are many

   137 examples of doctests in the standard Python test suite and libraries.

   138 Especially useful examples can be found in the standard test file

   139 :file:`Lib/test/test_doctest.py`.

   140 

   141 

   142 .. _doctest-simple-testmod:

   143 

   144 Simple Usage: Checking Examples in Docstrings

   145 ---------------------------------------------

   146 

   147 The simplest way to start using doctest (but not necessarily the way you'll

   148 continue to do it) is to end each module :mod:`M` with::

   149 

   150    if __name__ == "__main__":

   151        import doctest

   152        doctest.testmod()

   153 

   154 :mod:`doctest` then examines docstrings in module :mod:`M`.

   155 

   156 Running the module as a script causes the examples in the docstrings to get

   157 executed and verified::

   158 

   159    python M.py

   160 

   161 This won't display anything unless an example fails, in which case the failing

   162 example(s) and the cause(s) of the failure(s) are printed to stdout, and the

   163 final line of output is ``***Test Failed*** N failures.``, where *N* is the

   164 number of examples that failed.

   165 

   166 Run it with the :option:`-v` switch instead::

   167 

   168    python M.py -v

   169 

   170 and a detailed report of all examples tried is printed to standard output, along

   171 with assorted summaries at the end.

   172 

   173 You can force verbose mode by passing ``verbose=True`` to :func:`testmod`, or

   174 prohibit it by passing ``verbose=False``.  In either of those cases,

   175 ``sys.argv`` is not examined by :func:`testmod` (so passing :option:`-v` or not

   176 has no effect).

   177 

   178 Since Python 2.6, there is also a command line shortcut for running

   179 :func:`testmod`.  You can instruct the Python interpreter to run the doctest

   180 module directly from the standard library and pass the module name(s) on the

   181 command line::

   182 

   183    python -m doctest -v example.py

   184 

   185 This will import :file:`example.py` as a standalone module and run

   186 :func:`testmod` on it.  Note that this may not work correctly if the file is

   187 part of a package and imports other submodules from that package.

   188 

   189 For more information on :func:`testmod`, see section :ref:`doctest-basic-api`.

   190 

   191 

   192 .. _doctest-simple-testfile:

   193 

   194 Simple Usage: Checking Examples in a Text File

   195 ----------------------------------------------

   196 

   197 Another simple application of doctest is testing interactive examples in a text

   198 file.  This can be done with the :func:`testfile` function::

   199 

   200    import doctest

   201    doctest.testfile("example.txt")

   202 

   203 That short script executes and verifies any interactive Python examples

   204 contained in the file :file:`example.txt`.  The file content is treated as if it

   205 were a single giant docstring; the file doesn't need to contain a Python

   206 program!   For example, perhaps :file:`example.txt` contains this::

   207 

   208    The ``example`` module

   209    ======================

   210 

   211    Using ``factorial``

   212    -------------------

   213 

   214    This is an example text file in reStructuredText format.  First import

   215    ``factorial`` from the ``example`` module:

   216 

   217        >>> from example import factorial

   218 

   219    Now use it:

   220 

   221        >>> factorial(6)

   222        120

   223 

   224 Running ``doctest.testfile("example.txt")`` then finds the error in this

   225 documentation::

   226 

   227    File "./example.txt", line 14, in example.txt

   228    Failed example:

   229        factorial(6)

   230    Expected:

   231        120

   232    Got:

   233        720

   234 

   235 As with :func:`testmod`, :func:`testfile` won't display anything unless an

   236 example fails.  If an example does fail, then the failing example(s) and the

   237 cause(s) of the failure(s) are printed to stdout, using the same format as

   238 :func:`testmod`.

   239 

   240 By default, :func:`testfile` looks for files in the calling module's directory.

   241 See section :ref:`doctest-basic-api` for a description of the optional arguments

   242 that can be used to tell it to look for files in other locations.

   243 

   244 Like :func:`testmod`, :func:`testfile`'s verbosity can be set with the

   245 :option:`-v` command-line switch or with the optional keyword argument

   246 *verbose*.

   247 

   248 Since Python 2.6, there is also a command line shortcut for running

   249 :func:`testfile`.  You can instruct the Python interpreter to run the doctest

   250 module directly from the standard library and pass the file name(s) on the

   251 command line::

   252 

   253    python -m doctest -v example.txt

   254 

   255 Because the file name does not end with :file:`.py`, :mod:`doctest` infers that

   256 it must be run with :func:`testfile`, not :func:`testmod`.

   257 

   258 For more information on :func:`testfile`, see section :ref:`doctest-basic-api`.

   259 

   260 

   261 .. _doctest-how-it-works:

   262 

   263 How It Works

   264 ------------

   265 

   266 This section examines in detail how doctest works: which docstrings it looks at,

   267 how it finds interactive examples, what execution context it uses, how it

   268 handles exceptions, and how option flags can be used to control its behavior.

   269 This is the information that you need to know to write doctest examples; for

   270 information about actually running doctest on these examples, see the following

   271 sections.

   272 

   273 

   274 .. _doctest-which-docstrings:

   275 

   276 Which Docstrings Are Examined?

   277 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   278 

   279 The module docstring, and all function, class and method docstrings are

   280 searched.  Objects imported into the module are not searched.

   281 

   282 In addition, if ``M.__test__`` exists and "is true", it must be a dict, and each

   283 entry maps a (string) name to a function object, class object, or string.

   284 Function and class object docstrings found from ``M.__test__`` are searched, and

   285 strings are treated as if they were docstrings.  In output, a key ``K`` in

   286 ``M.__test__`` appears with name ::

   287 

   288    <name of M>.__test__.K

   289 

   290 Any classes found are recursively searched similarly, to test docstrings in

   291 their contained methods and nested classes.

   292 

   293 .. versionchanged:: 2.4

   294    A "private name" concept is deprecated and no longer documented.

   295 

   296 

   297 .. _doctest-finding-examples:

   298 

   299 How are Docstring Examples Recognized?

   300 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   301 

   302 In most cases a copy-and-paste of an interactive console session works fine, but

   303 doctest isn't trying to do an exact emulation of any specific Python shell.  All

   304 hard tab characters are expanded to spaces, using 8-column tab stops.  If you

   305 don't believe tabs should mean that, too bad:  don't use hard tabs, or write

   306 your own :class:`DocTestParser` class.

   307 

   308 .. versionchanged:: 2.4

   309    Expanding tabs to spaces is new; previous versions tried to preserve hard tabs,

   310    with confusing results.

   311 

   312 ::

   313 

   314    >>> # comments are ignored

   315    >>> x = 12

   316    >>> x

   317    12

   318    >>> if x == 13:

   319    ...     print "yes"

   320    ... else:

   321    ...     print "no"

   322    ...     print "NO"

   323    ...     print "NO!!!"

   324    ...

   325    no

   326    NO

   327    NO!!!

   328    >>>

   329 

   330 Any expected output must immediately follow the final ``'>>> '`` or ``'... '``

   331 line containing the code, and the expected output (if any) extends to the next

   332 ``'>>> '`` or all-whitespace line.

   333 

   334 The fine print:

   335 

   336 * Expected output cannot contain an all-whitespace line, since such a line is

   337   taken to signal the end of expected output.  If expected output does contain a

   338   blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line

   339   is expected.

   340 

   341   .. versionchanged:: 2.4

   342      ``<BLANKLINE>`` was added; there was no way to use expected output containing

   343      empty lines in previous versions.

   344 

   345 * Output to stdout is captured, but not output to stderr (exception tracebacks

   346   are captured via a different means).

   347 

   348 * If you continue a line via backslashing in an interactive session, or for any

   349   other reason use a backslash, you should use a raw docstring, which will

   350   preserve your backslashes exactly as you type them::

   351 

   352      >>> def f(x):

   353      ...     r'''Backslashes in a raw docstring: m\n'''

   354      >>> print f.__doc__

   355      Backslashes in a raw docstring: m\n

   356 

   357   Otherwise, the backslash will be interpreted as part of the string. For example,

   358   the "\\" above would be interpreted as a newline character.  Alternatively, you

   359   can double each backslash in the doctest version (and not use a raw string)::

   360 

   361      >>> def f(x):

   362      ...     '''Backslashes in a raw docstring: m\\n'''

   363      >>> print f.__doc__

   364      Backslashes in a raw docstring: m\n

   365 

   366 * The starting column doesn't matter::

   367 

   368      >>> assert "Easy!"

   369            >>> import math

   370                >>> math.floor(1.9)

   371                1.0

   372 

   373   and as many leading whitespace characters are stripped from the expected output

   374   as appeared in the initial ``'>>> '`` line that started the example.

   375 

   376 

   377 .. _doctest-execution-context:

   378 

   379 What's the Execution Context?

   380 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   381 

   382 By default, each time :mod:`doctest` finds a docstring to test, it uses a

   383 *shallow copy* of :mod:`M`'s globals, so that running tests doesn't change the

   384 module's real globals, and so that one test in :mod:`M` can't leave behind

   385 crumbs that accidentally allow another test to work.  This means examples can

   386 freely use any names defined at top-level in :mod:`M`, and names defined earlier

   387 in the docstring being run. Examples cannot see names defined in other

   388 docstrings.

   389 

   390 You can force use of your own dict as the execution context by passing

   391 ``globs=your_dict`` to :func:`testmod` or :func:`testfile` instead.

   392 

   393 

   394 .. _doctest-exceptions:

   395 

   396 What About Exceptions?

   397 ^^^^^^^^^^^^^^^^^^^^^^

   398 

   399 No problem, provided that the traceback is the only output produced by the

   400 example:  just paste in the traceback. [#]_ Since tracebacks contain details

   401 that are likely to change rapidly (for example, exact file paths and line

   402 numbers), this is one case where doctest works hard to be flexible in what it

   403 accepts.

   404 

   405 Simple example::

   406 

   407    >>> [1, 2, 3].remove(42)

   408    Traceback (most recent call last):

   409      File "<stdin>", line 1, in ?

   410    ValueError: list.remove(x): x not in list

   411 

   412 That doctest succeeds if :exc:`ValueError` is raised, with the ``list.remove(x):

   413 x not in list`` detail as shown.

   414 

   415 The expected output for an exception must start with a traceback header, which

   416 may be either of the following two lines, indented the same as the first line of

   417 the example::

   418 

   419    Traceback (most recent call last):

   420    Traceback (innermost last):

   421 

   422 The traceback header is followed by an optional traceback stack, whose contents

   423 are ignored by doctest.  The traceback stack is typically omitted, or copied

   424 verbatim from an interactive session.

   425 

   426 The traceback stack is followed by the most interesting part: the line(s)

   427 containing the exception type and detail.  This is usually the last line of a

   428 traceback, but can extend across multiple lines if the exception has a

   429 multi-line detail::

   430 

   431    >>> raise ValueError('multi\n    line\ndetail')

   432    Traceback (most recent call last):

   433      File "<stdin>", line 1, in ?

   434    ValueError: multi

   435        line

   436    detail

   437 

   438 The last three lines (starting with :exc:`ValueError`) are compared against the

   439 exception's type and detail, and the rest are ignored.

   440 

   441 Best practice is to omit the traceback stack, unless it adds significant

   442 documentation value to the example.  So the last example is probably better as::

   443 

   444    >>> raise ValueError('multi\n    line\ndetail')

   445    Traceback (most recent call last):

   446        ...

   447    ValueError: multi

   448        line

   449    detail

   450 

   451 Note that tracebacks are treated very specially.  In particular, in the

   452 rewritten example, the use of ``...`` is independent of doctest's

   453 :const:`ELLIPSIS` option.  The ellipsis in that example could be left out, or

   454 could just as well be three (or three hundred) commas or digits, or an indented

   455 transcript of a Monty Python skit.

   456 

   457 Some details you should read once, but won't need to remember:

   458 

   459 * Doctest can't guess whether your expected output came from an exception

   460   traceback or from ordinary printing.  So, e.g., an example that expects

   461   ``ValueError: 42 is prime`` will pass whether :exc:`ValueError` is actually

   462   raised or if the example merely prints that traceback text.  In practice,

   463   ordinary output rarely begins with a traceback header line, so this doesn't

   464   create real problems.

   465 

   466 * Each line of the traceback stack (if present) must be indented further than

   467   the first line of the example, *or* start with a non-alphanumeric character.

   468   The first line following the traceback header indented the same and starting

   469   with an alphanumeric is taken to be the start of the exception detail.  Of

   470   course this does the right thing for genuine tracebacks.

   471 

   472 * When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is is specified,

   473   everything following the leftmost colon is ignored.

   474 

   475 * The interactive shell omits the traceback header line for some

   476   :exc:`SyntaxError`\ s.  But doctest uses the traceback header line to

   477   distinguish exceptions from non-exceptions.  So in the rare case where you need

   478   to test a :exc:`SyntaxError` that omits the traceback header, you will need to

   479   manually add the traceback header line to your test example.

   480 

   481 * For some :exc:`SyntaxError`\ s, Python displays the character position of the

   482   syntax error, using a ``^`` marker::

   483 

   484      >>> 1 1

   485        File "<stdin>", line 1

   486          1 1

   487            ^

   488      SyntaxError: invalid syntax

   489 

   490   Since the lines showing the position of the error come before the exception type

   491   and detail, they are not checked by doctest.  For example, the following test

   492   would pass, even though it puts the ``^`` marker in the wrong location::

   493 

   494      >>> 1 1

   495      Traceback (most recent call last):

   496        File "<stdin>", line 1

   497          1 1

   498          ^

   499      SyntaxError: invalid syntax

   500 

   501 .. versionchanged:: 2.4

   502    The ability to handle a multi-line exception detail, and the

   503    :const:`IGNORE_EXCEPTION_DETAIL` doctest option, were added.

   504 

   505 

   506 .. _doctest-options:

   507 

   508 Option Flags and Directives

   509 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

   510 

   511 A number of option flags control various aspects of doctest's behavior.

   512 Symbolic names for the flags are supplied as module constants, which can be

   513 or'ed together and passed to various functions.  The names can also be used in

   514 doctest directives (see below).

   515 

   516 The first group of options define test semantics, controlling aspects of how

   517 doctest decides whether actual output matches an example's expected output:

   518 

   519 

   520 .. data:: DONT_ACCEPT_TRUE_FOR_1

   521 

   522    By default, if an expected output block contains just ``1``, an actual output

   523    block containing just ``1`` or just ``True`` is considered to be a match, and

   524    similarly for ``0`` versus ``False``.  When :const:`DONT_ACCEPT_TRUE_FOR_1` is

   525    specified, neither substitution is allowed.  The default behavior caters to that

   526    Python changed the return type of many functions from integer to boolean;

   527    doctests expecting "little integer" output still work in these cases.  This

   528    option will probably go away, but not for several years.

   529 

   530 

   531 .. data:: DONT_ACCEPT_BLANKLINE

   532 

   533    By default, if an expected output block contains a line containing only the

   534    string ``<BLANKLINE>``, then that line will match a blank line in the actual

   535    output.  Because a genuinely blank line delimits the expected output, this is

   536    the only way to communicate that a blank line is expected.  When

   537    :const:`DONT_ACCEPT_BLANKLINE` is specified, this substitution is not allowed.

   538 

   539 

   540 .. data:: NORMALIZE_WHITESPACE

   541 

   542    When specified, all sequences of whitespace (blanks and newlines) are treated as

   543    equal.  Any sequence of whitespace within the expected output will match any

   544    sequence of whitespace within the actual output. By default, whitespace must

   545    match exactly. :const:`NORMALIZE_WHITESPACE` is especially useful when a line of

   546    expected output is very long, and you want to wrap it across multiple lines in

   547    your source.

   548 

   549 

   550 .. data:: ELLIPSIS

   551 

   552    When specified, an ellipsis marker (``...``) in the expected output can match

   553    any substring in the actual output.  This includes substrings that span line

   554    boundaries, and empty substrings, so it's best to keep usage of this simple.

   555    Complicated uses can lead to the same kinds of "oops, it matched too much!"

   556    surprises that ``.*`` is prone to in regular expressions.

   557 

   558 

   559 .. data:: IGNORE_EXCEPTION_DETAIL

   560 

   561    When specified, an example that expects an exception passes if an exception of

   562    the expected type is raised, even if the exception detail does not match.  For

   563    example, an example expecting ``ValueError: 42`` will pass if the actual

   564    exception raised is ``ValueError: 3*14``, but will fail, e.g., if

   565    :exc:`TypeError` is raised.

   566 

   567    Note that a similar effect can be obtained using :const:`ELLIPSIS`, and

   568    :const:`IGNORE_EXCEPTION_DETAIL` may go away when Python releases prior to 2.4

   569    become uninteresting.  Until then, :const:`IGNORE_EXCEPTION_DETAIL` is the only

   570    clear way to write a doctest that doesn't care about the exception detail yet

   571    continues to pass under Python releases prior to 2.4 (doctest directives appear

   572    to be comments to them).  For example, ::

   573 

   574       >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL

   575       Traceback (most recent call last):

   576         File "<stdin>", line 1, in ?

   577       TypeError: object doesn't support item assignment

   578 

   579    passes under Python 2.4 and Python 2.3.  The detail changed in 2.4, to say "does

   580    not" instead of "doesn't".

   581 

   582 

   583 .. data:: SKIP

   584 

   585    When specified, do not run the example at all.  This can be useful in contexts

   586    where doctest examples serve as both documentation and test cases, and an

   587    example should be included for documentation purposes, but should not be

   588    checked.  E.g., the example's output might be random; or the example might

   589    depend on resources which would be unavailable to the test driver.

   590 

   591    The SKIP flag can also be used for temporarily "commenting out" examples.

   592 

   593 

   594 .. data:: COMPARISON_FLAGS

   595 

   596    A bitmask or'ing together all the comparison flags above.

   597 

   598 The second group of options controls how test failures are reported:

   599 

   600 

   601 .. data:: REPORT_UDIFF

   602 

   603    When specified, failures that involve multi-line expected and actual outputs are

   604    displayed using a unified diff.

   605 

   606 

   607 .. data:: REPORT_CDIFF

   608 

   609    When specified, failures that involve multi-line expected and actual outputs

   610    will be displayed using a context diff.

   611 

   612 

   613 .. data:: REPORT_NDIFF

   614 

   615    When specified, differences are computed by ``difflib.Differ``, using the same

   616    algorithm as the popular :file:`ndiff.py` utility. This is the only method that

   617    marks differences within lines as well as across lines.  For example, if a line

   618    of expected output contains digit ``1`` where actual output contains letter

   619    ``l``, a line is inserted with a caret marking the mismatching column positions.

   620 

   621 

   622 .. data:: REPORT_ONLY_FIRST_FAILURE

   623 

   624    When specified, display the first failing example in each doctest, but suppress

   625    output for all remaining examples.  This will prevent doctest from reporting

   626    correct examples that break because of earlier failures; but it might also hide

   627    incorrect examples that fail independently of the first failure.  When

   628    :const:`REPORT_ONLY_FIRST_FAILURE` is specified, the remaining examples are

   629    still run, and still count towards the total number of failures reported; only

   630    the output is suppressed.

   631 

   632 

   633 .. data:: REPORTING_FLAGS

   634 

   635    A bitmask or'ing together all the reporting flags above.

   636 

   637 "Doctest directives" may be used to modify the option flags for individual

   638 examples.  Doctest directives are expressed as a special Python comment

   639 following an example's source code:

   640 

   641 .. productionlist:: doctest

   642    directive: "#" "doctest:" `directive_options`

   643    directive_options: `directive_option` ("," `directive_option`)\*

   644    directive_option: `on_or_off` `directive_option_name`

   645    on_or_off: "+" \| "-"

   646    directive_option_name: "DONT_ACCEPT_BLANKLINE" \| "NORMALIZE_WHITESPACE" \| ...

   647 

   648 Whitespace is not allowed between the ``+`` or ``-`` and the directive option

   649 name.  The directive option name can be any of the option flag names explained

   650 above.

   651 

   652 An example's doctest directives modify doctest's behavior for that single

   653 example.  Use ``+`` to enable the named behavior, or ``-`` to disable it.

   654 

   655 For example, this test passes::

   656 

   657    >>> print range(20) #doctest: +NORMALIZE_WHITESPACE

   658    [0,   1,  2,  3,  4,  5,  6,  7,  8,  9,

   659    10,  11, 12, 13, 14, 15, 16, 17, 18, 19]

   660 

   661 Without the directive it would fail, both because the actual output doesn't have

   662 two blanks before the single-digit list elements, and because the actual output

   663 is on a single line.  This test also passes, and also requires a directive to do

   664 so::

   665 

   666    >>> print range(20) # doctest:+ELLIPSIS

   667    [0, 1, ..., 18, 19]

   668 

   669 Multiple directives can be used on a single physical line, separated by commas::

   670 

   671    >>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE

   672    [0,    1, ...,   18,    19]

   673 

   674 If multiple directive comments are used for a single example, then they are

   675 combined::

   676 

   677    >>> print range(20) # doctest: +ELLIPSIS

   678    ...                 # doctest: +NORMALIZE_WHITESPACE

   679    [0,    1, ...,   18,    19]

   680 

   681 As the previous example shows, you can add ``...`` lines to your example

   682 containing only directives.  This can be useful when an example is too long for

   683 a directive to comfortably fit on the same line::

   684 

   685    >>> print range(5) + range(10,20) + range(30,40) + range(50,60)

   686    ... # doctest: +ELLIPSIS

   687    [0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59]

   688 

   689 Note that since all options are disabled by default, and directives apply only

   690 to the example they appear in, enabling options (via ``+`` in a directive) is

   691 usually the only meaningful choice.  However, option flags can also be passed to

   692 functions that run doctests, establishing different defaults.  In such cases,

   693 disabling an option via ``-`` in a directive can be useful.

   694 

   695 .. versionchanged:: 2.4

   696    Constants :const:`DONT_ACCEPT_BLANKLINE`, :const:`NORMALIZE_WHITESPACE`,

   697    :const:`ELLIPSIS`, :const:`IGNORE_EXCEPTION_DETAIL`, :const:`REPORT_UDIFF`,

   698    :const:`REPORT_CDIFF`, :const:`REPORT_NDIFF`,

   699    :const:`REPORT_ONLY_FIRST_FAILURE`, :const:`COMPARISON_FLAGS` and

   700    :const:`REPORTING_FLAGS` were added; by default ``<BLANKLINE>`` in expected

   701    output matches an empty line in actual output; and doctest directives were

   702    added.

   703 

   704 .. versionchanged:: 2.5

   705    Constant :const:`SKIP` was added.

   706 

   707 There's also a way to register new option flag names, although this isn't useful

   708 unless you intend to extend :mod:`doctest` internals via subclassing:

   709 

   710 

   711 .. function:: register_optionflag(name)

   712 

   713    Create a new option flag with a given name, and return the new flag's integer

   714    value.  :func:`register_optionflag` can be used when subclassing

   715    :class:`OutputChecker` or :class:`DocTestRunner` to create new options that are

   716    supported by your subclasses.  :func:`register_optionflag` should always be

   717    called using the following idiom::

   718 

   719       MY_FLAG = register_optionflag('MY_FLAG')

   720 

   721    .. versionadded:: 2.4

   722 

   723 

   724 .. _doctest-warnings:

   725 

   726 Warnings

   727 ^^^^^^^^

   728 

   729 :mod:`doctest` is serious about requiring exact matches in expected output.  If

   730 even a single character doesn't match, the test fails.  This will probably

   731 surprise you a few times, as you learn exactly what Python does and doesn't

   732 guarantee about output.  For example, when printing a dict, Python doesn't

   733 guarantee that the key-value pairs will be printed in any particular order, so a

   734 test like ::

   735 

   736    >>> foo()

   737    {"Hermione": "hippogryph", "Harry": "broomstick"}

   738 

   739 is vulnerable!  One workaround is to do ::

   740 

   741    >>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"}

   742    True

   743 

   744 instead.  Another is to do ::

   745 

   746    >>> d = foo().items()

   747    >>> d.sort()

   748    >>> d

   749    [('Harry', 'broomstick'), ('Hermione', 'hippogryph')]

   750 

   751 There are others, but you get the idea.

   752 

   753 Another bad idea is to print things that embed an object address, like ::

   754 

   755    >>> id(1.0) # certain to fail some of the time

   756    7948648

   757    >>> class C: pass

   758    >>> C()   # the default repr() for instances embeds an address

   759    <__main__.C instance at 0x00AC18F0>

   760 

   761 The :const:`ELLIPSIS` directive gives a nice approach for the last example::

   762 

   763    >>> C() #doctest: +ELLIPSIS

   764    <__main__.C instance at 0x...>

   765 

   766 Floating-point numbers are also subject to small output variations across

   767 platforms, because Python defers to the platform C library for float formatting,

   768 and C libraries vary widely in quality here. ::

   769 

   770    >>> 1./7  # risky

   771    0.14285714285714285

   772    >>> print 1./7 # safer

   773    0.142857142857

   774    >>> print round(1./7, 6) # much safer

   775    0.142857

   776 

   777 Numbers of the form ``I/2.**J`` are safe across all platforms, and I often

   778 contrive doctest examples to produce numbers of that form::

   779 

   780    >>> 3./4  # utterly safe

   781    0.75

   782 

   783 Simple fractions are also easier for people to understand, and that makes for

   784 better documentation.

   785 

   786 

   787 .. _doctest-basic-api:

   788 

   789 Basic API

   790 ---------

   791 

   792 The functions :func:`testmod` and :func:`testfile` provide a simple interface to

   793 doctest that should be sufficient for most basic uses.  For a less formal

   794 introduction to these two functions, see sections :ref:`doctest-simple-testmod`

   795 and :ref:`doctest-simple-testfile`.

   796 

   797 

   798 .. function:: testfile(filename[, module_relative][, name][, package][, globs][, verbose][, report][, optionflags][, extraglobs][, raise_on_error][, parser][, encoding])

   799 

   800    All arguments except *filename* are optional, and should be specified in keyword

   801    form.

   802 

   803    Test examples in the file named *filename*.  Return ``(failure_count,

   804    test_count)``.

   805 

   806    Optional argument *module_relative* specifies how the filename should be

   807    interpreted:

   808 

   809    * If *module_relative* is ``True`` (the default), then *filename* specifies an

   810      OS-independent module-relative path.  By default, this path is relative to the

   811      calling module's directory; but if the *package* argument is specified, then it

   812      is relative to that package.  To ensure OS-independence, *filename* should use

   813      ``/`` characters to separate path segments, and may not be an absolute path

   814      (i.e., it may not begin with ``/``).

   815 

   816    * If *module_relative* is ``False``, then *filename* specifies an OS-specific

   817      path.  The path may be absolute or relative; relative paths are resolved with

   818      respect to the current working directory.

   819 

   820    Optional argument *name* gives the name of the test; by default, or if ``None``,

   821    ``os.path.basename(filename)`` is used.

   822 

   823    Optional argument *package* is a Python package or the name of a Python package

   824    whose directory should be used as the base directory for a module-relative

   825    filename.  If no package is specified, then the calling module's directory is

   826    used as the base directory for module-relative filenames.  It is an error to

   827    specify *package* if *module_relative* is ``False``.

   828 

   829    Optional argument *globs* gives a dict to be used as the globals when executing

   830    examples.  A new shallow copy of this dict is created for the doctest, so its

   831    examples start with a clean slate. By default, or if ``None``, a new empty dict

   832    is used.

   833 

   834    Optional argument *extraglobs* gives a dict merged into the globals used to

   835    execute examples.  This works like :meth:`dict.update`:  if *globs* and

   836    *extraglobs* have a common key, the associated value in *extraglobs* appears in

   837    the combined dict.  By default, or if ``None``, no extra globals are used.  This

   838    is an advanced feature that allows parameterization of doctests.  For example, a

   839    doctest can be written for a base class, using a generic name for the class,

   840    then reused to test any number of subclasses by passing an *extraglobs* dict

   841    mapping the generic name to the subclass to be tested.

   842 

   843    Optional argument *verbose* prints lots of stuff if true, and prints only

   844    failures if false; by default, or if ``None``, it's true if and only if ``'-v'``

   845    is in ``sys.argv``.

   846 

   847    Optional argument *report* prints a summary at the end when true, else prints

   848    nothing at the end.  In verbose mode, the summary is detailed, else the summary

   849    is very brief (in fact, empty if all tests passed).

   850 

   851    Optional argument *optionflags* or's together option flags.  See section

   852    :ref:`doctest-options`.

   853 

   854    Optional argument *raise_on_error* defaults to false.  If true, an exception is

   855    raised upon the first failure or unexpected exception in an example.  This

   856    allows failures to be post-mortem debugged. Default behavior is to continue

   857    running examples.

   858 

   859    Optional argument *parser* specifies a :class:`DocTestParser` (or subclass) that

   860    should be used to extract tests from the files.  It defaults to a normal parser

   861    (i.e., ``DocTestParser()``).

   862 

   863    Optional argument *encoding* specifies an encoding that should be used to

   864    convert the file to unicode.

   865 

   866    .. versionadded:: 2.4

   867 

   868    .. versionchanged:: 2.5

   869       The parameter *encoding* was added.

   870 

   871 

   872 .. function:: testmod([m][, name][, globs][, verbose][, report][, optionflags][, extraglobs][, raise_on_error][, exclude_empty])

   873 

   874    All arguments are optional, and all except for *m* should be specified in

   875    keyword form.

   876 

   877    Test examples in docstrings in functions and classes reachable from module *m*

   878    (or module :mod:`__main__` if *m* is not supplied or is ``None``), starting with

   879    ``m.__doc__``.

   880 

   881    Also test examples reachable from dict ``m.__test__``, if it exists and is not

   882    ``None``.  ``m.__test__`` maps names (strings) to functions, classes and

   883    strings; function and class docstrings are searched for examples; strings are

   884    searched directly, as if they were docstrings.

   885 

   886    Only docstrings attached to objects belonging to module *m* are searched.

   887 

   888    Return ``(failure_count, test_count)``.

   889 

   890    Optional argument *name* gives the name of the module; by default, or if

   891    ``None``, ``m.__name__`` is used.

   892 

   893    Optional argument *exclude_empty* defaults to false.  If true, objects for which

   894    no doctests are found are excluded from consideration. The default is a backward

   895    compatibility hack, so that code still using :meth:`doctest.master.summarize` in

   896    conjunction with :func:`testmod` continues to get output for objects with no

   897    tests. The *exclude_empty* argument to the newer :class:`DocTestFinder`

   898    constructor defaults to true.

   899 

   900    Optional arguments *extraglobs*, *verbose*, *report*, *optionflags*,

   901    *raise_on_error*, and *globs* are the same as for function :func:`testfile`

   902    above, except that *globs* defaults to ``m.__dict__``.

   903 

   904    .. versionchanged:: 2.3

   905       The parameter *optionflags* was added.

   906 

   907    .. versionchanged:: 2.4

   908       The parameters *extraglobs*, *raise_on_error* and *exclude_empty* were added.

   909 

   910    .. versionchanged:: 2.5

   911       The optional argument *isprivate*, deprecated in 2.4, was removed.

   912 

   913 There's also a function to run the doctests associated with a single object.

   914 This function is provided for backward compatibility.  There are no plans to

   915 deprecate it, but it's rarely useful:

   916 

   917 

   918 .. function:: run_docstring_examples(f, globs[, verbose][, name][, compileflags][, optionflags])

   919 

   920    Test examples associated with object *f*; for example, *f* may be a module,

   921    function, or class object.

   922 

   923    A shallow copy of dictionary argument *globs* is used for the execution context.

   924 

   925    Optional argument *name* is used in failure messages, and defaults to

   926    ``"NoName"``.

   927 

   928    If optional argument *verbose* is true, output is generated even if there are no

   929    failures.  By default, output is generated only in case of an example failure.

   930 

   931    Optional argument *compileflags* gives the set of flags that should be used by

   932    the Python compiler when running the examples.  By default, or if ``None``,

   933    flags are deduced corresponding to the set of future features found in *globs*.

   934 

   935    Optional argument *optionflags* works as for function :func:`testfile` above.

   936 

   937 

   938 .. _doctest-unittest-api:

   939 

   940 Unittest API

   941 ------------

   942 

   943 As your collection of doctest'ed modules grows, you'll want a way to run all

   944 their doctests systematically.  Prior to Python 2.4, :mod:`doctest` had a barely

   945 documented :class:`Tester` class that supplied a rudimentary way to combine

   946 doctests from multiple modules. :class:`Tester` was feeble, and in practice most

   947 serious Python testing frameworks build on the :mod:`unittest` module, which

   948 supplies many flexible ways to combine tests from multiple sources.  So, in

   949 Python 2.4, :mod:`doctest`'s :class:`Tester` class is deprecated, and

   950 :mod:`doctest` provides two functions that can be used to create :mod:`unittest`

   951 test suites from modules and text files containing doctests.  These test suites

   952 can then be run using :mod:`unittest` test runners::

   953 

   954    import unittest

   955    import doctest

   956    import my_module_with_doctests, and_another

   957 

   958    suite = unittest.TestSuite()

   959    for mod in my_module_with_doctests, and_another:

   960        suite.addTest(doctest.DocTestSuite(mod))

   961    runner = unittest.TextTestRunner()

   962    runner.run(suite)

   963 

   964 There are two main functions for creating :class:`unittest.TestSuite` instances

   965 from text files and modules with doctests:

   966 

   967 

   968 .. function:: DocFileSuite([module_relative][, package][, setUp][, tearDown][, globs][, optionflags][, parser][, encoding])

   969 

   970    Convert doctest tests from one or more text files to a

   971    :class:`unittest.TestSuite`.

   972 

   973    The returned :class:`unittest.TestSuite` is to be run by the unittest framework

   974    and runs the interactive examples in each file.  If an example in any file

   975    fails, then the synthesized unit test fails, and a :exc:`failureException`

   976    exception is raised showing the name of the file containing the test and a

   977    (sometimes approximate) line number.

   978 

   979    Pass one or more paths (as strings) to text files to be examined.

   980 

   981    Options may be provided as keyword arguments:

   982 

   983    Optional argument *module_relative* specifies how the filenames in *paths*

   984    should be interpreted:

   985 

   986    * If *module_relative* is ``True`` (the default), then each filename specifies

   987      an OS-independent module-relative path.  By default, this path is relative to

   988      the calling module's directory; but if the *package* argument is specified, then

   989      it is relative to that package.  To ensure OS-independence, each filename should

   990      use ``/`` characters to separate path segments, and may not be an absolute path

   991      (i.e., it may not begin with ``/``).

   992 

   993    * If *module_relative* is ``False``, then each filename specifies an OS-specific

   994      path.  The path may be absolute or relative; relative paths are resolved with

   995      respect to the current working directory.

   996 

   997    Optional argument *package* is a Python package or the name of a Python package

   998    whose directory should be used as the base directory for module-relative

   999    filenames.  If no package is specified, then the calling module's directory is

  1000    used as the base directory for module-relative filenames.  It is an error to

  1001    specify *package* if *module_relative* is ``False``.

  1002 

  1003    Optional argument *setUp* specifies a set-up function for the test suite.  This

  1004    is called before running the tests in each file.  The *setUp* function will be

  1005    passed a :class:`DocTest` object.  The setUp function can access the test

  1006    globals as the *globs* attribute of the test passed.

  1007 

  1008    Optional argument *tearDown* specifies a tear-down function for the test suite.

  1009    This is called after running the tests in each file.  The *tearDown* function

  1010    will be passed a :class:`DocTest` object.  The setUp function can access the

  1011    test globals as the *globs* attribute of the test passed.

  1012 

  1013    Optional argument *globs* is a dictionary containing the initial global

  1014    variables for the tests.  A new copy of this dictionary is created for each

  1015    test.  By default, *globs* is a new empty dictionary.

  1016 

  1017    Optional argument *optionflags* specifies the default doctest options for the

  1018    tests, created by or-ing together individual option flags.  See section

  1019    :ref:`doctest-options`. See function :func:`set_unittest_reportflags` below for

  1020    a better way to set reporting options.

  1021 

  1022    Optional argument *parser* specifies a :class:`DocTestParser` (or subclass) that

  1023    should be used to extract tests from the files.  It defaults to a normal parser

  1024    (i.e., ``DocTestParser()``).

  1025 

  1026    Optional argument *encoding* specifies an encoding that should be used to

  1027    convert the file to unicode.

  1028 

  1029    .. versionadded:: 2.4

  1030 

  1031    .. versionchanged:: 2.5

  1032       The global ``__file__`` was added to the globals provided to doctests loaded

  1033       from a text file using :func:`DocFileSuite`.

  1034 

  1035    .. versionchanged:: 2.5

  1036       The parameter *encoding* was added.

  1037 

  1038 

  1039 .. function:: DocTestSuite([module][, globs][, extraglobs][, test_finder][, setUp][, tearDown][, checker])

  1040 

  1041    Convert doctest tests for a module to a :class:`unittest.TestSuite`.

  1042 

  1043    The returned :class:`unittest.TestSuite` is to be run by the unittest framework

  1044    and runs each doctest in the module.  If any of the doctests fail, then the

  1045    synthesized unit test fails, and a :exc:`failureException` exception is raised

  1046    showing the name of the file containing the test and a (sometimes approximate)

  1047    line number.

  1048 

  1049    Optional argument *module* provides the module to be tested.  It can be a module

  1050    object or a (possibly dotted) module name.  If not specified, the module calling

  1051    this function is used.

  1052 

  1053    Optional argument *globs* is a dictionary containing the initial global

  1054    variables for the tests.  A new copy of this dictionary is created for each

  1055    test.  By default, *globs* is a new empty dictionary.

  1056 

  1057    Optional argument *extraglobs* specifies an extra set of global variables, which

  1058    is merged into *globs*.  By default, no extra globals are used.

  1059 

  1060    Optional argument *test_finder* is the :class:`DocTestFinder` object (or a

  1061    drop-in replacement) that is used to extract doctests from the module.

  1062 

  1063    Optional arguments *setUp*, *tearDown*, and *optionflags* are the same as for

  1064    function :func:`DocFileSuite` above.

  1065 

  1066    .. versionadded:: 2.3

  1067 

  1068    .. versionchanged:: 2.4

  1069       The parameters *globs*, *extraglobs*, *test_finder*, *setUp*, *tearDown*, and

  1070       *optionflags* were added; this function now uses the same search technique as

  1071       :func:`testmod`.

  1072 

  1073 Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out

  1074 of :class:`doctest.DocTestCase` instances, and :class:`DocTestCase` is a

  1075 subclass of :class:`unittest.TestCase`. :class:`DocTestCase` isn't documented

  1076 here (it's an internal detail), but studying its code can answer questions about

  1077 the exact details of :mod:`unittest` integration.

  1078 

  1079 Similarly, :func:`DocFileSuite` creates a :class:`unittest.TestSuite` out of

  1080 :class:`doctest.DocFileCase` instances, and :class:`DocFileCase` is a subclass

  1081 of :class:`DocTestCase`.

  1082 

  1083 So both ways of creating a :class:`unittest.TestSuite` run instances of

  1084 :class:`DocTestCase`.  This is important for a subtle reason: when you run

  1085 :mod:`doctest` functions yourself, you can control the :mod:`doctest` options in

  1086 use directly, by passing option flags to :mod:`doctest` functions.  However, if

  1087 you're writing a :mod:`unittest` framework, :mod:`unittest` ultimately controls

  1088 when and how tests get run.  The framework author typically wants to control

  1089 :mod:`doctest` reporting options (perhaps, e.g., specified by command line

  1090 options), but there's no way to pass options through :mod:`unittest` to

  1091 :mod:`doctest` test runners.

  1092 

  1093 For this reason, :mod:`doctest` also supports a notion of :mod:`doctest`

  1094 reporting flags specific to :mod:`unittest` support, via this function:

  1095 

  1096 

  1097 .. function:: set_unittest_reportflags(flags)

  1098 

  1099    Set the :mod:`doctest` reporting flags to use.

  1100 

  1101    Argument *flags* or's together option flags.  See section

  1102    :ref:`doctest-options`.  Only "reporting flags" can be used.

  1103 

  1104    This is a module-global setting, and affects all future doctests run by module

  1105    :mod:`unittest`:  the :meth:`runTest` method of :class:`DocTestCase` looks at

  1106    the option flags specified for the test case when the :class:`DocTestCase`

  1107    instance was constructed.  If no reporting flags were specified (which is the

  1108    typical and expected case), :mod:`doctest`'s :mod:`unittest` reporting flags are

  1109    or'ed into the option flags, and the option flags so augmented are passed to the

  1110    :class:`DocTestRunner` instance created to run the doctest.  If any reporting

  1111    flags were specified when the :class:`DocTestCase` instance was constructed,

  1112    :mod:`doctest`'s :mod:`unittest` reporting flags are ignored.

  1113 

  1114    The value of the :mod:`unittest` reporting flags in effect before the function

  1115    was called is returned by the function.

  1116 

  1117    .. versionadded:: 2.4

  1118 

  1119 

  1120 .. _doctest-advanced-api:

  1121 

  1122 Advanced API

  1123 ------------

  1124 

  1125 The basic API is a simple wrapper that's intended to make doctest easy to use.

  1126 It is fairly flexible, and should meet most users' needs; however, if you

  1127 require more fine-grained control over testing, or wish to extend doctest's

  1128 capabilities, then you should use the advanced API.

  1129 

  1130 The advanced API revolves around two container classes, which are used to store

  1131 the interactive examples extracted from doctest cases:

  1132 

  1133 * :class:`Example`: A single python :term:`statement`, paired with its expected

  1134   output.

  1135