:mod:`re` --- Regular expression operations

===========================================

.. module:: re

   :synopsis: Regular expression operations.

.. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com>

.. sectionauthor:: Andrew M. Kuchling <amk@amk.ca>

This module provides regular expression matching operations similar to

those found in Perl. Both patterns and strings to be searched can be

Unicode strings as well as 8-bit strings.  The :mod:`re` module is

always available.

Regular expressions use the backslash character (``'\'``) to indicate

special forms or to allow special characters to be used without invoking

their special meaning.  This collides with Python's usage of the same

character for the same purpose in string literals; for example, to match

a literal backslash, one might have to write ``'\\\\'`` as the pattern

string, because the regular expression must be ``\\``, and each

backslash must be expressed as ``\\`` inside a regular Python string

literal.

The solution is to use Python's raw string notation for regular expression

patterns; backslashes are not handled in any special way in a string literal

prefixed with ``'r'``.  So ``r"\n"`` is a two-character string containing

``'\'`` and ``'n'``, while ``"\n"`` is a one-character string containing a

newline.  Usually patterns will be expressed in Python code using this raw

string notation.

It is important to note that most regular expression operations are available as

module-level functions and :class:`RegexObject` methods.  The functions are

shortcuts that don't require you to compile a regex object first, but miss some

fine-tuning parameters.

.. seealso::

   Mastering Regular Expressions

      Book on regular expressions by Jeffrey Friedl, published by O'Reilly.  The

      second edition of the book no longer covers Python at all, but the first

      edition covered writing good regular expression patterns in great detail.

   `Kodos <http://kodos.sf.net/>`_

      is a graphical regular expression debugger written in Python.

.. _re-syntax:

Regular Expression Syntax

-------------------------

A regular expression (or RE) specifies a set of strings that matches it; the

functions in this module let you check if a particular string matches a given

regular expression (or if a given regular expression matches a particular

string, which comes down to the same thing).

Regular expressions can be concatenated to form new regular expressions; if *A*

and *B* are both regular expressions, then *AB* is also a regular expression.

In general, if a string *p* matches *A* and another string *q* matches *B*, the

string *pq* will match AB.  This holds unless *A* or *B* contain low precedence

operations; boundary conditions between *A* and *B*; or have numbered group

references.  Thus, complex expressions can easily be constructed from simpler

primitive expressions like the ones described here.  For details of the theory

and implementation of regular expressions, consult the Friedl book referenced

above, or almost any textbook about compiler construction.

A brief explanation of the format of regular expressions follows.  For further

information and a gentler presentation, consult the :ref:`regex-howto`.

Regular expressions can contain both special and ordinary characters. Most

ordinary characters, like ``'A'``, ``'a'``, or ``'0'``, are the simplest regular

expressions; they simply match themselves.  You can concatenate ordinary

characters, so ``last`` matches the string ``'last'``.  (In the rest of this

section, we'll write RE's in ``this special style``, usually without quotes, and

strings to be matched ``'in single quotes'``.)

Some characters, like ``'|'`` or ``'('``, are special. Special

characters either stand for classes of ordinary characters, or affect

how the regular expressions around them are interpreted. Regular

expression pattern strings may not contain null bytes, but can specify

the null byte using the ``\number`` notation, e.g., ``'\x00'``.

The special characters are:

``'.'``

   (Dot.)  In the default mode, this matches any character except a newline.  If

   the :const:`DOTALL` flag has been specified, this matches any character

   including a newline.

``'^'``

   (Caret.)  Matches the start of the string, and in :const:`MULTILINE` mode also

   matches immediately after each newline.

``'$'``

   Matches the end of the string or just before the newline at the end of the

   string, and in :const:`MULTILINE` mode also matches before a newline.  ``foo``

   matches both 'foo' and 'foobar', while the regular expression ``foo$`` matches

   only 'foo'.  More interestingly, searching for ``foo.$`` in ``'foo1\nfoo2\n'``

   matches 'foo2' normally, but 'foo1' in :const:`MULTILINE` mode; searching for

   a single ``$`` in ``'foo\n'`` will find two (empty) matches: one just before

   the newline, and one at the end of the string.

``'*'``

   Causes the resulting RE to match 0 or more repetitions of the preceding RE, as

   many repetitions as are possible.  ``ab*`` will match 'a', 'ab', or 'a' followed

   by any number of 'b's.

``'+'``

   Causes the resulting RE to match 1 or more repetitions of the preceding RE.

   ``ab+`` will match 'a' followed by any non-zero number of 'b's; it will not

   match just 'a'.

``'?'``

   Causes the resulting RE to match 0 or 1 repetitions of the preceding RE.

   ``ab?`` will match either 'a' or 'ab'.

``*?``, ``+?``, ``??``

   The ``'*'``, ``'+'``, and ``'?'`` qualifiers are all :dfn:`greedy`; they match

   as much text as possible.  Sometimes this behaviour isn't desired; if the RE

   ``<.*>`` is matched against ``'<H1>title</H1>'``, it will match the entire

   string, and not just ``'<H1>'``.  Adding ``'?'`` after the qualifier makes it

   perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few*

   characters as possible will be matched.  Using ``.*?`` in the previous

   expression will match only ``'<H1>'``.

``{m}``

   Specifies that exactly *m* copies of the previous RE should be matched; fewer

   matches cause the entire RE not to match.  For example, ``a{6}`` will match

   exactly six ``'a'`` characters, but not five.

``{m,n}``

   Causes the resulting RE to match from *m* to *n* repetitions of the preceding

   RE, attempting to match as many repetitions as possible.  For example,

   ``a{3,5}`` will match from 3 to 5 ``'a'`` characters.  Omitting *m* specifies a

   lower bound of zero,  and omitting *n* specifies an infinite upper bound.  As an

   example, ``a{4,}b`` will match ``aaaab`` or a thousand ``'a'`` characters

   followed by a ``b``, but not ``aaab``. The comma may not be omitted or the

   modifier would be confused with the previously described form.

``{m,n}?``

   Causes the resulting RE to match from *m* to *n* repetitions of the preceding

   RE, attempting to match as *few* repetitions as possible.  This is the

   non-greedy version of the previous qualifier.  For example, on the

   6-character string ``'aaaaaa'``, ``a{3,5}`` will match 5 ``'a'`` characters,

   while ``a{3,5}?`` will only match 3 characters.

``'\'``

   Either escapes special characters (permitting you to match characters like

   ``'*'``, ``'?'``, and so forth), or signals a special sequence; special

   sequences are discussed below.

   If you're not using a raw string to express the pattern, remember that Python

   also uses the backslash as an escape sequence in string literals; if the escape

   sequence isn't recognized by Python's parser, the backslash and subsequent

   character are included in the resulting string.  However, if Python would

   recognize the resulting sequence, the backslash should be repeated twice.  This

   is complicated and hard to understand, so it's highly recommended that you use

   raw strings for all but the simplest expressions.

``[]``

   Used to indicate a set of characters.  Characters can be listed individually, or

   a range of characters can be indicated by giving two characters and separating

   them by a ``'-'``.  Special characters are not active inside sets.  For example,

   ``[akm$]`` will match any of the characters ``'a'``, ``'k'``,

   ``'m'``, or ``'$'``; ``[a-z]`` will match any lowercase letter, and

   ``[a-zA-Z0-9]`` matches any letter or digit.  Character classes such

   as ``\w`` or ``\S`` (defined below) are also acceptable inside a

   range, although the characters they match depends on whether :const:`LOCALE`

   or  :const:`UNICODE` mode is in force.  If you want to include a

   ``']'`` or a ``'-'`` inside a set, precede it with a backslash, or

   place it as the first character.  The pattern ``[]]`` will match

   ``']'``, for example.

   You can match the characters not within a range by :dfn:`complementing` the set.

   This is indicated by including a ``'^'`` as the first character of the set;

   ``'^'`` elsewhere will simply match the ``'^'`` character.  For example,

   ``[^5]`` will match any character except ``'5'``, and ``[^^]`` will match any

   character except ``'^'``.

   Note that inside ``[]`` the special forms and special characters lose

   their meanings and only the syntaxes described here are valid. For

   example, ``+``, ``*``, ``(``, ``)``, and so on are treated as

   literals inside ``[]``, and backreferences cannot be used inside

   ``[]``.

``'|'``

   ``A|B``, where A and B can be arbitrary REs, creates a regular expression that

   will match either A or B.  An arbitrary number of REs can be separated by the

   ``'|'`` in this way.  This can be used inside groups (see below) as well.  As

   the target string is scanned, REs separated by ``'|'`` are tried from left to

   right. When one pattern completely matches, that branch is accepted. This means

   that once ``A`` matches, ``B`` will not be tested further, even if it would

   produce a longer overall match.  In other words, the ``'|'`` operator is never

   greedy.  To match a literal ``'|'``, use ``\|``, or enclose it inside a

   character class, as in ``[|]``.

``(...)``

   Matches whatever regular expression is inside the parentheses, and indicates the

   start and end of a group; the contents of a group can be retrieved after a match

   has been performed, and can be matched later in the string with the ``\number``

   special sequence, described below.  To match the literals ``'('`` or ``')'``,

   use ``\(`` or ``\)``, or enclose them inside a character class: ``[(] [)]``.

``(?...)``

   This is an extension notation (a ``'?'`` following a ``'('`` is not meaningful

   otherwise).  The first character after the ``'?'`` determines what the meaning

   and further syntax of the construct is. Extensions usually do not create a new

   group; ``(?P<name>...)`` is the only exception to this rule. Following are the

   currently supported extensions.

``(?iLmsux)``

   (One or more letters from the set ``'i'``, ``'L'``, ``'m'``, ``'s'``,

   ``'u'``, ``'x'``.)  The group matches the empty string; the letters

   set the corresponding flags: :const:`re.I` (ignore case),

   :const:`re.L` (locale dependent), :const:`re.M` (multi-line),

   :const:`re.S` (dot matches all), :const:`re.U` (Unicode dependent),

   and :const:`re.X` (verbose), for the entire regular expression. (The

   flags are described in :ref:`contents-of-module-re`.) This

   is useful if you wish to include the flags as part of the regular

   expression, instead of passing a *flag* argument to the

   :func:`compile` function.

   Note that the ``(?x)`` flag changes how the expression is parsed. It should be

   used first in the expression string, or after one or more whitespace characters.

   If there are non-whitespace characters before the flag, the results are

   undefined.

``(?:...)``

   A non-grouping version of regular parentheses. Matches whatever regular

   expression is inside the parentheses, but the substring matched by the group

   *cannot* be retrieved after performing a match or referenced later in the

   pattern.

``(?P<name>...)``

   Similar to regular parentheses, but the substring matched by the group is

   accessible via the symbolic group name *name*.  Group names must be valid Python

   identifiers, and each group name must be defined only once within a regular

   expression.  A symbolic group is also a numbered group, just as if the group

   were not named.  So the group named 'id' in the example below can also be

   referenced as the numbered group 1.

   For example, if the pattern is ``(?P<id>[a-zA-Z_]\w*)``, the group can be

   referenced by its name in arguments to methods of match objects, such as

   ``m.group('id')`` or ``m.end('id')``, and also by name in pattern text (for

   example, ``(?P=id)``) and replacement text (such as ``\g<id>``).

``(?P=name)``

   Matches whatever text was matched by the earlier group named *name*.

``(?#...)``

   A comment; the contents of the parentheses are simply ignored.

``(?=...)``

   Matches if ``...`` matches next, but doesn't consume any of the string.  This is

   called a lookahead assertion.  For example, ``Isaac (?=Asimov)`` will match

   ``'Isaac '`` only if it's followed by ``'Asimov'``.

``(?!...)``

   Matches if ``...`` doesn't match next.  This is a negative lookahead assertion.

   For example, ``Isaac (?!Asimov)`` will match ``'Isaac '`` only if it's *not*

   followed by ``'Asimov'``.

``(?<=...)``

   Matches if the current position in the string is preceded by a match for ``...``

   that ends at the current position.  This is called a :dfn:`positive lookbehind

   assertion`. ``(?<=abc)def`` will find a match in ``abcdef``, since the

   lookbehind will back up 3 characters and check if the contained pattern matches.

   The contained pattern must only match strings of some fixed length, meaning that

   ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not.  Note that

   patterns which start with positive lookbehind assertions will never match at the

   beginning of the string being searched; you will most likely want to use the

   :func:`search` function rather than the :func:`match` function:

      >>> import re

      >>> m = re.search('(?<=abc)def', 'abcdef')

      >>> m.group(0)

      'def'

   This example looks for a word following a hyphen:

      >>> m = re.search('(?<=-)\w+', 'spam-egg')

      >>> m.group(0)

      'egg'

``(?<!...)``

   Matches if the current position in the string is not preceded by a match for

   ``...``.  This is called a :dfn:`negative lookbehind assertion`.  Similar to

   positive lookbehind assertions, the contained pattern must only match strings of

   some fixed length.  Patterns which start with negative lookbehind assertions may

   match at the beginning of the string being searched.

``(?(id/name)yes-pattern|no-pattern)``

   Will try to match with ``yes-pattern`` if the group with given *id* or *name*

   exists, and with ``no-pattern`` if it doesn't. ``no-pattern`` is optional and

   can be omitted. For example,  ``(<)?(\w+@\w+(?:\.\w+)+)(?(1)>)`` is a poor email

   matching pattern, which will match with ``'<user@host.com>'`` as well as

   ``'user@host.com'``, but not with ``'<user@host.com'``.

   .. versionadded:: 2.4

The special sequences consist of ``'\'`` and a character from the list below.

If the ordinary character is not on the list, then the resulting RE will match

the second character.  For example, ``\$`` matches the character ``'$'``.

``\number``

   Matches the contents of the group of the same number.  Groups are numbered

   starting from 1.  For example, ``(.+) \1`` matches ``'the the'`` or ``'55 55'``,

   but not ``'the end'`` (note the space after the group).  This special sequence

   can only be used to match one of the first 99 groups.  If the first digit of

   *number* is 0, or *number* is 3 octal digits long, it will not be interpreted as

   a group match, but as the character with octal value *number*. Inside the

   ``'['`` and ``']'`` of a character class, all numeric escapes are treated as

   characters.

``\A``

   Matches only at the start of the string.

``\b``

   Matches the empty string, but only at the beginning or end of a word.  A word is

   defined as a sequence of alphanumeric or underscore characters, so the end of a

   word is indicated by whitespace or a non-alphanumeric, non-underscore character.

   Note that  ``\b`` is defined as the boundary between ``\w`` and ``\ W``, so the

   precise set of characters deemed to be alphanumeric depends on the values of the

   ``UNICODE`` and ``LOCALE`` flags.  Inside a character range, ``\b`` represents

   the backspace character, for compatibility with Python's string literals.

``\B``

   Matches the empty string, but only when it is *not* at the beginning or end of a

   word.  This is just the opposite of ``\b``, so is also subject to the settings

   of ``LOCALE`` and ``UNICODE``.

``\d``

   When the :const:`UNICODE` flag is not specified, matches any decimal digit; this

   is equivalent to the set ``[0-9]``.  With :const:`UNICODE`, it will match

   whatever is classified as a digit in the Unicode character properties database.

``\D``

   When the :const:`UNICODE` flag is not specified, matches any non-digit

   character; this is equivalent to the set  ``[^0-9]``.  With :const:`UNICODE`, it

   will match  anything other than character marked as digits in the Unicode

   character  properties database.

``\s``

   When the :const:`LOCALE` and :const:`UNICODE` flags are not specified, matches

   any whitespace character; this is equivalent to the set ``[ \t\n\r\f\v]``. With

   :const:`LOCALE`, it will match this set plus whatever characters are defined as

   space for the current locale. If :const:`UNICODE` is set, this will match the

   characters ``[ \t\n\r\f\v]`` plus whatever is classified as space in the Unicode

   character properties database.

``\S``

   When the :const:`LOCALE` and :const:`UNICODE` flags are not specified, matches

   any non-whitespace character; this is equivalent to the set ``[^ \t\n\r\f\v]``

   With :const:`LOCALE`, it will match any character not in this set, and not

   defined as space in the current locale. If :const:`UNICODE` is set, this will

   match anything other than ``[ \t\n\r\f\v]`` and characters marked as space in

   the Unicode character properties database.

``\w``

   When the :const:`LOCALE` and :const:`UNICODE` flags are not specified, matches

   any alphanumeric character and the underscore; this is equivalent to the set

   ``[a-zA-Z0-9_]``.  With :const:`LOCALE`, it will match the set ``[0-9_]`` plus

   whatever characters are defined as alphanumeric for the current locale.  If

   :const:`UNICODE` is set, this will match the characters ``[0-9_]`` plus whatever

   is classified as alphanumeric in the Unicode character properties database.

``\W``

   When the :const:`LOCALE` and :const:`UNICODE` flags are not specified, matches

   any non-alphanumeric character; this is equivalent to the set ``[^a-zA-Z0-9_]``.

   With :const:`LOCALE`, it will match any character not in the set ``[0-9_]``, and

   not defined as alphanumeric for the current locale. If :const:`UNICODE` is set,

   this will match anything other than ``[0-9_]`` and characters marked as

   alphanumeric in the Unicode character properties database.

``\Z``

   Matches only at the end of the string.

Most of the standard escapes supported by Python string literals are also

accepted by the regular expression parser::

   \a      \b      \f      \n

   \r      \t      \v      \x

   \\

Octal escapes are included in a limited form: If the first digit is a 0, or if

there are three octal digits, it is considered an octal escape. Otherwise, it is

a group reference.  As for string literals, octal escapes are always at most

three digits in length.

.. _matching-searching:

Matching vs Searching

---------------------

.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>

Python offers two different primitive operations based on regular expressions:

**match** checks for a match only at the beginning of the string, while

**search** checks for a match anywhere in the string (this is what Perl does

by default).

Note that match may differ from search even when using a regular expression

beginning with ``'^'``: ``'^'`` matches only at the start of the string, or in

:const:`MULTILINE` mode also immediately following a newline.  The "match"

operation succeeds only if the pattern matches at the start of the string

regardless of mode, or at the starting position given by the optional *pos*

argument regardless of whether a newline precedes it.

   >>> re.match("c", "abcdef")  # No match

   >>> re.search("c", "abcdef") # Match

   <_sre.SRE_Match object at ...>

.. _contents-of-module-re:

Module Contents

---------------

The module defines several functions, constants, and an exception. Some of the

functions are simplified versions of the full featured methods for compiled

regular expressions.  Most non-trivial applications always use the compiled

form.

.. function:: compile(pattern[, flags])

   Compile a regular expression pattern into a regular expression object, which

   can be used for matching using its :func:`match` and :func:`search` methods,

   described below.

   The expression's behaviour can be modified by specifying a *flags* value.

   Values can be any of the following variables, combined using bitwise OR (the

   ``|`` operator).

   The sequence ::

      prog = re.compile(pat)

      result = prog.match(str)

   is equivalent to ::

      result = re.match(pat, str)

   but the version using :func:`compile` is more efficient when the expression

   will be used several times in a single program.

   .. (The compiled version of the last pattern passed to :func:`re.match` or

      :func:`re.search` is cached, so programs that use only a single regular

      expression at a time needn't worry about compiling regular expressions.)

.. data:: I

          IGNORECASE

   Perform case-insensitive matching; expressions like ``[A-Z]`` will match

   lowercase letters, too.  This is not affected by the current locale.

.. data:: L

          LOCALE

   Make ``\w``, ``\W``, ``\b``, ``\B``, ``\s`` and ``\S`` dependent on the

   current locale.

.. data:: M

          MULTILINE

   When specified, the pattern character ``'^'`` matches at the beginning of the

   string and at the beginning of each line (immediately following each newline);

   and the pattern character ``'$'`` matches at the end of the string and at the

   end of each line (immediately preceding each newline).  By default, ``'^'``

   matches only at the beginning of the string, and ``'$'`` only at the end of the

   string and immediately before the newline (if any) at the end of the string.

.. data:: S

          DOTALL

   Make the ``'.'`` special character match any character at all, including a

   newline; without this flag, ``'.'`` will match anything *except* a newline.

.. data:: U

          UNICODE

   Make ``\w``, ``\W``, ``\b``, ``\B``, ``\d``, ``\D``, ``\s`` and ``\S`` dependent

   on the Unicode character properties database.

   .. versionadded:: 2.0

.. data:: X

          VERBOSE

   This flag allows you to write regular expressions that look nicer. Whitespace

   within the pattern is ignored, except when in a character class or preceded by

   an unescaped backslash, and, when a line contains a ``'#'`` neither in a

   character class or preceded by an unescaped backslash, all characters from the

   leftmost such ``'#'`` through the end of the line are ignored.

   That means that the two following regular expression objects that match a

   decimal number are functionally equal::

      a = re.compile(r"""\d +  # the integral part

                         \.    # the decimal point

                         \d *  # some fractional digits""", re.X)

      b = re.compile(r"\d+\.\d*")

.. function:: search(pattern, string[, flags])

   Scan through *string* looking for a location where the regular expression

   *pattern* produces a match, and return a corresponding :class:`MatchObject`

   instance. Return ``None`` if no position in the string matches the pattern; note

   that this is different from finding a zero-length match at some point in the

   string.

.. function:: match(pattern, string[, flags])

   If zero or more characters at the beginning of *string* match the regular