.. _lexical:

****************

Lexical analysis

****************

.. index::

   single: lexical analysis

   single: parser

   single: token

A Python program is read by a *parser*.  Input to the parser is a stream of

*tokens*, generated by the *lexical analyzer*.  This chapter describes how the

lexical analyzer breaks a file into tokens.

Python uses the 7-bit ASCII character set for program text.

.. versionadded:: 2.3

   An encoding declaration can be used to indicate that  string literals and

   comments use an encoding different from ASCII.

For compatibility with older versions, Python only warns if it finds 8-bit

characters; those warnings should be corrected by either declaring an explicit

encoding, or using escape sequences if those bytes are binary data, instead of

characters.

The run-time character set depends on the I/O devices connected to the program

but is generally a superset of ASCII.

**Future compatibility note:** It may be tempting to assume that the character

set for 8-bit characters is ISO Latin-1 (an ASCII superset that covers most

western languages that use the Latin alphabet), but it is possible that in the

future Unicode text editors will become common.  These generally use the UTF-8

encoding, which is also an ASCII superset, but with very different use for the

characters with ordinals 128-255.  While there is no consensus on this subject

yet, it is unwise to assume either Latin-1 or UTF-8, even though the current

implementation appears to favor Latin-1.  This applies both to the source

character set and the run-time character set.

.. _line-structure:

Line structure

==============

.. index:: single: line structure

A Python program is divided into a number of *logical lines*.

.. _logical:

Logical lines

-------------

.. index::

   single: logical line

   single: physical line

   single: line joining

   single: NEWLINE token

The end of a logical line is represented by the token NEWLINE.  Statements

cannot cross logical line boundaries except where NEWLINE is allowed by the

syntax (e.g., between statements in compound statements). A logical line is

constructed from one or more *physical lines* by following the explicit or

implicit *line joining* rules.

.. _physical:

Physical lines

--------------

A physical line is a sequence of characters terminated by an end-of-line

sequence.  In source files, any of the standard platform line termination

sequences can be used - the Unix form using ASCII LF (linefeed), the Windows

form using the ASCII sequence CR LF (return followed by linefeed), or the old

Macintosh form using the ASCII CR (return) character.  All of these forms can be

used equally, regardless of platform.

When embedding Python, source code strings should be passed to Python APIs using

the standard C conventions for newline characters (the ``\n`` character,

representing ASCII LF, is the line terminator).

.. _comments:

Comments

--------

.. index::

   single: comment

   single: hash character

A comment starts with a hash character (``#``) that is not part of a string

literal, and ends at the end of the physical line.  A comment signifies the end

of the logical line unless the implicit line joining rules are invoked. Comments

are ignored by the syntax; they are not tokens.

.. _encodings:

Encoding declarations

---------------------

.. index::

   single: source character set

   single: encodings

If a comment in the first or second line of the Python script matches the

regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an

encoding declaration; the first group of this expression names the encoding of

the source code file. The recommended forms of this expression are ::

   # -*- coding: <encoding-name> -*-

which is recognized also by GNU Emacs, and ::

   # vim:fileencoding=<encoding-name>

which is recognized by Bram Moolenaar's VIM. In addition, if the first bytes of

the file are the UTF-8 byte-order mark (``'\xef\xbb\xbf'``), the declared file

encoding is UTF-8 (this is supported, among others, by Microsoft's

:program:`notepad`).

If an encoding is declared, the encoding name must be recognized by Python. The

encoding is used for all lexical analysis, in particular to find the end of a

string, and to interpret the contents of Unicode literals. String literals are

converted to Unicode for syntactical analysis, then converted back to their

original encoding before interpretation starts. The encoding declaration must

appear on a line of its own.

.. XXX there should be a list of supported encodings.

.. _explicit-joining:

Explicit line joining

---------------------

.. index::

   single: physical line

   single: line joining

   single: line continuation

   single: backslash character

Two or more physical lines may be joined into logical lines using backslash

characters (``\``), as follows: when a physical line ends in a backslash that is

not part of a string literal or comment, it is joined with the following forming

a single logical line, deleting the backslash and the following end-of-line

character.  For example::

   if 1900 < year < 2100 and 1 <= month <= 12 \

      and 1 <= day <= 31 and 0 <= hour < 24 \

      and 0 <= minute < 60 and 0 <= second < 60:   # Looks like a valid date

           return 1

A line ending in a backslash cannot carry a comment.  A backslash does not

continue a comment.  A backslash does not continue a token except for string

literals (i.e., tokens other than string literals cannot be split across

physical lines using a backslash).  A backslash is illegal elsewhere on a line

outside a string literal.

.. _implicit-joining:

Implicit line joining

---------------------

Expressions in parentheses, square brackets or curly braces can be split over

more than one physical line without using backslashes. For example::

   month_names = ['Januari', 'Februari', 'Maart',      # These are the

                  'April',   'Mei',      'Juni',       # Dutch names

                  'Juli',    'Augustus', 'September',  # for the months

                  'Oktober', 'November', 'December']   # of the year

Implicitly continued lines can carry comments.  The indentation of the

continuation lines is not important.  Blank continuation lines are allowed.

There is no NEWLINE token between implicit continuation lines.  Implicitly

continued lines can also occur within triple-quoted strings (see below); in that

case they cannot carry comments.

.. _blank-lines:

Blank lines

-----------

.. index:: single: blank line

A logical line that contains only spaces, tabs, formfeeds and possibly a

comment, is ignored (i.e., no NEWLINE token is generated).  During interactive

input of statements, handling of a blank line may differ depending on the

implementation of the read-eval-print loop.  In the standard implementation, an

entirely blank logical line (i.e. one containing not even whitespace or a

comment) terminates a multi-line statement.

.. _indentation:

Indentation

-----------

.. index::

   single: indentation

   single: whitespace

   single: leading whitespace

   single: space

   single: tab

   single: grouping

   single: statement grouping

Leading whitespace (spaces and tabs) at the beginning of a logical line is used

to compute the indentation level of the line, which in turn is used to determine

the grouping of statements.

First, tabs are replaced (from left to right) by one to eight spaces such that

the total number of characters up to and including the replacement is a multiple

of eight (this is intended to be the same rule as used by Unix).  The total

number of spaces preceding the first non-blank character then determines the

line's indentation.  Indentation cannot be split over multiple physical lines

using backslashes; the whitespace up to the first backslash determines the

indentation.

**Cross-platform compatibility note:** because of the nature of text editors on

non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the

indentation in a single source file.  It should also be noted that different

platforms may explicitly limit the maximum indentation level.

A formfeed character may be present at the start of the line; it will be ignored

for the indentation calculations above.  Formfeed characters occurring elsewhere

in the leading whitespace have an undefined effect (for instance, they may reset

the space count to zero).

.. index::

   single: INDENT token

   single: DEDENT token

The indentation levels of consecutive lines are used to generate INDENT and

DEDENT tokens, using a stack, as follows.

Before the first line of the file is read, a single zero is pushed on the stack;

this will never be popped off again.  The numbers pushed on the stack will

always be strictly increasing from bottom to top.  At the beginning of each

logical line, the line's indentation level is compared to the top of the stack.

If it is equal, nothing happens. If it is larger, it is pushed on the stack, and

one INDENT token is generated.  If it is smaller, it *must* be one of the

numbers occurring on the stack; all numbers on the stack that are larger are

popped off, and for each number popped off a DEDENT token is generated.  At the

end of the file, a DEDENT token is generated for each number remaining on the

stack that is larger than zero.

Here is an example of a correctly (though confusingly) indented piece of Python

code::

   def perm(l):

           # Compute the list of all permutations of l

       if len(l) <= 1:

                     return [l]

       r = []

       for i in range(len(l)):

                s = l[:i] + l[i+1:]

                p = perm(s)

                for x in p:

                 r.append(l[i:i+1] + x)

       return r

The following example shows various indentation errors::

    def perm(l):                       # error: first line indented

   for i in range(len(l)):             # error: not indented

       s = l[:i] + l[i+1:]

           p = perm(l[:i] + l[i+1:])   # error: unexpected indent

           for x in p:

                   r.append(l[i:i+1] + x)

               return r                # error: inconsistent dedent

(Actually, the first three errors are detected by the parser; only the last

error is found by the lexical analyzer --- the indentation of ``return r`` does

not match a level popped off the stack.)

.. _whitespace:

Whitespace between tokens

-------------------------

Except at the beginning of a logical line or in string literals, the whitespace

characters space, tab and formfeed can be used interchangeably to separate

tokens.  Whitespace is needed between two tokens only if their concatenation

could otherwise be interpreted as a different token (e.g., ab is one token, but

a b is two tokens).

.. _other-tokens:

Other tokens

============

Besides NEWLINE, INDENT and DEDENT, the following categories of tokens exist:

*identifiers*, *keywords*, *literals*, *operators*, and *delimiters*. Whitespace

characters (other than line terminators, discussed earlier) are not tokens, but

serve to delimit tokens. Where ambiguity exists, a token comprises the longest

possible string that forms a legal token, when read from left to right.

.. _identifiers:

Identifiers and keywords

========================

.. index::

   single: identifier

   single: name

Identifiers (also referred to as *names*) are described by the following lexical

definitions:

.. productionlist::

   identifier: (`letter`|"_") (`letter` | `digit` | "_")*

   letter: `lowercase` | `uppercase`

   lowercase: "a"..."z"

   uppercase: "A"..."Z"

   digit: "0"..."9"

Identifiers are unlimited in length.  Case is significant.

.. _keywords:

Keywords

--------

.. index::

   single: keyword

   single: reserved word

The following identifiers are used as reserved words, or *keywords* of the

language, and cannot be used as ordinary identifiers.  They must be spelled

exactly as written here::

   and       del       from      not       while    

   as        elif      global    or        with     

   assert    else      if        pass      yield    

   break     except    import    print              

   class     exec      in        raise              

   continue  finally   is        return             

   def       for       lambda    try 

.. versionchanged:: 2.4

   :const:`None` became a constant and is now recognized by the compiler as a name

   for the built-in object :const:`None`.  Although it is not a keyword, you cannot

   assign a different object to it.

.. versionchanged:: 2.5

   Both :keyword:`as` and :keyword:`with` are only recognized when the

   ``with_statement`` future feature has been enabled. It will always be enabled in

   Python 2.6.  See section :ref:`with` for details.  Note that using :keyword:`as`

   and :keyword:`with` as identifiers will always issue a warning, even when the

   ``with_statement`` future directive is not in effect.

.. _id-classes:

Reserved classes of identifiers

-------------------------------

Certain classes of identifiers (besides keywords) have special meanings.  These

classes are identified by the patterns of leading and trailing underscore

characters:

``_*``

   Not imported by ``from module import *``.  The special identifier ``_`` is used

   in the interactive interpreter to store the result of the last evaluation; it is

   stored in the :mod:`__builtin__` module.  When not in interactive mode, ``_``

   has no special meaning and is not defined. See section :ref:`import`.

   .. note::

      The name ``_`` is often used in conjunction with internationalization;

      refer to the documentation for the :mod:`gettext` module for more

      information on this convention.

``__*__``

   System-defined names.  These names are defined by the interpreter and its

   implementation (including the standard library); applications should not expect

   to define additional names using this convention.  The set of names of this

   class defined by Python may be extended in future versions. See section

   :ref:`specialnames`.

``__*``

   Class-private names.  Names in this category, when used within the context of a

   class definition, are re-written to use a mangled form to help avoid name

   clashes between "private" attributes of base and derived classes. See section

   :ref:`atom-identifiers`.

.. _literals:

Literals

========

.. index::

   single: literal

   single: constant

Literals are notations for constant values of some built-in types.

.. _strings:

String literals

---------------

.. index:: single: string literal

String literals are described by the following lexical definitions:

.. index:: single: ASCII@ASCII

.. productionlist::

   stringliteral: [`stringprefix`](`shortstring` | `longstring`)

   stringprefix: "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR"

   shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"'

   longstring: "'''" `longstringitem`* "'''"

             : | '"""' `longstringitem`* '"""'

   shortstringitem: `shortstringchar` | `escapeseq`

   longstringitem: `longstringchar` | `escapeseq`

   shortstringchar: <any source character except "\" or newline or the quote>

   longstringchar: <any source character except "\">

   escapeseq: "\" <any ASCII character>

One syntactic restriction not indicated by these productions is that whitespace

is not allowed between the :token:`stringprefix` and the rest of the string

literal. The source character set is defined by the encoding declaration; it is

ASCII if no encoding declaration is given in the source file; see section

:ref:`encodings`.

.. index::

   single: triple-quoted string

   single: Unicode Consortium

   single: string; Unicode

   single: raw string

In plain English: String literals can be enclosed in matching single quotes

(``'``) or double quotes (``"``).  They can also be enclosed in matching groups

of three single or double quotes (these are generally referred to as

*triple-quoted strings*).  The backslash (``\``) character is used to escape

characters that otherwise have a special meaning, such as newline, backslash

itself, or the quote character.  String literals may optionally be prefixed with

a letter ``'r'`` or ``'R'``; such strings are called :dfn:`raw strings` and use

different rules for interpreting backslash escape sequences.  A prefix of

``'u'`` or ``'U'`` makes the string a Unicode string.  Unicode strings use the

Unicode character set as defined by the Unicode Consortium and ISO 10646.  Some

additional escape sequences, described below, are available in Unicode strings.

The two prefix characters may be combined; in this case, ``'u'`` must appear

before ``'r'``.

In triple-quoted strings, unescaped newlines and quotes are allowed (and are

retained), except that three unescaped quotes in a row terminate the string.  (A

"quote" is the character used to open the string, i.e. either ``'`` or ``"``.)

.. index::

   single: physical line

   single: escape sequence

   single: Standard C

   single: C

Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in strings are

interpreted according to rules similar to those used by Standard C.  The

recognized escape sequences are:

+-----------------+---------------------------------+-------+

| Escape Sequence | Meaning                         | Notes |

+=================+=================================+=======+

| ``\newline``    | Ignored                         |       |

+-----------------+---------------------------------+-------+

| ``\\``          | Backslash (``\``)               |       |

+-----------------+---------------------------------+-------+

| ``\'``          | Single quote (``'``)            |       |

+-----------------+---------------------------------+-------+

| ``\"``          | Double quote (``"``)            |       |

+-----------------+---------------------------------+-------+

| ``\a``          | ASCII Bell (BEL)                |       |

+-----------------+---------------------------------+-------+

| ``\b``          | ASCII Backspace (BS)            |       |

+-----------------+---------------------------------+-------+

| ``\f``          | ASCII Formfeed (FF)             |       |

+-----------------+---------------------------------+-------+

| ``\n``          | ASCII Linefeed (LF)             |       |

+-----------------+---------------------------------+-------+

| ``\N{name}``    | Character named *name* in the   |       |

|                 | Unicode database (Unicode only) |       |

+-----------------+---------------------------------+-------+

| ``\r``          | ASCII Carriage Return (CR)      |       |

+-----------------+---------------------------------+-------+

| ``\t``          | ASCII Horizontal Tab (TAB)      |       |

+-----------------+---------------------------------+-------+

| ``\uxxxx``      | Character with 16-bit hex value | \(1)  |

|                 | *xxxx* (Unicode only)           |       |

+-----------------+---------------------------------+-------+

| ``\Uxxxxxxxx``  | Character with 32-bit hex value | \(2)  |

|                 | *xxxxxxxx* (Unicode only)       |       |

+-----------------+---------------------------------+-------+

| ``\v``          | ASCII Vertical Tab (VT)         |       |

+-----------------+---------------------------------+-------+

| ``\ooo``        | Character with octal value      | (3,5) |

|                 | *ooo*                           |       |

+-----------------+---------------------------------+-------+

| ``\xhh``        | Character with hex value *hh*   | (4,5) |

+-----------------+---------------------------------+-------+

.. index:: single: ASCII@ASCII

Notes:

(1)

   Individual code units which form parts of a surrogate pair can be encoded using

   this escape sequence.

(2)

   Any Unicode character can be encoded this way, but characters outside the Basic

   Multilingual Plane (BMP) will be encoded using a surrogate pair if Python is

   compiled to use 16-bit code units (the default).  Individual code units which

   form parts of a surrogate pair can be encoded using this escape sequence.