****************************

  What's New in Python 2.3  

****************************

:Author: A.M. Kuchling

.. |release| replace:: 1.01

.. $Id: whatsnew23.tex 54631 2007-03-31 11:58:36Z georg.brandl $

This article explains the new features in Python 2.3.  Python 2.3 was released

on July 29, 2003.

The main themes for Python 2.3 are polishing some of the features added in 2.2,

adding various small but useful enhancements to the core language, and expanding

the standard library.  The new object model introduced in the previous version

has benefited from 18 months of bugfixes and from optimization efforts that have

improved the performance of new-style classes.  A few new built-in functions

have been added such as :func:`sum` and :func:`enumerate`.  The :keyword:`in`

operator can now be used for substring searches (e.g. ``"ab" in "abc"`` returns

:const:`True`).

Some of the many new library features include Boolean, set, heap, and date/time

data types, the ability to import modules from ZIP-format archives, metadata

support for the long-awaited Python catalog, an updated version of IDLE, and

modules for logging messages, wrapping text, parsing CSV files, processing

command-line options, using BerkeleyDB databases...  the list of new and

enhanced modules is lengthy.

This article doesn't attempt to provide a complete specification of the new

features, but instead provides a convenient overview.  For full details, you

should refer to the documentation for Python 2.3, such as the Python Library

Reference and the Python Reference Manual.  If you want to understand the

complete implementation and design rationale, refer to the PEP for a particular

new feature.

.. ======================================================================

PEP 218: A Standard Set Datatype

================================

The new :mod:`sets` module contains an implementation of a set datatype.  The

:class:`Set` class is for mutable sets, sets that can have members added and

removed.  The :class:`ImmutableSet` class is for sets that can't be modified,

and instances of :class:`ImmutableSet` can therefore be used as dictionary keys.

Sets are built on top of dictionaries, so the elements within a set must be

hashable.

Here's a simple example::

   >>> import sets

   >>> S = sets.Set([1,2,3])

   >>> S

   Set([1, 2, 3])

   >>> 1 in S

   True

   >>> 0 in S

   False

   >>> S.add(5)

   >>> S.remove(3)

   >>> S

   Set([1, 2, 5])

   >>>

The union and intersection of sets can be computed with the :meth:`union` and

:meth:`intersection` methods; an alternative notation uses the bitwise operators

``&`` and ``|``. Mutable sets also have in-place versions of these methods,

:meth:`union_update` and :meth:`intersection_update`. ::

   >>> S1 = sets.Set([1,2,3])

   >>> S2 = sets.Set([4,5,6])

   >>> S1.union(S2)

   Set([1, 2, 3, 4, 5, 6])

   >>> S1 | S2                  # Alternative notation

   Set([1, 2, 3, 4, 5, 6])

   >>> S1.intersection(S2)

   Set([])

   >>> S1 & S2                  # Alternative notation

   Set([])

   >>> S1.union_update(S2)

   >>> S1

   Set([1, 2, 3, 4, 5, 6])

   >>>

It's also possible to take the symmetric difference of two sets.  This is the

set of all elements in the union that aren't in the intersection.  Another way

of putting it is that the symmetric difference contains all elements that are in

exactly one set.  Again, there's an alternative notation (``^``), and an in-

place version with the ungainly name :meth:`symmetric_difference_update`. ::

   >>> S1 = sets.Set([1,2,3,4])

   >>> S2 = sets.Set([3,4,5,6])

   >>> S1.symmetric_difference(S2)

   Set([1, 2, 5, 6])

   >>> S1 ^ S2

   Set([1, 2, 5, 6])

   >>>

There are also :meth:`issubset` and :meth:`issuperset` methods for checking

whether one set is a subset or superset of another::

   >>> S1 = sets.Set([1,2,3])

   >>> S2 = sets.Set([2,3])

   >>> S2.issubset(S1)

   True

   >>> S1.issubset(S2)

   False

   >>> S1.issuperset(S2)

   True

   >>>

.. seealso::

   :pep:`218` - Adding a Built-In Set Object Type

      PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex Martelli, and

      GvR.

.. ======================================================================

.. _section-generators:

PEP 255: Simple Generators

==========================

In Python 2.2, generators were added as an optional feature, to be enabled by a

``from __future__ import generators`` directive.  In 2.3 generators no longer

need to be specially enabled, and are now always present; this means that

:keyword:`yield` is now always a keyword.  The rest of this section is a copy of

the description of generators from the "What's New in Python 2.2" document; if

you read it back when Python 2.2 came out, you can skip the rest of this

section.

You're doubtless familiar with how function calls work in Python or C. When you

call a function, it gets a private namespace where its local variables are

created.  When the function reaches a :keyword:`return` statement, the local

variables are destroyed and the resulting value is returned to the caller.  A

later call to the same function will get a fresh new set of local variables.

But, what if the local variables weren't thrown away on exiting a function?

What if you could later resume the function where it left off?  This is what

generators provide; they can be thought of as resumable functions.

Here's the simplest example of a generator function::

   def generate_ints(N):

       for i in range(N):

           yield i

A new keyword, :keyword:`yield`, was introduced for generators.  Any function

containing a :keyword:`yield` statement is a generator function; this is

detected by Python's bytecode compiler which compiles the function specially as

a result.

When you call a generator function, it doesn't return a single value; instead it

returns a generator object that supports the iterator protocol.  On executing

the :keyword:`yield` statement, the generator outputs the value of ``i``,

similar to a :keyword:`return` statement.  The big difference between

:keyword:`yield` and a :keyword:`return` statement is that on reaching a

:keyword:`yield` the generator's state of execution is suspended and local

variables are preserved.  On the next call to the generator's ``.next()``

method, the function will resume executing immediately after the

:keyword:`yield` statement.  (For complicated reasons, the :keyword:`yield`

statement isn't allowed inside the :keyword:`try` block of a :keyword:`try`...\

:keyword:`finally` statement; read :pep:`255` for a full explanation of the

interaction between :keyword:`yield` and exceptions.)

Here's a sample usage of the :func:`generate_ints` generator::

   >>> gen = generate_ints(3)

   >>> gen

   <generator object at 0x8117f90>

   >>> gen.next()

   0

   >>> gen.next()

   1

   >>> gen.next()

   2

   >>> gen.next()

   Traceback (most recent call last):

     File "stdin", line 1, in ?

     File "stdin", line 2, in generate_ints

   StopIteration

You could equally write ``for i in generate_ints(5)``, or ``a,b,c =

generate_ints(3)``.

Inside a generator function, the :keyword:`return` statement can only be used

without a value, and signals the end of the procession of values; afterwards the

generator cannot return any further values. :keyword:`return` with a value, such

as ``return 5``, is a syntax error inside a generator function.  The end of the

generator's results can also be indicated by raising :exc:`StopIteration`

manually, or by just letting the flow of execution fall off the bottom of the

function.

You could achieve the effect of generators manually by writing your own class

and storing all the local variables of the generator as instance variables.  For

example, returning a list of integers could be done by setting ``self.count`` to

0, and having the :meth:`next` method increment ``self.count`` and return it.

However, for a moderately complicated generator, writing a corresponding class

would be much messier. :file:`Lib/test/test_generators.py` contains a number of

more interesting examples.  The simplest one implements an in-order traversal of

a tree using generators recursively. ::

   # A recursive generator that generates Tree leaves in in-order.

   def inorder(t):

       if t:

           for x in inorder(t.left):

               yield x

           yield t.label

           for x in inorder(t.right):

               yield x

Two other examples in :file:`Lib/test/test_generators.py` produce solutions for

the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no

queen threatens another) and the Knight's Tour (a route that takes a knight to

every square of an $NxN$ chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon

(http://www.cs.arizona.edu/icon/), where the idea of generators is central.  In

Icon, every expression and function call behaves like a generator.  One example

from "An Overview of the Icon Programming Language" at

http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks

like::

   sentence := "Store it in the neighboring harbor"

   if (i := find("or", sentence)) > 5 then write(i)

In Icon the :func:`find` function returns the indexes at which the substring

"or" is found: 3, 23, 33.  In the :keyword:`if` statement, ``i`` is first

assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon

retries it with the second value of 23.  23 is greater than 5, so the comparison

now succeeds, and the code prints the value 23 to the screen.

Python doesn't go nearly as far as Icon in adopting generators as a central

concept.  Generators are considered part of the core Python language, but

learning or using them isn't compulsory; if they don't solve any problems that

you have, feel free to ignore them. One novel feature of Python's interface as

compared to Icon's is that a generator's state is represented as a concrete

object (the iterator) that can be passed around to other functions or stored in

a data structure.

.. seealso::

   :pep:`255` - Simple Generators

      Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.  Implemented mostly

      by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.

.. ======================================================================

.. _section-encodings:

PEP 263: Source Code Encodings

==============================

Python source files can now be declared as being in different character set

encodings.  Encodings are declared by including a specially formatted comment in

the first or second line of the source file.  For example, a UTF-8 file can be

declared with::

   #!/usr/bin/env python

   # -*- coding: UTF-8 -*-

Without such an encoding declaration, the default encoding used is 7-bit ASCII.

Executing or importing modules that contain string literals with 8-bit

characters and have no encoding declaration will result in a

:exc:`DeprecationWarning` being signalled by Python 2.3; in 2.4 this will be a

syntax error.

The encoding declaration only affects Unicode string literals, which will be

converted to Unicode using the specified encoding.  Note that Python identifiers

are still restricted to ASCII characters, so you can't have variable names that

use characters outside of the usual alphanumerics.

.. seealso::

   :pep:`263` - Defining Python Source Code Encodings

      Written by Marc-André Lemburg and Martin von Löwis; implemented by Suzuki Hisao

      and Martin von Löwis.

.. ======================================================================

PEP 273: Importing Modules from ZIP Archives

============================================

The new :mod:`zipimport` module adds support for importing modules from a ZIP-

format archive.  You don't need to import the module explicitly; it will be

automatically imported if a ZIP archive's filename is added to ``sys.path``.

For example::

   amk@nyman:~/src/python$ unzip -l /tmp/example.zip

   Archive:  /tmp/example.zip

     Length     Date   Time    Name

    --------    ----   ----    ----

        8467  11-26-02 22:30   jwzthreading.py

    --------                   -------

        8467                   1 file

   amk@nyman:~/src/python$ ./python

   Python 2.3 (#1, Aug 1 2003, 19:54:32) 

   >>> import sys

   >>> sys.path.insert(0, '/tmp/example.zip')  # Add .zip file to front of path

   >>> import jwzthreading

   >>> jwzthreading.__file__

   '/tmp/example.zip/jwzthreading.py'

   >>>

An entry in ``sys.path`` can now be the filename of a ZIP archive. The ZIP

archive can contain any kind of files, but only files named :file:`\*.py`,

:file:`\*.pyc`, or :file:`\*.pyo` can be imported.  If an archive only contains

:file:`\*.py` files, Python will not attempt to modify the archive by adding the

corresponding :file:`\*.pyc` file, meaning that if a ZIP archive doesn't contain

:file:`\*.pyc` files, importing may be rather slow.

A path within the archive can also be specified to only import from a

subdirectory; for example, the path :file:`/tmp/example.zip/lib/` would only

import from the :file:`lib/` subdirectory within the archive.

.. seealso::

   :pep:`273` - Import Modules from Zip Archives

      Written by James C. Ahlstrom,  who also provided an implementation. Python 2.3

      follows the specification in :pep:`273`,  but uses an implementation written by

      Just van Rossum  that uses the import hooks described in :pep:`302`. See section

      :ref:`section-pep302` for a description of the new import hooks.

.. ======================================================================

PEP 277: Unicode file name support for Windows NT

=================================================

On Windows NT, 2000, and XP, the system stores file names as Unicode strings.

Traditionally, Python has represented file names as byte strings, which is

inadequate because it renders some file names inaccessible.

Python now allows using arbitrary Unicode strings (within the limitations of the

file system) for all functions that expect file names, most notably the

:func:`open` built-in function. If a Unicode string is passed to

:func:`os.listdir`, Python now returns a list of Unicode strings.  A new

function, :func:`os.getcwdu`, returns the current directory as a Unicode string.

Byte strings still work as file names, and on Windows Python will transparently

convert them to Unicode using the ``mbcs`` encoding.

Other systems also allow Unicode strings as file names but convert them to byte

strings before passing them to the system, which can cause a :exc:`UnicodeError`

to be raised. Applications can test whether arbitrary Unicode strings are

supported as file names by checking :attr:`os.path.supports_unicode_filenames`,

a Boolean value.

Under MacOS, :func:`os.listdir` may now return Unicode filenames.

.. seealso::

   :pep:`277` - Unicode file name support for Windows NT

      Written by Neil Hodgson; implemented by Neil Hodgson, Martin von Löwis, and Mark

      Hammond.

.. ======================================================================

PEP 278: Universal Newline Support

==================================

The three major operating systems used today are Microsoft Windows, Apple's

Macintosh OS, and the various Unix derivatives.  A minor irritation of cross-

platform work  is that these three platforms all use different characters to

mark the ends of lines in text files.  Unix uses the linefeed (ASCII character

10), MacOS uses the carriage return (ASCII character 13), and Windows uses a

two-character sequence of a carriage return plus a newline.

Python's file objects can now support end of line conventions other than the one

followed by the platform on which Python is running. Opening a file with the

mode ``'U'`` or ``'rU'`` will open a file for reading in universal newline mode.

All three line ending conventions will be translated to a ``'\n'`` in the

strings returned by the various file methods such as :meth:`read` and

:meth:`readline`.

Universal newline support is also used when importing modules and when executing

a file with the :func:`execfile` function.  This means that Python modules can

be shared between all three operating systems without needing to convert the

line-endings.

This feature can be disabled when compiling Python by specifying the

:option:`--without-universal-newlines` switch when running Python's

:program:`configure` script.

.. seealso::

   :pep:`278` - Universal Newline Support

      Written and implemented by Jack Jansen.

.. ======================================================================

.. _section-enumerate:

PEP 279: enumerate()

====================

A new built-in function, :func:`enumerate`, will make certain loops a bit

clearer.  ``enumerate(thing)``, where *thing* is either an iterator or a

sequence, returns a iterator that will return ``(0, thing[0])``, ``(1,

thing[1])``, ``(2, thing[2])``, and so forth.

A common idiom to change every element of a list looks like this::

   for i in range(len(L)):

       item = L[i]

       # ... compute some result based on item ...

       L[i] = result

This can be rewritten using :func:`enumerate` as::

   for i, item in enumerate(L):

       # ... compute some result based on item ...

       L[i] = result

.. seealso::

   :pep:`279` - The enumerate() built-in function

      Written and implemented by Raymond D. Hettinger.

.. ======================================================================

PEP 282: The logging Package

============================

A standard package for writing logs, :mod:`logging`, has been added to Python

2.3.  It provides a powerful and flexible mechanism for generating logging

output which can then be filtered and processed in various ways.  A

configuration file written in a standard format can be used to control the

logging behavior of a program.  Python includes handlers that will write log

records to standard error or to a file or socket, send them to the system log,

or even e-mail them to a particular address; of course, it's also possible to

write your own handler classes.

The :class:`Logger` class is the primary class. Most application code will deal

with one or more :class:`Logger` objects, each one used by a particular

subsystem of the application. Each :class:`Logger` is identified by a name, and

names are organized into a hierarchy using ``.``  as the component separator.

For example, you might have :class:`Logger` instances named ``server``,

``server.auth`` and ``server.network``.  The latter two instances are below

``server`` in the hierarchy.  This means that if you turn up the verbosity for

``server`` or direct ``server`` messages to a different handler, the changes

will also apply to records logged to ``server.auth`` and ``server.network``.

There's also a root :class:`Logger` that's the parent of all other loggers.

For simple uses, the :mod:`logging` package contains some convenience functions

that always use the root log::

   import logging

   logging.debug('Debugging information')

   logging.info('Informational message')

   logging.warning('Warning:config file %s not found', 'server.conf')

   logging.error('Error occurred')

   logging.critical('Critical error -- shutting down')

This produces the following output::

   WARNING:root:Warning:config file server.conf not found

   ERROR:root:Error occurred

   CRITICAL:root:Critical error -- shutting down

In the default configuration, informational and debugging messages are

suppressed and the output is sent to standard error.  You can enable the display

of informational and debugging messages by calling the :meth:`setLevel` method

on the root logger.

Notice the :func:`warning` call's use of string formatting operators; all of the

functions for logging messages take the arguments ``(msg, arg1, arg2, ...)`` and

log the string resulting from ``msg % (arg1, arg2, ...)``.

There's also an :func:`exception` function that records the most recent

traceback.  Any of the other functions will also record the traceback if you

specify a true value for the keyword argument *exc_info*. ::

   def f():

       try:    1/0

       except: logging.exception('Problem recorded')

   f()

This produces the following output::

   ERROR:root:Problem recorded

   Traceback (most recent call last):

     File "t.py", line 6, in f

       1/0

   ZeroDivisionError: integer division or modulo by zero

Slightly more advanced programs will use a logger other than the root logger.

The :func:`getLogger(name)` function is used to get a particular log, creating

it if it doesn't exist yet. :func:`getLogger(None)` returns the root logger. ::

   log = logging.getLogger('server')

    ...

   log.info('Listening on port %i', port)

    ...

   log.critical('Disk full')

    ...

Log records are usually propagated up the hierarchy, so a message logged to

``server.auth`` is also seen by ``server`` and ``root``, but a :class:`Logger`

can prevent this by setting its :attr:`propagate` attribute to :const:`False`.

There are more classes provided by the :mod:`logging` package that can be

customized.  When a :class:`Logger` instance is told to log a message, it

creates a :class:`LogRecord` instance that is sent to any number of different

:class:`Handler` instances.  Loggers and handlers can also have an attached list

of filters, and each filter can cause the :class:`LogRecord` to be ignored or

can modify the record before passing it along.  When they're finally output,

:class:`LogRecord` instances are converted to text by a :class:`Formatter`

class.  All of these classes can be replaced by your own specially-written

classes.

With all of these features the :mod:`logging` package should provide enough