****************************

  What's New in Python 2.5  

****************************

:Author: A.M. Kuchling

.. |release| replace:: 1.01

.. $Id: whatsnew25.tex 56611 2007-07-29 08:26:10Z georg.brandl $

.. Fix XXX comments

This article explains the new features in Python 2.5.  The final release of

Python 2.5 is scheduled for August 2006; :pep:`356` describes the planned

release schedule.

The changes in Python 2.5 are an interesting mix of language and library

improvements. The library enhancements will be more important to Python's user

community, I think, because several widely-useful packages were added.  New

modules include ElementTree for XML processing (:mod:`xml.etree`),

the SQLite database module (:mod:`sqlite`), and the :mod:`ctypes`

module for calling C functions.

The language changes are of middling significance.  Some pleasant new features

were added, but most of them aren't features that you'll use every day.

Conditional expressions were finally added to the language using a novel syntax;

see section :ref:`pep-308`.  The new ':keyword:`with`' statement will make

writing cleanup code easier (section :ref:`pep-343`).  Values can now be passed

into generators (section :ref:`pep-342`).  Imports are now visible as either

absolute or relative (section :ref:`pep-328`).  Some corner cases of exception

handling are handled better (section :ref:`pep-341`).  All these improvements

are worthwhile, but they're improvements to one specific language feature or

another; none of them are broad modifications to Python's semantics.

As well as the language and library additions, other improvements and bugfixes

were made throughout the source tree.  A search through the SVN change logs

finds there were 353 patches applied and 458 bugs fixed between Python 2.4 and

2.5.  (Both figures are likely to be underestimates.)

This article doesn't try to be a complete specification of the new features;

instead changes are briefly introduced using helpful examples.  For full

details, you should always refer to the documentation for Python 2.5 at

http://docs.python.org. If you want to understand the complete implementation

and design rationale, refer to the PEP for a particular new feature.

Comments, suggestions, and error reports for this document are welcome; please

e-mail them to the author or open a bug in the Python bug tracker.

.. ======================================================================

.. _pep-308:

PEP 308: Conditional Expressions

================================

For a long time, people have been requesting a way to write conditional

expressions, which are expressions that return value A or value B depending on

whether a Boolean value is true or false.  A conditional expression lets you

write a single assignment statement that has the same effect as the following::

   if condition:

       x = true_value

   else:

       x = false_value

There have been endless tedious discussions of syntax on both python-dev and

comp.lang.python.  A vote was even held that found the majority of voters wanted

conditional expressions in some form, but there was no syntax that was preferred

by a clear majority. Candidates included C's ``cond ? true_v : false_v``, ``if

cond then true_v else false_v``, and 16 other variations.

Guido van Rossum eventually chose a surprising syntax::

   x = true_value if condition else false_value

Evaluation is still lazy as in existing Boolean expressions, so the order of

evaluation jumps around a bit.  The *condition* expression in the middle is

evaluated first, and the *true_value* expression is evaluated only if the

condition was true.  Similarly, the *false_value* expression is only evaluated

when the condition is false.

This syntax may seem strange and backwards; why does the condition go in the

*middle* of the expression, and not in the front as in C's ``c ? x : y``?  The

decision was checked by applying the new syntax to the modules in the standard

library and seeing how the resulting code read.  In many cases where a

conditional expression is used, one value seems to be the 'common case' and one

value is an 'exceptional case', used only on rarer occasions when the condition

isn't met.  The conditional syntax makes this pattern a bit more obvious::

   contents = ((doc + '\n') if doc else '')

I read the above statement as meaning "here *contents* is  usually assigned a

value of ``doc+'\n'``; sometimes  *doc* is empty, in which special case an empty

string is returned."   I doubt I will use conditional expressions very often

where there  isn't a clear common and uncommon case.

There was some discussion of whether the language should require surrounding

conditional expressions with parentheses.  The decision was made to *not*

require parentheses in the Python language's grammar, but as a matter of style I

think you should always use them. Consider these two statements::

   # First version -- no parens

   level = 1 if logging else 0

   # Second version -- with parens

   level = (1 if logging else 0)

In the first version, I think a reader's eye might group the statement into

'level = 1', 'if logging', 'else 0', and think that the condition decides

whether the assignment to *level* is performed.  The second version reads

better, in my opinion, because it makes it clear that the assignment is always

performed and the choice is being made between two values.

Another reason for including the brackets: a few odd combinations of list

comprehensions and lambdas could look like incorrect conditional expressions.

See :pep:`308` for some examples.  If you put parentheses around your

conditional expressions, you won't run into this case.

.. seealso::

   :pep:`308` - Conditional Expressions

      PEP written by Guido van Rossum and Raymond D. Hettinger; implemented by Thomas

      Wouters.

.. ======================================================================

.. _pep-309:

PEP 309: Partial Function Application

=====================================

The :mod:`functools` module is intended to contain tools for functional-style

programming.

One useful tool in this module is the :func:`partial` function. For programs

written in a functional style, you'll sometimes want to construct variants of

existing functions that have some of the parameters filled in.  Consider a

Python function ``f(a, b, c)``; you could create a new function ``g(b, c)`` that

was equivalent to ``f(1, b, c)``.  This is called "partial function

application".

:func:`partial` takes the arguments ``(function, arg1, arg2, ... kwarg1=value1,

kwarg2=value2)``.  The resulting object is callable, so you can just call it to

invoke *function* with the filled-in arguments.

Here's a small but realistic example::

   import functools

   def log (message, subsystem):

       "Write the contents of 'message' to the specified subsystem."

       print '%s: %s' % (subsystem, message)

       ...

   server_log = functools.partial(log, subsystem='server')

   server_log('Unable to open socket')

Here's another example, from a program that uses PyGTK.  Here a context-

sensitive pop-up menu is being constructed dynamically.  The callback provided

for the menu option is a partially applied version of the :meth:`open_item`

method, where the first argument has been provided. ::

   ...

   class Application:

       def open_item(self, path):

          ...

       def init (self):

           open_func = functools.partial(self.open_item, item_path)

           popup_menu.append( ("Open", open_func, 1) )

Another function in the :mod:`functools` module is the

:func:`update_wrapper(wrapper, wrapped)` function that helps you write well-

behaved decorators.  :func:`update_wrapper` copies the name, module, and

docstring attribute to a wrapper function so that tracebacks inside the wrapped

function are easier to understand.  For example, you might write::

   def my_decorator(f):

       def wrapper(*args, **kwds):

           print 'Calling decorated function'

           return f(*args, **kwds)

       functools.update_wrapper(wrapper, f)

       return wrapper

:func:`wraps` is a decorator that can be used inside your own decorators to copy

the wrapped function's information.  An alternate  version of the previous

example would be::

   def my_decorator(f):

       @functools.wraps(f)

       def wrapper(*args, **kwds):

           print 'Calling decorated function'

           return f(*args, **kwds)

       return wrapper

.. seealso::

   :pep:`309` - Partial Function Application

      PEP proposed and written by Peter Harris; implemented by Hye-Shik Chang and Nick

      Coghlan, with adaptations by Raymond Hettinger.

.. ======================================================================

.. _pep-314:

PEP 314: Metadata for Python Software Packages v1.1

===================================================

Some simple dependency support was added to Distutils.  The :func:`setup`

function now has ``requires``, ``provides``, and ``obsoletes`` keyword

parameters.  When you build a source distribution using the ``sdist`` command,

the dependency information will be recorded in the :file:`PKG-INFO` file.

Another new keyword parameter is ``download_url``, which should be set to a URL

for the package's source code.  This means it's now possible to look up an entry

in the package index, determine the dependencies for a package, and download the

required packages. ::

   VERSION = '1.0'

   setup(name='PyPackage', 

         version=VERSION,

         requires=['numarray', 'zlib (>=1.1.4)'],

         obsoletes=['OldPackage']

         download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'

                       % VERSION),

        )

Another new enhancement to the Python package index at

http://cheeseshop.python.org is storing source and binary archives for a

package.  The new :command:`upload` Distutils command will upload a package to

the repository.

Before a package can be uploaded, you must be able to build a distribution using

the :command:`sdist` Distutils command.  Once that works, you can run ``python

setup.py upload`` to add your package to the PyPI archive.  Optionally you can

GPG-sign the package by supplying the :option:`--sign` and :option:`--identity`

options.

Package uploading was implemented by Martin von Löwis and Richard Jones.

.. seealso::

   :pep:`314` - Metadata for Python Software Packages v1.1

      PEP proposed and written by A.M. Kuchling, Richard Jones, and Fred Drake;

      implemented by Richard Jones and Fred Drake.

.. ======================================================================

.. _pep-328:

PEP 328: Absolute and Relative Imports

======================================

The simpler part of PEP 328 was implemented in Python 2.4: parentheses could now

be used to enclose the names imported from a module using the ``from ... import

...`` statement, making it easier to import many different names.

The more complicated part has been implemented in Python 2.5: importing a module

can be specified to use absolute or package-relative imports.  The plan is to

move toward making absolute imports the default in future versions of Python.

Let's say you have a package directory like this::

   pkg/

   pkg/__init__.py

   pkg/main.py

   pkg/string.py

This defines a package named :mod:`pkg` containing the :mod:`pkg.main` and

:mod:`pkg.string` submodules.

Consider the code in the :file:`main.py` module.  What happens if it executes

the statement ``import string``?  In Python 2.4 and earlier, it will first look

in the package's directory to perform a relative import, finds

:file:`pkg/string.py`, imports the contents of that file as the

:mod:`pkg.string` module, and that module is bound to the name ``string`` in the

:mod:`pkg.main` module's namespace.

That's fine if :mod:`pkg.string` was what you wanted.  But what if you wanted

Python's standard :mod:`string` module?  There's no clean way to ignore

:mod:`pkg.string` and look for the standard module; generally you had to look at

the contents of ``sys.modules``, which is slightly unclean.    Holger Krekel's

:mod:`py.std` package provides a tidier way to perform imports from the standard

library, ``import py ; py.std.string.join()``, but that package isn't available

on all Python installations.

Reading code which relies on relative imports is also less clear, because a

reader may be confused about which module, :mod:`string` or :mod:`pkg.string`,

is intended to be used.  Python users soon learned not to duplicate the names of

standard library modules in the names of their packages' submodules, but you

can't protect against having your submodule's name being used for a new module

added in a future version of Python.

In Python 2.5, you can switch :keyword:`import`'s behaviour to  absolute imports

using a ``from __future__ import absolute_import`` directive.  This absolute-

import behaviour will become the default in a future version (probably Python

2.7).  Once absolute imports  are the default, ``import string`` will always

find the standard library's version. It's suggested that users should begin

using absolute imports as much as possible, so it's preferable to begin writing

``from pkg import string`` in your code.

Relative imports are still possible by adding a leading period  to the module

name when using the ``from ... import`` form::

   # Import names from pkg.string

   from .string import name1, name2

   # Import pkg.string

   from . import string

This imports the :mod:`string` module relative to the current package, so in

:mod:`pkg.main` this will import *name1* and *name2* from :mod:`pkg.string`.

Additional leading periods perform the relative import starting from the parent

of the current package.  For example, code in the :mod:`A.B.C` module can do::

   from . import D                 # Imports A.B.D

   from .. import E                # Imports A.E

   from ..F import G               # Imports A.F.G

Leading periods cannot be used with the ``import modname``  form of the import

statement, only the ``from ... import`` form.

.. seealso::

   :pep:`328` - Imports: Multi-Line and Absolute/Relative

      PEP written by Aahz; implemented by Thomas Wouters.

   http://codespeak.net/py/current/doc/index.html

      The py library by Holger Krekel, which contains the :mod:`py.std` package.

.. ======================================================================

.. _pep-338:

PEP 338: Executing Modules as Scripts

=====================================

The :option:`-m` switch added in Python 2.4 to execute a module as a script

gained a few more abilities.  Instead of being implemented in C code inside the

Python interpreter, the switch now uses an implementation in a new module,

:mod:`runpy`.

The :mod:`runpy` module implements a more sophisticated import mechanism so that

it's now possible to run modules in a package such as :mod:`pychecker.checker`.

The module also supports alternative import mechanisms such as the

:mod:`zipimport` module.  This means you can add a .zip archive's path to

``sys.path`` and then use the :option:`-m` switch to execute code from the

archive.

.. seealso::

   :pep:`338` - Executing modules as scripts

      PEP written and  implemented by Nick Coghlan.

.. ======================================================================

.. _pep-341:

PEP 341: Unified try/except/finally

===================================

Until Python 2.5, the :keyword:`try` statement came in two flavours. You could

use a :keyword:`finally` block to ensure that code is always executed, or one or

more :keyword:`except` blocks to catch  specific exceptions.  You couldn't

combine both :keyword:`except` blocks and a :keyword:`finally` block, because

generating the right bytecode for the combined version was complicated and it

wasn't clear what the semantics of the combined statement should be.

Guido van Rossum spent some time working with Java, which does support the

equivalent of combining :keyword:`except` blocks and a :keyword:`finally` block,

and this clarified what the statement should mean.  In Python 2.5, you can now

write::

   try:

       block-1 ...

   except Exception1:

       handler-1 ...

   except Exception2:

       handler-2 ...

   else:

       else-block

   finally:

       final-block 

The code in *block-1* is executed.  If the code raises an exception, the various

:keyword:`except` blocks are tested: if the exception is of class

:class:`Exception1`, *handler-1* is executed; otherwise if it's of class

:class:`Exception2`, *handler-2* is executed, and so forth.  If no exception is

raised, the *else-block* is executed.

No matter what happened previously, the *final-block* is executed once the code

block is complete and any raised exceptions handled. Even if there's an error in

an exception handler or the *else-block* and a new exception is raised, the code

in the *final-block* is still run.

.. seealso::

   :pep:`341` - Unifying try-except and try-finally

      PEP written by Georg Brandl;  implementation by Thomas Lee.

.. ======================================================================

.. _pep-342:

PEP 342: New Generator Features

===============================

Python 2.5 adds a simple way to pass values *into* a generator. As introduced in

Python 2.3, generators only produce output; once a generator's code was invoked

to create an iterator, there was no way to pass any new information into the

function when its execution is resumed.  Sometimes the ability to pass in some

information would be useful.  Hackish solutions to this include making the

generator's code look at a global variable and then changing the global

variable's value, or passing in some mutable object that callers then modify.

To refresh your memory of basic generators, here's a simple example::

   def counter (maximum):

       i = 0

       while i < maximum:

           yield i

           i += 1

When you call ``counter(10)``, the result is an iterator that returns the values

from 0 up to 9.  On encountering the :keyword:`yield` statement, the iterator

returns the provided value and suspends the function's execution, preserving the

local variables. Execution resumes on the following call to the iterator's

:meth:`next` method, picking up after the :keyword:`yield` statement.

In Python 2.3, :keyword:`yield` was a statement; it didn't return any value.  In

2.5, :keyword:`yield` is now an expression, returning a value that can be

assigned to a variable or otherwise operated on::

   val = (yield i)

I recommend that you always put parentheses around a :keyword:`yield` expression

when you're doing something with the returned value, as in the above example.

The parentheses aren't always necessary, but it's easier to always add them

instead of having to remember when they're needed.

(:pep:`342` explains the exact rules, which are that a :keyword:`yield`\

-expression must always be parenthesized except when it occurs at the top-level

expression on the right-hand side of an assignment.  This means you can write

``val = yield i`` but have to use parentheses when there's an operation, as in

``val = (yield i) + 12``.)

Values are sent into a generator by calling its :meth:`send(value)` method.  The

generator's code is then resumed and the :keyword:`yield` expression returns the

specified *value*.  If the regular :meth:`next` method is called, the

:keyword:`yield` returns :const:`None`.

Here's the previous example, modified to allow changing the value of the

internal counter. ::

   def counter (maximum):

       i = 0

       while i < maximum:

           val = (yield i)

           # If value provided, change counter

           if val is not None:

               i = val

           else:

               i += 1

And here's an example of changing the counter::

   >>> it = counter(10)

   >>> print it.next()

   0

   >>> print it.next()

   1

   >>> print it.send(8)

   8

   >>> print it.next()

   9

   >>> print it.next()

   Traceback (most recent call last):

     File ``t.py'', line 15, in ?

       print it.next()

   StopIteration

:keyword:`yield` will usually return :const:`None`, so you should always check

for this case.  Don't just use its value in expressions unless you're sure that

the :meth:`send` method will be the only method used to resume your generator

function.

In addition to :meth:`send`, there are two other new methods on generators:

* :meth:`throw(type, value=None, traceback=None)` is used to raise an exception

  inside the generator; the exception is raised by the :keyword:`yield` expression

  where the generator's execution is paused.

* :meth:`close` raises a new :exc:`GeneratorExit` exception inside the generator

  to terminate the iteration.  On receiving this exception, the generator's code

  must either raise :exc:`GeneratorExit` or :exc:`StopIteration`.  Catching the

  :exc:`GeneratorExit` exception and returning a value is illegal and will trigger

  a :exc:`RuntimeError`; if the function raises some other exception, that

  exception is propagated to the caller.  :meth:`close` will also be called by

  Python's garbage collector when the generator is garbage-collected.

  If you need to run cleanup code when a :exc:`GeneratorExit` occurs, I suggest

  using a ``try: ... finally:`` suite instead of  catching :exc:`GeneratorExit`.

The cumulative effect of these changes is to turn generators from one-way

producers of information into both producers and consumers.

Generators also become *coroutines*, a more generalized form of subroutines.

Subroutines are entered at one point and exited at another point (the top of the

function, and a :keyword:`return` statement), but coroutines can be entered,

exited, and resumed at many different points (the :keyword:`yield` statements).

We'll have to figure out patterns for using coroutines effectively in Python.

The addition of the :meth:`close` method has one side effect that isn't obvious.

:meth:`close` is called when a generator is garbage-collected, so this means the

generator's code gets one last chance to run before the generator is destroyed.

This last chance means that ``try...finally`` statements in generators can now

be guaranteed to work; the :keyword:`finally` clause will now always get a

chance to run.  The syntactic restriction that you couldn't mix :keyword:`yield`