****************************

  What's New in Python 2.1  

****************************

:Author: A.M. Kuchling

.. |release| replace:: 1.01

.. $Id: whatsnew21.tex 50964 2006-07-30 03:03:43Z fred.drake $

Introduction

============

This article explains the new features in Python 2.1.  While there aren't as

many changes in 2.1 as there were in Python 2.0, there are still some pleasant

surprises in store.  2.1 is the first release to be steered through the use of

Python Enhancement Proposals, or PEPs, so most of the sizable changes have

accompanying PEPs that provide more complete documentation and a design

rationale for the change.  This article doesn't attempt to document the new

features completely, but simply provides an overview of the new features for

Python programmers. Refer to the Python 2.1 documentation, or to the specific

PEP, for more details about any new feature that particularly interests you.

One recent goal of the Python development team has been to accelerate the pace

of new releases, with a new release coming every 6 to 9 months. 2.1 is the first

release to come out at this faster pace, with the first alpha appearing in

January, 3 months after the final version of 2.0 was released.

The final release of Python 2.1 was made on April 17, 2001.

.. ======================================================================

PEP 227: Nested Scopes

======================

The largest change in Python 2.1 is to Python's scoping rules.  In Python 2.0,

at any given time there are at most three namespaces used to look up variable

names: local, module-level, and the built-in namespace.  This often surprised

people because it didn't match their intuitive expectations.  For example, a

nested recursive function definition doesn't work::

   def f():

       ...

       def g(value):

           ...

           return g(value-1) + 1

       ...

The function :func:`g` will always raise a :exc:`NameError` exception, because

the binding of the name ``g`` isn't in either its local namespace or in the

module-level namespace.  This isn't much of a problem in practice (how often do

you recursively define interior functions like this?), but this also made using

the :keyword:`lambda` statement clumsier, and this was a problem in practice.

In code which uses :keyword:`lambda` you can often find local variables being

copied by passing them as the default values of arguments. ::

   def find(self, name):

       "Return list of any entries equal to 'name'"

       L = filter(lambda x, name=name: x == name,

                  self.list_attribute)

       return L

The readability of Python code written in a strongly functional style suffers

greatly as a result.

The most significant change to Python 2.1 is that static scoping has been added

to the language to fix this problem.  As a first effect, the ``name=name``

default argument is now unnecessary in the above example.  Put simply, when a

given variable name is not assigned a value within a function (by an assignment,

or the :keyword:`def`, :keyword:`class`, or :keyword:`import` statements),

references to the variable will be looked up in the local namespace of the

enclosing scope.  A more detailed explanation of the rules, and a dissection of

the implementation, can be found in the PEP.

This change may cause some compatibility problems for code where the same

variable name is used both at the module level and as a local variable within a

function that contains further function definitions. This seems rather unlikely

though, since such code would have been pretty confusing to read in the first

place.

One side effect of the change is that the ``from module import *`` and

:keyword:`exec` statements have been made illegal inside a function scope under

certain conditions.  The Python reference manual has said all along that ``from

module import *`` is only legal at the top level of a module, but the CPython

interpreter has never enforced this before.  As part of the implementation of

nested scopes, the compiler which turns Python source into bytecodes has to

generate different code to access variables in a containing scope.  ``from

module import *`` and :keyword:`exec` make it impossible for the compiler to

figure this out, because they add names to the local namespace that are

unknowable at compile time. Therefore, if a function contains function

definitions or :keyword:`lambda` expressions with free variables, the compiler

will flag this by raising a :exc:`SyntaxError` exception.

To make the preceding explanation a bit clearer, here's an example::

   x = 1

   def f():

       # The next line is a syntax error

       exec 'x=2'  

       def g():

           return x

Line 4 containing the :keyword:`exec` statement is a syntax error, since

:keyword:`exec` would define a new local variable named ``x`` whose value should

be accessed by :func:`g`.

This shouldn't be much of a limitation, since :keyword:`exec` is rarely used in

most Python code (and when it is used, it's often a sign of a poor design

anyway).

Compatibility concerns have led to nested scopes being introduced gradually; in

Python 2.1, they aren't enabled by default, but can be turned on within a module

by using a future statement as described in PEP 236.  (See the following section

for further discussion of PEP 236.)  In Python 2.2, nested scopes will become

the default and there will be no way to turn them off, but users will have had

all of 2.1's lifetime to fix any breakage resulting from their introduction.

.. seealso::

   :pep:`227` - Statically Nested Scopes

      Written and implemented by Jeremy Hylton.

.. ======================================================================

PEP 236: __future__ Directives

==============================

The reaction to nested scopes was widespread concern about the dangers of

breaking code with the 2.1 release, and it was strong enough to make the

Pythoneers take a more conservative approach.  This approach consists of

introducing a convention for enabling optional functionality in release N that

will become compulsory in release N+1.

The syntax uses a ``from...import`` statement using the reserved module name

:mod:`__future__`.  Nested scopes can be enabled by the following statement::

   from __future__ import nested_scopes

While it looks like a normal :keyword:`import` statement, it's not; there are

strict rules on where such a future statement can be put. They can only be at

the top of a module, and must precede any Python code or regular

:keyword:`import` statements.  This is because such statements can affect how

the Python bytecode compiler parses code and generates bytecode, so they must

precede any statement that will result in bytecodes being produced.

.. seealso::

   :pep:`236` - Back to the :mod:`__future__`

      Written by Tim Peters, and primarily implemented by Jeremy Hylton.

.. ======================================================================

PEP 207: Rich Comparisons

=========================

In earlier versions, Python's support for implementing comparisons on user-

defined classes and extension types was quite simple. Classes could implement a

:meth:`__cmp__` method that was given two instances of a class, and could only

return 0 if they were equal or +1 or -1 if they weren't; the method couldn't

raise an exception or return anything other than a Boolean value.  Users of

Numeric Python often found this model too weak and restrictive, because in the

number-crunching programs that numeric Python is used for, it would be more

useful to be able to perform elementwise comparisons of two matrices, returning

a matrix containing the results of a given comparison for each element.  If the

two matrices are of different sizes, then the compare has to be able to raise an

exception to signal the error.

In Python 2.1, rich comparisons were added in order to support this need.

Python classes can now individually overload each of the ``<``, ``<=``, ``>``,

``>=``, ``==``, and ``!=`` operations.  The new magic method names are:

+-----------+----------------+

| Operation | Method name    |

+===========+================+

| ``<``     | :meth:`__lt__` |

+-----------+----------------+

| ``<=``    | :meth:`__le__` |

+-----------+----------------+

| ``>``     | :meth:`__gt__` |

+-----------+----------------+

| ``>=``    | :meth:`__ge__` |

+-----------+----------------+

| ``==``    | :meth:`__eq__` |

+-----------+----------------+

| ``!=``    | :meth:`__ne__` |

+-----------+----------------+

(The magic methods are named after the corresponding Fortran operators ``.LT.``.

``.LE.``, &c.  Numeric programmers are almost certainly quite familiar with

these names and will find them easy to remember.)

Each of these magic methods is of the form ``method(self, other)``, where

``self`` will be the object on the left-hand side of the operator, while

``other`` will be the object on the right-hand side.  For example, the

expression ``A < B`` will cause ``A.__lt__(B)`` to be called.

Each of these magic methods can return anything at all: a Boolean, a matrix, a

list, or any other Python object.  Alternatively they can raise an exception if

the comparison is impossible, inconsistent, or otherwise meaningless.

The built-in :func:`cmp(A,B)` function can use the rich comparison machinery,

and now accepts an optional argument specifying which comparison operation to

use; this is given as one of the strings ``"<"``, ``"<="``, ``">"``, ``">="``,

``"=="``, or ``"!="``.  If called without the optional third argument,

:func:`cmp` will only return -1, 0, or +1 as in previous versions of Python;

otherwise it will call the appropriate method and can return any Python object.

There are also corresponding changes of interest to C programmers; there's a new

slot ``tp_richcmp`` in type objects and an API for performing a given rich

comparison.  I won't cover the C API here, but will refer you to PEP 207, or to

2.1's C API documentation, for the full list of related functions.

.. seealso::

   :pep:`207` - Rich Comparisions

      Written by Guido van Rossum, heavily based on earlier work by David Ascher, and

      implemented by Guido van Rossum.

.. ======================================================================

PEP 230: Warning Framework

==========================

Over its 10 years of existence, Python has accumulated a certain number of

obsolete modules and features along the way.  It's difficult to know when a

feature is safe to remove, since there's no way of knowing how much code uses it

--- perhaps no programs depend on the feature, or perhaps many do.  To enable

removing old features in a more structured way, a warning framework was added.

When the Python developers want to get rid of a feature, it will first trigger a

warning in the next version of Python.  The following Python version can then

drop the feature, and users will have had a full release cycle to remove uses of

the old feature.

Python 2.1 adds the warning framework to be used in this scheme.  It adds a

:mod:`warnings` module that provide functions to issue warnings, and to filter

out warnings that you don't want to be displayed. Third-party modules can also

use this framework to deprecate old features that they no longer wish to

support.

For example, in Python 2.1 the :mod:`regex` module is deprecated, so importing

it causes a warning to be printed::

   >>> import regex

   __main__:1: DeprecationWarning: the regex module

            is deprecated; please use the re module

   >>>

Warnings can be issued by calling the :func:`warnings.warn` function::

   warnings.warn("feature X no longer supported")

The first parameter is the warning message; an additional optional parameters

can be used to specify a particular warning category.

Filters can be added to disable certain warnings; a regular expression pattern

can be applied to the message or to the module name in order to suppress a

warning.  For example, you may have a program that uses the :mod:`regex` module

and not want to spare the time to convert it to use the :mod:`re` module right

now.  The warning can be suppressed by calling ::

   import warnings

   warnings.filterwarnings(action = 'ignore',

                           message='.*regex module is deprecated',

                           category=DeprecationWarning,

                           module = '__main__')

This adds a filter that will apply only to warnings of the class

:class:`DeprecationWarning` triggered in the :mod:`__main__` module, and applies

a regular expression to only match the message about the :mod:`regex` module

being deprecated, and will cause such warnings to be ignored.  Warnings can also

be printed only once, printed every time the offending code is executed, or

turned into exceptions that will cause the program to stop (unless the

exceptions are caught in the usual way, of course).

Functions were also added to Python's C API for issuing warnings; refer to PEP

230 or to Python's API documentation for the details.

.. seealso::

   :pep:`5` - Guidelines for Language Evolution

      Written by Paul Prescod, to specify procedures to be followed when removing old

      features from Python.  The policy described in this PEP hasn't been officially

      adopted, but the eventual policy probably won't be too different from Prescod's

      proposal.

   :pep:`230` - Warning Framework

      Written and implemented by Guido van Rossum.

.. ======================================================================

PEP 229: New Build System

=========================

When compiling Python, the user had to go in and edit the :file:`Modules/Setup`

file in order to enable various additional modules; the default set is

relatively small and limited to modules that compile on most Unix platforms.

This means that on Unix platforms with many more features, most notably Linux,

Python installations often don't contain all useful modules they could.

Python 2.0 added the Distutils, a set of modules for distributing and installing

extensions.  In Python 2.1, the Distutils are used to compile much of the

standard library of extension modules, autodetecting which ones are supported on

the current machine.  It's hoped that this will make Python installations easier

and more featureful.

Instead of having to edit the :file:`Modules/Setup` file in order to enable

modules, a :file:`setup.py` script in the top directory of the Python source

distribution is run at build time, and attempts to discover which modules can be

enabled by examining the modules and header files on the system.  If a module is

configured in :file:`Modules/Setup`, the :file:`setup.py` script won't attempt

to compile that module and will defer to the :file:`Modules/Setup` file's

contents.  This provides a way to specific any strange command-line flags or

libraries that are required for a specific platform.

In another far-reaching change to the build mechanism, Neil Schemenauer

restructured things so Python now uses a single makefile that isn't recursive,

instead of makefiles in the top directory and in each of the :file:`Python/`,

:file:`Parser/`, :file:`Objects/`, and :file:`Modules/` subdirectories.  This

makes building Python faster and also makes hacking the Makefiles clearer and

simpler.

.. seealso::

   :pep:`229` - Using Distutils to Build Python

      Written and implemented by A.M. Kuchling.

.. ======================================================================

PEP 205: Weak References

========================

Weak references, available through the :mod:`weakref` module, are a minor but

useful new data type in the Python programmer's toolbox.

Storing a reference to an object (say, in a dictionary or a list) has the side

effect of keeping that object alive forever.  There are a few specific cases

where this behaviour is undesirable, object caches being the most common one,

and another being circular references in data structures such as trees.

For example, consider a memoizing function that caches the results of another

function :func:`f(x)` by storing the function's argument and its result in a

dictionary::

   _cache = {}

   def memoize(x):

       if _cache.has_key(x):

           return _cache[x]

       retval = f(x)

       # Cache the returned object

       _cache[x] = retval

       return retval

This version works for simple things such as integers, but it has a side effect;

the ``_cache`` dictionary holds a reference to the return values, so they'll

never be deallocated until the Python process exits and cleans up This isn't

very noticeable for integers, but if :func:`f` returns an object, or a data

structure that takes up a lot of memory, this can be a problem.

Weak references provide a way to implement a cache that won't keep objects alive

beyond their time.  If an object is only accessible through weak references, the

object will be deallocated and the weak references will now indicate that the

object it referred to no longer exists.  A weak reference to an object *obj* is

created by calling ``wr = weakref.ref(obj)``.  The object being referred to is

returned by calling the weak reference as if it were a function: ``wr()``.  It

will return the referenced object, or ``None`` if the object no longer exists.

This makes it possible to write a :func:`memoize` function whose cache doesn't

keep objects alive, by storing weak references in the cache. ::

   _cache = {}

   def memoize(x):

       if _cache.has_key(x):

           obj = _cache[x]()

           # If weak reference object still exists,

           # return it

           if obj is not None: return obj

       retval = f(x)

       # Cache a weak reference

       _cache[x] = weakref.ref(retval)

       return retval

The :mod:`weakref` module also allows creating proxy objects which behave like

weak references --- an object referenced only by proxy objects is deallocated --

but instead of requiring an explicit call to retrieve the object, the proxy

transparently forwards all operations to the object as long as the object still

exists.  If the object is deallocated, attempting to use a proxy will cause a

:exc:`weakref.ReferenceError` exception to be raised. ::

   proxy = weakref.proxy(obj)

   proxy.attr   # Equivalent to obj.attr

   proxy.meth() # Equivalent to obj.meth()

   del obj

   proxy.attr   # raises weakref.ReferenceError

.. seealso::

   :pep:`205` - Weak References

      Written and implemented by Fred L. Drake, Jr.

.. ======================================================================

PEP 232: Function Attributes

============================

In Python 2.1, functions can now have arbitrary information attached to them.

People were often using docstrings to hold information about functions and

methods, because the ``__doc__`` attribute was the only way of attaching any

information to a function.  For example, in the Zope Web application server,

functions are marked as safe for public access by having a docstring, and in

John Aycock's SPARK parsing framework, docstrings hold parts of the BNF grammar

to be parsed.  This overloading is unfortunate, since docstrings are really

intended to hold a function's documentation; for example, it means you can't

properly document functions intended for private use in Zope.

Arbitrary attributes can now be set and retrieved on functions using the regular

Python syntax::

   def f(): pass

   f.publish = 1

   f.secure = 1

   f.grammar = "A ::= B (C D)*"

The dictionary containing attributes can be accessed as the function's

:attr:`__dict__`. Unlike the :attr:`__dict__` attribute of class instances, in

functions you can actually assign a new dictionary to :attr:`__dict__`, though

the new value is restricted to a regular Python dictionary; you *can't* be

tricky and set it to a :class:`UserDict` instance, or any other random object

that behaves like a mapping.

.. seealso::

   :pep:`232` - Function Attributes

      Written and implemented by Barry Warsaw.

.. ======================================================================

PEP 235: Importing Modules on Case-Insensitive Platforms

========================================================

Some operating systems have filesystems that are case-insensitive, MacOS and

Windows being the primary examples; on these systems, it's impossible to

distinguish the filenames ``FILE.PY`` and ``file.py``, even though they do store

the file's name  in its original case (they're case-preserving, too).

In Python 2.1, the :keyword:`import` statement will work to simulate case-

sensitivity on case-insensitive platforms.  Python will now search for the first

case-sensitive match by default, raising an :exc:`ImportError` if no such file

is found, so ``import file`` will not import a module named ``FILE.PY``.  Case-

insensitive matching can be requested by setting the :envvar:`PYTHONCASEOK`

environment variable before starting the Python interpreter.

.. ======================================================================

PEP 217: Interactive Display Hook

=================================

When using the Python interpreter interactively, the output of commands is

displayed using the built-in :func:`repr` function. In Python 2.1, the variable

:func:`sys.displayhook` can be set to a callable object which will be called

instead of :func:`repr`. For example, you can set it to a special pretty-

printing function::

   >>> # Create a recursive data structure

   ... L = [1,2,3]

   >>> L.append(L)

   >>> L # Show Python's default output

   [1, 2, 3, [...]]

   >>> # Use pprint.pprint() as the display function

   ... import sys, pprint

   >>> sys.displayhook = pprint.pprint

   >>> L

   [1, 2, 3,  <Recursion on list with id=135143996>]

   >>>

.. seealso::

   :pep:`217` - Display Hook for Interactive Use

      Written and implemented by Moshe Zadka.

.. ======================================================================

PEP 208: New Coercion Model

===========================

How numeric coercion is done at the C level was significantly modified.  This

will only affect the authors of C extensions to Python, allowing them more

flexibility in writing extension types that support numeric operations.

Extension types can now set the type flag ``Py_TPFLAGS_CHECKTYPES`` in their

``PyTypeObject`` structure to indicate that they support the new coercion model.

In such extension types, the numeric slot functions can no longer assume that

they'll be passed two arguments of the same type; instead they may be passed two

arguments of differing types, and can then perform their own internal coercion.

If the slot function is passed a type it can't handle, it can indicate the

failure by returning a reference to the ``Py_NotImplemented`` singleton value.

The numeric functions of the other type will then be tried, and perhaps they can

handle the operation; if the other type also returns ``Py_NotImplemented``, then

a :exc:`TypeError` will be raised.  Numeric methods written in Python can also

return ``Py_NotImplemented``, causing the interpreter to act as if the method

did not exist (perhaps raising a :exc:`TypeError`, perhaps trying another

object's numeric methods).