****************************

  What's New in Python 2.0  

****************************

:Author: A.M. Kuchling and Moshe Zadka

.. |release| replace:: 1.02

.. $Id: whatsnew20.tex 50964 2006-07-30 03:03:43Z fred.drake $

Introduction

============

A new release of Python, version 2.0, was released on October 16, 2000. This

article covers the exciting new features in 2.0, highlights some other useful

changes, and points out a few incompatible changes that may require rewriting

code.

Python's development never completely stops between releases, and a steady flow

of bug fixes and improvements are always being submitted. A host of minor fixes,

a few optimizations, additional docstrings, and better error messages went into

2.0; to list them all would be impossible, but they're certainly significant.

Consult the publicly-available CVS logs if you want to see the full list.  This

progress is due to the five developers working for  PythonLabs are now getting

paid to spend their days fixing bugs, and also due to the improved communication

resulting  from moving to SourceForge.

.. ======================================================================

What About Python 1.6?

======================

Python 1.6 can be thought of as the Contractual Obligations Python release.

After the core development team left CNRI in May 2000, CNRI requested that a 1.6

release be created, containing all the work on Python that had been performed at

CNRI.  Python 1.6 therefore represents the state of the CVS tree as of May 2000,

with the most significant new feature being Unicode support.  Development

continued after May, of course, so the 1.6 tree received a few fixes to ensure

that it's forward-compatible with Python 2.0.  1.6 is therefore part of Python's

evolution, and not a side branch.

So, should you take much interest in Python 1.6?  Probably not.  The 1.6final

and 2.0beta1 releases were made on the same day (September 5, 2000), the plan

being to finalize Python 2.0 within a month or so.  If you have applications to

maintain, there seems little point in breaking things by moving to 1.6, fixing

them, and then having another round of breakage within a month by moving to 2.0;

you're better off just going straight to 2.0.  Most of the really interesting

features described in this document are only in 2.0, because a lot of work was

done between May and September.

.. ======================================================================

New Development Process

=======================

The most important change in Python 2.0 may not be to the code at all, but to

how Python is developed: in May 2000 the Python developers began using the tools

made available by SourceForge for storing  source code, tracking bug reports,

and managing the queue of patch submissions.  To report bugs or submit patches

for Python 2.0, use the bug tracking and patch manager tools available from

Python's project page, located at http://sourceforge.net/projects/python/.

The most important of the services now hosted at SourceForge is the Python CVS

tree, the version-controlled repository containing the source code for Python.

Previously, there were roughly 7 or so people who had write access to the CVS

tree, and all patches had to be inspected and checked in by one of the people on

this short list. Obviously, this wasn't very scalable.  By moving the CVS tree

to SourceForge, it became possible to grant write access to more people; as of

September 2000 there were 27 people able to check in changes, a fourfold

increase.  This makes possible large-scale changes that wouldn't be attempted if

they'd have to be filtered through the small group of core developers.  For

example, one day Peter Schneider-Kamp took it into his head to drop K&R C

compatibility and convert the C source for Python to ANSI C. After getting

approval on the python-dev mailing list, he launched into a flurry of checkins

that lasted about a week, other developers joined in to help, and the job was

done.  If there were only 5 people with write access, probably that task would

have been viewed as "nice, but not worth the time and effort needed" and it

would never have gotten done.

The shift to using SourceForge's services has resulted in a remarkable increase

in the speed of development.  Patches now get submitted, commented on, revised

by people other than the original submitter, and bounced back and forth between

people until the patch is deemed worth checking in.  Bugs are tracked in one

central location and can be assigned to a specific person for fixing, and we can

count the number of open bugs to measure progress.  This didn't come without a

cost: developers now have more e-mail to deal with, more mailing lists to

follow, and special tools had to be written for the new environment. For

example, SourceForge sends default patch and bug notification e-mail messages

that are completely unhelpful, so Ka-Ping Yee wrote an HTML screen-scraper that

sends more useful messages.

The ease of adding code caused a few initial growing pains, such as code was

checked in before it was ready or without getting clear agreement from the

developer group.  The approval process that has emerged is somewhat similar to

that used by the Apache group. Developers can vote +1, +0, -0, or -1 on a patch;

+1 and -1 denote acceptance or rejection, while +0 and -0 mean the developer is

mostly indifferent to the change, though with a slight positive or negative

slant.  The most significant change from the Apache model is that the voting is

essentially advisory, letting Guido van Rossum, who has Benevolent Dictator For

Life status, know what the general opinion is. He can still ignore the result of

a vote, and approve or reject a change even if the community disagrees with him.

Producing an actual patch is the last step in adding a new feature, and is

usually easy compared to the earlier task of coming up with a good design.

Discussions of new features can often explode into lengthy mailing list threads,

making the discussion hard to follow, and no one can read every posting to

python-dev.  Therefore, a relatively formal process has been set up to write

Python Enhancement Proposals (PEPs), modelled on the Internet RFC process.  PEPs

are draft documents that describe a proposed new feature, and are continually

revised until the community reaches a consensus, either accepting or rejecting

the proposal.  Quoting from the introduction to PEP 1, "PEP Purpose and

Guidelines":

.. epigraph::

   PEP stands for Python Enhancement Proposal.  A PEP is a design document

   providing information to the Python community, or describing a new feature for

   Python.  The PEP should provide a concise technical specification of the feature

   and a rationale for the feature.

   We intend PEPs to be the primary mechanisms for proposing new features, for

   collecting community input on an issue, and for documenting the design decisions

   that have gone into Python.  The PEP author is responsible for building

   consensus within the community and documenting dissenting opinions.

Read the rest of PEP 1 for the details of the PEP editorial process, style, and

format.  PEPs are kept in the Python CVS tree on SourceForge, though they're not

part of the Python 2.0 distribution, and are also available in HTML form from

http://www.python.org/peps/.  As of September 2000, there are 25 PEPS, ranging

from PEP 201, "Lockstep Iteration", to PEP 225, "Elementwise/Objectwise

Operators".

.. ======================================================================

Unicode

=======

The largest new feature in Python 2.0 is a new fundamental data type: Unicode

strings.  Unicode uses 16-bit numbers to represent characters instead of the

8-bit number used by ASCII, meaning that 65,536 distinct characters can be

supported.

The final interface for Unicode support was arrived at through countless often-

stormy discussions on the python-dev mailing list, and mostly implemented by

Marc-André Lemburg, based on a Unicode string type implementation by Fredrik

Lundh.  A detailed explanation of the interface was written up as :pep:`100`,

"Python Unicode Integration". This article will simply cover the most

significant points about the Unicode interfaces.

In Python source code, Unicode strings are written as ``u"string"``.  Arbitrary

Unicode characters can be written using a new escape sequence, ``\uHHHH``, where

*HHHH* is a 4-digit hexadecimal number from 0000 to FFFF.  The existing

``\xHHHH`` escape sequence can also be used, and octal escapes can be used for

characters up to U+01FF, which is represented by ``\777``.

Unicode strings, just like regular strings, are an immutable sequence type.

They can be indexed and sliced, but not modified in place. Unicode strings have

an ``encode( [encoding] )`` method that returns an 8-bit string in the desired

encoding.  Encodings are named by strings, such as ``'ascii'``, ``'utf-8'``,

``'iso-8859-1'``, or whatever.  A codec API is defined for implementing and

registering new encodings that are then available throughout a Python program.

If an encoding isn't specified, the default encoding is usually 7-bit ASCII,

though it can be changed for your Python installation by calling the

:func:`sys.setdefaultencoding(encoding)` function in a customised version of

:file:`site.py`.

Combining 8-bit and Unicode strings always coerces to Unicode, using the default

ASCII encoding; the result of ``'a' + u'bc'`` is ``u'abc'``.

New built-in functions have been added, and existing built-ins modified to

support Unicode:

* ``unichr(ch)`` returns a Unicode string 1 character long, containing the

  character *ch*.

* ``ord(u)``, where *u* is a 1-character regular or Unicode string, returns the

  number of the character as an integer.

* ``unicode(string [, encoding]  [, errors] )`` creates a Unicode string

  from an 8-bit string.  ``encoding`` is a string naming the encoding to use. The

  ``errors`` parameter specifies the treatment of characters that are invalid for

  the current encoding; passing ``'strict'`` as the value causes an exception to

  be raised on any encoding error, while ``'ignore'`` causes errors to be silently

  ignored and ``'replace'`` uses U+FFFD, the official replacement character, in

  case of any problems.

* The :keyword:`exec` statement, and various built-ins such as ``eval()``,

  ``getattr()``, and ``setattr()`` will also accept Unicode strings as well as

  regular strings.  (It's possible that the process of fixing this missed some

  built-ins; if you find a built-in function that accepts strings but doesn't

  accept Unicode strings at all, please report it as a bug.)

A new module, :mod:`unicodedata`, provides an interface to Unicode character

properties.  For example, ``unicodedata.category(u'A')`` returns the 2-character

string 'Lu', the 'L' denoting it's a letter, and 'u' meaning that it's

uppercase. ``unicodedata.bidirectional(u'\u0660')`` returns 'AN', meaning that

U+0660 is an Arabic number.

The :mod:`codecs` module contains functions to look up existing encodings and

register new ones.  Unless you want to implement a new encoding, you'll most

often use the :func:`codecs.lookup(encoding)` function, which returns a

4-element tuple: ``(encode_func, decode_func, stream_reader, stream_writer)``.

* *encode_func* is a function that takes a Unicode string, and returns a 2-tuple

  ``(string, length)``.  *string* is an 8-bit string containing a portion (perhaps

  all) of the Unicode string converted into the given encoding, and *length* tells

  you how much of the Unicode string was converted.

* *decode_func* is the opposite of *encode_func*, taking an 8-bit string and

  returning a 2-tuple ``(ustring, length)``, consisting of the resulting Unicode

  string *ustring* and the integer *length* telling how much of the 8-bit string

  was consumed.

* *stream_reader* is a class that supports decoding input from a stream.

  *stream_reader(file_obj)* returns an object that supports the :meth:`read`,

  :meth:`readline`, and :meth:`readlines` methods.  These methods will all

  translate from the given encoding and return Unicode strings.

* *stream_writer*, similarly, is a class that supports encoding output to a

  stream.  *stream_writer(file_obj)* returns an object that supports the

  :meth:`write` and :meth:`writelines` methods.  These methods expect Unicode

  strings, translating them to the given encoding on output.

For example, the following code writes a Unicode string into a file,  encoding

it as UTF-8::

   import codecs

   unistr = u'\u0660\u2000ab ...'

   (UTF8_encode, UTF8_decode,

    UTF8_streamreader, UTF8_streamwriter) = codecs.lookup('UTF-8')

   output = UTF8_streamwriter( open( '/tmp/output', 'wb') )

   output.write( unistr )

   output.close()

The following code would then read UTF-8 input from the file::

   input = UTF8_streamreader( open( '/tmp/output', 'rb') )

   print repr(input.read())

   input.close()

Unicode-aware regular expressions are available through the :mod:`re` module,

which has a new underlying implementation called SRE written by Fredrik Lundh of

Secret Labs AB.

A ``-U`` command line option was added which causes the Python compiler to

interpret all string literals as Unicode string literals. This is intended to be

used in testing and future-proofing your Python code, since some future version

of Python may drop support for 8-bit strings and provide only Unicode strings.

.. ======================================================================

List Comprehensions

===================

Lists are a workhorse data type in Python, and many programs manipulate a list

at some point.  Two common operations on lists are to loop over them, and either

pick out the elements that meet a certain criterion, or apply some function to

each element.  For example, given a list of strings, you might want to pull out

all the strings containing a given substring, or strip off trailing whitespace

from each line.

The existing :func:`map` and :func:`filter` functions can be used for this

purpose, but they require a function as one of their arguments.  This is fine if

there's an existing built-in function that can be passed directly, but if there

isn't, you have to create a little function to do the required work, and

Python's scoping rules make the result ugly if the little function needs

additional information.  Take the first example in the previous paragraph,

finding all the strings in the list containing a given substring.  You could

write the following to do it::

   # Given the list L, make a list of all strings 

   # containing the substring S.

   sublist = filter( lambda s, substring=S: 

                        string.find(s, substring) != -1,

   	          L)

Because of Python's scoping rules, a default argument is used so that the

anonymous function created by the :keyword:`lambda` statement knows what

substring is being searched for.  List comprehensions make this cleaner::

   sublist = [ s for s in L if string.find(s, S) != -1 ]

List comprehensions have the form::

   [ expression for expr in sequence1 

                for expr2 in sequence2 ...

   	     for exprN in sequenceN

                if condition ]

The :keyword:`for`...\ :keyword:`in` clauses contain the sequences to be

iterated over.  The sequences do not have to be the same length, because they

are *not* iterated over in parallel, but from left to right; this is explained

more clearly in the following paragraphs.  The elements of the generated list

will be the successive values of *expression*.  The final :keyword:`if` clause

is optional; if present, *expression* is only evaluated and added to the result

if *condition* is true.

To make the semantics very clear, a list comprehension is equivalent to the

following Python code::

   for expr1 in sequence1:

       for expr2 in sequence2:

       ...

           for exprN in sequenceN:

                if (condition):

                     # Append the value of 

                     # the expression to the 

                     # resulting list.

This means that when there are multiple :keyword:`for`...\ :keyword:`in`

clauses, the resulting list will be equal to the product of the lengths of all

the sequences.  If you have two lists of length 3, the output list is 9 elements

long::

   seq1 = 'abc'

   seq2 = (1,2,3)

   >>> [ (x,y) for x in seq1 for y in seq2]

   [('a', 1), ('a', 2), ('a', 3), ('b', 1), ('b', 2), ('b', 3), ('c', 1),

   ('c', 2), ('c', 3)]

To avoid introducing an ambiguity into Python's grammar, if *expression* is

creating a tuple, it must be surrounded with parentheses.  The first list

comprehension below is a syntax error, while the second one is correct::

   # Syntax error

   [ x,y for x in seq1 for y in seq2]

   # Correct

   [ (x,y) for x in seq1 for y in seq2]

The idea of list comprehensions originally comes from the functional programming

language Haskell (http://www.haskell.org).  Greg Ewing argued most effectively

for adding them to Python and wrote the initial list comprehension patch, which

was then discussed for a seemingly endless time on the python-dev mailing list

and kept up-to-date by Skip Montanaro.

.. ======================================================================

Augmented Assignment

====================

Augmented assignment operators, another long-requested feature, have been added

to Python 2.0.  Augmented assignment operators include ``+=``, ``-=``, ``*=``,

and so forth.  For example, the statement ``a += 2`` increments the value of the

variable  ``a`` by 2, equivalent to the slightly lengthier ``a = a + 2``.

The full list of supported assignment operators is ``+=``, ``-=``, ``*=``,

``/=``, ``%=``, ``**=``, ``&=``, ``|=``, ``^=``, ``>>=``, and ``<<=``.  Python

classes can override the augmented assignment operators by defining methods

named :meth:`__iadd__`, :meth:`__isub__`, etc.  For example, the following

:class:`Number` class stores a number and supports using += to create a new

instance with an incremented value.

.. The empty groups below prevent conversion to guillemets.

::

   class Number:

       def __init__(self, value):

           self.value = value

       def __iadd__(self, increment):

   	return Number( self.value + increment)

   n = Number(5)

   n += 3

   print n.value

The :meth:`__iadd__` special method is called with the value of the increment,

and should return a new instance with an appropriately modified value; this

return value is bound as the new value of the variable on the left-hand side.

Augmented assignment operators were first introduced in the C programming

language, and most C-derived languages, such as :program:`awk`, C++, Java, Perl,

and PHP also support them.  The augmented assignment patch was implemented by

Thomas Wouters.

.. ======================================================================

String Methods

==============

Until now string-manipulation functionality was in the :mod:`string` module,

which was usually a front-end for the :mod:`strop` module written in C.  The

addition of Unicode posed a difficulty for the :mod:`strop` module, because the

functions would all need to be rewritten in order to accept either 8-bit or

Unicode strings.  For functions such as :func:`string.replace`, which takes 3

string arguments, that means eight possible permutations, and correspondingly

complicated code.

Instead, Python 2.0 pushes the problem onto the string type, making string

manipulation functionality available through methods on both 8-bit strings and

Unicode strings.   ::

   >>> 'andrew'.capitalize()

   'Andrew'

   >>> 'hostname'.replace('os', 'linux')

   'hlinuxtname'

   >>> 'moshe'.find('sh')

   2

One thing that hasn't changed, a noteworthy April Fools' joke notwithstanding,

is that Python strings are immutable. Thus, the string methods return new

strings, and do not modify the string on which they operate.

The old :mod:`string` module is still around for backwards compatibility, but it

mostly acts as a front-end to the new string methods.

Two methods which have no parallel in pre-2.0 versions, although they did exist

in JPython for quite some time, are :meth:`startswith` and :meth:`endswith`.

``s.startswith(t)`` is equivalent to ``s[:len(t)] == t``, while

``s.endswith(t)`` is equivalent to ``s[-len(t):] == t``.

One other method which deserves special mention is :meth:`join`.  The

:meth:`join` method of a string receives one parameter, a sequence of strings,

and is equivalent to the :func:`string.join` function from the old :mod:`string`

module, with the arguments reversed. In other words, ``s.join(seq)`` is

equivalent to the old ``string.join(seq, s)``.

.. ======================================================================

Garbage Collection of Cycles

============================

The C implementation of Python uses reference counting to implement garbage

collection.  Every Python object maintains a count of the number of references

pointing to itself, and adjusts the count as references are created or

destroyed.  Once the reference count reaches zero, the object is no longer

accessible, since you need to have a reference to an object to access it, and if

the count is zero, no references exist any longer.

Reference counting has some pleasant properties: it's easy to understand and

implement, and the resulting implementation is portable, fairly fast, and reacts

well with other libraries that implement their own memory handling schemes.  The

major problem with reference counting is that it sometimes doesn't realise that

objects are no longer accessible, resulting in a memory leak.  This happens when

there are cycles of references.

Consider the simplest possible cycle,  a class instance which has a reference to

itself::

   instance = SomeClass()

   instance.myself = instance

After the above two lines of code have been executed, the reference count of

``instance`` is 2; one reference is from the variable named ``'instance'``, and

the other is from the ``myself`` attribute of the instance.

If the next line of code is ``del instance``, what happens?  The reference count

of ``instance`` is decreased by 1, so it has a reference count of 1; the

reference in the ``myself`` attribute still exists.  Yet the instance is no

longer accessible through Python code, and it could be deleted.  Several objects

can participate in a cycle if they have references to each other, causing all of

the objects to be leaked.

Python 2.0 fixes this problem by periodically executing a cycle detection

algorithm which looks for inaccessible cycles and deletes the objects involved.

A new :mod:`gc` module provides functions to perform a garbage collection,

obtain debugging statistics, and tuning the collector's parameters.

Running the cycle detection algorithm takes some time, and therefore will result

in some additional overhead.  It is hoped that after we've gotten experience

with the cycle collection from using 2.0, Python 2.1 will be able to minimize

the overhead with careful tuning.  It's not yet obvious how much performance is

lost, because benchmarking this is tricky and depends crucially on how often the

program creates and destroys objects.  The detection of cycles can be disabled

when Python is compiled, if you can't afford even a tiny speed penalty or

suspect that the cycle collection is buggy, by specifying the

:option:`--without-cycle-gc` switch when running the :program:`configure`

script.

Several people tackled this problem and contributed to a solution.  An early

implementation of the cycle detection approach was written by Toby Kelsey.  The

current algorithm was suggested by Eric Tiedemann during a visit to CNRI, and

Guido van Rossum and Neil Schemenauer wrote two different implementations, which

were later integrated by Neil.  Lots of other people offered suggestions along

the way; the March 2000 archives of the python-dev mailing list contain most of

the relevant discussion, especially in the threads titled "Reference cycle

collection for Python" and "Finalization again".

.. ======================================================================

Other Core Changes

==================

Various minor changes have been made to Python's syntax and built-in functions.

None of the changes are very far-reaching, but they're handy conveniences.

Minor Language Changes

----------------------

A new syntax makes it more convenient to call a given function with a tuple of

arguments and/or a dictionary of keyword arguments. In Python 1.5 and earlier,

you'd use the :func:`apply` built-in function: ``apply(f, args, kw)`` calls the

function :func:`f` with the argument tuple *args* and the keyword arguments in

the dictionary *kw*.  :func:`apply`  is the same in 2.0, but thanks to a patch

from Greg Ewing, ``f(*args, **kw)`` as a shorter and clearer way to achieve the

same effect.  This syntax is symmetrical with the syntax for defining

functions::

   def f(*args, **kw):

       # args is a tuple of positional args,

       # kw is a dictionary of keyword args

       ...

The :keyword:`print` statement can now have its output directed to a file-like

object by following the :keyword:`print` with  ``>> file``, similar to the

redirection operator in Unix shells. Previously you'd either have to use the

:meth:`write` method of the file-like object, which lacks the convenience and

simplicity of :keyword:`print`, or you could assign a new value to

``sys.stdout`` and then restore the old value.  For sending output to standard

error, it's much easier to write this::

   print >> sys.stderr, "Warning: action field not supplied"

Modules can now be renamed on importing them, using the syntax ``import module