python core notes

8/9/2019 Python Core Notes

1/191

Page 1of 191

g

ArPython

Version 1.1.7 20 Februrary 2007

Contents

References ......................................................................................................................................... 4

Overview ............................................................................................. Error! Bookmark not defined.

Python language essentials ................................................................................................................ 4

Lexical structure ............................................................................................................................. 4

Objects ........................................................................................................................................... 5

Names ........................................................................................................................................ 7

Assignment ................................................................................................................................. 7

lists .............................................................................................. Error! Bookmark not defined.

Dictionaries ................................................................................. Error! Bookmark not defined.

sets ........................................................................................................................................... 15

Flow control ................................................................................................................................. 17

while ......................................................................................................................................... 17

for ............................................................................................................................................. 17

iterators and generators .......................................................................................................... 25

list comprehensions ................................................................................................................. 32


2/191

Page 2of 191

g

With .......................................................................................................................................... 33

Functions ...................................................................................................................................... 43

Classes .......................................................................................................................................... 50

Why Decorate? ........................................................................................................................ 58

Creating Decorators ................................................................................................................. 59

Objects as Decorators .............................................................................................................. 59

Stacking Decorators ................................................................................................................. 60

Functions as Decorators .......................................................................................................... 61

Decorators with Arguments ..................................................................................................... 62

Function Attributes .................................................................................................................. 63

Practicing "Safe Decs" .............................................................................................................. 63

Putting It All Together .............................................................................................................. 64

Conclusion ................................................................................................................................ 66

Modules, Packages, paths and imports ........................................................................................... 74

Setting import path of a module to any directory ....................................................................... 74

Notes on .pth files ........................................................................................................................ 75

Documentation ................................................................................................................................ 75

Pydoc ............................................................................................................................................ 75

Docutils ........................................................................................................................................ 75

Distributions of Python Software .................................................................................................... 75

The Working Set ........................................................................................................................... 77

The Environment .......................................................................................................................... 77

Python Eggs .................................................................................................................................. 78

Initialization, Development, and Deployment ............................................................................. 78


3/191

Page 3of 191

g

Disutils .......................................................................................................................................... 80

Conversion to Executables ............................................................................................................... 80

PyInstaller .................................................................................................................................... 80

Py2exe .......................................................................................................................................... 81

Recipe for plain python (console) ............................................................................................ 81

Recipe for wxpython (windows) .............................................................................................. 82

How does py2exe work and what are all those files? ............................................................. 82

CGI .................................................................................................................................................... 84

CGI Web Applications with Python, Part One.............................................................................. 86

Opening word documents from Python ........................................................................................ 107

Quick receipt .............................................................................................................................. 107

more detailed receipt ................................................................................................................ 107

Printing word files ...................................................................................................................... 108

Patterns .......................................................................................................................................... 109

Typology ..................................................................................................................................... 109

mvc ......................................................................................................................................... 119

DQueues ......................................................................................................................................... 119

deque objects ........................................................................................................................... 119

Recipes ................................................................................................................................... 123

Heap queue .................................................................................................................................... 125

Event scheduler .............................................................................................................................. 130

Mutual exclusion support .............................................................................................................. 131

Coroutines and Threading ............................................................................................................. 132

Threading ................................................................................................................................... 132


4/191

Page 4of 191

g

Queue ....................................................................................................................................... 133

Python threads - a first example ............................................................................................ 135

THE SAME APPLICATION, WRITTEN USING PYTHON THREADS ............................................. 137

Understanding Threading in Python .......................................................................................... 141

My own coroutine stuff ............................................................................................................. 166

Module threadpool .................................................................................................................... 177

APPENDIX: Type Hierarchy ............................................................................................................. 177

Basic date and time types .............................................................................................................. 188

Decimal floating point arithmetic .................................................................................................. 190

References

PyERef Python Essential Reference (Beazley)

PyNut "Python in a Nutshell 2nd

ed" by Alex Martelli

PyCBK Python cookbook 2nd

ed " by Alex Martelli

PyLib Python 2.5 library

PracPy Practical Python

PyTut Python 2.5 Tutorial

PyLib Python 2.5 Library

Python language essentials

Lexical structure logical lines (consisting of one or more physical lines) with indentation of contiguous to

denote blocks of statements. Each indentation level uses four spaces. Tabs are replaced

by up to 8 spaces. Best is to expand automatically tabs to spaces when editing.

Character set is ascii per default; other character sets are allowed if the first comment

defines the codec (utf-8 or iso-8859-1) with the following coding directives:


5/191

Page 5of 191

g

# -*- coding:utf-8 -*-

or

# encoding: utf-8

Tokens are elementary lexical components resulting as breakup of a logical line:

identifiers, keywords, operators, delimiters, literals

A source file is a sequence of simple and compound statements. The former lies entirely within a

logical line and can be an expression or assignment. The latter controls one or more other

statements and controls their executions, by means of one or more clauses. A clause has a

header of the form : and a body made out of a single or a block of statements.

ObjectsAll data values are objects, having a

Type, which determines which operations are supported, as well as the attributes

(properties) and items associated to the object (for instance, methods –as covered

later on) and whether the object´s value can be altered (mutability).

Unique identity, which is an integer that corresponds to the object´s location in

memory

Content ( also called value)

An object that contains references to other objects is said to be a container or collection.


6/191

Page 6of 191

g

Some objects allow you to change their content (without changing the identity or the type, that

is). Some objects don't allow you to change their content (more below).

You cannot change the identity.

You cannot change the type. Actually, this is quite subtle:

The type is represented by a type object, which knows more about objects of this type

(how many bytes of memory they usually occupy, what methods they have, etc).( actually,

you can change the type under some rather limited circumstances1.)

1 so you can in fact change the type, as long as the internal structure (the "C-level type") is

identical. the types involved must (to quote the checkin messages):

- have the same basic size

- have the same item size

- have the same dict offset

- have the same weaklist offset

- have the same GC flag bit

- have a common base that is the same except for maybe the

dict and weaklist (which may have been added separately at

the same offsets in both types)

- both be heap types

this basically limits the feature to classes defined at the Python level (just like before the

unification); most attempts to use arbitrary types will fail. e.g.

>>> x.__class__ = dict

Traceback (most recent call last):

File "", line 1, in ?

TypeError: __class__ assignment: only for heap types

http://article.gmane.org/gmane.comp.python.general/427452http://article.gmane.org/gmane.comp.python.general/427452http://article.gmane.org/gmane.comp.python.general/427452http://article.gmane.org/gmane.comp.python.general/427452


7/191

Page 7of 191

g

Objects may also have this:

zero or more methods (provided by the type object) zero or more names

Some objects have methods that allow you to change the contents of the object (modify it in place, that is).

Some objects only have methods that allow you to access the contents, not change it. Some objects don't have any

methods at all. Even if they have methods, you can never change the type, nor the identity.

Things like attribute assignment and item references are just syntactic sugar (more below).

Names

The names are a bit different — they're not really properties of the object, and the object itself doesn't know what it's

called. An object can have any number of names, or no name at all.

Names live in namespaces (such as a module namespace, an instance namespace, a function's local namespace).

Namespaces are collections of (name, object reference) pairs (implemented using dictionaries).

When you call a function or a method, its namespace is initialized with the arguments you call it with (the names are

taken from the function's argument list, the objects are those you pass in).

Assignment

Assignment statements modify namespaces, not objects. In other words,

name = 10

>>> class c(list):

... pass

...

>>> x.__class__ = c


File "", line 1, in ?

TypeError: __class__ assignment: 'a' object layout differs from 'c'


8/191

Page 8of 191

g

means that you're adding the name "name" to your local namespace, and making it refer to an integer object containing

the value 10.

If the name is already present, the assignment replaces the original name:

name = 10

name = 20

means that you're first adding the name "name" to the local namespace, and making it refer to an integer object

containing the value 10. You're then replacing the name, making it point to an integer object containing the value 20.

The original "10" object isn't affected by this operation, and it doesn't care.

In contrast, if you do:

name = []

name.append(1)

you're first adding the name "name" to the local namespace, making it refer to an empty list object. This modifies the

namespace. You're then calling a method on that object, telling it to append an integer object to itself. This modifies the

content of the list object, but it doesn't touch the namespace, and it doesn't touch the integer object.

Things like name.attr and name[index] are just syntactic sugar for method calls. The first corresponds to

__setattr__/ __getattr__, the second to __setitem__/ __getitem__ (depending on which side of the assignment they

appear).

The built-in object() returns a bland object , which is the abstract base class for all classes and

objects.

Types may be built-in or user defined. Built-in types are:

None

Numbers int,long,float,complex,bool

Sequences list,tuple,str,xrange,unicode,basestring

Mapping dict

Sets set,frozenset


9/191

Page 9of 191

g

Callable object, types… (see Appendix)

Modules types.ModuleType

Classes object

Types type

Files file

Internal types…. (see Appendix)

Classic Classes types.ClassType , types.InstanceType

The built-in type(obj) accepts any object as argument and returns the type object that is the type

of obj (built in or in the types module). Be aware of the polyphormic feature of the type built-in,

that also accepts the form type(name,bases,dict) to create a new type object (same as defining a

new class).

t=(1,2,3)

l=[1,2,3]

d=dict(a=1,b=2)

s=set()

class C: pass

class Cnew(object): pass

def name(x): return x.__name__

print "type: %s" %type(1)

print "type: %s" %type("")

print "type: %s" %type(l)

print "type: %s" %type(t)

print "type: %s" %type(s)

print "type: %s" %type(d)


10/191

Page 10of 191

g

print "type: %s" %type(object)

print "type: %s" %type(name)

print "type: %s" %type(C)

print "type: %s" %type(Cnew)

emits:

type:

type:

type:

type:

type:

type:

type:

type:

type:

type:

The attributes associated to an object may be retrieved using the dir() built-in. Some objects have

the attribute __name__, like classes, while others do not. Some objects allow that the __name__

attribute be set –in particular the user defined objects.

For instance,

def setName(obj,x):

obj.__name__=str(x)

def getName(x):


11/191

Page 11of 191

g

return x.__name__

c=C()

#c.__name__='my name is c'

#setName(c,'my name is bim bam bola')

print "name: %s" %getName(C)

print "name: %s" %getName(c)

#emits:

name: C


…

File "D:\Python\lib\dump.py", line 47, in getName

return x.__name__

AttributeError: 'C' object has no attribute '__name_

However, if any of the comments setting name is enabled:

#emits:

name: C

name: my name is bim bam bola

There are objects that do not allow to set the __name__ attribute, like built-in types (as lists): you

get the following error when trying to do that on a list l=[1,2,3]:



12/191

Page 12of 191

g

…

l.__setattr__('__name__','hallo')

AttributeError: 'list' object has no attribute '__name__'

and dir(l) renders:

['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__',

'__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__', '__getslice__', '__gt__',

'__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__

', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',

'__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__str__', 'append','count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

a similar thing happens with a dictionary or a set

['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__', '__doc__', '__eq__',

'__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__',

'__le__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduc

e_ex__', '__repr__', '__setattr__', '__setitem__', '__str__', 'clear', 'copy', 'fromkeys', 'get',

'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault',

'update', 'values']


13/191

Page 13of 191

g

The following test script renders interesting results:

#testattributes.py

#class L(object,list):# would raise TypeError: Error when calling the metaclass bases

# Cannot create a consistent method resolution order (MRO) for bases

object, list

class L(list):

pass

l=L([1,2,3])

x=l.pop()

print l

print dir(L)#does not return attribute __name__ although it exists

print L.__name__

#L.__setattr__(L,'__name__','ANTONIO')# would raise: TypeError: can't apply this

__setattr__ to type object

#l.__setattr__('__name__','ANTONIO')#would raise: AttributeError: 'list' object has no

attribute '__name__'

#print l.__name__# would raise: AttributeError: 'L' object has no attribute '__name__'

l.__dict__['__name__']='antonio' #works

print l.__name__

Li=[1,2,3]

print dir (Li)

#print Li.__name__ #would raise: attributeError: 'list' object has no attribute

'__name__'

#Li.__dict__['__name__']='antonio'#would raise: attributeError: 'list' object has no

attribute '__dict__'


14/191

Page 14of 191

g

"""

results:

[1, 2]


'__dict__', '__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__',

'__getslice__',

'__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__',

'__len__',

'__lt__', '__module__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',

'__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__',

'__str__', '__weakref__', 'append', 'count', 'extend', 'index', 'insert', 'pop',

'remove', 'reverse', 'sort']

L

antonio


'__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__', '__getslice__',

'__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__',

'__len__',

'__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',

'__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__',

'__str__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse',

'sort']

"""

Data values (objects) are accessed through references, which is a name that refers to the location

in memory, taking the form of variables, object attributes and items.

A name is bound to an object through an assignment statement (name=value in a simplified form

or more generically target=expression) . Generically, the target may be an identifier , an

attribute reference (obj.name, whereby name is here an attribute name of the object), an

indexing (obj[expr]) or a slicing (obj[start:stop:stride]).


15/191

Page 15of 191

g

References may be unbound using a del statement, followed by one or more taget references.

Objects can be (arbitrarily) categorized as

collections

o lists

o dictionaries

o sets

sets

A set object is an unordered collection of immutable values. Common uses include membership

testing, removing duplicates from a sequence, and computing mathematical operations such as

intersection, union, difference, and symmetric difference.

Like other collections, sets support x in set , len(set ), and for x in set . Being an unordered collection,

sets do not record element position or order of insertion. Accordingly, sets do not support

indexing, slicing, or other sequence-like behavior.

There are currently two builtin set types, set and frozenset. The set type is mutable -- the

contents can be changed using methods like add() and remove(). Since it is mutable, it has

no hash value and cannot be used as either a dictionary key or as an element of another set. The

frozenset type is immutable and hashable -- its contents cannot be altered after is created;

however, it can be used as a dictionary key or as an element of another set.

set( [iterable])


16/191

Page 16of 191

g

return a set whose elements are taken from iterable. The elements must be immutable.

To represent sets of sets, the inner sets should be frozenset objects. If iterable is not

specified, returns a new empty set, set([]).

Instances of set and frozenset provide the following operations:

Operation Equivalent Result

len(s) cardinality of set s

x in s test x for membership in s

x not in s test x for non-membership in s

s.issubset(t ) s = t test whether every element in t is in s

s.union(t ) s | t new set with elements from both s and t

s.intersection(t ) s & t new set with elements common to s and t

s.difference(t ) s - t new set with elements in s but not in t

s.symmetric_difference(t ) s ^ t new set with elements in either s or t but not both

s.copy() new set with a shallow copy of s


17/191

Page 17of 191

g

Flow controlA program's control flow is the order in which the program's code executes. The control flow of a

Python program is regulated by conditional statements, loops, and function calls.

Conditionan and loop control flow statements are: if, while, for .

while

The while statement in Python supports repeated execution of a statement or block of

statements that are controlled by a conditional expression. Here's the syntax for the while statement:

while expression:

statement(s)

A while statement can also include an else clause

for

The for statement in Python supports repeated execution of a statement, or block of statements,

controlled by an iterable expression. Here's the syntax for the for statement:

for target in iterable:

statement(s)

The in keyword is part of the syntax of the for statement and is distinct from the in operator,

which tests membership. A for statement can also include an else clause.

iterable may be any Python expression suitable as an argument to built-in function iter, which

returns an iterator object ( is covered below ). In particular, any sequence is iterable.

target is normally an identifier that names the control variable of the loop; the for statement

successively rebinds this variable to each item of the iterator, in order.


18/191

Page 18of 191

g

The statement or statements that make up the loop body execute once for each item in

iterable (unless the loop ends because an exception is raised or a break or return statement

executes). Note that, since the loop body may contain a break statement to terminate the loop,

this is one case in which you may want to use an unbounded iterable-one that, per se, would

never cease yielding items.

You can also have a target with multiple identifiers, as with an unpacking assignment. In this case,

the iterator's items must then be iterables, each with exactly as many items as there are

identifiers in the target.

Some interesting comments:

understanding Python's "for" statement

One of the things I noticed when skimming through the various reactions to my recent "with"-article

is that some people seem to have a somewhat fuzzy understanding of Python's other block

statement, the good old for-in loop statement. The with statement didn't introduce code blocks in

Python; they've always been there. To rectify this, for-in probably deserves it's own article, so here

we go (but be warned that the following is a bit rough; I reserve the right to tweak it a little over the

next few days).

On the surface, Python's for-in statement is taken right away from Python's predecessor ABC,

where it's described as:

FOR name,... IN train:

commands

Take each element of train in turn

In ABC, what's called statements in Python are known as commands, and sequences are known as

trains. (The whole language is like that, by the way; lots of common mechanisms described using

less-common names. Maybe they thought that renaming everything would make it easier for people

to pick up the subtle details of the language, instead of assuming that everything worked exactly as

other seemingly similar languages, or maybe it only makes sense if you're Dutch.)

http://online.effbot.org/2006_10_01_archive.htm#withhttp://homepages.cwi.nl/~steven/abc/http://homepages.cwi.nl/~steven/abc/http://online.effbot.org/2006_10_01_archive.htm#with


19/191

Page 19of 191

g

Anyway, to take each element (item) from a train (sequence) in turn, we can simply do (using a

psuedo-Python syntax):

name = train[0]

do something with name

name = train[1]


name = train[2]


... etc ...

and keep doing that until we run out of items. When we do, we'll get an IndexError exception, which

tells us that it's time to stop.

And in its simplest and original form, this is exactly what the for-in statement does; when you write

for name in train:


the interpreter will simply fetch train[0] and assign it to name, and then execute the code block. It'll

then fetch train[1], train[2], and so on, until it gets an IndexError.

The code inside the for-in loop is executed in the same scope as the surrounding code; in the

following example:

train = 1, 2, 3

for name in train:

value = name * 10

print value

the variables train, name, and value all live in the same namespace.


20/191

Page 20of 191

g

This is pretty straightforward, of course, but it immediately gets a bit more interesting once you

realize that you can use custom objects as trains. Just implement the __getitem__ method, and you

can control how the loop behaves. The following code:

class MyTrain:

def __getitem__(self, index):

if not condition:

raise IndexError("that's enough!")

value = fetch item identified by index

return value # hand control back to the block

for name in MyTrain():


will run the loop as long as the given condition is true, with values provided by the custom train. In

other words, the do something part is turned into a block of code that's being executed under the

control of the custom sequence object. The above is equivalent to:

index = 0

while True: # run forever

if not condition:

break

name = fetch item identified by index


index = index + 1

except that index is a hidden variable, and the controlling code is placed in a separate object.

You can use this mechanism for everything from generating sequence elements on the fly (like

xrange):


21/191

Page 21of 191

g

class MySequence:


if index > 10:

raise IndexError("that's enough!")

return value * 10 # returns 0, 10, 20, ..., 100

and fetching data from an external source:

class MyTable:


value = fetch item index from database table

if value not found:

raise IndexError("not found")

return value

or from a stream:

class MyFileIterator:


text = get next line from file

if end of file:

raise IndexError("end of file")

return text

to fetching data from some other source:

class MyEventSource:


event = get next event

if event == terminate:

raise IndexError

return event

for event in MyEventSource():


22/191

Page 22of 191

g

process event

It's more explicit in the latter examples, but in all these examples, the code in __getitem__ is

basically treating the block of code inside the for-in loop as an in-lined callback.

Also note how the last two examples don't even bother to look at the index; they just keep calling

the for-in block until they run out of data. Or, less obvious, until they run out of bits in the internal

index variable.

To deal with this, and also avoid the issue with having objects that looks a lot as sequences, but

doesn't support random access, the for-in statement was redesigned in Python 2.2. Instead of using

the __getitem__ interface, for-in now starts by looking for an __iter__ hook. If present, this method

is called, and the resulting object is then used to fetch items, one by one. This new protocol

behaves like this:

obj = train.__iter__()

name = obj.next()


name = obj.next()


...

where obj is an internal variable, and the next method indicates end of data by raising the

StopIterator exception, instead of IndexError. Using a custom object can look something like:

class MyTrain:

def __iter__(self):

return self

def next(self):

if not condition:

raise StopIteration

value = calculate next value

return value # hand control over to the block


23/191

Page 23of 191

g

for name in MyTrain():


(Here, the MyTrain object returns itself, which means that the for-in statement will call MyTrain's

own next method to do the actual work. In some cases, it makes more sense to use an independent

object for the iteration).

Using this mechanism, we can now rewrite the file iterator from above as:

class MyFileIterator:

def __iter__(self):

return self # use myself

def next():

text = get next line from file

if end of file:

raise StopIteration()

return text

and, with very little work, get an object that doesn't support normal indexing, and doesn't break

down if used on a file with more than 2 billion lines.

But what about ordinary sequences, you ask? That's of course easily handled by a wrapper object,

that keeps an internal counter, and maps next calls to __getitem__ calls, in exactly the same way

as the original for-in statement did. Python provides a standard implementation of such an object,

iter, which is used automatically if __iter__ doesn't exist.

This wasn't very difficult, was it?

Footnote: In Python 2.2 and later, several non-sequence objects have been extended to support

the new protocol. For example, you can loop over both text files and dictionaries; the former return

lines of text, the latter dictionary keys.

for line in open("file.txt"):


24/191

Page 24of 191

g

do something with line

for key in my_dict:

do something with key


25/191

Page 25of 191

g

iterators and generators

Interesting objects are iterators and generators.

To iterate means to repeat something several times- what you do with loops. In Python you can

iterate over special objects, iterators, representing a stream of data and returning the data one

element at a time. It must support a method called next() always returning the next element

of the stream: you can only go forward and there is no way to get the previous element.

Formally, an iterator is an object i such that you can call i.next( ) with no arguments.

i.next( ) returns the next item of iterator i or, when iterator i has no more items, raises a

StopIteration exception. The built-in iter() function takes an arbitrary object and tries to

return an iterator that will return the object´s contents or elements raising TypeError if the

object does not support iteration. An object is said to be iterable if you can get an iterator for it.

Any Python sequrence type (lists, tuples, strings) as well as dictionaries are iterable – they will

automatically support creation for an iterator. When you write a class you can allow instances of

the class to be iterators by defining a method next and the attribute __iter__().

Iterators can be built by implicit or explicit calls to built-in function iter. Calling a generator

also returns an iterator.

Notice that after consuming all of the iterator´s output the iterator is exhausted: if you need to

do something different with the same stream, you need to create a new (different) iterator.

L= [1,2,3]; C= (1,2,3); S= set(L) ; D= dict(x=1,y=2,z=3)

class K:

def __init__(self,values):

self.values=values

def next(self):

try:

return self.values.pop()

except:

raise StopIteration

def __iter__(self):

return self

k=K(L)


26/191

Page 26of 191

g

X=k

it1 = iter(X)

print it1

for item in it1: print item

print it1.next()

emits the following, since the last statement referred to an exhausted iterator :

3

2

1


File "D:\Python\TESTS\test.py", line 24, in

print it1.next()

File "D:\Python\TESTS\test.py", line 13, in next

raise StopIteration

StopIteration

The situation does not change making the following change:

for item in it1: print item

it1 = iter(X) # does not change anything – the iterator is exhausted!

print it1.next()

The following file provides iterator tests.

http://localhost/var/www/apps/conversion/tmp/TESTS/itertests.pyhttp://localhost/var/www/apps/conversion/tmp/TESTS/itertests.pyhttp://localhost/var/www/apps/conversion/tmp/TESTS/itertests.py


27/191

Page 27of 191

g

A generator2 is a function whose body contains one or more occurrences of the keyword yield.

When the generator is called, the function body does not execute: instead, it returns a special

2 Generators and iterators are related to the concept of coroutines. Coroutines are more generic than subroutines.

The start of a subroutine is the only point of entry; the start of a coroutine is the first point of entry, and places

within a coroutine following returns (yields) are subsequent points of entry. Subroutines can return only once; in

contrast, coroutines can return (yield) several times. The lifespan of subroutines is dictated by last in, first out (the

last subroutine called is the first to return); in contrast, the lifespan of coroutines is dictated entirely by their use and

need.

Here's a simple example of how coroutines can be useful. Suppose you have a consumer-producer relationship

where one routine creates items and adds them to a queue and another removes items from the queue and uses

them. For reasons of efficiency, you want to add and remove several items at once. The code might look like this:

var q := new queue

coroutine produce

loop

while q is not full

create some new items

add the items to q

yield to consume

coroutine consume

loop

while q is not empty

remove some items from q

use the items

yield to produce

Each coroutine does as much work as it can before yielding control to the other using the yield command. The yield

causes control in the other coroutine to pick up where it left off, but now with the queue modified so that it can do

more work. Although this example is often used to introduce multithreading, it's not necessary to have two threads to

effect this dynamic: the yield statement can be implemented by a branch directly from one routine into the other.

http://en.wikipedia.org/wiki/LIFOhttp://en.wikipedia.org/wiki/LIFOhttp://en.wikipedia.org/wiki/LIFOhttp://en.wikipedia.org/wiki/Multithreadinghttp://en.wikipedia.org/wiki/Multithreadinghttp://en.wikipedia.org/wiki/Multithreadinghttp://en.wikipedia.org/wiki/Multithreadinghttp://en.wikipedia.org/wiki/LIFO


28/191

Page 28of 191

g

iterator object that wraps the function body, its local variables (including its parameters) and

the current point of execution, which is initially the start of the function; when the next method

of this iterator object is called, the function body executes up to the next yield statement (yield

expression). At this point, the function execution is frozen, with current point of execution and

local variables intact, and the expression following yield is returned as the result of the next

method. When next is called again, execution of the function body resumes where it left off,again up to the next yield statement. If the function body ends or executes a return statement,

the iterator raises a StopIteration exception, to indicate that the iteration is finished. Return

should have no parameters3. A generator may be unbounded (returning an infinite stream o

results) or bounded. In Python 2.5 a generator object has the methods send, throw and close.

Python's generators provide a convenient way to implement the iterator protocol. If a container

object's __iter__() method is implemented as a generator, it will automatically return an iterator

object (technically, a generator object) supplying the __iter__() and next() methods.

One framework for coroutines is spasmodic, supporting asynchronous I/O (and other tasks). The SpasmodicEngine

selects tasks (spasmoids) from a (heapqueue based) priority queue. The tasks are Python 2.5 extended generators

(some call them coroutines: PEP 342).

The engine calls task.send() with an appropriate argument. One of the library of tasks is Pollster. Pollster calls poll()

with for tasks that are waiting I/O. Tasks that are ready for I/O are fed to the priority queue.

Spasmodic provides an efficient way to manage a large number of sockets and/or files. Other processing works welltoo, if it can be subdivided into brief spasms.

3 def updown(N):

""" small generator function"""

for x in xrange(1, N): yield x

if x==3: return 5

for x in xrange(N, 0, -1): yield x

for i in updown(4): print I

renders:

>>> File "", line 4 SyntaxError: 'return' with argument inside generator


29/191

Page 29of 191

g

The for statement implicitly calls iter to get an iterator. The following statement:

for x in c:

statement(s)

is exactly equivalent to:

_temporary_iterator = iter(c)

while True:

try: x = _temporary_iterator.next( )

except StopIteration: break

statement(s)

where _temporary_iterator is some arbitrary name that is not used elsewhere in the current scope.

Thus, if iter(c) returns an iterator i such that i.next( ) never raises StopIteration (an unbounded

iterator ), the loop for x in c never terminates (unless the statements in the loop body include suitable

break or return statements, or raise or propagate exceptions). iter(c), in turn, calls special method

c._ _iter_ _( ) to obtain and return an iterator on c.

Many of the best ways to build and manipulate iterators are found in standard library module

itertools.

while and for statements may optionally have a trailing else clause. The statement or block

under that else executes when the loop terminates naturally (at the end of the for iterator, or

when the while loop condition becomes false), but not when the loop terminates prematurely

(via break, return, or an exception). When a loop contains one or more break statements, you

often need to check whether the loop terminates naturally or prematurely. You can use an else

clause on the loop for this purpose:

for x in some_container:

if is_ok(x): break # item x is satisfactory, terminate

loop

else:

print "Warning: no satisfactory item was found in container"

x = None


30/191

Page 30of 191

g

any and all built-in functions in Python 2.5

$ python2.5

Python 2.5 (r25:51908, Sep 27 2006, 12:21:46)

[GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> any

built-in function any

>>> all

built-in function all

>>> print any.__doc__

any(iterable) -> bool

Return True if bool(x) is True for any x in the iterable.

>>> print all.__doc__

all(iterable) -> bool

Return True if bool(x) is True for all values x in the iterable.

>>> any([1,2])

True

>>> all([1,2])

True

>>> any([1,0])

True


31/191

Page 31of 191

g

>>> all([1,0])

False


32/191

Page 32of 191

g

list comprehensions

A common use of a for loop is to inspect each item in an iterable and build a new list by

appending the results of an expression computed on some or all of the items. The expression

form known as a list comprehension lets you code this common idiom concisely and directly.Since a list comprehension is an expression (rather than a block of statements), you can use it

wherever you need an expression (e.g., as an argument in a function call, in a return statement,

or as a subexpression for some other expression).

A list comprehension has the following syntax:

[ expression for target in

iterable lc-clauses ]

target and iterable are the same as in a regular for statement. You must enclose the

expression in parentheses if it indicates a tuple.

lc-clauses is a series of zero or more clauses, each with one of the following forms:

for target in iterable

if expression

target and iterable in each for clause of a list comprehension have the same syntax and

meaning as those in a regular for statement, and the expression in each if clause of a list

comprehension has the same syntax and meaning as the expression in a regular if statement.

A list comprehension is equivalent to a for loop that builds the same list by repeated calls to theresulting list's append method. For example (assigning the list comprehension result to a variable

for clarity):

result1 = [x+1 for x in some_sequence]

is the same as the for loop:

result2 = []

for x in some_sequence:

result2.append(x+1)

Here's a list comprehension that uses an if clause:

result3 = [x+1 for x in some_sequence if x>23]

This list comprehension is the same as a for loop that contains an if statement:

result4 = []

for x in some_sequence:


33/191

Page 33of 191

g

if x>23:

result4.append(x+1)

And here's a list comprehension that uses a for clause:

result5 = [x+y for x in alist for y in another]

This is the same as a for loop with another for loop nested inside:

result6 = []

for x in alist:

for y in another:

result6.append(x+y)

As these examples show, the order of for and if in a list comprehension is the same as in the

equivalent loop, but in the list comprehension, the nesting remains implicit.

With

The with statement is used to wrap the execution of a block with functionality provided by a

separate guard object (see context-managers). This allows common try-except-finally usage

patterns to be encapsulated for convenient reuse.

from __future__ import with_statement # To enable in 2.5

Syntax:

with expression [as target ] :

suite

or:

with expression [as ( target list ) ] :

suite

The expression is evaluated once, and should yield a context guard , which is used to control

execution of the suite. The guard can provide execution-specific data, which is assigned to the

target (or target list).

Note that if a target list is used instead of a single target, the list must be parenthesized.

Here's a more detailed description:

http://pyref.infogami.com/context-managershttp://pyref.infogami.com/context-managershttp://pyref.infogami.com/context-managershttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-suitehttp://pyref.infogami.com/syntax-suitehttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-suitehttp://pyref.infogami.com/syntax-suitehttp://pyref.infogami.com/syntax-suitehttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/syntax-suitehttp://pyref.infogami.com/syntax-targethttp://pyref.infogami.com/syntax-expressionhttp://pyref.infogami.com/context-managers


34/191

Page 34of 191

g

1. The context expression is evaluated, to obtain a context guard .

2. The guard object's __enter__ method is invoked to obtain the context entry value.

3. The context entry value is assigned to the target or target list, if present.

4. The suite is executed.

5. No matter what happens in the suite, the guard object's __exit__ method is invoked.

If an exception caused the suite to be exited, its type, value, and traceback are passed

as arguments to __exit__. Otherwise, three None arguments are supplied.

If the suite was exited due to an exception, and the return value from the __exit__ method is

false, the exception is reraised. If the return value is true, the exception is suppressed, and

execution continues with the statement following the with statement.

If the suite was exited for any reason other than an exception (e.g., by falling off the end of

the suite, or via return, break, or continue), the return value from __exit__ is ignored, and

execution proceeds at the normal location for the kind of exit that was taken.

Note: The with statement guarantees that if the __enter__ method returns without an error,

then __exit__ will always be called. Thus, if an error occurs during the assignment to the

target list, it will be treated the same as an error occurring within the suite would be. See

step 5 above.

Note: In Python 2.5, the with statement is only allowed when the with_statement feature

has been enabled. It will always be enabled in Python 2.6. This __future__ import statement

can be used to enable the feature (see future):

from __future__ import with_statement

See Also: PEP 0343, The "with" statement The specification, background, and examples for

the Python with statement.

with statement considered hard?

The documentation to context managers. The term comes from the draft documentation but

doesn't explain it further. From that draft:

Context managers ensure a particular action is taken to establish the context before the

contained suite is entered, and a second action to clean up the context when the suite is exited.

and from the PEP

the actual state modifications made by the context manager

http://pyref.infogami.com/__enter__http://pyref.infogami.com/__enter__http://pyref.infogami.com/__enter__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/Nonehttp://pyref.infogami.com/Nonehttp://pyref.infogami.com/Nonehttp://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/returnhttp://pyref.infogami.com/returnhttp://pyref.infogami.com/returnhttp://pyref.infogami.com/breakhttp://pyref.infogami.com/breakhttp://pyref.infogami.com/breakhttp://pyref.infogami.com/continuehttp://pyref.infogami.com/continuehttp://pyref.infogami.com/continuehttp://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__enter__http://pyref.infogami.com/__enter__http://pyref.infogami.com/__enter__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/with_statementhttp://pyref.infogami.com/with_statementhttp://pyref.infogami.com/with_statementhttp://pyref.infogami.com/__future__http://pyref.infogami.com/__future__http://pyref.infogami.com/__future__http://pyref.infogami.com/futurehttp://pyref.infogami.com/futurehttp://pyref.infogami.com/futurehttp://www.python.org/peps/pep-0343.htmlhttp://www.python.org/peps/pep-0343.htmlhttp://www.python.org/peps/pep-0343.htmlhttp://www.python.org/peps/pep-0343.htmlhttp://mail.python.org/pipermail/python-dev/2005-July/054658.htmlhttp://mail.python.org/pipermail/python-dev/2005-July/054658.htmlhttp://mail.python.org/pipermail/python-dev/2005-July/054658.htmlhttp://mail.python.org/pipermail/python-dev/2005-July/054658.htmlhttp://www.python.org/peps/pep-0343.htmlhttp://pyref.infogami.com/futurehttp://pyref.infogami.com/__future__http://pyref.infogami.com/with_statementhttp://pyref.infogami.com/__exit__http://pyref.infogami.com/__enter__http://pyref.infogami.com/__exit__http://pyref.infogami.com/continuehttp://pyref.infogami.com/breakhttp://pyref.infogami.com/returnhttp://pyref.infogami.com/__exit__http://pyref.infogami.com/Nonehttp://pyref.infogami.com/__exit__http://pyref.infogami.com/__exit__http://pyref.infogami.com/__enter__


35/191

Page 35of 191

g

All of the examples in the PEP and the what's new document use the with statement for things

like locks, ensuring a socket or file is closed, database transactions and temporarily modifying a

system setting. These are heavy duty things. Deep things. Things most people don't want to messwith. The what's new document even says

Under the hood, the 'with' statement is fairly complicated. Most people will only use 'with' in

company with existing objects and don't need to know these details

I think this viewpoint is wrong, or at least overly limited. I think the PEP is more generally useful

than those examples show and the term "context manager" is too abstract. I also conjecture that

the people working on the PEP were systems developers and not applications developers, hence

the bias towards system/state modification examples. ;)

The with Statement is very much similar to the using block in C#.

consider an example of opening a file and making sure that it is closed after the block ends

with open('/etc/passwd', 'r') as f:

for line in f:

print line

so after the with block ends the f object is completly distroyed and the file is closed.

and like C# the f object should implement some kind of cleanup functions to make it possible for

the with block to call these functions automatically, in C# the object should implement

IDisposable in python

The expression is evaluated and should result in an object called a ``context manager''.

The context manager must have __enter__() and __exit__() methods.

The context manager's __enter__() method is called. The value returned is assigned to

VAR. If no 'as VAR' clause is present, the value is simply discarded.

The code in BLOCK is executed.

If BLOCK raises an exception, the __exit__(type, value, traceback) is called with the

exception details, the same values returned by sys.exc_info(). The method's return value

controls whether the exception is re-raised: any false value re-raises the exception, and

True will result in suppressing it. You'll only rarely want to suppress the exception,

because if you do the author of the code containing the 'with' statement will never realize

anything went wrong.

If BLOCK didn't raise an exception, the __exit__() method is still called, but type, value,

and traceback are all None.


36/191

Page 36of 191

g

the new “with” statement handles the part I care about: making it easier to write code that works

correctly in the case of failures.

The 3 typical use cases are a file that needs to be closed, a lock that needs to be released, and

and a database transaction that needs to be either committed or rolled back. The database case

is the most interesting, since you need to handle success and failure differently, and before

version 2.5 python would not allow you to have try/except/finally. You had to pick either

try/except or try/finally. Python 2.5 also provides a unified try/except/finally, but the “with”

statement is easier to write, and easier to read.

I’ve borrowed an example of what user code would look like for a database connection using the

new “with” statement from the python docs. The idea is that the block of code should run, and

then the transaction should either be committed or rolled back depending on whether the block

exited normally or with an exception:

db_connection = DatabaseConnection()

with db_connection as cursor:

cursor.execute('insert into ...')

cursor.execute('delete from ...')

# ... more operations ...

In order for this to work, it looks like the classes that you are working with need to properly

support a context manager which defines what should happen for success and error. But, not all

classes will need to implement a full blown context manager, the contextlib module allows for an

easy way to add support to existing objects without the need to write a new class.

from contextlib import contextmanager

@contextmanager

def db_transaction (connection):

http://www.python.org/dev/peps/pep-0343/http://www.python.org/dev/peps/pep-0343/http://www.python.org/dev/peps/pep-0341/http://www.python.org/dev/peps/pep-0341/http://www.python.org/dev/peps/pep-0341/http://docs.python.org/dev/whatsnew/pep-343.htmlhttp://docs.python.org/dev/whatsnew/pep-343.htmlhttp://docs.python.org/dev/whatsnew/pep-343.htmlhttp://docs.python.org/dev/whatsnew/pep-343.htmlhttp://www.python.org/dev/peps/pep-0341/http://www.python.org/dev/peps/pep-0343/


37/191

Page 37of 191

g

cursor = connection.cursor()

try:

yield cursor

except:

connection.rollback()

raise

else:

connection.commit()

db = DatabaseConnection()

with db_transaction(db) as cursor:

...

This was one area where Ruby code was much cleaner than Python, so it’s great to see the new

functionality. It’s pretty hard for me to write code which doesn’t touch a file, a database, or

threads, so it will be used a lot.

Examples:

# Public domain

from __future__ import with_statement

import contextlib, time

@contextlib.contextmanager

def accum_time(L):


38/191

Page 38of 191

g

"""

Add time used inside a with block to the value of L[0].

"""

start = time.clock()

try:

yield

finally:

end = time.clock()

L[0] += end - start

# Example: measure time to execute code inside with blocks.

t = [0]

with accum_time(t):

print sum(range(1000000))

with accum_time(t):

print sum(range(2000000))

print 'Time:', t[0]

another description (efBot):

understanding the "with" statement #

Judging from comp.lang.python and other forums, Python 2.5's new with statement seems to be

a bit confusing even for experienced Python programmers.

http://online.effbot.org/2006_10_01_archive.htm#withhttp://online.effbot.org/2006_10_01_archive.htm#withhttp://online.effbot.org/2006_10_01_archive.htm#withhttp://effbot.org/pyref/with.htmhttp://effbot.org/pyref/with.htmhttp://effbot.org/pyref/with.htmhttp://effbot.org/pyref/with.htmhttp://effbot.org/pyref/with.htmhttp://online.effbot.org/2006_10_01_archive.htm#with


39/191

Page 39of 191

g

As most other things in Python, the with statement is actually very simple, once you understand

the problem it's trying to solve. Consider this piece of code:

set things up

try:

do something

finally:

tear things down

Here, "set things up" could be opening a file, or acquiring some sort of external resource, and

"tear things down" would then be closing the file, or releasing or removing the resource. The try-

finally construct guarantees that the "tear things down" part is always executed, even if the code

that does the work doesn't finish.

If you do this a lot, it would be quite convenient if you could put the "set things up" and "tear

things down" code in a library function, to make it easy to reuse. You can of course do something

like

def controlled_execution(callback):

set things up

try:

callback(thing)

finally:

tear things down

def my_function(thing):

do something


40/191

Page 40of 191

g

controlled_execution(my_function)

But that's a bit verbose, especially if you need to modify local variables. Another approach is to

use a one-shot generator, and use the for-in statement to "wrap" the code:

def controlled_execution():

set things up

try:

yield thing

finally:

tear things down

for thing in controlled_execution():

do something with thing

But yield isn't even allowed inside a try-finally in 2.4 and earlier. And while that could be fixed

(and it has been fixed in 2.5), it's still a bit weird to use a loop construct when you know that you

only want to execute something once.

So after contemplating a number of alternatives, GvR and the python-dev team finally came up

with a generalization of the latter, using an object instead of a generator to control the behaviour

of an external piece of code:

class controlled_execution:

def __enter__(self):

set things up

return thing


41/191

Page 41of 191

g

def __exit__(self, type, value, traceback):

tear things down

with controlled_execution() as thing:

some code

Now, when the "with" statement is executed, Python evaluates the expression, calls the

__enter__ method on the resulting value (which is called a "context guard"), and assigns

whatever __enter__ returns to the variable given by as. Python will then execute the code body,

and no matter what happens in that code , call the guard object's __exit__ method.

As an extra bonus, the __exit__ method can look at the exception, if any, and suppress it or act

on it as necessary. To suppress the exception, just return a true value. For example, the following

__exit__ method swallows any TypeError, but lets all other exceptions through:

def __exit__(self, type, value, traceback):

return isinstance(value, TypeError)

In Python 2.5, the file object has been equipped with __enter__ and __exit__ methods; the

former simply returns the file object itself, and the latter closes the file:

>>> f = open("x.txt")

>>> f

>>> f.__enter__()

>>> f.read(1)


42/191

Page 42of 191

g

'X'

>>> f.__exit__(None, None, None)

>>> f.read(1)


File "", line 1, in

ValueError: I/O operation on closed file

so to open a file, process its contents, and make sure to close it, you can simply do:

with open("x.txt") as f:

data = f.read()

do something with data


43/191

Page 43of 191

g

Functions

A function is a group of statements that execute upon request. Python provides many built-in

functions and allows programmers to define their own functions. A request to execute a function

is known as a function call. When you call a function, you can pass arguments that specify dataupon which the function performs its computation. In Python, a function always returns a result

value, either None or a value that represents the results of the computation. Functions defined

within class statements are also known as methods. Functions are objects (values) that are

handled like other objects. Thus, you can pass a function as an argument in a call to another

function. Similarly, a function can return another function as the result of a call. A function, just

like any other object, can be bound to a variable, an item in a container, or an attribute of an

object. Functions can also be keys into a dictionary. For example, if you need to quickly find a

function's inverse given the function, you could define a dictionary whose keys and values are

functions and then make the dictionary bidirectional:inverse = {sin:asin, cos:acos, tan:atan, log:exp}

for f in inverse.keys( ): inverse[inverse[f]] = f

The def statement is the most common way to define a function. def is a single-clause

compound statement with the following syntax:

def function-name( parameters):

statement(s)

function-name is an identifier. It is a variable that gets bound (or rebound) to the function

object when def executes.

parameters is an optional list of identifiers, known as formal parameters or just parameters, that

get bound to the values supplied as arguments when the function is called. In the simplest case, a

function doesn't have any formal parameters, which means the function doesn't take any

arguments when it is called. In this case, the function definition has empty parentheses after

function-name.

When a function does take arguments, parameters

contains one or more identifiers, separated

by commas (,). In this case, each call to the function supplies values, known as arguments,

corresponding to the parameters listed in the function definition. The parameters are local

variables of the function.


44/191

Page 44of 191

g

The def statement sets some attributes of a function object. The attribute func_name, also

accessible as _ _name_ _, refers to the identifier string given as the function name in the def

statement. You may rebind the attribute to any string value, but trying to unbind it raises an

exception. The attribute func_defaults, which you may freely rebind or unbind, refers to the

tuple of default values for the optional parameters (or the empty tuple, if the function has no

optional parameters).

Another function attribute is the documentation string, also known as the docstring. You may use

or rebind a function's docstring attribute as either func_doc or _ _doc_ _. If the first statement

in the function body is a string literal, the compiler binds that string as the function's docstring

attribute

In addition to its predefined attributes, a function object may have other arbitrary attributes. To

create an attribute of a function object, bind a value to the appropriate attribute reference in an

assignment statement after the def statement executes. For example, a function could counthow many times it gets called:

def counter( ):

counter.count += 1

return counter.count counter.count = 0

Note that this is not common usage. More often, when you want to group together some state

(data) and some behavior (code), you should use the object-oriented mechanisms .

The return statement is allowed only inside a function body and can optionally be followed by

an expression. When return executes, the function terminates, and the value of the expression

is the function's result. A function returns None if it terminates by reaching the end of its body or

by executing a return statement that has no expression (or, of course, by executing return

None).

As a matter of style, you should never write a return statement without an expression at the end

of a function body. If some return statements in a function have an expression, all return

statements should have an expression. return None should only be written explicitly to meet

this style requirement. Python does not enforce these stylistic conventions.

A function call is an expression with the following syntax:function-object(arguments)

function-object may be any reference to a function (or other callable) object; most often, it's

the function's name. The parentheses denote the function-call operation itself. arguments, in the

simplest case, is a series of zero or more expressions separated by commas (,), giving values for

the function's corresponding parameters. When the function call executes, the parameters are


45/191

Page 45of 191

g

bound to the argument values, the function body executes, and the value of the function-call

expression is whatever the function returns.

Note that just mentioning a function (or other callable object) does not call it. To call a function

(or other object) without arguments, you must use ( ) after the function's name.

In traditional terms, all argument passing in Python is by value. For example, if you pass a variable

as an argument, Python passes to the function the object (value) to which the variable currently

refers, not "the variable itself." Thus, a function cannot rebind the caller's variables. However, if

you pass a mutable object as an argument, the function may request changes to that object

because Python passes the object itself, not a copy. Rebinding a variable and mutating an object

are totally disjoint concepts. For example:

def f(x, y):

x = 23

y.append(42)

a = 77

b = [99]

f(a, b)

print a, b # prints: 77 [99, 42]

The print statement shows that a is still bound to 77. Function f's rebinding of its parameter x

to23

has no effect onf

's caller, nor, in particular, on the binding of the caller's variable that

happened to be used to pass 77 as the parameter's value. However, the print statement also

shows that b is now bound to [99, 42]. b is still bound to the same list object as before the call,

but that object has mutated, as f has appended 42 to that list object. In either case, f has not

altered the caller's bindings, nor can f alter the number 77, since numbers are immutable.

However, f can alter a list object, since list objects are mutable. In this example, f mutates the

list object that the caller passes to f as the second argument by calling the object's append

method.

Arguments that are just expressions are known as positional arguments. Each positional

argument supplies the value for the parameter that corresponds to it by position (order) in thefunction definition.

In a function call, zero or more positional arguments may be followed by zero or more named

arguments, each with the following syntax:

identifier =expression


46/191

Page 46of 191

g

The identifier must be one of the parameter names used in the def statement for the

function. The expression supplies the value for the parameter of that name. Most built-in

functions do not accept named arguments, you must call such functions with positional

arguments only. However, all normal functions coded in Python accept named as well as

positional arguments, so you may call them in different ways.

A function call must supply, via a positional or a named argument, exactly one value for each

mandatory parameter, and zero or one value for each optional parameter. For example:

def divide(divisor, dividend):

return dividend // divisor print divide(12, 94)

# prints: 7

print divide(dividend=94, divisor=12) # prints: 7

As you can see, the two calls to divide are equivalent. You can pass named arguments for

readability purposes whenever you think that identifying the role of each argument and

controlling the order of arguments enhances your code's clarity.

A common use of named arguments is to bind some optional parameters to specific values, while

letting other optional parameters take default values:

def f(middle, begin='init', end='finis'):

return begin+middle+end print f('tini', end='') #

prints: inittini

Thanks to named argument end='', the caller can specify a value, the empty string '', for f's

third parameter, end, and still let f's second parameter, begin, use its default value, the string

'init'.

At the end of the arguments in a function call, you may optionally use either or both of the

special forms *seq and **dct. If both forms are present, the form with two asterisks must be

last. *seq passes the items of seq to the function as positional arguments (after the normal

positional arguments, if any, that the call gives with the usual syntax). seq may be any iterable.

**dct passes the items of dct to the function as named arguments, where dct must be a

dictionary whose keys are all strings. Each item's key is a parameter name, and the item's value is

the argument's value.


47/191

Page 47of 191

g

Sometimes you want to pass an argument of the form *seq or **dct when the parameters use

similar forms. For example, using the function sum_args defined in that section (and shown again

here), you may want to print the sum of all the values in dictionary d . This is easy with *seq :

def sum_args(*numbers):

return sum(numbers)

print sum_args(*d.values( ))

(Of course, in this case, print sum(d.values( )) would be simpler and more direct!)

However, you may also pass arguments of the form *seq or **dct when calling a function that

does not use the corresponding forms in its parameters. In that case, of course, you must ensure

that iterable seq has the right number of items, or, respectively, that dictionary dct uses the right

names as its keys; otherwise, the call operation raises an exception.

A function's parameters, plus any variables that are bound (by assignment or by other binding

statements, such as def) in the function body, make up the function's local namespace, also

known as local scope. Each of these variables is known as a local variable of the function.

Variables that are not local are known as global variables (in the absence of nested function

definitions). Global variables are attributes of the module object: Whenever a function's local

variable has the same name as a global variable, that name, within the function body, refers to

the local variable, not the global one. We express this by saying that the local variable hides the

global variable of the same name throughout the function body.By default, any variable that is bound within a function body is a local variable of the function. If a

function needs to rebind some global variables, the first statement of the function must be:

global identifiers

where identifiers is one or more identifiers separated by commas (,). The identifiers listed in a

global statement refer to the global variables (i.e., attributes of the module object) that the

function needs to rebind.

Don't use global if the function body just uses a global variable (including mutating the object

bound to that variable if the object is mutable). Use a global statement only if the function body

rebinds a global variable (generally by assigning to the variable's name). As a matter of style,

don't use global unless it's strictly necessary, as its presence will cause readers of your program

to assume the statement is there for some useful purpose. In particular, never use global except

as the first statement in a function body.


48/191

Page 48of 191

g

A def statement within a function body defines a nested function, and the function whose body

includes the def is known as an outer function to the nested one. Code in a nested function's

body may access (but not rebind) local variables of an outer function, also known as free

variables of the nested function.

The simplest way to let a nested function access a value is often not to rely on nested scopes, but

rather to explicitly pass that value as one of the function's arguments. If necessary, the

argument's value can be bound when the nested function is defined by using the value as the

default for an optional argument. For example:

def percent1(a, b, c):

def pc(x, total=a+b+c): return (x*100.0) / total

print "Percentages are:", pc(a), pc(b), pc(c)

Here's the same functionality using nested scopes:def percent2(a, b, c):

def pc(x): return (x*100.0) / (a+b+c)

print "Percentages are:", pc(a), pc(b), pc(c)

In this specific case, percent1 has a tiny advantage: the computation of a+b+c happens only

once, while percent2's inner function pc repeats the computation three times. However, if the

outer function rebinds its local variables between calls to the nested function, repeating the

computation can be necessary. It's therefore advisable to be aware of both approaches, and

choose the most appropriate one case by case.

A nested function that accesses values from outer local variables is also known as a closure. The

following example shows how to build a closure:

def make_adder(augend):

def add(addend):

return addend+augend

return add

Closures are an exception to the general rule that the object-oriented mechanisms are the best

way to bundle together data and code. When you need specifically to construct callable objects,

with some parameters fixed at object construction time, closures can be simpler and more

effective than classes. For example, the result of make_adder(7) is a function that accepts a

single argument and adds 7 to that argument. An outer function that returns a closure is a

"factory" for members of a family of functions distinguished by some parameters, such as the


49/191

Page 49of 191

g

value of argument augend in the previous example, and may often help you avoid code

duplication.


50/191

Page 50of 191

g

Classes

A class is a Python object with several characteristics:

You can call a class object as if it were a function. The call returns another object, known

as an instance of the class; the class is also known as the type of the instance.

A class has arbitrarily named attributes that you can bind and reference.

The values of class attributes can be descriptors (including functions or normal data

objects.

Class attributes bound to functions are also known as methods of the class.

A method can have a special Python-defined name with two leading and two trailing

underscores. Python implicitly invokes such special methods, if a class supplies them,

when various kinds of operations take place on instances of that class. A class can inherit from other classes, meaning it delegates to other class objects the

lookup of attributes that are not found in the class itself.

An instance of a class is a Python object with arbitrarily named attributes that you can bind and

reference. An instance object implicitly delegates to its class the lookup of attributes not found in

the instance itself. The class, in turn, may delegate the lookup to the classes from which it

inherits, if any.

In Python, classes are objects (values) and are handled like other objects. Thus, you can pass a

class as an argument in a call to a function. Similarly, a function can return a class as the result of

a call. A class, just like any other object, can be bound to a variable (local or global), an item in a

container, or an attribute of an object. Classes can also be keys into a dictionary. The fact that

classes are ordinary objects in Python is often expressed by saying that classes are first-class

objects.

A descriptor is any new-style object whose class supplies a special method named _ _get_ _.

Descriptors that are class attributes control the semantics of accessing and setting attributes on

instances of that class. Roughly speaking, when you access an instance attribute, Python obtains

the attribute's value by calling _ _get_ _ on the corresponding descriptor, if any.

If a descriptor's class also supplies a special method named _ _set_ _, then the descriptor is

known as an overriding descriptor (or, by an older and slightly confusing terminology, a data

descriptor); if the descriptor's class supplies only _ _get_ _, and not _ _set_ _, then the

descriptor is known as a nonoverriding (or nondata) descriptor. For example, the class of function


51/191

Page 51of 191

g

objects supplies _ _get_ _, but not _ _set_ _; therefore, function objects are nonoverriding

descriptors. Roughly speaking, when you assign a value to an instance attribute with a

corresponding descriptor that is overriding, Python sets the attribute value by calling _ _set_ _

on the descriptor.

A common task is to create instances of different classes depending on some condition, or to

avoid creating a new instance if an existing one is available for reuse. A common misconception is

that such needs might be met by having _ _init_ _ return a particular object, but such an

approach is absolutely unfeasible: Python raises an exception when _ _init_ _ returns any

value other than None. The best way to implement flexible object creation is by using an ordinary

function rather than calling the class object directly. A function used in this role is known as a

factory function.

Calling a factory function is a flexible approach: a function may return an existing reusable

instance, or create a new instance by calling whatever class is appropriate. Say you have two

almost interchangeable classes (SpecialCase and NormalCase) and want to flexibly generate

instances of either one of them, depending on an argument. The following appropriateCase

factory function allows you to do just that

class SpecialCase(object):

def amethod(self): print "special"

class NormalCase(object):

def amethod(self): print "normal"

def appropriateCase(isnormal=True):

if isnormal: return NormalCase( )

else: return SpecialCase( )

aninstance = appropriateCase(isnormal=False)

aninstance.amethod( ) # prints "special", as desired

The built-in object type is the ancestor of all built-in types and new-style classes. The object

type defines some special methods that implement the default semantics of objects:

_ _new_ _ _ _init_ _


52/191


53/191

Page 53of 191

g

A property is an instance attribute with special functionality. You reference, bind, or unbind the

attribute with the normal syntax (e.g.,

python core notes

Documents