metaclass programming in python

Metaclass Programming in PythonAdvanced OO concepts

Juan Manuel Gimeno Illa

[email protected]

December 2008

J.M.Gimeno ([email protected]) Metaclasses December 2008 1 / 21

This is dangerous, so don’t do it at home :-)

[Metaclasses] are deeper magic than 99% of users should everworry about. If you wonder whether you need them, you don’t(the people who actually need them know with certainty thatthey need them, and don’t need an explanation about why).

Tim Peters (c.l.p post 2002-12-22)

Outline

1 Basic Concepts

2 How classes are createdInside class statementMetaclass ProgrammingMetaclass Conflicts

3 More Examples

4 Concluding Remarks

Basic Concepts

Objects and Classes

Classes are templates used to create objectsI that share structure (attributes) and behaviour (methods)I but have different state (attributes’ values)

For object obj, type(obj) is the class or type of obj

For object obj, type(type(obj)) is the metaclass or metatype ofobj

(Sometimes type(type(obj)) is referred as the metaclass oftype(obj) but this is incorrect)

The same way that classes shape objects, metaclasses shape classes.

For instance, in Python, the difference between new and old classesresides in the metaclass used to create the classes

(Note that the difference in metaclass can affect the objects of theclasses. For instance, instances of new-style classes behave differentlythan those of old-style ones)

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

Objects and Classes

Basic Concepts

New and Old Classes

>>> class New(object): pass...>>> class Old: pass...>>> n = New()>>> o = Old()>>> type(n)<class ’ main .New’>>>> type(o)<type ’instance’>>>> type(New)<type ’type’>>>> type(Old)<type ’classobj’>

Differences in new andold style objects (forinstance, the mro ofattributes) . . .

. . . are due to thedifferent metaclassesthey have

Basic Concepts

New and Old Classes

Basic Concepts

New and Old Classes

Basic Concepts

New and Old Classes

Basic Concepts

New and Old Classes

How classes are created Inside class statement

The built-in type

type(name, bases, attribs)(a.k.a types.TypeType) creates anew-style class object

name is a string containing the name of the class(C. name )

bases is a tuple containing the base classes for the new class(C. bases )

attribs is a dictionary pairing the attribute’s names withtheir corresponding values

So the usual class statement (when dealing with new-style classes)

class Class(base1, base2, ...):statements

calls type with parameters ‘‘Class’’, (base1, base2, ...) anda dictionary containing the bindings created by the execution of thestatements.

The built-in type

Inside the class statement

When one defines a new-style class:

>>> class Class1(object):... attrib = 1... def __init__(self, n):... self.n = n

One actually executes:

>>> class_name = "Class1">>> class_bases = (object,)>>> class_dict = {}>>> class_body = """attrib = 1def __init__(self, n):

self.n = n""">>> exec class_body in globals(), class_dict>>> locals()[class_name] = type(class_name, class_bases, class_dict)

The full truth of the class statement

To execute a class statement:

1 Python first collects the class name in a string n, the base classes intoa tuple t and executes the body in dictionary d

2 Then determines the metaclass to use for the new class object C1 When metaclass is a key in d, then M = d[’ metaclass ’]2 When t is nonempty, then M is the leafmost1 metaclass among all the

metaclasses of C’s bases that are not old-style classes3 If the current module has a global variable metaclass , then M is

the value of this variable4 Failing all of these, M defaults to types.ClassType, that is, an

old-style class is created

3 Finally, C = M(n, t, d)

1If the metaclasses of C’s bases’ do not form an inheritance lattice including its lowerbound - i.e. if there is no leafmost metaclass - Python raises and exception diagnosingthis metatype conflict.

How classes are created Metaclass Programming

A very simple metaclass

Redefining method repr in class changes the way python show itsinstancesSo to change the way python shows a class, we redefine repr in themetaclass

>>> class MetaPretty(type):... def __repr__(cls):... return "I’m the class %s" % cls.__name__...>>> class Ugly: pass>>> Ugly<class ’__main__.Ugly’>>>> class Pretty(object):... __metaclass__ = MetaPretty>>> PrettyI’m the class Pretty

Metaclass programming is easier than you thought !!!

Automatic property creation I

We don’t want to tell python the properties defined in a class if there aremethods in the class that evidently want a property defined !!!We implement it redefining init

class MetaAutoProp(type):def __init__(cls, name, bases, dict):

super(MetaAutoProp, cls).__init__(name, bases, dict)props = set(name[5:] for name in dict.iterkeys()

if name.startswith("_get_") or \name.startswith("_set_"))

for name in props:fget = getattr(cls, "_get_%s" % name, None)fset = getattr(cls, "_set_%s" % name, None)setattr(cls, name, property(fget, fset))

class AutoProp(object): __metaclass__ = MetaAutoPropclass UseX(AutoProp):

def __init__(self): self.__x = 0def _get_x(self): return self.__xdef _set_x(self, x): self.__x = x

Automatic property creation IIThe same as before but this time the implementation redefines new

class MetaAutoProp(type):def __new__(mcl, classname, bases, dict):

props = set(name[5:] for name in dict.iterkeys()if name.startswith("_get_") or \

name.startswith("_set_"))for name in props:

fget = dict.get("_get_%s" % name)fset = dict.get("_set_%s" % name)dict[name] = property(fget, fset)

super_ = super(MetaAutoProp, mcl)return super_.__new__(mcl, classname, bases, dict)

Deciding which way to go is not clearly defined

Automatic Initialization of Instance Attributesimport inspectclass auto_attr(object):

def __init__(self, factory, *args, **kwargs):self.creation_data = (factory, args, kwargs)

def is_auto_attr(attr):return isinstance(attr, auto_attr)

class MetaAutoAttr(type):def __call__(cls, *args, **kwargs):

obj = super(MetaAutoAttr, cls).__call__(*args, **kwargs)for attr, value in inspect.getmembers(cls, is_auto_attr):

factory, a, k = value.creation_datasetattr(obj, attr, factory(*a, **k))

return objclass AutoAttr(object): __metaclass__ = MetaAutoAttrclass Recorder(AutoAttr):

count = 0 # is immutable so no shared among instancesevents = auto_attr(list)def record(self, event):

self.count += 1self.events.append((self.count, event))

How classes are created Metaclass Conflicts

But, sometimes, conflicts happen

Imagine one wants a class with both behaviours. If one tries:

from autoprop import AutoPropfrom autoattr import AutoAttr, auto_attrclass Both(AutoProp, AutoAttr):

__x = auto_attr(list)def _get_x(self): return self.__xdef _set_x(self, v): self.__x = v

One gets:

TypeError: Error when calling the metaclass basesmetaclass conflict: the metaclass of a derived class must be a (non-strict)

subclass of the metaclasses of all its bases

The problem is that python needs to find a metaclass that brings bothbehaviours at the same time(This is the application of the Liskov Substitution Principle applied tometaclasses)

One gets:

Solving the metaclass conflictTo ensured it, the metaclass to use must be a subclass of all those of thebases of the class(Technically it must be minimal in the inheritance subgraph of the metaclassesof the bases considered as a lattice)Some languages create an on-the-fly this minimal metaclass if neededPython needs you to solve the problem , so:

from autoprop import MetaAutoPropfrom autoattr import MetaAutoAttr, auto_attrclass MetaBoth(MetaAutoAttr, MetaAutoProp): passclass Both(object):

__metaclass__ = MetaBoth__x = auto_attr(list)def _get_x(self):

return self.__xdef _set_x(self, value):

self.__x = value

(A recipe exists in the Python Cookbook that solves this problem by defininga metaclass factory method which creates the appropriate metaclass whenneeded)

self.__x = value

More Examples

Avoiding multiple initialization of SingletonsThe Singleton class that was presented before needed init to beidempotent. Changing its metaclass solves it:

# Singleton classes are only init’ed onceclass Singleton(object):

class __metaclass__(type):_initialized = set()def __call__(cls, *args, **kwargs):

inits = Singleton.__metaclass__._initializedobj = cls.__new__(cls, *args, **kwargs)if isinstance(obj, cls) and \

cls not in inits:inits.add(cls)obj.__init__(*args, **kwargs)

return obj_singletons = {}def __new__(cls, *args, **kwds):

if cls not in cls._singletons:s = super(Singleton, cls)cls._singletons[cls] = \

super(Singleton, cls).__new__(cls, *args, **kwds)return cls._singletons[cls]

More Examples

Avoiding multiple initialization of SingletonsThe Singleton class that was presented before needed init to beidempotent. Changing its metaclass solves it:

# Singleton classes are only init’ed onceclass Singleton(object):

class __metaclass__(type):_initialized = set()def __call__(cls, *args, **kwargs):

inits = Singleton.__metaclass__._initializedobj = cls.__new__(cls, *args, **kwargs)if isinstance(obj, cls) and \

cls not in inits:inits.add(cls)obj.__init__(*args, **kwargs)

return obj_singletons = {}def __new__(cls, *args, **kwds):

if cls not in cls._singletons:s = super(Singleton, cls)cls._singletons[cls] = \

super(Singleton, cls).__new__(cls, *args, **kwds)return cls._singletons[cls]

More Examples

Enforcing naming rules

We don’t want classes with attributes that are not lowercase to be evercreated !!!!

class InvalidAttribName(TypeError): passclass MetaEnsureAttribNames(type):

def __new__(mcl, name, bases, attrs):invalids = [attr for attr in attrs

if not attr.islower()]if invalids:

msg = "Invalid Attributes: " + ", ".join(invalids)raise InvalidAttribName, msg

super_ = super(MetaEnsureAttribNames, mcl)return super_.__new__(mcl, name, bases, attrs)

class EnsureAttribNames(object):__metaclass__ = MetaEnsureAttribNames

(You can experiment with different rules for methods, rules for class names,etc.)

More Examples

Checking Whether Interfaces are Implemented

We want to extend python capabilities to respresent the idea of an interfaceand to ensure that a class that declares implementing one, actually does itInterfaces are represented by simple classes

class IConnectable(object):def send(self, msg): passdef receive(self): pass

Declaring an interface is done with a class’ attribute and is ensured by ametaclass

class Channel(object):__metaclass__ = MetaInterfaceChecker__implements__ = IConnectabledef send(self, msg): ....def receive(self): ....

More Examples

The following metaclass checks if all the methods in all declared interfaces aredefined in the class(The implementation is naıve, so you can enhance it. hint: use moduleinspect)

import setclass InterfaceOmission(TypeError): passclass MetaInterfaceChecker(type):

def __init__(cls, classname, bases, classdict):super(MetaInterfaceChecker, cls).__init__(classname,

bases,classdict)

cls_defines = set(dir(cls))for interface in cls.__implements__:

itf_requires = set(dir(interface))if not itf_requires.issubset(cls_defines):

raise (InterfaceOmission,list(itf_requires - cls_defines))

More Examples

bases,classdict)

More Examples

bases,classdict)

Concluding Remarks

Conclusions

Metaclasses must not be abusedI they put you dangerously near to the dark sideI if there’s a simple way to perform the same, don’t use themI but sometimes they simplify thingsI or lead to more efficient solutions

But understanding themI shows you the whole picture of python’s OOI gives you new ways to think about solutions

And they are fun !!!

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Concluding Remarks

Conclusions

Bibliography

Shalahb Chaturvedi, Python Types and Objects

Mike Fletcher, Metaclasses, Who, Why, When, Python Conference(Pycon) 2004.

Ira R. Forman and Scott Danforth, Putting Metaclasses to Work.Addison-Wesley, 1999.

Alex Martelli, Python in a Nutshell (2nd Edition), O’Reilly Media Inc,2006.

Alex Martelli, Python Metaclasses

Alex Martelli, Anna Martelli, Python Cookbook (2nd Edition),O’Reilly Media Inc, 2005.The recipes are based on those in ActiveState Python Cookbook.

David Mertz, A Primer on Python Metaclass Programming

Guido Van Rossum, Unifying types and classes in Python 2.2

License

Aquesta obra esta subjecta a una llicencia Reconeixement-Compartir ambla mateixa llicencia 2.5 Espanya de Creative Commons.Per veure’n una copia, visiteu

http://creativecommons.org/licenses/by-sa/2.5/es/

o envieu una carta a

Creative Commons559 Nathan Abbott WayStanfordCalifornia 94305USA

metaclass programming in python

Education

classes classes

style classes

old classes

metaclasses shape classes

object obj

metaclass of typeobj

metatype of obj

basic concepts objects