Top python Questions

List of Tags
1485
Alex. S.

What is the use of the yield keyword in Python? What does it do?

For example, I'm trying to understand this code (**):

def node._get_child_candidates(self, distance, min_dist, max_dist):
    if self._leftchild and distance - max_dist < self._median:
        yield self._leftchild
    if self._rightchild and distance + max_dist >= self._median:
        yield self._rightchild  

And this is the caller:

result, candidates = list(), [self]
while candidates:
    node = candidates.pop()
    distance = node._get_dist(obj)
    if distance <= max_dist and distance >= min_dist:
        result.extend(node._values)
    candidates.extend(node._get_child_candidates(distance, min_dist, max_dist))
return result

What happens when the method _get_child_candidates is called? A list is returned? A single element is returned? Is it called again? When subsequent calls do stop?

** The code comes from Jochen Schulz (jrschulz), who made a great Python library for metric spaces. This is the link to the complete source: http://well-adjusted.de/~jrschulz/mspace/.

Answered By: e-satis ( 2695)

To understand what yield does, you must understand what generators are. And before generators come iterables.

Iterables

When you create a list, you can read its items one by one, and it's called iteration:

>>> mylist = [1, 2, 3]
>>> for i in mylist:
...    print(i)
1
2
3

Mylist is an iterable. When you use a comprehension list, you create a list, and so an iterable:

>>> mylist = [x*x for x in range(3)]
>>> for i in mylist:
...    print(i)
0
1
4

Everything you can use "for... in..." on is an iterable: lists, strings, files... These iterables are handy because you can read them as much as you wish, but you store all the values in memory and it's not always what you want when you have a lot of values.

Generators

Generators are iterables, but you can only read them once. It's because they do not store all the values in memory, they generate the values on the fly:

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator:
...    print(i)
0
1
4

It just the same except you used () instead of []. BUT, you can not perform for i in mygenerator a second time since generators can only be used once: they calculate 0, then forget about it and calculate 1 and ends calculating 4, one by one.

Yield

Yield is a keyword that is used like return, except the function will return a generator.

>>> def createGenerator():
...    mylist = range(3)
...    for i in mylist:
...        yield i*i
...
>>> mygenerator = createGenerator() # create a generator
>>> print(mygenerator) # mygenerator is an object!
<generator object createGenerator at 0xb7555c34>
>>> for i in mygenerator:
...     print(i)
0
1
4

Here it's a useless example, but it's handy when you know your function will return a huge set of values that you will only need to read once.

To master yield, you must understand that when you call the function, the code you have written in the function body does not run. The function only returns the generator object, this is bit tricky :-)

Then, your code will be run each time the for uses the generator.

Now the hard part:

The first time your function will run, it will run from the beginning until it hits yield, then it'll return the first value of the loop. Then, each other call will run the loop you have written in the function one more time, and return the next value, until there is no value to return.

The generator is considered empty once the function runs but does not hit yield anymore. It can be because the loop had come to ends, or because you do not satisfy a "if/else" anymore.

Your code explained

Generator:

# Here you create the method of the node object that will return the generator
def node._get_child_candidates(self, distance, min_dist, max_dist):

  # Here is the code that will be called each time you use the generator object:

  # If there is still a child of the node object on its left
  # AND if distance is ok, return the next child
  if self._leftchild and distance - max_dist < self._median:
                yield self._leftchild

  # If there is still a child of the node object on its right
  # AND if distance is ok, return the next child
  if self._rightchild and distance + max_dist >= self._median:
                yield self._rightchild

  # If the function arrives here, the generator will be considered empty
  # there is no more than two values: the left and the right children

Caller:

# Create an empty list and a list with the current object reference
result, candidates = list(), [self]

# Loop on candidates (they contain only one element at the beginning) 
while candidates:

    # Get the last candidate and remove it from the list
    node = candidates.pop()

    # Get the distance between obj and the candidate
    distance = node._get_dist(obj)

    # If distance is ok, then you can fill the result
    if distance <= max_dist and distance >= min_dist:
        result.extend(node._values)

    # Add the children of the candidate in the candidates list 
    # so the loop will keep running until it will have looked
    # at all the children of the children of the children, etc. of the candidate
    candidates.extend(node._get_child_candidates(distance, min_dist, max_dist))

return result

This code contains several smart parts:

  • The loop iterates on a list but the list expands while the loop is being iterated :-) It's a concise way to go through all these nested data even if it's a bit dangerous since you can end up with an infinite loop. In this case, candidates.extend(node._get_child_candidates(distance, min_dist, max_dist)) exhausts all the values of the generator, but while keep creating new generators object which will produce different values from the previous ones since it's not applied on the same node.

  • The extend() method is a list object method that expects an iterable and adds its values to the list.

Usually we pass a list to it:

>>> a = [1, 2]
>>> b = [3, 4]
>>> a.extend(b)
>>> print(a)
[1, 2, 3, 4]

But in your code it gets a generator, which is good because:

  1. You don't need to read the values twice.
  2. You can have a lot of children and you don't want them all stored in memory.

And it works because Python does not care if the argument of a method is a list or not. Python expects iterables so it will work with strings, lists, tuples and generators ! This is called duck typing and is one of the reason why Python is so cool. But this is another story, for another question...

You can stop here, or read a little bit to see a advanced use of generator:

Controlling a generator exhaustion

>>> class Bank(): # let's create a bank, building ATMs
...    crisis = False
...    def create_atm(self):
...        while not self.crisis:
...            yield "$100"
>>> hsbc = Bank() # when everything's ok the ATM gives you as much as you want
>>> corner_street_atm = hsbc.create_atm()
>>> print(corner_street_atm.next())
$100
>>> print(corner_street_atm.next())
$100
>>> print([corner_street_atm.next() for cash in range(5)])
['$100', '$100', '$100', '$100', '$100']
>>> hsbc.crisis = True # crisis is coming, no more money!
>>> print(corner_street_atm.next())
<type 'exceptions.StopIteration'>
>>> wall_street_atm = hsbc.create_atm() # it's even true for new ATMs
>>> print(wall_street_atm.next())
<type 'exceptions.StopIteration'>
>>> hsbc.crisis = False # trouble is, even post-crisis the ATM remains empty
>>> print(corner_street_atm.next())
<type 'exceptions.StopIteration'>
>>> brand_new_atm = hsbc.create_atm() # build a new one to get back in business
>>> for cash in brand_new_atm:
...    print cash
$100
$100
$100
$100
$100
$100
$100
$100
$100
...

It can be useful for various things like controlling access to a resource.

Itertools, your best friend

The itertools module contains special functions to manipulate iterables. Ever wish to duplicate a generator ? Chain a two generators ? Groups values in nested list with a one liner ? Map / Zip without creating another list ?

Then just import itertools.

An example ? Let's see the possible orders of arrival for a 4 horses race:

>>> horses = [1, 2, 3, 4]
>>> races = itertools.permutations(horses)
>>> print(races)
<itertools.permutations object at 0xb754f1dc>
>>> print(list(itertools.permutations(horses)))
[(1, 2, 3, 4),
 (1, 2, 4, 3),
 (1, 3, 2, 4),
 (1, 3, 4, 2),
 (1, 4, 2, 3),
 (1, 4, 3, 2),
 (2, 1, 3, 4),
 (2, 1, 4, 3),
 (2, 3, 1, 4),
 (2, 3, 4, 1),
 (2, 4, 1, 3),
 (2, 4, 3, 1),
 (3, 1, 2, 4),
 (3, 1, 4, 2),
 (3, 2, 1, 4),
 (3, 2, 4, 1),
 (3, 4, 1, 2),
 (3, 4, 2, 1),
 (4, 1, 2, 3),
 (4, 1, 3, 2),
 (4, 2, 1, 3),
 (4, 2, 3, 1),
 (4, 3, 1, 2),
 (4, 3, 2, 1)]

Understanding the inner mechanisms of iteration

Iteration is a process implying iterables (implementing the __iter__() method) and iterators (implementing the __next__() method). Iterables are any objects you can get an iterator from. Iterators are objects that let you iterate on iterables.

More about it in in this article about how does the for loop work.

Oh, and if you liked this answer, you'll probably like my explanation for decorators and metaclasses.

1423
jelovirt
Answered By: Thomas Wouters ( 743)

Chaining comparison operators:

>>> x = 5
>>> 1 < x < 10
True
>>> 10 < x < 20 
False
>>> x < 10 < x*10 < 100
True
>>> 10 > x <= 9
True
>>> 5 == x > 4
True

In case you're thinking it's doing 1 < x, which comes out as True, and then comparing True < 10, which is also True, then no, that's really not what happens (see the last example.) It's really translating into 1 < x and x < 10, and x < 10 and 10 < x * 10 and x*10 < 100, but with less typing and each term is only evaluated once.

1033
husayt

Asking as a Python beginner, what IDEs ("GUIs/editors") do others use for Python coding?

If you can just give the name (for example, Textpad, Eclipse ..) that will be enough. If it is already mentioned, you can just vote for it.

But if you can also give some more comparative information, that will be much appreciated.


Update: Results so far

                                       Rapid Application Development -.
                                           Integrated DB Support -+   |
                                                GUI Designer  -+  |   |
                                             Unit Testing -+   |  |   |
                                        Code Templates -.  |   |  |   |
                                       Code Folding -+  |  |   |  |   |
                           UML Editing / Viewing -+  |  |  |   |  |   |
                              Line Numbering -+   |  |  |  |   |  |   |
                         Bracket Matching -+  |   |  |  |  |   |  |   |
                          Smart Indent -+  |  |   |  |  |  |   |  |   |
         Source Control Integration -+  |  |  |   |  |  |  |   |  |   |
                   Error Markup  -+  |  |  |  |   |  |  |  |   |  |   |
  Integrated Python Debugging -+  |  |  |  |  |   |  |  |  |   |  |   |
    Multi-Language Support -+  |  |  |  |  |  |   |  |  |  |   |  |   |
  Auto Code Completion -+   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 Commercial / Free --+  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 Cross Platform -+   |  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
                _|___|__|___|__|__|__|__|__|__|___|__|__|__|___|__|___|_
                |CP|C/F|AC|MLS|PD|EM|SC|SI|BM|LN|UML|CF|CT|UT|UID|DB|RAD|comments
                +--+---+--+---+--+--+--+--+--+--+---+--+--+--+---+--+---+
 BlackAdder     |Y | C |  |   |  |  |  |Y |  |  |   |Y |  |  |   |  |   |
 BlueFish       |L |   |  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 Boa Constructor|Y | F |Y |   |Y |Y |  |Y |Y |Y | Y |Y |Y |  |   |  |   |
 ConTEXT        |W | C |  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 DABO           |Y |   |  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 DreamPie       |  | F |  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 Dr.Python      |  | F |  |   |  |Y |  |  |  |  |   |  |  |  |   |  |   |
 Editra         |Y | F |Y | Y |  |  |Y |Y |Y |Y |   |Y |  |  |   |  |   |
 Emacs          |Y | F |Y | Y |Y |Y |Y |Y |Y |Y | Y |Y |Y |Y |   |  |   |
 Eric Ide       |Y | F |Y |   |Y |Y |  |Y |  |Y |   |Y |  |Y |   |  |   |
 E-Texteditor   |W |   |  |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 Geany          |Y | F |Y*| Y |  |  |  |Y |Y |Y |   |Y |  |  |   |  |   |* very limited
 Gedit          |Y | F |Y¹| Y |  |  |  |Y |Y |Y |   |  |Y²|  |   |  |   |¹ with plugin ² sort of
 Idle           |Y | F |Y |   |  |  |  |  |  |  |   |  |  |  |   |  |   |
 JEdit          |Y | F |  | Y |  |  |  |  |Y |Y |   |Y |  |  |   |  |   |
 KDevelop       |Y | F |  | Y |  |  |Y |Y |Y |Y |   |Y |  |  |   |  |   |
 Komodo         |Y |C/F|Y | Y |Y |Y |Y |Y |Y |Y |   |Y |Y |Y |   |Y |   |
 NetBeans       |Y | F |Y | Y |Y |  |Y |Y |Y |Y | Y |Y |Y |Y |   |  | Y |
 NotePad++      |W | F |  | Y |  |  |  |  |  |Y |   |  |  |  |   |  |   |
 Pfaide         |W | C |Y | Y |  |  |  |Y |Y |Y |   |Y |Y |  |   |  |   |
 PIDA           |LW| F |Y | Y |  |  |  |Y |Y |Y |   |Y |  |  |   |  |   |VIM based
 PTVS           |W | F |Y | Y |Y |Y |Y |Y |Y |Y |   |Y |  |  |Y* |  | Y |*WPF bsed
 PyCharm        |Y | C |Y | Y*|Y |  |Y |Y |Y |Y |   |Y |  |Y |   |  |   |* javascript
 PyDev(Eclipse) |Y | F |Y | Y |Y |Y |Y |Y |Y |Y | Y |Y |Y |Y |   |  |   |
 Pyscripter     |W | F |Y |   |Y |Y |  |Y |  |Y |   |  |Y |Y |   |  |   |
 PythonWin      |W | F |Y |   |Y |  |  |Y |Y |  |   |Y |  |  |   |  |   |
 SciTE          |Y | F |  | Y |  |Y |  |  |Y |Y |   |Y |Y |  |   |  |   |
 ScriptDev      |W | C |Y | Y |Y |Y |  |Y |Y |Y |   |Y |Y |  |   |  |   |
 SPE            |  | F |Y |   |  |  |  |  |  |  | Y |  |  |  |   |  |   |
 Spyder         |Y | F |Y |   |Y |Y |  |Y |Y |Y |   |  |  |  |   |  |   |
 Sublime Text   |Y | C |Y | Y |  |  |  |Y |Y |Y |   |  |Y |  |   |  |   |extensible w/python
 TextMate       |M |   |  | Y |  |  |  |Y |Y |Y |   |Y |Y |  |   |  |   |
 UliPad         |Y | F |Y | Y |Y |  |  |Y |Y |  |   |  |Y |Y |   |  |   |
 Vim            |Y | F |Y | Y |Y |Y |Y |Y |Y |Y |   |Y |Y |Y |   |  |   |
 WingIde        |Y | C |Y | Y*|Y |Y |Y |Y |Y |Y |   |Y |Y |Y |   |  |   |* support for C
 Zeus           |W | C |  |   |  |  |Y |Y |Y |Y |   |Y |Y |  |   |  |   |
                +--+---+--+---+--+--+--+--+--+--+---+--+--+--+---+--+---+
                |CP|C/F|AC|MLS|PD|EM|SC|SI|BM|LN|UML|CF|CT|UT|UID|DB|RAD|
                |__|___|__|___|__|__|__|__|__|__|___|__|__|__|___|__|___|

Acronyms used:

  • CP - Cross Platform
  • C - Commercial
  • F - Free
  • AC - Automatic Code-completion
  • MLS - Multi-Language Support
  • PD - Integrated Python Debugging
  • EM - ErrorMarkup
  • SC - Source Control integration
  • SI - Smart Indent
  • BM - Bracket Matching
  • LN - Line Numbering
  • UML - UML editing / viewing
  • CF - Code Folding
  • CT - Code Templates
  • UT - Unit Testing
  • UID - GUI Designer (for example, Qt, Eric, ..)
  • DB - integrated database support
  • RAD - Rapid application development support
  • L - Linux
  • W - Windows
  • M - Mac

I don't mention basics like syntax highlighting as I expect these by default.


This is a just dry list reflecting your feedback and comments, I am not advocating any of these tools. I will keep updating this list as you keep posting your answers.

PS. Can you help me to add features of the above editors to the list (like auto-complete, debugging, etc.)?

Answered By: Rafał Rawicki ( 376)

Vim - For me it's the best choice, regardless of technology I'm using at the moment. It's not so hard to learn as it looks and during the work you are becoming more and more productive.

I've mastered almost all the Python concepts (well, let's say they're just OO concepts :-)) but this one is tricky.

I know it has something to do with introspection but it's still unclear to me.

So what are metaclasses? What do you use them for?

Concrete examples, including snippets, much appreciated!

Answered By: e-satis ( 1730)

Now that I know better, and due to demands in the answers on the yield keyword and the decorators, I'm going to answer this question with a lot of detail.

Disclaimer: very long post.

Classes as objects

Before understanding metaclasses, you need to master classes in Python. And Python has a very peculiar idea of what classes are, borrowed from the Smalltalk language.

In most languages, classes are just pieces of code that describe how to produce an object. That's kinda true in Python too:

  >>> class ObjectCreator(object):
  ...       pass
  ... 

  >>> my_object = ObjectCreator()
  >>> print my_object
  <__main__.ObjectCreator object at 0x8974f2c>

But classes are more than that in Python. Classes are objects too.

Yes, objects.

As soon as you use the keyword class, Python executes it and creates an OBJECT. The instruction

  >>> class ObjectCreator(object):
  ...       pass
  ... 

creates in memory an object with the name ObjectCreator.

This object (the class) is itself capable of creating objects (the instances), and this is why it's a class.

But still, it's an object, and therefore:

  • you can assign it to a variable
  • you can copy it
  • you can add attributes to it
  • you can pass it as a function parameter

e.g.:

  >>> print ObjectCreator # you can print a class because it's an object
  <class '__main__.ObjectCreator'>
  >>> def echo(o):
  ...       print o
  ... 
  >>> echo(ObjectCreator) # you can pass a class as a parameter
  <class '__main__.ObjectCreator'>
  >>> print hasattr(ObjectCreator, 'new_attribute')
  False
  >>> ObjectCreator.new_attribute = 'foo' # you can add attributes to a class
  >>> print hasattr(ObjectCreator, 'new_attribute')
  True
  >>> print ObjectCreator.new_attribute
  foo
  >>> ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable
  >>> print ObjectCreatorMirror.new_attribute
  foo
  >>> print ObjectCreatorMirror()
  <__main__.ObjectCreator object at 0x8997b4c>

Creating classes dynamically

Since classes are objects, you can create them on the fly, like any object.

First, you can create a class in a function using class:

  >>> def choose_class(name):
  ...     if name == 'foo':
  ...         class Foo(object):
  ...             pass
  ...         return Foo # return the class, not an instance
  ...     else:
  ...         class Bar(object):
  ...             pass
  ...         return Bar
  ...     
  >>> MyClass = choose_class('foo') 
  >>> print MyClass # the function returns a class, not an instance
  <class '__main__.Foo'>
  >>> print MyClass() # you can create an object from this class
  <__main__.Foo object at 0x89c6d4c>

But it's not so dynamic, since you still have to write the whole class yourself.

Since classes are objects, they must be generated by something.

When you use the class keyword, Python creates this object automatically. But as with most things in Python, it gives you a way to do it manually.

Remember the function type? The good old function that lets you know what type an object is:

>>> print type(1)
<type 'int'>
>>> print type("1")
<type 'str'>
>>> print type(ObjectCreator)
<type 'type'>
>>> print type(ObjectCreator())
<class '__main__.ObjectCreator'>

Well, type has a completely different ability, it can also create classes on the fly. type can take the description of a class as parameters, and return a class.

(I know, it's silly that the same function can have two completely different uses according to the parameters you pass to it. It's an issue due to backwards compatibility in Python)

type works this way:

  type(name of the class, 
       tuple of the parent class (for inheritance, can be empty), 
       dictionary containing attributes names and values)

e.g.:

>>> class MyShinyClass(object):
...       pass

can be created manually this way:

  >>> MyShinyClass = type('MyShinyClass', (), {}) # returns a class object
  >>> print MyShinyClass
  <class '__main__.MyShinyClass'>
  >>> print MyShinyClass() # create an instance with the class
  <__main__.MyShinyClass object at 0x8997cec>

You'll notice that we use "MyShinyClass" as the name of the class and as the variable to hold the class reference. They can be different, but there is no reason to complicate things.

type accepts a dictionary to define the attributes of the class. So:

>>> class Foo(object):
...       bar = True

Can be translated to:

  >>> Foo = type('Foo', (), {'bar':True})

And used as a normal class:

  >>> print Foo
  <class '__main__.Foo'>
  >>> print Foo.bar
  True
  >>> f = Foo()
  >>> print f
  <__main__.Foo object at 0x8a9b84c>
  >>> print f.bar
  True

And of course, you can inherit from it, so:

  >>>   class FooChild(Foo):
  ...         pass

would be:

  >>> FooChild = type('FooChild', (Foo,), {})
  >>> print FooChild
  <class '__main__.FooChild'>
  >>> print FooChild.bar # bar is inherited from Foo
  True

Eventually you'll want to add methods to your class. Just define a function with the proper signature and assign it as an attribute.

>>> def echo_bar(self):
...       print self.bar
... 
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True

You see where we are going: in Python, classes are objects, and you can create a class on the fly, dynamically.

This is what Python does when you use the keyword class, and it does so by using a metaclass.

What are metaclasses (finally)

Metaclasses are the 'stuff' that creates classes.

You define classes in order to create objects, right?

But we learned that Python classes are objects.

Well, metaclasses are what create these objects. They are the classes' classes, you can picture them this way:

  MyClass = MetaClass()
  MyObject = MyClass()

You've seen that type lets you do something like this:

  MyClass = type('MyClass', (), {})

It's because the function type is in fact a metaclass. type is the metaclass Python uses to create all classes behind the scenes.

Now you wonder why the heck is it written in lowercase, and not Type?

Well, I guess it's a matter of consistency with str, the class that creates strings objects, and int the class that creates integer objects. type is just the class that creates class objects.

You see that by checking the __class__ attribute.

Everything, and I mean everything, is an object in Python. That includes ints, strings, functions and classes. All of them are objects. And all of them have been created from a class:

  >>> age = 35
  >>> age.__class__
  <type 'int'>
  >>> name = 'bob'
  >>> name.__class__
  <type 'str'>
  >>> def foo(): pass
  >>> foo.__class__
  <type 'function'>
  >>> class Bar(object): pass
  >>> b = Bar()
  >>> b.__class__
  <class '__main__.Bar'>

Now, what is the __class__ of any __class__ ?

  >>> a.__class__.__class__
  <type 'type'>
  >>> age.__class__.__class__
  <type 'type'>
  >>> foo.__class__.__class__
  <type 'type'>
  >>> b.__class__.__class__
  <type 'type'>

So, a metaclass is just the stuff that creates class objects.

You can call it a 'class factory' if you wish.

type is the built-in metaclass Python uses, but of course, you can create your own metaclass.

The __metaclass__ attribute

You can add a __metaclass__ attribute when you write a class:

class Foo(object):
  __metaclass__ = something...
  [...]

If you do so, Python will use the metaclass to create the class Foo.

Careful, it's tricky.

You write class Foo(object) first, but the class object Foo is not created in memory yet.

Python will look for __metaclass__ in the class definition. If it finds it, it will use it to create the object class Foo. If it doesn't, it will use type to create the class.

Read that several times.

When you do:

class Foo(Bar):
  pass

Python does the following:

Is there a __metaclass__ attribute in Foo?

If yes, create in memory a class object (I said a class object, stay with me here), with the name Foo by using what is in __metaclass__.

If Python can't find __metaclass__, it will look for a __metaclass__ in Bar (the parent class), and try to do the same.

If Python can't find __metaclass__ in any parent, it will look for a __metaclass__ at the MODULE level, and try to do the same.

Then if it can't find any __metaclass__ at all, it will use type to create the class object.

Now the big question is, what can you put in __metaclass__ ?

The answer is: something that can create a class.

And what can create a class? type, or anything that subclasses or uses it.

Custom metaclasses

The main purpose of a metaclass is to change the class automatically, when it's created.

You usually do this for APIs, where you want to create classes matching the current context.

Imagine a stupid example, where you decide that all classes in your module should have their attributes written in uppercase. There are several ways to do this, but one way is to set __metaclass__ at the module level.

This way, all classes of this module will be created using this metaclass, and we just have to tell the metaclass to turn all attributes to uppercase.

Luckily, __metaclass__ can actually be any callable, it doesn't need to be a formal class (I know, something with 'class' in its name doesn't need to be a class, go figure... but it's helpful).

So we will start with a simple example, by using a function.

# the metaclass will automatically get passed the same argument
# that you usually pass to `type`
def upper_attr(future_class_name, future_class_parents, future_class_attr):
  """
    Return a class object, with the list of its attribute turned 
    into uppercase.
  """

  # pick up any attribute that doesn't start with '__'
  attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
  # turn them into uppercase
  uppercase_attr = dict((name.upper(), value) for name, value in attrs)

  # let `type` do the class creation
  return type(future_class_name, future_class_parents, uppercase_attr)

__metaclass__ = upper_attr # this will affect all classes in the module

class Foo(): # global __metaclass__ won't work with "object" though
  # but we can define __metaclass__ here instead to affect only this class
  # and this will work with "object" children
  bar = 'bip'

print hasattr(Foo, 'bar')
# Out: False
print hasattr(Foo, 'BAR')
# Out: True

f = Foo()
print f.BAR
# Out: 'bip'

Now, let's do exactly the same, but using a real class for a metaclass:

# remember that `type` is actually a class like `str` and `int`
# so you can inherit from it
class UpperAttrMetaclass(type): 
    # __new__ is the method called before __init__
    # it's the method that creates the object and returns it
    # while __init__ just initializes the object passed as parameter
    # you rarely use __new__, except when you want to control how the object
    # is created.
    # here the created object is the class, and we want to customize it
    # so we override __new__
    # you can do some stuff in __init__ too if you wish
    # some advanced use involves overriding __call__ as well, but we won't
    # see this
    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):

        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        return type(future_class_name, future_class_parents, uppercase_attr)

But this is not really OOP. We call type directly and we don't override call the parent __new__. Let's do it:

class UpperAttrMetaclass(type): 

    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):

        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        # reuse the type.__new__ method
        # this is basic OOP, nothing magic in there
        return type.__new__(upperattr_metaclass, future_class_name, 
                            future_class_parents, uppercase_attr)

You may have noticed the extra argument upperattr_metaclass. There is nothing special about it: a method always receives the current instance as first parameter. Just like you have self for ordinary methods.

Of course, the names I used here are long for the sake of clarity, but like for self, all the arguments have conventional names. So a real production metaclass would look like this:

class UpperAttrMetaclass(type): 

    def __new__(cls, name, bases, dct):

        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        return type.__new__(cls, name, bases, uppercase_attr)

We can make it even cleaner by using super, which will ease inheritance (because yes, you can have metaclasses, inheriting from metaclasses, inheriting from type):

class UpperAttrMetaclass(type): 

    def __new__(cls, name, bases, dct):

        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        return super(UpperAttrMetaclass, cls).__new__(cls, name, bases, uppercase_attr)

That's it. There is really nothing more about metaclasses.

The reason behind the complexity of the code using metaclasses is not because of metaclasses, it's because you usually use metaclasses to do twisted stuff relying on introspection, manipulating inheritance, vars such as __dict__, etc.

Indeed, metaclasses are especially useful to do black magic, and therefore complicated stuff. But by themselves, they are simple:

  • intercept a class creation
  • modify the class
  • return the modified class

Why would you use metaclasses classes instead of functions?

Since __metaclass__ can accept any callable, why would you use a class since it's obviously more complicated?

There are several reasons to do so:

  • The intention is clear. When you read UpperAttrMetaclass(type), you know what's going to follow
  • You can use OOP. Metaclass can inherit from metaclass, override parent methods. Metaclasses can even use metaclasses.
  • You can structure your code better. You never use metaclasses for something as trivial as the above example. It's usually for something complicated. Having the ability to make several methods and group them in one class is very useful to make the code easier to read.
  • You can hook on __new__, __init__ and __call__. Which will allow you to do different stuff. Even if usually you can do it all in __new__, some people are just more comfortable using __init__.
  • These are called metaclasses, damn it! It must mean something!

Why the hell would you use metaclasses?

Now the big question. Why would you use some obscure error prone feature?

Well, usually you don't:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why).

Python Guru Tim Peters

The main use case for a metaclass is creating an API. A typical example of this is the Django ORM.

It allows you to define something like this:

  class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

But if you do this:

  guy = Person(name='bob', age='35')
  print guy.age

It won't return an IntegerField object. It will return an int, and can even take it directly from the database.

This is possible because models.Model defines __metaclass__ and it uses some magic that will turn the Person you just defined with simple statements into a complex hook to a database field.

Django makes something complex look simple by exposing a simple API and using metaclasses, recreating code from this API to do the real job behind the scenes.

The last word

First, you know that classes are objects that can create instances.

Well in fact, classes are themselves instances. Of metaclasses.

  >>> class Foo(object): pass
  >>> id(Foo)
  142630324

Everything is an object in Python, and they are all either instances of classes or instances of metaclasses.

Except for type.

type is actually its own metaclass. This is not something you could reproduce in pure Python, and is done by cheating a little bit at the implementation level.

Secondly, metaclasses are complicated. You may not want to use them for very simple class alterations. You can change classes by using two different techniques:

  • monkey patching
  • class decorators

99% of the time you need class alteration, you are better off using these.

But 99% of the time, you don't need class alteration at all :-)

804
Imran

How can I make a decorator in Python that would do the following.

@makebold
@makeitalic
def say():
   return "Hello"

which should return

<b><i>Hello</i></b>

I'm not trying to make HTML this way in a real application, just trying to understand how decorators and decorator chaining works.

Answered By: e-satis ( 1737)

Beware LONG POST!!!

Since this answer explaining yield has been quite a success, I think a little tutorial about Python decorators could help as well.

If you are not into long explanations, see Paolo Bergantino's answer.

Python's functions are objects

To understand decorators, you must first understand that functions are objects in Python. This has important consequences. Let's see why with a simple example :

def shout(word="yes"):
    return word.capitalize()+"!"

print shout()
# outputs : 'Yes!'

# As an object, you can assign the function to a variable like any
# other object

scream = shout

# Notice we don't use parentheses: we are not calling the function, we are
# putting the function "shout" into the variable "scream".
# It means you can then call "shout" from "scream":

print scream()
# outputs : 'Yes!'

# More than that, it means you can remove the old name 'shout', and
# the function will still be accessible from 'scream'

del shout
try:
    print shout()
except NameError, e:
    print e
    #outputs: "name 'shout' is not defined"

print scream()
# outputs: 'Yes!'

OK, keep that in mind, we are going back to it soon. Another interesting property of Python functions is they can be defined... inside another function!

def talk():

    # You can define a function on the fly in "talk" ...
    def whisper(word="yes"):
        return word.lower()+"..."

    # ... and use it right away!

    print whisper()

# You call "talk", that defines "whisper" EVERY TIME you call it, then
# "whisper" is called in "talk".
talk()
# outputs:
# "yes..."

# But "whisper" DOES NOT EXIST outside "talk":

try:
    print whisper()
except NameError, e:
    print e
    #outputs : "name 'whisper' is not defined"*

Functions references

OK, still here? Now the fun part, you've seen that functions are objects and therefore:

  • can be assigned to a variable;
  • can be defined in another function.

Well, that means that a function can return another function :-) Have a look:

def getTalk(type="shout"):

    # We define functions on the fly
    def shout(word="yes"):
        return word.capitalize()+"!"

    def whisper(word="yes") :
        return word.lower()+"...";

    # Then we return one of them
    if type == "shout":
        # We don't use "()", we are not calling the function,
        # we are returning the function object
        return shout
    else:
        return whisper

# How do you use this strange beast?

# Get the function and assign it to a variable
talk = getTalk()

# You can see that "talk" is here a function object:
print talk
#outputs : <function shout at 0xb7ea817c>

# The object is the one returned by the function:
print talk()
#outputs : Yes!

# And you can even use it directly if you feel wild:
print getTalk("whisper")()
#outputs : yes...

But wait, there is more. If you can return a function, then you can pass one as a parameter:

def doSomethingBefore(func):
    print "I do something before then I call the function you gave me"
    print func()

doSomethingBefore(scream)
#outputs:
#I do something before then I call the function you gave me
#Yes!

Well, you just have everything needed to understand decorators. You see, decorators are wrappers which means that they let you execute code before and after the function they decorate without the need to modify the function itself.

Handcrafted decorators

How you would do it manually:

# A decorator is a function that expects ANOTHER function as parameter
def my_shiny_new_decorator(a_function_to_decorate):

    # Inside, the decorator defines a function on the fly: the wrapper.
    # This function is going to be wrapped around the original function
    # so it can execute code before and after it.
    def the_wrapper_around_the_original_function():

        # Put here the code you want to be executed BEFORE the original
        # function is called
        print "Before the function runs"

        # Call the function here (using parentheses)
        a_function_to_decorate()

        # Put here the code you want to be executed AFTER the original
        # function is called
        print "After the function runs"

    # At this point, "a_function_to_decorate" HAS NEVER BEEN EXECUTED.
    # We return the wrapper function we have just created.
    # The wrapper contains the function and the code to execute before
    # and after. It's ready to use!
    return the_wrapper_around_the_original_function

# Now imagine you create a function you don't want to ever touch again.
def a_stand_alone_function():
    print "I am a stand alone function, don't you dare modify me"

a_stand_alone_function()
#outputs: I am a stand alone function, don't you dare modify me

# Well, you can decorate it to extend its behavior.
# Just pass it to the decorator, it will wrap it dynamically in
# any code you want and return you a new function ready to be used:

a_stand_alone_function_decorated = my_shiny_new_decorator(a_stand_alone_function)
a_stand_alone_function_decorated()
#outputs:
#Before the function runs
#I am a stand alone function, don't you dare modify me
#After the function runs

Now, you probably want that every time you call a_stand_alone_function, a_stand_alone_function_decorated is called instead. That's easy, just overwrite a_stand_alone_function with the function returned by my_shiny_new_decorator:

a_stand_alone_function = my_shiny_new_decorator(a_stand_alone_function)
a_stand_alone_function()
#outputs:
#Before the function runs
#I am a stand alone function, don't you dare modify me
#After the function runs

# And guess what? That's EXACTLY what decorators do!

Decorators demystified

The previous example, using the decorator syntax:

@my_shiny_new_decorator
def another_stand_alone_function():
    print "Leave me alone"

another_stand_alone_function()
#outputs:
#Before the function runs
#Leave me alone
#After the function runs

Yes, that's all, it's that simple. @decorator is just a shortcut to:

another_stand_alone_function = my_shiny_new_decorator(another_stand_alone_function)

Decorators are just a pythonic variant of the decorator design pattern. There are several classic design patterns embedded in Python to ease development, like iterators.

Of course, you can cumulate decorators:

def bread(func):
    def wrapper():
        print "</''''''\>"
        func()
        print "<\______/>"
    return wrapper

def ingredients(func):
    def wrapper():
        print "#tomatoes#"
        func()
        print "~salad~"
    return wrapper

def sandwich(food="--ham--"):
    print food

sandwich()
#outputs: --ham--
sandwich = bread(ingredients(sandwich))
sandwich()
#outputs:
#</''''''\>
# #tomatoes#
# --ham--
# ~salad~
#<\______/>

Using the Python decorator syntax:

@bread
@ingredients
def sandwich(food="--ham--"):
    print food

sandwich()
#outputs:
#</''''''\>
# #tomatoes#
# --ham--
# ~salad~
#<\______/>

The order you set the decorators MATTERS:

@ingredients
@bread
def strange_sandwich(food="--ham--"):
    print food

strange_sandwich()
#outputs:
##tomatoes#
#</''''''\>
# --ham--
#<\______/>
# ~salad~

Eventually answering the question

As a conclusion, you can easily see how to answer the question:

# The decorator to make it bold
def makebold(fn):
    # The new function the decorator returns
    def wrapper():
        # Insertion of some code before and after
        return "<b>" + fn() + "</b>"
    return wrapper

# The decorator to make it italic
def makeitalic(fn):
    # The new function the decorator returns
    def wrapper():
        # Insertion of some code before and after
        return "<i>" + fn() + "</i>"
    return wrapper

@makebold
@makeitalic
def say():
    return "hello"

print say()
#outputs: <b><i>hello</i></b>

# This is the exact equivalent to
def say():
    return "hello"
say = makebold(makeitalic(say))

print say()
#outputs: <b><i>hello</i></b>

You can now just leave happy, or burn your brain a little bit more and see advanced uses of decorators.

Passing arguments to the decorated function

# It's not black magic, you just have to let the wrapper
# pass the argument:

def a_decorator_passing_arguments(function_to_decorate):
    def a_wrapper_accepting_arguments(arg1, arg2):
        print "I got args! Look:", arg1, arg2
        function_to_decorate(arg1, arg2)
    return a_wrapper_accepting_arguments

# Since when you are calling the function returned by the decorator, you are
# calling the wrapper, passing arguments to the wrapper will let it pass them to
# the decorated function

@a_decorator_passing_arguments
def print_full_name(first_name, last_name):
    print "My name is", first_name, last_name

print_full_name("Peter", "Venkman")
# outputs:
#I got args! Look: Peter Venkman
#My name is Peter Venkman

Decorating methods

What's great with Python is that methods and functions are really the same, except methods expect their first parameter to be a reference to the current object (self). It means you can build a decorator for methods the same way, just remember to take self in consideration:

def method_friendly_decorator(method_to_decorate):
    def wrapper(self, lie):
        lie = lie - 3 # very friendly, decrease age even more :-)
        return method_to_decorate(self, lie)
    return wrapper


class Lucy(object):

    def __init__(self):
        self.age = 32

    @method_friendly_decorator
    def sayYourAge(self, lie):
        print "I am %s, what did you think?" % (self.age + lie)

l = Lucy()
l.sayYourAge(-3)
#outputs: I am 26, what did you think?

Of course, if you make a very general decorator and want to apply it to any function or method, no matter its arguments, then just use *args, **kwargs:

def a_decorator_passing_arbitrary_arguments(function_to_decorate):
    # The wrapper accepts any arguments
    def a_wrapper_accepting_arbitrary_arguments(*args, **kwargs):
        print "Do I have args?:"
        print args
        print kwargs
        # Then you unpack the arguments, here *args, **kwargs
        # If you are not familiar with unpacking, check:
        # http://www.saltycrane.com/blog/2008/01/how-to-use-args-and-kwargs-in-python/
        function_to_decorate(*args, **kwargs)
    return a_wrapper_accepting_arbitrary_arguments

@a_decorator_passing_arbitrary_arguments
def function_with_no_argument():
    print "Python is cool, no argument here."

function_with_no_argument()
#outputs
#Do I have args?:
#()
#{}
#Python is cool, no argument here.

@a_decorator_passing_arbitrary_arguments
def function_with_arguments(a, b, c):
    print a, b, c

function_with_arguments(1,2,3)
#outputs
#Do I have args?:
#(1, 2, 3)
#{}
#1 2 3

@a_decorator_passing_arbitrary_arguments
def function_with_named_arguments(a, b, c, platypus="Why not ?"):
    print "Do %s, %s and %s like platypus? %s" %\
    (a, b, c, platypus)

function_with_named_arguments("Bill", "Linus", "Steve", platypus="Indeed!")
#outputs
#Do I have args ? :
#('Bill', 'Linus', 'Steve')
#{'platypus': 'Indeed!'}
#Do Bill, Linus and Steve like platypus? Indeed!

class Mary(object):

    def __init__(self):
        self.age = 31

    @a_decorator_passing_arbitrary_arguments
    def sayYourAge(self, lie=-3): # You can now add a default value
        print "I am %s, what did you think ?" % (self.age + lie)

m = Mary()
m.sayYourAge()
#outputs
# Do I have args?:
#(<__main__.Mary object at 0xb7d303ac>,)
#{}
#I am 28, what did you think?

Passing arguments to the decorator

Great, now what would you say about passing arguments to the decorator itself? Well this is a bit twisted because a decorator must accept a function as an argument and therefore, you cannot pass the decorated function arguments directly to the decorator.

Before rushing to the solution, let's write a little reminder:

# Decorators are ORDINARY functions
def my_decorator(func):
    print "I am a ordinary function"
    def wrapper():
        print "I am function returned by the decorator"
        func()
    return wrapper

# Therefore, you can call it without any "@"

def lazy_function():
    print "zzzzzzzz"

decorated_function = my_decorator(lazy_function)
#outputs: I am a ordinary function

# It outputs "I am a ordinary function", because that's just what you do:
# calling a function. Nothing magic.

@my_decorator
def lazy_function():
    print "zzzzzzzz"

#outputs: I am a ordinary function

It's exactly the same. "my_decorator" is called. So when you @my_decorator, you are telling Python to call the function 'labeled by the variable "my_decorator"'. It's important, because the label you give can point directly to the decorator... or not! Let's start to be evil!

def decorator_maker():

    print "I make decorators! I am executed only once: "+\
          "when you make me create a decorator."

    def my_decorator(func):

        print "I am a decorator! I am executed only when you decorate a function."

        def wrapped():
            print ("I am the wrapper around the decorated function. "
                  "I am called when you call the decorated function. "
                  "As the wrapper, I return the RESULT of the decorated function.")
            return func()

        print "As the decorator, I return the wrapped function."

        return wrapped

    print "As a decorator maker, I return a decorator"
    return my_decorator

# Let's create a decorator. It's just a new function after all.
new_decorator = decorator_maker()
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator

# Then we decorate the function

def decorated_function():
    print "I am the decorated function."

decorated_function = new_decorator(decorated_function)
#outputs:
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function

# Let's call the function:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.

No surprise here. Let's do EXACTLY the same thing, but skipping intermediate variables:

def decorated_function():
    print "I am the decorated function."
decorated_function = decorator_maker()(decorated_function)
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function.

# Finally:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.

Let's make it AGAIN, even shorter:

@decorator_maker()
def decorated_function():
    print "I am the decorated function."
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function.

#Eventually:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.

Hey, did you see that? We used a function call with the "@" syntax :-)

So back to decorators with arguments. If we can use functions to generate the decorator on the fly, we can pass arguments to that function, right?

def decorator_maker_with_arguments(decorator_arg1, decorator_arg2):

    print "I make decorators! And I accept arguments:", decorator_arg1, decorator_arg2

    def my_decorator(func):
        # The ability to pass arguments here is a gift from closures.
        # If you are not comfortable with closures, you can assume it's ok,
        # or read: http://stackoverflow.com/questions/13857/can-you-explain-closures-as-they-relate-to-python
        print "I am the decorator. Somehow you passed me arguments:", decorator_arg1, decorator_arg2

        # Don't confuse decorator arguments and function arguments!
        def wrapped(function_arg1, function_arg2) :
            print ("I am the wrapper around the decorated function.\n"
                  "I can access all the variables\n"
                  "\t- from the decorator: {0} {1}\n"
                  "\t- from the function call: {2} {3}\n"
                  "Then I can pass them to the decorated function"
                  .format(decorator_arg1, decorator_arg2,
                          function_arg1, function_arg2))
            return func(function_arg1, function_arg2)

        return wrapped

    return my_decorator

@decorator_maker_with_arguments("Leonard", "Sheldon")
def decorated_function_with_arguments(function_arg1, function_arg2):
    print ("I am the decorated function and only knows about my arguments: {0}"
           " {1}".format(function_arg1, function_arg2))

decorated_function_with_arguments("Rajesh", "Howard")
#outputs:
#I make decorators! And I accept arguments: Leonard Sheldon
#I am the decorator. Somehow you passed me arguments: Leonard Sheldon
#I am the wrapper around the decorated function.
#I can access all the variables
#   - from the decorator: Leonard Sheldon
#   - from the function call: Rajesh Howard
#Then I can pass them to the decorated function
#I am the decorated function and only knows about my arguments: Rajesh Howard

Here it is, a decorator with arguments. Arguments can be set as variable:

c1 = "Penny"
c2 = "Leslie"

@decorator_maker_with_arguments("Leonard", c1)
def decorated_function_with_arguments(function_arg1, function_arg2):
    print ("I am the decorated function and only knows about my arguments:"
           " {0} {1}".format(function_arg1, function_arg2))

decorated_function_with_arguments(c2, "Howard")
#outputs:
#I make decorators! And I accept arguments: Leonard Penny
#I am the decorator. Somehow you passed me arguments: Leonard Penny
#I am the wrapper around the decorated function.
#I can access all the variables
#   - from the decorator: Leonard Penny
#   - from the function call: Leslie Howard
#Then I can pass them to the decorated function
#I am the decorated function and only knows about my arguments: Leslie Howard

As you can see, you can pass arguments to the decorator like any function using this trick. You can even use *args, **kwargs if you wish. But remember decorators are called only once. Just when Python imports the script. You can't dynamically set the arguments afterwards. When you do "import x", the function is already decorated, so you can't change anything.

Let's practice: a decorator to decorate a decorator

OK, as a bonus, I'll give you a snippet to make any decorator accept generically any argument. After all, in order to accept arguments, we created our decorator using another function. We wrapped the decorator. Anything else we saw recently that wrapped function? Oh yes, decorators! Let's have some fun and write a decorator for the decorators:

def decorator_with_args(decorator_to_enhance):
    """
    This function is supposed to be used as a decorator.
    It must decorate an other function, that is intended to be used as a decorator.
    Take a cup of coffee.
    It will allow any decorator to accept an arbitrary number of arguments,
    saving you the headache to remember how to do that every time.
    """

    # We use the same trick we did to pass arguments
    def decorator_maker(*args, **kwargs):

        # We create on the fly a decorator that accepts only a function
        # but keeps the passed arguments from the maker.
        def decorator_wrapper(func):

            # We return the result of the original decorator, which, after all,
            # IS JUST AN ORDINARY FUNCTION (which returns a function).
            # Only pitfall: the decorator must have this specific signature or it won't work:
            return decorator_to_enhance(func, *args, **kwargs)

        return decorator_wrapper

    return decorator_maker

It can be used as follows:

# You create the function you will use as a decorator. And stick a decorator on it :-)
# Don't forget, the signature is "decorator(func, *args, **kwargs)"
@decorator_with_args
def decorated_decorator(func, *args, **kwargs):
    def wrapper(function_arg1, function_arg2):
        print "Decorated with", args, kwargs
        return func(function_arg1, function_arg2)
    return wrapper

# Then you decorate the functions you wish with your brand new decorated decorator.

@decorated_decorator(42, 404, 1024)
def decorated_function(function_arg1, function_arg2):
    print "Hello", function_arg1, function_arg2

decorated_function("Universe and", "everything")
#outputs:
#Decorated with (42, 404, 1024) {}
#Hello Universe and everything

# Whoooot!

I know, the last time you had this feeling, it was after listening a guy saying: "before understanding recursion, you must first understand recursion". But now, don't you feel good about mastering this?

Best practices while using decorators

  • They are new as of Python 2.4, so be sure that's what your code is running on.
  • Decorators slow down the function call. Keep that in mind.
  • You can not un-decorate a function. There are hacks to create decorators that can be removed but nobody uses them. So once a function is decorated, it's done. For all the code.
  • Decorators wrap functions, which can make them hard to debug.

Python 2.5 solves this last issue by providing the functools module including functools.wraps that copies the name, module and docstring of any wrapped function to it's wrapper. Fun fact, functools.wraps is a decorator :-)

# For debugging, the stacktrace prints you the function __name__
def foo():
    print "foo"

print foo.__name__
#outputs: foo

# With a decorator, it gets messy
def bar(func):
    def wrapper():
        print "bar"
        return func()
    return wrapper

@bar
def foo():
    print "foo"

print foo.__name__
#outputs: wrapper

# "functools" can help for that

import functools

def bar(func):
    # We say that "wrapper", is wrapping "func"
    # and the magic begins
    @functools.wraps(func)
    def wrapper():
        print "bar"
        return func()
    return wrapper

@bar
def foo():
    print "foo"

print foo.__name__
#outputs: foo

How can the decorators be useful?

Now the big question: what can I use decorators for? Seem cool and powerful, but a practical example would be great. Well, there are 1000 possibilities. Classic uses are extending a function behavior from an external lib (you can't modify it) or for a debug purpose (you don't want to modify it because it's temporary). You can use them to extends several functions with the same code without rewriting it every time, for DRY's sake. E.g.:

def benchmark(func):
    """
    A decorator that prints the time a function takes
    to execute.
    """
    import time
    def wrapper(*args, **kwargs):
        t = time.clock()
        res = func(*args, **kwargs)
        print func.__name__, time.clock()-t
        return res
    return wrapper


def logging(func):
    """
    A decorator that logs the activity of the script.
    (it actually just prints it, but it could be logging!)
    """
    def wrapper(*args, **kwargs):
        res = func(*args, **kwargs)
        print func.__name__, args, kwargs
        return res
    return wrapper


def counter(func):
    """
    A decorator that counts and prints the number of times a function has been executed
    """
    def wrapper(*args, **kwargs):
        wrapper.count = wrapper.count + 1
        res = func(*args, **kwargs)
        print "{0} has been used: {1}x".format(func.__name__, wrapper.count)
        return res
    wrapper.count = 0
    return wrapper

@counter
@benchmark
@logging
def reverse_string(string):
    return str(reversed(string))

print reverse_string("Able was I ere I saw Elba")
print reverse_string("A man, a plan, a canoe, pasta, heros, rajahs, a coloratura, maps, snipe, percale, macaroni, a gag, a banana bag, a tan, a tag, a banana bag again (or a camel), a crepe, pins, Spam, a rut, a Rolo, cash, a jar, sore hats, a peon, a canal: Panama!")

#outputs:
#reverse_string ('Able was I ere I saw Elba',) {}
#wrapper 0.0
#wrapper has been used: 1x
#ablE was I ere I saw elbA
#reverse_string ('A man, a plan, a canoe, pasta, heros, rajahs, a coloratura, maps, snipe, percale, macaroni, a gag, a banana bag, a tan, a tag, a banana bag again (or a camel), a crepe, pins, Spam, a rut, a Rolo, cash, a jar, sore hats, a peon, a canal: Panama!',) {}
#wrapper 0.0
#wrapper has been used: 2x
#!amanaP :lanac a ,noep a ,stah eros ,raj a ,hsac ,oloR a ,tur a ,mapS ,snip ,eperc a ,)lemac a ro( niaga gab ananab a ,gat a ,nat a ,gab ananab a ,gag a ,inoracam ,elacrep ,epins ,spam ,arutaroloc a ,shajar ,soreh ,atsap ,eonac a ,nalp a ,nam A

Of course the good thing with decorators is that you can use them right away on almost anything without rewriting. DRY, I said:

@counter
@benchmark
@logging
def get_random_futurama_quote():
    import httplib
    conn = httplib.HTTPConnection("slashdot.org:80")
    conn.request("HEAD", "/index.html")
    for key, value in conn.getresponse().getheaders():
        if key.startswith("x-b") or key.startswith("x-f"):
            return value
    return "No, I'm ... doesn't!"

print get_random_furturama_quote()
print get_random_furturama_quote()

#outputs:
#get_random_futurama_quote () {}
#wrapper 0.02
#wrapper has been used: 1x
#The laws of science be a harsh mistress.
#get_random_futurama_quote () {}
#wrapper 0.01
#wrapper has been used: 2x
#Curse you, merciful Poseidon!

Python itself provides several decorators: property, staticmethod, etc. Django use decorators to manage caching and view permissions. Twisted to fake inlining asynchronous functions calls. This really is a large playground.

EDIT: given the success of this answer, and people asking me to do the same with metaclasses, I did.

I like the Android platform. Actually, with some friends, we even participate to the ADC with the Spoxt project.

But Java is not my favourite language at all. We are working on a S60 version and this platform has a nice Python API. Of course there is nothing official about Python on Android, but since Jython exists, does anybody know a way to let the snake and the robot work together ?

Answered By: Heat Miser ( 169)

There is also the new ASE project, it looks awesome, and has some integration with native Android components. Android Scripting Environment

583
sectrean

I'm mainly a C# developer, but I'm currently working on a project in Python.

What's the best way to implement the equivalent of an enum in Python?

Answered By: Alec Thomas ( 505)

Here's yet another way:

def enum(**enums):
    return type('Enum', (), enums)

Used like so:

>>> Numbers = enum(ONE=1, TWO=2, THREE='three')
>>> Numbers.ONE
1
>>> Numbers.TWO
2
>>> Numbers.THREE
'three'

You can also easily support automatic enumeration with something like this:

def enum(*sequential, **named):
    enums = dict(zip(sequential, range(len(sequential))), **named)
    return type('Enum', (), enums)

Used like so:

>>> Numbers = enum('ZERO', 'ONE', 'TWO')
>>> Numbers.ZERO
0
>>> Numbers.ONE
1

Support for converting the values back to names can be added this way:

def enum(*sequential, **named):
    enums = dict(zip(sequential, range(len(sequential))), **named)
    reverse = dict((value, key) for key, value in enums.iteritems())
    enums['reverse_mapping'] = reverse
    return type('Enum', (), enums)

Obviously, this overwrites anything with that name, but it is useful for rendering your enums in output. It will throw KeyError if the reverse mapping doesn't exist. With the first example:

>>> Numbers.reverse_mapping['three']
'THREE'
521
spence91

How do I check if a file exists, using Python, without using a try: statement?

Answered By: Brian ( 518)

Just to add to the answers - it's usually safer to use the following approach:

try:
   with open('filename'): pass
except IOError:
   print 'Oh dear.'

os.path.exists() only tells you that the file existed at that point. In the tiny interval between that and running code that depends on it, it is possible that someone will have created or deleted the file.

This is a race condition that can often lead to security vulnerabilities. An attacker can create a symlink to an arbitrary file immediately after the program checks no file exists. This way arbitrary files can be read or overwritten with the privilege level your program runs with.

488
freshWoWer

How can I call an external command in Python?

Answered By: Eli Courtwright ( 571)

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

  1. os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
    os.system("some_command < input_file | another_command > output_file")
    However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
    http://docs.python.org/lib/os-process.html

  2. stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
    http://docs.python.org/lib/os-newstreams.html

  3. The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
    print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
    instead of
    print os.popen("echo Hello World").read()
    but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
    http://docs.python.org/lib/node528.html

  4. The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
    return_code = call("echo Hello World", shell=True)
    http://docs.python.org/lib/node529.html

  5. The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

The subprocess module should probably be what you use.

Does Python have a ternary conditional operator? If not, is it possible to simulate one concisely using other language constructs?

Answered By: Vinko Vrsalovic ( 560)

Yes, it was added in version 2.5. It's frowned upon by some pythonistas, so keep that in mind.
The syntax is:

a if test else b

First test is evaluated, then either a or b is returned based on the Boolean value of test;
if test evaluates to True a is returned, else b is returned.

For example:

>>> 'true' if True else 'false'
'true'
>>> 'true' if False else 'false'
'false'

Official documentation: