the dam

List comprehensions

List comprehensions give Python users a very concise and powerful syntax to build a list from another list (or any iterable object). The syntax is the following:

result_list = [expression(item) for item in original_list if condition(item)]

which means that result_list will be a list containing expression(item) (an expression computed from item) for each item element of original_list for which condition(item) (a boolean expression involving item) is True. The boolean condition which allows you to filer the list is optional.

For example, to compute the list of squares of elements in a list, instead of:

l = [1, 2, 3]
result = []
for i in l:
    result.append(i*i)

which is particularly inefficient because of the repeated use of the append function, one could use a list comprehension:

l = [1, 2, 3]
result = [i*i for i in l] # result = [1, 4, 9]

In addition to being shorter, the above code is also faster (around 3x improvement) because you build the list in one instruction.

Another example to compute the list of square roots of all non-negative elements of a list (you could get in big troubles computing the square root of a negative element):

from math import sqrt

l = [4, -3, 9]
result = [sqrt(i) for i in l if i >= 0] # result = [2, 3]

The same syntax also exists for dictionaries, this is called dict comprehensions (very original, isn't it?). For example, to transform a list of (name, phone number) pairs into a dictionary, for faster lookup:

l = [("Barthes", "+33 6 29 64 91 12"), ("John", "+001 650 472 4243")]
d = {name: phone for (name, phone) in l}

You can get more details about list comprehensions on the dedicated section of the official documentation.

The multiples faces of in

The in keyword has many different meanings and makes Python code so easy to write that people often forget to use it.

• in gives a universal syntax to iterate over iterable objects. For example, to iterate over a list, instead of:

  l = [1, 2, 3]
  for i in range(len(l)):
      print l[i]
  

  you could simply write:

  l = [1, 2,3]
  for i in l:
      print i
  

  similarly, to iterate over a dictionary, instead of:

  d = { ... }
  for key in d.keys():
      print d[key]
  

  you could write:

  d = { ... }
  for key in d:
      print d[key]
  

• in also allows you to test whether an element belongs to some structure: list, dictionary (or any iterable object), occurrence of a substring inside a string. For example:

  l = [line for line in open("server.log") if "Connected" in line]
  

  will return the list of lines from the file server.log containing Connected as a substring.

Manipulating lists with atomic instructions

More generally, it is advised to avoid iterating over a list with a for loop. for loops are slow in Python and writing an operation over a list as a single instruction allows Python to optimize the execution of the code internally.

List comprehensions often help in replacing an iteration by a single instruction. Here are a few other functions which can be helpful in this regard:

• join can be useful to format a list. For example, to print the list of words whose first letter is a in a list of words. Instead of:

  l = [ ... ]
  result = ""
  for word in l:
      if word[0] == 'a':
          result += word + " "
  print result
  

  you could do:

  l = [ ... ]
  print " ".join([word for word in l if word[0] == 'a'])
  

• sum, to sum the elements of a list.

• map, to apply a given function to all elements in a list. For example to reverse all the words in a list:

  l = ["Adam", "Eve"]
  
  def reverse(word):
      return word[::-1]
  
  m = map(reverse, l) # m = ['madA', 'evE']
  

slices are also very useful when it comes to manipulating lists (or sublists) in blocks. Remember that if l is a list (or any iterable) l[begin:end:step] will extract all the elements from index begin (included) to index end (excluded) with a step of step (this last parameter being optional).

If the begin parameter is omitted, it is given 0 as default value. Similarly, the default value of end when unspecified is len(l) (the numbers of elements in l). A negative value for begin or end will be subtracted from the end of the list. For example, to extract all the element but the last one:

l = [1, 2, 3]
m = l[:-1] # m = [1, 2]

Using a negative value for the step parameter can be useful to walk through an iterable object in reverse order as shown in the example given above to take the mirror image of a word:

word = "dumbo"
drow = word[::-1] # drow = "obmud"

which compensates for the scandalous lack of a reverse function for strings in Python.

Exceptions

Exceptions provide a powerful tool found in many high-level programming languages which is often under-used. They allow for a less defensive programming style by handling errors as they appear instead of making test beforehand to prevent them from happening.

In Python, every time you are trying to execute an illegal operation (e. g. trying to access an element outside a list's boundaries, dividing by zero, etc.) instead of simply crashing the program, Python raises an exception which can be caught, giving the programmer a last chance to fix the problem before the program ultimately crashes.

The syntax to catch exceptions in Python is the following:

try:
    .... # piece of code potentially raising the exception named Kaboum
except Kaboum:
    .... # piece of code to be executed if the above code raises Kaboum

For example, if a line of code contains a division by a number which could seldom be equal to zero, instead of systematically checking that the number is non zero, it is much more efficient to encapsulate the line within a try ... except ZeroDivisionError: to handle specifically the rare cases when the number will be zero. This is the well-known principle: better ask for absolution than permission.

Another example, when trying to access an unbound key in a dictionary, Python raises the KeyError exception. This exception can be used to initialize the value associated with the unbound key. For example, to compute a dictionary of word counts in a text, you can often find:

text = "..."
result = {}
for word in text.split():
    if word in result:
        result[word] += 1
    else:
        result[word] = 1

You could instead use the KeyError exception to your advantage to avoid the systematic if test:

test = "..."
result = {}
for word in text.split():
    try:
        result[word] += 1
    except KeyError:
        result[word] = 1

The difference with the previous code is that most of the time, this code will behave exactly as if the body of the for loop only contained the instruction result[word] += 1. This gives a significant speedup compared to the first code where a test was computed for each iteration of the loop.

See the dedicated page in the official documentation.

Values equivalent to True or False

If test is a boolean variable (equal to True or False), we know that it is redundant to write:

if test == True:
    ...

instead of:

if test:
    ...

More generally, Python has automatic conversion rules from standard types to booleans. This can be used to shorten the syntax in conditional tests:

• as in the vast majority of programming languages, a positive integer is converted to True and zero is converted to False.

• a string is converted to False if and only if it is empty. For example, to test whether a string title is empty, you can simply write:

  if title:
      ...
  

  instead of:

  if len(title) > 0:
       ...
  

• the None value, a constant used to initialize unspecified variables, is converted to False. To test that a variable var is not equal to None, you can write:

  if not var:
      ...
  

  Beware, the above code will not allow you to distinguish the case where var is None from the case where var has a value which is converted to False by Python (for example, an empty string or list). You need to be careful that this is really what you are trying to test.

Generators

Generators provide an easy way to create iterator objects (objects over which you can iterate) and can be created in several ways.

l = [1, 2, 3]
m = (i*i for i in l)
print '\n'.join(m)

m = [i*i for i in l]

l = [1, 2, 3]
total = sum((i*i for i in l))

total = sum(i*i for i in l)

def min_max(filename):
    with open(filename) as f:
        for line in f:
            l = map(int, line.split())
            yield min(l), max(l)

for (inf, sup) in min_max(filename):
    print (inf + sup)/2.

with open(filename) as f:
    for line in f:
        l = map(int, line.split())
        inf, sup = min(l), max(l)
        print (inf + sup)/2.

Decorators

Decorators provide a very powerful way to alter the behavior of a function without redifining it. The syntax is the following:

@logging
def f(x):
    return x + 1

In the above example, we say that f has been decorated with logging. logging must be a function taking another function as an argument. The result of this decoration is equivalent to this piece of code:

def f(x):
    return x + 1

f = logging(f)

which means that by decorating f with logging, f now behaves as the composite function logging(f).

def logging(fun):
    def aux(*args, **kwargs):
        print "Calling", fun.__name__
        fun(*args, **kwargs)
    return aux

from functools import wraps

def logging(fun):
    @wraps(fun)
    def aux(*args, **kwargs):
        print "Calling", fun.__name__
        fun(*args, **kwargs)
    return aux

@logging
def f(x):
    return x + 1

print f.__name__

from functools import wraps

def logging(fun):
    a = 0
    @wraps(fun)
    def aux(*args, **kwargs):
        a = a + 1
        print "{0} has been called {1} times".format(fun.__name__, a)
        fun(*args, **kwargs)
    return aux

a = a + 1

from functools import wraps

def logging(fun):
    a = [0]
    @wraps(fun)
    def aux(*args, **kwargs):
        a[0] = a[0] + 1
        print "{0} has been called {1} times".format(fun.__name__, a[0])
        fun(*args, **kwargs)
    return aux

from functools import wraps

def memoize(fun):
    cache = {}
    @wraps(fun)
    def aux(x):
        if x in cache:
            return cache[x]
        else:
            a = fun(x)
            cache[x] = a
            return a
    return aux

@memoize
def f(x):
    ... # very long and heavy computation

Classes with two methods

Let us briefly recall how classes work in Python. A class is defined like this:

class Cipher:

    def __init__(self, key):
        self.key = key

    def decrypt(self, message):
        return (message & self.key)

all the methods of a class take as their first argument the instance on which the method is being called. By convention, this first arguments is always named self. If a is an instance of Cipher, the instruction a.decrypt(message) is equivalent to decrypt(a, message).

The special function __init__ is the class constructor and is called every time an instance of the class is created. Its typical use is to initialize some attributes of the instance. An instance of Cipher class can be created like this:

d = Cipher(key)

A flaw commonly found in code written by people coming from object-oriented programing languages is to create classes for everything. This often leads to classes containing only two methods, one being the __init__ function. This is the case in the class written above as an example. By looking to this example a bit closer, you can see that it is possible to completely get rid of the class definition: a decrypt function taking the key as an additional argument is sufficient:

def decrypt(key, message):
    return (message & key)

Some people could object that it still makes sense to use a class in the example above, if we plan to extend the Cypher class in the future, for example by adding an encrypt function. In my opinion, it is better to start by writing your code as simply as possible. If you really need to extend the code, then you can start restructuring it and group several related functions in a class.

PEP 8

When writing about good practices in Python, it is impossible not to mention the PEP8. It is a set of recommendations on coding style in Python. These recommendations are of course not absolute rules and should be taken as advice. However, I noticed that following these recommendations generally leads to greater code readability. Moreover, as many people who code in Python also follow these recommendations, adopting them reduces the gap between your code and code written by others: this will save you some time when reading code.

Here are a few points extracted from the PEP8:

• you should follow English typographic rules: no space before a colon, no space before a comma, but a space after, etc.
• you should put spaces around operators like the equal sign, plus sign, etc.
• you should try to limit the length of the lines of code to a maximum of 80 characters.

More details on the PEP8 page.