1. Acknowledgements

I would love to take the opportunity to thank all who have reviewed and spotted issues in the manuscript. This includes but is not limited to Ngozi Nwosu for taking the time out to review the whole manual and point out a whole load of grammatical errors, Olivia Enewally, Roman Turna and Abhen Ng for pointing out some factual and grammatical errors.A whole lot of other people on Reddit have pointed out errors and to those people I am really grateful.

Without the input of all these people, this manuscript would be worth less than it currently is. Thank you all!

2. An Introduction

The Python Programming language has been around for quite a while. Development work was started on the first version of Python by Guido Van Rossum in 1989. Since then, it has grown to become a highly loved and revered language that has been used and continues to be used in a host of different application types.

The Python interpreter and the extensive standard library that come with the interpreter are available for free in source or binary form for all major platforms from the Python Web site. This site also contains distributions of and pointers to many free third party Python modules, programs and tools, and additional documentation.

The Python interpreter can easily be extended with new functions and data types implemented in C, C++ or any other language that is callable from C. Python is also suitable as an extension language for customisable applications. One of the most notable feature of python is the easy and white-space aware syntax.

This book is intended as a concise intermediate level treatise on the Python programming language. There is a need for this due to the lack of availability of materials for python programmers at this level. The material contained in this book is targeted at the programmer that has been through a beginner level introduction to the Python programming language or that has some experience in a different object oriented programming language such as Java and wants to gain a more in-depth understanding of the Python programming language in a holistic manner. It is not intended as an introductory tutorial for beginners although programmers with some experience in other languages may find the very short tutorial included instructive.

The book covers only a handful of topics but tries to provide a holistic and in-depth coverage of these topics. It starts with a short tutorial introduction to get the reader up to speed with the basics of Python; experienced programmers from other object oriented languages such as Java may find that this is all the introduction to Python that they need. This is followed by a discussion of the Python object model then it moves on to discussing object oriented programming in Python. With a firm understanding of the Python object model, it goes ahead to discuss functions and functional programming. This is followed by a discussion of meta-programming techniques and their applications. The remaining chapters cover generators, a complex but very interesting topic in Python, modules and packaging, and python runtime services. In between, intermezzos are used to discuss topics are worth knowing because of the added understanding they provide.

I hope the content of the book achieves the purpose for the writing of this book. I welcome all feedback readers may have and actively encourage readers to provide such feedback.

2.1 The Evolution of Python

In December 1989, Guido Van Rossum started work on a language that he christened Python. Guido Van Rossum had been part of the team that worked on the ABC programming language as part of the Amoeba operating systems in the 1980s at CWI (Centrum Wiskunde & Informatica) in Amsterdam and although he liked the ABC programming language, he was frustrated by a number of features or lack of thereof. Guido wanted a high level programming language that would speed up the development of utilities for the the Amoeba project and ABC was not the answer. The ABC programming language would however play a very influential role in the development of python as Guido took parts he liked from the language and provided solutions for aspects of the ABC programming language that he found frustrating.

Guido released the first version of the Python programming language in February 1991. This release was object oriented, had a module system, included exception handling, functions, and the core data types of list, dict, str and others. Python version 1.0 was released in January 1994 and this release included functional programming constructs such as lambda, map, filter and reduce.

Python 1.2 was the last version released while Guido was still at CWI. In 1995, Van Rossum continued his work on Python at the Corporation for National Research Initiatives (CNRI) in Reston, Virginia where he released several versions of the language with indirect funding support from DARPRA.

By version 1.4, Python had acquired several new features including the Modula-3 inspired keyword arguments and built-in support for complex numbers. It also included a basic form of data hiding by name mangling. Python 1.5 was released on December 31, 1997 while python 1.6 followed on September 5, 2000.

Python 2.0 was released on October 16, 2000 and it introduced list comprehensions, a feature borrowed from the functional programming languages SETL and Haskell as well as a garbage collection system capable of collecting reference cycles.

Python 2.2 was the first major update to the Python type system. This update saw the unification of Python’s in-built types and user defined classes written in Python into one hierarchy. This single unification made Python’s object model purely and consistently object oriented. This update to the class system of Python added a number of features that improved the programming experience. These included:

1. The ability to subclass in built types such as dicts and lists.
2. The addition of static and class methods
3. The addition of properties defined by get and set methods.
4. An update to meta-classes, __new__() and super() methods, and MRO algorithms.

The next major milestone was Python 3 released on December 2, 2008. This was designed to rectify certain fundamental design flaws in the language that could not be implemented while maintaining full backwards compatibility with the 2.x series.

2.2 Python 2 vs Python 3

Perhaps the most visible and disruptive change to the Python ecosystem has been the introduction of Python 3. The major changes introduced into Python 3 include the following:

1. print() is now a function
2. Some well known python APIs such as range(), dict.keys(), dict.values() return views and iterators rather than lists improving efficiency when they are used.
3. The rules for ordering comparisons have been simplified. For example, a heterogeneous list cannot be sorted, because all the elements of a list must be comparable to each other.
4. The integer types have been whittled down to only one, i.e. int. long is also an int.
5. The division of two integers returns a float instead of an integer. // can be used to return an integer when division takes place.
6. All texts are now unicode but encoded unicode text is represented as binary data and any attempt to mix text and data will result in an exception. This breaks backwards compatibility with python 2.x versions.
7. Python 3 also saw the introduction of some new syntax such as function annotations, the nonlocal statement, extended iterable unpacking, set literals, dictionary comprehensions etc.
8. Python 3 also saw update to some syntax such as that of exception handling, meta-class specification, list comprehensions etc.

The full details on changes from python 2 to python 3 can be viewed on the python website. The rest of the book will assumes the use of Python 3.4.

2.3 The Python Programming Language

The Python programming language refers to the language as documented in the language reference. There is no official language specification but the language reference provides enough details to guide anyone implementing the Python programming language. The implementation of the Python programming language available on the Python website is an implementation written in C and commonly referred to as CPython. This is normally used as the reference implementation. However, there are other Python implementations in different languages. Popular among these are PyPy: python implemented in python and Jython: python implemented in Java. For the rest of this book, the reference CPython version that is freely distributed through the Python website is used.

3. A Very Short Tutorial

This short tutorial introduces the reader informally to the basic concepts and features of the Python programming language. It is not intended to be a comprehensive introductory tutorial to programming for a complete novice but rather assumes the reader has had previous experience programming with an object oriented programming language.

3.1 Using Python

Python is installed by default on most Unix-based systems including the Mac OS and various Linux-based distributions. To check if python is installed, open the command-line and type python. If not installed then python can be installed by visiting the python language website and following instructions on how to install python for the given platform.

The Python Interpreter

The python interpreter can be invoked by opening the command-line and typing in python. In the case that it has been installed but the command cannot be found then the full path to the installation should be used or added to the path. Invoking the interpreter using python brings up the python interactive session with an REPL prompt. The primary prompt, >>>, signals a user to enter statements while the secondary prompt, ..., signals a continuation line as shown in the following example.

    >>> def hello():
    ...     print("Hello world")
    ... 
    >>> 

A user can type in python statements at the interpreter prompt and get instant feedback. For example, we can evaluate expressions at the REPL and get values for such expressions as in the following example.

    Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
    [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> var_1 = 3
    >>> var_2 = 3
    >>> var_1*var_2
    9
    >>> 

Typing Ctrl-D at the primary prompt causes the interpreter to exit the session.

3.2 Python Statements, Line Structure and Indentation

A program in Python is composed of a number of logical lines and each of these logical lines is delimited by the NEWLINE token. Each logical line is equivalent to a valid statement. Compound statements however can be made up of multiple logical lines. A logical line is made from one or more physical lines using the explicit or implicit line joining rules. A physical line is a sequence of characters terminated by an end-of-line sequence. Python implicitly sees physical lines as logical lines eliminating the explicit need for semi-colons in terminating statements as in Java. Semi-colons however play a role in python; it is possible to have multiple logical lines on the same physical line by separating the logical lines with semi-colons such as shown below:

    >>> i = 5; print i;
    5

Multiple physical lines can be explicitly joined into a single logical line by use of the line continuation character, \, as shown below:

    >>> name = "Obi Ike-Nwosu"
    >>> cleaned_name = name.replace("-", " "). \
    ...                 replace(" ", "")
    >>> cleaned_name
    'ObiIkeNwosu'
    >>> 

Lines are joined implicitly, thus eliminating the need for line continuation characters, when expressions in triple quoted strings, enclosed in parenthesis (…), brackets [….] or braces {…} spans multiple lines.

From discussions above, it can be inferred that there are two types of statements in python:

1. Simple statements that span a single logical line. These include statements such as assignment statements, yield statements etc. A simple statement can be summarized as follows:

    simple_stmt ::=  expression_stmt
                 | assert_stmt
                 | assignment_stmt
                 | augmented_assignment_stmt
                 | pass_stmt
                 | del_stmt
                 | return_stmt
                 | yield_stmt
                 | raise_stmt
                 | break_stmt
                 | continue_stmt
                 | import_stmt
                 | global_stmt
                 | nonlocal_stmt
                 
    ::= means is defined as
    | means or

1. Compound statements that span multiple logical lines statements. These include statements such as the while and for statements. A compound statement is summarized as thus in python:

    compound_stmt ::=  if_stmt
                   | while_stmt
                   | for_stmt
                   | try_stmt
                   | with_stmt
                   | funcdef
                   | classdef
    suite         ::=  stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
    statement     ::=  stmt_list NEWLINE | compound_stmt
    stmt_list     ::=  simple_stmt (";" simple_stmt)* [";"]

Compound statements are made up of one or more clauses. A clause is made up of a header and a suite. The clause headers for a given compound statement are all at the same indentation level; they begin with a unique identifier, while, if etc., and end with a colon. The suite execution is controlled by the header. This is illustrate with the example below:

    >>> num = 6
    # if statement is a compound statement
        # clause header controls execution of indented block that follows
    >>> if num % 2 == 0: 
            # indented suite block
    ...     print("The number {} is even".format(num))
    ... 
    The number 6 is even
    >>> 

The suite may be a set of one or more statements that follow the header’s colon with each statement separated from the previous by a semi-colon as shown in the following example.

```python
    >>> x = 1
    >>> y = 2
    >>> z = 3
    >>> if x < y < z: print(x); print(y); print(z)
    ... 
    1
    2
    3
``` 

The suite is conventionally written as one or more indented statements on subsequent lines that follow the header such as below:

```python
    >>> x = 1
    >>> y = 2
    >>> z = 3
    >>> if x < y < z: 
    ...    print(x)
    ...    print(y); 
    ...    print(z)
    ... 
    1
    2
    3
```     

Indentations are used to denote code blocks such as function bodies, conditionals, loops and classes. Leading white-space at the beginning of a logical line is used to compute the indentation level for that line, which in turn is used to determine the grouping of statements. Indentation used within the code body must always match the indentation of the the first statement of the block of code.

3.3 Strings

Strings are represented in Python using double "..." or single '...' quotes. Special characters can be used within a string by escaping them with \ as shown in the following example:

    # the quote is used as an apostrophe so we escape it for python to 
    # treat is as an apostrophe rather than the closing quote for a string
    >>> name = 'men\'s'
    >>> name
    "men's"
    >>> 

To avoid the interpretation of characters as special characters, the character, r, is added before the opening quote for the string as shown in the following example.

    >>> print('C:\some\name')  # here \n means newline!
    C:\some
    ame
    >>> print(r'C:\some\name')  # note the r before the quote
    C:\some\name

String literals that span multiple lines can be created with the triple quotes but newlines are automatically added at the end of a line as shown in the following snippet.

    >>> para = """hello world I am putting together a 
    ... book for beginners to get to the next level in python"""
    # notice the new line character 
    >>> para
    'hello world I am putting together a \nbook for beginners to get to the next level in python'
    # printing this will cause the string to go on multiple lines
    >>> print(para)
    hello world I am putting together a 
    book for beginners to get to the next level in python
    >>> 

To avoid the inclusion of a newline, the continuation character \ should be used at the end of a line as shown in the following example.

    >>> para = """hello world I am putting together a \
    ... book for beginners to get to the next level in python"""
    >>> para
    'hello world I am putting together a book for beginners to get to the next level in python'
    >>> print(para)
    hello world I am putting together a book for beginners to get to the next level in python
    >>> 

String are immutable so once created they cannot be modified. There is no character type so characters are assumed to be strings of length, 1. Strings are sequence types so support sequence type operations except assignment due to their immutability. Strings can be indexed with integers as shown in the following snippet:

    >>> name = 'obiesie'
    >>> name[1]
    'b'
    >>> 

Strings can be concatenated to create new strings as shown in the following example

    >>> name = 'obiesie'
    >>> surname = " Ike-Nwosu"
    >>> full_name = name + surname
    >>> full_name
    'obiesie Ike-Nwosu'
    >>> 

One or more string literals can be concatenated together by writing them next to each other as shown in the following snippet:

    >>> 'Py' 'thon'
    'Python'
    >>> 

The built-in method len can also be used to get the length of a string as shown in the following snippet.

    >>> name = "obi"
    >>> len(name)
    3
    >>> 

3.4 Flow Control

if-else and if-elif-else statements

Python supports the if statement for conditional execution of a code block.

    >>> name = "obi"
    >>> if name == "obi":
    ...     print("Hello Obi")
    ... 
    Hello Obi
    >>> 

The if statement can be followed by zero or more elif statements and an optional else statement that is executed when none of the conditions in the if or elif statements have been met.

    >>> if name == "obi":
    ...     print("Hello Obi")
    ... elif name == "chuks":
    ...     print("Hello chuks")
    ... else:
    ...    print("Hello Stranger")
    Hello Stranger
    >>> 

for and range statements

The while and for statements constitute the main looping constructs provided by python.

The for statement in python is used to iterate over sequence types (lists, sets, tuples etc.). More generally, the for loop is used to iterate over any object that implements the python iterator protocol. This will be discussed further in chapters that follow. Example usage of the for loop is shown by the following snippet:

    >>> names = ["Joe", "Obi", "Chris", "Nkem"]
    >>> for name in names:
    ...     print(name)
    ... 
    Joe
    Obi
    Chris
    Nkem
    >>> 

Most programming languages have a syntax similar to the following for iterating over a progression of numbers:

    for(int x = 10; x < 20; x = x+1) {
             // do something here
          }

Python replaces the above with the simpler range() statement that is used to generate an arithmetic progression of integers. For example:

    >>> for i in range(10, 20):
    ...     print i
    ... 
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    >>> 

The range statement has a syntax of range(start, stop, step). The stop value is never part of the progression that is returned.

while statement

The while statement executes the statements in its suite as long as the condition expression in the while statement evaluates to true.

    >>> counter = 10
    >>> while counter > 0: # the conditional expression is 'counter>0'
    ...     print(counter)
    ...     counter = counter - 1
    ... 
    10
    9
    8
    7
    6
    5
    4
    3
    2
    1

break and continue statements

The break keyword is used to escape from an enclosing loop. Whenever the break keyword is encountered during the execution of a loop, the loop is abruptly exited and no other statement within the loop is executed.

    >>> for i in range(10):
    ...     if i == 5:
    ...             break
    ...     else:
    ...             print(i)
    ... 
    0
    1
    2
    3
    4

The continue keyword is used to force the start of the next iteration of a loop. When used the interpreter ignores all statements that come after the continue statement and continues with the next iteration of the loop.

    >>> for i in range(10):
            # if i is 5 then continue so print statement is ignored and the next iteration
            # continues with i set to 6
    ...     if i == 5:
    ...             continue
    ...     print("The value is " + str(i))
    ... 
    The value is 0
    The value is 1
    The value is 2
    The value is 3
    The value is 4
    # no printed value for i == 6
    The value is 6
    The value is 7
    The value is 8
    The value is 9

In the example above, it can be observed that the number 5 is not printed due to the use of continue when the value is 5 however all subsequent values are printed.

else clause with looping constructs

Python has a quirky feature in which the else keyword can be used with looping constructs. When an else keyword is used with a looping construct such as while or for, the statements within the suite of the else statement are executed as long as the looping construct was not ended by a break statement.

    # loop exits normally 
    >>> for i in range(10):
    ...     print(i)
    ... else:
    ...     print("I am in quirky else loop")
    ... 
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    I am in quirky else loop
    >>> 

If the loop was exited by a break statement, the execution of the suite of the else statement is skipped as shown in the following example:

    >>> for i in range(10):
    ...     if i == 5:
    ...             break
    ...     print(i)
    ... else:
    ...     print("I am in quirky else loop")
    ... 
    0
    1
    2
    3
    4
    >>> 

Enumerate

Sometimes, when iterating over a list, a tuple or a sequence in general, having access to the index of the item, as well as the item being enumerated over maybe necessary. This could achieved using a while loop as shown in the following snippet:

    >>> names = ["Joe", "Obi", "Chris", "Jamie"]
    >>> name_count = len(names)
    >>> index = 0
    >>> while index < name_count:
    ...     print("{}. {}".format(index, names[index]))
    ...     index = index + 1
    ... 
    0. Joe
    1. Obi
    2. Chris
    3. Jamie

The above solution is how one would go about it in most languages but python has a better alternative to such in the form of the enumerate keyword. The above solution can be reworked beautifully in python as shown in the following snippet:

    >>> for index, name in enumerate(names):
    ...     print("{}. {}".format(index, name))
    ... 
    0. Joe
    1. Obi
    2. Chris
    3. Jamie
    >>> 

3.5 Functions

Named functions are defined with the def keyword which must be followed by the function name and the parenthesized list of formal parameters. The returnkeyword is used to return a value from a function definition. A python function definition is shown in the example below:

    def full_name(first_name, last_name):
        return " ".join((first_name, last_name))

Functions are invoked by calling the function name with required arguments in parenthesis for example full_name("Obi", "Ike-Nwosu"). Python functions can return multiple values by returning a tuple of the required values as shown in the example below in which we return the quotient and remainder from a division operation:

    >>> def divide(a, b):
    ...     return divmod(q, r)
    ... 
    >>> divide(7, 2)
    (3, 1)
    >>> 

Python functions can be defined without return keyword. In that case the default returned value is None as shown in the following snippet:

    >>> def print_name(first_name, last_name):
    ...     print(" ".join((first_name, last_name)))
    ... 
    >>> print_name("Obi", "Ike-Nwosu")
    Obi Ike-Nwosu
    >>> x = print_name("Obi", "Ike-Nwosu")
    Obi Ike-Nwosu
    >>> x
    >>> type(x)
    <type 'NoneType'>
    >>> 

The return keyword does not even have to return a value in python as shown in the following example.

    >>> def dont_return_value():
    ...     print("How to use return keyword without a value")
    ...     return
    ... 
    >>> dont_return_value()
    How to use return keyword without a value

Python also supports anonymous functions defined with the lambda keyword. Python’s lambda support is rather limited, crippled a few people may say, because it supports only a single expression in the body of the lambda expression. Lambda expressions are another form of syntactic sugar and are equivalent to conventional named function definition. An example of a lambda expression is the following:

    >>> square_of_number = lambda x: x**2
    >>> square_of_number
    <function <lambda> at 0x101a07158>
    >>> square_of_number(2)
    4
    >>> 

3.6 Data Structures

Python has a number of built-in data structures that make programming easy. The built-in data structures include lists, tuples and dictionaries.

1. Lists: Lists are created using square brackets, [] or the list() function. The empty list is denoted by []. Lists preserve the order of items as they are created or insert into the list. Lists are sequence types so support integer indexing and all other sequence type subscripting that will be discussed in chapters that follow. Lists are indexed by integers starting with zero and going up to the length of the list minus one.

    >>> name = ["obi", "ike", "nwosu"]
    >>> name[0]
    'obi'
    >>> 
   

Items can be added to a list by appending to the list.

>>>     name = ["obi", "ike", "nwosu"]
>>> name.append("nkem")
>>> names 
["obi", "ike", "nwosu", "nkem"]

Elements can also be added to other parts of a list not just the end using `insert` method.

>>> name = ["obi", "ike", "nwosu"]
>>> name.insert(1, "nkem")
>>> names 
["obi", "nkem", "ike", "nwosu"]

Two or more lists can be concatenated together with the `+` operator.

>>> name = ["obi", "ike", "nwosu"]
>>> name1 = ["James"]
>>> name + name1 
["obi", "ike", "nwosu", "James"]

To get a full listing of all methods of the list, run the help command with list as argument.

1. Tuples: These are also another type of sequence structures. A tuple consists of a number of comma separated objects for example.

>>> companies = "Google", "Microsoft", "Tesla"
>>> companies
('Google', 'Microsoft', 'Tesla')
>>> 

When defining a non-empty tuple the parenthesis is optional but when the tuple is part of a larger expression, the parenthesis is required. The parenthesis come in handy when defining an empty tuple for instance:

>>> companies = ()
>>> type(companies)
<class 'tuple'>
>>> 

Tuples have a quirky syntax that some people may find surprising. When defining a single element tuple, the comma must be included after the single element regardless of whether or not parenthesis are included. If the comma is left out then the result of the expression is not a tuple. For instance:

>>> company = "Google",
>>> type(company)
<class 'tuple'>
>>> 
>>> company = ("Google",)
>>> type(company)
<class 'tuple'>
# absence of the comma returns the value contained within the parenthesis
>>> company = ("Google")
>>> company
'Google'
>>> type(company)
<class 'str'>
>>> 

Tuples are integer indexed just like lists but are immutable; once created the contents cannot be changed by any means such as by assignment. For instance:

    >>> companies = ("Google", "Microsoft", "Palantir")
    >>> companies[0]
    'Google'
    >>> companies[0] = "Boeing"
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'tuple' object does not support item assignment
    >>> 

However, if the object in a tuple is a mutable object such as a list, such object can be changed as shown in the following example:

    >>> companies = (["lockheedMartin", "Boeing"], ["Google", "Microsoft"])
    >>> companies
    (['lockheedMartin', 'Boeing'], ['Google', 'Microsoft'])
    >>> companies[0].append("SpaceX")
    >>> companies
    (['lockheedMartin', 'Boeing', 'SpaceX'], ['Google', 'Microsoft'])
    >>> 

1. Sets: A set is an unordered collection of objects that does not contain any duplicates. An empty set is created using set() or by using curly braces, {}. Sets are unordered so unlike tuples or lists they cannot be indexed by integers. However sets, with the exception of frozen sets, are mutable so one can add, update or remove from a set as shown in the following:

    >>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
    >>> basket_set = set()
    >>> basket_set
    set()
    >>> basket_set.update(basket)
    >>> basket_set
    {'pear', 'orange', 'apple', 'banana'}
    >>> basket_set.add("clementine")
    >>> basket_set
    {'pear', 'orange', 'apple', 'banana', 'clementine'}
    >>> basket_set.remove("apple")
    >>> basket_set
    {'pear', 'orange', 'banana', 'clementine'}
    >>> 
   

2. Dictionary: This is a mapping data structure that is commonly referred to as an associative array or a hash table in other languages. Dictionaries or dicts as they are commonly called are indexed by keys that must be immutable. A pair of braces, {...} or method dict() is used to create a dict. Dictionaries are unordered set of key:value pairs, in which the keys are unique. A dictionary can be initialized by placing a set of key:value pairs within the braces as shown in the following example.

    ages = {"obi": 24,
            "nkem": 23,
            "Chris": 23
            }
   

   The primary operations of interest that are offered by dictionaries are the storage of a value by the key and retrieval of stored values also by key. Values are retrieved by using indexing the dictionary with the key using square brackets as shown in the following example.

    >>> ages["obi"]
    24
   

   Dictionaries are mutable so the values indexed by a key can be changed, keys can be deleted and added to the dict.

Python’s data structures are not limited to just those listed in this section. For example the collections module provides additional data structures such as queues and deques however the data structures listed in this section form the workhorse for most Python applications. To get better insight into the capabilities of a data structure, the help() function is used with the name of the data structure as argument for example, help(list).

3.7 Classes

The class statement is used to define new types in python as shown in the following example:

    class Account:
        # class variable that is common to all instances of a class
    	num_accounts = 0

    	def __init__(self, name, balance):
    	    # start of instance variable
    		self.name = name 
    		self.balance = balance 
    		# end of instance variables
    		Account.num_accounts += 1
    		
    	def deposit(self, amt):
    		self.balance = self.balance + amt 

    	def withdraw(self, amt):
    		self.balance = self.balance - amt 

    	def inquiry(self):
    		return "Name={}, balance={}".format(self.name, self.balance) 

    	@classmethod 
    	def from_dict(cls, params):
    		params_dict = json.loads(params)
    		return cls(params_dict.get("name"), params_dict.get("balance"))

Classes in python just like classes in other languages have class variables, instance variables, class methods, static methods and instance methods. When defining classes, the base classes are included in the parenthesis that follows the class name. For those that are familiar with Java, the __init__ method is something similar to a constructor; it is in this method that instance variables are initialized. The above defined class can be initialized by calling the defined class with required arguments to __init__ in parenthesis ignoring the self argument as shown in the following example.

>>> acct = Account("obie", 10000000)

Methods in a class that are defined with self as first argument are instance methods. The self argument is similar to this in java and refers to the object instance. Methods are called in python using the dot notation syntax as shown below:

>>> acct = Account("obie", 10000000)
>>>account.inquiry()
Name=obie, balance=10000000

Python comes with built-in function, dir, for introspection of objects. The dir function can be called with an object as argument and it returns a list of all attributes, methods and variables, of a class.

3.8 Modules

Functions and classes provide mean for structuring your Python code but as the code grows in size and complexity, there is a need for such code to be split into multiple files with each source file containing related definitions. The source files can then be imported as needed in order to access definitions in any of such source file. In python, we refer to source files as modules and modules have the .py extensions.

For example, the Account class definition from the previous section can be saved to a module called Account.py. To use this module else where, the import statement is used to import the module as shown in the following example:

>>> import Account
>>> acct = Account.Account("obie", 10000000)

Note that the import statement takes the name of the module without the .py extension. Using the import statement creates a name-space, in this case the Account name-space and all definitions in the module are available in such name-space. The dot notation (.) is used to access the definitions as required. An alias for an imported module can also be created using the as keyword so the example from above can be reformulated as shown in the following snippet:

    >>> import Account as acct
    >>> account = acct.Account("obie". 10000000)

It is also possible to import only the definitions that are needed from the module resulting in the following:

>>> from Account import Account
>>> account = Account("obie", 10000000)

All the definitions in a module can also be imported by using the wild card symbol a shown below:

>>> from Account import *

This method of imports is not always advised as it can result in name clashes when one of the name definitions being imported is already used in the current name-space. This is avoided by importing the module as a whole. Modules are also objects in Python so we can introspect on them using the dir introspection function. Python modules can be further grouped together into packages. Modules and packages are discussed in depth in a subsequent chapter that follows.

3.9 Exceptions

Python has support for exceptions and exception handling. For example, when an attempt is made to divide by zero, a ZeroDivisionError is thrown by the python interpreter as shown in the following example.

>>> 2/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero
>>> 

During the execution of a program, an exception is raised when an error occurs; if the exception is not handled, a trace-back is dumped to the screen. Errors that are not handled will normally cause an executing program to terminate.

Exceptions can be handled in Python by using the try...catch statements. For example, the divide by zero exception from above could be handled as shown in the following snippet.

    >>> try:
    ...     2/0
    ... except ZeroDivisionError as e:
    ...     print("Attempting to divide by 0. Not allowed")
    ... 
    Attempting to divide by 0. Not allowed
    >>> 

Exceptions in python can be of different types. For example, if an attempt was made to catch an IOError in the previous snippet, the program would terminate because the resulting exception type is a ZeroDivisionError exception. To catch all types of exceptions with a single handler, try...catch Exception is used but this is advised against as it becomes impossible to tell what kind of exception has occurred thus masking the exception

Custom exceptions can be defined to handle custom exceptions in our code. To do this, define a custom exception class that inherits from the Exception base class.

3.10 Input and Output

Python as expected has support for reading and writing to and from input and output sources. The file on the hard drive is the most popular IO device. The content of a file can be opened and read from using the snippet below:

f = open("afile.txt")
line = f.readline()
while line:
    print(line)
    line = f.readline()

The open method returns a file object or throws an exception if the file does not exist. The file object supports a number of methods such as read that reads the whole content of the file into a string or readline that reads the contents of the file one line at a time. Python supports the following syntactic sugar for iterating through the lines of a file.

for line in open("afile.txt"):
    print(line)

Python supports writing to a file as shown below:

f = open("out.txt", "w")
contents = ["I", "love", "python"]
for content in contents:
    f.write(content)
f.close()

Python also has support for writing to standard input and standard output. This can be done using the sys.stdout.write() or the sys.stdin.readline() from the sys module.

3.11 Getting Help

The python programming language has a very detailed set of documentation that can be obtained at the interpreter prompt by using the help method. To get more information about a syntactic construct or data structure, pass it as an argument to the help function for example help(list).

4. Intermezzo: Glossary

A number of terms and esoteric python functions are used throughout this book and a good understanding of these terms is integral to gaining a better. and deeper understanding of python. A description of these terms and functions is provided in the sections that follow.

4.1 Names and Binding

In python, objects are referenced by names. names are analogous to variables in C++ and Java.

    >>> x = 5

In the above, example, x is a name that references the object, 5. The process of assigning a reference to 5 to x is called binding. A binding causes a name to be associated with an object in the innermost scope of the currently executing program. Bindings may occur during a number of instances such as during variable assignment or function or method call when the supplied parameter is bound to the argument. It is important to note that names are just symbols and they have no type associated with them; names are just references to objects that actually have types

4.2 Code Blocks

A code block is a piece of program code that is executed as a single unit in python. Modules, functions and classes are all examples of code blocks. Commands typed in interactively at the REPL, script commands run with the -c option are also code blocks. A code block has a number of name-spaces associated with it. For example, a module code block has access to the globalname-space while a function code block has access to the local as well as the global name-spaces.

4.3 Name-spaces

A name-space as the name implies is a context in which a given set of names is bound to objects. Name-spaces in python are currently implemented as dictionary mappings. The built-in name-space is an example of a name-space that contains all the built-in functions and this can be accessed by entering __builtins__.__dict__ at the terminal (the result is of a considerable amount). The interpreter has access to multiple name-spaces including the global name-space, the built-in name-space and the local name-space. These name-spaces are created at different times and have different lifetimes. For example, a new local name-space is created at the invocation of a function and forgotten when the function exits or returns. The global name-space is created at the start of the execution of a module and all names defined in this name-space are available module-wide while the built-in name-space comes into existence when the interpeter is invoked and contains all the built-in names. These three name-spaces are the main name-space available to the interpreter.

4.4 Scopes

A scope is an area of a program in which a set of name bindings (name-spaces) is visible and directly accessible. Direct access is an important characteristic of a scope as will be explained when classes are discussed. This simply means that a name, name, can be used as is, without the need for dot notation such as SomeClassOrModule.name to access it. At runtime, the following scopes may be available.

1. Inner most scope with local names
2. The scope of enclosing functions if any (this is applicable for nested functions)
3. The current module’s globals scope
4. The scope containing the builtin name-space.

Whan a name is used in python, the interpreter searches the name-spaces of the scopes in ascending order as listed above and if the name is not found in any of the name-spaces, an exception is raised. Python supports static scoping also known as lexical scoping; this means that the visibility of a set of name bindings can be inferred by only inspecting the program text.

Note

Python has a quirky scoping rule that prevents a reference to an object in the global scope from being modified in a local scope; such an attempt will throw an UnboundLocalError exception. In order to modify an object from the global scope within a local scope, the global keyword has to be used with the object name before modification is attempted. The following example illustrates this.

    >>> a = 1
    >>> def inc_a(): a += 2
    ... 
    >>> inc_a()
    Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<stdin>", line 1, in inc_a
    UnboundLocalError: local variable 'a' referenced before assignment

In order to modify the object from the global scope, the global statement is used as shown in the following snippet.

    >>> a = 1
    >>> def inc_a():
    ...     global a
    ...     a += 1
    ... 
    >>> inc_a()
    >>> a
    2

Python also has the nonlocal keyword that is used when there is a need to modify a variable bound in an outer non-global scope from an inner scope. This proves very handy when working with nested functions (also referred to as closures). A very trivial illustration of the nonlocal keyword in action is shown in the following snippet that defines a simple counter object that counts in ascending order.

    >>> def make_counter():
    ...     count = 0
    ...     def counter():
    ...         nonlocal count # nonlocal captures the count binding from enclosing scope not global\
 scope
    ...         count += 1
    ...         return count
    ...     return counter
    ... 
    >>> counter_1 = make_counter()
    >>> counter_2 = make_counter()
    >>> counter_1()
    1
    >>> counter_1()
    2
    >>> counter_2()
    1
    >>> counter_2()
    2

4.5 eval()

eval is a python built-in method for dynamically executing python expressions in a string (the content of the string must be a valid python expression) or code objects. The function has the following signature eval(expression, globals=None, locals=None). If supplied, the globals argument to the eval function must be a dictionary while the locals argument can be any mapping. The evaluation of the supplied expression is done using the globals and locals dictionaries as the global and local name-spaces. If the __builtins__ is absent from the globals dictionary, the current globals are copied into globals before expression is parsed. This means that the expression will have either full or restricted access to the standard built-ins depending on the execution environment; this way the exection environment of eval can be restricted or sandboxed. eval when called returns the result of executing the expression or code object for example:

```python
>>> eval("2 + 1") # note the expression is in a string
3
```

Since eval can take arbitrary code obects as argument and return the value of executing such expressions, it along with exec, is used in executing arbitrary Python code that has been compiled into code objects using the compile method. Online Python interpreters are able to execute python code supplied by their users using both eval and exec among other methods.

4.6 exec()

exec is the counterpart to eval. This executes a string interpreted as a suite of python statements or a code object. The code supplied is supposed to be valid as file input in both cases. exec has the following signature: exec(object[, globals[, locals]]). The following is an example of exec using a string and the current name-spaces.

Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  5 2014, 20:42:22) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
# the acct.py file is located somewhere on file
>>> cont = open('acct.py', 'r').read()
>>> cont
'class Account:\n    """base class for representing user accounts"""\n    num_accounts = 0\n\n    de\
f __init__(self, name, balance):\n        self.name = name \n        self.balance = balance \n      \
  Account.num_accounts += 1\n\n    def del_account(self):\n        Account.num_accounts -= 1\n\n    \
def __getattr__(self, name):\n        """handle attribute reference for non-existent attribute"""\n \
       return "Hey I dont see any attribute called {}".format(name)\n\n    def deposit(self, amt):\n\
        self.balance = self.balance + amt \n\n    def withdraw(self, amt):\n        self.balance = s\
elf.balance - amt \n\n    def inquiry(self):\n        return "Name={}, balance={}".format(self.name,\
 self.balance) \n\n'
>>> exec(cont)
# exec content of file using the default name-spaces
>>> Account # we can now reference the account class
<class '__main__.Account'>
>>> 

In all instances, if optional arguments are omitted, the code is executed in the current scope. If only the globals argument is provided, it has to be a dictionary, that is used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, the locals argument can be any mapping object. If the globals dictionary does not contain a value for the key __builtins__, a reference to the dictionary of the built-in module builtins is inserted under that key. One can control the builtins that are available to the executed code by inserting custom __builtins__ dictionary into globals before passing it to exec() thus creating a sandbox.

5. Objects 201

Python objects are the basic abstraction over data in python; every value is an object in python. Every object has an identity, a type and a value. An object’s identity never changes once it has been created. The id(obj) function returns an integer representing the obj's identity. The is operator compares the identity of two objects returning a boolean. In CPython, the id() function returns an integer that is a memory location for the object thus uniquely identifying such object. This is an implementation detail and implementations of Python are free to return whatever value uniquely identifies objects within the interpreter.

The type() function returns an object’s type; the type of an object is also an object itself. An object’s type is also normally unchangeable. An object’s type determines the operations that the object supports and also defines the possible values for objects of that type. Python is a dynamic language because types are not associated with variables so a variable, x, may refer to a string and later refer to an integer as shown in the following example.

x = 1
x = "Nkem"

However, Python unlike dynamic languages such as Javascript is strongly typed because the interpreter will never change the type of an object. This means that actions such as adding a string to a number will cause an exception in Python as shown in the following snippet:

>>> x = "Nkem"
>>> x + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't convert 'int' object to str implicitly

This is unlike Javascript where the above succeeds because the interpreter implicitly converts the integer to a string then adds it to the supplied string.

Python objects are either one of the following:

1. Mutable objects: These refer to objects whose value can change. For example a list is a mutable data structure as we can grow or shrink the list at will.

        >>> x = [1, 2, 4]
        >>> y = [5, 6, 7]
        >>> x = x + y
        >>> x
        [1, 2, 4, 5, 6, 7]
        >>> 
   

Programmers new to Python from other languages may find some behavior of mutable object puzzling; Python is a pass-by-object-reference language which means that the values of object references are the values passed to function or method calls and names bound to variables refer to these reference values. For example consider the snippets shown in the following example.

>>> x
[1, 2, 3]
# now x and y refer to the same list
>>> y = x
# a change to x will also be reflected in y
>>> x.extend([4, 5, 6])
>>> y
[1, 2, 3, 4, 5, 6]

y and x refer to the same object so a change to x is reflected in y. To fully understand why this is so, it must be noted that the variable, x does not actually hold the list, [1, 2, 3], rather it holds a reference that points to the location of that object so when the variable, y is bound to the value contained in x, it now also contains the reference to the original list, [1, 2, 3]. Any operation on x finds the list that x refers to and carries out the operation on the list; y also refers to the same list thus the change is also reflected in the variable, y.

1. Immutable objects: These objects have values that cannot be changed. A tuple is an example of an immutable data structure because once created we can not change the constituent objects as shown below:

        >>> x = (1, 2, 3, 4)
        >>> x[0]
        1
        >>> x[0] = 10
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        TypeError: 'tuple' object does not support item assignment
        >>> 
   

However if an immutable object contains a mutable object the mutable object can have its value changed even if it is part of an immutable object. For example, a tuple is an immutable data structure however if a tuple contains a list object, a mutable object, then we can change the value of the list object as shown in the following snippet.

```python 
    >>> t = [1, 2, 3, 4]
    >>> x = t,
    >>> x
    ([1, 2, 3, 4],)
    >>> x[0]
    [1, 2, 3, 4]
    >>> x[0].append(10)
    >>> x
    ([1, 2, 3, 4, 10],)
    >>> 
```

5.1 Strong and Weak Object References

Python objects get references when they are bound to names. This binding can be in form of an assignment, a function or method call that binds objects to argument names etc. Every time an object gets a reference, the reference count is increased. In fact the reference count for an object can be found using the sys.getrefcount method as shown in the following example.

>>> import sys
>>> l = []
>>> m = l
# note that there are 3 references to the list object, l, m and the binding 
# to the object argument for sys.getrefcount function
>>> sys.getrefcount(l)
3

Two kind of references, strong and weak references, exist in Python but when discussing references, it is almost certainly the strong reference that is being referred to. The previous example for instance, has three references and these are all strong references. The defining characteristic of a strong reference in Python is that whenever a new strong reference is created, the reference count for the referenced object is incremented by 1. This means that the garbage collector will never collect an object that is strongly referenced because the garbage collector collects only objects that have a reference count of 0. Weak references on the other hand do not increase the reference count of the referenced object. Weak referencing is provided by the weakref module. The following snippet shows weak referencing in action.

    >>> class Foo:
    ...     pass
    ... 
    >>> a = Foo()
    >>> b = a
    >>> sys.getrefcount(a)
    3
    >>> c = weakref.ref(a)
    >>> sys.getrefcount(a)
    3
    >>> c()
    <__main__.Foo object at 0x1012d6828>
    >>> type(c)
    <class 'weakref'>

The weakref.ref function returns an object that when called returns the weakly referenced object. The weakref module the weakref.proxy alternative to the weakref.ref function for creating weak references. This method creates a proxy object that can be used just like the original object without the need for a call as shown in the following snippet.

    >>> d = weakref.proxy(a)
    >>> d
    <weakproxy at 0x10138ba98 to Foo at 0x1012d6828>
    >>> d.__dict__
    {}

When all the strong references to an object have deleted then the weak reference looses it reference to the original object and the object is ready for garbage collection. This is shown in the following example.

    >>> del a 
    >>> del b
    >>> d
    <weakproxy at 0x10138ba98 to NoneType at 0x1002040d0>
    >>> d.__dict__
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ReferenceError: weakly-referenced object no longer exists
    >>> c()
    >>> c().__dict__
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'NoneType' object has no attribute '__dict__'

5.2 The Type Hierarchy

Python comes with its own set of built-in types and these built-in types broadly fall into one of the following categories:

None Type

The None type is a singleton object that has a single value and this value is accessed through the built-in name None. It is used to signify the absence of a value in many situations, e.g., it is returned by functions that don’t explicitly return a value as illustrated below:

```python 
    >>> def print_name(name):
    ...     print(name)
    ... 
    >>> name = print_name("nkem")
    nkem
    >>> name
    >>> type(name)
    <class 'NoneType'>
    >>> 
```

The None type has a truth value of false.

NotImplemented Type

The NotImplemented type is another singleton object that has a single value. The value of this object is accessed through the built-in name NotImplemented. This object should be returned when we want to delegate the search for the implementation of a method to the interpreter rather than throwing a runtime NotImplementedError exception. For example, consider the two types, Foo and Bar below:

    class Foo:
        def __init__(self, value):
            self.value = value

        def __eq__(self, other):
            if isinstance(other, Foo):
                print('Comparing an instance of Foo with another instance of Foo')
                return other.value == self.value
            elif isinstance(other, Bar):
                print('Comparing an instance of Foo with an instance of Bar')
                return other.value == self.value
            print('Could not compare an instance of Foo with the other class')
            return NotImplemented

    class Bar:
        def __init__(self, value):
            self.value = value

        def __eq__(self, other):
            if isinstance(other, Bar):
                print('Comparing an instance of Bar with another instance of Bar')
                return other.value == self.value
            print('Could not compare an instance of Bar with the other class')
            return NotImplemented

When an attempt is made at comparisons, the effect of returning NotImplemented can be clearly observed. In Python, a == b results in a call to a.__eq__(b). In this example, instance of Foo and Bar have implementations for comparing themselves to other instance of the same class, for example:

    >>> f = Foo(1)
    >>> b = Bar(1)
    >>> f == b
    Comparing an instance of Foo with an instance of Bar
    True
    >>> f == f
    Comparing an instance of Foo with another instance of Foo
    True
    >>> b == b
    Comparing an instance of Bar with another instance of Bar
    True
    >>> 

What actually happens when we compare f with b? The implementation of __eq__() in Foo checks that the other argument is an instance of Bar and handles it accordingly returning a value of True:

    >>> f == b
    Comparing an instance of Foo with an instance of Bar
    True

If b is compared with f then b.__eq__(f) is invoked and the NotImplemented object is returned because the implementation of __eq__() in Bar only supports comparison with a Bar instances. However, it can be seen in the following snippet that the comparison operation actually succeeds; what has happened?

    >>> b == f
    Could not compare an instance of Bar with the other class
    Comparing an instance of Foo with an instance of Bar
    True
    >>> 

The call to b.__eq__(f) method returned NotImplemented causing the python interpreter to invoke the __eq__() method in Foo and since a comparison between Foo and Bar is defined in the implementation of the __eq__() method in Foo the correct result, True, is returned.

The NotImplmented object has a truth value of true.

    >>>bool(NotImplemented)
    True

Ellipsis Type

This is another singleton object type that has a single value. The value of this object is accessed through the literal ... or the built-in name Ellipsis. The truth value for the Ellipsis object is true. The Ellipsis object is mainly used in numeric python for indexing and slicing matrices. The numpy documentation provides more insight into how the Ellipsis object is used.

Numeric Type

Numeric types are otherwise referred to as numbers. Numeric objects are immutable thus once created their value cannot be changed. Python numbers fall into one of the following categories:

1. Integers: These represent elements from the set of positive and negative integers. These fall into one of the following types:
   1. Plain integers: These are numbers in the range of -2147483648 through 2147483647 on a 32-bit machine; the range value is dependent on machine word size. Long integers are returned when results of operations fall outside the range of plain integers and in some cases, the exception OverflowError is raised. For the purpose of shift and mask operations, integers are assumed to have a binary, 2’s complement notation using 32 or more bits, and hiding no bits from the user.
   2. Long integers: Long integers are used to hold integer values that are as large as the virtual memory on a system can handle. This is illustrated in the following example.

       >>> 238**238
       422003234274091507517421795325920182528086611140712666297183769
       390925685510755057402680778036236427150019987694212157636287196
       316333783750877563193837256416303318957733860108662430281598286
       073858990878489423027387093434036402502753142182439305674327314
       588077348865742839689189553235732976315624152928932760343933360
       660521328084551181052724703073395502160912535704170505456773718
       101922384718032634785464920586864837524059460946069784113790792
       337938047537052436442366076757495221197683115845225278869129420
       5907022278985117566190920525466326339246613410508288691503104L
      

      It is important to note that from the perspective of a user, there is no difference between the plain and long integers as all conversions if any are done under covers by the interpreter.

   3. Booleans: These represent the truth values False and True. The Boolean type is a subtype of plain integers. The False and True Boolean values behave like 0 and 1 values respectively except when converted to a string, then the strings “False” or “True” are returned respectively. For example:

           >>> x = 1
           >>> y = True
           >>> x + y
           2
           >>> a = 1
           >>> b = False
           >>> a + b
           1
           >>> b == 0
           True
           >>> y == 1
           True
           >>> 
           >>> str(True)
           'True'
           >>> str(False)
           'False'
      

2. Float: These represent machine-level only double precision floating point numbers. The underlying machine architecture and specific python implementation determines the accepted range and the handling of overflow; so CPython will be limited by the underlying C language while Jython will be limited by the underlying Java language.
3. Complex Numbers: These represent complex numbers as a pair of machine-level double precision floating point numbers. The same caveats apply as for floating point numbers. Complex numbers can be created using the complex keyword as shown in the following example.

    >>> complex(1,2)
    (1+2j)
    >>> 
   

Complex numbers can also be created by using a number literal prefixed with a j. For instance, the previous complex number example can be created by the expression, 1+2j. The real and imaginary parts of a complex number z can be retrieved through the read-only attributes z.real and z.imag.

Sequence Type

Sequence types are finite ordered collections of objects that can be indexed by integers; using negative indices in python is legal. Sequences fall into two categories - mutable and immutable sequences.

1. Immutable sequences: An immutable sequence type object is one whose value cannot change once it is created. This means that the collection of objects that are directly referenced by an immutable sequence is fixed. The collection of objects referenced by an immutable sequence maybe composed of mutable objects whose value may change at runtime but the mutable object itself that is directly referenced by an immutable sequence cannot be changed. For example, a tuple is an immutable sequence but if one of the elements in the tuple is a list, a mutable sequence, then the list can change but the reference to the list object that tuple holds cannot be changed as shown below:

    >>> t = [1, 2, 3], "obi", "ike"
    >>> type(t)
    <class 'tuple'>
    >>> t[0].append(4) # mutate the list
    >>> t
    ([1, 2, 3, 4], 'obi', 'ike')
    >>> t[0] = [] # attempt to change the reference in tuple
    Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
    TypeError: 'tuple' object does not support item assignment
   

The following are built-in immutable sequence types:

1. Strings: A string is an immutable sequence of Unicode code points or more informally an immutable sequence of characters. There is no char type in python so a character is just a string of length, 1. Strings in python can represent all unicode code points in the range U+0000 - U+10FFFF. All text in python is Unicode and the type of the objects used to hold such text is str.
2. Bytes: A bytes object is an immutable sequence of 8-bit bytes. Each bytes is represented by an integer in the range 0 <= x < 256. Bytes literals such as b'abc' and the built-in function bytes() are used to create bytes objects. Bytes object have an intimate relationship with strings. Strings are abstractions over text representation used in the computer; text is represented internally using binary or bytes. Strings are just sequences of bytes that have been decoded using an encoding such as UTF-8. The abstract characters of a string can also be encoded using available encodings such as UTF-8 to get the binary representation of the string in bytes objects. The relationship between bytes and stringsis illustrated with the following example.

          >>> b = b'abc'
          >>> b
          b'abc'
          >>> type(b)
          <class 'bytes'>
          >>> b = bytes('abc', 'utf-16') # encode a string to bytes using UTF-16 encoding
          >>> b
          b'\xff\xfea\x00b\x00c\x00'
          >>> b
          b'\xff\xfea\x00b\x00c\x00'
          >>> b.decode("utf-8") # decoding fails as encoding has been done with utf-16
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
          >>> b.decode("utf-16") # decoding to string passes
          'abc'
          >>> type(b.decode("utf-16"))
          <class 'str'>
   

3. Tuple: A tuple is a sequence of arbitrary python objects. Tuples of two or more items are formed by comma-separated lists of expressions. A tuple of one item is formed by affixing a comma to an expression while an empty tuple is formed by an empty pair of parentheses. This is illustrated in the following example.

      >>> names = "Obi",  # tuple of 1
      >>> names
      ('Obi',)
      >>> type(names)
      <class 'tuple'>    
   
      >>> names = ()  # tuple of 0
      >>> names
      ()
      >>> type(names)
      <class 'tuple'>  
   
   
      >>> names = "Obi", "Ike", 1  # tuple of 2 or more
      >>> names
      ('Obi', "Ike", 1)
      >>> type(names)
      <class 'tuple'>  
   

4. Mutable sequences: An immutable sequence type is one whose value can change after it has created. There are currently two built-in mutable sequence types - byte arrays and lists
   1. Byte Arrays: Bytearray objects are mutable arrays of bytes. Byte arrays are created using the built-in bytearray() constructor. Apart from being mutable and thus unhashable, byte arrays provide the same interface and functionality as immutable byte objects. Bytearrays are very useful when the efficiency offered by their mutability is required. For example, when receiving an unknown amount of data over a network, byte arrays are more efficient because the array can be extended as more data is received without having to allocate new objects as would be the case if the immutable byte type was used.
   2. Lists: Lists are a sequence of arbitrary Python objects. Lists are formed by placing a comma-separated list of expressions in square brackets. The empty list is formed with the empty square bracket, []. A list can be created from any iterable by passing such iterable to the list method. The list data structure is one of the most widely used data type in python.

Sequence types have some operations that are common to all sequence types. These are described in the following table; x is an object, s and t are sequences and n, i, j, k are integers.

Note

1. Values of n that are less than 0 are treated as 0 and this yields an empty sequence of the same type as s such as below:

    >>> x = "obi"
    >>> x*-2
    ''
   

2. Copies made from using the * operation are shallow copies; any nested structures are not copied. This can result in some confusion when trying to create copies of a structure such as a nested list.

   >>> lists = [[]] * 3 # shallow copy
   >>> lists
   [[], [], []] # all three copies reference the same list
   >>> lists[0].append(3)
   >>> lists
   [[3], [3], [3]]
   

   To avoid shallow copies when dealing with nested lists, the following method can be adopted

   ```python
   >>> lists = [[] for i in range(3)]
   >>> lists[0].append(3)
   >>> lists[1].append(5)
   >>> lists[2].append(7)
   >>> lists
   [[3], [5], [7]]
   ```
   

3. When i or j is negative, the index is relative to the end of the string thus len(s) + i or len(s) + j is substituted for the negative value of i or j.
4. Concatenating immutable sequences such as strings always results in a new object for example:

    >>> name = "Obi"
    >>> id(name)
    4330660336
    >>> name += "Obi" + " Ike-Nwosu"
    >>> id(name)
    4330641208
   

Python defines the interfaces (thats the closest word that can be used) - Sequences and MutableSequences in the collections library and these define all the methods a type must implement to be considered a mutable or immutable sequence; when abstract base classes are discussed, this concept will become much clearer.

Set

These are unordered, finite collection of unique python objects. Sets are unordered so they cannot be indexed by integers. The members of a set must be hash-able so only immutable objects can be members of a set. This is so because sets in python are implemented using a hash table; a hash table uses some kind of hash function to compute an index into a slot. If a mutable value is used then the index calculated will change when this object changes thus mutable values are not allowed in sets. Sets provide efficient solutions for membership testing, de-duplication, computing of intersections, union and differences. Sets can be iterated over, and the built-in function len() returns the number of items in a set. There are currently two intrinsic set types:- the mutable set type and the immutable frozenset type. Both have a number of common methods that are shown in the following table.

1. Frozen set: This represents an immutable set. A frozen set is created by the built-in frozenset() constructor. A frozenset is immutable and thus hashable so it can be used as an element of another set, or as a dictionary key.
2. Set: This represents a mutable set and it is created using the built-in set() constructor. The mutable set is not hashable and cannot be part of another set. A set can also be created using the set literal {}. Methods unique to the mutable set include:

Mapping

A python mapping is a finite set of objects (values) indexed by a set of immutable python objects (keys). The keys in the mapping must be hashable for the same reason given previously in describing set members thus eliminating mutable types like lists, frozensets, mappings etc. The expression, a[k], selects the item indexed by the key, k, from the mapping a and can be used as in assignments or del statements. The dictionary mostly called dict for convenience is the only intrinsic mapping type built into python:

1. Dictionary: Dictionaries can be created by placing a comma-separated sequence of key: value pairs within braces, for example: {'name': "obi", 'age': 18}, or by the dict() constructor. The main operations supported by the dictionary type is the addition, deletion and selection of values using a given key. When adding a key that is already in use within a dict, the old value associated with that key is forgotten. Attempting to access a value with a non-existent key will result in a KeyError exception. Dictionaries are perhaps one of the most important types within the interpreter. Without explicitly making use of a dictionary, the interpreter is already using them in a number of different places. For example, the namespaces, namespaces are discussed in a subsequent chapter, in python are implemented using dictionaries; this means that every time a symbol is referenced within a program, a dictionary access occurs. Objects are layered on dictionaries in python; all attributes of python objects are stored in a dictionary attribute, __dict__. These are but a few applications of this type within the python interpreter.

Python supplies more advanced forms of the dictionary type in its collections library. These are the OrderedDict that introduces order into a dictionary thus remembering the order in which items were insert and the defaultdict that takes a factory function that is called to produce a value when a key is missing. If a key is missing from a defaultdict instance, the factory function is called to produce a value for the key and the dictionary is updated with this key, value pair and the created value is returned. For example,

```python
    >>> from collections import defaultdict
    >>> d = defaultdict(int)
    >>> d
    defaultdict(<class 'int'>, {})
    >>> d[7]
    0
    >>>d
    defaultdict(<class 'int'>, {7: 0})
```

Callable Types

These are types that support the function call operation. The function call operation is the use of () after the type name. In the example below, the function is print_name and the function call is when the () is appended to the function name as such print_name().

    def print_name(name):
        print(name)

Functions are not the only callable types in python; any object type that implements the __call__ special method is a callable type. The function, callable(type), is used to check that a given type is callable. The following are built-in callable python types:

1. User-defined functions: these are functions that a user defines with the def statement such as the print_name function from the previous section.
2. Methods: these are functions defined within a class and accessible within the scope of the class or a class instance. These methods could either be instance methods, static or class methods.
3. Built-in functions: These are functions available within the interpreter core such as the len function.
4. Classes: Classes are also callable types. The process of creating a class instance involves calling the class such as Foo().

Each of the above types is covered in detail in subsequent chapters.

Custom Type

Custom types are created using the class statements. Custom class objects have a type of type. These are types created by user defined programs and they are discussed in the chapter on object oriented programming.

Module Type

A module is one of the organizational units of Python code just like functions or classes. A module is also an object just like every other value in the python. The module type is created by the import system as invoked either by the import statement, or by calling functions such as importlib.import_module() and built-in __import__().

File/IO Types

A file object represents an open file. Files are created using the open built-in functions that opens and returns a file object on the local file system; the file object can be open in either binary or text mode. Other methods for creating file objects include:

1. os.fdopen that takes a file descriptor and create a file object from it. The os.open method not to be confused with the open built-in function is used to create a file descriptor that can then be passed to the os.fdopen method to create a file object as shown in the following example.

       >>> import os
       >> fd = os.open("test.txt", os.O_RDWR|os.O_CREAT)
       >>> type(fd)
       <class 'int'>
       >>> fd
       3
       >>> fo = os.fdopen(fd, "w")
       >>> fo
       <_io.TextIOWrapper name=3 mode='w' encoding='UTF-8'>
       >>> type(fo)
       <class '_io.TextIOWrapper'>
   

2. os.popen(): this is marked for deprecation.
3. makefile() method of a socket object that opens and returns a file object that is associated with the socket on which it was called.

The built-in objects, sys.stdin, sys.stdout and sys.stderr, are also file objects corresponding to the python interpreter’s standard input, output and error streams.

Built-in Types

These are objects used internally by the python interpreter but accessible by a user program. They include traceback objects, code objects, frame objects and slice objects

Code Objects

Code objects represent compiled executable Python code, or bytecode. Code objects are machine code for the python virtual machine along with all that is necessary for the execution of the bytecode they represent. They are normally created when a block of code is compiled. This executable piece of code can only be executed using the exec or eval python methods. To give a concrete understanding of code objects we define a very simple function below and dissect the code object.

def return_author_name():
    return "obi Ike-Nwosu"

The code object for the above function can be obtained from the function object by assessing its __code__ attribute as shown below:

    >>> return_author_name.__code__
    <code object return_author_name at 0x102279270, file "<stdin>", line 1>

We can go further and inspect the code object using the dir function to see the attributes of the code object.

     >>> dir(return_author_name.__code__)
    ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattri\
bute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '_\
_reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount'\
, 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars',\
 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames\
']

Of particular interest to us at this point in time are the non-special methods that is methods that do not start with an underscore. We give a brief description of each of these non-special methods in the following table

We can view the bytecode string for the function using the co_code method of the code object as shown below.

    >>> return_author_name.__code__.co_code
    b'd\x01\x00S'

The bytecode returned however is basically of no use to someone investigating code objects. This is where the python dis module comes into play. The dis module can be used to generate a human readable version of the code object. We use the dis function from the dis module to generate the code object for return_author_name function.

    >>> dis.dis(return_author_name)
      2           0 LOAD_CONST               1 ('obi Ike-Nwosu')
                  3 RETURN_VALUE

The above shows the human readable version of the the python code object. The LOAD_CONST instruction reads a value from the co_consts tuple, and pushes it onto the top of the stack (the CPython interpreter is a stack based virtual machine). The RETURN_VALUE instruction pops the top of the stack, and returns this to the calling scope signalling the end of the execution of that python code block.

Code objects serve a number of purposes while programming. They contain information that can aid in interactive debugging while programming and can provide us with readable tracebacks during an exception.

Frame Objects

Frame objects represent execution frames. Python code blocks are executed in execution frames. The call stack of the interpreter stores information about currently executing subroutines and the call stack is made up of stack frame objects. Frame objects on the stack have a one-to-one mapping with subroutine calls by the program executing or the interpreter. The frame object contains code objects and all necessary information, including references to the local and global name-spaces, necessary for the runtime execution environment. The frame objects are linked together to form the call stack. To simplify how this all fits together a bit, the call stack can be thought of as a stack data structure (it actually is), every time a subroutine is called, a frame object is created and inserted into the stack and then the code object contained within the frame is executed. Some special read-only attributes of frame objects include:

1. f_back is to the previous stack frame towards the caller, or None if this is the bottom stack frame.
2. f_code is the code object being executed in this frame.
3. f_locals is the dictionary used to look up local variables.
4. f_globals is used for global variables.
5. f_builtins is used for built-in names.
6. f_lasti gives the precise instruction - it is an index into the bytecode string of the code object.

Some special writable attributes include:

1. f_trace: If this is not None, this is a function called at the start of each source code line.
2. f_lineno: This is the current line number of the frame. Writing to this from within a trace function jumps to the given line only for the bottom-most frame. A debugger can implement a Jump command by writing to f_lineno.

Frame objects support one method:

1. frame.clear(): This method clears all references to local variables held by the frame. If the frame belonged to a generator, the generator is finalized. This helps break reference cycles involving frame objects. A RuntimeError is raised if the frame is currently executing.

Traceback Objects

Traceback objects represent the stack trace of an exception. A traceback object is created when an exception occurs. The interpreter searches for an exception handler by continuously popping the execution stack and inserting a traceback object in front of the current traceback for each frame popped. When an exception handler is encountered, the stack trace is made available to the program. The stack trace object is accessible as the third item of the tuple returned by sys.exc_info(). When the program contains no suitable handler, the stack trace is written to the standard error stream; if the interpreter is interactive, it is also made available to the user as sys.last_traceback. A few important attributes of a traceback object is shown in the following table.

Slice Objects

Slice objects represent slices for __getitem__() methods of sequence-like objects (more on special methods such as __getitem__() in the chapter on object oriented programming). Slice object return a subset of the sequence they are applied to as shown below.

    >>> t = [i for i in range(10)]
    >>> t
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> t[:10:2]
    [0, 2, 4, 6, 8]
    >>> 

They are also created by the built-in slice([start,], stop [,step]) function. The returned object can be used in between the square brackets as a regular slice object.

    >>> t = [i for i in range(10)]
    >>> t
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> t[:10:2]
    [0, 2, 4, 6, 8] 
    >>> s = slice(None, 10, 2)
    >>> s
    slice(None, 10, 2)
    >>> t[s]
    [0, 2, 4, 6, 8]

Slice object read-only attributes include:

Each of the optional attributes is None if omitted. Slices can take a number of forms in addition to the standard slice(start, stop [,step]). Other forms include

    a[start:end] # items start to end-1 equivalent to slice(start, stop)
    a[start:]    # items start to end-1 equivalent to slice(start)
    a[:end]      # items from the beginning to end-1 equivalent to slice(None, end)
    a[:]         # a shallow copy of the whole array equivalent to slice(None, None)

The start or end values may also be negative in which case we count from the end of the array as shown below:

    a[-1]    # last item in the array equivalent to slice(-1)
    a[-2:]   # last two items in the array equivalent to slice(-2)
    a[:-2]   # everything except the last two items equivalent to slice(None, -2)

Slice objects support one method:

1. slice.indices(self, length): This method takes a single integer argument, length, and returns a tuple of three integers - (start, stop, stride) that indicates how the slice would apply to the given length. The start and stop indices are actual indices they would be in a sequence of length given by the length argument. An example is shown below:

```python
>>> s = slice(10, 30, 1) 
# applying slice(10, 30, 1) to sequence of length 100 gives [10:30]
>>> s.indices(100)  
(10, 30, 1)
# applying slice(10, 30, 1) to sequence of length 15 gives [10:15]
>>> s.indices(15)
(10, 15, 1)
# applying slice(10, 30, 1) to sequence of length 1 gives [1:1]
>>> s.indices(1)
(1, 1, 1)
>>> s.indices(0)
(0, 0, 1)
```

Generator Objects

Generator objects are created by the invocation of generator functions; these are functions that use the keyword, yield. This type is discussed in detail in the chapter on Sequences and Generators.

With a strong understanding of the built-in type hierarchy, the stage is now set for examining object oriented programming and how users can create their own type hierarchy and even make such types behave like built-in types.

6. Object Oriented Programming

Classes are the basis of object oriented programming in python and are one of the basic organizational units in a python program.

6.1 The Mechanics of Class Definitions

The class statement is used to define a new type. The class statement defines a set of attributes, variables and methods, that are associated with and shared by a collection of instances of such a class. A simple class definition is given below:

    class Account(object):
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            Account.num_accounts += 1

        def del_account(self):
            Account.num_accounts -= 1

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return self.balance 

Class definitions introduce class objects, instance objects and method objects.

Class Objects

The execution of a class statement creates a class object. At the start of the execution of a class statement, a new name-space is created and this serves as the name-space into which all class attributes go; unlike languages like Java, this name-space does not create a new local scope that can be used by class methods hence the need for fully qualified names when accessing attributes. The Account class from the previous section illustrates this; a method trying to access the num_accounts variable must use the fully qualified name, Account.num_accounts else an error results such as when the fully qualified name is not used in the __init__ method as shown below:

    class Account(object):
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            num_accounts += 1

        def del_account(self):
            Account.num_accounts -= 1

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return self.balance 

    >>> acct = Account('obi', 10)
    Traceback (most recent call last):
      File "python", line 1, in <module>
      File "python", line 9, in __init__
    UnboundLocalError: local variable 'num_accounts' referenced before assignment

At the end of the execution of a class statement, a class object is created; the scope preceding the class definition is reinstated, and the class object is bound in this scope to the class name given in the class definition header.

A little diversion here. One may ask, if the class created is an object then what is the class of the class object?. In accordance with the Python philosophy that every value is an object , the class object does indeed have a class which it is created from; this is the type class.

    >>> type(Account)
    <class 'type'>

So just to confuse you a bit, the type of a type, the Account type, is type. To get a better understanding of the fact that a class is indeed an object with its own class we go behind the scenes to explain what really goes on during the execution of a class statement using the Account example from above.

    >>>class_name = "Account"
    >>>class_parents = (object,)
    >>>class_body = """
    num_accounts = 0

    def __init__(self, name, balance):
        self.name = name 
        self.balance = balance 
        num_accounts += 1

    def del_account(self):
        Account.num_accounts -= 1

    def deposit(self, amt):
        self.balance = self.balance + amt 

    def withdraw(self, amt):
        self.balance = self.balance - amt 

    def inquiry(self):
        return self.balance 
    """
    # a new dict is used as local name-space
    >>>class_dict = {}

    #the body of the class is executed using dict from above as local 
    # name-space 
    >>>exec(class_body, globals(), class_dict)

    # viewing the class dict reveals the name bindings from class body
    >>> class_dict
    {'del_account': <function del_account at 0x106be60c8>, 'num_accounts': 0, 'inquiry': <function i\
nquiry at 0x106beac80>, 'deposit': <function deposit at 0x106be66e0>, 'withdraw': <function withdraw\
 at 0x106be6de8>, '__init__': <function __init__ at 0x106be2c08>}

    # final step of class creation
    >>>Foo = type(class_name, class_parents, class_dict)
    # view created class object
    >>>Account
    <class '__main__.Account'>
    >>>type(Account)
    <type 'type'>

During the execution of class statement, the interpreter kind of carries out the following steps behind the scene (greatly simplified here):

1. The body of the class statement is isolated into a code object.
2. A class dictionary representing the name-space for the class is created.
3. The code object representing the body of the class is executed within this name-space.
4. During the final step, the class object is created by instantiating the type class, passing in the class name, base classes and class dictionary as arguments. The type class used here in creating the Account class object is a meta-class, the class of a class. The meta-class used in the class object creation can be explicitly specified by supplying the metaclass keyword argument in the class definition. In the case that this is not supplied, the base classes if any are checked for a meta-class. If no base classes are supplied, then the default type() metaclass is used. More about meta-classes is discussed in subsequent chapters.

Class objects support attribute reference and object instantiation. Attributes are referenced using the standard dot syntax; an object followed by dot and then attribute name: obj.name. Valid attribute names are all the variable and method names present in the class’ name-space when the class object was created. For example:

    >>> Account.num_accounts
    0
    >>> Account.deposit
    >>> <unbound method Account.deposit>

Object instantiation is carried out by calling the class object like a normal function with required parameters for the __init__ method of the class as shown in the following example:

    >>> Account("obi", 0)

An instance object that has been initialized with supplied arguments is returned from instantiation of a class object. In the case of the Account class, the account name and account balance are set and, the number of instances is incremented by 1 in the __init__ method.

Instance Objects

If class objects are the cookie cutters then instance objects are the cookies that are the result of instantiating class objects. Instance objects are returned after the correct initialization of a class just as shown in the previous section. Attribute references are the only operations that are valid on instance objects. Instance attributes are either data attribute, better known as instance variables in languages like Java, or method attributes.

Method Objects

If x is an instance of the Account class, x.deposit is an example of a method object. Method objects are similar to functions however during a method definition, an extra argument is included in the arguments list, the self argument. This self argument refers to an instance of the class but why do we have to pass an instance as an argument to a method? This is best illustrated by a method call such as the following.

    >>> x = Account()
    >>> x.inquiry()
    10

But what exactly happens when an instance method is called? It can be observed that x.inquiry() is called without an argument above, even though the method definition for inquiry() requires the self argument. What happened to this argument?

In the example from above, the call to x.inquiry() is exactly equivalent to Account.inquiry(x); notice that the object instance, x, is being passed as argument to the method - this is the self argument. Invoking an object method with an argument list is equivalent to invoking the corresponding method from the object’s class with an argument list that is created by inserting the method’s object at the start of the list of argument. In order to understand this, note that methods are stored as functions in class dicts.

    >>> type(Account.inquiry)
    <class 'function'>

To fully understand how this transformation takes place one has to understand descriptors and Python’s attribute references algorithm. These are discussed in subsequent sections of this chapter. In summary, the method object is a wrapper around a function object; when the method object is called with an argument list, a new argument list is constructed from the instance object and the argument list, and the underlying function object is called with this new argument list. This applies to all instance method objects including the __init__ method. Note that the self argument is actually not a keyword; the name, self is just a convention and any valid argument name can be used as shown in the Account class definition below.

    class Account(object):
        num_accounts = 0

        def __init__(obj, name, balance):
            obj.name = name 
            obj.balance = balance 
            Account.num_accounts += 1

        def del_account(obj):
            Account.num_accounts -= 1

        def deposit(obj, amt):
            obj.balance = obj.balance + amt 

        def withdraw(obj, amt):
            obj.balance = obj.balance - amt 

        def inquiry(obj):
            return obj.balance 

    >>> Account.num_accounts
    0
    >>> x = Account('obi', 0)
    >>> x.deposit(10)
    >>> Account.inquiry(x)
    10

6.2 Customizing User-defined Types

Python is a very flexible language providing user with the ability to customize classes in ways that are unimaginable in other languages. Attribute access, class creation and object initialization are a few examples of ways in which classes can be customized. User defined types can also be customized to behave like built-in types and support special operators and syntax such as *, +, -, [] etc.

All these customization is possible because of methods that are called special or magic methods. Python special methods are just ordinary python methods with double underscores as prefix and suffix to the method names. Special methods have already encountered in this book. An example is the __init__ method that is called to initialize class instances; another is the __getitem__ method invoked by the index, [] operator; an index such as a[i] is translated by the interpreter to a call to type(a).__getitem__(a, i). Methods with the double underscore as prefix and suffix are just ordinary python methods; users can define their own class methods with method names prefixed and suffixed with the double underscore and use it just like normal python methods. This is however not the conventional approach to defining normal user methods.

User defined classes can also implement these special methods; a corollary of this is that built-in operators such as + or [] can be adapted for use by user defined classes. This is one of the essence of polymorphism in Python. In this book, special methods are grouped according to the functions they serve. These groups include:

Special methods for instance creation

The __new__ and __init__ special methods are the two methods that are integral to instance creation. New class instances are created in a two step process; first the static method, __new__, is called to create and return a new class instance then the __init__ method is called to to initialize the newly created object with supplied arguments. A very important instance in which there is a need to override the __new__ method is when sub-classing built-in immutable types. Any initialization that is done in the sub-class must be done before object creation. This is because once an immutable object is created, its value cannot be changed so it makes no sense trying to carry out any function that modifies the created object in an __init__ method. An example of sub-classing is shown in the following snippet in which whatever value is supplied is rounded up to the next integer.

    >>> import math
    >>> class NextInteger(int):
    ...     def __new__(cls, val):
    ...         return int.__new__(cls, math.ceil(val))
    ... 
    >>> NextInteger(2.2)
    3
    >>> 

Attempting to do the math.ceil operation in an __init__ method will cause the object initialization to fail. The __new__ method can also be overridden to create a Singleton super class; subclasses of this class can only ever have a single instance throughout the execution of a program; the following example illustrates this.

    class Singleton:

        def __new__(cls, *args, **kwds):
            it = cls.__dict__.get("__it__")
            if it is None:
                return it
            cls.__it__ = it = object.__new__(cls)
            it.init(*args, **kwds)
            return it

        def __init__(self, *args, **kwsds):
            pass

It is worth noting that when implementing the __new__ method, the implementation must call its base class’ __new__ and the implementation method must return an object.

Users are already familiar with defining the __init__ method; the __init__ method is overridden to perform attribute initialization for an instance of a mutable types.

Special methods for attribute access

The special methods in this category provide means for customizing attribute references; this maybe in order to access or set such an attribute. This set of special methods available for this include:

1. __getattr__: This method can be implemented to handle situations in which a referenced attribute cannot be found. This method is only called when an attribute that is referenced is neither an instance attribute nor is it found in the class tree of that object. This method should return some value for the attribute or raise an AttributeError exception. For example, if x is an instance of the Account class defined above, trying to access an attribute that does not exist will result in a call to this method as shown in the following snippet

    class Account(object):
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            Account.num_accounts += 1

        def del_account(self):
            Account.num_accounts -= 1

        def __getattr__(self, name):
            return "Hey I don't see any attribute called {}".format(name)

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return "Name={}, balance={}".format(self.name, self.balance) 

    >>> x = Account('obi', 0)
    >>> x.balaance
    Hey I dont see any attribute called balaance 

Care should be taken with the implementation of __getattr__ because if the implementation references an instance attribute that does not exist, an infinite loop may occur because the __getattr__ method is called successively without end.

    class Account(object):
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            Account.num_accounts += 1

        def del_account(self):
            Account.num_accounts -= 1

        def __getattr__(self, name):
            return self.namee # trying to acess a variable that doesnt exist will result in __getatt\
r___ calling itself over and over again

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return "Name={}, balance={}".format(self.name, self.balance) 

    >>> x = Account('obi', 0)
    >>> x.balaance # this will result in a RuntimeError: maximum recursion depth exceeded while call\
ing a Python object exception 

1. __getattribute__: This method is implemented to customize the attribute access for a class. This method is always called unconditionally during attribute access for instances of a class.
2. __setattr__: This method is implemented to unconditionally handle all attribute assignment. __setattr__ should insert the value being assigned into the dictionary of the instance attributes rather than using self.name=value which results in an infinite recursive call. When __setattr__() is used for instance attribute assignment, the base class method with the same name should be called such as super().__setattr__(self, name, value).
3. __delattr__: This is implemented to customize the process of deleting an instance of a class. it is invoked whenever del obj is called.
4. __dir__: This is implemented to customize the list of object attributes returned by a call to dir(obj).

Special methods for Type Emulation

Built-in types in python have special operators that work with them. For example, numeric types support the + operator for adding two numbers, numeric types also support the - operator for subtracting two numbers, sequence and mapping types support the [] operator for indexing values held. Sequence types even also have support for the + operator for concatenating such sequences. User defined classes can be customized to behave like these built-in types where it makes sense. This can be done by implementing the special methods that are invoked by the interpreter when these special operators are encountered. The special methods that provide these functionalities for emulating built-in types can be broadly grouped into one of the following:

Numeric Type Special Methods

The following table shows some of the basic operators and the special methods invoked when these operators are encountered.

Python has the concept of reflected operations; this was covered in the section on the NotImplemented of previous chapter. The idea behind this concept is that if the left operand of a binary arithmetic operation does not support a required operation and returns NotImplemented then an attempt is made to call the corresponding reflected operation on the right operand provided the type of both operands differ. An example of this rarely used functionality is shown in the following trivial example for emphasis.

        class MyNumber(object):
            def __init__(self, x):
                self.x = x

            def __str__(self):
                return str(self.x)

        >>> 10 - MyNumber(9) # int type, 10, does not know how to subtract MyNumber type and MyNumbe\
r does not know how to handle the operation too
        Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        TypeError: unsupported operand type(s) for -: 'int' and 'MyNumber'

In the next snippet the class implements the reflected special method and this reflected method is called by the interpreter.

        class MyFixedNumber(MyNumber):
            def __rsub__(self, other): # reflected operation implemented
                return MyNumber(other - self.val)

        >>> (10 - MyFixedNumber(9)).val
        1

The following special methods implement reflected binary arithmetic operations.

Another set of operators that work with numeric types are the augmented assignment operators. An example of an augmented operation is shown in the following code snippet.

    >>> val = 10
    >>> val += 90
    >>> val
    100
    >>> 

A few of the special methods for implementing augmented arithmetic operations are listed in the following table.

Sequence and Mapping Types Special Methods

Sequence and mapping are often referred to as container types because they can hold references to other objects. User-defined classes can emulate container types to the extent that this makes sense if such classes implement the special methods listed in the following table.

Sequence types such as lists support the addition (for concatenating lists) and multiplication operators (for creating copies), + and * respectively, by defining the methods __add__(), __radd__(), __iadd__(), __mul__(), __rmul__() and __imul__(). Sequence types also implement the __reversed__ method that implements the reversed() method that is used for reverse iteration over a sequence. User defined classes can implement these special methods to get the required functionality.

Emulating Callable Types

Callable types support the function call syntax, (args). Classes that implement the __call__(self[, args...]) method are callable. User defined classes for which this functionality makes sense can implement this method to make class instances callable. The following example shows a class implementing the __call__(self[, args...]) method and how instances of this class can be called using the function call syntax.

    class Account(object):
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            Account.num_accounts += 1
 
        def __call__(self, arg):
            return "I was called with \'{}\'".format(arg)

        def del_account(self):
            Account.num_accounts -= 1

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return self.balance 

    >>> acct = Account()
    >>> acct("Testing function call on instance object")
    I was called with 'Testing function call on instance object'    

Special Methods for comparing objects

User-defined classes can provide custom implementation for the special methods invoked by the five object comparison operators in python, <, >, >=, <=, = in order to control how these operators work. These special methods are given in the following table.

In Python, x==y is True does not imply that x!=y is False so __eq__() should be defined along with __ne__() so that the operators are well behaved. __lt__() and __gt__(), and __le__() and __ge__() are each other’s reflection while __eq__() and __ne__() are their own reflection; this means that if a call to the implementation of any of these methods on the left argument returns NotImplemented, the reflected operator is is used.

Special Methods and Attributes for Miscellaneous Customizations

1. __slots__: This is a special attribute rather than a method. It is an optimization trick that is used by the interpreter to efficiently store object attributes. Objects by default store all attributes in a dictionary (the __dict__ attribute) and this is very inefficient when objects with few attributes are created in large numbers. __slots__ make use of a static iterable that reserves just enough space for each attribute rather than the dynamic __dict__ attribute. The iterable representing the __slot__ variable can also be a string made up of the attribute names. The following example shows how __slots__ works.

    class Account:
        """base class for representing user accounts"""

        # we can also use __slots__ = "name balance"
        __slots__ = ['name', 'balance']
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            Account.num_accounts += 1

        def del_account(self):
            Account.num_accounts -= 1

        def __getattr__(self, name):
            """handle attribute reference for non-existent attribute"""
            return "Hey I dont see any attribute called {}".format(name)

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return "Name={}, balance={}".format(self.name, self.balance) 

    >>>acct = Account("obi", 10)
    >>>acct.__dict__ # __dict__ attribute is gone
    Hey I dont see any attribute called __dict__
    >>>acct.x = 10
    Traceback (most recent call last):
    File "acct.py", line 32, in <module>
      acct.x = 10
    AttributeError: 'Account' object has no attribute 'x'
    >>>acct.__slots__
    ['name', 'balance']

A few things that are worth noting about __slots__ include the following:

1. If a superclass has the __dict__ attribute then using __slots__ in sub-classes is of no use as the dictionary is available.
2. If __slots__ are used then attempting to assign to a variable not in the __slots__ variable will result in an AttributeError as shown in the previous example.
3. Sub-classes will have a __dict__ even if they inherit from a base class with a __slots__ declaration; subclasses have to define their own __slots__ attribute which must contain only the additional names in order to avoid having the __dict__ for storing names.
4. Subclasses with “variable-length” built-in types as base class cannot have a non-empty __slots__ variable.
5. __bool__: This method implements the truth value testing for a given class; it is invoked by the built-in operation bool() and should return a True or False value. In the absence of an implementation, __len__() is called and if __len__ is implemented, the object’s truth value is considered to be True if result of the call to __len__ is non-zero. If neither __len__() nor __bool__() are defined by a class then all its instances are considered to be True.
6. __repr__ and __str__: These are two closely related methods as they both return string representations for a given object and only differ subtly in the intent behind their creation. Both are invoked by a call to repr and str methods respectively. The __repr__ method implementation should return an unambiguous string representation of the object it is being called on. Ideally, the representation that is returned should be an expression that when evaluated by the eval method returns the given object; when this is not possible the representation returned should be as unambiguous as possible. On the other hand, __str__ exists to provide a human readable version of an object; a version that would make sense to some one reading the output but that doesn’t necessarily understand the semantics of the language. A very good illustration of how both methods differ is shown below by calling both methods on a data object.

        >>> import datetime
        >>> today = datetime.datetime.now()
        >>> str(today)
        '2015-07-05 20:55:58.642018' # human readable version of datetime object
        >>> repr(today)
        'datetime.datetime(2015, 7, 5, 20, 55, 58, 642018)' # eval will return the datetime object
   

   When using string interpolation, %r makes a call to repr while %s makes a call to str.

7. __bytes__: This is invoked by a call to the bytes() built-in and it should return a byte string representation for an object. The byte string should be a bytes object.
8. __hash__: This is invoked by the hash() built-in. It is also used by operations that work on types such as set, frozenset, and dict that make use of object hash values. Providing __hash__ implementation for user defined classes is an involved and delicate act that should be carried out with care as will be seen. Immutable built-in types are hashable while mutable types such as lists are not. For example, the hash of a number is the value of the number as shown in the following snippet.

    >>> hash(1)
    1
    >>> hash(12345)
    12345
    >>> 

User defined classes have a default hash value that is derived from their id() value. Any __hash__() implementation must return an integer and objects that are equal by comparison must have the same hash value so for two object, a and b, (a==b and hash(a)==hash(b)) must be true. A few rules for implementing a __hash__() method include the following: 1. A class should only define the __hash__() method if it also defines the __eq__() method.

1. The absence of an implementation for the __hash__() method in a class renders its instances unhashable.
2. The interpreter provides user-defined classes with default implementations for __eq__() and __hash__(). By default, all objects compare unequal except with themselves and x.__hash__() returns a value such that (x == y and x is y and hash(x) == hash(y)) is always true. In CPython, the default __hash__() implementation returns a value derived from the id() of the object.
3. Overriding the __eq__() method without defining the __hash__() method sets the __hash__() method to None in the class. When the __hash__() method of a class is None, an instance of the class will raise an appropriate TypeError when an attempt is made to retrieve its hash value. The object will also be correctly identified as unhashable when checking isinstance(obj, collections.Hashable).
4. If a class overrides the __eq__() and needs to keep the implementation of __hash__() from a base class, this must be done explicitly by setting __hash__ = BaseClass.__hash__.
5. A class that does not override the __eq__() can suppress hash support by setting __hash__ to None. If a class defines its own __hash__() method that explicitly raises a TypeError, instances of such class will be incorrectly identified as hashable by an isinstance(obj, collections.Hashable) test.

6.3 A Vector class

In this section, a complete example of the use of special methods to emulate built-in types is provided by a Vector class. The Vector class provides support for performing vector arithmetic operations.

        # Copyright 2013 Philip N. Klein
        class Vec:
            """
            A vector has two fields:
            D - the domain (a set)
            f - a dictionary mapping (some) domain elements to field elements
                elements of D not appearing in f are implicitly mapped to zero
            """
            def __init__(self, labels, function):
                assert isinstance(labels, set)
                assert isinstance(function, dict)
                self.D = labels
                self.f = function


            def __getitem__(self, key):
                """
                Return the value of entry k in v.
                Be sure getitem(v,k) returns 0 if k is not represented in v.f.

                >>> v = Vec({'a','b','c', 'd'},{'a':2,'c':1,'d':3})
                >>> v['d']
                3
                >>> v['b']
                0
                """
                assert key in self.D
                if key in self.f: 
                    return self.f[key]
                return 0

            def __setitem__(self, key, val):
                """
                Set the element of v with label d to be val.
                setitem(v,d,val) should set the value for key d even if d
                is not previously represented in v.f, and even if val is 0.

                >>> v = Vec({'a', 'b', 'c'}, {'b':0})
                >>> v['b'] = 5
                >>> v['b']
                5
                >>> v['a'] = 1
                >>> v['a']
                1
                >>> v['a'] = 0
                >>> v['a']
                0
                """
                assert key in self.D
                self.f[key] = val

            def __neg__(self):
                """
                Returns the negation of a vector.

                >>> u = Vec({1,3,5,7},{1:1,3:2,5:3,7:4})
                >>> -u
                Vec({1, 3, 5, 7},{1: -1, 3: -2, 5: -3, 7: -4})
                >>> u == Vec({1,3,5,7},{1:1,3:2,5:3,7:4})
                True
                >>> -Vec({'a','b','c'}, {'a':1}) == Vec({'a','b','c'}, {'a':-1})
                True
                """
                return Vec(self.D, {key:-self[key] for key in self.D})

            def __rmul__(self, alpha):
                """
                Returns the scalar-vector product alpha times v.

                >>> zero = Vec({'x','y','z','w'}, {})
                >>> u = Vec({'x','y','z','w'},{'x':1,'y':2,'z':3,'w':4})
                >>> 0*u == zero
                True
                >>> 1*u == u
                True
                >>> 0.5*u == Vec({'x','y','z','w'},{'x':0.5,'y':1,'z':1.5,'w':2})
                True
                >>> u == Vec({'x','y','z','w'},{'x':1,'y':2,'z':3,'w':4})
                True
                """
                return Vec(self.D, {key : alpha*self[key] for key in self.D })


            def __mul__(self,other):
                #If other is a vector, returns the dot product of self and other
                if isinstance(other, Vec):
                    return dot(self,other)
                else:
                    return NotImplemented  #  Will cause other.__rmul__(self) to be invoked

            def __truediv__(self,other):  # Scalar division
                return (1/other)*self

            def __add__(self, other):
                """
                Returns the sum of the two vectors.
                
                Make sure to add together values for all keys from u.f and v.f even if some keys in \
u.f do not
                exist in v.f (or vice versa)

                >>> a = Vec({'a','e','i','o','u'}, {'a':0,'e':1,'i':2})
                >>> b = Vec({'a','e','i','o','u'}, {'o':4,'u':7})
                >>> c = Vec({'a','e','i','o','u'}, {'a':0,'e':1,'i':2,'o':4,'u':7})
                >>> a + b == c
                True
                >>> a == Vec({'a','e','i','o','u'}, {'a':0,'e':1,'i':2})
                True
                >>> b == Vec({'a','e','i','o','u'}, {'o':4,'u':7})
                True
                >>> d = Vec({'x','y','z'}, {'x':2,'y':1})
                >>> e = Vec({'x','y','z'}, {'z':4,'y':-1})
                >>> f = Vec({'x','y','z'}, {'x':2,'y':0,'z':4})
                >>> d + e == f
                True
                >>> d == Vec({'x','y','z'}, {'x':2,'y':1})
                True
                >>> e == Vec({'x','y','z'}, {'z':4,'y':-1})
                True
                >>> b + Vec({'a','e','i','o','u'}, {}) == b
                True
                """
                assert self.D == other.D
                return Vec(self.D, {key: self[key] + other[key] for key in self.D})

            def __radd__(self, other):
                "Hack to allow sum(...) to work with vectors"
                if other == 0:
                    return self

            def __sub__(a,b):
                "Returns a vector which is the difference of a and b."
                return a+(-b)


            def __eq__(self, other):
                """
                Return true iff u is equal to v.

                Consider using brackets notation u[...] and v[...] in your procedure
                to access entries of the input vectors.  This avoids some sparsity bugs.

                >>> Vec({'a', 'b', 'c'}, {'a':0}) == Vec({'a', 'b', 'c'}, {'b':0})
                True
                >>> Vec({'a', 'b', 'c'}, {'a': 0}) == Vec({'a', 'b', 'c'}, {})
                True
                >>> Vec({'a', 'b', 'c'}, {}) == Vec({'a', 'b', 'c'}, {'a': 0})
                True

                Be sure that equal(u, v) checks equalities for all keys from u.f and v.f even if
                some keys in u.f do not exist in v.f (or vice versa)

                >>> Vec({'x','y','z'},{'y':1,'x':2}) == Vec({'x','y','z'},{'y':1,'z':0})
                False
                >>> Vec({'a','b','c'}, {'a':0,'c':1}) == Vec({'a','b','c'}, {'a':0,'c':1,'b':4})
                False
                >>> Vec({'a','b','c'}, {'a':0,'c':1,'b':4}) == Vec({'a','b','c'}, {'a':0,'c':1})
                False

                The keys matter:
                >>> Vec({'a','b'},{'a':1}) == Vec({'a','b'},{'b':1})
                False

                The values matter:
                >>> Vec({'a','b'},{'a':1}) == Vec({'a','b'},{'a':2})
                False
                """
                assert self.D == other.D
                return all([self[key] == other[key] for key in self.D])

            def is_almost_zero(self):
                s = 0
                for x in self.f.values():
                    if isinstance(x, int) or isinstance(x, float):
                        s += x*x
                    elif isinstance(x, complex):
                        y = abs(x)
                        s += y*y
                    else: return False
                return s < 1e-20

            def __str__(v):
                "pretty-printing"
                D_list = sorted(v.D, key=repr)
                numdec = 3
                wd = dict([(k,(1+max(len(str(k)), len('{0:.{1}G}'.format(v[k], numdec))))) if isinst\
ance(v[k], int) or isinstance(v[k], float) else (k,(1+max(len(str(k)), len(str(v[k]))))) for k in D_\
list])
                s1 = ''.join(['{0:>{1}}'.format(str(k),wd[k]) for k in D_list])
                s2 = ''.join(['{0:>{1}.{2}G}'.format(v[k],wd[k],numdec) if isinstance(v[k], int) or \
isinstance(v[k], float) else '{0:>{1}}'.format(v[k], wd[k]) for k in D_list])
                return "\n" + s1 + "\n" + '-'*sum(wd.values()) +"\n" + s2

            def __hash__(self):
                "Here we pretend Vecs are immutable so we can form sets of them"
                h = hash(frozenset(self.D))
                for k,v in sorted(self.f.items(), key = lambda x:repr(x[0])):
                    if v != 0:
                        h = hash((h, hash(v)))
                return h

            def __repr__(self):
                return "Vec(" + str(self.D) + "," + str(self.f) + ")"

            def copy(self):
                "Don't make a new copy of the domain D"
                return Vec(self.D, self.f.copy())

            def __iter__(self):
                raise TypeError('%r object is not iterable' % self.__class__.__name__)

        if __name__ == "__main__":
            import doctest
            doctest.testmod()

6.4 Inheritance

Inheritance is one of the basic tenets of object oriented programming and python supports multiple inheritance just like C++. Inheritance provides a mechanism for creating new classes that specialise or modify a base class thereby introducing new functionality. We call the base class the parent class or the super class. An example of a class inheriting from a base class in python is given in the following example.

    class Account:
        """base class for representing user accounts"""
        num_accounts = 0

        def __init__(self, name, balance):
            self.name = name 
            self.balance = balance 
            Account.num_accounts += 1

        def del_account(self):
            Account.num_accounts -= 1

        def __getattr__(self, name):
            """handle attribute reference for non-existent attribute"""
            return "Hey I dont see any attribute called {}".format(name)

        def deposit(self, amt):
            self.balance = self.balance + amt 

        def withdraw(self, amt):
            self.balance = self.balance - amt 

        def inquiry(self):
            return "Name={}, balance={}".format(self.name, self.balance) 

    class SavingsAccount(Account):

        def __init__(self, name, balance, rate):
            super().__init__(name, balance)
            self.rate = rate


        def __repr__(self):
                return "SavingsAccount({}, {}, {})".format(self.name, self.balance, self.rate)


    >>>acct = SavingsAccount("Obi", 10, 1)
    >>>repr(acct)
    SavingsAccount(Obi, 10, 1)

The super keyword

The super keyword plays an integral part in python inheritance. In a single inheritance hierarchy, the super keyword is used to refer to the parent/super class without explicitly naming it. This is similar to the super method in Java. This comes into play when overriding a method and there is a need to also call the parent version of such method as shown in the above example in which the __init__ method in the SavingsAccount class is overridden but the __init__ method of the parent class is also called using the super method. The super keyword plays a more integral role in python inheritance when a multiple inheritance hierarchy exists.

Multiple Inheritance

In multiple inheritance, a class can have multiple parent classes. This type of hierarchy is strongly discouraged. One of the issues with this kind of inheritance is the complexity involved in properly resolving methods when called. Imagine a class, D, that inherits from two classes, B and C and there is a need to call a method from the parent classes however both parent classes implement the same method. How is the order in which classes are searched for the method determined ? A Method Resolution Order algorithm determines how a method is found in a class or any of the class’ base classes. In Python, the resolution order is calculated at class definition time and stored in the class __dict__ as the __mro__ attribute. To illustrate this, imagine a class hierarchy with multiple inheritance such as that showed in the following example.

    >>> class A:
    ...     def meth(self): return "A"
    ... 
    >>> class B(A):
    ...     def meth(self): return "B"
    ... 
    >>> class C(A):
    ...     def meth(self): return "C"
    ... 
    >>> class D(B, C):
    ...     def meth(self): return "X"
    ... 
    >>> 
    >>> D.__mro__
    (<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class \
'object'>)
    >>> 

To obtain an mro, the interpreter method resolution algorithm carries out a left to right depth first listing of all classes in the hierarchy. In the trivial example above, this results in the following class list, [D, B, A, C, A, object]. Note that all objects will inherit from the root object class if no parent class is supplied during class definition. Finally, for each class that occurs multiple times, all occurrences are removed except the last occurrence resulting in an mro of [D, B, C, A, object] for the previous class hierarchy. This result is the order in which classes would be searched for attributes for a given instance of D.

Cooperative method calls with super

This section will show the power of the super keyword in a multiple inheritance hierarchy. The class hierarchy from the previous section is used. This example is from the excellent write up by Guido Van Rossum on Type Unification. Imagine that A defines a method that is overridden by B, C and D. Suppose that there is a requirement that all the methods are called; the method may be a save method that saves an attribute for each type it is defined for, so missing any call will result in some unsaved data in the hierarchy. A combination of super and __mro__ provide the ammunition for solving this problem. This solution is referred to as the call-next method by Guido van Rossum and is shown in the following snippet:

    class A(object):
        def meth(self): 
            "save A's data"
            print("saving A's data")

    class B(A):
        def meth(self): 
            "save B's data" 
            super(B, self).meth()
            print("saving B's data")

    class C(A):
        def meth(self): 
            "save C's data" 
            super(C, self).meth()
            print("saving C's data")

    class D(B, C):
        def meth(self): 
            "save D's data"
            super(D, self).meth()
            print("saving D's data")

When self.meth() is called by an instance of D for example, super(D, self).meth() will find and call B.meth(self), since B is the first base class following D that defines meth in D.__mro__. Now in B.meth, super(B, self).m() is called and since self is an instance of D, the next class after B is C (__mro__ is [D, B, C, A]) and the search for a definition of meth continues here. This finds C.meth which is called, and which in turn calls super(C, self).m(). Using the same MRO, the next class after C is A, and thus A.meth is called. This is the original definition of m, so no further super() call is made at this point. Using super and method resolution order, the interpreter has been able to find and call all version of the meth method implemented by each of the classes in the hierarchy. However, multiple inheritance is best avoided because for more complex class hierarchies, the calls may be way more complicated than this.