Demystifying the magic in Python

Why

Python has many inbuilt magic methods which can be overridden in your own custom classes to create more magic. In the official Python docs these methods are scattered all over the place and does not reflect a proper understanding. Here I am trying to get the gist out of all the major magic methods and how a Pythonista can use them in their code.

What are magic methods

In Python magic methods are those which start with double underscore and end with double underscore ( For Ex: __init__ ). These are special methods that you can define in your classes. And they work seamlessly with some built in keywords.

Methods

__new__

        The first method that's called during object instantiation. Takes a class as argument and additional arguments are passed on the to the __init__ method. It’s used very rarely and has very limited usefulness in Python. One use case for __new__ is to allow subclassing for immutable types.

class inch(float):
        def __new__(cls, arg=0.00):
                return float.__new__(cls, arg*0.02)

print(inch(12))
>> 0.3

class inch(float):
        def __init__(self, arg=0.00):
                float.__init__(self, arg*0.02)

print(inch(12))
>> Error

__init__

        Init is the constructor method of a class during object instantiation. When you call a class with arguments, they are passed to the init method. __init__ is almost used universally in Python. Takes the self as the first argument, it represents the the instance of the object itself. Setting variables as self would make them exist for the lifetime of the object. Otherwise they would cease to exist when the init method goes out of scope.

class point:
        i = 100
        def __init__(self):
                self.i = 200

print(point.i)
>> 100
a = point()
print(a.i)
>> 200

__init__ vs __new__

        __new__ is a static method where as __init__ is an instance method. __new__ takes the cls as the first parameter and __init__ takes self as the first parameter. Until the instance has been created there is no self. __new__ is executed before __init__. __new__ is also the place where per-class customisation of the "allocate a new instance" part is done. For example, this is place a singleton design pattern can be implemented.

class Singleton(type):
       def __new__(cls, *args, **kwds):
            it = cls.__dict__.get("__it__")
            if it is not None:
                return it
            cls.__it__ = it = object.__new__(cls)
            it.init(*args, **kwds)
            return it
        def init(self, *args, **kwds):
            Pass

__del__

        If __new__ is used for constructing, then __del__ is used for destructing. Behaviour needed when object is garbage collected can be written here. Although one should not rely on __del__ method for cleaning up sockets or open files because there is no guarantee that it would be called when interpreter exits. Just like __new__,  __del__ too is not a beloved method in Python because there is no guarantee that it would be called and it has some other odd issues. Hence manually handling connections to sockets and files, especially using context managers is a better option.
        Also not that if you call del x, and somewhere else x is referenced then x object’s __del__ method would not be called. It would only decrease the reference count by one.

class T:
        def __del__(self):
                print("deleted")

a = T()
b = a
del a
del b
>> deleted

__iter__ and __next__

        To full implemented the iterator protocol a class must define both the __iter__ and the __next__ methods. The __iter__ is called during the initialization of a iteration, for example, for loop. The __iter__ method should return an object that has implemented the __next__ method. The __next__ method is called whenever the global next() is called using an iterator object. In a for loop Python implicitly calls the next(). And the __next__ method should raise the StopIteration exception to signal end of iteration.

        Note that if a class has the __iter__ method but not the __next__ method, then its an iterable and not an iterator.

class OddNum:

  def __init__(self, num = 0):
    self.num = num
  def __init__(self, num = 0):
    self.num = num

  def __iter__(self):
    self.x = 1
    return self

  def __next__(self):
    if self.x <= self.num:
      odd_num = self.x
      self.x += 2
      return odd_num
    else:
      raise StopIteration

for num in OddNum(10):
    print(num)

>> 1
>> 3
>> 5
>> 7
>> 9

__call__

        The __call__ method allows the instance of an object to be called as a function itself. This method can also take in variable number of arguments. The __call__ method is an intuitive way to modify the instance’s state.

class plane:
        def __init__(self, x, y):
                self.x = x
                self.y = y
                print(self.x)
                print(self.y)

        def __call__(self, x, y):
                self.x = x
                self.y = y
                print(self.x)
                print(self.y)

a = plane(5, 10)
>> 5
>> 10
a(15, 20)
>> 15
>> 20

__enter__ and __exit__

        Sometimes there is a need to execute a pair of operations, for example, opening and closing of file or locking a data structure and then releasing it for others or opening and closing sockets. To achieve this, since Python 2.5, we can use the with keyword. The with keyword is an inbuilt context manager which implements the context manager protocol. To implement the context manager protocol one needs to define the __enter__ and __exit__ methods of a class.

The enter method should return the resource object so its bound to the code within the with block.  First __enter__ is executed, then the code block inside the with statement and then the __exit__ method. This is ideal in use cases where we would want to cleanup open resources.

class Closer:

    def __init__(self, obj):
        self.obj = obj

    def __enter__(self):
        return self.obj

    def __exit__(self, exception_type, exception_val, trace):
        try:
           self.obj.close()
        except AttributeError:
           print 'Not closable.'
           return True

Other types:

1. Comparison

        Python has a whole slew of magic methods designed to implement intuitive comparisons between objects using operators.

[__cmp__, __eq__, __ne__, __lt__, __gt__, __le__, __ge__]

2. Numeric

        Just like you can create ways for instances of your class to be compared with comparison operators, you can define behavior for numeric operators.

        There are a lot numeric methods, listing down the categories here,

• Unary
• Normal Arithmetic operators
• Reflected Arithmetic operators
• Augmented assignment
• Type conversion magic methods

3. Representation

        It's often useful to have a string representation of a class. In Python, there are a few methods that you can implement in your class definition to customize how built in functions that return representations of your class behave.

• __str__ : Defines behaviour when str() is called on instance of your class
• __repr__: Defined behaviour when repr() is called on instance of your class. The major difference between __str__ and __repr__ is the audience. __str__ is for human readability and __repr__ is for machine readable and in many cases the later is even valid Python code.
• __unicode__: Define behaviour when unicode() is called on instance of your class. Note that when __str__ is not defined and it str() is called then __unicode__ won’t suffice.
• There are others too, listing them here,
• __dir__, __has__, __sizeof__, __format__

4. Access Control

        Many people coming to Python from other languages complain that it lacks true encapsulation for classes, that is, there's no way to define private attributes with public getter and setters. This couldn't be farther than the truth: it just happens that Python accomplishes a great deal of encapsulation through "magic", instead of explicit modifiers for methods or fields.

• __getattr__: This is called when the property or method is not found. This could be useful in cases where a default value can be returned as a fail safe. However this is not really encapsulation.
• __setattr__: Opposite of the above, this enables encapsulation because when you want to set the value of a property you can add rules in this method to control the behaviour.
• __delattr__: Just like __setattr__ but this is for deletion. Like __setattr__ care needs to be taken when using __delattr__ as it could easily lead of infinite recursion.

def __setattr__(self, name, value):
    self.name = value # this will cause recursion

def __setattr__(self, name, value):
    super(Class, self).__setattr__(name, value) # this will not cause recursion

Takeaway

Using magic methods can help us write more elegant, feature rich and Pythonic style code. Magic methods also help us write popular design patterns like singleton, factory, adapter etc. This article for beginners can help in understanding and writing in a correct way and for experienced developers is a good refresher.

References

https://rszalski.github.io/magicmethods/#comparisons

Intermediate Python Book by Obi Ike-Nwosu

https://www.python.org/

---