Python's magic methods - also known as dunder (double underscore) methods - can be used to implement a lot of cool things. Most of the time we use them for simple stuff, such as constructors (__init__), string representation (__str__, __repr__) or arithmetic operators (__add__/__mul__). There are however many more magic methods which you probably haven't heard about and in this article we will explore all of them (even the hidden and undocumented)!
Iterator Length
We all know the __len__ method that you can use to implement len() function on your container classes. What if you want to get length of a class object that implements an iterator, though?
it = iter(range(100))
print(it.__length_hint__())
# 100
next(it)
print(it.__length_hint__())
# 99
a = [1, 2, 3, 4, 5]
it = iter(a)
print(it.__length_hint__())
# 5
next(it)
print(it.__length_hint__())
# 4
a.append(6)
print(it.__length_hint__())
# 5
All you need to do is implement __length_hint__ method, which is also present on builtin iterators (but not generators) as you can see above. Additionally, as you can see here, it also supports dynamic length changes. With that said though - as the name suggests - it's really just a _hint_ and can be wholly inaccurate - for the list iterator you will get exact results, for other iterators not necessarily. However, even if it's not accurate, it can be very helpful for optimizations as explained in the PEP 424 which introduced it a while back.
Meta Programming
Bulk of the magic methods you rarely see being used are related to meta-programming and while meta-programming is something you probably don't have to use every day, there are some handy tricks you can use it for.
One such trick is using __init_subclass__ as a shortcut to extend the functionalities of a base class without having to handle metaclasses:
class Pet:
def __init_subclass__(cls, /, default_breed, **kwargs):
super().__init_subclass__(**kwargs)
cls.default_breed = default_breed
class Dog(Pet, default_name="German Shepherd"):
pass
Here we use it to add keyword argument to a base class, which can be set when defining the child. In the real world use cases, you would probably use this in a situation where you want to process the provided argument, rather than just assigning to an attribute.
While this might seem very obscure and rarely useful, you probably encountered it many times already, as it can be used when building an API where users subclass your parent class like in SQLAlchemy models or Flask Views.
Another metaclass magic method that you might find use for is __call__. This method allows you to customize what happens when you call a class instance:
class CallableClass:
def __call__(self, *args, **kwargs):
print("I was called!")
instance = CallableClass()
instance()
# I was called!
Funnily, you can use this to create class that cannot be called:
class NoInstances(type):
def __call__(cls, *args, **kwargs):
raise TypeError("Can't create instance of this class")
class SomeClass(metaclass=NoInstances):
@staticmethod
def func(x):
print('A static method')
instance = SomeClass()
# TypeError: Can't create instance of this class
This can be useful if you have a class that only have static methods and therefore there's no good reason to create instances of said class.
Another similar use case that comes to mind is singleton pattern - a class that can have at most single instance:
class Singleton(type):
def __init__(cls, *args, **kwargs):
cls.__instance = None
super().__init__(*args, **kwargs)
def __call__(cls, *args, **kwargs):
if cls.__instance is None:
cls.__instance = super().__call__(*args, **kwargs)
return cls.__instance
else:
return cls.__instance
class Logger(metaclass=Singleton):
def __init__(self):
print("Creating global Logger instance")
Here we demonstrate this by implementing a global logger class that there can be only one instance of. The concept might look a little complicated, but this implementation is pretty simple - the Singleton class holds a private __instance - if there's none, it gets created and assigned to the attribute, if it already exists, it just gets returned.
Now, let's say you have a class and you want to create an instance of it without invoking __init__. The __new__ magic method can help with that:
class Document:
def __init__(self, text):
self.text = text
bare_document = Document.__new__(Document)
print(bare_document.text)
# AttributeError: 'Document' object has no attribute 'text'
setattr(bare_document, "text", "Text of the document")
There are situations where you might need to bypass the usual process of creating instance and the above code shows how you can do that. Instead of calling Document(...) we invoke Document.__new__(Document) which creates a bare instance without invoking __init__. Because of that, the instance attribute(s) - in this case text - aren't initialized, to fix that, we can use setattr function (which - by the way - is also a magic method - __setattr__).
You might be wondering why would you ever want to do that. One example would be implementing alternative constructor, such as this:
class Document:
def __init__(self, text):
self.text = text
@classmethod
def from_file(cls, file): # Alternative constructor
d = cls.__new__(cls)
# Do stuff...
return d
Here we define from_file method which serves as a constructor by first creating instance with __new__ and then configuring it without invoking __init__.
Next meta-programming related magic method we will take a look at here is __getattr__. This method gets called when normal attribute access fails. This can be leveraged to delegate access/calls to missing methods to another class:
class String:
def __init__(self, value):
self._value = str(value)
def custom_operation(self):
pass
def __getattr__(self, name):
return getattr(self._value, name)
s = String("some text")
s.custom_operation() # Calls String.custom_operation()
print(s.split()) # Calls String.__getattr__("split") and delegates to str.split
# ['some', 'text']
print("some text" + "more text")
# ... works
print(s + "more text")
# TypeError: unsupported operand type(s) for +: 'String' and 'str'
Let's suppose that we want to define custom implementation of string with some extra functions such as custom_operation above. We however, don't want to re-implement every single string method such split, join, capitalize, and so on. Therefore, we use __getattr__ to call these existing string methods in case they're not found on our class.
While this works great for normal methods, notice that in the example above the operations such as concatenation provided by magic method __add__ doesn't get delegated. So, if we wanted those to work as well, then we would have to re-implement them.
Introspection
Final meta-programming related magic method we will try out is __getattribute__. This one looks very similar to the previous __getattr__. There's however a slight difference - as already mentioned __getattr__ gets invoked only when attribute lookup fails, while __getattribute__ is invoked before attribute lookup is attempted.
You can therefore use __getattribute__ to control access to attributes, or you can for example create a decorator that logs every attempt to access instance attribute:
def logger(cls):
original_getattribute = cls.__getattribute__
def getattribute(self, name):
print(f"Getting: '{name}'")
return original_getattribute(self, name)
cls.__getattribute__ = getattribute
return cls
@logger
class SomeClass:
def __init__(self, attr):
self.attr = attr
def func(self):
...
instance = SomeClass("value")
instance.attr
# Getting: 'attr'
instance.func()
# Getting: 'func'
The logger decorator function start by taking note of the original __getattribute__ method of the class it decorates. It then replaces it with custom method that first logs the name of the attribute being accessed before calling the original __getattribute__ method.
Magic Attributes
So far, we've talked only about magic methods, but there are also quite a few magic variables/attributes in Python. One of them is __all__:
# some_module/__init__.py
__all__ = ["func", "some_var"]
some_var = "data"
some_other_var = "more data"
def func():
return "hello"
# -----------
from some_module import *
print(some_var)
# "data"
print(func())
# "hello"
print(some_other_var)
# Exception, "some_other_var" is not exported by the module
This magic attribute can be used to define which variables and function get exported from a module. In the example we create a Python module in .../some_module/ with single file (__init__.py). In this file we define 2 variables and one function of which we export only 2 (func and some_var). If we then try to import contents of some_module in other Python program we only get the 2 exported ones.
Be aware though, that the __all__ variable only effects the * import shown above, you're still able to import the un-exported functions and variables with imports like import some_other_var from some_module.
Another double underscore variable (module attribute) that you might have seen is __file__. This variable simply identifies path to the file it's accessed from:
from pathlib import Path
print(__file__)
print(Path(__file__).resolve())
# /home/.../directory/examples.py
# Or the old way:
import os
print(os.path.dirname(os.path.abspath(__file__)))
# /home/.../directory/
And combining the __all__ and __file__, you can for example load all modules in a folder:
# Directory structure:
# .
# |____some_dir
# |____module_three.py
# |____module_two.py
# |____module_one.py
from pathlib import Path, PurePath
modules = list(Path(__file__).parent.glob("*.py"))
print([PurePath(f).stem for f in modules if f.is_file() and not f.name == "__init__.py"])
# ['module_one', 'module_two', 'module_three']
And one last we will try out is __debug__ attribute. This one - obviously - can be used for debugging, but more specifically it can be used to better control assertions:
# example.py
def func():
if __debug__:
print("debugging logs")
# Do stuff...
func()
If we run this piece of code normally with python example.py, we will see the "debugging logs" printed out, however if we use python3 -O example.py, the optimization flag (-O) will set __debug__ to false and strip out the debugging messages. Therefore, if you run your code with -O in production environment, you won't have to worry about forgotten print calls left from debugging, as they will be all stripped out.
Hidden and Undocumented
All of the above methods and attributes might be somewhat unknown, but they're all in Python docs. There are however a couple that are not clearly documented and/or somewhat hidden.
You can for example run the following code to discover a couple new ones:
import struct
dir(struct)
# ['Struct', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__',
# '__spec__', '_clearcache', 'calcsize', 'error', 'iter_unpack', 'pack', 'pack_into', 'unpack', 'unpack_from']
Besides these, there are quite a few more as listed in Python bug tracker BPO 23639. As pointed out there though, most of them are implementation details or private names which should not be accessed. So them being not documented is probably for the best.
Making Your Own?
Now, with so many magic methods and attributes, could you actually make your own? Well, you could, but you should not.
The double underscore names are reserved for future extensions of the Python language and should not be used for your own code. If you decide to use such name in your code anyway, then you're running a risk of them getting added to the Python interpreter in the future, which would more than likely break your code.
Closing Thoughts
In this article we looked at the lesser known magic methods and attribute that I find useful or interesting, there are however more of them listed in docs that might be useful for you. Most of them can be found in Python Data Model docs. If you however want to dig deeper, you can try searching for "__" in Python docs, which will turn up many more methods and attributes to explore and play with.