Using Mix-ins with Python

Mix-in programming is a style of software development where units of functionality are created in a class and then mixed in with other classes. This might sound like simple inheritance at first, but a mix-in differs from a traditional class in one or more of the following ways. Often a mix-in is not the “primary” superclass of any given class, does not care what class it is used with, is used with many classes scattered throughout the class hierarchy and is introduced dynamically at runtime.

There are several reasons to use mix-ins: they extend existing classes in new areas without having to edit, maintain or merge with their source code; they keep project components (such as domain frameworks and interface frameworks) separate; they ease the creation of new classes by providing a grab bag of functionalities that can be combined as needed; and they overcome a limitation of subclassing, whereby a new subclass has no effect if objects of the original class are still being created in other parts of the software.

So while mix-ins are not a distinct technical feature of Python, the benefits of this technique are worth studying.

Python provides an ideal language for mix-in development because it supports multiple inheritance, supports full-dynamic binding and allows dynamic changes to classes. Before we dive into Python, let me admit that mix-ins are old hat. The first time I saw mix-in programming by that name was when reviewing the now-defunct Taligent Project, known for its Pink operating system and CommonPoint application framework. However, since C++ does not support language feature #2, full-dynamic binding, or language feature #3, dynamic changes at runtime, I'm not surprised that the approach didn't bring to fruition all its inventors had hoped for.

I have also seen another instance of mix-in programming under a different name. Objective-C has a nifty language feature called categories that allows you to add and replace methods of existing classes, even without access to their source code.

This is great for repairing existing system classes and extending their capabilities. Also, combined with an ability to load libraries dynamically, categories are quite effective in improving the structure of applications and reducing code.

The grapevine informs me that Symbolics' object-oriented Flavors system is most likely the earliest appearance of bona fide mix-ins. The designers were inspired by Steve's Ice Cream Parlor in Cambridge, Massachusetts where customers started with a basic flavor of ice cream (vanilla, chocolate, etc.) and added any combination of mix-ins (nuts, fudge, chocolate chips, etc.). In the Symbolics system, large, standalone classes were known as flavors while smaller helper classes designed for enhancing other classes were known as mix-ins. A reference can be found on the Web at www.kirkrader.com/examples/cpp/mixin.htm.

Objective-C: I Knew Him Well

Python 2.0

Having paid our respects to the dead (Taligent), nearly dead (Objective-C) and legendary (Symbolics), let's start digging into the features that make Python a great language for mix-in programming. For one, Python supports multiple inheritance. That is, in Python, a class can inherit more than one class:

class Server(Object, Configurable):
    pass

also, Python supports full-dynamic binding. When passing a message to an object such as:

obj.load(filename)

One thing to keep in mind is the order of searching with regard to multiple inheritance. The search order goes from left to right through the base classes, and for any given base class, goes deep into its ancestor classes.

When you create mix-ins, keep in mind the potential for method names to clash. By creating distinct mix-ins with well-named methods you can generally avoid any surprises. Lastly, Python supports dynamic changes to the class hierarchy.

Most Python “things”, whether they are lists, dictionaries, classes or instances, have a set of accessible attributes. Python classes have an attribute named __bases__, which is a tuple of their base classes. Consistent with Python design, you can play with it at runtime. In the following session with the Python interactive interpreter seen in Listing 1, we create two classes and then later change the inheritance. Our person in Listing 1 isn't very friendly so let's change it. In fact, let's change all people so that we'll never have this problem again:

<<< Person.__bases__ += (Friendly,)
<<< p.hello()
Hello

Listing 1. Installing a Mix-in Dynamically

The first statement above changes the base classes of Person. By using += (as opposed to =) we avoid accidentally removing existing base classes, especially if a future version of the code makes Person inherit from another class. Also, the funny looking expression, (Friendly,), specifies a tuple that would normally simply be enclosed in parenthesis. However, while Python readily recognizes <If“Courier”>(x,y)<I$f$> as a tuple of two elements, it recognizes <If“Courier”>(x)<I$f$> as a parenthesized expression. Appending the comma forces tuple recognition.

The most straightforward way to apply mix-ins is at design time within the construction of a module. One of the more famous third-party modules for Python, MySQLdb, does exactly this.

Python defines a standard programmatic interface for database access named DB API (http://www.python.org/topics/database/). Andy Dustman's MySQLdb module implements this interface so that Python programmers can make connections and send queries to a MySQL server. It can be found at http://dustman.net/andy/python/MySQLdb/.

MySQLdb provides three major features for the cursor objects it creates. It reports warnings when necessary; it stores result sets on the client side or uses them on the server side as needed, and it returns results as tuples (e.g., immutable lists) or dictionaries.

Rather than combining all of these into one monolithic class, MySQLdb defines mix-in classes for each of them:

class CursorWarningMixIn:
class CursorStoreResultMixIn:
class CursorUseResultMixIn
class CursorTupleRowsMixIn:
class CursorDictRowsMixIn(CursorTupleRowsMixIn):

Remember that mix-ins are classes, so they can take advantage of inheritance, as we see with CursorDictRowsMixIn, which inherits CursorTupleRowsMixIn.

None of the mix-ins above can stand on their own: a BaseCursor class provides the required core functionality for any type of cursor. Using these mix-ins in combination with BaseCursor, MySQLdb offers every combination of warnings, storage and result types (eight in all). When creating a database connection, you can pass the cursor class you desire:

conn = MySQLdb.connection (cursorclass=MySQLdb.DictCursor)

Mix-ins don't only help in the creation of MySQLdb itself. They also make it more extensible by allowing you to pick and choose features for your own custom cursor classes.

Note that these class names are suffixed with MixIn to emphasize their nature. Another common convention is to append “-able” or “-ible” to the end of the name as in Configurable or NamedValueAccessible.

Let's use that last one as an example. The NamedValueAccessible mix-in adds the method valueForKey( ) to whatever class with which it is joined. For obj.valueForKey(name), this method will return one of the following:

In other words, valueForKey( ) looks for methods or attributes, either public or private, in order to return a value for the given key. The design of this method reflects the fact that Python objects often provide information through both attributes and methods. See Listing 2 for the implementation.

Listing 2. A Mix-in for Uniform Value Access

A useful application of this mix-in is to implement generic code for writing logs (see Listing 3).

Listing 3. Applying the NamedValueAccessible Mix-in for Logging

By simply adding new keys to the logColumns( ) method, the log can be expanded without having to modify the code that generates it, which is found in logEntry( ). More importantly, you can imagine that logColumns( ) could read its list of fields from a simple configuration file.

The transaction object itself is free to provide the given values via either methods or attributes, due to the flexibility of the valueForKey( ) method. Making mix-ins flexible increases their utility and is an art that can be developed over time.

So far we have seen examples of using mix-ins during the construction of classes. However, Python's dynamic nature also allows us to mix in functionality at runtime. The simplest technique for doing so is to modify the base classes of the given class, as described earlier. A function allows us to keep this operation opaque and enhance it later if need be:

def MixIn(pyClass, mixInClass):
    pyClass.__bases__ += mixInClass

Let's consider a situation that makes the utility of MixIn( ) obvious. In the construction of internet applications, keeping domain classes separate from interface classes is generally a good idea. Domain classes represent the concepts, data and operations of a specific application. They are independent of operating system, user interface, database, etc. Some writers refer to domain objects as business objects, model objects or problem space objects.

Keeping the domain and interface separate makes sense for various reasons. An individual focus is created for two key areas that are largely independent: What is the subject material of the problem? And, how should that be presented? New interfaces can be constructed without modifying or rewriting the domain classes. In fact, multiple interfaces can be provided.

Domain classes for a story publishing system might include Story, Author and Site. These classes contain essential attributes (such as title, body, name, e-mail, etc.) and various operations (save, load, publish, etc.).

One interface for such a system could be a web site that allows users to create, edit, delete and publish these stories. When developing such a site, it would be useful if our domain classes, such as Story, have the methods renderView( ) and renderForm( ), which write HTML for either displaying the story or editing it with a form.

Using mix-ins, we can develop such functionality outside of the domain classes:

class StoryInterface:
    def renderView(self):
        # write the HTML representation of the story
        pass
    def renderForm(self):
        # write the HTML form to edit the story
        pass

And within the code that backs the web site, mix it in like so:

from MixIn import MixIn
from Domain.Story import Story
MixIn(Story, StoryInterface)

One could argue that a new class might be created to bring the two together:

class StoryInterface:
    ...
from Domain.Story import Story
class Story(Story, StoryInterface): pass

Or one could argue that StoryInterface might be made a subclass of Story in order to achieve the same benefit. However, consider the case when Story already has other domain subclasses:

class Story: ...
class Editorial(Story): ...
class Feature(Story): ...
class Column(Story): ...

Also, consider the case where Story objects are created in parts of the code where Story is hard-coded. While poor practice, this is common. In this situation, creating subclasses of Story will have no effect on the code that ignores them.

One warning regarding dynamic mix-ins: they can change the behavior of existing objects (because they change the classes of those objects). This could lead to unpredictable results, as most classes are not designed with that type of change in mind. The safe way to use dynamic mix-ins is to install them when the application first starts, before any objects are created.

The first enhancement we can add to MixIn( ) is to check that we're not mixing in the same class twice:

def MixIn(pyClass, mixInClass):
    if mixInClass not in pyClass.__bases__
        pyClass.__bases__ += (mixInClass,)

In practice, I find more often than not, that I want my mix-in methods to take a high priority, even superseding inherited methods if needed. The next version of the function puts the mix-in class at the front of the sequence of base classes but allows you to override this behavior with an optional argument:

def MixIn(pyClass, mixInClass, makeLast=0):
  if mixInClass not in pyClass.__bases__
    if makeLast:
      pyClass.__bases__ += (mixInClass,)
    else:
      pyClass.__bases__ = (mixInClass,) + pyClass.__bases__

# not so readable:
MixIn(Story, StoryInterface, 1)
# much better:
MixIn(Story, StoryInterface, makeLast=1)

This new version still doesn't allow methods in the actual class to be overridden with methods in the mix-in. In order to accomplish that, the mix-in methods must actually be installed in the class. Fortunately, Python is dynamic enough to accomplish this. Listing 4 gives the source code for our final version of MixIn( ). By default it will install the methods of the mix-in directly into the target class, even taking care to traverse the base classes of the mix-in. The invocation is the same:

MixIn(Story, StoryInterface)

An extra makeAncestor=1 argument can be provided for the new MixIn( ) to get the old semantics (e.g., make the mix-in a base class of the target class). The ability to put the mix-in at the end of the base classes has been removed, since I have never needed this in practice.

An even more sophisticated version of this function could return (perhaps optionally) a list of methods that clash between the two, or raise an exception accompanied by such a list, if the overlap exists.

When making heavy use of after-the-fact mix-ins, invocations of the MixIn( ) function become repetitious. For example, a GUI application might have a mix-in for every domain class in existence, thereby requiring a call such as this for each one:

from Domain.User import User
MixIn(User, UserMixIn)

One solution is to bind the mix-ins to their target classes by name and have the application install these at startup. For example, all mix-ins could be named directly after the class they modify and put into a MixIns/ directory. The code in Listing 5 will install them.

Listing 5. Detecting and Installing Mix-ins Named after Their Classes

While it's fun to explore more sophisticated versions of the MixIn( ) function, the most important key is the ability to apply them in order to improve your software. Here are some additional uses to stimulate your imagination:

Mix-ins are great for improving modularity and enhancing existing classes without having to get intimate with their source code. This in turn supports other design paradigms, like separation of domain and interface, dynamic configuration and plug-ins. Python's inherent support for multiple inheritance, dynamic binding and dynamic changes to classes enables a very powerful technique. As you continue to write Python code, consider ways in which mix-ins can enhance your software.