A Proposal for Pluggable Modules

Notice: This proposal has been integrated in Trac 0.9. See TracDev/ComponentArchitecture for the latest version of this documentation.

Different projects, organizations or companies have different requirements for a tool such as Trac. For example, while some software development shops have some kind of system for automated nightly builds or continuous integration in place, many others do not. Those who do would like to integrate their build system with Trac, but adding such features to the core would inevitably result in enormous bloat, turning away other potential users who just don't need all the features. Arguably, this is already the case for people who want to, for example, use Trac as Wiki and issue tracker, but don't need the SCM integration — for whatever reason.

Adding new functional areas (aka modules) to Trac currently requires some heavy lifting, touching various parts of the code base. Extending Trac on specific sites will therefore quickly become a maintenance nightmare, because those customized and/or extended installs will not be easily upgradeable.

For these reasons it would be highly desirable for the Trac project to provide support for pluggable modules, so that non-core functionality could be maintained separately from Trac (at least to a certain degree).

Introduction

If functionality is supposed to be made pluggable (and thus optional), we need to figure out what functionality sits at the core of the system. And at that point you will probably already get different opinions. Arguably, you should be able to use Trac without deploying the issue tracker. Quite certainly, the dependency on Subversion should be made optional. The Wiki functionality is central to the system, but do Wiki pages as such and their management also belong to the core system?

In an ideal architecture, there is as little core functionality as absolutely required. Anything that can be separated from the core system should be separated, for a variety of (mostly obvious) reasons:

• Loose coupling between the core system and the individual components allows the addition or modification of functionality with minimum impact on the code-base, thus improving stability and customizability.
• It becomes more likely that individual functional units can be tested in isolation (unit tests).
• The user has more control over what functionality is deployed to a site by choosing only those modules that are absolutely necessary.

The ​Eclipse IDE is an example of a large application that is based almost entirely on individual plug-ins. As Eclipse is based on Java, the development team had a lot of hoops to jump through to create such a dynamic system with a statically typed, compiled language. For example, they rely on plugin descriptors that describe the runtime requirements, what other plug-ins it extends in what way, and what extension points the plug-in itself offers to other plug-ins. I suspect that much of this complexity could be done away with in Trac, while still adopting the basic model of plug-ins that can extend — and can be extended — by other plug-ins.

In the concrete case of Trac, the timeline module rethought as a plug-in might look like this:

• Adds a button to the navigation bar, and extends the URL namespace with the /timeline path.
• Allows other plug-ins to add timed events to the timeline. Each timed event would have a title, a description and a URL.
• In addition, plug-ins extending the timeline could establish "filters" that would allow the user to restrict the type of events displayed (such as you currently can choose from "Ticket changes", "Repository check-ins", etc.)

So what is the core in such a system? For Eclipse, the answer is that the core is actually just the plug-in manager and a couple of essential interfaces that allow "communication" between plug-ins. In any other plug-in based system, the answer should be the same.

So assuming this basic model of operation, let's move on to discussing ideas for how this might be implemented.

Introducing the new trac.core

The first step towards a plugin-based architecture is a revised trac.core module: as the heart of the new architecture it implements a minimal component kernel that allows components to easily extend each others functionality. It provides a "meta-plugin-API": every component can easily offer its own plugin API by declaring "extension points".

What is a Component?

For our purposes, a component is an object that provides a certain type of service within the context of the application. There is at most one instance of any component: components are singletons. That implies that a component does not map to an entity of the application's object model; instead, components represent functional subsystems.

Components can declare "extension points" that other components can “plug in” to. This allows one component to enhance the functionality of the component it extends, without the extended component even knowing that the extending component exists. All that is needed is that the original component exposes – and uses – one or more extension points.

A component can extend any number of other components and still offer its own extension points. This allows a plugin to itself offer a plugin API (i.e. extension point). This feature is the basis for a plugin-based architecture.

The actual functionality and APIs are defined by individual components. The component kernel provides the “magic glue” to hook the different subsystems together – without them necessarily knowing about each other.

Public classes

trac.core.ComponentManager

Manages component life cycle, instantiating registered components on demand

trac.core.Component

Abstract base class for components.

trac.core.ExtensionPoint

Declares an extension point on a component that other components can plug in to.

trac.core.Interface

Every extension point specifies the contract that extenders must conform to via an Interface subclass.

Declaring a component

The simplest possible component is an empty class derived from trac.core.Component:

from trac.core import *

class MyComponent(Component):
    pass

In the context of a component manager, this component can already be used:

    comp_mgr = ComponentManager()
    my_comp = MyComponent(comp_mgr)

Remember that components follow the singleton pattern. There is only one active instance of any component per component manager. The component constructor is "magic" in that it checks with the component manager whether there's already an active instance before allocating a new instance. If the component was already instantiated, the existing instance is returned:

    my_comp1 = MyComponent(comp_mgr)
    my_comp2 = MyComponent(comp_mgr)
    assert id(my_comp1) == id(my_comp2)

If a component needs to initialize data members, it can override the __init__ method. But because the component manager needs to be able to instantiate the component on demand, __init__ must not require any extra parameters, including the reference to the component manager passed into the constructor:

    from trac.core import *

    class MyComponent(Component):
        def __init__(self):
            self.data = {}

    comp_mgr = ComponentManager()
    my_comp = MyComponent(comp_mgr)

Direct Component sub-classes also do not need to worry about invoking the base classes __init__ method (which is empty).

Declaring an extension point

The component above doesn't actually do anything. Making an object a component only makes it act as a singleton in the scope of a component manager, which isn't that exciting in itself.

The real value of components becomes clearer when the facilities for extensions are used. As a simple example, the following component provides an extension point that lets other components listen to changes to the data it manages (in this case a list of to-do items) – following the widely known observable pattern:

    from trac.core import *
    
    class ITodoObserver(Interface):
        def todo_added(name, description):
            """Called when a to-do item is added."""

    class TodoList(Component):
        observers = ExtensionPoint(ITodoObserver)

        def __init__(self):
            self.todos = {}

        def add(self, name, description):
            assert not name in self.todos, 'To-do already in list'
            self.todos[name] = description
            for observer in self.observers:
                observer.todo_added(name, description)

Here, the TodoList class declares an extension point called observers with the interface ITodoObserver. The interface defines the contract that extending components need to conform to.

The TodoList component notifies the observers inside the add() method by iterating over self.observers and calling the todo_added() method for each. The Component class performs some magic to enable that: Conceptually, it intercepts access to the extension point attribute and finds all registered components that declare to extend the extension point. For each of those components, it gets the instance from the component manager, potentially activating it if it is getting accessed for the first time.

Plugging in to an extension point

Now that we have an extendable component, let's add another component that extends it:

    class TodoPrinter(Component):
        implements(ITodoObserver)

        def todo_added(self, name, description):
            print 'TODO:', name
            print '     ', description

This class implements the ITodoObserver interface declared above, and simply prints every new to-do item to the console. By declaring to implement the interface, it transparently registers itself as an extension of the TodoList class.

Note that you don't actually derive the component from the interface it implements. That is because conformance to an interface is orthogonal to inheritance; and because Python doesn't have static typing, there's no need to explicitly mark the component as implementing an interface.

You can specify multiple extension point interfaces to extend with the implements method by simply passing them as additional arguments.

Putting it together

Now that we've declared both a component exposing an extension point, and another component extending that extension point, let's use the to-do list example to see what happens:

    comp_mgr = ComponentManager()
    todo_list = TodoList(comp_mgr)

    todo_list.add('Make coffee',
                  'Really need to make some coffee')
    todo_list.add('Bug triage',
                  'Double-check that all known issues were addressed')

Running this script will produce the following output:

    TODO: Make coffee
          Really need to make some coffee
    TODO: Bug triage
          Double-check that all known issues were addressed

This output obviously comes from the TodoPrinter. Note however that the code snippet above doesn't even mention that class. All that is needed to have it participating in the action is to declare the class. (That implies that an extending class needs to be imported by a python script to be registered. The aspect of loading components is however separate from the extension mechanism itself.)

Current Status

The architecture explained above is implemented for Trac in version 0.9. There are currently three extension points exposed:

• IRequestHandler: allows plugins to process HTTP requests
• INavigationContributor: allows plugins to contribute links to the Trac navigation menus
• ITimelineEventProvider: allows plugins to add events to the timeline

The branch also provides an overview of installed plugins, and what extension points are provided and extended:

​http://projects.edgewall.com/trac/attachment/wiki/TracPluggableModules/about_plugins.png?format=raw

Other Aspects of Plug-Ins

Plug-ins will rarely consist solely of python code. Instead, they will almost always include templates, style sheets and images. I do not propose any kind of single-directory or even single-file deployment of plug-ins. Rather the user should be responsible for copying these files into the appropriate directories.

Another aspect not discussed yet are the database requirements of modules. The plug-in approach moves us away from the monolithic database approach currently in use. Individual plug-ins may need their own tables in the database. They may need to upgrade those tables separately from the rest of the database. Possibly using a per-table version number instead of a global database version number will suffice here.

Plug-ins should also be able to not only hook into the web-interface, but also into TracAdmin. The approach for this would be similar to the one discussed above, meaning that modules should be able to contribute commands.