A Proposal for Pluggable Modules

Different projects, organizations or companies have different requirements for a tool such as Trac. For example, while some software development shops have some kind of system for automated nightly builds or continuous integration in place, many others do not. Those who do would like to integrate their build system with Trac, but adding such features to the core would inevitably result in enormous bloat, turning away other potential users who just don't need all the features. Arguably, this is already the case for people who want to, for example, use Trac as Wiki and issue tracker, but don't need the SCM integration — for whatever reason.

Adding new functional areas (aka modules) to Trac currently requires some heavy lifting, touching various parts of the code base. Extending Trac on specific sites will therefore quickly become a maintenance nightmare, because those customized and/or extended installs will not be easily upgradeable.

For these reasons it would be highly desirable for the Trac project to provide support for pluggable modules, so that non-core functionality could be maintained separately from Trac (at least to a certain degree). This would for example be used by the Wiki module to build the list of names of existing wiki pages.

Introduction

If functionality is supposed to be made pluggable (and thus optional), we need to figure out what functionality sits at the core of the system. And at that point you will probably already get different opinions. Arguably, you should be able to use Trac without deploying the issue tracker. Quite certainly, the dependency on Subversion should be made optional. The Wiki functionality is central to the system, but do Wiki pages as such and their management also belong to the core system?

In an ideal architecture, there is as little core functionality as absolutely required. Anything that can be separated from the core system, should be separated, for a variety of (mostly obvious) reasons:

• Loose coupling between the core system and the individual components allows the addition or modification of functionality with minimum impact on the code-base, thus improving stability and customizability.
• It becomes more likely that individual functional units can be tested in isolation (unit tests).
• The user has more control over what functionality is deployed to a site by choosing only those modules that are absolutely necessary.

The ​Eclipse IDE is an example of a large application that is based almost entirely on individual plug-ins. As Eclipse is based on Java, the development team had a lot of hoops to jump through to create such a dynamic system with a statically typed, compiled language. For example, they rely on plugin descriptors that describe the runtime requirements, what other plug-ins it extends in what way, and what extension points the plug-in itself offers to other plug-ins. I suspect that much of this complexity could be done away with in Trac, while still adopting the basic model of plug-ins that can extend — and can be extended — by other plug-ins.

In the concrete case of Trac, the timeline module rethought as a plug-in might look like this:

• Adds a button to the navigation bar, and extends the URL namespace with the /timeline path.
• Allows other plug-ins to add timed events to the timeline. Each timed event would have a title, a description and a URL.
• In addition, plug-ins extending the timeline could establish "filters" that would allow the user to restrict the type of events displayed (such as you currently can choose from "Ticket changes", "Repository check-ins", etc.)

So what is the core in such a system? For Eclipse, the answer is that the core is actually just the plug-in manager and a couple of essential interfaces that allow "communication" between plug-ins. In any other plug-in based system, the answer should be the same.

So assuming this basic model of operation, let's move on to discussing ideas for how this might be implemented.

The Module Manager

There needs to be a central facility that manages the discovery and loading of plug-ins as well as the communication between them. We'll call this the ModuleManager for now.

The module manager should probably be part of the environment, or in other words, for any environment their should be one module registry. Module discovery would presumably be based on a given paths (list of directories) that contain python modules. The registry class would find all classes that somehow identify themselves as Trac modules (for example by sub-classing a central Module class, or by defining some module-scope variable). There could also be a 'plugins' folder in Trac environments that would be added the path by default.

An alternative and possibly more efficient but also more cumbersome approach would be to require the user to specify the "fully qualified" name of every additional module, for example in TracIni. Modules such as the timeline would be in the list by default, but could be disabled by removing them.

Modules would get loaded as any other python module, but should also be able to hook into the initialization of the system via an exported function.

The Module Interface

In the current Trac architecture, modules are basically HTTP request processors that live for exactly one request. In the envisioned system, the modules should be able to outlive a request, and also participate in the request processing of other modules, so as to be able to contribute to their functionality.

So instead of getting the request object as argument to their constructor, they'd have an interface like:

class XxxModule(Module):

    def __init__(self, env):
        Module.__init__(self, env)
        # do whatever other initialization may be needed
        pass

    def process(self, req):
        # do whatever needs to be done
        pass

In an environment such as tracd or mod_python, the process method might be called concurrently on the same module object, so module implementations should not store request-specific data as instance variables, but rather pass such data through as parameters to other methods.

Renaming the current "render" to "process" here simply because the latter sounds more appropriate.

Module Cooperation

As discussed above, a basic requirement is that the modules should be able to:

• contribute functionality to other modules
• provide hooks so that other modules can contribute to its functionality

The supporting model for such cooperation between modules should be functions defined by the contributing plug-in, and called by the plug-in to which we contribute. For example, let's assume the timeline module documents the following hook functions (or extension points in Eclipse jargon):

    def get_timeline_events(self, start, to, filters):
        # return a list of TimedEvent objects in the time span delimited
        # by the start and stop parameters, only containing events of the
        # types included in the filters list

    def get_timeline_filters(self):
        # return a list of (name,label) tuples that define the filters available
        # for this module. The ticket module might export ('ticket','Ticket changes'),
        # for example.

The wiki module would contribute to the timeline simply by appropriately implementing these functions. Same for any other module that wants to put events in the timeline.

The timeline module would use a convenience function in the Module base class to collect the events from all available plug-ins. That might look like the following:

    def process(self, req):
        # ...
        events = self.extensions.get_timeline_events(start, stop, filters)

Here, I'd like the 'extensions' object to be a somewhat magic proxy that would call a given function on all modules defining that function, and returning the individual results as a list.

A couple of drawbacks become visible here:

• The current timeline code is very efficient by basically consisting of a single SQL query over all the related tables. Under the plug-in model, we'd have at least one query per contributing module.
• Similarly, ordering of the events will need to be done in Python in the timeline module after it has collected all events.

This is probably just the price to pay for a cleaner architecture, but optimizations may be possible.

Other Aspects of Plug-Ins

Plug-ins will rarely consist solely of python code. Instead, they will almost always include templates, style sheets and images. I do not propose any kind of single-directory or even single-file deployment of plug-ins. Rather the user should be responsible for copying these files into the appropriate directories.

Another aspect not discussed yet are the database requirements of modules. The plug-in approach moves us away from the monolithic database approach currently in use. Individual plug-ins may need their own tables in the database. They may need to upgrade those tables separately from the rest of the database. Possibly using a per-table version number instead of a global database version number will suffice here.

Plug-ins should also be able to not only hook into the web-interface, but also into TracAdmin. The approach for this would be similar to the one discussed above, meaning that modules should be able to contribute commands.

Your Ideas Wanted

Comments, alternative approaches, constructive criticism, etc. Bring it on! —ChristopherLenz