A Proposal for Pluggable Modules

Different projects, organizations or companies have different requirements for a tool such as Trac. For example, while some software development shops have some kind of system for automated nightly builds or continuous integration in place, many others do not. Those who do would like to integrate their build system with Trac, but adding such features to the core would inevitably result in enormous bloat, turning away other potential users who just don't need all the features. Arguably, this is already the case for people who want to, for example, use Trac as Wiki and issue tracker, but don't need the SCM integration — for whatever reason.

Adding new functional areas (aka modules) to Trac currently requires some heavy lifting, touching various parts of the code base. Extending Trac on specific sites will therefore quickly become a maintenance nightmare, because those customized and/or extended installs will not be easily upgradeable.

For these reasons it would be highly desirable for the Trac project to provide support for pluggable modules, so that non-core functionality could be maintained separately from Trac (at least to a certain degree).

Introduction

If functionality is supposed to be made pluggable (and thus optional), we need to figure out what functionality sits at the core of the system. And at that point you will probably already get different opinions. Arguably, you should be able to use Trac without deploying the issue tracker. Quite certainly, the dependency on Subversion should be made optional. The Wiki functionality is central to the system, but do Wiki pages as such and their management also belong to the core system?

In an ideal architecture, there is as little core functionality as absolutely required. Anything that can be separated from the core system, should be separated, for a variety of (mostly obvious) reasons:

• Loose coupling between the core system and the individual components allows the addition or modification of functionality with minimum impact on the code-base, thus improving stability and customizability.
• It becomes more likely that individual functional units can be tested in isolation (unit tests).
• The user has more control over what functionality is deployed to a site by choosing only those modules that are absolutely necessary.

The ​Eclipse IDE is an example of a large application that is based almost entirely on individual plug-ins. As Eclipse is based on Java, the development team had a lot of hoops to jump through to create such a dynamic system with a statically typed, compiled language. For example, they rely on plugin descriptors that describe the runtime requirements, what other plug-ins it extends in what way, and what extension points the plug-in itself offers to other plug-ins. I suspect that much of this complexity could be done away with in Trac, while still adopting the basic model of plug-ins that can extend — and can be extended — by other plug-ins.

In the concrete case of Trac, the timeline module rethought as a plug-in might look like this:

• Adds a button to the navigation bar, and extends the URL namespace with the /timeline path.
• Allows other plug-ins to add timed events to the timeline. Each timed event would have a title, a description and a URL.
• In addition, plug-ins extending the timeline could establish "filters" that would allow the user to restrict the type of events displayed (such as you currently can choose from "Ticket changes", "Repository check-ins", etc.)

So what is the core in such a system? For Eclipse, the answer is that the core is actually just the plug-in manager and a couple of essential interfaces that allow "communication" between plug-ins. In any other plug-in based system, the answer should be the same.

So assuming this basic model of operation, let's move on to discussing ideas for how this might be implemented.

The Plug-in Manager

There needs to be a central facility that manages the discovery and loading of plug-ins as well as the communication between them. We'll call this the PluginManager for now.

The plug-in manager should probably be part of the environment, or in other words, for any environment there should be one plug-in manager. Plug-in discovery would presumably be based on a given path (list of directories) that contain python modules. The PluginManager class would collect all classes that somehow identify themselves as Trac plug-ins (for example by sub-classing a central Plugin class). There could also be a 'plugins' folder in Trac environments that would be added the path by default.

The Plug-in Interface

In the current Trac architecture, modules are basically HTTP request processors that live for exactly one request. In the envisioned system, plug-ins should be able to outlive a request, and also participate in the request processing of other plug-ins, so as to be able to contribute to their functionality.

So instead of getting the request object as argument to their constructor, they'd have an interface like:

class XxxPlugin(Plugin):

    def __init__(self, env):
        Module.__init__(self, env)
        # do whatever other initialization may be needed
        pass

    def processRequest(self, req):
        # do whatever needs to be done
        pass

In an environment such as tracd or mod_python, the processRequest method might be called concurrently on the same module object, so plug-in implementations should not store request-specific data as instance variables, but rather pass such data through as parameters to other methods.

Plug-in Cooperation

As discussed above, a basic requirement is that the modules should be able to:

• contribute functionality to other modules
• provide hooks so that other modules can contribute to its functionality

For this to work, plug-ins can declare their own extension points and also declare themselves as extending one or more extension points of other plug-ins:

from protocols import *
from trac.plugin import *

import time

class TimelinePlugin(Plugin):

    _name = 'timeline'
    eventProviders = ExtensionPoint(IEventProvider)

    def processRequest(self, req):
        if not filters:
            filters = []
            for provider in self.eventProviders:
                filters += provider.getTimelineFilters()
        events = []
        for provider in self.eventProviders:
            events += provider.getTimelineEvents(start, stop, filters)

        # render the page etc.


class ChangesetPlugin(Plugin):

    _name = 'changeset'
    _extends = ['timeline.eventProviders']
    advise(instancesProvide=[timeline.IEventProvider])

    def getTimelineEvents(self, start, stop, filters):
        if 'changesets' in filters:
            return [timeline.Event(time.time(), 'Changeset [1]'),
                    timeline.Event(time.time(), 'Changeset [2]')]

    def getTimelineFilters(self):
        return ['changesets']

In this example, the TimelinePlugin declares the extension point eventProviders. Extensions to that extension point must support the interface IEventProvider (not shown here). This code currently uses ​PyProtocols to provide a flexible mechanism for interfaces and adaptation. The ​zope.interface package has been brought up as an alternative that we're looking into.

The ChangesetPlugin implements the IEventProvider interface with the methods getTimelineEvents() and getTimelineFilters().

The interesting piece is in TimelinePlugin.processRequest(): Here we can access a "magic" attribute with the same name as the extension point we declared, and iterate over it to access the individual plugins that extend it. Note: the plugins are automatically adapted to the required interface before being returned, so the plugin class does not have to implement the interface itself, but could also register an apprioprate adapter.

A couple of drawbacks become visible here:

• The current timeline code is very efficient by basically consisting of a single SQL query over all the related tables. Under the plug-in model, we'd have at least one query per contributing module. Not necessarily: each contributing module could contribute the appropriate SQL fragment for the query, as well as a post-processing routine. The current timeline code could be easily refactored that way, without loosing its efficiency.
• Similarly, ordering of the events will need to be done in Python in the timeline module after it has collected all events. See above.

This is probably just the price to pay for a cleaner architecture, but optimizations may be possible.

Other Aspects of Plug-Ins

Plug-ins will rarely consist solely of python code. Instead, they will almost always include templates, style sheets and images. I do not propose any kind of single-directory or even single-file deployment of plug-ins. Rather the user should be responsible for copying these files into the appropriate directories.

Another aspect not discussed yet are the database requirements of modules. The plug-in approach moves us away from the monolithic database approach currently in use. Individual plug-ins may need their own tables in the database. They may need to upgrade those tables separately from the rest of the database. Possibly using a per-table version number instead of a global database version number will suffice here.

Plug-ins should also be able to not only hook into the web-interface, but also into TracAdmin. The approach for this would be similar to the one discussed above, meaning that modules should be able to contribute commands.

Status

Previous snapshots of this code have been added as attachments to this page, but the "real" work has now started on the new rearch branch. That code is fully functional with the following exceptions:

• The mod_python handler and tracd are most likely broken, only way to use the branch is CGI.
• Sessions won't work.

What we do have:

• A working plug-in system as described above, minus the configuration of which plug-ins are to be loaded.
• A new (and as of yet incomplete) abstraction layer between the Trac web-application and the web-server. Unlike the current trac.core.Request class, it provides separate request and response objects. It is based on ​WSGI with a simple adapter for CGI. This doesn't really have much to do with the plug-in system, it's just another refactoring that I'm playing with in the same sandbox.
• A new abstraction layer on top of the ClearSilver templating engine. It allows populating the HDF using native Python data structures such as lists and dicts.
• ClearSilver templating is a plug-in (trac/web/clearsilver.py), request dispatching is a plug-in (trac/web/dispatcher.py) and building the web-site chrome (e.g. the navigation) is another plug-in (trac/web/chrome.py). Hooking into HTTP authentication is also a plug-in (trac/web/ext_auth.py). The idea with the latter is that we could provide an alternative plug-in that would do form-based authentication.
• A plug-in that runs the current set of modules in a compatability layer (trac/plugins/compat.py), providing them the environment they expect.

None of the actual modules have been migrated to plug-ins yet, they all run under the compatibility layer. They will be migrated one after another in the next couple of days/weeks.

To use the CGI, configure your web-server the same way as for a regular Trac instance. The CGI script is in cgi-bin/trac.cgi just like before. You will however need to add the following snippet to your trac.ini file:

[web]
default = compat
filters = external_auth, clearsilver, compat, chrome

Here, default tells the request dispatcher which plug-in to use on a request to the root URL (i.e. SCRIPT_NAME without further PATH_INFO). It basically designates the welcome page. filters determines the order in which the request filters will be run (so yes, order does matter — a lot).

You will also need to have ​PyProtocols installed.

Your Ideas Wanted

Comments, alternative approaches, constructive criticism, etc. Bring it on! —ChristopherLenz

Trac Modules and Trac Objects

The approach of Pluggable Module described here would be complementary to the Trac Objects idea described here: TracObjectModelProposal.

—ChristianBoos

I don't see how the two ideas are complementary, other than both are refactoring discussions. —ChristopherLenz

Identity Crisis?

Eclipse is a fantastic tool and an excellent example of a very impressive plug-in architecture… but, Eclipse is a "Rich Client Platform" - a universal tool. It wants to be everything to everyone. I think a better model for Trac to follow would be jEdit which has a wealth of plug-ins, but jEdit remains a text editor, not a plug-in manager with a text editing plug-in. (I believe my example is correct, I've never looked under the hood of jEdit).

Eclipse is first and foremost an IDE featuring an extremely modular architecture (the whole RCP thing was slapped on much later). Whether you call the pieces of the software "components", "modules" or "plug-ins" doesn't really matter. Anyway, I might have taken the analogy with Eclipse too far, please don't let that distract you from the rest of the message. —ChristopherLenz

Trac, as defined in the logo at the top of this page, is an SCM Issue Tracker/Project Manager. It makes good design sense to have a modular architecture (as it is now) and a plug-in interface for non-core extensions, but to make everything a plug-in?? What would Trac become? It would be an empty shell that loads Python modules - but we already have Python to load Python modules.

Not an empty shell that loads Python modules, but a shell designed to contain SCM-related modules, combined with a collection of useful modules focused around project management.

The way I see it, Trac would be the sum of the modules that make up the functionality we have now. The fact that the actual core would just consist of a plug-in manager and request dispatcher is an implementation detail that serves to make the architecture more modular. It should have no immediate effect on the user or administrator apart from allowing more flexible deployments. —ChristopherLenz

I would say that the features available in Trac 0.8 are pretty close to what I would consider core functionality - but that is because I want the total package, one-stop shopping: SCM Issue Tracker/Project Management. I can spare the 50 or 100k of disk space per module, even if I don't use them.

If a user doesn't want to use the TracRoadmap or the TracBrowser, etc - then why not have some boolean settings in TracIni to disable their appearance? Its a poor-man's way to eliminate "bloat", but it would keep the current "core" intact.

What benefit is there to maintaining a high level of coupling between the core modules?

I'm all for a plug-in interface for TracReleaselist and whatever anyone else can dream up (Anthill or CruiseControl plug-in, anyone?) Personally, I do not think that refactoring the core into individual plug-ins provides any real benefit.

Refactoring the core would make it easier to abstract the frontend away from the backend, so that other frontends (an XML-RPC interface, say) can be constructed more easily.

Developing a good plug-in extension interface for the existing core, however, would be very worthwhile. And to develop a good interface is going to be tricky enough.

—JamesMoger

Re-read of proposal

What you've got so far sounds like a good start - although I still think the Module Manager should not be the top-level component, but a peer of timeline, roadmap, etc. and at that point perhaps a more appropriate name would be Plugin Manager. :)

—James Moger

James, the idea I was trying to get across with this proposal is that modularizing the current core of Trac, while allowing the mechanism to be used for plugging in new or custom functionality, would probably be easier and than developing a well designed plug-in interface, and come with more benefits. —ChristopherLenz