A Proposal for Pluggable Modules

Different projects, organizations or companies have different requirements for a tool such as Trac. For example, while some software development shops have some kind of system for automated nightly builds or continuous integration in place, many others do not. Those who do would like to integrate their build system with Trac, but adding such features to the core would inevitably result in enormous bloat, turning away other potential users who just don't need all the features. Arguably, this is already the case for people who want to, for example, use Trac as Wiki and issue tracker, but don't need the SCM integration — for whatever reason.

Adding new functional areas (aka modules) to Trac currently requires some heavy lifting, touching various parts of the code base. Extending Trac on specific sites will therefore quickly become a maintenance nightmare, because those customized and/or extended installs will not be easily upgradeable.

For these reasons it would be highly desirable for the Trac project to provide support for pluggable modules, so that non-core functionality could be maintained separately from Trac (at least to a certain degree).

Introduction

If functionality is supposed to be made pluggable (and thus optional), we need to figure out what functionality sits at the core of the system. And at that point you will probably already get different opinions. Arguably, you should be able to use Trac without deploying the issue tracker. Quite certainly, the dependency on Subversion should be made optional. The Wiki functionality is central to the system, but do Wiki pages as such and their management also belong to the core system?

In an ideal architecture, there is as little core functionality as absolutely required. Anything that can be separated from the core system, should be separated, for a variety of (mostly obvious) reasons:

• Loose coupling between the core system and the individual components allows the addition or modification of functionality with minimum impact on the code-base, thus improving stability and customizability.
• It becomes more likely that individual functional units can be tested in isolation (unit tests).
• The user has more control over what functionality is deployed to a site by choosing only those modules that are absolutely necessary.

The ​Eclipse IDE is an example of a large application that is based almost entirely on individual plug-ins. As Eclipse is based on Java, the development team had a lot of hoops to jump through to create such a dynamic system with a statically typed, compiled language. For example, they rely on plugin descriptors that describe the runtime requirements, what other plug-ins it extends in what way, and what extension points the plug-in itself offers to other plug-ins. I suspect that much of this complexity could be done away with in Trac, while still adopting the basic model of plug-ins that can extend — and can be extended — by other plug-ins.

In the concrete case of Trac, the timeline module rethought as a plug-in might look like this:

• Adds a button to the navigation bar, and extends the URL namespace with the /timeline path.
• Allows other plug-ins to add timed events to the timeline. Each timed event would have a title, a description and a URL.
• In addition, plug-ins extending the timeline could establish "filters" that would allow the user to restrict the type of events displayed (such as you currently can choose from "Ticket changes", "Repository check-ins", etc.)

So what is the core in such a system? For Eclipse, the answer is that the core is actually just the plug-in manager and a couple of essential interfaces that allow "communication" between plug-ins. In any other plug-in based system, the answer should be the same.

So assuming this basic model of operation, let's move on to discussing ideas for how this might be implemented.

The Module Manager

There needs to be a central facility that manages the discovery and loading of plug-ins as well as the communication between them. We'll call this the ModuleManager for now.

The module manager should probably be part of the environment, or in other words, for any environment their should be one module registry. Module discovery would presumably be based on a given paths (list of directories) that contain python modules. The registry class would find all classes that somehow identify themselves as Trac modules (for example by sub-classing a central Module class, or by defining some module-scope variable). There could also be a 'plugins' folder in Trac environments that would be added the path by default.

An alternative and possibly more efficient but also more cumbersome approach would be to require the user to specify the "fully qualified" name of every additional module, for example in TracIni. Modules such as the timeline would be in the list by default, but could be disabled by removing them.

Modules would get loaded as any other python module, but should also be able to hook into the initialization of the system via an "exported" function. This would for example be used by the Wiki module to build the list of names of existing wiki pages.

The Module Interface

In the current Trac architecture, modules are basically HTTP request processors that live for exactly one request. In the envisioned system, the modules should be able to outlive a request, and also participate in the request processing of other modules, so as to be able to contribute to their functionality.

So instead of getting the request object as argument to their constructor, they'd have an interface like:

class XxxModule(Module):

    def __init__(self, env):
        Module.__init__(self, env)
        # do whatever other initialization may be needed
        pass

    def process(self, req):
        # do whatever needs to be done
        pass

In an environment such as tracd or mod_python, the process method might be called concurrently on the same module object, so module implementations should not store request-specific data as instance variables, but rather pass such data through as parameters to other methods.

Renaming the current "render" to "process" here simply because the latter sounds more appropriate.

Module Cooperation

As discussed above, a basic requirement is that the modules should be able to:

• contribute functionality to other modules
• provide hooks so that other modules can contribute to its functionality

By using Function Hooks

The supporting model for such cooperation between modules should be functions defined by the contributing plug-in, and called by the plug-in to which we contribute. For example, let's assume the timeline module documents the following hook functions (or extension points in Eclipse jargon):

    def get_timeline_events(self, start, to, filters):
        # return a list of TimedEvent objects in the time span delimited
        # by the start and stop parameters, only containing events of the
        # types included in the filters list

    def get_timeline_filters(self):
        # return a list of (name,label) tuples that define the filters available
        # for this module. The ticket module might export ('ticket','Ticket changes'),
        # for example.

The wiki module would contribute to the timeline simply by appropriately implementing these functions. Same for any other module that wants to put events in the timeline.

The timeline module would use a convenience function in the Module base class to collect the events from all available plug-ins. That might look like the following:

    def process(self, req):
        # ...
        events = self.extensions.get_timeline_events(start, stop, filters)

Here, I'd like the 'extensions' object to be a somewhat magic proxy that would call a given function on all modules defining that function, and returning the individual results as a list.

A couple of drawbacks become visible here:

• The current timeline code is very efficient by basically consisting of a single SQL query over all the related tables. Under the plug-in model, we'd have at least one query per contributing module. Not necessarily: each contributing module could contribute the appropriate SQL fragment for the query, as well as a post-processing routine. The current timeline code could be easily refactored that way, without loosing its efficiency.
• Similarly, ordering of the events will need to be done in Python in the timeline module after it has collected all events. See above.

This is probably just the price to pay for a cleaner architecture, but optimizations may be possible.

See the attachment moduly.py (at the bottom of this page) for an example how this might work. Note that the example uses new-style Python classes and meta-classes, and as such requires at least Python 2.2. I've only tested it on Python 2.3 myself.

By using ExtensionPoint Interfaces

Alternatively, a Module could define one or several client interfaces (all ExtensionPoint subclasses), and the other modules could provide zero, one or more implementations for this Client interface.

This approach would provide both clarity and flexibility (if not even a simpler programming model) than the hook functions approach.

See a more detailed description below Your Ideas Wanted / Modules and Configurability

Other Aspects of Plug-Ins

Plug-ins will rarely consist solely of python code. Instead, they will almost always include templates, style sheets and images. I do not propose any kind of single-directory or even single-file deployment of plug-ins. Rather the user should be responsible for copying these files into the appropriate directories.

Another aspect not discussed yet are the database requirements of modules. The plug-in approach moves us away from the monolithic database approach currently in use. Individual plug-ins may need their own tables in the database. They may need to upgrade those tables separately from the rest of the database. Possibly using a per-table version number instead of a global database version number will suffice here.

Plug-ins should also be able to not only hook into the web-interface, but also into TracAdmin. The approach for this would be similar to the one discussed above, meaning that modules should be able to contribute commands.

Your Ideas Wanted

Comments, alternative approaches, constructive criticism, etc. Bring it on! —ChristopherLenz

Trac Modules and Trac Objects

The approach of Pluggable Module described here would be complementary to the Trac Objects idea described here: TracObjectModelProposal.

—ChristianBoos

I don't see how the two ideas are complementary, other than both are refactoring discussions. —ChristopherLenz

Modules and Configurability

In some cases, one would like to have a fine control on how a module works. I'll take an example. The Timeline module interacts with the Ticket module. The latter provide several timeline entries to be displayed by the former, as well as specific filters for composing the filter panel.

But let's say I would be interested in having the Ticket Contributions displayed (See #187 and my patch at #890), whereas others wouldn't want this option to be available. How to solve this?

• Subclassing the Ticket module? (For the hook function approach: not very flexible, what if there are multiple optional features)
• A configuration flag for the Ticket module? (This would be more flexible, but, well, would be spaghetti code in the end!)
• Have the Ticket module provide a (configurable) list of TimelineEventProvider objects (the, or one of the ExtensionPoint interface that the Timeline module would offer)?

The last idea seems the most interesting. The Timeline module expects that the other modules will provide zero or more TimelineEventProvider object.

The Ticket module would implement all of the following subclasses of TimelineEventProvider class:

• NewAndCloseTicketsInTimeline
• ReopenedTicketsInTimeline
• TicketContributionsInTimeline

After being configured (either from the trac.ini file, or any other mean, even dynamically!), the Ticket module could decide which instances of these subclasses will be given to the Timeline instance.

The TimelineEventProvider and ExtensionPoint could be defined as:

class ExtensionPoint:
    def module(self): 
        """The module providing this client for the timeline"""
        pass
    ...

class TimelineEventProvider(ExtensionPoint):
    def query_string(self):
        """The SQL fragment which provides:
 * time: the timestamp of the event
 * idata: the unique identifier for this event
 * ... and so on
"""
        pass

    def render_item(self,item):
        """Populate the item dictionary with hdf data"""
        pass
    ...

—ChristianBoos

I don't see the original problem, to be honest. If you wanted to add a plug-in that adds e.g. ticket comments to the timeline, just do that. No need to subclass the Ticket module, or add a configuration flag. Just add a new plug-in:

class TicketCommentsInTimeline(Module):

    def get_timeline_events(self, start, to, filters):
        if 'comments' in filters:
            # get all comments from the ticket_change table and return them
        return []

    def get_timeline_filters(self):
        return [ ('comments', 'Ticket comments') ]

This is, in my humble opinion, a lot simpler than adding typed extension points and what not.

Identity Crisis?

Eclipse is a fantastic tool and an excellent example of a very impressive plug-in architecture… but, Eclipse is a "Rich Client Platform" - a universal tool. It wants to be everything to everyone. I think a better model for Trac to follow would be jEdit which has a wealth of plug-ins, but jEdit remains a text editor, not a plug-in manager with a text editing plug-in. (I believe my example is correct, I've never looked under the hood of jEdit).

Eclipes is first and foremost an IDE featuring an extremely modular architecture (the whole RCP thing was slapped on much later). Whether you call the pieces of the software "components", "modules" or "plug-ins" doesn't really matter. Anyway, I might have taken the analogy with Eclipse too far, please don't let that distract you from the rest of the message. —ChristopherLenz

Trac, as defined in the logo at the top of this page, is an SCM Issue Tracker/Project Manager. It makes good design sense to have a modular architecture (as it is now) and a plug-in interface for non-core extensions, but to make everything a plug-in?? What would Trac become? It would be an empty shell that loads Python modules - but we already have Python to load Python modules.

Not an empty shell that loads Python modules, but a shell designed to contain SCM-related modules, combined with a collection of useful modules focused around project management.

The way I see it, Trac would be the sum of the modules that make up the functionality we have now. The fact that the actual core would just consist of a plug-in manager and request dispatcher is an implementation detail that serves to make the architecture more modular. It should have no immediate effect on the user or administrator apart from allowing more flexible deployments. —ChristopherLenz

I would say that the features available in Trac 0.8 are pretty close to what I would consider core functionality - but that is because I want the total package, one-stop shopping: SCM Issue Tracker/Project Management. I can spare the 50 or 100k of disk space per module, even if I don't use them.

If a user doesn't want to use the TracRoadmap or the TracBrowser, etc - then why not have some boolean settings in TracIni to disable their appearance? Its a poor-man's way to eliminate "bloat", but it would keep the current "core" intact.

What benefit is there to maintaining a high level of coupling between the core modules?

I'm all for a plug-in interface for TracReleaselist and whatever anyone else can dream up (Anthill or CruiseControl plug-in, anyone?) Personally, I do not think that refactoring the core into individual plug-ins provides any real benefit.

Refactoring the core would make it easier to abstract the frontend away from the backend, so that other frontends (an XML-RPC interface, say) can be constructed more easily.

Developing a good plug-in extension interface for the existing core, however, would be very worthwhile. And to develop a good interface is going to be tricky enough.

—JamesMoger

Re-read of proposal

What you've got so far sounds like a good start - although I still think the Module Manager should not be the top-level component, but a peer of timeline, roadmap, etc. and at that point perhaps a more appropriate name would be Plugin Manager. :)

—James Moger

James, the idea I was trying to get across with this proposal is that modularizing the current core of Trac, while allowing the mechanism to be used for plugging in new or custom functionality, would probably be easier and than developing a well designed plug-in interface, and come with more benefits. —ChristopherLenz