= A Proposal for Pluggable Modules = Different projects, organizations or companies have different requirements for a tool such as Trac. For example, while some software development shops have some kind of system for automated nightly builds or continuous integration in place, many others do not. Those who do would like to integrate their build system with Trac, but adding such features to the core would inevitably result in enormous bloat, turning away other potential users who just don't need all the features. Arguably, this is already the case for people who want to, for example, use Trac as Wiki and issue tracker, but don't need the SCM integration -- for whatever reason. Adding new functional areas (aka modules) to Trac currently requires some heavy lifting, touching various parts of the code base. Extending Trac on specific sites will therefore quickly become a maintenance nightmare, because those customized and/or extended installs will not be easily upgradeable. For these reasons it would be highly desirable for the Trac project to provide support for ''pluggable modules'', so that non-core functionality could be maintained separately from Trac (at least to a certain degree). == Introduction == If functionality is supposed to be made pluggable (and thus optional), we need to figure out what functionality sits at the core of the system. And at that point you will probably already get different opinions. Arguably, you should be able to use Trac without deploying the issue tracker. Quite certainly, the dependency on Subversion should be made optional. The Wiki functionality is central to the system, but do Wiki pages as such and their management also belong to the core system? In an ideal architecture, there is as little core functionality as absolutely required. Anything that '''can''' be separated from the core system, '''should''' be separated, for a variety of (mostly obvious) reasons: * Loose coupling between the core system and the individual components allows the addition or modification of functionality with minimum impact on the code-base, thus improving stability and customizability. * It becomes more likely that individual functional units can be tested in isolation (unit tests). * The user has more control over what functionality is deployed to a site by choosing only those modules that are absolutely necessary. The [http://www.eclipse.org/ Eclipse] IDE is an example of a large application that is based almost entirely on individual plug-ins. As Eclipse is based on Java, the development team had a lot of hoops to jump through to create such a dynamic system with a statically typed, compiled language. For example, they rely on plugin descriptors that describe the runtime requirements, what other plug-ins it extends in what way, and what ''extension points'' the plug-in itself offers to other plug-ins. I suspect that much of this complexity could be done away with in Trac, while still adopting the basic model of plug-ins that can extend -- and can be extended -- by other plug-ins. In the concrete case of Trac, the [wiki:TracTimeline timeline] module rethought as a plug-in might look like this: * Adds a button to the navigation bar, and extends the URL namespace with the {{{/timeline}}} path. * Allows other plug-ins to add timed events to the timeline. Each timed event would have a title, a description and a URL. * In addition, plug-ins extending the timeline could establish "filters" that would allow the user to restrict the type of events displayed (such as you currently can choose from "Ticket changes", "Repository check-ins", etc.) So what '''is''' the core in such a system? For Eclipse, the answer is that the core is actually just the plug-in manager and a couple of essential interfaces that allow "communication" between plug-ins. In any other plug-in based system, the answer should be the same. So assuming this basic model of operation, let's move on to discussing ideas for how this might be implemented. == The Module Manager == There needs to be a central facility that manages the discovery and loading of plug-ins as well as the communication between them. We'll call this the {{{ModuleManager}}} for now. The module manager should probably be part of the environment, or in other words, for any environment their should be one module registry. Module discovery would presumably be based on a given paths (list of directories) that contain python modules. The registry class would find all classes that somehow identify themselves as Trac modules (for example by sub-classing a central {{{Module}}} class, or by defining some module-scope variable). There could also be a 'plugins' folder in Trac environments that would be added the path by default. ''An alternative and possibly more efficient but also more cumbersome approach would be to require the user to specify the "fully qualified" name of every additional module, for example in TracIni. Modules such as the timeline would be in the list by default, but could be disabled by removing them.'' Modules would get loaded as any other python module, but should also be able to hook into the initialization of the system via an "exported" function. This would for example be used by the Wiki module to build the list of names of existing wiki pages. == The Module Interface == In the current Trac architecture, modules are basically HTTP request processors that live for exactly one request. In the envisioned system, the modules should be able to outlive a request, and also participate in the request processing of other modules, so as to be able to contribute to their functionality. So instead of getting the request object as argument to their constructor, they'd have an interface like: {{{ class XxxModule(Module): def __init__(self, env): Module.__init__(self, env) # do whatever other initialization may be needed pass def process(self, req): # do whatever needs to be done pass }}} In an environment such as [wiki:TracStandalone tracd] or [wiki:TracModPython mod_python], the {{{process}}} method might be called concurrently on the same module object, so module implementations should not store request-specific data as instance variables, but rather pass such data through as parameters to other methods. ''Renaming the current "{{{render}}}" to "{{{process}}}" here simply because the latter sounds more appropriate.''' == Module Cooperation == As discussed above, a basic requirement is that the modules should be able to: * contribute functionality to other modules * provide hooks so that other modules can contribute to its functionality === By using Function Hooks === The supporting model for such cooperation between modules should be functions defined by the contributing plug-in, and called by the plug-in to which we contribute. For example, let's assume the timeline module documents the following hook functions (or ''extension points'' in Eclipse jargon): {{{ def get_timeline_events(self, start, to, filters): # return a list of TimedEvent objects in the time span delimited # by the start and stop parameters, only containing events of the # types included in the filters list def get_timeline_filters(self): # return a list of (name,label) tuples that define the filters available # for this module. The ticket module might export ('ticket','Ticket changes'), # for example. }}} The wiki module would contribute to the timeline simply by appropriately implementing these functions. Same for any other module that wants to put events in the timeline. The timeline module would use a convenience function in the {{{Module}}} base class to collect the events from all available plug-ins. That might look like the following: {{{ def process(self, req): # ... events = self.extensions.get_timeline_events(start, stop, filters) }}} Here, I'd like the 'extensions' object to be a somewhat magic ''proxy'' that would call a given function on all modules defining that function, and returning the individual results as a list. A couple of drawbacks become visible here: * The current timeline code is very efficient by basically consisting of a single SQL query over all the related tables. Under the plug-in model, we'd have at least one query per contributing module. ''Not necessarily: each contributing module could contribute the appropriate SQL fragment for the query, as well as a post-processing routine. The current timeline code could be easily refactored that way, without loosing its efficiency.'' * Similarly, ordering of the events will need to be done in Python in the timeline module after it has collected all events. ''See above.'' This is probably just the price to pay for a cleaner architecture, but optimizations may be possible. === By using ExtensionPoint Interfaces === Alternatively, a Module could define one or several client interfaces (all ExtensionPoint subclasses), and the other modules could provide zero, one or more implementations for this Client interface. This approach would provide both clarity and flexibility (if not even a simpler programming model) than the ''hook functions'' approach. See a more detailed description below '''Your Ideas Wanted / Modules and Configurability''' == Other Aspects of Plug-Ins == Plug-ins will rarely consist solely of python code. Instead, they will almost always include templates, style sheets and images. I do not propose any kind of single-directory or even single-file deployment of plug-ins. Rather the user should be responsible for copying these files into the appropriate directories. Another aspect not discussed yet are the database requirements of modules. The plug-in approach moves us away from the monolithic database approach currently in use. Individual plug-ins may need their own tables in the database. They may need to upgrade those tables separately from the rest of the database. Possibly using a per-table version number instead of a global database version number will suffice here. Plug-ins should also be able to not only hook into the web-interface, but also into TracAdmin. The approach for this would be similar to the one discussed above, meaning that modules should be able to contribute commands. == Your Ideas Wanted == Comments, alternative approaches, constructive criticism, etc. Bring it on! --ChristopherLenz === Trac Modules and Trac Objects === The approach of Pluggable Module described here would be complementary to the Trac Objects idea described here: TracObjectModelProposal. --ChristianBoos === Modules and Configurability === In some cases, one would like to have a fine control on how a module works. I'll take an example. The Timeline module interacts with the Ticket module. The latter provide several timeline entries to be displayed by the former, as well as specific filters for composing the filter panel. But let's say I would be interested in having the Ticket Contributions displayed (See #187 and my patch at #890), whereas others wouldn't want this option to be available. How to solve this? * Subclassing the Ticket module? (For the ''hook function approach'': not very flexible, what if there are multiple optional features) * A configuration flag for the Ticket module? (This would be more flexible, but, well, would be spaghetti code in the end!) * Have the Ticket module provide a (configurable) list of !TimelineEventProvider objects (the, or one of the ExtensionPoint interface that the Timeline module would offer)? The last idea seems the most interesting. The Timeline module expects that the other modules will provide zero or more !TimelineEventProvider object. The Ticket module would implement all of the following subclasses of !TimelineEventProvider class: * !NewAndCloseTicketsInTimeline * !ReopenedTicketsInTimeline * !TicketContributionsInTimeline After being configured (either from the {{{trac.ini}}} file, or any other mean, even dynamically!), the Ticket module could decide which instances of these subclasses will be given to the Timeline instance. The !TimelineEventProvider and ExtensionPoint could be defined as: {{{ #!python class ExtensionPoint: def module(self): """The module providing this client for the timeline""" pass ... class TimelineEventProvider(ExtensionPoint): def query_string(self): """The SQL fragment which provides: * time: the timestamp of the event * idata: the unique identifier for this event * ... and so on """ pass def render_item(self,item): """Populate the item dictionary with hdf data""" pass ... }}} --ChristianBoos == Identity Crisis? == Eclipse is a fantastic tool and an excellent example of a very impressive plug-in architecture... but, Eclipse is a "Rich Client Platform" - a universal tool. It wants to be everything to everyone. I think a better model for Trac to follow would be jEdit which has a wealth of plug-ins, but jEdit remains a text editor, not a plug-in manager with a text editing plug-in. (I believe my example is correct, I've never looked under the hood of jEdit). Trac, as defined in the logo at the top of this page, is an SCM Issue Tracker/Project Manager. It makes good design sense to have a modular architecture (as it is now) and a plug-in interface for non-core extensions, but to make everything a plug-in?? What would Trac become? It would be an empty shell that loads Python modules - but we already have Python to load Python modules. ''Not an empty shell that loads Python modules, but a shell designed to contain SCM-related modules, combined with a collection of useful modules focused around project management.'' I would say that the features available in Trac 0.8 are pretty close to what I would consider core functionality - but that is because I want the total package, one-stop shopping: SCM Issue Tracker/Project Management. I can spare the 50 or 100k of disk space per module, even if I don't use them. If a user doesn't want to use the TracRoadmap or the TracBrowser, etc - then why not have some boolean settings in TracIni to disable their appearance? Its a poor-man's way to eliminate "bloat", but it would keep the current "core" intact. ''What benefit is there to maintaining a high level of coupling between the core modules?'' I'm all for a plug-in interface for TracReleaselist and whatever anyone else can dream up (Anthill or CruiseControl plug-in, anyone?) Personally, I do not think that refactoring the core into individual plug-ins provides any real benefit. ''Refactoring the core would make it easier to abstract the frontend away from the backend, so that other frontends (an XML-RPC interface, say) can be constructed more easily.'' Developing a good plug-in extension interface for the existing core, however, would be very worthwhile. And to develop a good interface is going to be tricky enough. --JamesMoger == Re-read of proposal == What you've got so far sounds like a good start - although I still think the Module Manager should not be the top-level component, but a peer of timeline, roadmap, etc. and at that point perhaps a more appropriate name would be Plugin Manager. :) --James Moger