A Comprehensive Multi-Project Solution

Abstract: we present a comprehensive solution for many of the problems of maintaining or participating in multiple projects under Trac. We briefly analyze some of the other approaches and explain why we're not using them.

This method supports:

• Cross-project ticket queries. Find out what your most important issues are in one swell foop.

• Single sign-on. Login once and skip around between Trac instances at will (subject to per-instance permissions, of course).

• Central configuration. You'll no longer have to edit multiple trac.ini files just to make a configuration change across many Trac instances.

• Common permissions. User permissions can be granted across all projects or on a per-project basis [this is misphrased; I haven't figured out how to say what I actually mean precisely, but it's good]

Note: this solution obviously owes much to the work of others; I'm not claiming to be an innovator here, just an integrator.

Caveats

• This page is a work in progress. I've solved the technical issues on my local development server and am composing this page as I port my real Trac instances over to the new system.
  • Just a quick feedback to mention that at this point, what your solution is isn't immediately clear ;-). From the requirements section below it appears that it is based on TracForge, so maybe this should be made more clear from the start. Note that TracForge itself is probably implementing something quite close to what is discussed in TracMultipleProjects/MultipleEnvironments.
    — cboos
  • Yes, I know. As I said, "work in progress," which I hope to have finished in the next few days. I'm actually combining stuff from TracForge, various other plugins from TracHacks, stuff from TracMultipleProjects/MultipleEnvironmentsSingleDatabase, and some of my own code. The approach for supporting cross-project queries is entirely different from what TracForge does, so characterizing it as "based on TracForge" won't really tell people what it is.
    — dave
  • If you can tell me more than 3 persons who could understand how tracforge did this i pay you a beer, and based on tracforge is confusing. otherwise not … —ThurnerRupert.

• This method requires the use of not-yet-released versions of Trac, ​Genshi, and several plugins, available from those projects' Subversion repositories.

• This approach isn't going to work for everybody. Ticket #130 contains a long, rambling, and extremely enlightening discussion of what people need from multi-project support. Anyone who has read that thread, and the other pages proposing different ways to support multiple projects, knows that no single approach is going to satisfy everone. Nonetheless, I think I've hit a "sweet spot" that gives most people most of what they're looking for, and more importantly, can be (relatively) easily set up today by anyone who is willing to use bleeding-edge Trac software and my choice of database technology.

Basic Strategy

• Use ​PostGreSQL to collect all projects into a single database using separate PostGreSQL "schemas."
• Use ​trachacks:MultipleProjectQueryFilterPlugin to fix up urls in those query results to point to the right tickets.
• Use ​TracForge to get:
  • Single Sign-On
  • An improved project list
• …

Requirements

• My port of the ​trachacks:TracForgePlugin to Trac 0.11dev (​svn, ​browse).
• ​Genshi is required for Trac-0.11
• ​ClearSilver is no longer used by Trac-0.11 since the switch to Genshi. However, it contains a ClearSilver compatibility layer and the TracForge plugin hasn't been ported to Genshi yet.
• ​PostGreSQL
• If you are currently using SQLite, You'll need ​trachacks:wiki:SqliteToPgScript with ​this patch.

Niceties

• ​Pygments is the new default syntax highlighter for Trac, but you may need to install it separately.

Motivation

When you host multiple trac projects, its very common to have users that are members of more than one of these projects. If, like ​me, you're using your Tracs to support multiple customers, you probably need to be a member of each trac instance. Even when using a TRAC_ENV_PARENT_DIR setup, one typically ends up having to log in to three or four Trac instances a day as you review your most active projects. Just having to log in over and over is problematic in itself. Worse, if you're tracking tickets in several projects, there's no way to see them all together in stock Trac. You actually need to log into each project and check your tickets there, and it becomes very likely that you'll miss something really important.

The other problem is that, when administering multiple similar tracs, you end up repeating the same configuration steps over and over. The tedious jobs include:

• creation and initialization of new trac instances
• setting up common user permissions
• making trac.ini adjustments across all projects

What's Wrong With Sharing a Single Trac?

There's been some work on managing multiple projects within a single Trac instance, but Trac can't yet give my customers the privacy they need in a shared environment. Sometimes we're working on proprietary software that must be kept private, but even when we're working on open source, customers generally feel uncomfortable when all their issues are automatically exposed to the world.

There are at least two areas that need to be improved in Trac to make a shared Trac workable:

• TracDev/SecurityBranch (which, despite its name, is on Trac's trunk now) needs to be extended to cover Trac resources other than Wiki pages.
• We need a flexible and automatic way to attach these permissions to resources upon creation. In my usage model, when a customer enters a ticket, it should be visible to and writable by everyone in his company and everyone in my company, but nobody else. Also, I occasionally need to create a ticket myself, with those same properties, and assign it to the customer. These capabilities are outside the scope of TracDev/SecurityBranch, so they need to be addressed separately. The ​Private Tickets Plugin can do the first part of the job for me (using the old permissions system), but not the second. See also #1316.

Naturally, the same kinds of issues apply to other resources such as Milestones. Upon creation by a customer, they need to be private to that customer and (some subset of) my people. When I create these resources there needs to be an easy way to make them private to a particular customer's company. And I'm sure that other people will have vastly different "permission workflow" requirements than I do, so the system probably needs to be more flexible than what I've described.

What's Wrong With Previous Attempts to Leverage PostGreSQL Schemas?

TracMultipleProjects/MultipleEnvironmentsSingleDatabase describes two approaches to using PostGreSQL schemas for doing multi-project reports:

• The first (main) approach probably works, but it fundamentally changes the layout of the database, so trac-admin upgrade will fail, maybe even spectacularly. That wasn't acceptable to me.

• The second (alternative) approach is much simpler, but it also has a fatal flaw: although you can get a report, clicking any of the tickets still takes you to the ticket with the same ID as the ticket you clicked, but in the current project.

What's Wrong With TracForge Ticket Subscriptions?

​TracForge contains a featuere called "subscriptions" that replicates data from one trac instance to another. One could try to use that to make all those tickets available in a master project where they can be queried. However,

1. Once the subscription is established, tickets are only replicated across Trac instances when new tickets are created. That means I'm out-of-luck when it comes to querying across my pre-existing Trac instances.
2. All that data replication is wasteful (this is minor) and probably slow (could be significant).
3. This approach has basically the same flaw as the alternative approach above using PostGreSQL, just one level removed: yes, if you click on a ticket in the report, it takes you to a ticket with the right content… but that ticket is still in the wrong context… so any project-local links to wiki pages changesets, etc. that are embedded in the ticket will take you to the wrong place. One could try to patch the subscription code to fix them up, but in my opinion that is a hopeless fight. To handle all the possible wiki syntax, you'd have to do a reverse translation for all the macros, plugins, etc. that introduce new wiki syntax. There are just too many to possibly account for all of them, and trying to keep up with new ones would be a maintenance nightmare.