Multi-Project Support for Trac

The current design of Trac associates one project to one environment.

More often than not, software projects tend to split themselves in separate sub-projects, sharing common libraries and components. In enterprise setups, it is practical to consolidate the development knowledge base in one Wiki, yet to be able manage the different products or family of products independently. Moreover developers are often working on different products at once, and would like to be able to have an overview of the different projects they're involved in. At other times, the same developers would like to have the ability to fully focus on a single project. Widening the scope and looking at a whole family of products or even at the entire set of projects is also useful when one wants to get an overview or doesn't know where to start searching for an information.

Trac has to be adaptable to these different views. The current approach of setting up different Trac environments only allows to get a fixed view on one project. By using InterTrac links, it has been possible to relate resources one to another across Tracs, but the interaction and sharing of information couldn't go beyond linking: no common search or ticket queries, no shared timeline, no possibility to conveniently move resources from one project to the next while retaining the full history, etc.

In addition, for some database backends, the single project per environment design impose practical limits to the number of projects that can be run concurrently, due to connection sharing concerns (#8058).

The next step is therefore to provide the ability to create multiple Trac projects and sub-projects inside a single environment and to offer some convenient ways to form "views" on which projects should be active for a given request.

In our usual tradition of backward compatibility, when an environment only contains the toplevel project everything should continue to work like it always did. This will allow alternate implementations of multiple project setups to continue working unmodified (see the various TracMultipleProjects sub-pages, esp. TracMultipleProjects/SingleEnvironment which contains a proposal quite similar to this: adding a project field and a Projects table).

The Project Tree

A very natural way to group projects and sub-projects is to use a hierarchy of projects. The top-level project contains all the other projects, the leaf projects can focus on some appropriate subset of the development activity (e.g. a given library) and the intermediate project nodes in this the project tree are used to group related sub-projects together.

This approach accommodates with the simplest case of a the toplevel project being the only project, supports quite well the situation of having many independent projects by creating all projects as sub-projects of the toplevel project, but also scales to more complex development environments, e.g. where there are different families of products and possibly customer specific sub-projects for each of the products.

Here are some typical setups:

Single project

• "" or "/", the toplevel project

Several independent projects

• "" or "/", the toplevel project
  • "project_a"
  • "project_b"
  • "project_c"

Hierarchy of projects

• "" or "/", the toplevel project
  • "Family A"
    • "Family A/product a"
    • "Family A/product b"
  • "Family B"
    • "Family B/product c"
    • "Family B/product d"
  • "Tools"
  • "3rdParty"
    • …

The hierarchy doesn't limit the possible ways of grouping projects. In the example above, it should still be possible to search for tickets in "Family A" and "Family B" but not in "Tools" and "3rdParty" when needed, using a reasonably simple user interface.

However the hierarchy can help to intuitively select a meaningful subset of the existing projects. For example, a project selector user interface could allow for any subset of projects to be selected, yet provide a straightforward way to pick a project which becomes the center of interest. This means selecting not only that project, but also its ancestor projects and sub-projects.

For example, if that center of interest is set to "Family B" in the above example tree, then the selected projects will be the toplevel one, "Family B" itself, "product c" and "product b".

This means that when looking for tickets, one would see all the tickets from sub-projects (products c and d), but also tickets that address issues concerning the "Family B" as a whole (e.g. installer issues), and tickets that concerns all the projects (e.g. bugs in a base library used by every project).

In the end this is mainly a matter of user interface, and picking up any other subset of projects should be easily possible as well by just toggling a few checkboxes (among other things, it should also be possible to select only "Family B" without selecting its ancestor projects or sub-projects).

Example:

Project Selector

• Top-level
  • Family A
    • product a
    • product b
  • Family B
    • product c
    • product d
  • Tools
  • 3rdParty

The user interface could be as simple as changing the center of interest when clicking on the name of the project. Clicking on the checkboxes could be used to select/deselect projects on an individual basis.

On a different level, the hierarchy can also be used to organize resource sharing between projects. For example, in the Ticket system we could imagine that milestones belonging to ancestor projects are also visible in their sub-projects. In our hierarchy example above, imagine setting up a milestone for a "B-1" release in the "Family B" project. In sub-projects "product c" and "product d", no redundant milestone setup needs to be done, "product c" issues could be directly targeted to this "B-1" milestone.

In the typical center-of-interest setup, looking at the roadmap when "Family B/product_c" is the center of interest, the "B-1" milestone will show up and the tickets for "Family B/product_c", "Family B" and toplevel "/" will be shown. The tickets for "Family B/product_d" on the other hand won't be visible in that view.

Conversely, when "Family B" is the center of interest, the "B-1" milestone will show the tickets for "Family B/product_c" and "Family B/product_d" (all the sub-projects), "Family B" itself and the toplevel ones associated to this milestone.

In case the project selection is restricted to "Family B" alone (no ancestors, no sub-projects), then the "B-1" milestone will naturally only show the tickets belonging to the "Family B" project.

So we see that the current project selection is always used to filter the set of resources that are taken into account for any kind of data retrieval. The default project selection obtained by selecting a "center of interest" project (focal project?), consisting of the project itself, its ancestors and its sub-projects provides a view on all the resources which should be generally relevant for someone who is working in that project.

With this "center-of-interest" model, the users will also have a clear guideline for selecting the project in which they should create their resource (be it a ticket, wiki page, milestone, version, component, etc.): they should pick the project for which it makes sense to see the resource from all the project's sub-projects.

At this early stage of the proposal, this organization seems to be at the same time simple, intuitive yet very powerful.

The Data Model

The main idea is to attach a project identifier to each resource.

This can be achieved either by using an extra .project field for each resource or by parenting the resources in a project resource. The second way is probably better, as some resources might be project independent (like the Changeset and Source which belong to a given repository - see MultiRepos).

The empty string '' could be used to qualify the toplevel project, which is always there. If no other project has been defined, then Trac behaves as usual, one environment is one project. As soon as other projects are defined, the environment becomes a multi-project one.

Independently from the GenericTrac specification, the MultiProject changes to the data model are quite simple. The only backward incompatible addition needed is a project field to most of the resource related tables.

An additional project table is also needed, and it can take the form of a resource table specialization from the GenericTrac. Projects should have at least a description field, an owner and a status field. Projects can be linked explicitly to shared data. For example, each projects can be linked to its own set of repositories.

For resource specific SQL queries, the appropriate project identifier will have to be specified in addition to the realm (if needed) and id information.

For SQL queries that may cover more than one project, a helper method can be used to form the include/exclude constraints, given the project pattern list that has to be used.

On the API side, the multiproject nature will be mostly transparent except when instantiating and manipulating resources, which will have an additional project field. Legacy resources and legacy code not taking the project into account will simply fallback to accessing resources at the global level, which will work just fine in single project environments.

Web User Interface Changes

needs to be amended to take #ProjectTree into account

There should be a few standard ways to switch the project, either a pull-down menu or a tab-bar (in the style of edgewall.org).

The toplevel project should also be able to present a Projects main navigation entries, showing a list of the available projects, much similar to the list of Milestones.

An admin module can be used to create new projects (eventually copying settings from another one), renaming projects and deleting them.

Existing modules that lend themselves to cover multiple projects should be adapted and provide a standard way to specify multiple projects when accessed from the toplevel project. This can take the form of a list of checkboxes, a multiple selection field, or a text field for entering a pattern. When those modules are accessed within a dedicated project (parameter 'project'), they should behave similarly to the single project Trac, i.e. not offer the possibility to cover different projects. But when the project is the toplevel one (project == '') or there are already multiple projects specified, then the possibility to select different projects should be offered.

Utilities

Scripts for importing and exporting projects between environments should be provided. This can be useful for archiving a project before deleting it, for merging multiple single environment Tracs into one, etc.

URL space for different projects

Several kinds of mappings can be defined for mapping URLs to projects, before normal request dispatching takes place. The two default ones are:

• host mapping: the name of the (virtual) host used to access the Trac environment is taken to be the name of the project. A given host can be configured to be an alias for the default, top-level project.
• prefix mapping: if the PATH_INFO starts with the name of a project, this will be taken to be the name of a project. If none such project is detected, then the request is targeted at the top-level project.

In addition to a single project specification, there can sometimes also be multi-project specifications in a given URL. The request parameter 'project' can contain more than one project, and if the request processor can take advantage of this, the scope of the request will cover those projects. The special value '*' can be used to cover all the projects and more generally can be used in a glob pattern to cover different projects at once, and a '!' prefix for a project name or pattern can be used to exclude those projects. Typical users of multi-project specifications are the Timeline, the Search, the Custom Query…

Data Sharing

As briefly mentioned above, some data structure will be accessible globally and are shared between project. This will be the case for the Version Control Subsystem, where the repositories and the associated data are global objects that can be accessed indifferently from different projects. Of course, projects can define which repository they want to cover, eventually none or just one. They can also specify which repository will be the default one for them (the reponame to use when no reponame has been explicitly specified in links or queries).

Session and user data will also be shared across different environments, to some extent: preferences at a global level can be overridden by preferences set at the project level.

Permission will also be shared, though the system of fine-grained permission can be used to limit the permissions to a given project.

Workflow

The question came up whether workflow should be per-project, or per-project group… and how to reuse them (ticket:130#comment:157 and follow-up).

Having one workflow per project could quickly make the maintenance of workflows quite complex. To balance this complexity, in hierarchical setups, one could define a workflow at the upper level, and have sub-projects use the workflow defined closest in the hierarchy. For example, one could define a workflow on the root project, and every other project uses this. If one exception has to be defined for a given project, that project can define its own.

Alternatively (ticket:130#comment:159), workflows could be defined as independent entities, and a project will have to be associated to a given workflow (a bit like repositories will be associated to projects). Having workflows as separate entities would allow :

• sharing the same workflow between different projects, but still having the possibility to set another workflow for some specific projects;
• (might) allow (the user?) to specify explicitely how to manage moving a ticket from one installed workflow to another installed workflow - whatever the project;
• ease sharing workflows between users: having workflows encapsulated encourages implementation of several generic workflow solutions (that can be easily tweaked for specific use) that could be worked and provided by default or in a file repository - that would even ease to choose the starting workflow of a new TRAC user;