better support for multiple repositories

[anonymous]

Trac does not support multiple source repositories in a single Trac instance; this is limiting for some projects that have different branches is separate repositories.

Also, other version control systems has a notion of branch == repo == working directory, resulting in that Trac will only be able to follow a single branch.

To solve this probably a repo member has to be added to the revision and node_change tables. The Repository class also has to know about the repository name, so it can query the tables correctly.

[Christian Boos]

Keeping one repository per Trac environment has certainly its advantages:

• no ambiguity when referring to a changeset or a file
• no big changes to the versioncontrol API needed

Also, by using InterTrac links, it would be quite easy to refer to other Trac instances, and thus, to other branches.

However, there would be a big loss: the ability to diff between branches.

I would say it could be possible to extend the arbitrary diff interface (anydiff.cs in the TracDiff branch) to support diffing between branches/repositories.

So, while I understand the motivation for this ticket, improving the support of multiple, related repositories, the proposed approach is not the best. I would rather stick with one repository per environment, use the InterTrac feature when working on those environments and extend the TracDiff feature to be able to do comparisons across environments (InterDiff feature?).

[jrydberg@…]

I can not really agree that using InterTrac links would solve this the best way; but it might be a good short term solution.

Right now there exist 11 branches in the Trac SVN repo. That would mean 11 trac instances, which is both quite a waste of resources.

Regarding no ambiguity when referring to a changeset or a file; my initial idea was to add an extra indirection to changesets name. So instead of refering to [3] you refer to ![repo@3].

A short term solution for backends such as darcs and bzr is to add a "proxy repository" that behave as a single repo but really provide access to several repositories. I would rather note have to do this, though, since it can become quite ugly.

[eric-trac@…]

I'm not sure if bzr or darcs can do this, but I assume they can…

Just combine all the repositories together into one big repository that has multiple heads revisions. That way you can unambiguously refer to a particular changeset, and diff between branches.

I don't like the ![repo@3] idea that much as I think it makes the links more complicated which greatly reduces their utility.

[Christian Boos]

Actually, we discussed the Wiki syntax a while ago on IRC:

• there should be no problem with the source: link syntax, as the repository name could be used as a toplevel path component, e.g. source:repos1/dir1/fileA will link to path dir1/fileA within the repos1 repository
• as an extension to the above scheme, link to changesets could be written [repos1/<the_changeset>], in case there could be an ambiguity in the changeset identifiers. But generally, the better SCMs would provide some kind of changeset identifier that is repository independent, right? (at least Mercurial does it that way)
  In this situation, there wouldn't even be a need to disambiguate the changeset.

Also, there was the idea that the changes needed to support multiple repositories could actually be used to work on a compound view of a Subversion repository (the trunk/calc, branches/calc, tags/calc situation, which one would like to remap to simply trunk, branches and tags within a calc project, see #1830). In that case too, the changesets wouldn't need any extra qualifier.

[zooko@…]

Trac+darcs works okay.

​http://www.darcs.net/DarcsWiki/TracOnDarcs

[Christian Boos]

I just thought about a workaround that could work for the Mercurial backend:

In order to track multiple related repositories (i.e. clones),

one could create another clone (call it tracrepos), and from there, pull in the changes from all the other clones! This "meta"-repository would give access to all the existing branches, in all the clones.

The pull could either be done externally, by a cron job,

or attempted in the sync method of the cache (to be done during vc-refactoring phase 2).

Would a similar approach work for other backends as well?

[anonymous]

SVN supports multiple branches per Repository. Darcs does not.

Trac is expected to provide a whole project view with multiple branches. So it should (I would say needs to) implement this via multiple repos for the darcs plugin. The one repo restriction of Trac only makes sense with One Repo/Multiple Branches SCMSs like subversion.

For implementation, maybe just require darcs repos to live somewhere below a common base dir, and look for branch dirs (which are also proper darcs repos on their own, sharing some common revision in the past) recursively by looking for the _darcs subdir. The path from base to the repo/branch dir could be mapped to the branch name. That way, you might not need to store extra data (in core or whatever) about multiple repos, just append the branch name to the base dir to get the real repo dir whenever accessing a branch.

I have not looked at Trac code yet, I don't know how hard this would be, but this is kind of a "must have" for serious Trac deployment here. Not having it means Trac+Darcs is crippled in comparison to Trac+SVN.

[anonymous]

I have a need for multiple repositories because I need different security permissions for each project, but I'd like to have them share one Trac instance. I don't need to secure the bug reports, just the source code, so it would be nice to have a single Trac instance for all the repositories so that, e.g., I could see unified reporting across all modules/repositories.

[anonymous]

I have a number of developer subgroups that will each use TRAC+SVN repos for each package they develop. I'd like a rollup TRAC that covers the whole project. Each individual package has interwoven milestones, requirements, and dependencies that make project tracking at the individual package level very difficult. Having a top-level (project-level) TRAC is the obvious solution to this problem. Each subgroup could then follow the TRAC that they are concerned with and maintain the package TRACs with their own resources. The project-level TRAC would then pick up meta-data from each individual package-level TRAC, and track source code for multi-package applications. We may also depend on a third party partner that also uses TRAC+SVN, and they will also depend on us. It is a tight partnership, and I would expect we could use the project-level TRAC to recieve information from their project TRAC as well. It would be especially useful to see a project-level rollup in the timeline, milestones, and roadmap. I believe this is similar to the essence of what is being requested above — I just wanted to emphasize a key feature that I believe is missing in the current version of TRAC.

[anarcat@…]

This is interesting. I'm looking for ways to make Trac more "forge-like", basically to create some kind of community around multiple Trac environment on a single install.

For me, the default "listing of projects" when you access the root of the trac install could be a trac itself, with its own wiki, templates and bugtracker. It could have special plugins for tracking project and user activity, as gforge does for example.

I would be interested in finding ways of making this a reality in Trac. We would like to start a forge here, but I find gforge has a … well… charged history and is a bit bloated. I like the minimalist "no bullshit" approach trac has.

For me the main issues to make this work at this point are:

1. have a way for the account manager plugin to manage apache or authz_svn "groups", the same way it currently manages user accounts. this is critical to have credentials isolation from a project to the other
2. make the "root trac listing" be a bit more fleshed out. this can be either:
   1. by making that a mostly "static" (as in not modifiable without hacking the source) content that would display projects statistics
   2. by making that a complete trac install with custom plugins for activity

As for the original idea of the bug, I think a simple solution would be to allow InterWiki linking through the difference projects. Just prefix the usual links by the project name and you're done. So for example, you have 2 projects environment on your trac install: projecta and projectb.

• to link to a wiki page, you simply use: projecta:WikiStart or projectb:WikiStart
• to link to a bug: projecta:#349
• to link to a file: projecta:source:trunk/
• to link to a revision: projecta:[1234]
• etc…

Wikis have been networked like this forever and it works like a charm.

[telenieko@…]

adding my self.

[anonymous]

add myself to cc:

[Christian Boos]

Some related changes to the API are currently being discussed in this ​trac-dev thread.

The support for multiple repositories itself will probably happen in 0.12 only, sorry for the delays.

[anonymous]

It would also be nice to view tickets from all projects together in one list, as a developer.

[anonymous]

Could you please explain what commands you would run? I'm trying to merge disparate, unrelated repositories into one "meta" repository, but I have no idea what commands would actually make it happen.

No matter what I try, my parent repository never seems to be aware of the child repositories.

Replying to cboos:

I just thought about a workaround that could work for the Mercurial backend:

In order to track multiple related repositories (i.e. clones),

one could create another clone (call it tracrepos), and from there, pull in the changes from all the other clones! This "meta"-repository would give access to all the existing branches, in all the clones.

The pull could either be done externally, by a cron job,

or attempted in the sync method of the cache (to be done during vc-refactoring phase 2).

Would a similar approach work for other backends as well?

[Christian Boos]

Replying to anonymous:

Could you please explain what commands you would run? I'm trying to merge disparate, unrelated repositories into one "meta" repository

(preliminary note: this whole discussion is about Mercurial)

The idea I exposed above wouldn't work so well for unrelated repositories, as all they have in common is the initial null revision (you'll need to provide the -f option to hg pull, in order to force the pull). Browsing will be inconvenient, as you'll get to the tree corresponding to the repository you last pulled from.

Note that even for related repositories (cloned from each other) you'll have the same inconvenience for browsing, unless you're using named branches for the clones, then at least you'll have the quickjump feature for switching from one branch to the other.

At best this is a workaround waiting for a better general solution (VcRefactoring#SupportforMultipleRepositories).

Example: let's create two unrelated repositories a and b and artificially merge them in a third meta repository:

$ hg init a
$ cd a/
$ echo A > A
$ hg ci -Am "repo A"
adding A
$ cd ..

$ hg init b
$ cd b
$ echo B>B
$ mkdir c
$ echo b > c/a
$ echo b > c/b
$ hg ci -Am "repo B"
adding B
adding c/a
adding c/b
$ cd ..

$ hg init meta
$ ls
a/  b/  meta/
$ cd meta/
$ hg pull ../a
pulling from ../a
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
(run 'hg update' to get a working copy)
$ hg pull ../b
pulling from ../b
searching for changes
abort: repository is unrelated
$ hg pull -f ../b
pulling from ../b
searching for changes
warning: repository is unrelated
adding changesets
adding manifests
adding file changes
added 1 changesets with 3 changes to 3 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)

Now [trac] repository_dir can point to the meta repository and Trac will show changesets from a and b.

[anonymous]

added myself..

[anonymous]

In a corporate environment it is common that users working a particular project are not authorized to access all code. One way to handle this is to have several repositories (also it is never good to have to large repositories,better to partition them wisely into smaller ones).

When reading ​http://trac.edgewall.org/wiki/VcRefactoring#SupportforMultipleRepositories "use the cache as it is, merging all the repositories in a kind of virtual repository; the first component of the path would be the name of the repository."

I wonder how the access control would work if the caches is merged to one cache? In other words suggestion 2 or 3 looks better. I have been waiting a long time for this feature, look forward to it.

[sstarr]

cc: Add myself to this bug for tracking

[anonymous]

cc: Add myself to this bug for tracking

[ches]

This clearly doesn't apply to a general solution to the multiple repositories issue in Trac, but for the Mercurial discussion on this thread, the ​Forest extension may be worth a look.

It specifically addresses the composite- or meta-repository approach discussed here. I haven't tried it, much less use with TracMercurial, but it looks promising so I thought I'd point it out for others.

[mirko-trac@…]

I just tried the TracMercurial with the ​Forest extension and it did not work. The source browser correctly just finds no revisions in the parent repository.

[Christian Boos]

I know that someone is working on Forest support. Stay tuned, we'll update this ticket when there's something ready.

[anonymous]

cc: Add myself to this bug for tracking

[vadim]

cc: Add myself to this bug for tracking

[Christian Boos]

There's now a pretty complete first stab at this feature, described in MultipleRepositorySupport and available in the source:sandbox/multirepos branch.

If you're using Mercurial, you will also need a newer version of the plugin, available in source:sandbox/mercurial-plugin-0.12.

If you're using Subversion, be aware that only non-cached repositories are supported for now, i.e. you'll need to specify direct-svnfs as the repository type.

If you're using both… then no problem, you can now happily mix repository types in the same environment!

[Remy Blank]

#7723 tracks the implementation of the multirepos cache.

[richard@…]

Added CC. Thanks for all the great work!

[Christian Boos]

Getting closer…

[Fred McTaker <TakeyMcTaker@…>]

I'm glad to see progress is being made here. Thanks cboos! I'm just wondering — has that InterDiff feature mentioned above been implemented? I want to experiment with multi-server hosted Subversion repositories using tools like SVK and Pushmi, or even SVN-Mercurial converters. InterTrac and InterDiff seem like good solutions for possibly linking Tracs across multiple servers. Also, I like the ideas above about having a MasterTrac and a tree of SubTrac's. Some other tool for merging and diffing across Tracs could also be cool, possibly at the sqlite level, but I'm getting ahead of myself. The more flexible the better!

[Christian Boos]

The sandbox/multirepos branch landed on trunk in r9125.

There are no big changes expected for MultipleRepositorySupport until the 0.12 release, so this can now be considered feature complete.

[Christian Boos]

And of course, big thanks to Remy Blank, as without him this little experiment wouldn't have turned into reality ;-)

[Remy Blank]

Woohoo! Congratulations Christian! And thanks for your hard work!

(Small clarification: you had already done most of the work when I started hacking on Trac. The only thing I really brought to the branch (besides a few caching and permission fixes) was to push you regularly for merging ;-)

[mike.sherov@…]

Congrats on the merge. I can't wait till 0.12 is released with this!