Maintain git mirror (on GitHub?)

[jake@…]

While Trac is already a very feature-complete system for my use and its plugin support is very robust, I find myself wanting to modify core features. If the subversion repository was mirrored on a socially orientated DVCS site (such as the great GitHub) it would allow users like myself to fork and maintain our own flavors of Trac, porting in features from the official repository as they come.

There are an increasing amount of projects which are mirroring their internally hosted git and subversion repositories on GitHub (see ​http://github.com/apache/ fpr an example). Mirroring subversion repositories with git is an extremely easy process and could be fully automated after a small setup.

1. Run git svn clone http://svn.edgewall.com/repos/trac/
2. Run git gc to shrink the space of the git repo by A LOT.
3. Copy the .git folder from the newly created Trac folder to Trac.git in the parent directory and then delete the Trac folder. You now have a bare git mirror of the svn repository.
4. Create an account on GitHub, set up an SSH key, and add a new repo.
5. Run git remote add origin git@github.com:Edgewall/Trac.git (assuming Edgewall is the account name and Trac is the repo name) in the Trac.git directory.
6. Set up a cron job to run the following script every 10 or 15 minutes:

   #!/bin/bash
   export GIT_DIR=/path/to/Trac.git
   git svn fetch -q &>> /var/log/git_mirror.log
   git push --all origin
   

7. Have a cup of tea, you're done!

I strongly believe that a lot of the people who like to hack on Trac would benefit from this greatly.

[Remy Blank]

I think this is a nice idea. And while we're at it, also do a Mercurial mirror on BitBucket?

[anatoly techtonik <techtonik@…>]

It is not that hard at all to provide Trac mirror on Bitbucket. The only question is where to setup automatic sync?

[jake@…]

You can script the fetch/push in an SVN post-commit hook on the repository server for instant updating of all mirrors you create. Just remember to use a lockfile so quick back-to-back commits don't create multiple instances of the hook running at the same time.

[Christian Boos]

Just a quick note to say we actively work on this topic, stay tuned!

[daltonmatos@…]

This will be great! I'm looking forward to see an official github repository for Trac. I'll fork it for sure! =D

[Christian Boos]

There's now an official Git mirror, on GitHub: ​http://github.com/edgewall/trac.

We still have some remaining things to do before announcing the Mercurial mirror.

[Christian Boos]

Now there's also an official Mercurial mirror available on BitBucket: ​http://bitbucket.org/edgewall/trac

[osimons]

Replying to cboos:

Now there's also an official Mercurial mirror available on BitBucket

Thanks. I've maintained a mirror there for many years, and find the addition of the Subversion revision numbers to be very important. Without it, it just becomes so much harder to cross-reference with the main Trac and checkin/change history:

​http://bitbucket.org/osimons/trac-trunk

Anyway, just my 2c. Very good getting official mirrors - thanks :-)

[Christian Boos]

Replying to osimons:

… the addition of the Subversion revision numbers to be very important. Without it, it just becomes so much harder to cross-reference with the main Trac and checkin/change history

Good idea! But as a result, we're going to redo the conversion. In the meantime the edgewall/trac repository has been made private.

[Remy Blank]

Replying to osimons:

I've maintained a mirror there for many years, and find the addition of the Subversion revision numbers to be very important. Without it, it just becomes so much harder to cross-reference with the main Trac and checkin/change history:

Yes, I thought about that in the beginning, and I even found the location where Mercurial has to be (monkey-)patched to achieve that, but somehow forgot about it. We'll probably re-create the mirror with that suggestion, so please don't use that mirror just yet (I see Christian has removed it already).

[Christian Boos]

Replying to rblank:

We'll probably re-create the mirror with that suggestion, so please don't use that mirror just yet

Ok, so this is now "fixed", messages are reformatted like that: "[T%(rev)s] %(log)s", see the demo-0.12:source:/ for example.

Another change was to keep the branch named trunk instead of mapping it to default. That way the hg mirror will have the same branch names as the git mirror.

But before pushing this bitbucket.org, I wonder if we shouldn't also do a last enhancement concerning the tags. For now hg convert adds all the tags at once at the end of the initial conversion. Instead, we should have tag changesets part of the linear history of the branch being tagged.

[Remy Blank]

Replying to osimons:

Thanks. I've maintained a mirror there for many years, and find the addition of the Subversion revision numbers to be very important. Without it, it just becomes so much harder to cross-reference with the main Trac and checkin/change history:

After some discussion, we have agreed (or let's say I have bugged Christian enough that he stopped resisting :) on not mangling the changeset message on the mirror, but to take an alternative approach:

• The SVN revision is already added in a revision property during the conversion.
• mercurial-plugin already renders that property as a link to the original SVN changeset, provided the source repository is in the same Trac instance. I will extend this to also allow linking to external Trac instances through InterTrac.
• I have started a small Mercurial plugin that gathers those revision properties and creates "virtual" tags, so that the SVN revisions become visible in "hg log", "hg graphlog" and even the web interface.

The extension can be found here:

​http://rc.c-space.org/hg/HgSvnUtils/

The performance of the extension is currently just acceptable, and I'll work on optimizing it, both for usability and performance.

So the remaining work to be done before creating the final Mercurial mirror is:

• A small patch to keep the branch name "trunk" for trunk instead of "default" (already done).
• A patch (or possibly an extension) to have the conversion process create Mercurial tags at the point in history where the source tag was created, instead of adding a single tagging changeset at the end of the conversion.

[Remy Blank]

Performance and usability for the "virtual tags" have been greatly improved:

• The virtual tags are now cached, and the cache is only updated when adding or stripping changesets. Reading the cached tags adds about 0.07s to the commands that need the tags (e.g. "hg log") on my machine, so it has become negligible.
• The virtual tags don't appear in the list shown by "hg tags" unless the --verbose is provided.

Also, the remaining quirks for the conversion process to Mercurial have been fixed in the HgSvnUtils extension. We now have a mirror with the following characteristics:

• The trunk is in branch "trunk".
• All branches from 0.7-stable up are converted to named branches.
• All tags are converted, and tagging occurs in a separate, disconnected branch "tags", so the tagging doesn't pollute the changelog.

A test conversion can be found here: ​https://svn.edgewall.org/hg/trac/mirror3

If nobody objects, I would like to make this the official mirror and push it to BitBucket.

[Christian Boos]

Replying to rblank:

• All tags are converted, and tagging occurs in a separate, disconnected branch "tags", so the tagging doesn't pollute the changelog.

Give me this week-end to check by myself if this really can't be fixed in hg convert (have tags happen in the svn order, on the branch the tag belongs to).

[Remy Blank]

Replying to cboos:

Give me this week-end to check by myself if this really can't be fixed in hg convert (have tags happen in the svn order, on the branch the tag belongs to).

Be my guest. The thing is, tagging is a completely separate step in the conversion process. The order of the steps are source- and destination-independent. Making the tagging part of the "content" conversion would require a significant patch on the convert extension, and I couldn't find a reasonable way to solve the issue in HgSvnUtils. Except maybe by generating a "virtual" file named .hgtags and interleaving its changes in the content, but then you would have to fiddle with the changeset parent relations of the following changesets.

And I actually prefer having the tagging in a separate branch, for the following reasons:

• The content of a Mercurial working copy is identical to that of a SVN working copy, and there's no additional .hgtags file in the former.
• The tags already appear in the changelog as tags, so I see no use for an explicit tagging changeset interleaved with the normal content.

Even better would be to have the tagging occurring at the point in history where the tag was created, but still in a separate branch. This is going to happen with future tags anyway. But as this is purely cosmetic, and the cost (of patching the convert extension) is high, it's just not worth it.

[Christian Boos]

I made some progress on this topic and have a first patch (incrementaltagging-hg16.patch​) which refactors slightly the tagging to also allow for an alternate incremental tagging mode, which effectively implements the tagging occurring at the point in history where the tag was created, but still in a separate branch feature you proposed above. It probably needs some more testing though.

About the separate "tags" branch and the two reasons you proposed for keeping it:

• I don't see the presence of the .hgtags file to be an issue; we already have .hgignore, .gitignore, one more .file is not going to hurt
• What do you mean by the tags already appear in the changelog as tags? The fact that there's a tag "label" for the corresponding changeset, in the hg changelog?

I personally prefer to have the tagging changesets be part of the branches they're tagging, as they conceptually belong there (e.g. when cloning one branch only, you'd still get the tags belonging to that branch), and it's the standard Mercurial usage after all.

[Christian Boos]

convert: add incremental tagging mode

[Remy Blank]

Replying to cboos:

• I don't see the presence of the .hgtags file to be an issue; we already have .hgignore, .gitignore, one more .file is not going to hurt

Yes, but those other files are tracked in SVN as well, whereas the .hgtags file would be tracked only in Mercurial and not in SVN. So the content of both repositories becomes different.

• What do you mean by the tags already appear in the changelog as tags? The fact that there's a tag "label" for the corresponding changeset, in the hg changelog?

Yes, the tags appear as labels and that's all they need to do. I see no need for them to appear as a separate changeset.

I personally prefer to have the tagging changesets be part of the branches they're tagging, as they conceptually belong there (e.g. when cloning one branch only, you'd still get the tags belonging to that branch), and it's the standard Mercurial usage after all.

I guess it's a matter of personal taste: to me tags are conceptually out-of-band information and not part of the content to be tracked. AFAICT it's only standard Mercurial usage because it made the implementation much simpler than tracking a separate type of object.

If you insert tagging changesets in-between "content" changesets, you are modifying the revision graph compared to SVN. Sure, this is already the case for changesets that become no-ops in Mercurial (e.g. property changes), but those are only deletions.

I do see the advantage of being able to clone a single branch and still getting the tags. If they are in a separate branch, you have to pull the tags "by hand". I still prefer having them separate.

Ultimately, I'd rather spend time working on Trac features rather than arguing about infrastructure details. Again, we're not migrating to Mercurial, we're providing an infrastructure for easier collaboration. Tags will probably hardly ever used at all, so I'd rather KISS and go the route that requires the least amount of work, i.e. one-shot tagging.

[Remy Blank]

Any news on this? I'd really like to start using the Mercurial mirror to replace my previous "SVN branch" system.

[Christian Boos]

Well, sorry for not following up earlier, but I've been busy lately.

So last week, I finished up a set of patches that allow me to have an incremental mirror, with the svn changesets for tags (like r9872) mirrored as Hg changesets, complete with author and commit message (instead of convert-repo / updating tags), and appearing in the proper date order, on the same branch as the changeset they're tagging. I'll eventually polish those patches and submit them upstream, as I think they can be generally useful.

As for the tags being "conceptually out-of-band information and not part of the content to be tracked", it's not the Mercurial way of thinking about tags. In Mercurial, tags are part of the history by design (see e.g. ​http://www.selenic.com/pipermail/mercurial/2009-January/023271.html, cited from ​mercurial:GitConcepts#Tag_model).

Back to our topic of a Mercurial mirror: if you really want to promote mirror3 to mirror and take this as our official edgewall/trac mirror on bitbucket, it's OK for me even if I really dislike having a separate "tags" branch, but we need to move forward.

But now that I know the convert code a little better, I also have some ideas about how to get a complete mirror, with all the branches we ever created directly or indirectly from trunk. At that point, we might reopen the discussion :-)

[Remy Blank]

Replying to cboos:

As for the tags being "conceptually out-of-band information and not part of the content to be tracked", it's not the Mercurial way of thinking about tags. In Mercurial, tags are part of the history by design (see e.g. ​http://www.selenic.com/pipermail/mercurial/2009-January/023271.html, cited from ​mercurial:GitConcepts#Tag_model).

That message is in line with my concept of tags. From comment:17 (added emphasis):

I guess it's a matter of personal taste: to me tags are conceptually out-of-band information and not part of the content to be tracked.

Tags are metadata and therefore (to me) conceptually not part of the content. Yes, they must be part of the history. The fact that they appear in separate changesets is an implementation detail, and it was probably the simplest possible implementation ("the most reasonable thing to do is use the same mechanism"). The rest of the thread confirms this.

Back to our topic of a Mercurial mirror: if you really want to promote mirror3 to mirror and take this as our official edgewall/trac mirror on bitbucket, it's OK for me even if I really dislike having a separate "tags" branch, but we need to move forward.

I would like to move forward, yes. We have been playing with this mirror for over two months now, and I have the feeling that, considering the goal of the operation (facilitating collaboration, helping to keep local patches), we haven't gained much for the added effort. It has also been rather distracting from "real", productive Trac work.

But now that I know the convert code a little better, I also have some ideas about how to get a complete mirror, with all the branches we ever created directly or indirectly from trunk.

I was afraid that you would be saying that. I still don't get what the added value of a full conversion is (unless we want to switch to Mercurial, that is). Sure, hacking Mercurial is fun, and we can always make the conversion more "perfect", but in the end, how does it help our goal? Maybe we have different goals, mind you, so we should probably have defined that earlier.

The only advantage I see to have the perfect conversion now is to avoid having to rebase local changes on top of the final repository, the day we switch to Mercurial. But this won't happen anytime soon (if ever, we could also switch to git FWIW), and until then, the conversion process will have changed even more, so rebasing will probably be necessary anyway.

Ok, let's step back a bit. I see three options now:

1. We set up mirror3 as the official mirror.
2. We create a new mirror with your patched conversion and set that as the official mirror.
3. We delay the mirror even further until we have the perfect, complete conversion.

You're the boss, you get to decide. I don't care about 1. or 2., as long as we have an official mirror. Should we go for 3., I'll use my own local conversion. But please, let's make a decision now.

[Christian Boos]

Replying to rblank:

But please, let's make a decision now.

Well, I thought I did so when I said "OK" for mirror3.

I may attempt to have a complete conversion at some point in the future, … or not, but this shouldn't prevent us to have a working mirror now.

[Remy Blank]

Replying to cboos:

Well, I thought I did so when I said "OK" for mirror3.

Oh, sorry. I guess I put too much weight on "even if I really dislike having a separate "tags" branch" :)

All right, I'll set up the conversion and the mirror on BitBucket.

[Christian Boos]

I may attempt to have a complete conversion at some point in the future, … or not, but this shouldn't prevent us to have a working mirror now.

… and to add one more requirement to this future hypothetical "perfect" conversion, it should also try to mirror the merges, just like the git-svn conversion does ;-)

[Remy Blank]

The Mercurial mirror is up and running on BitBucket:

​http://bitbucket.org/edgewall/trac

[Remy Blank]

And the use of ​HgSvnUtils is documented in SubversionRepository@11.