SubversionMergePropertyDiffRenderer is slow

[ebouaziz@…]

The repository has around 40000 revisions, and a lot of mergeinfo on the trunk.

When viewing a changeset that affects mergeinfo on the trunk, it takes 20 seconds to render the changes. render_property_diff() loops on every revision contained in the new mergeinfo property and makes one or more database requests for each (_get_node_revs()), which makes thousands in this case. The displayed information as a result is something like (with links):

trunk 
Property svn:mergeinfo changed
/sandboxes/t6823 (added)	merged: 38954-38956

Note that svn returns the same information in 0.1 second

$ svn diff -c 38957 svn://svn.xxx.yyy/sdk/
(...)
Property changes on: trunk
___________________________________________________________________
Modified: svn:mergeinfo
   Merged /sandboxes/t6823:r38954-38956

Sadly, it seams that the versioncontrol API does not allow to fetch property diffs, only file diffs.

This is with Trac 1.1.2dev and Postgres 9.1

[Christian Boos]

Yeah, sorry for that, the code in that area leaves much to be desired w.r.t. performance. We already have #8459, but here you narrowed the problem somehow.

If you would follow-up on your analysis and produce a patch fixing this, you'd become an instant Trac-hero ;-)

[ebouaziz@…]

I though that maybe we could get the information from svn through

repos.get_changes(old_context.resource.realm,
                  old_context.resource.version,
                  new_context.resource.realm,
                  new_context.resource.version)

but for some reason, has_node() in svn_fs.py returns False. I tried to look into the python-subversion module to find why, but I don't even find the actual code for fs.check_path(), it all wrapped etc, so I'm stuck.

[Jun Omae]

I've tried to reduce time for _get_node_revs() in log:jomae.git:ticket11219/1.0-stable.

After the changes, the method search only the revisions in differences between svn:mergeinfo properties for each path. Also, the method divides single revision range into small multiple ranges and search the ranges in node_change records.

Thoughts?

[Jun Omae]

jomae.git:ticket11219/1.0-stable is very slow on MySQL…. Reworked at jomae.git:ticket11219.2/1.0-stable.

Timing of render_property_diff for svn:mergeinfo with [11654] in a clone of ​http://svn.edgewall.com/repos/trac.

Repository type	Database	before	after
direct-svnfs		1.203752s	0.106252s
svn	sqlite	0.368621s	0.031234s
svn	postgres	0.182075s	0.023396s
svn	mysql	4.226078s	0.043819s

[Jun Omae]

Reworked in jomae.git@t11219.2_0.12 ([4938066f/jomae.git]), again.

In our case, svn:mergeinfo has 25 lines and node_change table has about 150,000 records. Viewing a changeset is very slow. I want to fix it on 0.12-stable.

[Jun Omae]

Reworked in jomae.git@t11219.2_0.12 ([4938066f/jomae.git]), again.

Committed in [12959] and merged in [12960-12961].

I have a idea to reduce calls and time of _get_node_revs(). I try it for both this ticket and #8459.

[Jun Omae]

Comitted in [13015,13018] and merged in [13016-13017,13019-13020].

[ebouaziz@…]

Thanks for this fix! For information, it now takes only 2 seconds to display the mergeinfo properties on a repository with 50000+ revisions (Postgres 9.1)