Edgewall Software
Modify

Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#11219 closed defect (fixed)

SubversionMergePropertyDiffRenderer is slow

Reported by: ebouaziz@… Owned by: Jun Omae
Priority: normal Milestone: 0.12.6
Component: version control/changeset view Version:
Severity: major Keywords: performance svn mergeinfo
Cc: Jun Omae, mpotter@…
Release Notes:

Reduce execution time of rendering svn:mergeinfo in changeset view.

API Changes:

Description

The repository has around 40000 revisions, and a lot of mergeinfo on the trunk.

When viewing a changeset that affects mergeinfo on the trunk, it takes 20 seconds to render the changes. render_property_diff() loops on every revision contained in the new mergeinfo property and makes one or more database requests for each (_get_node_revs()), which makes thousands in this case. The displayed information as a result is something like (with links):

trunk 
Property svn:mergeinfo changed
/sandboxes/t6823 (added)	merged: 38954-38956

Note that svn returns the same information in 0.1 second

$ svn diff -c 38957 svn://svn.xxx.yyy/sdk/
(...)
Property changes on: trunk
___________________________________________________________________
Modified: svn:mergeinfo
   Merged /sandboxes/t6823:r38954-38956

Sadly, it seams that the versioncontrol API does not allow to fetch property diffs, only file diffs.

This is with Trac 1.1.2dev and Postgres 9.1

Attachments (0)

Change History (10)

comment:1 Changed 5 years ago by Christian Boos

Keywords: performance svn mergeinfo added
Milestone: next-stable-1.0.x
Priority: lownormal
Severity: normalmajor

Yeah, sorry for that, the code in that area leaves much to be desired w.r.t. performance. We already have #8459, but here you narrowed the problem somehow.

If you would follow-up on your analysis and produce a patch fixing this, you'd become an instant Trac-hero ;-)

comment:2 Changed 5 years ago by ebouaziz@…

I though that maybe we could get the information from svn through

repos.get_changes(old_context.resource.realm,
                  old_context.resource.version,
                  new_context.resource.realm,
                  new_context.resource.version)

but for some reason, has_node() in svn_fs.py returns False. I tried to look into the python-subversion module to find why, but I don't even find the actual code for fs.check_path(), it all wrapped etc, so I'm stuck.

comment:3 Changed 5 years ago by Jun Omae

I've tried to reduce time for _get_node_revs() in log:jomae.git:ticket11219/1.0-stable.

After the changes, the method search only the revisions in differences between svn:mergeinfo properties for each path. Also, the method divides single revision range into small multiple ranges and search the ranges in node_change records.

Thoughts?

comment:4 Changed 5 years ago by Jun Omae

jomae.git:ticket11219/1.0-stable is very slow on MySQL…. Reworked at jomae.git:ticket11219.2/1.0-stable.

Timing of render_property_diff for svn:mergeinfo with [11654] in a clone of http://svn.edgewall.com/repos/trac.

Repository type Database before after
direct-svnfs 1.203752s 0.106252s
svn sqlite 0.368621s 0.031234s
svn postgres 0.182075s 0.023396s
svn mysql 4.226078s 0.043819s
Last edited 5 years ago by Jun Omae (previous) (diff)

comment:5 Changed 5 years ago by Jun Omae

Cc: Jun Omae added

comment:6 Changed 5 years ago by Mark Potter <mpotter@…>

Cc: mpotter@… added

comment:7 Changed 4 years ago by Jun Omae

Milestone: next-stable-1.0.x0.12.6
Owner: set to Jun Omae
Status: newassigned

Reworked in jomae.git@t11219.2_0.12 ([4938066f/jomae.git]), again.

In our case, svn:mergeinfo has 25 lines and node_change table has about 150,000 records. Viewing a changeset is very slow. I want to fix it on 0.12-stable.

Last edited 4 years ago by Jun Omae (previous) (diff)

comment:8 in reply to:  7 Changed 4 years ago by Jun Omae

Reworked in jomae.git@t11219.2_0.12 ([4938066f/jomae.git]), again.

Committed in [12959] and merged in [12960-12961].

I have a idea to reduce calls and time of _get_node_revs(). I try it for both this ticket and #8459.

comment:9 Changed 4 years ago by Jun Omae

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Comitted in [13015,13018] and merged in [13016-13017,13019-13020].

comment:10 Changed 4 years ago by ebouaziz@…

Thanks for this fix! For information, it now takes only 2 seconds to display the mergeinfo properties on a repository with 50000+ revisions (Postgres 9.1)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.