Edgewall Software

Opened 4 years ago

Closed 4 years ago

#12694 closed defect (fixed)

Browser page with large git repository is pretty slow since git 2.9

Reported by: Jun Omae Owned by: Jun Omae
Priority: normal Milestone: 1.0.14
Component: plugin/git Version: 1.0.11
Severity: normal Keywords: performance
Cc: Branch:
Release Notes:

Fix slowness of browser page with large git repository since git 2.9.

API Changes:
Internal Changes:


After upgrading to git 2.11.1, I get speed degradation of browser page with large git repository in production environment.

The browser page calls GitNode.get_entries() for git repository. The get_entries() internally executes git log ... command at tags/trac-1.0.13/tracopt/versioncontrol/git/PyGIT.py@:914-915#L906. However, git log ... command is pretty slow caused by detecting renames in commit.

$ time /tmp/git/2.8.4/bin/git --git-dir /path/to/git/host/reponame log --pretty=format:%n%H --name-status master -- . >/dev/null

real    0m1.596s
user    0m0.613s
sys     0m0.103s

$ time /tmp/git/2.9.3/bin/git --git-dir /path/to/git/host/reponame log --pretty=format:%n%H --name-status master -- . >/dev/null
warning: inexact rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 3652 and retry the command.

real    0m42.107s
user    0m17.918s
sys     0m4.408s

Work around: adding --no-renames option to the git log ... command.

$ time /tmp/git/2.8.4/bin/git --git-dir /path/to/git/host/reponame log --pretty=format:%n%H --no-renames --name-status master -- . >/dev/null

real    0m0.288s
user    0m0.203s
sys     0m0.020s

$ time /tmp/git/2.9.3/bin/git --git-dir /path/to/git/host/reponame log --pretty=format:%n%H --no-renames --name-status master -- . >/dev/null

real    0m0.527s
user    0m0.300s
sys     0m0.044s

Large git repository:

$ du -sh /path/to/git/host/reponame
3.3G    /path/to/git/host/reponame
$ git --git-dir /path/to/git/host/reponame rev-list --all | wc -l

Test script:

from time import time
from trac.env import Environment

env = Environment('/path/to/trac/host/env')
from trac.versioncontrol.api import RepositoryManager
from tracopt.versioncontrol.git.git_fs import GitConnector
version = GitConnector(env)._version['v_str']
rm = RepositoryManager(env)
repos = rm.get_repository('reponame')
node = repos.get_node('/')

t = time()
    print('%s - %.3fs' % (version, time() - t))

Patch which unit tests pass with git - 2.11.1:

  • trac/tracopt/versioncontrol/git/PyGIT.py

    diff --git a/trac/tracopt/versioncontrol/git/PyGIT.py b/trac/tracopt/versioncontrol/git/PyGIT.py
    index fc61319e..c08fbb8a 100644
    a b class Storage(object):  
    902902        base_path = self._fs_from_unicode(base_path)
    904904        def name_status_gen():
    905             p[:] = [self.repo.log_pipe('--pretty=format:%n%H',
     905            p[:] = [self.repo.log_pipe('--pretty=format:%n%H', '--no-renames',
    906906                                       '--name-status', sha, '--', base_path)]
    907907            f = p[0].stdout
    908908            for l in f:

Timing of GitNode.get_entries():

Version w/o patch w/ patch 0.353s 0.317s
2.3.10 0.707s 0.325s
2.4.11 0.318s 0.317s
2.5.5 0.308s 0.303s
2.7.4 0.317s 0.305s
2.8.4 0.323s 0.304s
2.9.3 46.758s 0.306s
2.10.2 47.422s 0.370s
2.11.1 49.967s 0.310s

Attachments (0)

Change History (2)

comment:1 by Jun Omae, 4 years ago

Milestone: next-stable-1.0.x1.0.14
Owner: set to Jun Omae
Status: newassigned

Proposed changes in [7594cae88/jomae.git].

comment:2 by Jun Omae, 4 years ago

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Fixed in [15576] and merged in [15577-15578].

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain Jun Omae.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Jun Omae to the specified user.

Add Comment

E-mail address and name can be saved in the Preferences .
Note: See TracTickets for help on using tickets.