Edgewall Software
Modify

Opened 12 years ago

Closed 10 years ago

Last modified 3 years ago

#6588 closed defect (fixed)

trac-admin resync leaks memory FAST!

Reported by: marcin.owsiany@… Owned by: Christian Boos
Priority: normal Milestone: 0.11.5
Component: version control Version: 0.10.4
Severity: normal Keywords: svn resync memory
Cc: Branch:
Release Notes:
API Changes:

Description

I am upgrading a trac installation from 0.9.5 to 0.10.4.

As recommended, I'm trying to run a resync in our environment as the last stage of the upgrade. Unfortunately this fails about 4% before finishing, apparently due to running out of memory. The repository our project uses has over 40k revisions, and my estimation is that trac-admin resync leaks about 40 MB of memory per 1000 revisions. Well, maybe not all this is being leaked, but it grew from 30 MB (resident size) shortly after starting to over 150 MB at revision 3000.

I just created a huge swapfile to see if it succeds with more memory available.

We are using a sqlite backend.

Attachments (0)

Change History (7)

comment:1 by marcin.owsiany@…, 12 years ago

Unfortunately adding more swap does not help. I think trac-admin could just be hitting the maximum python heap size on 32-bit systems. Adding a print_exc() to the exception handler gives me:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/scripts/admin.py", line 97, in onecmd
    rv = cmd.Cmd.onecmd(self, line) or 0
  File "/usr/lib/python2.3/cmd.py", line 210, in onecmd
    return func(arg)
  File "/usr/lib/python2.3/site-packages/trac/scripts/admin.py", line 707, in do_resync
    repos = self._resync(env.get_repository())
  File "/usr/lib/python2.3/site-packages/trac/scripts/admin.py", line 682, in _resync
    repos.sync(_feedback)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 185, in sync
    (str(next_youngest),
  File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/lib/python2.3/site-packages/sqlite/main.py", line 255, in execute
    self.rs = self.con.db.execute(SQL % parms)
MemoryError

Also, forgot to mention that we're running it on python 2.3 on RHEL4.

comment:2 by Christian Boos, 12 years ago

Keywords: memory added
Milestone: 0.11.1
Severity: normalminor

While this error is annoying, it shouldn't be a show stopper. Once you've started the resync, the tables are cleared and populated again with each missing changeset, normally in one go. But if that fails, the next automatic resync attempt (i.e. not the resync command, the one triggered by accessing the project through the web interface) will resume the unfinished work.

comment:3 by Christian Boos, 11 years ago

Component: generalversion control
Keywords: verify added; sqlite removed
Milestone: 0.11.2not applicable

Could be related to the following bug in the Python bindings for Subversion:

(while the summary of the issue indicates svn_ra, the fix and the changeset comment indicate it's a more general issue)

comment:4 by techtonik <techtonik@…>, 11 years ago

Is it possible to run this test under profiler to get more information about memory usage and other potential problems?

comment:5 by Christian Boos, 10 years ago

Keywords: verify removed
Milestone: not applicable0.11.5
Owner: changed from Jonas Borgström to Christian Boos
Severity: minornormal

At first this seemed to work for me: using MultiRepos branch (which should be the same as trunk or even 0.11-stable in this respect) and Python 2.4.4 with SVN 1.4.3 bindings, I could resync a 26k revision repository (tried both at top-level and with scopes), with a memory usage of 25Mb growing to 30Mb and staying there for the major time of the resync (RSS).

But then, the scope used was trunk or some subfolder from trunk. As soon as I used some path which was copied from trunk (directly or as a copy of a branch), then the memory usage increased dramatically. That increase happened while there was still nothing to insert in the tables. Indeed, going back to a trunk subfolder shows the same effect when that subfolder is created late in the history of the project.

So it seems the problem is:

self.repos.next_rev(next_youngest, find_initial_rev=True)

#5213 again …

comment:6 by Christian Boos, 10 years ago

Status: newassigned

The root cause for this issue was a leak in fs.node_history(root,path), when the given path didn't exist in root. Calling fs.check_path(root,path) first avoids the problem.

comment:7 by Christian Boos, 10 years ago

Resolution: fixed
Status: assignedclosed

Fixed in r7998. Note that I first tried to add a try/except around the problematic fs.node_history, with a pool cleanup in the except: clause, but that didn't prevent the leak.

Additional clean-up done in r7999.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christian Boos.
The resolution will be deleted. Next status will be 'reopened'.
to as closed The owner will be changed from Christian Boos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.