trac-admin resync leaks memory FAST!

[marcin.owsiany@…]

I am upgrading a trac installation from 0.9.5 to 0.10.4.

As recommended, I'm trying to run a resync in our environment as the last stage of the upgrade. Unfortunately this fails about 4% before finishing, apparently due to running out of memory. The repository our project uses has over 40k revisions, and my estimation is that trac-admin resync leaks about 40 MB of memory per 1000 revisions. Well, maybe not all this is being leaked, but it grew from 30 MB (resident size) shortly after starting to over 150 MB at revision 3000.

I just created a huge swapfile to see if it succeds with more memory available.

We are using a sqlite backend.

[marcin.owsiany@…]

Unfortunately adding more swap does not help. I think trac-admin could just be hitting the maximum python heap size on 32-bit systems. Adding a print_exc() to the exception handler gives me:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/scripts/admin.py", line 97, in onecmd
    rv = cmd.Cmd.onecmd(self, line) or 0
  File "/usr/lib/python2.3/cmd.py", line 210, in onecmd
    return func(arg)
  File "/usr/lib/python2.3/site-packages/trac/scripts/admin.py", line 707, in do_resync
    repos = self._resync(env.get_repository())
  File "/usr/lib/python2.3/site-packages/trac/scripts/admin.py", line 682, in _resync
    repos.sync(_feedback)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 185, in sync
    (str(next_youngest),
  File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/lib/python2.3/site-packages/sqlite/main.py", line 255, in execute
    self.rs = self.con.db.execute(SQL % parms)
MemoryError

Also, forgot to mention that we're running it on python 2.3 on RHEL4.

[Christian Boos]

While this error is annoying, it shouldn't be a show stopper. Once you've started the resync, the tables are cleared and populated again with each missing changeset, normally in one go. But if that fails, the next automatic resync attempt (i.e. not the resync command, the one triggered by accessing the project through the web interface) will resume the unfinished work.

[Christian Boos]

Could be related to the following bug in the Python bindings for Subversion:

• issue ​3052
• fixed in ​r28544 (in svn 1.5 only for now)

(while the summary of the issue indicates svn_ra, the fix and the changeset comment indicate it's a more general issue)

[techtonik <techtonik@…>]

Is it possible to run this test under profiler to get more information about memory usage and other potential problems?

[Christian Boos]

At first this seemed to work for me: using MultiRepos branch (which should be the same as trunk or even 0.11-stable in this respect) and Python 2.4.4 with SVN 1.4.3 bindings, I could resync a 26k revision repository (tried both at top-level and with scopes), with a memory usage of 25Mb growing to 30Mb and staying there for the major time of the resync (RSS).

But then, the scope used was trunk or some subfolder from trunk. As soon as I used some path which was copied from trunk (directly or as a copy of a branch), then the memory usage increased dramatically. That increase happened while there was still nothing to insert in the tables. Indeed, going back to a trunk subfolder shows the same effect when that subfolder is created late in the history of the project.

So it seems the problem is:

self.repos.next_rev(next_youngest, find_initial_rev=True)

#5213 again …

[Christian Boos]

The root cause for this issue was a leak in fs.node_history(root,path), when the given path didn't exist in root. Calling fs.check_path(root,path) first avoids the problem.

[Christian Boos]

Fixed in r7998. Note that I first tried to add a try/except around the problematic fs.node_history, with a pool cleanup in the except: clause, but that didn't prevent the leak.

Additional clean-up done in r7999.