trac-admin resync leaks memory

[turnopil@…]

I am using a trac installation 1.2.3 version. Was installed via easy_install

subversion-python27 Version: 1.9.7
python27-psycopg2   Version: 2.5.1
subversion          Version: 1.9.7

I'm trying to run a resync in our environment. Unfortunately this fails apparently due to running out of memory. The repository our project uses has over 100k revisions, and my estimation is that trac-admin resync leaks about 40 MB of memory per 1000 revisions. But after 90K revisions he used all 8 gigabytes of RAM.

-bash: fork: Cannot allocate memory

PID   USER     PR NI VIRT  RES  SHR  S %CPU %MEM TIME+    COMMAND
401   root     20 0  0     0      0  D 8.3  0.0  0:02.91  kswapd0
10228 root     30 10 7804m 7.0g 1168 D 3.6  93.7 11:59.68 trac-admin
9736  postgres 20 0  328m  133m 132m S 0.7  1.7  0:00.56  postmaster

We are using a postgres 9.6 backend.

[Jun Omae]

I think the issue is the same of #13112. Please try to use 1.2-stable or apply [16835] to Trac 1.2.3.

[turnopil@…]

Replying to Jun Omae:

I think the issue is the same of #13112. Please try to use 1.2-stable or apply [16835] to Trac 1.2.3.

Than you! I will try

[turnopil@…]

Confirmed. Memory leaks still exists but 50% less.

Is it planned to solve this problem completely? Or it's technically impossible?

[Jun Omae]

Thanks for the feedback.

I spent more time to investigate other memory leaks on #13112, however I couldn't find. I guess that Subversion Python bindings has the causes of the memory leaks, but no evidence.

[Jun Omae]

The root cause is that svn.delta.make_editor() leads ref-count leak, see comment:6:ticket:13356 for the details.

I'm trying to create a fix for the leak in Subversion Python bindings, but it is not finished.

Workaround:

tracopt/versioncontrol/svn/svn_fs.py

@@ -736 +736 @@
             deltas = sorted(((_from_svn(path), kind, change)
                              for path, kind, change in editor.deltas),
                             key=lambda entry: entry[0])
+
+            # Workaround for ref-count leak due to delta.make_editor()
+            editor.__dict__.clear()
+
             for path, kind, change in deltas:
                 old_node = new_node = None
                 if change != Changeset.ADD:
@@ -1086 +1090 @@
             changes.append([path, kind, action, base_path, base_rev])
             idx += 1
 
+        # Workaround for ref-count leak due to delta.make_editor()
+        editor.__dict__.clear()
+
         moves = []
         # a MOVE is a COPY whose `base_path` corresponds to a `new_path`
         # which has been deleted

Memory profile:

• Python 2.7.12
• Trac 1.0-stable
• Subversion 1.9.7
• Mirror of Trac Subversion repository (latest rev r17519)

	Increment on repos.sync()
Before the patch	194.1 MiB
After the patch	24.1 MiB

[Jun Omae]

Committed in [17552-17554].