Problem with SVN bindings (SVN 1.3.0, Trac r2771)

[Manuzhai]

Every morning when I first try to get to Trac, I get this error on the first request:

Mod_python error: "PythonHandler trac.web.modpython_frontend"

Traceback (most recent call last):
...
  File "/home/manuzhai/dev/trac/trac/versioncontrol/svn_fs.py", line 31, in ?
    from svn import fs, repos, core, delta
  File "/usr/lib/python2.4/site-packages/svn/fs.py", line 19, in ?
    from libsvn.fs import *
  File "/usr/lib/python2.4/site-packages/libsvn/fs.py", line 29, in ?
    import core
  File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 3049, in ?
    svn_pool_create()
  File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 1098, in svn_pool_create
    return apply(_core.svn_pool_create, args)

TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance()' is received

Similar symptoms:

[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0xb6924c8c>)' is received

Subsequent requests seem to just work…

Note

This is now a well-known issue having to do with the way the SVN Python bindings behaves when multiple Python interpreters are used within the same process, as this can be the case with mod_python e.g. when more than one virtual host is used.

Most SVN binding versions are affected, though there's no confirmed report of this with 1.2.x (the bug started to be reported with 1.3.0) and reverting to SVN 1.2.4 is sometimes mentioned as a workaround.

There are other documented workarounds where one can keep using SVN 1.3.x and 1.4.x:

• see #3455
• see comment:72

Related issue: #3371

[Christian Boos]

See also #2472, where this error was also reported and discussed. As that other ticket was about another issue with the 1.3.0 bindings, it has been closed and this ticket should be used to focus on the 'apr_pool_t *' is expected issue.

[Manuzhai]

This error seems to appear quite often on editing a wiki page. Refreshing the edit (resending POST data) gives the same error again, going back to the wiki page, then sending it again too, hard-refreshing the page then redoing the changes and submitting them still seems to work. Not sure whether this is always true…

[Christian Boos]

I've not been able to investigate and reproduce this myself, but could you try the following?

Index: svn_fs.py
===================================================================
--- svn_fs.py	(revision 2819)
+++ svn_fs.py	(working copy)
@@ -28,7 +28,7 @@
 from trac.versioncontrol.svn_authz import SubversionAuthorizer
 
 try:
-    from svn import fs, repos, core, delta
+    from svn import core, fs, repos, delta
     has_subversion = True
 except ImportError:
     has_subversion = False

[Manuzhai]

I've made the change, and will report back to see if it helps.

[anonymous]

I installed mod_python 3.1.4 / Python 2.3 / Subversion 1.3.0 on a single processor RHEL4 machine and I can't reproduce this issue. Ideas?

[pjenvey@…]

I am seeing this problem with python 2.4. The original bug reported to the subversion-users list also occured under python 2.4

Myabe the fact that you're using python 2.3 is a factor, or possibly some other dependency version mismatch?

[Manuzhai]

Yes, I'm also using Python 2.4. The change of import order doesn't solve all of the problem, though it does seem that it doesn't happen as often now. That might have to do with my usage pattern, though…

[Christian Boos]

Can you try to split the import in several lines, i.e.

try:
    from svn import core
    from svn import fs
    # etc.

[Manuzhai]

Done…

[Christian Boos]

I think I understand the issue: having all the imports in one line allows two threads (A and B) to start simultaneously the import process:

• A: trac/versioncontrol/svn_fs.py: from svn import fs, repos, core, delta
• B: trac/versioncontrol/svn_fs.py: from svn import fs, repos, core, delta
• A: in svn/fs.py, line 19: from libsvn.core import *,
• A: now in libsvn/core.py, is going through the > 3000 lines…
• B: in svn/fs.py, line 19: from libsvn.core import *,
• B: now in libsvn/core.py, is going through the > 3000 lines…
• A: in libsvn/core.py, line 3049: does a _core.svn_pool_create
  and here, _core is really _core (i.e. the native extension)
• A: in svn/fs.py, line 29: import svn.core as _core
• B: in libsvn/core.py, line 3049: does a _core.svn_pool_create
  and now, _core is not _core but svn.core and therefore, _core.svn_pool_create is not the native function but the (deprecated) compatibility function, which expects a pool as first argument… hence the error.

Hopefully splitting the imports does solve the issue, but the bindings should really be more robust about that and not rename different things to a same name…

[james82@…]

I have not been able to reproduce the problem (even with Python 2.4 and mod_python 3.1.3), but, I think I have figured out how to fix it. Please try the below patch (for Subversion 1.3.0).

subversion/bindings/swig/include/svn_types.swg

@@ -701 +701 @@
 
 %exception {
 #ifdef SWIGPYTHON
-    svn_swig_py_release_py_lock();
+    int released_lock = 0;
+    if (_global_svn_swig_py_pool != NULL) {
+      released_lock = 1;
+      svn_swig_py_release_py_lock();
+    }
 #endif
     $action
 #ifdef SWIGPYTHON
-    svn_swig_py_acquire_py_lock();
+    if (released_lock) {
+      svn_swig_py_acquire_py_lock();
+    }
 #endif
 }
 

[james82@…]

Important note: In order for the above patch to take effect, you'll need to install SWIG 1.3.25, and run the following commands:

   make extraclean
   ./autogen.sh
   ./configure
   make swig-py
   make install-swig-py

Note that you need to make extraclean to remove any old headers shipped with the tarball.

[pjenvey@…]

Manuzhai -

Did splitting the import into multiple lines solve the problem?

If not, I'll see if I can give james' patch a shot later today

[Manuzhai]

Splitting the import didn't solve the problem.

James' fix is a little involved for me, I just install SVN & bindings through emerge, I'm afraid. I could try it, but I'm not sure when I get to it.

[pjenvey@…]

james - I upgraded to swig 1.3.27, applied your patch and did the whole make extraclean/autogen. Now I am noticing a different, similar error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 206, in handler
    dispatch_request(mpr.path_info, mpr, env)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/web_ui/browser.py", line 91, in process_request
    repos = self.env.get_repository(req.authname)
  File "/usr/local/lib/python2.4/site-packages/trac/env.py", line 157, in get_repository
    authname)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/api.py", line 61, in get_repository
    return self._connector.get_repository(repos_type, repos_dir, authname)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 205, in get_repository
    repos = SubversionRepository(dir, authz, self.log)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 224, in __init__
    self.path = repos.svn_repos_find_root_path(path, self.pool())
  File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43, in svn_repos_find_root_path
    return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x8b88f6c>)' is received

Instead of seeing the stack trace in my error log (where I saw the original apr_pool_t exception), it shows up on an actual trac "Oops… Trac detected an internal error:" error page. I can't tell if this is due to your patch, the swig upgrade, or both

This new error is intermittent like the old one. Occasionally I receive either 404 responses or blank pages instead of the error message. Nothing in the error_log

Also: I got the impression the make extraclean/reautogen would end up creating a brand new subversion/includes/svn_types.h file corresponding to your change to svn_types.swg. That was not the case — the patch was successfully applied, but I ended up with the same svn_types.h file. Let me know if that is actually a problem

[David James <james82@…>]

Hi pjenvey,

It looks like you applied the patch correctly, but my patch did not help. I am not surprised by your new error message, because it is the same as the one that was reported earlier on the users list.

Could you send the full details for your system, and the full details of your apache configuration including your httpd.conf and a description of which "mpm" you are using (e.g. the worker mpm, or the prefork mpm). I would like to try to reproduce the problem by copying your configuration. Are you using Linux, Mac OS, or something else?

Here's a question for the group: In mod_python, does the "global interpreter lock" lock just the current interpreter or does it lock all interpreters? If the "global interpreter lock" only locks the current interpreter, then it explains our problem. We will need to use a different locking system in order to prevent race conditions during pool initialization. Perhaps an APR lock would do the trick?

Cheers,

David

[Manuzhai]

From ​http://docs.python.org/api/threads.html (and my understanding), the GIL just locks all but one threads within one interpreter. So if multiple interpreters are at work, we need a different locking mechanism (in which case the APR lock would seem like the best solution).

[David James <james82@…>]

Hi pjenvey, Manzuhai:

After thinking about this problem some more, I have written a patch which uses Python locking to solve the race condition in application pool initialization. Could you give this patch a try?

subversion/bindings/swig/include/proxy_apr.swg

@@ -60 +60 @@
  */
 
 %pythoncode %{
+import threading
+
 application_pool = None
+application_pool_lock = threading.Lock()
 class GenericSWIGWrapper:
   def __init__(self, this, pool):
     """Create new Generic SWIG wrapper object"""
@@ -98 +101 @@
       def set_parent_pool(self, parent_pool=None):
         """Create a new memory pool"""
         global application_pool
+
+        try:
+          application_pool_lock.acquire()
+
+          self._parent_pool = parent_pool or application_pool
+          self._mark_valid()
+    
+          # Protect important functions from GC
+          self._apr_pool_destroy = _core.apr_pool_destroy
+          self._svn_swig_py_clear_application_pool = \
+            _core.svn_swig_py_clear_application_pool
+    
+          # If we are an application-level pool,
+          # then set this pool to be the application-level pool
+          if not self._parent_pool:
+            svn_swig_py_set_application_pool(self, self)
+            application_pool = self
+        finally:
+          application_pool_lock.release()
   
-        self._parent_pool = parent_pool or application_pool
-        self._mark_valid()
-  
-        # Protect important functions from GC
-        self._apr_pool_destroy = _core.apr_pool_destroy
-        self._svn_swig_py_clear_application_pool = \
-          _core.svn_swig_py_clear_application_pool
-  
-        # If we are an application-level pool,
-        # then set this pool to be the application-level pool
-        if not self._parent_pool:
-          svn_swig_py_set_application_pool(self, self)
-          application_pool = self
-  
       def valid(self):
         """Check whether this memory pool and its parents
         are still valid"""

Important note: Once again, in order for this patch to take effect, you'll need to type:

   make extraclean
   ./autogen.sh
   ./configure
   make swig-py
   make install-swig-py

[Manuzhai]

Could you attach this patch to the ticket insteading of showing it inline? I'm having a hard time getting it applied correctly. Thanks!

[David James <james82@…>]

Fix race condition in application pool initialization

[David James <james82@…>]

Hi Manzuhai,

The patch is attached. I've also posted it to the Subversion developers list, at ​http://svn.haxx.se/dev/archive-2006-01/0946.shtml

Cheers,

David

[pjenvey@…]

David -

Unfortunately I'm getting the same error I reported last time with your latest patch. (scroll up for the entire thing)

 File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43, in svn_repos_find_root_path
    return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x92633ac>)' is received

Here's my specs:

FreeBSD 4
python 2.4.2
trac 1.0dev (r2721)
subversion 1.3.0 (with or without your latest patch, and cboos's small __del__ patch in #2472 which shouldn't make a difference)
swig 1.3.27
apr 0.9.7 (from apache-2.0.55)
neon 0.24.7

[pjenvey@…]

Also —

prefork MPM

[Christian Boos]

I've been unable to actually reproduce the problem (apache-2.0.55, mod_python-3.1.4, Subversion-1.3.0 and Python-2.4.2 on Linux SLES9 64bits (did I say it's reasonably fast?)).

So I'm leaving this open for further investigation or feedback, but post-poning it.

[Manuzhai]

This is still annoying. I'll see if I can try the patch tomorrow.

[Manuzhai]

I thought, what the hell, I'll give it a whirl right now.

And it seems to work alright! We should probably give it some more time, but it looks good. I used a clean 1.3.x Subversion checkout from revision 18687 to apply the patch to, and then followed David's recipe.

(I still wonder why other people haven't been able to reproduce it, though.)

[mark@…]

I have been able to reproduce this. See the previous bug reports on that topic.

Due to this we still use SVN 1.2.x …

[Manuzhai]

Well, sure, some have had the same errors, but not cboos or djames.

[anonymous]

Manzuhai, can you still reproduce the issue without my patch? Is it clear that the patch fixes the problem?

Neither me nor cboos have been able to reproduce this bug, with or without the patches, so it's possible that the problem went away on your machine for an unrelated reason.

[Manuzhai]

I changed nothing else except your patch, I've been seeing the problem for ages, and it is now gone. I'm fairly certain that it's your patch that fixed it.

[dg]

Just 2 days ago I discovered the very same problem. It occurs only rarely, so I am not suprised not everyone has seen it. I have seen it on our sever machine (single CPU linux), and on my development machine (single CPU linux)

• mod_python-3.2.7
• python-2.4.2
• subversion 1.3.0
• swig 1.3.27
• apache 2 with prefork MPM
• linux

Everything upatched as yet.

Error from log:

[Sat Mar 04 13:47:20 2006] [error] Error(TypeError): argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x430bb02c>)' is received at /usr/local/lib/svn-python/libsvn/core.py line 1098
[Sat Mar 04 13:47:20 2006] [notice] child pid 15649 exit signal Segmentation fault (11)

I am attempting to use the python subversion bindings in a custom application to manage text document revisions - until I saw this problem, everything seemed just great.

Now I am not sure what to do - should I abandon using subversion at all for this?

[mark@…]

We can exclude Apache's MPM: I've managed to reproduce this issue on MPM worker.

As version 1.3.0 is a constant, here, you could downgrade SVN to 1.2.3.

And well, none of the above patches worked for me.

[dg]

Well, good news. I applied the patch according to the recipe (application_pool_race_condition_fix.txt), and the problem seems to have gone away!

(same configuration as before, using 1.3.0)

I have not tested this on our server yet, but on my local machine running 10 browser-emulation processes banging away at the local server app (each making 200 requests), not a single error! (2000 requests in about 10 minutes)

Will try this on our server late tonight.

[Manuzhai]

mark@…: it would seem that yours is a different problem, then? As the patches fixes the problem both for me and dg.

[chasetheblues@…]

I'm experiencing the same problems after upgrading

• Bindings to SVN 1.3.0
• Python from 2.3 to 2.4
• SWIG from 1.3.2x(?) to 1.3.28.

Using

• Trac 0.9.4 with
• mod_python 3.2.8
• Python 2.4.

[chasetheblues@…]

forgot:

• Python 2.4.2
• apr 0.9.7 (from apache-2.0.55)
• Debian Linux

[Christian Boos]

chasetheblues: have you tried to apply the attachment:application_pool_race_condition_fix.txt​ and rebuild your bindings as explained in the comment above?

[chasetheblues]

yes, i applied it, and had the make procedure. but it doesn't help. i hence cannot except 100% the possibility of something went wrong at fixing. i got some compile errors, before the make procedure went through in the end.

on some small test projects, the error is shown 10 times in a row before the page is loaded correctly, on large test projects i have to try several dozen page request before any problem.

[Manuzhai]

The same error does still turn up at times for me too, although it now seems much less frequent than before. *shrug* It's kinda weird, it seems really quite unpredictable.

[Manuzhai]

Okay, I have obviously been too optimistic about the patch.

Subversion still dies on me regularly with the same error. djames, how can I help you diagnose this problem? I can probably provide you with access to one of my servers if that would help.

[dg]

Just an update - after applying the patch March 4, we haven't seen the problem since. And we get pretty extensive use of the library, as our online editing system is based on this package.

Wish I could help with something more concrete….

[ctb]

The Problem still exists, even after upgrading to svn 1.3.1 and downgrading to swig 1.3.25 and reinstalling trac.

[anonymous]

Did you already report this to the ​Subversion's issue tracker?

[Christian Boos]

The problems seems to be there also for 1.3.1 on windows, see #2965.

[Christian Boos]

In order to get some progress on this topic, I'd like that the people who are still seeing the issue report here if they are using multiple virtual hosts, as #2965 suggests that this might trigger the problem.

Also mention if you're using:

1. svn 1.3.1
2. svn 1.3.0 + patch​, built following those ​instructions
3. something else (hm… you should have tried one of the above!)

[blaufalke@…]

• SVN 1.3.0 (no patching so far).
• Several virtual hosts on Apache 2.2.0 (SuSE Linux 9.3).
• Only one virtual host used for trac (via name based subdomains)

• Subdomains configured like that:

   <Directory /path/to/httpsdocs>
    Satisfy Any
    Require valid-user
    AuthType Basic
    AuthName "Subversion@mydomain.com"
    AuthUserFile /path/to/.htpasswd
   </Directory>
  
   RedirectMatch permanent ^/(projects)$ https://repos.mydomain.com/$1/
   RedirectMatch permanent ^/(projects)/([^/]+)$ https://repos.mydomain.com/$1/$2/
  
   <Location /svn/>
    DAV svn
    SVNParentPath /path/to/repositories
    AuthzSVNAccessFile /path/to/.policy
   </Location>
  
   <Location /projects/>
    SSLRequireSSL
    setHandler mod_python
    PythonPath "sys.path + ['/usr/share/trac']"
    PythonHandler trac.web.modpython_frontend
    PythonOption TracEnvParentDir /path/to/projects/
    PythonOption TracUriRoot /projects/
    SetEnv PYTHON_EGG_CACHE /usr/lib/python2.4/site-packages
    SVNParentPath /path/to/repositories
    AuthzSVNAccessFile /path/to/.policy
   </Location>
  

[mark@…]

Common settings:

• Several virtual hosts.
  Every configured as described in TracModPython.
• Trac 0.9.[2-5]
• Python 2.4.2
• SQLite 3
• Apache: mpm_worker, PHP 5.1.x

Tested configurations at which the errors occurs:

• Subversion 1.3.0 (with and without patch, no relief)
  • Apache 2.0.55, SWIG 1.3.21, APR 0.9.7
  • Apache 2.0.55, SWIG 1.3.21, APR 0.9.12
  • Apache 2.0.58, SWIG 1.3.25, APR 0.9.12
  • Apache 2.2.0, SWIG 1.3.21, APR 1.2.2
  • Apache 2.2.2, SWIG 1.3.25, APR 1.2.7
• Subversion 1.3.1 (with/without)
  • Apache 2.0.58, SWIG 1.3.25, APR 0.9.12
  • Apache 2.2.0, SWIG 1.3.21, APR 1.2.2
  • Apache 2.2.0, SWIG 1.3.25, APR 1.2.2
  • Apache 2.2.0, SWIG 1.3.25, APR 1.2.7

Drop me a line if I got the "tester's award".

[mark@…]

That's what I've forgotten to mention:

• Apache 2.0.x: mod_python 3.1.4
• Apache 2.2.x: mod_python 3.2.8

Everything compiled under ​Gentoo with GCC 3.4.x, -fomit-frame-pointer -pipe and either:

• -Os -march=athlon-mp -ffast-math
• -O2 -mtune=pentium-m -fforce-addr -frename-registers -falign-functions=64

[mark@…]

SVN 1.3.1 has always been without patch - It didn't even apply.

[Christian Boos]

Actually, that patch was merged in Subversion 1.3.1, see TracSubversion for details.

Mark, just in order to verify the hypothesis that using multiple virtual hosts could trigger the problem, would it be possible for you to setup a test environment with one/no virtual hosts?

(of course, that will earn you even more points for the tester-of-the-year contest :) )

[mark@…]

I've had only the time for one test, so here it is:

Environment

• Apache 2.0.58, APR 0.9.12, mod_python 3.1.4 (r1)
• Subversion 1.3.1, SWIG 1.3.21

one vhost:

Fire up Apache, click every of the items in mainnav (wiki, timeline…). No error, no matter how much items you invoke.

second vhost:

preparations

Stop Apache, copy 10_trac.conf to 10_something.conf and adjust path and hostname.
Create a new subversion repo, import something into it and run trac-admin /var/lib/trac/something initenv.

Add another line to /etc/hosts to match your new vhost:

127.0.0.1 something.localhost

invocation

Fire up Apache, click through the first installed vhost. No errors will appear, believe me.

And now has the time come to open up the second vhost. Type something.localhost in your browser, get the first page displayed and change into "timeline" or "browse source"! Voila, fatal error appears.

[Christian Boos]

Ok, thanks for the confirmation and for the steps to reproduce the issue.

[Manuzhai]

There's something to the vhost diagnosis, but it's not everything.

On my old server, I had a single vhost containing different Location sections, for different projects, replicating the full setup for each config (e.g. SetHandler, several PythonOption directives, and authentication). I encountered the errors there, which is why I opened this ticket. On my new server, however, I have set it up differently, just using the multiple env support from Trac within a vhost, and I get no more errors.

[anonymous]

with subversion 1.3.2,this problem still exists.

[Alec Thomas]

I upgraded TracHacks to SVN 1.3.1 and was not able to get Apache to serve any Trac requests at all. Quite aggravating. Downgraded to 1.2.4 until fix is merged.

[Manuzhai]

This is really a Subversion problem. We should still gather more information here, but let's set the milestone to 0.11; it really doesn't block 0.10.

[maxb1@…]

I have what seems to be a reliable reproduction recipe: You must have an apache setup which contains Trac instances in *multiple* mod_python subinterpreters. Restart apache. The first request to a svn-using page (i.e. "Browse Source") within the first subinterpreter to be accessed will SUCCEED. The first request to a svn-using page within a second subinterpreter will fail with the error.

Additionally, just as an interesting data point: I actually got this error running a totally non-Trac-related command line script, once (it was ViewVC's svndbadmin). Sadly, this was not reproducable.

[maxb1@…]

If anyone has seen this error with SWIG 1.3.28 or 1.3.29, please comment!

The reliable reproduction I mentioned in the previous comment was using the pregenerated SWIG files shipped with subversion 1.3.2 (i.e. SWIG 1.3.25 IIRC). I retested with SWIG 1.3.28, and the problem seems to have disappeared!

Further, I have isolated the source of the TypeError to the following code segment in _wrap_svn_pool_create:

        if (_global_svn_swig_py_pool != NULL &&
            !PyObject_HasAttrString(_global_svn_swig_py_pool,
                                    (char *)"_mark_valid"))
        {
            int argnum = PyTuple_GET_SIZE(args);
            SWIG_Python_TypeError(SWIG_TypePrettyName(SWIGTYPE_p_apr_pool_t),
                                  _global_svn_swig_py_pool);
            SWIG_arg_fail(argnum);
            SWIG_fail;
        }

[marc@…]

I have the same problem, and I'm running

• apache 2.0.54
• mod_python 3.1.4
• subversion 1.3.2
• python 2.4.1
• trac 0.9.4

I have about 15 tracs running under a virtualhost each (on port 81), and get intermittent errors. I have an RSS reader configured to request the Timeline page for each trac and it always gets an error for at least one of the tracs.

When I tried the setup with a single virtualhost I got no errors, but I have not tested that extensively.

If there's anything else that I could do or provide just let me know and I will try.

[anonymous]

Replying to marc@guidance.nl:

I have the same problem, and I'm running

• apache 2.0.54
• mod_python 3.1.4
• subversion 1.3.2
• python 2.4.1
• trac 0.9.4

I have about 15 tracs running under a virtualhost each (on port 81), and get intermittent errors. I have an RSS reader configured to request the Timeline page for each trac and it always gets an error for at least one of the tracs.

When I tried the setup with a single virtualhost I got no errors, but I have not tested that extensively.

If there's anything else that I could do or provide just let me know and I will try.

Instead of using virtual hosts, you can use another apache "instance" with another config file and on another TCP port or IP address. Something like this: /usr/sbin/apachectl -f /etc/httpd/conf/httpd.trac.conf

For Trac I have one apache (default) instance for intranet running on port 80, and another apache instance for internet running on port 81, and it works fine.

Before this I had only one apache instance running on ports 80 and 81, and it give this error.

[Christian Boos]

In light of ​#3455, it seems clear that the fix in attachment:application_pool_race_condition_fix.txt​ is not working in case of multiple Python subinterpreters (as this is the case by default with virtual hosts).

The svn_swig_py_set_application_pool(self, self) will be called for each sub-interpreter and therefore at the C level one pool object coming from one subinterpreter will be used by all the other subinterpreters. This probably leads to an invalid GC, which destroys that object prematurely, and that deletes the "_is_valid" string which triggers the message we're seeing, see comment:64).

So… the workaround for #3455 should most probably also work here.

It remains to be confirmed if the 1.3.2 bindings generated with SWIG ≥ 1.3.28 are handling the multiple sub-interpreter scenario correctly.

[anonymous]

We have mod_python-3.1.4, apache-2.0.58, swig-1.3.29, subversion-1.3.2, python-2.4.3, trac-0.10rc1. (gcc 4.1.1)

I have been working on converting a bunch of trac sites from 0.8 to 0.10rc1 (we will be moving to 0.10). We have hitting this since converting to gcc-4.1.1. That was a one-way change, but this is crazy. I have no idea what to do =(.

Apache is in mpm-prefork. We have disactivated about everything that we can, and I just implented the

PythonInterpreter "trac"

in my config. We have 5 sites.

[anonymous]

BTW: I guess I should mention - we are hitting this bug.

[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: Traceback (most recent call last):
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch\n    result = object(req)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 87, in handler\n    gateway.run(dispatch_request)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 304, in dispatch_request\n    env = _open_environment(env_path, run_once=environ['wsgi.run_once'])
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 57, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/env.py", line 431, in open_environment\n    env = Environment(env_path)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/env.py", line 121, in __init__\n    load_components(self)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/loader.py", line 140, in load_components\n    __import__(module)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 56, in ?\n    from svn import fs, repos, core, delta
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/svn/fs.py", line 19, in ?\n    from libsvn.fs import *
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/fs.py", line 29, in ?\n    import core
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 3057, in ?\n    svn_pool_create()
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 1098, in svn_pool_create\n    return apply(_core.svn_pool_create, args)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0xb6924c8c>)' is received

[Christian Boos]

Replying to anonymous:

We have mod_python-3.1.4, apache-2.0.58, swig-1.3.29, subversion-1.3.2, python-2.4.3, trac-0.10rc1. (gcc 4.1.1)

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

PythonInterpreter "trac"

Try "main_interpreter", as suggested in #3455.

[joshland@…]

Replying to cboos:

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

I think this worked. We are returning to mpm-worker mode. Our current software loadout is:

swig-1.3.25, subversion-1.3.2-r3, apache-2.0.58 (mpm-worker) and mod_python-3.1.4

Try "main_interpreter", as suggested in #3455.

my mod_python config is now

    SetHandler    mod_python
    PythonInterpreter main_interpreter
    PythonPath    "['/var/trac/lib/python2.4/site-packages'] + sys.path"
    PythonHandler trac.web.modpython_frontend
    PythonOption  TracEnvParentDir            "/var/live/trac"
    PythonOption  TracUriRoot                 /
    SetEnv        PYTHON_EGG_CACHE            "/var/live/egg_cache"

[anonymous]

Replying to joshland@gmail.com:

Replying to cboos:

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

I think this worked. We are returning to mpm-worker mode. Our current software loadout is:

swig-1.3.25, subversion-1.3.2-r3, apache-2.0.58 (mpm-worker) and mod_python-3.1.4

Try "main_interpreter", as suggested in #3455.

my mod_python config is now

    SetHandler    mod_python
    PythonInterpreter main_interpreter
    PythonPath    "['/var/trac/lib/python2.4/site-packages'] + sys.path"
    PythonHandler trac.web.modpython_frontend
    PythonOption  TracEnvParentDir            "/var/live/trac"
    PythonOption  TracUriRoot                 /
    SetEnv        PYTHON_EGG_CACHE            "/var/live/egg_cache"

Because this person is using mod_python 3.1.4 and not mod_python 3.2.8 or later, using 'main_interpreter' will be no different to having used 'trac' as the interpreter name.

There are really two parts to trying to reduce these sorts of problems, that is even if this is a valid workaround for the original problem this person is having.

1. Force all use of Trac/subversion to run under the same Python interpreter instance under mod_python. This would be necessary where for some reason one or more Trac instances were being accessed through different VirtualHost configurations within the Apache setup. This avoids issue where internal Python objects associated with the subversion bindings are somehow bound to a specific Python interpreter instance and therefore cannot be used across multiple Python interpreter instances. This is only going to be effective where access was through multiple contexts, such as VirtualHost, where mod_python would have executed the request within a different interpreter instance context. Doing this would otherwise make no difference.

2. If using mod_python 3.2.8 or later, you can further increase your chances of things working by avoiding possible GIL issues caused by use of simplified Python locking API for the GIL by Python extensions, by forcing the very first Python interpreter instance created by mod_python to be used. This particular interpreter instance is called 'main_interpreter'. If using mod_python prior to 3.2.8, there is no means of specifically indicating the first interpreter created should be used, using 'main_interpreter' doesn't indicate that for those older versions. This change may have benefit even if access isn't through multiple interpreters in the first place, but it will only make a difference if you have mod_python 3.2.8 or later.

[joshland@…]

BTW: It didn't work, it just lasted for about 30 minutes. It is definatly still failing. I am looking at the last post, and I will attempt to fix this problem following your guidelines.

[joshland@…]

We upgraded to mod_python 3.2, and downgraded to svn 1.2. Problems have since gone away.

[Christian Boos]

Right, the problem appeared with Subversion 1.3, so it's likely that downgrading to 1.2.4 makes the problem go away… However, it's a bit sad that this wasn't addressed in newer Subversion releases (there's been 1.3.1, 1.3.2 and now 1.4.0 in the meantime).

I wonder if djames is still seeing updates to this ticket (and if he's still taking care of the python bindings for Subversion).

[neofutur]

Same problem here whith 1 vhosts

When i deleted the second vhost, no more problem I had to change the second vhost to another server

[anonymous]

I downgraded to svn 1.2.3 until this is resolved.

[joshland@…]

I just hit a similar segfault w/ 1.2.3 I am running Gentoo, Apache-2.2 and Mod Python 3.2. I don't have any tracebacks or anything. Is anyone else experiencing problems with this?

I am about to switch to nginx + tracd and see if that fixes everything.

[Christian Boos]

Replying to joshland@gmail.com:

I just hit a similar segfault w/ 1.2.3 I am running Gentoo, Apache-2.2 and Mod Python 3.2. I don't have any tracebacks or anything. Is anyone else experiencing problems with this?

It might be something different… this issue was never reported against 1.2.3 so far. So if you don't have the TypeError: argument ... error, I bet it's something else.

I am about to switch to nginx + tracd and see if that fixes everything.

TracStandalone is not affected by this issue, so if the segfault happens again, you'll know that it was something else. In case it happens, you might want to follow the advices given in TracTroubleshooting in order to locate the cause of the problem.

[boris@…]

I have a hunch that it is in fact related to multiple vhosts running on the same apache instance. I'm starting to see errors of this nature crop up in my logs only after adding a second TracModPython virtual host.

I tried Trac 0.10, and SVN 1.3.1, 1.3.2.

[joshland@…]

We will setup nginx + tracd this week and, provided that everything continues to work as expected, post a recipe on the nginx wiki (and the Trac wiki if possible).

[Christian Boos]

Shifting the milestone to 2.0, as there's nothing we can do for 0.11.

Closing this issue is not an option either, as this is a recurring problem in a basic piece of infrastructure often needed by Trac. Probably an "Undefined" milestone would be appropriate here.