Edgewall Software

Opened 16 years ago

Last modified 2 years ago

#2611 new defect

Problem with SVN bindings (SVN 1.3.0, Trac r2771) — at Version 84

Reported by: Manuzhai Owned by: Christian Boos
Priority: high Milestone: not applicable
Component: version control Version: 0.10.4
Severity: major Keywords: svn13 svn14 svn15 svn16
Cc: joshland@…, manuzhai@…, odela01@…, pjenvey@…, shishz@…, james82@…, mark@…, dgeller@…, chasetheblues@…, brian@…, blaufalke@…, marc@…, neofutur@…, bock@…, boris@…, migs.ho@…, progrium@…, Thijs Triemstra Branch:
Release Notes:
API Changes:
Internal Changes:

Description (last modified by Christian Boos)

Every morning when I first try to get to Trac, I get this error on the first request:

Mod_python error: "PythonHandler trac.web.modpython_frontend"

Traceback (most recent call last):
...
  File "/home/manuzhai/dev/trac/trac/versioncontrol/svn_fs.py", line 31, in ?
    from svn import fs, repos, core, delta
  File "/usr/lib/python2.4/site-packages/svn/fs.py", line 19, in ?
    from libsvn.fs import *
  File "/usr/lib/python2.4/site-packages/libsvn/fs.py", line 29, in ?
    import core
  File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 3049, in ?
    svn_pool_create()
  File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 1098, in svn_pool_create
    return apply(_core.svn_pool_create, args)

TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance()' is received

Similar symptoms:

[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0xb6924c8c>)' is received

Subsequent requests seem to just work…

Note

This is now a well-known issue having to do with the way the SVN Python bindings behaves when multiple Python interpreters are used within the same process, as this can be the case with mod_python e.g. when more than one virtual host is used.

Most SVN binding versions are affected, though there's no confirmed report of this with 1.2.x (the bug started to be reported with 1.3.0) and reverting to SVN 1.2.4 is sometimes mentioned as a workaround.

There are other documented workarounds where one can keep using SVN 1.3.x and 1.4.x:

Related issue: #3371

Change History (85)

comment:1 by Christian Boos, 16 years ago

Milestone: 0.9.4
Priority: normalhigh
Severity: normalmajor

See also #2472, where this error was also reported and discussed. As that other ticket was about another issue with the 1.3.0 bindings, it has been closed and this ticket should be used to focus on the 'apr_pool_t *' is expected issue.

comment:2 by anonymous, 16 years ago

Cc: odela01@… added

comment:3 by pjenvey@…, 16 years ago

Cc: pjenvey@… added

comment:4 by anonymous, 16 years ago

Cc: shishz@… added

comment:5 by Manuzhai, 16 years ago

This error seems to appear quite often on editing a wiki page. Refreshing the edit (resending POST data) gives the same error again, going back to the wiki page, then sending it again too, hard-refreshing the page then redoing the changes and submitting them still seems to work. Not sure whether this is always true…

comment:6 by Christian Boos, 16 years ago

I've not been able to investigate and reproduce this myself, but could you try the following?

Index: svn_fs.py
===================================================================
--- svn_fs.py	(revision 2819)
+++ svn_fs.py	(working copy)
@@ -28,7 +28,7 @@
 from trac.versioncontrol.svn_authz import SubversionAuthorizer
 
 try:
-    from svn import fs, repos, core, delta
+    from svn import core, fs, repos, delta
     has_subversion = True
 except ImportError:
     has_subversion = False

comment:7 by Manuzhai, 16 years ago

I've made the change, and will report back to see if it helps.

comment:8 by anonymous, 16 years ago

Cc: james82@… added

I installed mod_python 3.1.4 / Python 2.3 / Subversion 1.3.0 on a single processor RHEL4 machine and I can't reproduce this issue. Ideas?

comment:9 by pjenvey@…, 16 years ago

I am seeing this problem with python 2.4. The original bug reported to the subversion-users list also occured under python 2.4

Myabe the fact that you're using python 2.3 is a factor, or possibly some other dependency version mismatch?

comment:10 by Manuzhai, 16 years ago

Yes, I'm also using Python 2.4. The change of import order doesn't solve all of the problem, though it does seem that it doesn't happen as often now. That might have to do with my usage pattern, though…

comment:11 by Christian Boos, 16 years ago

Can you try to split the import in several lines, i.e.

try:
    from svn import core
    from svn import fs
    # etc.

comment:12 by Manuzhai, 16 years ago

Done…

comment:13 by Christian Boos, 16 years ago

Owner: changed from Christopher Lenz to Christian Boos

I think I understand the issue: having all the imports in one line allows two threads (A and B) to start simultaneously the import process:

  • A: trac/versioncontrol/svn_fs.py: from svn import fs, repos, core, delta
  • B: trac/versioncontrol/svn_fs.py: from svn import fs, repos, core, delta
  • A: in svn/fs.py, line 19: from libsvn.core import *,
  • A: now in libsvn/core.py, is going through the > 3000 lines…
  • B: in svn/fs.py, line 19: from libsvn.core import *,
  • B: now in libsvn/core.py, is going through the > 3000 lines…
  • A: in libsvn/core.py, line 3049: does a _core.svn_pool_create
    and here, _core is really _core (i.e. the native extension)
  • A: in svn/fs.py, line 29: import svn.core as _core
  • B: in libsvn/core.py, line 3049: does a _core.svn_pool_create
    and now, _core is not _core but svn.core and therefore, _core.svn_pool_create is not the native function but the (deprecated) compatibility function, which expects a pool as first argument… hence the error.

Hopefully splitting the imports does solve the issue, but the bindings should really be more robust about that and not rename different things to a same name…

comment:14 by mark@…, 16 years ago

Cc: mark@… added

comment:15 by james82@…, 16 years ago

I have not been able to reproduce the problem (even with Python 2.4 and mod_python 3.1.3), but, I think I have figured out how to fix it. Please try the below patch (for Subversion 1.3.0).

  • subversion/bindings/swig/include/svn_types.swg

     
    701701
    702702%exception {
    703703#ifdef SWIGPYTHON
    704     svn_swig_py_release_py_lock();
     704    int released_lock = 0;
     705    if (_global_svn_swig_py_pool != NULL) {
     706      released_lock = 1;
     707      svn_swig_py_release_py_lock();
     708    }
    705709#endif
    706710    $action
    707711#ifdef SWIGPYTHON
    708     svn_swig_py_acquire_py_lock();
     712    if (released_lock) {
     713      svn_swig_py_acquire_py_lock();
     714    }
    709715#endif
    710716}
    711717

comment:16 by james82@…, 16 years ago

Important note: In order for the above patch to take effect, you'll need to install SWIG 1.3.25, and run the following commands:

   make extraclean
   ./autogen.sh
   ./configure
   make swig-py
   make install-swig-py

Note that you need to make extraclean to remove any old headers shipped with the tarball.

comment:17 by pjenvey@…, 16 years ago

Manuzhai -

Did splitting the import into multiple lines solve the problem?

If not, I'll see if I can give james' patch a shot later today

comment:18 by Manuzhai, 16 years ago

Splitting the import didn't solve the problem.

James' fix is a little involved for me, I just install SVN & bindings through emerge, I'm afraid. I could try it, but I'm not sure when I get to it.

comment:19 by pjenvey@…, 16 years ago

james - I upgraded to swig 1.3.27, applied your patch and did the whole make extraclean/autogen. Now I am noticing a different, similar error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 206, in handler
    dispatch_request(mpr.path_info, mpr, env)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/web_ui/browser.py", line 91, in process_request
    repos = self.env.get_repository(req.authname)
  File "/usr/local/lib/python2.4/site-packages/trac/env.py", line 157, in get_repository
    authname)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/api.py", line 61, in get_repository
    return self._connector.get_repository(repos_type, repos_dir, authname)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 205, in get_repository
    repos = SubversionRepository(dir, authz, self.log)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 224, in __init__
    self.path = repos.svn_repos_find_root_path(path, self.pool())
  File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43, in svn_repos_find_root_path
    return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x8b88f6c>)' is received

Instead of seeing the stack trace in my error log (where I saw the original apr_pool_t exception), it shows up on an actual trac "Oops… Trac detected an internal error:" error page. I can't tell if this is due to your patch, the swig upgrade, or both

This new error is intermittent like the old one. Occasionally I receive either 404 responses or blank pages instead of the error message. Nothing in the error_log

Also: I got the impression the make extraclean/reautogen would end up creating a brand new subversion/includes/svn_types.h file corresponding to your change to svn_types.swg. That was not the case — the patch was successfully applied, but I ended up with the same svn_types.h file. Let me know if that is actually a problem

comment:20 by David James <james82@…>, 16 years ago

Hi pjenvey,

It looks like you applied the patch correctly, but my patch did not help. I am not surprised by your new error message, because it is the same as the one that was reported earlier on the users list.

Could you send the full details for your system, and the full details of your apache configuration including your httpd.conf and a description of which "mpm" you are using (e.g. the worker mpm, or the prefork mpm). I would like to try to reproduce the problem by copying your configuration. Are you using Linux, Mac OS, or something else?

Here's a question for the group: In mod_python, does the "global interpreter lock" lock just the current interpreter or does it lock all interpreters? If the "global interpreter lock" only locks the current interpreter, then it explains our problem. We will need to use a different locking system in order to prevent race conditions during pool initialization. Perhaps an APR lock would do the trick?

Cheers,

David

comment:21 by Manuzhai, 16 years ago

From http://docs.python.org/api/threads.html (and my understanding), the GIL just locks all but one threads within one interpreter. So if multiple interpreters are at work, we need a different locking mechanism (in which case the APR lock would seem like the best solution).

comment:22 by David James <james82@…>, 16 years ago

Hi pjenvey, Manzuhai:

After thinking about this problem some more, I have written a patch which uses Python locking to solve the race condition in application pool initialization. Could you give this patch a try?

  • subversion/bindings/swig/include/proxy_apr.swg

     
    6060 */
    6161
    6262%pythoncode %{
     63import threading
     64
    6365application_pool = None
     66application_pool_lock = threading.Lock()
    6467class GenericSWIGWrapper:
    6568  def __init__(self, this, pool):
    6669    """Create new Generic SWIG wrapper object"""
     
    98101      def set_parent_pool(self, parent_pool=None):
    99102        """Create a new memory pool"""
    100103        global application_pool
     104
     105        try:
     106          application_pool_lock.acquire()
     107
     108          self._parent_pool = parent_pool or application_pool
     109          self._mark_valid()
     110   
     111          # Protect important functions from GC
     112          self._apr_pool_destroy = _core.apr_pool_destroy
     113          self._svn_swig_py_clear_application_pool = \
     114            _core.svn_swig_py_clear_application_pool
     115   
     116          # If we are an application-level pool,
     117          # then set this pool to be the application-level pool
     118          if not self._parent_pool:
     119            svn_swig_py_set_application_pool(self, self)
     120            application_pool = self
     121        finally:
     122          application_pool_lock.release()
    101123 
    102         self._parent_pool = parent_pool or application_pool
    103         self._mark_valid()
    104  
    105         # Protect important functions from GC
    106         self._apr_pool_destroy = _core.apr_pool_destroy
    107         self._svn_swig_py_clear_application_pool = \
    108           _core.svn_swig_py_clear_application_pool
    109  
    110         # If we are an application-level pool,
    111         # then set this pool to be the application-level pool
    112         if not self._parent_pool:
    113           svn_swig_py_set_application_pool(self, self)
    114           application_pool = self
    115  
    116124      def valid(self):
    117125        """Check whether this memory pool and its parents
    118126        are still valid"""

Important note: Once again, in order for this patch to take effect, you'll need to type:

   make extraclean
   ./autogen.sh
   ./configure
   make swig-py
   make install-swig-py

comment:23 by Manuzhai, 16 years ago

Could you attach this patch to the ticket insteading of showing it inline? I'm having a hard time getting it applied correctly. Thanks!

by David James <james82@…>, 16 years ago

Fix race condition in application pool initialization

comment:24 by David James <james82@…>, 16 years ago

Hi Manzuhai,

The patch is attached. I've also posted it to the Subversion developers list, at http://svn.haxx.se/dev/archive-2006-01/0946.shtml

Cheers,

David

comment:25 by pjenvey@…, 16 years ago

David -

Unfortunately I'm getting the same error I reported last time with your latest patch. (scroll up for the entire thing)

 File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43, in svn_repos_find_root_path
    return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x92633ac>)' is received

Here's my specs:

FreeBSD 4
python 2.4.2
trac 1.0dev (r2721)
subversion 1.3.0 (with or without your latest patch, and cboos's small __del__ patch in #2472 which shouldn't make a difference)
swig 1.3.27
apr 0.9.7 (from apache-2.0.55)
neon 0.24.7

comment:26 by pjenvey@…, 16 years ago

Summary: Problem with SVN bindings (SVN 1.3.0, Trac r2771)Problem with SVN bindings (SVN 1.3.0, Trac r2771)

Also —

prefork MPM

comment:27 by Christian Boos, 15 years ago

Milestone: 0.9.40.10
Summary: Problem with SVN bindings (SVN 1.3.0, Trac r2771)Problem with SVN bindings (SVN 1.3.0, Trac r2771)

I've been unable to actually reproduce the problem (apache-2.0.55, mod_python-3.1.4, Subversion-1.3.0 and Python-2.4.2 on Linux SLES9 64bits (did I say it's reasonably fast?)).

So I'm leaving this open for further investigation or feedback, but post-poning it.

comment:28 by Manuzhai, 15 years ago

This is still annoying. I'll see if I can try the patch tomorrow.

comment:29 by Manuzhai, 15 years ago

I thought, what the hell, I'll give it a whirl right now.

And it seems to work alright! We should probably give it some more time, but it looks good. I used a clean 1.3.x Subversion checkout from revision 18687 to apply the patch to, and then followed David's recipe.

(I still wonder why other people haven't been able to reproduce it, though.)

comment:30 by mark@…, 15 years ago

I have been able to reproduce this. See the previous bug reports on that topic.

Due to this we still use SVN 1.2.x …

comment:31 by Manuzhai, 15 years ago

Well, sure, some have had the same errors, but not cboos or djames.

comment:32 by anonymous, 15 years ago

Manzuhai, can you still reproduce the issue without my patch? Is it clear that the patch fixes the problem?

Neither me nor cboos have been able to reproduce this bug, with or without the patches, so it's possible that the problem went away on your machine for an unrelated reason.

comment:33 by Manuzhai, 15 years ago

I changed nothing else except your patch, I've been seeing the problem for ages, and it is now gone. I'm fairly certain that it's your patch that fixed it.

comment:34 by dg, 15 years ago

Just 2 days ago I discovered the very same problem. It occurs only rarely, so I am not suprised not everyone has seen it. I have seen it on our sever machine (single CPU linux), and on my development machine (single CPU linux)

  • mod_python-3.2.7
  • python-2.4.2
  • subversion 1.3.0
  • swig 1.3.27
  • apache 2 with prefork MPM
  • linux

Everything upatched as yet.

Error from log:

[Sat Mar 04 13:47:20 2006] [error] Error(TypeError): argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x430bb02c>)' is received at /usr/local/lib/svn-python/libsvn/core.py line 1098
[Sat Mar 04 13:47:20 2006] [notice] child pid 15649 exit signal Segmentation fault (11)

I am attempting to use the python subversion bindings in a custom application to manage text document revisions - until I saw this problem, everything seemed just great.

Now I am not sure what to do - should I abandon using subversion at all for this?

comment:35 by dg, 15 years ago

Cc: dgeller@… added

comment:36 by mark@…, 15 years ago

We can exclude Apache's MPM: I've managed to reproduce this issue on MPM worker.

As version 1.3.0 is a constant, here, you could downgrade SVN to 1.2.3.

And well, none of the above patches worked for me.

comment:37 by dg, 15 years ago

Well, good news. I applied the patch according to the recipe (application_pool_race_condition_fix.txt), and the problem seems to have gone away!

(same configuration as before, using 1.3.0)

I have not tested this on our server yet, but on my local machine running 10 browser-emulation processes banging away at the local server app (each making 200 requests), not a single error! (2000 requests in about 10 minutes)

Will try this on our server late tonight.

comment:38 by Manuzhai, 15 years ago

mark@…: it would seem that yours is a different problem, then? As the patches fixes the problem both for me and dg.

comment:39 by chasetheblues@…, 15 years ago

Cc: chasetheblues@… added

I'm experiencing the same problems after upgrading

  • Bindings to SVN 1.3.0
  • Python from 2.3 to 2.4
  • SWIG from 1.3.2x(?) to 1.3.28.

Using

  • Trac 0.9.4 with
  • mod_python 3.2.8
  • Python 2.4.

comment:40 by chasetheblues@…, 15 years ago

forgot:

  • Python 2.4.2
  • apr 0.9.7 (from apache-2.0.55)
  • Debian Linux

comment:41 by Christian Boos, 15 years ago

chasetheblues: have you tried to apply the attachment:application_pool_race_condition_fix.txt and rebuild your bindings as explained in the comment above?

comment:42 by chasetheblues, 15 years ago

yes, i applied it, and had the make procedure. but it doesn't help. i hence cannot except 100% the possibility of something went wrong at fixing. i got some compile errors, before the make procedure went through in the end.

on some small test projects, the error is shown 10 times in a row before the page is loaded correctly, on large test projects i have to try several dozen page request before any problem.

comment:43 by Manuzhai, 15 years ago

The same error does still turn up at times for me too, although it now seems much less frequent than before. *shrug* It's kinda weird, it seems really quite unpredictable.

comment:44 by Manuzhai, 15 years ago

Okay, I have obviously been too optimistic about the patch.

Subversion still dies on me regularly with the same error. djames, how can I help you diagnose this problem? I can probably provide you with access to one of my servers if that would help.

comment:45 by dg, 15 years ago

Just an update - after applying the patch March 4, we haven't seen the problem since. And we get pretty extensive use of the library, as our online editing system is based on this package.

Wish I could help with something more concrete….

comment:46 by ctb, 15 years ago

The Problem still exists, even after upgrading to svn 1.3.1 and downgrading to swig 1.3.25 and reinstalling trac.

comment:47 by anonymous, 15 years ago

Did you already report this to the Subversion's issue tracker?

comment:48 by Christian Boos, 15 years ago

The problems seems to be there also for 1.3.1 on windows, see #2965.

comment:49 by anonymous, 15 years ago

Cc: brian@… added

comment:50 by anonymous, 15 years ago

Cc: blaufalke@… added

comment:51 by Christian Boos, 15 years ago

Keywords: svn130 added

In order to get some progress on this topic, I'd like that the people who are still seeing the issue report here if they are using multiple virtual hosts, as #2965 suggests that this might trigger the problem.

Also mention if you're using:

  1. svn 1.3.1
  2. svn 1.3.0 + patch, built following those instructions
  3. something else (hm… you should have tried one of the above!)

comment:52 by blaufalke@…, 15 years ago

  • SVN 1.3.0 (no patching so far).
  • Several virtual hosts on Apache 2.2.0 (SuSE Linux 9.3).
  • Only one virtual host used for trac (via name based subdomains)
  • Subdomains configured like that:
     <Directory /path/to/httpsdocs>
      Satisfy Any
      Require valid-user
      AuthType Basic
      AuthName "Subversion@mydomain.com"
      AuthUserFile /path/to/.htpasswd
     </Directory>
    
     RedirectMatch permanent ^/(projects)$ https://repos.mydomain.com/$1/
     RedirectMatch permanent ^/(projects)/([^/]+)$ https://repos.mydomain.com/$1/$2/
    
     <Location /svn/>
      DAV svn
      SVNParentPath /path/to/repositories
      AuthzSVNAccessFile /path/to/.policy
     </Location>
    
     <Location /projects/>
      SSLRequireSSL
      setHandler mod_python
      PythonPath "sys.path + ['/usr/share/trac']"
      PythonHandler trac.web.modpython_frontend
      PythonOption TracEnvParentDir /path/to/projects/
      PythonOption TracUriRoot /projects/
      SetEnv PYTHON_EGG_CACHE /usr/lib/python2.4/site-packages
      SVNParentPath /path/to/repositories
      AuthzSVNAccessFile /path/to/.policy
     </Location>
    

comment:53 by mark@…, 15 years ago

Common settings:

  • Several virtual hosts.
    Every configured as described in TracModPython.
  • Trac 0.9.[2-5]
  • Python 2.4.2
  • SQLite 3
  • Apache: mpm_worker, PHP 5.1.x

Tested configurations at which the errors occurs:

  • Subversion 1.3.0 (with and without patch, no relief)
    • Apache 2.0.55, SWIG 1.3.21, APR 0.9.7
    • Apache 2.0.55, SWIG 1.3.21, APR 0.9.12
    • Apache 2.0.58, SWIG 1.3.25, APR 0.9.12
    • Apache 2.2.0, SWIG 1.3.21, APR 1.2.2
    • Apache 2.2.2, SWIG 1.3.25, APR 1.2.7
  • Subversion 1.3.1 (with/without)
    • Apache 2.0.58, SWIG 1.3.25, APR 0.9.12
    • Apache 2.2.0, SWIG 1.3.21, APR 1.2.2
    • Apache 2.2.0, SWIG 1.3.25, APR 1.2.2
    • Apache 2.2.0, SWIG 1.3.25, APR 1.2.7

Drop me a line if I got the "tester's award".

comment:54 by mark@…, 15 years ago

That's what I've forgotten to mention:

  • Apache 2.0.x: mod_python 3.1.4
  • Apache 2.2.x: mod_python 3.2.8

Everything compiled under Gentoo with GCC 3.4.x, -fomit-frame-pointer -pipe and either:

  • -Os -march=athlon-mp -ffast-math
  • -O2 -mtune=pentium-m -fforce-addr -frename-registers -falign-functions=64

comment:55 by mark@…, 15 years ago

SVN 1.3.1 has always been without patch - It didn't even apply.

comment:56 by Christian Boos, 15 years ago

Actually, that patch was merged in Subversion 1.3.1, see TracSubversion for details.

Mark, just in order to verify the hypothesis that using multiple virtual hosts could trigger the problem, would it be possible for you to setup a test environment with one/no virtual hosts?

(of course, that will earn you even more points for the tester-of-the-year contest :) )

comment:57 by mark@…, 15 years ago

I've had only the time for one test, so here it is:

Environment

  • Apache 2.0.58, APR 0.9.12, mod_python 3.1.4 (r1)
  • Subversion 1.3.1, SWIG 1.3.21

one vhost:

Fire up Apache, click every of the items in mainnav (wiki, timeline…). No error, no matter how much items you invoke.

second vhost:

preparations

Stop Apache, copy 10_trac.conf to 10_something.conf and adjust path and hostname.
Create a new subversion repo, import something into it and run trac-admin /var/lib/trac/something initenv.

Add another line to /etc/hosts to match your new vhost:

127.0.0.1 something.localhost

invocation

Fire up Apache, click through the first installed vhost. No errors will appear, believe me.

And now has the time come to open up the second vhost. Type something.localhost in your browser, get the first page displayed and change into "timeline" or "browse source"! Voila, fatal error appears.

comment:58 by Christian Boos, 15 years ago

Ok, thanks for the confirmation and for the steps to reproduce the issue.

comment:59 by Manuzhai, 15 years ago

There's something to the vhost diagnosis, but it's not everything.

On my old server, I had a single vhost containing different Location sections, for different projects, replicating the full setup for each config (e.g. SetHandler, several PythonOption directives, and authentication). I encountered the errors there, which is why I opened this ticket. On my new server, however, I have set it up differently, just using the multiple env support from Trac within a vhost, and I get no more errors.

comment:60 by anonymous, 15 years ago

with subversion 1.3.2,this problem still exists.

comment:61 by Alec Thomas, 15 years ago

I upgraded TracHacks to SVN 1.3.1 and was not able to get Apache to serve any Trac requests at all. Quite aggravating. Downgraded to 1.2.4 until fix is merged.

comment:62 by Manuzhai, 15 years ago

Milestone: 0.100.11

This is really a Subversion problem. We should still gather more information here, but let's set the milestone to 0.11; it really doesn't block 0.10.

comment:63 by maxb1@…, 15 years ago

I have what seems to be a reliable reproduction recipe: You must have an apache setup which contains Trac instances in *multiple* mod_python subinterpreters. Restart apache. The first request to a svn-using page (i.e. "Browse Source") within the first subinterpreter to be accessed will SUCCEED. The first request to a svn-using page within a second subinterpreter will fail with the error.

Additionally, just as an interesting data point: I actually got this error running a totally non-Trac-related command line script, once (it was ViewVC's svndbadmin). Sadly, this was not reproducable.

comment:64 by maxb1@…, 15 years ago

If anyone has seen this error with SWIG 1.3.28 or 1.3.29, please comment!

The reliable reproduction I mentioned in the previous comment was using the pregenerated SWIG files shipped with subversion 1.3.2 (i.e. SWIG 1.3.25 IIRC). I retested with SWIG 1.3.28, and the problem seems to have disappeared!

Further, I have isolated the source of the TypeError to the following code segment in _wrap_svn_pool_create:

        if (_global_svn_swig_py_pool != NULL &&
            !PyObject_HasAttrString(_global_svn_swig_py_pool,
                                    (char *)"_mark_valid"))
        {
            int argnum = PyTuple_GET_SIZE(args);
            SWIG_Python_TypeError(SWIG_TypePrettyName(SWIGTYPE_p_apr_pool_t),
                                  _global_svn_swig_py_pool);
            SWIG_arg_fail(argnum);
            SWIG_fail;
        }

comment:65 by marc@…, 15 years ago

Cc: marc@… added

I have the same problem, and I'm running

  • apache 2.0.54
  • mod_python 3.1.4
  • subversion 1.3.2
  • python 2.4.1
  • trac 0.9.4

I have about 15 tracs running under a virtualhost each (on port 81), and get intermittent errors. I have an RSS reader configured to request the Timeline page for each trac and it always gets an error for at least one of the tracs.

When I tried the setup with a single virtualhost I got no errors, but I have not tested that extensively.

If there's anything else that I could do or provide just let me know and I will try.

in reply to:  65 comment:66 by anonymous, 15 years ago

Replying to marc@guidance.nl:

I have the same problem, and I'm running

  • apache 2.0.54
  • mod_python 3.1.4
  • subversion 1.3.2
  • python 2.4.1
  • trac 0.9.4

I have about 15 tracs running under a virtualhost each (on port 81), and get intermittent errors. I have an RSS reader configured to request the Timeline page for each trac and it always gets an error for at least one of the tracs.

When I tried the setup with a single virtualhost I got no errors, but I have not tested that extensively.

If there's anything else that I could do or provide just let me know and I will try.

Instead of using virtual hosts, you can use another apache "instance" with another config file and on another TCP port or IP address. Something like this: /usr/sbin/apachectl -f /etc/httpd/conf/httpd.trac.conf

For Trac I have one apache (default) instance for intranet running on port 80, and another apache instance for internet running on port 81, and it works fine.

Before this I had only one apache instance running on ports 80 and 81, and it give this error.

comment:67 by Christian Boos, 15 years ago

In light of #3455, it seems clear that the fix in attachment:application_pool_race_condition_fix.txt is not working in case of multiple Python subinterpreters (as this is the case by default with virtual hosts).

The svn_swig_py_set_application_pool(self, self) will be called for each sub-interpreter and therefore at the C level one pool object coming from one subinterpreter will be used by all the other subinterpreters. This probably leads to an invalid GC, which destroys that object prematurely, and that deletes the "_is_valid" string which triggers the message we're seeing, see comment:64).

So… the workaround for #3455 should most probably also work here.

It remains to be confirmed if the 1.3.2 bindings generated with SWIG ≥ 1.3.28 are handling the multiple sub-interpreter scenario correctly.

comment:68 by anonymous, 15 years ago

We have mod_python-3.1.4, apache-2.0.58, swig-1.3.29, subversion-1.3.2, python-2.4.3, trac-0.10rc1. (gcc 4.1.1)

I have been working on converting a bunch of trac sites from 0.8 to 0.10rc1 (we will be moving to 0.10). We have hitting this since converting to gcc-4.1.1. That was a one-way change, but this is crazy. I have no idea what to do =(.

Apache is in mpm-prefork. We have disactivated about everything that we can, and I just implented the

PythonInterpreter "trac"

in my config. We have 5 sites.

comment:69 by anonymous, 15 years ago

Cc: joshland@… added

BTW: I guess I should mention - we are hitting this bug.

[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: Traceback (most recent call last):
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch\n    result = object(req)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 87, in handler\n    gateway.run(dispatch_request)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 304, in dispatch_request\n    env = _open_environment(env_path, run_once=environ['wsgi.run_once'])
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 57, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/env.py", line 431, in open_environment\n    env = Environment(env_path)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/env.py", line 121, in __init__\n    load_components(self)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/loader.py", line 140, in load_components\n    __import__(module)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 56, in ?\n    from svn import fs, repos, core, delta
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/svn/fs.py", line 19, in ?\n    from libsvn.fs import *
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/fs.py", line 29, in ?\n    import core
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 3057, in ?\n    svn_pool_create()
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 1098, in svn_pool_create\n    return apply(_core.svn_pool_create, args)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0xb6924c8c>)' is received

in reply to:  68 ; comment:70 by Christian Boos, 15 years ago

Replying to anonymous:

We have mod_python-3.1.4, apache-2.0.58, swig-1.3.29, subversion-1.3.2, python-2.4.3, trac-0.10rc1. (gcc 4.1.1)

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

PythonInterpreter "trac"

Try "main_interpreter", as suggested in #3455.

in reply to:  70 ; comment:71 by joshland@…, 15 years ago

Replying to cboos:

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

I think this worked. We are returning to mpm-worker mode. Our current software loadout is:

swig-1.3.25, subversion-1.3.2-r3, apache-2.0.58 (mpm-worker) and mod_python-3.1.4

Try "main_interpreter", as suggested in #3455.

my mod_python config is now

    SetHandler    mod_python
    PythonInterpreter main_interpreter
    PythonPath    "['/var/trac/lib/python2.4/site-packages'] + sys.path"
    PythonHandler trac.web.modpython_frontend
    PythonOption  TracEnvParentDir            "/var/live/trac"
    PythonOption  TracUriRoot                 /
    SetEnv        PYTHON_EGG_CACHE            "/var/live/egg_cache"

in reply to:  71 comment:72 by anonymous, 15 years ago

Replying to joshland@gmail.com:

Replying to cboos:

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

I think this worked. We are returning to mpm-worker mode. Our current software loadout is:

swig-1.3.25, subversion-1.3.2-r3, apache-2.0.58 (mpm-worker) and mod_python-3.1.4

Try "main_interpreter", as suggested in #3455.

my mod_python config is now

    SetHandler    mod_python
    PythonInterpreter main_interpreter
    PythonPath    "['/var/trac/lib/python2.4/site-packages'] + sys.path"
    PythonHandler trac.web.modpython_frontend
    PythonOption  TracEnvParentDir            "/var/live/trac"
    PythonOption  TracUriRoot                 /
    SetEnv        PYTHON_EGG_CACHE            "/var/live/egg_cache"

Because this person is using mod_python 3.1.4 and not mod_python 3.2.8 or later, using 'main_interpreter' will be no different to having used 'trac' as the interpreter name.

There are really two parts to trying to reduce these sorts of problems, that is even if this is a valid workaround for the original problem this person is having.

  1. Force all use of Trac/subversion to run under the same Python interpreter instance under mod_python. This would be necessary where for some reason one or more Trac instances were being accessed through different VirtualHost configurations within the Apache setup. This avoids issue where internal Python objects associated with the subversion bindings are somehow bound to a specific Python interpreter instance and therefore cannot be used across multiple Python interpreter instances. This is only going to be effective where access was through multiple contexts, such as VirtualHost, where mod_python would have executed the request within a different interpreter instance context. Doing this would otherwise make no difference.
  1. If using mod_python 3.2.8 or later, you can further increase your chances of things working by avoiding possible GIL issues caused by use of simplified Python locking API for the GIL by Python extensions, by forcing the very first Python interpreter instance created by mod_python to be used. This particular interpreter instance is called 'main_interpreter'. If using mod_python prior to 3.2.8, there is no means of specifically indicating the first interpreter created should be used, using 'main_interpreter' doesn't indicate that for those older versions. This change may have benefit even if access isn't through multiple interpreters in the first place, but it will only make a difference if you have mod_python 3.2.8 or later.

comment:73 by joshland@…, 15 years ago

BTW: It didn't work, it just lasted for about 30 minutes. It is definatly still failing. I am looking at the last post, and I will attempt to fix this problem following your guidelines.

comment:74 by joshland@…, 15 years ago

We upgraded to mod_python 3.2, and downgraded to svn 1.2. Problems have since gone away.

comment:75 by Christian Boos, 15 years ago

Right, the problem appeared with Subversion 1.3, so it's likely that downgrading to 1.2.4 makes the problem go away… However, it's a bit sad that this wasn't addressed in newer Subversion releases (there's been 1.3.1, 1.3.2 and now 1.4.0 in the meantime).

I wonder if djames is still seeing updates to this ticket (and if he's still taking care of the python bindings for Subversion).

comment:76 by neofutur, 15 years ago

Cc: neofutur@… added

Same problem here whith 1 vhosts

When i deleted the second vhost, no more problem I had to change the second vhost to another server

comment:77 by anonymous, 15 years ago

Cc: bock@… added

comment:78 by anonymous, 15 years ago

Cc: andrew.fecheyr@… added

I downgraded to svn 1.2.3 until this is resolved.

comment:79 by joshland@…, 15 years ago

I just hit a similar segfault w/ 1.2.3 I am running Gentoo, Apache-2.2 and Mod Python 3.2. I don't have any tracebacks or anything. Is anyone else experiencing problems with this?

I am about to switch to nginx + tracd and see if that fixes everything.

in reply to:  79 comment:80 by Christian Boos, 15 years ago

Replying to joshland@gmail.com:

I just hit a similar segfault w/ 1.2.3 I am running Gentoo, Apache-2.2 and Mod Python 3.2. I don't have any tracebacks or anything. Is anyone else experiencing problems with this?

It might be something different… this issue was never reported against 1.2.3 so far. So if you don't have the TypeError: argument ... error, I bet it's something else.

I am about to switch to nginx + tracd and see if that fixes everything.

TracStandalone is not affected by this issue, so if the segfault happens again, you'll know that it was something else. In case it happens, you might want to follow the advices given in TracTroubleshooting in order to locate the cause of the problem.

comment:81 by anonymous, 15 years ago

Cc: boris@… added

comment:82 by boris@…, 15 years ago

I have a hunch that it is in fact related to multiple vhosts running on the same apache instance. I'm starting to see errors of this nature crop up in my logs only after adding a second TracModPython virtual host.

I tried Trac 0.10, and SVN 1.3.1, 1.3.2.

comment:83 by joshland@…, 15 years ago

We will setup nginx + tracd this week and, provided that everything continues to work as expected, post a recipe on the nginx wiki (and the Trac wiki if possible).

comment:84 by Christian Boos, 15 years ago

Description: modified (diff)
Keywords: svn13 svn14 added; svn130 removed
Milestone: 0.112.0

Shifting the milestone to 2.0, as there's nothing we can do for 0.11.

Closing this issue is not an option either, as this is a recurring problem in a basic piece of infrastructure often needed by Trac. Probably an "Undefined" milestone would be appropriate here.

Note: See TracTickets for help on using tickets.