Seg fault potentially SWIG related?

[cameron@…]

Hi everybody,

Not sure if this is the place to post this, but I didn't get any response from the mailing list, and it does seem like potentially a Trac issue, not an installation issue.

I've been running a Trac install for about a month on multiple repositories just using tracd and it's been working fine. Then, a few days ago, it crashed. I restarted, and worked for a while. Then crashed again, and then again, etc. Now it seems to serve up a couple pages then crash. And it's never totally consistent, sometimes it'll crash on a wiki page, sometimes not…the only consistent part is I haven't been able to see the svn revision log page on any project since this started.

So I've tried a bunch of things, to no avail. Simplifying it down to 1 project and the error still occurs. The logs in the project folders (I turned logging on at the highest level) don't seem to show anything. When I run it not as a daemon, it says it has had a segmentation fault, but nothing more.

I've now run it under gdb:

$ gdb $(which python)
(gdb) run /opt/trac-0.10/scripts/tracd -p 8080 /srv/trac/yourproject

It still fails, and outputs this little bit:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213224016 (LWP 7891)]
SWIG_Python_NewPointerObj (ptr=0x8797dc0, type=0x0, own=0) at /usr/
local/share/swig/1.3.24/python/pyrun.swg:226
226     {

Typing "bt" to get a backtrace (I'm a gdb nuby), I get this (here's the whole, long thing):

#0  SWIG_Python_NewPointerObj (ptr=0x8ce5dc0, type=0x0, own=0) at /usr/ local/share/swig/1.3.24/python/pyrun.swg:226
#1  0x002e2065 in make_pointer (typename=0x2e4aed "svn_fs_root_t *",ptr=0x8ce5dc0) at /usr/local/share/swig/1.3.24/runtime.swg:19
#2  0x0042ba3f in Py_InitModule4 () from /usr/lib/libpython2.3.so.1.0
#3  0x0042bb05 in Py_InitModule4 () from /usr/lib/libpython2.3.so.1.0
#4  0x0042bbe9 in Py_VaBuildValue () from /usr/lib/libpython2.3.so.1.0
#5  0x003bc737 in PyObject_CallFunction () from /usr/lib/libpython2.3.so.1.0
#6  0x002e3f82 in svn_swig_py_repos_authz_func (allowed=0xb7aa8e94,root=0x8ce5dc0, path=0xb7c5cb54 "/", baton=0xb6fb2e2c, pool=0x8cddd68)
    at /root/subversion-1.2.1/subversion/bindings/swig/python/libsvn_swig_py/swigutil_py.c:1521
#7  0x00c3f890 in svn_repos_history2 (fs=0x8cd39e8, path=0xb7c5cb54"/", history_func=0x2e4038 <svn_swig_py_repos_history_func>,history_baton=0xb6fb2dbc,
    authz_read_func=0x2e3f1c <svn_swig_py_repos_authz_func>,authz_read_baton=0xb6fb2e2c, start=45, end=45, cross_copies=1,pool=0x8cddd68)
    at subversion/libsvn_repos/rev_hunt.c:238
#8  0x00714d7a in _wrap_svn_repos_history2 (self=0x0, args=0xb6fb2e2c)at subversion/bindings/swig/python/svn_repos.c:2669
#9  0x003df961 in PyCFunction_Call () from /usr/lib/libpython2.3.so.1.0
#10 0x004124ec in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#11 0x004143a2 in PyEval_EvalCode () from /usr/lib/libpython2.3.so.1.0
#12 0x003bc3cf in PyIter_Next () from /usr/lib/libpython2.3.so.1.0
#13 0x0040f6d2 in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#14 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#15 0x00412d69 in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#16 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#17 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#18 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#19 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#20 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#21 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#22 0x003f493c in _PyObject_SlotCompare () from /usr/lib/libpython2.3.so.1.0
#23 0x003edd1c in PyType_IsSubtype () from /usr/lib/libpython2.3.so.1.0
#24 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#25 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#26 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#27 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#28 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#29 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#30 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#31 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#32 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#33 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#34 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#35 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#36 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#37 0x003f4235 in _PyObject_SlotCompare () from /usr/lib/libpython2.3.so.1.0
#38 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#39 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#40 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#41 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#42 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#43 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#44 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#45 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#46 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#47 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#48 0x0040e180 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.3.so.1.0
#49 0x003bfacb in PyInstance_New () from /usr/lib/libpython2.3.so.1.0
#50 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#51 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#52 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#53 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#54 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#55 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#56 0x00411968 in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#57 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#58 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#59 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#60 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#61 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#62 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#63 0x0040e180 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.3.so.1.0
#64 0x00436052 in _PyObject_GC_Del () from /usr/lib/libpython2.3.so.1.0
#65 0x00b55371 in start_thread () from /lib/tls/libpthread.so.0
#66 0x00a9dffe in clone () from /lib/tls/libc.so.6

My specs are:

• Centos 4.x
• Trac 0.10.3
• Python 2.3.4
• SVN 1.2.1
• swig 1.3.24

Not sure what else is needed…any advice would be amazing, I've been tearing my hair out over this, as it seems so strange that it was working one day and not another.

And I have tried running it as a CGI, and get similar situations..though it seems harder to debug, as I don't get anything useful in the web browser (only "misconfiguration..blah blah"), and my apache error log also doesn't seem to show much of anything either.

Thanks in advance!

Cameron

[Christian Boos]

Hm, well, using the authz callback to store the revisions can be called "a hack", and that used to work well so far. But perhaps it's a bit fragile in some situations…

But now that I've looked at the code of svn_repos_history2 in ​rev_hunt.c, I see that nothing more fancy than calling svn_fs_node_history / svn_fs_history_prev is done there, so I should have used that in the first place.

[cameron@…]

Hi again. In case it helps, I was wondering if the version of SWIG was an issue, so I just updated my SWIG to 1.3.25, reinstalling the same SVN and SVN SWIG bindings at the same time.

The stack trace (from gdb) seems pretty much the same this time, except the exact failure point is slighly diffferent:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1212998736 (LWP 10090)]
0x001a5331 in SWIG_TypeQueryModule (start=0x0, end=0x0, name=0x1a8c79 "svn_fs_root_t *") at /usr/local/share/swig/1.3.25/swigrun.swg:254
254         if (iter->size) {

and the difference in the backtrace is basically just the first 2 lines:

#0  0x001a5331 in SWIG_TypeQueryModule (start=0x0, end=0x0, name=0x1a8c79 "svn_fs_root_t *") at /usr/local/share/swig/1.3.25/swigrun.swg:254
#1  0x001a6209 in make_pointer (typename=0x1a8c79 "svn_fs_root_t *", ptr=0x88614a8) at /usr/local/share/swig/1.3.25/runtime.swg:28
#2  0x0042ba3f in Py_InitModule4 () from /usr/lib/libpython2.3.so.1.0
#3  0x0042bb05 in Py_InitModule4 () from /usr/lib/libpython2.3.so.1.0
#4  0x0042bbe9 in Py_VaBuildValue () from /usr/lib/libpython2.3.so.1.0
#5  0x003bc737 in PyObject_CallFunction () from /usr/lib/libpython2.3.so.1.0
#6  0x001a8136 in svn_swig_py_repos_authz_func (allowed=0xb7b2d914, root=0x88614a8, path=0xb7ce1b54 "/", baton=0xb7037f0c, pool=0x8859450)
    at /root/subversion-1.2.1/subversion/bindings/swig/python/libsvn_swig_py/swigutil_py.c:1521
#7  0x00304890 in svn_repos_history2 (fs=0x884f978, path=0xb7ce1b54 "/", history_func=0x1a81ec <svn_swig_py_repos_history_func>, history_baton=0xb7037e64, 
    authz_read_func=0x1a80d0 <svn_swig_py_repos_authz_func>, authz_read_baton=0xb7037f0c, start=45, end=45, cross_copies=1, pool=0x8859450)
    at subversion/libsvn_repos/rev_hunt.c:238
#8  0x00823d7a in _wrap_svn_repos_history2 (self=0x0, args=0xb7037f0c) at subversion/bindings/swig/python/svn_repos.c:2669
....seems the same below...

Thanks!

[Christian Boos]

Follow-up to comment:1:

Hm, well, using the authz callback to store the revisions can be called "a hack",

Precision: I actually used the authz callback to stop earlier the history, once we reached "limit" revision, there's an other user callback used for storing the revisions. Anyway, the problem is not there, but in swig crashing while preparing the first argument (root) for the authz callback.

[Christian Boos]

Replying to cameron@cdbdesign.net:

Hi again. In case it helps, I was wondering if the version of SWIG was an issue, so I just updated my SWIG to 1.3.25, reinstalling the same SVN and SVN SWIG bindings at the same time.

Well, Subversion 1.2.1 is pretty old, are you sure that SWIG 1.3.24 or 1.3.25 was the recommended version at that time? Looks a bit too recent to me…

[cameron@…]

Yes, it's an older SVN for sure. But looking in the SVN source "subversion-1.2.1/subversion/bindings/swig/INSTALL" it says:

Install a suitable version of SWIG (which is currently swig versions 1.3.19 - 1.3.21, or 1.3.24 or above).

I will try to install a newer SVN right now and see if that makes a difference. I'm very hesitant to jump to the 1.4 branch of it because I understand the repository layout changed, and isn't backwards compatible. Maybe I'll try something in the 1.3.x range. I'll play around, and respond here ;-)

Thanks again!

[Christian Boos]

Replying to cameron@cdbdesign.net:

Yes, it's an older SVN for sure. But looking in the SVN source "subversion-1.2.1/subversion/bindings/swig/INSTALL" it says:

Install a suitable version of SWIG (which is currently swig versions 1.3.19 - 1.3.21, or 1.3.24 or above).

Ok, though I wouldn't trust the "or above" part too much ;)

I'm very hesitant to jump to the 1.4 branch of it because I understand the repository layout changed, and isn't backwards compatible.

You shouldn't have any trouble upgrading your repository to the 1.4.x line. I've successfully carried the same repository from 0.26 up to 1.4.3 without trouble (or maybe it was even older, can't remember ;-) ).

[Christian Boos]

Patch on trunk, replacing the use of repos.svn_repos_history by lower-level fs.` history calls. Applies cleanly on 0.10.3 as well

[Christian Boos]

Could you please test attachment:history-t4683.diff​, even with your original 1.2.1 bindings built with SWIG 1.3.24?

If everything goes well, I'll apply that tomorrow, after some further testings. The _history function should not only be more robust now, but also quite faster, as we avoid the callbacks.

[cameron@…]

OK, something is acting weird. I applied that patch, and at first it seemed to work. But I quickly realized that it is only working on some of my projects. Others still crashed.

I tried to investigate which ones were crashing vs not, and find a pattern, and the only pattern I could find was that the ones crashing all have had SVN commits to them in the last 3 days (ie. they are our currently active projects).

Some things that weren't related to which ones crashed:

• They use a mix of berkeley and FSFS SVN repo type
• They range from only having a few commits in them, to a few hundred

I also tried dumping and rebuilding one of the failing repositories using "svnadmin dump….svnadmin load", and I also tried creating a new trac project but the failures still seem to happen.

Does that make any sense at all?

[Christian Boos]

Replying to cameron@cdbdesign.net:

… Others still crashed.

The stack trace must be quite different, can you attach one of those?

… the only pattern I could find was that the ones crashing all have had SVN commits to them in the last 3 days (ie. they are our currently active projects).

Is there a mix of Subversion install? i.e. are the bindings used by Trac not older than the Subversion server software (i.e. either svnserve or mod_dav_svn, those which recently wrote in the problematic repositories)?

[Christian Boos]

Second version, get rid of the limit parameter in the _history method, as its now a real generator

[Christian Boos]

OK, I think the clean-up of the _history function was worth it (even if it ends up not being related to the original issue - I'm still waiting for the new stacktraces!).

I'll test attachment:history-t4683.2.diff​ a bit more tomorrow and then commit it if everything seems to be OK.

[cameron@…]

Sorry, got caught up in other work.

I've just applied this second page instead of the first patch, and still same symptoms (working on all projects except recently committed to ones). Here's the backtrace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213355088 (LWP 10040)]
0x001e5331 in SWIG_TypeQueryModule (start=0x0, end=0x0, name=0x1e8c36 "apr_pool_t *") at /usr/local/share/swig/1.3.25/swigrun.swg:254
254         if (iter->size) {
(gdb) bt
#0  0x001e5331 in SWIG_TypeQueryModule (start=0x0, end=0x0, name=0x1e8c36 "apr_pool_t *") at /usr/local/share/swig/1.3.25/swigrun.swg:254
#1  0x001e6209 in make_pointer (typename=0x1e8c36 "apr_pool_t *", ptr=0xa043368) at /usr/local/share/swig/1.3.25/runtime.swg:28
#2  0x0042ba3f in Py_InitModule4 () from /usr/lib/libpython2.3.so.1.0
#3  0x0042bb05 in Py_InitModule4 () from /usr/lib/libpython2.3.so.1.0
#4  0x0042bbe9 in Py_VaBuildValue () from /usr/lib/libpython2.3.so.1.0
#5  0x003bc8d9 in PyObject_CallMethod () from /usr/lib/libpython2.3.so.1.0
#6  0x001e6faf in open_root (edit_baton=0xa03e640, base_revision=-1, dir_pool=0xa043368, root_baton=0x0)
    at /root/subversion-1.2.1/subversion/bindings/swig/python/libsvn_swig_py/swigutil_py.c:772
#7  0x001ed985 in svn_delta_path_driver (editor=0xa03e600, edit_baton=0xa03e640, revision=-1, paths=0xa042928, callback_func=0x9800b4 <path_driver_cb_func>, 
    callback_baton=0xb7ad6b80, pool=0xa02f160) at subversion/libsvn_delta/path_driver.c:165
#8  0x00980508 in svn_repos_replay (root=0xa039378, editor=0xa03e600, edit_baton=0xa03e640, pool=0xa02f160) at subversion/libsvn_repos/replay.c:300
#9  0x004dc21c in _wrap_svn_repos_replay (self=0x0, args=0xb6f90b94) at subversion/bindings/swig/python/svn_repos.c:2273
#10 0x003df961 in PyCFunction_Call () from /usr/lib/libpython2.3.so.1.0
#11 0x004124ec in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#12 0x004143a2 in PyEval_EvalCode () from /usr/lib/libpython2.3.so.1.0
#13 0x003bc3cf in PyIter_Next () from /usr/lib/libpython2.3.so.1.0
#14 0x0040f6d2 in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#15 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#16 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#17 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#18 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#19 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#20 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#21 0x003f493c in _PyObject_SlotCompare () from /usr/lib/libpython2.3.so.1.0
#22 0x003edd1c in PyType_IsSubtype () from /usr/lib/libpython2.3.so.1.0
#23 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#24 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#25 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#26 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#27 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#28 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#29 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#30 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#31 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#32 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#33 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#34 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#35 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#36 0x003f4235 in _PyObject_SlotCompare () from /usr/lib/libpython2.3.so.1.0
#37 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#38 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#39 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#40 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#41 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#42 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#43 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#44 0x003f4235 in _PyObject_SlotCompare () from /usr/lib/libpython2.3.so.1.0
#45 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#46 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#47 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#48 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#49 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#50 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#51 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#52 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#53 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#54 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#55 0x0040e180 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.3.so.1.0
#56 0x003bfacb in PyInstance_New () from /usr/lib/libpython2.3.so.1.0
#57 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#58 0x00411b4f in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#59 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#60 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#61 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#62 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#63 0x00411968 in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#64 0x0041395b in _PyEval_SliceIndex () from /usr/lib/libpython2.3.so.1.0
#65 0x00414076 in PyEval_EvalCodeEx () from /usr/lib/libpython2.3.so.1.0
#66 0x003cfe6e in PyFunction_SetClosure () from /usr/lib/libpython2.3.so.1.0
#67 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#68 0x003c3dc8 in PyMethod_New () from /usr/lib/libpython2.3.so.1.0
#69 0x003bc637 in PyObject_Call () from /usr/lib/libpython2.3.so.1.0
#70 0x0040e180 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.3.so.1.0
#71 0x00436052 in _PyObject_GC_Del () from /usr/lib/libpython2.3.so.1.0
#72 0x00b55371 in start_thread () from /lib/tls/libpthread.so.0
#73 0x00a9dffe in clone () from /lib/tls/libc.so.6

You had suggested a possible mixing of Subversion installs, which is a possibility. I did try upgrading to 1.3.2, and then went back after. However if I run 'svn —version' or 'svnserve —version' I get the same numbers both times (1.2.1), not sure if I should completely remove it, then reinstall (and I don't actually know how to do that).

Any thoughts would be great, thanks!

Cameron

[Christian Boos]

Replying to cameron@cdbdesign.net:

I've just applied this second page instead of the first patch, and still same symptoms

Ok, so this was not something specific to the svn_repos_history call, but it does seem to have to do with callbacks, though, as the stack trace shows.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213355088 (LWP 10040)]

in a thread…

#5 0x003bc8d9 in PyObject_CallMethod () from /usr/lib/libpython2.3.so.1.0

(python callback)

#6 0x001e6faf in open_root (edit_baton=0xa03e640, base_revision=-1, dir_pool=0xa043368, root_baton=0x0)

(called from the Subversion function)

So could it be that one component was not build thread safe? Are you sure your Python build is thread-safe?

Did something change on the server side recently? Upgrade of Apache, change of mpm (from prefork to worker?), rebuild of Python? Any additional hint?

Side note: the changes in the patches are working great, so I've applied them even if that won't help for your particular issue… (r4710 and r4711).

[cameron@…]

Hello again,

To answer your questions, no, not much has changed on the server recently, except me playing around with the versions of SVN, SWIG, and the SWIG-PY bindings. No changes to Apache, mpm (don't know what that is) or python.

I installed python via "yum install python" I think, way back when, and I don't know if it was set to be thread-safe or not. Should I recompile it from source instead? Is there a flag to pass to set it to be thread-safe? If I reinstall it, should I then reinstall the rest of the stack? (svn/swig/etc)?

Cameron

[cameron@…]

Hey again,

So, here's what I've done. I've upgraded python to 2.5, and reinstalled most of the pieces:

• Python 2.5
• ClearSilver 10.4
• PySQLite
• SWIG 1.3.27
• SVN 1.3.2
• SVN Swig bindings
• Trac 10.4dev

And now here's the weird thing. When I run tracd under gdb, it works fine. I can browse my revision logs, and everything else without any problems.

However, when I run tracd on it's own (not within gdb) it crashes immediately, even on the project listing page. All that it outputs is "segmentation fault". And like I said, I can't figure out how to debug it because it runs fine under the debugger?

Any ideas? It's very strange to me….

Thanks in advance!

Cameron

[ThurnerRupert]

pls reopen if it occurs with new versions too.

[anonymous]

(don't forget to clear the milestone when closing with a resolution different than fixed)