Trac Troubleshooting or "When Things Go Wrong"

Here's a collection of advices you can put into profit when troubleshooting Trac and for helping us to fix bugs.

RTFW (Read the Fine Wiki)

Trac comes with an extensive documentation in the form of a collection of Wiki pages. Some are present in each Trac installation (those part of the TracGuide), most are in this Trac. Among the latter, be sure to go through the TracFaq.

A few subsystems have their dedicated page with a troubleshooting section:

• PySqlite#Troubleshooting
• TracSubversion#Troubleshooting
• TracModPython#Troubleshooting

Then, use the TracSearch to dig through existing tickets, looking for past issues similar to yours thanks to carefully selected keywords. The Trac MailingLists are also usually of some help.

And don't forget it's a Wiki, so feel free to enhance its content while you're at it.

Now, assuming you're facing a new problem, the next step is to try to find out what's really wrong.

Debugging Trac

System Errors

By system errors, we mean serious issues like segmentation faults or process hangs. This usually involves some C extensions used by Trac, either due to a bug or a misconfiguration in those libraries, or to an incorrect usage within Trac.

In this case, the first thing to do is to identify the subsystem involved, by getting the stack trace of the process. This can be done using gdb on Unix or ​WinDbg on Windows (or MS Developer Studio).

Debugging Segmentation Faults

Getting a backtrace for tracd

$ gdb $(which python)
(gdb) run /opt/trac-0.10/scripts/tracd -p 8080 /srv/trac/yourproject

(of course, adapt the paths and the options to match what's relevant for you)

Getting a backtrace for Apache's httpd can be done in a similar way:

$ apachectl -k stop
$ gdb $(which httpd)
(gdb) run -X

Debugging a Hanging Process

Here it might be interesting to just "attach" to an already running process:

$ ps -ef | grep httpd 
...
$ # note the PID of interest, e.g. 28221
$ gdb
(gdb) attach 28221
Attaching to process 28221
Reading symbols from /opt/apache-2.0.55/bin/httpd...done.
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
Reading symbols from /opt/apache-2.0.55/lib/libaprutil-0.so.0...done.
Loaded symbols for ...
...
(gdb) bt
#0  0x0000002a962905af in __accept_nocancel () from /lib64/tls/libpthread.so.0
#1  0x0000002a95bc1a84 in apr_socket_accept (new=0x7fbfffd538, sock=0x5b27b0,
    connection_context=0xbd2d58) at sockets.c:164
#2  0x0000000000477c4d in unixd_accept (accepted=0x7fbfffd560,
...
(gdb) cont
Continuing.

<here you can press Ctrl-C>

Program received signal SIGINT, Interrupt.
[Switching to Thread 182911242432 (LWP 28221)]
0x0000002a962905af in __accept_nocancel () from /lib64/tls/libpthread.so.0
(gdb) detach
Detaching from program: /opt/apache-2.0.55/bin/httpd, process 28221
(gdb) ^D
$

In the above, the Loaded symbols ... part usually already interesting. You may find out that the libraries actually used are not the one that you expected, or you might notice interesting things, like the presence of mod_php triggering the load of an alternate sqlite library, etc. That kind of information is also available without gdb by looking in the /proc/28221/maps file (to reuse the same pid from the above example).

The bt command is what gives you the "backtrace" of the program, usually the most interesting bit of information. You can also resume execution of the program (using cont) and interrupt the process a bit later, to see if it remains hanged in the same area. In case there's no hang (you "attached" to it just for curiosity), you can also detach from the process and it will continue to work unaffected.