Contents
Trac Troubleshooting
This is a collection of recipes you can use to troubleshoot Trac and help us to fix bugs.
Read the Wiki
Trac comes with extensive documentation in form of Wiki pages. Some are present in each Trac installation, for example those that are part of the TracGuide, while most are only found on this site, for example the Frequently Asked Questions.
Some subsystems have their dedicated page with a troubleshooting section:
- PySqlite#Troubleshooting
- MySqlDb#Troubleshooting
- TracModWSGI#Troubleshooting
- TracModPython#Troubleshooting
- TracPlugins#Troubleshooting
- TracNotification#Troubleshooting
- TracRepositoryAdmin#Troubleshooting
- TracSubversion#Troubleshooting
- TracGit#Troubleshooting
- TracMercurial#Troubleshooting
Then, use the TracSearch to dig through existing tickets describing issues similar to yours. The Trac MailingLists are also usually of some help.
And because this a Wiki, feel free to enhance its contents while you're at it.
If you're facing a new problem, the next step is to try to find out the root cause.
Check the Logs
When a feature is unavailable it's often the case that the Trac or plugin Component that implements the feature has failed to load. This could be due to a programming error, missing dependency or installation problem. Component loading is logged at DEBUG level when restarting tracd or the web server. Additional debug information and tracebacks may also be logged.
Set the LogLevel to DEBUG, restart tracd or the web server and then inspect the log. The log location is configured in trac.ini. The default location is log/trac.log in the environment directory.
See TracLogging for additional information.
Check the Browser Console
JavaScript error and debug messages will be logged to the browser console. See the documentation for your browser for instructions on accessing the browser console.
Debugging Trac
Python errors
Trac assists with debugging internal errors by providing a stack trace. Provided you have the TRAC_ADMIN privilege, the error page will display the faulty line of code in its context and the values of local variables.
Now, before describing the specific debugging techniques you can use, a basic understanding of Trac's architecture is needed.
Trac uses a model-view-controller approach. The controller is a Python class called a "module" which inherits from the Component class and implements the IRequestHandler interface. A Component is the basic building block in the TracDev/ComponentArchitecture, it implements some Interfaces. The controller reacts on user requests and prepares the data that will be used by a template engine to fill the adequate template, in order to render the view which is sent back to the user.
The TracDev collection of development guides can be useful here.
Inspecting the Template data
The template data dictionary can be inspected by appending?hdfdump=1 to the URL, or &hdfdump=1, in case other parameters are already present in the URL. The acronym hdf is a holdover from before Trac 0.11, when ClearSilver was used with its so-called Hierarchical Data Format.
Starting with 0.11, Trac uses the Genshi template engine, and starting with 1.3.2 Trac uses Jinja2.
As the template engines are written in Python, no specific data format is needed and a simple Python dictionary is used to feed the engine. It's very easy to inspect any part of this data by modifying the template and inserting ${pprint(...)} statements, possibly in between <pre>...</pre> tags.
Each modification to a template will be detected on the fly and you'll be able to see the result of the change immediately, provided you have the following configuration:
[trac] auto_reload = yes
Modifying the Code
TracStandalone is the indispensable companion whether you intend to develop or debug Trac. In particular, check out the -r (auto-reload) feature, which will make Trac notice any change to one of its source file and restart automatically. You can therefore see Trac react immediately to your code changes, provided you don't have syntax errors outside of a method.
In this setup, you're free to try out modifications, dump additional information to the log or insert direct print statements; an ugly but effective way of debugging.
It may even be possible to run tracd with a debugger, but that is not explored here.
Best is to start from a checkout of the pristine source code you're interested to debug or develop for. Then, you can run the standalone server by doing:
$ tracd <options>
Note that the very first time in a fresh working copy, you'll have to install the code in develop mode:
$ python setup.py develop
System Errors
System errors are serious issues like segmentation faults or process hangs. This usually involves some C extensions used by Trac, either due to a bug or a misconfiguration in those libraries, or to an incorrect usage within Trac.
In this case, the first thing to do is to identify the subsystem involved, by getting the stack trace of the process. This can be done using gdb on Unix or WinDbg on Windows or MS Developer Studio.
With GDB you can also make the link between the C stack trace and the Python backtrace. For this, you need to teach gdb a few additional commands: Get the gdbinit script, and "source" it.
You should be able to issue interesting commands like pystack, pyframe, pylocals, etc.
If you are using a gdb 7.x version built --with-python, you can instead source libpython.py and use py-bt. See also Scripting gdb using Python.
Debugging Segmentation Faults
Getting a backtrace for tracd:
$ gdb $(which python) (gdb) run /opt/trac-0.10/scripts/tracd -p 8080 /srv/trac/yourproject
Of course, adapt the paths and the options to match what is relevant for you.
Getting a backtrace for Apache's httpd can be done in a similar way:
$ apachectl -k stop $ gdb $(which httpd) (gdb) run -X
When it crashes, do bt in order to get a full backtrace.
Another very useful command is info shared, which gives you a list of the shared libraries actually used.
Debugging a Hanging Process
Here it might be interesting to just "attach" to an already running process:
$ ps -ef | grep httpd ... $ # note the PID of interest, e.g. 28221 $ gdb (gdb) attach 28221 Attaching to process 28221 Reading symbols from /opt/apache-2.0.55/bin/httpd...done. Using host libthread_db library "/lib64/tls/libthread_db.so.1". Reading symbols from /opt/apache-2.0.55/lib/libaprutil-0.so.0...done. Loaded symbols for ... ... (gdb) bt #0 0x0000002a962905af in __accept_nocancel () from /lib64/tls/libpthread.so.0 #1 0x0000002a95bc1a84 in apr_socket_accept (new=0x7fbfffd538, sock=0x5b27b0, connection_context=0xbd2d58) at sockets.c:164 #2 0x0000000000477c4d in unixd_accept (accepted=0x7fbfffd560, ... (gdb) cont Continuing. <here you can press Ctrl-C> Program received signal SIGINT, Interrupt. [Switching to Thread 182911242432 (LWP 28221)] 0x0000002a962905af in __accept_nocancel () from /lib64/tls/libpthread.so.0 (gdb) detach Detaching from program: /opt/apache-2.0.55/bin/httpd, process 28221 (gdb) ^D $
The Loaded symbols ... part usually provides interesting information.
You may find out that the libraries actually used are not the one that you expected, or you might notice the presence of mod_php triggering the load of an alternate sqlite library.
That kind of information is also available without gdb by looking in the /proc/28221/maps file (to reuse the same pid from the above example).
The bt command is what gives you the "backtrace" of the program, usually the most interesting bit of information. You can also resume execution of the program (using cont) and interrupt the process a bit later, to see if it remains hanged in the same area. In case there is no hang (you "attached" to it just for curiosity), you can also detach from the process and it will continue to work unaffected.
