[[PageOutline(2-5,Contents)]] = Trac Troubleshooting This is a collection of recipes you can use to troubleshoot Trac and help us to fix bugs. == Read the Wiki Trac comes with extensive documentation in form of [http://trac.edgewall.org/wiki Wiki pages]. Some are present in each Trac installation, for example those that are part of the TracGuide, while most are only found on this site, for example the [TracFaq Frequently Asked Questions]. Some subsystems have their dedicated page with a troubleshooting section: - PySqlite#Troubleshooting - MySqlDb#Troubleshooting - [[TracModWSGI#Troubleshooting]] - TracModPython#Troubleshooting - TracPlugins#Troubleshooting - TracNotification#Troubleshooting - TracRepositoryAdmin#Troubleshooting - TracSubversion#Troubleshooting - TracGit#Troubleshooting - TracMercurial#Troubleshooting Then, use the TracSearch to dig through existing tickets describing issues similar to yours. The Trac ["MailingList"]s are also usually of some help. And because this a Wiki, feel free to enhance its contents while you're at it. If you're facing a new problem, the next step is to try to find out the root cause. == Check the Logs The log output is an excellent source of diagnostic information. When a feature is unavailable it's often the case that the Trac or plugin //Component// that implements the feature has failed to load. This could be due to a programming error, missing dependency or installation problem. //Component// loading is logged at //DEBUG// level when restarting tracd or the web server. Additional debug information and tracebacks may also be logged. Set the [TracLogging#LogLevels LogLevel] to //DEBUG//, restart tracd or the web server and then inspect the log. See TracLogging for additional information. == Check the Browser Console JavaScript error and debug messages will be logged to the browser console. See the documentation for your browser for instructions on accessing the browser console. * [https://developer.mozilla.org/en-US/docs/Tools/Browser_Console Firefox] * [https://developers.google.com/web/tools/chrome-devtools/console/ Chrome] * [https://docs.microsoft.com/en-us/microsoft-edge/f12-devtools-guide/console Internet Explorer] == Debugging Trac === Python errors Trac assists with debugging internal errors by providing a stack trace. Provided you have the `TRAC_ADMIN` privilege, the error page will display the faulty line of code in its context and the values of local variables. Now, before describing the specific debugging techniques you can use, a basic understanding of Trac's architecture is needed. Trac uses a [http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller model-view-controller approach]. The ''controller'' is a Python class called a "module" which inherits from the Component class and implements the `IRequestHandler` interface. A ''Component'' is the basic building block in the TracDev/ComponentArchitecture, it ''implements'' some ''Interface''s. The controller reacts on user requests and prepares the data that will be used by a template engine to fill the adequate template, in order to render the ''view'' which is sent back to the user. The TracDev collection of development guides can be useful here. ==== Inspecting the Template data The template data dictionary can be inspected by appending`?hdfdump=1` to the URL, or `&hdfdump=1`, in case other parameters are already present in the URL. The acronym `hdf` is a holdover from before Trac 0.11, when ClearSilver was used with its so-called ''Hierarchical Data Format''. Starting with [milestone:0.11], Trac uses the [http://genshi.edgewall.org Genshi] template engine, and starting with [milestone:1.3.2] Trac uses [wiki:TracDev/Proposals/Jinja Jinja2]. As the template engines are written in Python, no specific data format is needed and a simple Python dictionary is used to feed the engine. It's very easy to inspect any part of this data by modifying the template and inserting `${pprint(...)}` statements, possibly in between `
...
` tags. Each modification to a template will be detected on the fly and you'll be able to see the result of the change immediately, provided you have the following [TracIni#trac-auto_reload-option configuration]: {{{#!ini [trac] auto_reload = yes }}} ==== Modifying the Code TracStandalone is the indispensable companion whether you intend to develop or debug Trac. In particular, check out the `-r` (auto-reload) feature, which will make Trac notice any change to one of its source file and restart automatically. You can therefore see Trac react immediately to your code changes, provided you don't have syntax errors outside of a method. In this setup, you're free to try out modifications, dump additional information to the log or insert direct `print` statements; an ugly but effective way of debugging. It may even be possible to run `tracd` with a debugger, but that is not explored here. Best is to start from a [TracSubversion checkout] of the pristine source code you're interested to debug or develop for. Then, you can run the standalone server by doing: {{{#!sh $ tracd }}} Note that the very first time in a fresh working copy, you'll have to install the code in develop mode: {{{#!sh $ python setup.py develop }}} === System Errors System errors are serious issues like segmentation faults or process hangs. This usually involves some C extensions used by Trac, either due to a bug or a misconfiguration in those libraries, or to an incorrect usage within Trac. In this case, the first thing to do is to identify the subsystem involved, by getting the ''stack trace'' of the process. This can be done using `gdb` on Unix or [http://www.microsoft.com/whdc/devtools/debugging/default.mspx WinDbg] on Windows or MS Developer Studio. With [https://www.gnu.org/software/gdb/ GDB] you can also make the link between the C stack trace and the Python backtrace. For this, you need to teach `gdb` a few additional commands: Get the [http://hg.python.org/cpython/file/2.7/Misc/gdbinit gdbinit script], and "source" it. You should be able to issue interesting commands like `pystack`, `pyframe`, `pylocals`, etc. If you are using a gdb 7.x version built `--with-python`, you can instead `source` [http://hg.python.org/cpython/file/2.7/Tools/gdb/libpython.py libpython.py] and use `py-bt`. See also [http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html Scripting gdb using Python]. ==== Debugging Segmentation Faults Getting a backtrace for `tracd`: {{{#!sh $ gdb $(which python) (gdb) run /opt/trac-0.10/scripts/tracd -p 8080 /srv/trac/yourproject }}} Of course, adapt the paths and the options to match what is relevant for you. Getting a backtrace for Apache's `httpd` can be done in a similar way: {{{#!sh $ apachectl -k stop $ gdb $(which httpd) (gdb) run -X }}} When it crashes, do `bt` in order to get a full backtrace. Another very useful command is `info shared`, which gives you a list of the shared libraries actually used. ==== Debugging a Hanging Process Here it might be interesting to just "attach" to an already running process: {{{#!sh $ ps -ef | grep httpd ... $ # note the PID of interest, e.g. 28221 $ gdb (gdb) attach 28221 Attaching to process 28221 Reading symbols from /opt/apache-2.0.55/bin/httpd...done. Using host libthread_db library "/lib64/tls/libthread_db.so.1". Reading symbols from /opt/apache-2.0.55/lib/libaprutil-0.so.0...done. Loaded symbols for ... ... (gdb) bt #0 0x0000002a962905af in __accept_nocancel () from /lib64/tls/libpthread.so.0 #1 0x0000002a95bc1a84 in apr_socket_accept (new=0x7fbfffd538, sock=0x5b27b0, connection_context=0xbd2d58) at sockets.c:164 #2 0x0000000000477c4d in unixd_accept (accepted=0x7fbfffd560, ... (gdb) cont Continuing. Program received signal SIGINT, Interrupt. [Switching to Thread 182911242432 (LWP 28221)] 0x0000002a962905af in __accept_nocancel () from /lib64/tls/libpthread.so.0 (gdb) detach Detaching from program: /opt/apache-2.0.55/bin/httpd, process 28221 (gdb) ^D $ }}} The `Loaded symbols ... ` part usually provides interesting information. You may find out that the libraries actually used are not the one that you expected, or you might notice the presence of mod_php triggering the load of an alternate sqlite library. That kind of information is also available without `gdb` by looking in the `/proc/28221/maps` file (to reuse the same pid from the above example). The `bt` command is what gives you the "backtrace" of the program, usually the most interesting bit of information. You can also resume execution of the program (using `cont`) and interrupt the process a bit later, to see if it remains hanged in the same area. In case there is no hang (you "attached" to it just for curiosity), you can also `detach` from the process and it will continue to work unaffected.