HTTPNotFound Errors, Very High CPU Usage, Git Defunct Processes

[nicholas.h.helseth@…]

We'd been having problems with trac 0.12 crashing and leaving behind defunct git processes, so we recently upgraded to Trac 1.0.1, hoping it'd solve the problem, but it appears to continue.

I've looked at #10422, #10414, #11174, and a couple others, but the solutions offered there don't seem to help.

We get the following errors:

2014-09-20 06:36:45,475 Trac[main] WARNING: [137.136.20.156] HTTPForbidden: 403 Forbidden (TICKET_CREATE privileges are required to perform this operation. You don't have the required permissions.)
2014-09-20 07:01:22,373 Trac[main] ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build/bdist.linux-x86_64/egg/trac/ticket/query.py", line 963, in process_request
    return self.display_html(req, query)
  File "build/bdist.linux-x86_64/egg/trac/ticket/query.py", line 1079, in display_html
    TicketSystem(self.env).eventually_restrict_owner(owner_field[0])
  File "build/bdist.linux-x86_64/egg/trac/ticket/api.py", line 408, in eventually_restrict_owner
    .get_users_with_permission('TICKET_MODIFY'):
  File "build/bdist.linux-x86_64/egg/trac/perm.py", line 436, in get_users_with_permission
    perms = self.store.get_users_with_permissions(satisfying_perms) or []
  File "build/bdist.linux-x86_64/egg/trac/perm.py", line 211, in get_users_with_permissions
    users = set([u[0] for u in self.env.get_known_users()])
  File "build/bdist.linux-x86_64/egg/trac/env.py", line 667, in get_known_users
    """):
  File "build/bdist.linux-x86_64/egg/trac/db/api.py", line 121, in execute
    with self as db:
  File "build/bdist.linux-x86_64/egg/trac/db/api.py", line 174, in __enter__
    db = self.dbmgr.get_connection(readonly=True)
  File "build/bdist.linux-x86_64/egg/trac/db/api.py", line 262, in get_connection
    db = self._cnx_pool.get_cnx(self.timeout or None)
  File "build/bdist.linux-x86_64/egg/trac/db/pool.py", line 213, in get_cnx
    return _backend.get_cnx(self._connector, self._kwargs, timeout)
  File "build/bdist.linux-x86_64/egg/trac/db/pool.py", line 134, in get_cnx
    raise TimeoutError(errmsg)
TimeoutError: Unable to get database connection within 0 seconds.

and:

2014-09-20 07:31:35,140 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node iov/gvs9000_bin at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/iov/gvs9000_bin?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:34:47,166 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node master/frc40 at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/master/frc40?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:35:44,021 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node master/raptor at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/master/raptor?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:35:55,444 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node master/gem04d at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/master/gem04d?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:36:04,673 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node master/include at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/master/include?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:36:09,677 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node master/src at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/master/src?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:36:38,123 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node master/makeall at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/master/makeall?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:37:27,644 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node iov/include at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/iov/include?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:37:40,138 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node iov/linux_include at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/iov/linux_include?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:43:47,905 Trac[main] WARNING: [137.136.20.156] HTTPNotFound: 404 Trac Error (<p class="message">No node iov/src at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/iov/src?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2014-09-20 07:58:44,935 Trac[main] WARNING: [137.136.20.155] HTTPNotFound: 404 Trac Error (<p class="message">No node gem04d_lib/src/Make_gem04d at revision 278c40ce804273d4e29ce166dd31f2345e24ddc0</p><p>You can <a href="/dms/log/dms/gem04d_lib/src/Make_gem04d?rev=278c40ce804273d4e29ce166dd31f2345e24ddc0&amp;mode=path_history">search</a> in the repository history to see if that path existed but was later removed</p>)
2

That commit (278c40c) is the most recent in our development branch and those files were removed a long time ago (on the order of a year, or at least 100 commmits).

Additionally, top shows that tracd is taking up more than 100% of the cpu on a regular basis, and occasionally git processes are left behind (strace doesn't show anything in them and ps won't show their argument list).

tracd is started in the following way:

/ots/sw/python-2.7.0/bin/python /ots/sw/python-2.7.0/bin/tracd -d -p 8080 --basic-auth=*,/project/dms/trac/global/.htpasswd,boeing -e /project/dms/trac/projects

Here's my system configuration:

Trac	   1.0.1
Babel	   0.9.6
Genshi	   0.7 (with speedups)
GIT	   1.7.1
pysqlite   2.6.3
Python	   2.7 (r27:82500, Aug 20 2012, 15:23:18) [GCC 3.4.6 20060404 (Red Hat 3.4.6-4)]
setuptools 0.6c11
SQLite	   3.3.6
jQuery	   1.7.2

My OS version is RHEL 5.9.

I've performed a full repo resync.

Thanks so much for your help.

[nicholas.h.helseth@…]

Internal IT crawler was hitting HTTP interface really hard and trac fell behind as a result. Blocked crawler and everything's fine. Sorry for the trouble.