Severe memory issues/leaks.

[josh@…]

My company has Trac and SVN set up to use locally for our internal development. However, we're having issues whereby Trac's memory usage is slowly building up over a period of around 1-3 days, to the point where it consumes all available RAM, forcing us to restart Apache, or worse, do a hard-reboot of the server.

Running top reveals that it's the httpd (Apache) processes which are eating the RAM. They are also gradually increacing their CPU usage, to the point where the machine is generally idling at around ~10% cpu.

The Swap partition is also not being utilised much, suggesting(?) the actual cause of the slowdown is probably thrashing.

Obviously, this is not ideal, and we'd quite like to identify the cause of the problem and fix it ASAP.

Details of the setup are:

• Machine
  • OS: Virtualised Fedora 12 64-bit Linux
  • Memory: 1Gb
  • Swap: 2Gb
  • CPU: Intel 2.4GHz Dual-core

• Software
  • Apache: 2.2.14
  • Python: 2.6.2
  • Trac: 0.11.7
  • SVN: 1.6.9
  • mod_wsgi: 3.1

• Apache
  • Running 10 instances. (i.e. Whatever that build defaults to, since I didn't configure that.)

• Trac
  • ~20 individual projects (all w/own SVN repos).
  • Plugins: AdvancedTicketWorkflow, Developer, MasterTickets, NeverNotifyUpdater, PrivateTickets, TimingAndEstimation, VirtualTicketPermissions, Worklog, WYSIWYG.

With the exception of Trac and it's plugins, all the software is installed from the Fedora repos. Trac and plugins are installed from source.

There's no other signficant software running on that server, so it's fairly certain that Trac is the cause.

We're quite happy to provide more details/feedback if it helps solve this issue.

[Remy Blank]

Can you try to strace the apache processes idling at 10%? That may give an indication about what't they are doing.

As a workaround, you could run mod_wsgi in daemon mode, and limit the number of requests before restarting the process (see the maximum-requests argument to WSGIDaemonProcess).