Severe memory issues/leaks.

[josh@…]

My company has Trac and SVN set up to use locally for our internal development. However, we're having issues whereby Trac's memory usage is slowly building up over a period of around 1-3 days, to the point where it consumes all available RAM, forcing us to restart Apache, or worse, do a hard-reboot of the server.

Running top reveals that it's the httpd (Apache) processes which are eating the RAM. They are also gradually increacing their CPU usage, to the point where the machine is generally idling at around ~10% cpu.

The Swap partition is also not being utilised much, suggesting(?) the actual cause of the slowdown is probably thrashing.

Obviously, this is not ideal, and we'd quite like to identify the cause of the problem and fix it ASAP.

Details of the setup are:

• Machine
  • OS: Virtualised Fedora 12 64-bit Linux
  • Memory: 1Gb
  • Swap: 2Gb
  • CPU: Intel 2.4GHz Dual-core

• Software
  • Apache: 2.2.14
  • Python: 2.6.2
  • Trac: 0.11.7
  • SVN: 1.6.9
  • mod_wsgi: 3.1

• Apache
  • Running 10 instances. (i.e. Whatever that build defaults to, since I didn't configure that.)

• Trac
  • ~20 individual projects (all w/own SVN repos).
  • Plugins: AdvancedTicketWorkflow?, Developer, MasterTickets?, NeverNotifyUpdater?, PrivateTickets?, TimingAndEstimation?, VirtualTicketPermissions?, Worklog, WYSIWYG.

With the exception of Trac and it's plugins, all the software is installed from the Fedora repos. Trac and plugins are installed from source.

There's no other signficant software running on that server, so it's fairly certain that Trac is the cause.

We're quite happy to provide more details/feedback if it helps solve this issue.

[rblank]

Can you try to strace the apache processes idling at 10%? That may give an indication about what't they are doing.

As a workaround, you could run mod_wsgi in daemon mode, and limit the number of requests before restarting the process (see the maximum-requests argument to WSGIDaemonProcess).

[cboos]

Also, try running Trac without any plugins. If there's no issues when no plugin are used, reintroduce them one by one, in order to find the culprit.

[josh@…]

strace on httpd process

[josh@…]

I've attached the output of running strace on one of the httpd child processes. It's worth noting that it's no one process that is eating 10% of the CPU, that's spread across 8-16 processes.

Tweaking the MPM settings for the prefork Apache module (not mod_wsgi, haven't gotten around to that yet) seems to slow the problem down, but hasn't stopped it yet. Will try running WSGI in demon mode next.

[cboos]

You should try to upgrade to 0.12-stable, the fixes for #9111 should also prove effective in your situation.

[cboos]

No feedback in more than 3 months, closing.

Probably fixed by r10002 and friends.