Slow trac performance

[jerk]

just have multiple instances of trac projects running with trac 0.11b2 and do have a very bad performance. the db is about 500MB big, but according to top not the bottleneck. moreover the trac.cgi process creates a 100% load for about 30-45 seconds before the page is displayed and finally the machine is a 512 MB, 1.6 GHz Xeon Processor.

Secondly I tried running tracd alternatively to my mod_python/apache setup and had the same performance and also deactivating the plugins did not significantly improve the sites performance.

[cboos]

Chances are that with only 512MB available the server is swapping, as there are known memory issues (see #6614). Can you move the server to a machine with more RAM? (at least 1GB). If not, you should at least use Python 2.5 which is somewhat better at giving memory back to the system.

A "100% load for about 30-45 second" is unfortunately not uncommon with lengthy pages like big changesets now that they're rendered with Genshi. There are ongoing efforts to speed things up at the Genshi level, including a GSoC proposal.

In order to be able to do something more specific about your issue, we need to know additional details: for which pages do you see this bad performance, all? some? If it's the timeline, you can reduce the default "daysback" parameter. For big changesets, there are other parameters to tune (TracIni#changeset-section), etc.

[cboos]

#7035 closed as duplicate. The increased time needed to render big changesets can even lead to timeouts in Safari.

[jerk]

Replying to cboos:

Chances are that with only 512MB available the server is swapping, as there are known memory issues (see #6614). Can you move the server to a machine with more RAM? (at least 1GB). If not, you should at least use Python 2.5 which is somewhat better at giving memory back to the system.

The machine is not swapping. I checked this already.

A "100% load for about 30-45 second" is unfortunately not uncommon with lengthy pages like big changesets now that they're rendered with Genshi. There are ongoing efforts to speed things up at the Genshi level, including a GSoC proposal. In order to be able to do something more specific about your issue, we need to know additional details: for which pages do you see this bad performance, all? some? If it's the timeline, you can reduce the default "daysback" parameter. For big changesets, there are other parameters to tune (TracIni#changeset-section), etc.

The performance issue is esp. regarding viewing and creating tickets. It is in no way related to view change sets.

[cboos]

My bad, too much time spent on debugging memory related issues :-)

So OK, your particular problem has probably nothing to do with the rendering. Are you sure this happens also for simply viewing tickets, not only when creating and modifying them? If this is the latter case, then I strongly suspect issues related to the notification (like in #3220). Please try disabling ticket notifications and if this is indeed the problem, check your SMTP server configuration.

[jerk]

I just tried this, but it did not help. I disabled the whole notification settings and created a new ticket. this was still very slow. further more things like viewing the report list and viewing some ticket lists is slow.

though I still think this is more like a general issue. I also checked the performance of the timeline, wiki, roadmap, admin etc... and the are all slow...

[anonymous]

here is just my top output when I'm loading some trac page... does this give you a better idea?

top - 10:37:56 up 9 days, 20:04,  1 user,  load average: 0.39, 0.80, 0.56
Tasks:  45 total,   4 running,  41 sleeping,   0 stopped,   0 zombie
Cpu(s): 16.6%us, 10.3%sy,  0.0%ni, 72.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Mem:    524440k total,   459008k used,    65432k free,    99456k buffers
Swap:        0k total,        0k used,        0k free,   195188k cached
 Unknown command - try 'h' for help
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30987 www-data  17   0 20080  11m 3648 R 26.3  2.3   0:00.79 trac.cgi
    1 root      16   0  1952  664  568 S  0.0  0.1   0:00.24 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 R  0.0  0.0   0:00.53 ksoftirqd/0
    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.33 events/0
    6 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
    7 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
    8 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 xenwatch
    9 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 xenbus
   14 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
   44 root      15   0     0    0    0 S  0.0  0.0   0:46.37 pdflush
   45 root      15   0     0    0    0 S  0.0  0.0   0:53.27 pdflush
   47 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
   46 root      15   0     0    0    0 S  0.0  0.0   0:00.49 kswapd0
  568 root      17  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
 1492 root      15   0     0    0    0 S  0.0  0.0   0:41.50 kjournald
 1754 root      12  -5     0    0    0 S  0.0  0.0   0:00.00 kmirrord
 1938 root      16   0  1752  560  388 S  0.0  0.1   0:00.13 dhclient
 2073 root      16   0  1632  640  508 S  0.0  0.1   0:02.92 syslogd
 2080 root      16   0  1584  400  328 S  0.0  0.1   0:00.00 klogd
 2104 postgres  16   0 19576 2284 1900 S  0.0  0.4   0:29.48 postmaster
 2107 postgres  15   0 10376 1856  484 S  0.0  0.4   0:24.53 postmaster
 2108 postgres  15   0  9520 1080  584 S  0.0  0.2   0:17.68 postmaster
 2120 postgres  16   0  4524 1720 1392 S  0.0  0.3   0:54.71 pg_autovacuum
 2173 Debian-e  16   0  5340  988  692 S  0.0  0.2   0:00.09 exim4
 2197 root      16   0  4932 1100  768 S  0.0  0.2   0:00.00 sshd
 2229 daemon    15   0  1836  424  300 S  0.0  0.1   0:00.00 atd
 2236 root      16   0  2196  896  712 S  0.0  0.2   0:00.94 cron
 2301 root      16   0  6440 4744 1556 S  0.0  0.9   0:07.28 munin-node
 2330 root      17   0  1576  500  432 S  0.0  0.1   0:00.00 getty
21860 root      16   0 24028 9112 5192 S  0.0  1.7   0:00.64 apache2
24420 www-data  16   0 29744  12m 2860 S  0.0  2.3   0:01.34 apache2
15150 www-data  16   0 30116  11m 2852 S  0.0  2.3   0:01.06 apache2
15457 www-data  17   0 35656  17m 2856 S  0.0  3.4   0:06.60 apache2
15481 www-data  15   0 28128  10m 2868 S  0.0  2.1   0:04.33 apache2
15492 www-data  17   0 32512  14m 2580 S  0.0  2.9   0:05.20 apache2
15493 www-data  16   0 34636  16m 2868 S  0.0  3.2   0:12.34 apache2
15566 www-data  16   0 27696  10m 2864 S  0.0  2.0   0:00.62 apache2
15567 www-data  16   0 32320  14m 2860 S  0.0  2.8   0:04.99 apache2
15576 www-data  16   0 27468 9840 2580 S  0.0  1.9   0:00.66 apache2
 2981 www-data  16   0 24164 5644 1500 S  0.0  1.1   0:00.03 apache2
30611 root      16   0  7864 2336 1888 R  0.0  0.4   0:00.05 sshd
30614 root      15   0  5104 1736 1336 S  0.0  0.3   0:00.01 bash
30986 root      16   0  2232 1112  860 R  0.0  0.2   0:00.00 top

[cboos]

Well, first don't expect to get any good performance when using TracCgi, having a slow response time for any page is expected there, as there's a lot of modules to load and initialization overhead that has to be repeated for each request (even more so if you misconfigured the rendering of static documents and use Trac's chrome module to render them instead of letting Apache do it - see TracCgi#MappingStaticResources).

You should really try to use either one of the faster deployment mode for Trac (basically anything but CGI, like TracModPython, TracModWsgi, TracStandalone).

You mentioned that you already tried with tracd, and that this didn't provide any performance gain. But can you try to monitor the tracd log output (with log_type = stderr and log_level = DEBUG, see TracLogging) and then try to figure out where tracd spends its time?

[jerk]

Dear, thanks for this great helpful response. I will try this these days and be back with some more debugging output ;). Thanks in advance...

[jerk]

I just enabled the debug output and got quite some warnings and errors. I will clean up everything and post here, if the performance stays that way...

[jerk]

ok, cleaning up the debug messages (resolving the sources of the warnings and errors) as well als moving to the apache (proxy) and tracd duo produced acceptable speed.

thanks for help

[cboos]

(for fixing resolution)

[cboos]

So now that it works, if you don't mind you can also summarize what where those errors which made tracd very slow for you.

[jerk]

I'm not sure. I could not really determine what it made slow. I seems to me like that tracd is getting faster after about 1d of operating... before its as slow as the other stuff... there might be any caching reasons for...