Improving Trac Performance

This is the developer-oriented side of TracPerformance. While in the latter we try to analyse the different factors that come into play, here we'll try to discuss about the practical solutions we can imagine.

For a start, here's a raw list of the tickets tagged with the performance keyword:

Ticket	Summary	Status	Keywords	Owner	Type	Severity
#10606	GitPlugin: Improve performance	new	performance		enhancement	critical
#4245	[PATCH] Tickets are slow with large amount of users and permissions	new	patch performance slow restrict_owner get_users_with_permission		defect	major
#6986	MySQL schema changes for performance	new	performance mysql database patch		enhancement	major
#8813	next_rev is slow, particularly in the direct-svnfs case	new	slow performance newcache svn	anonymous	defect	major
#10267	Version controls diffs of large text files kill the system	new	diff performance genshi		defect	major
#11185	improve speed of TicketQuery for format=table	new	performance genshi jinja2 javascript		enhancement	major
#1216	wiki pages caching	new	objectstore performance		enhancement	normal
#2636	browsing a directory with many files takes a long time	new	performance		enhancement	normal
#2827	Preview attachments in main ticket (or wiki) window	new	thumbail image attachment performance		enhancement	normal
#5499	performance issues when viewing larger files	new	blame performance		defect	normal
#6128	[Patch] Pagination for the Timeline	new	patch pagination performance		enhancement	normal
#6638	Allow file exclusions in diffs	new	bitesized performance		enhancement	normal
#6807	Option to skip revisions with lots of added files	new	performance		enhancement	normal
#7055	Add "toggle line numbers" JavaScript to source browser	new	javascript performance patch		enhancement	normal
#7739	trac & memcached	new	memcached performance		enhancement	normal
#8058	Smarter db connection pool	new	performance, schema, postgresql		enhancement	normal
#8509	[PATCH] use precompiled regex for match_request	new	patch performance		defect	normal
#9938	Improve on static resources	new	performance		enhancement	normal
#10672	Automatically minify Javascript and CSS	new	javascript css performance		enhancement	normal
#11024	Losslessly compress images	new	performance		enhancement	normal
#11206	Reduce join in query process	new	mysql, performance		enhancement	normal
#12745	Changeset view with single revision which has many files is so slow	new	git performance		enhancement	normal
#8639	"View latest revision" links to non-existent (404) page	new	newcache performance		defect	minor
#9172	Trac diff highlighting start and end points	new	performance diff		enhancement	minor
#11826	Optimize ticket.query.execute	new	performance		enhancement	minor

Performance Analysis

Load Testing

• using JMeter; see TracPerformance/LoadTestingWithJmeter
• using ab (see 0.11.5)

For timing template generation I used ApacheBench and tossed out "warmup" requests, sometimes testing with keep alive (getting slightly better req/s)

ab [-k] -c 1 -n 10 url

Profiling

• #7490 contains some profiling data.
• Shane Caraveo gave some instructions about profiling Trac (0.11.5); see also #8507 which contains his scripts.

Improvement Opportunities

Genshi

The impact of Genshi is especially important for requests generating a big amount of data, like the changeset view or the file browser view, especially when compared to ClearSilver.

From 0.11.5, we can get the following ideas:

• revert to out.write(_encode(u''.join(list(iterator)))) instead of using the StringIO (that was a change done during #6614, but we could have a favor_speed_over_memory setting) -
• avoid the whitespace filter (same setting)

Additionally, there's the idea to increase the size of the template cache for Genshi (#7842), also something that could be done if a favor_speed_over_memory configuration is active.

We could also possibly gain some speed by not passing all the content through all the filters, but pre-render some parts in a faster way, then wrap them in Markup objects. That would make them opaque to the filters, but that would be most of the time OK (or the plugins that really need that could somehow selectively turn this optimization off). See #5499 (browser) and #7975 (changeset).

gc.collect

According to Shane's analysis, the systematic gc.collect() call after every request is one of the most critical performance killer for the average request (0.11.5). See proposed implementation of a secondary thread taking care of this, in #8507.