Changes between Version 16 and Version 17 of TracDev/Performance
- Timestamp:
- Feb 6, 2015, 8:53:28 AM (9 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
TracDev/Performance
v16 v17 1 = Improving Trac Performance =1 = Improving Trac Performance 2 2 3 This is the developer-oriented side of TracPerformance. 4 While in the latter we try to analyse the different factors that 5 come into play, here we'll try to discuss about the practical solutions 6 we can imagine. 7 8 For a start, here's a raw list of the tickets tagged with the ''performance'' keyword: 3 On TracPerformance some ideas for improving the performance of a Trac install have been discussed, this page collects some suggestions for performance meansurement and implementation ideas. Here's a list of the tickets tagged with the ''performance'' keyword: 9 4 [[TicketQuery(status=!closed,keywords=~performance,order=severity,format=table)]] 10 5 11 == Performance Analysis == 12 === Load Testing === 13 - using JMeter; see TracPerformance/LoadTestingWithJmeter 14 - using `ab` (see [./0.11.5#HowItimed]) 15 ''For timing template generation I used ApacheBench and tossed out "warmup" requests, sometimes testing with keep alive (getting slightly better req/s)'' 6 == Performance Analysis 7 8 === Load Testing 9 10 Large loads can be simulated in the following ways: 11 - using [http://jmeter.apache.org/ JMeter], see TracPerformance/LoadTestingWithJmeter 12 - using [http://httpd.apache.org/docs/2.2/programs/ab.html ApacheBench] `ab`, see [./0.11.5#HowItimed]. 13 ''For timing template generation, you can use !ApacheBench and toss out "warmup" requests, sometimes testing with keep alive (getting slightly better req/s)'' 16 14 {{{ 17 15 ab [-k] -c 1 -n 10 url 18 16 }}} 19 17 20 === Profiling === 18 === Profiling 19 20 Page load measurements can be collected by using profiling tools: 21 21 - #7490 contains some profiling data. 22 - Shane Caraveo gave some instructions about profiling Trac ([./0.11.5#Profilingissues]); see also #8507 which contains his scripts.23 - here's some particularlyinteresting profiling data from ticket:7490#comment:106, typical of a request triggering an environment reload. Edited to make it more readable:22 - Instructions about profiling Trac: [./0.11.5#Profilingissues]; see also #8507 which contains scripts by Shane Caraveo. 23 - Some interesting profiling data from ticket:7490#comment:106, typical of a request triggering an environment reload. Edited to make it more readable: 24 24 {{{ 25 25 935179 function calls (910454 primitive calls) in 6.699 CPU seconds … … 65 65 - trac corresponds to Trac-0.11.5rc2-py2.5.egg and genshi to Genshi-0.5.1-py2.5-freebsd-6.1-RELEASE-i386.egg 66 66 - `Environment.is_component_enabled` is unexpectedly slow 67 - as Shane found out, config is really taking more time than expected. Only 0.1ms per call to get/options, but as this happens 11000 times ...1.1s.68 - also slow due to the following (line 236)67 - as Shane found out, config is really taking more time than expected. Only 0.1ms per call to get/options, but as this happens 11000 times, it is 1.1s. 68 - also slow as a result of the following (line 236): 69 69 {{{ 70 70 #!python … … 72 72 }}} 73 73 There is some potential for caching `rules` here. 74 - The problem should be fixed by r8644:8645, but I still have to confirm this by performing the profiling myself...74 - The problem should be fixed by r8644:8645, but this is to be confirmed by performing the profiling. 75 75 76 == Improvement Opportunities 76 77 77 == Improvement Opportunities == 78 79 === web front-end 78 === Web front-end 80 79 81 80 r8215 introduced HTTP1.1 support for tracd, but at the cost of making the `Content-Length` header mandatory. We should have added chunked-encoding support at the same time. Now that we have it (via #717), we could revert to streaming some of the responses (see ticket:717#comment:48). 82 81 83 === Genshi === 82 === Genshi 83 84 84 The impact of Genshi is especially important for requests generating a big amount of data, like the changeset view or the file browser view, especially when compared to ClearSilver. 85 85 86 86 From [./0.11.5#Genshi], we can get the following ideas: 87 - revert to `out.write(_encode(u''.join(list(iterator))))` instead of using the StringIO (that was a change done during #6614, but we could have a `favor_speed_over_memory` setting) -88 - avoid the whitespace filter (same setting)87 - revert to `out.write(_encode(u''.join(list(iterator))))` instead of using the StringIO; that was a change done during #6614, but we could have a `favor_speed_over_memory` setting 88 - avoid the whitespace filter, same setting 89 89 90 Additionally, there's the idea to increase the size of the template cache for Genshi (#7842), also something that could be done if a 90 Additionally, there's the idea to increase the size of the template cache for Genshi (#7842), also something that could be done if a `favor_speed_over_memory` configuration is active. 91 91 92 We could also possibly gain some speed by not passing all the content through all the filters, but pre-render some parts in a faster way, then wrap them in `Markup` objects. That would make them opaque to the filters, but that would be most of the time OK (or the plugins that really need that could somehow selectively turn this optimization off). See #5499 (browser) and #7975 (changeset).92 We could also gain some speed by not passing all the content through all the filters, but pre-render some parts in a faster way, then wrap them in `Markup` objects. That would make them opaque to the filters, but that would be most of the time OK, or the plugins that really need that could somehow selectively turn this optimization off. See #5499 (browser) and #7975 (changeset). 93 93 94 Also, the generated XHTML itself could certainly be improved: for example, using <pre> instead of tables rows for rendering lines in the browser view (#7055).94 Also, the generated XHTML itself could certainly be improved: for example, using <pre> instead of tables rows for rendering lines in the browser view. See #7055. 95 95 96 Let's not forget alternatives: 97 - Proposals/Jinja 98 - [http://kajiki.pythonisito.com Kajiki] 99 - [http://groups.google.com/group/genshi/browse_thread/thread/4f66c2ca5302b807# Genshi Compiler] 96 === gc.collect 100 97 101 === gc.collect === 102 According to Shane's analysis, the systematic `gc.collect()` call after every request is one of the most critical performance killer for the average request ([./0.11.5#gc.collect]). 103 See proposed implementation of a secondary thread taking care of this, in #8507. 98 According to Shane's analysis, the systematic `gc.collect()` call after every request is one of the most critical performance killers for the average request: [./0.11.5#gc.collect]. 99 See proposed implementation of a secondary thread taking care of this: #8507. 104 100 105 === database level optimizations === 106 - #4425 101 === Database level optimizations 102 103 - #4425 - report of all unwritten pages 107 104 - Some MySQL specific changes have been suggested, see #6986. 108 105 - #6654 - pad revision numbers with leading zeroes … … 115 112 On the topic of transactions, we should probably use the transaction idea from the WikiRename branch, as refined by rblank on Trac-dev (googlegroups:trac-dev:21d21ad9866fc12b). This would help to visualize the span of write transactions in the code and better handle query failures (#8379). 116 113 117 === permission checking ===114 === Permission checking 118 115 119 116 The whole TracFineGrainedPermissions was not designed with performance in mind and this shows in several places, like the ad-hoc additions of various caches in source:trunk/trac/perm.py. … … 121 118 Some places are still up for optimization, in particular the AuthzPolicy, which graduated from sample plugin to optional component. See ticket:9348#comment:6 for hints about what could be done. 122 119 123 === misc. === 124 - some more modules could use paginated results (#6128 - timeline, #6101 - browser) or on-demand display (#515 - changeset view), or... no display at all (#5170) 125 - there was once the suspicion that DEBUG logging level could significantly impact performance (get some numbers) 120 === Miscellaneous 126 121 127 === "Page speed" improvements === 122 - Some more modules could use paginated results (#6128 - timeline, #6101 - browser) or on-demand display (#515 - changeset view), or... no display at all (#5170) 123 - There was once the suspicion that DEBUG logging level could significantly impact performance. We need some numbers on this. 124 125 === "Page speed" improvements 128 126 129 127 Improve static resource delivery: 130 128 - better use of caching headers: 131 129 - add `Cache-Control: public` 132 - add `Expires: `''(future)'', 130 - add `Expires: `''(future)'', `Cache-Control: max-age` together with URL fingerprinting, #9936 133 131 - fix `Vary:` header #6367 134 132 - verify we call `req.check_modified()` as appropriate #4022