Version 8 (modified by 8 years ago) ( diff ) | ,
---|
Add Support for the Jinja2 Template Engine
We've decided some time ago to remove the legacy support for the ClearSilver template engine, for Trac 1.0 (r10570). Clearsilver had its share of inconveniences, enough that we decided to switch to the nicer Genshi template engine in 0.11, but to be honest ClearSilver was very fast and memory lenient. While we managed to keep Genshi memory usage somewhat in control (remember #6614?), the speed was never really adequate, especially for big changesets and for displaying source files over a few thousand lines of code (see TracDev/Performance#Genshi for details).
So one solution would be to switch once again, to a template engine that would combine the advantages of Genshi (pure Python, nice templates, flexible) and ClearSilver (speed!). Such a beast seems to exist now: Jinja2.
Several points remain to be clarified:
- what will be the upgrade path for plugins that came to rely on
IStreamFilter
s? - how to handle themeing?
- should we rewrite tag builders or use lightweight string templates?
- others?
See also this Trac-Dev discussion from 2010, which is still pertinent. Well, obviously we managed to release Genshi 0.6 since then, but the issue is a recurring one, see this recent (2016-01) Genshi question on Trac-Users.
Experimenting with Jinja2 (2.8)
Nothing like a few numbers to make a point ;-)
These are the timings for rendering r3871, with the diff options set to side-by-side, in place modifications, served by tracd on my development laptop. This generates a page weighing 11.5MB (Genshi) to 10.3MB (Jinja2) in size.
Genshi | Jinja2 | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
stream | blob | generate | stream (5) | stream (10) | stream (100) | stream (1000) | blob | |||||||||
1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | |
TTFB | 16600 | 15670 | 25530 | 24460 | 2020 | 1160 | 2030 | 1160 | 2070 | 1170 | 2150 | 1230 | 2280 | 1230 | 3370 | 2450 |
CD | 16090 | 16050 | 387 | 1240 | 2820 | 2720 | 2730 | 2640 | 2730 | 2680 | 2470 | 2390 | 2350 | 2250 | 488 | 1060 |
Total | 32690 | 31720 | 25917 | 25700 | 4840 | 3880 | 4760 | 3800 | 4800 | 3850 | 4620 | 3620 | 4630 | 3480 | 3850 | 3510 |
Rdr | — | — | 23533 | 23273 | — | — | — | — | — | — | — | — | — | — | 1477 | 1263 |
Some explanations:
- Genshi (0.7 with speedups)
- stream means we return content via
Stream.serialize
and send chunks as we have them - blob means we first generate all the content in memory with
Stream.render
, then send it at once
- stream means we return content via
- Jinja2 (2.8 with speedups)
- generate means we use
Template.generate
and send chunks as we have them - stream means we use the
TemplateBuffer
wrapper on the above, which groups a few chunks (given by the number in parenthesis) together before we send them; for a chunk size of 100, we get the best compromise: still a very low TTFB and a reduced Content download time; actually the sweet spot is probably between 10 and 100, and will most certainly depend on the actual content (I just tested 75 which gives 1160/2430 for example) - blob means we first generate all the content in memory with
Template.render
- generate means we use
- both:
- 1st, is the time in ms for the first request, sent right after a server restart
- 2nd, is the time in ms for the second request, sent just after the first (usually the 3rd and subsequent requests would show the same results as this 2nd request)
We measure:
- TTFB (Time to first byte), as given by Chrome network panel in the developer window
- CD (Content download), idem
- Rdr (template rendering time), mostly significant for the "blob" method otherwise it also takes the network latency into account
Note that even if the total "blob" time seems better than the total "stream" one, the lower TTFB is nevertheless a major benefit for the streaming variant, as this means the secondary requests can start earlier (and in this case, finish before the main request).
In addition, while I didn't measure precisely the memory usage, Genshi made the python.exe process jump from 109MB to 239MB while rendering the request (blob). The memory seems to be freed afterwards (there were no concurrent requests). By contrast, with Jinja2 the memory spike was 106MB to 126MB.
In summary, this means that for the big problematic pages, we can easily have a 10x speedup and more, by migrating to Jinja2,and this with a much lighter memory footprint. For smaller pages, the speed-up is between 5x to 10x as well.
Implementation details
Themeing
Genshi thene
I've never really tried the TH:ThemeEnginePlugin plugin, or alternatives, so I can't be sure if I got it right, but from what I can see in Trac's code base itself, the idea with Genshi-based themeing (and page architecture in general) was the following:
Let's take the example of a simple "end user" page (e.g. search.html):
- it starts by including the layout.html page:
<xi:include href="layout.html" />
- the layout.html page:
- defines
<py:match>
filters for transforming the content provided in<head>
and<body>
elements - then dynamically includes some "theme" page:
<xi:include href="$chrome.theme"><xi:fallback /></xi:include>
- by default, this will be our
theme.html:
- it simply defines a
<py:match>
filter on the<body>
tag (the one that will be produced by the previously applied filters, i.e. the output of the<py:match>
from the layout.html)
- it simply defines a
- by default, this will be our
theme.html:
- defines
- the layout.html page:
- the search.html page then provide the
<head>
and<body>
elements specific to that search page; these elements will be processed by the<py:match>
es filters defined so far in the order in which they have been included (so first the ones defined in layout.html, then the one for thebody
defined in theme.html)
The scheme can be extended with more intermediate levels, for example what we have for the admin panels, e.g the admin_basics.html panel:
- it starts by including the admin.html page:
<xi:include href="admin.html" />
- the admin.html page
- defines
<py:match>
filters for<head>
and<body>
, which in turn will produce modified<head>
and<body>
elements - it then includes the layout.html page (see above)
- defines
- the admin.html page
- then it provides the
<head>
and<body>
elements specific to that panel (they will be process by the<py:match>
filters in the order seen, so first those from admin.html, then those from layout.html and finally those from the dynamically included theme page
Feel free to brush up your Genshi craft by reading (GenshiTutorial#AddingaLayoutTemplate), as I just did ;-)
So this dynamic theme template has the last say, and can theoretically re-order
the content generated by the previous filters any way it likes.
In practice (at least in our default theme.html), it's just a <py:match once>
template which appends a footer to the content of the <body>
element produced
so far.
An example of another theme page: trac-bootstrap-theme's theme.html.