Version 32 (modified by 8 years ago) ( diff ) | ,
---|
Contents
Switch to the Jinja2 Template Engine
Background
Jinja2 is a template engine that combines the advantages of Genshi (pure Python, nice templates, flexible) and ClearSilver (speed!).
We've decided some time ago to remove the legacy support for the ClearSilver template engine for Trac 1.0 (r10570), because of its many inconveniences, and to switch to the nicer Genshi template engine in Trac 0.11. While we managed to keep Genshi memory usage somewhat in control (#6614), the speed was never really adequate, especially for big changesets and for displaying source files over a few thousand lines of code (see TracDev/Performance#Genshi for details). Hence the proposal to look into an alternative template engine.
The Jinja2 template engine has the peripheral benefit of being used by the Django community: trac-dev/KqWPQWuZ63k/GPfda0_PDgAJ.
Overview of activities and progress
There is an experimental branch which supports this proposal: cboos.git@jinja2. A mirror is available in github.
Automatic builds for the jinja2
branch: on Travis,
on AppVeyor. (Current build failures are unrelated to jinja2; they are due to the use of a recent pytz package together with an outdated "trunk" basis for the branch)
Status of the branch (2016-07-30):
- TODO rebase on current trunk
- TODO verify what happens with templates in plugins (e.g. SpamFilter plugin)
- ported all Genshi templates from trac/tracopt (10703 Jinja2 lines corresponding to 8059 Genshi lines)
- detailed porting guide: PortingFromGenshiToJinja
- DONE accesskey support (e.g. use
${accesskey("f")}
in templates) - DONE clarify upgrade path for plugins that came to rely on
ITemplateStreamFilter
s? 127/898 plugins (14.1%) on trac-hacks.org usefilter_stream()
→ see replacing ITemplateStreamFilter - DONE clarify how to handle themeing? → see HtmlTemplates#Jinjaarchitecture
- DONE rewrite tag builders
or use lightweight string templates? → tagFragment
/Element
builder API has been reimplemented - DONE hack
ITemplateStreamFilter
support for Jinja2 templates - DONE site.html replacement, for example try to reproduce t.e.o customizations → TracInterfaceCustomization
- others?
See also this Trac-Dev discussion from 2010, which is still pertinent. While we managed to release Genshi 0.6 since then, the issue is a recurring one, see this recent (2016-01) Genshi question on Trac-Users.
The topic is now (Feb / March 2016) again discussed on trac-dev.
Experimenting with Jinja2 (2.8)
Nothing like a few numbers to make a point.
These are the timings for rendering r3871 (don't try this one here, please), with the diff options set to side-by-side, in place modifications, served by tracd on my development laptop. This generates a page weighing from 11.5MB (Genshi) to 10.3MB (Jinja2) in size.
Genshi | Jinja2 | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
stream | blob | generate | stream (5) | stream (10) | stream (100) | stream (1000) | blob | |||||||||
1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | |
TTFB | 16600 | 15670 | 25530 | 24460 | 2020 | 1160 | 2030 | 1160 | 2070 | 1170 | 2150 | 1230 | 2280 | 1230 | 3370 | 2450 |
CD | 16090 | 16050 | 387 | 1240 | 2820 | 2720 | 2730 | 2640 | 2730 | 2680 | 2470 | 2390 | 2350 | 2250 | 488 | 1060 |
Total | 32690 | 31720 | 25917 | 25700 | 4840 | 3880 | 4760 | 3800 | 4800 | 3850 | 4620 | 3620 | 4630 | 3480 | 3850 | 3510 |
Rdr | — | — | 23533 | 23273 | — | — | — | — | — | — | — | — | — | — | 1477 | 1263 |
Some explanations:
- Genshi (0.7 with speedups)
- stream means we return content via
Stream.serialize
and send chunks as we have them - blob means we first generate all the content in memory with
Stream.render
, then send it at once
- stream means we return content via
- Jinja2 (2.8 with speedups)
- generate means we use
Template.generate
and send chunks as we have them - stream means we use the
TemplateBuffer
wrapper on the above, which groups a few chunks (given by the number in parentheses) together before we send them; for a chunk size of 100, we get the best compromise: still a very low TTFB and a reduced Content download time; actually the sweet spot is probably between 10 and 100, and will most certainly depend on the actual content (I just tested 75 which gives 1160/2430 for example) - blob means we first generate all the content in memory with
Template.render
- generate means we use
- both:
- 1st, is the time in ms for the first request, sent right after a server restart
- 2nd, is the time in ms for the second request, sent just after the first (usually the 3rd and subsequent requests would show the same results as this 2nd request)
We measure:
- TTFB (Time to first byte), as given by Chrome network panel in the developer window
- CD (Content download), idem
- Rdr (template rendering time), mostly significant for the "blob" method otherwise it also takes the network latency into account
All values are given in milliseconds.
Note that even if the total "blob" time seems better than the total "stream" one, the lower TTFB is nevertheless a major benefit for the streaming variant, as this means the secondary requests can start earlier. In this case, it can finish before the main request.
In addition, while I didn't measure precisely the memory usage, Genshi made the python.exe process jump from 109MB to 239MB while rendering the request (blob). The memory seems to be freed afterwards (there were no concurrent requests). By contrast, with Jinja2 the memory spike was 106MB to 126MB.
In another experiment, I used the memory_profiler on Windows, which provided the following results when rendering a big changeset, for a rendered page weighing 4.48MB (it was a side-by-side diff):
Genshi xhtml: 293.612 total (load=0.017, generate=0.017, filter=0.000, render=293.578) Filename: d:\Trac\repos\trunk\trac\web\chrome.py Line # Mem usage Increment Line Contents ================================================ 1283 101.6 MiB 0.0 MiB @profile 1284 def genshi(): 1285 101.6 MiB 0.0 MiB buffer = StringIO() 1286 101.6 MiB 0.0 MiB t5 = time.time() 1287 101.6 MiB 0.0 MiB stream.render(method, doctype=doctype, out=buffer, 1288 153.2 MiB 51.6 MiB encoding='utf-8') 1289 158.0 MiB 4.9 MiB gs = buffer.getvalue().translate(_translate_nop, 1290 158.0 MiB 0.0 MiB _invalid_control_chars) 1291 158.0 MiB 0.0 MiB t6 = time.time() 1292 158.0 MiB 0.0 MiB show_times('Genshi', t2 - t1, t4 - t3, t5a - t4a, t6 - t5, 1293 158.1 MiB 0.0 MiB method) 1294 158.1 MiB 0.0 MiB return gs
vs.
Line # Mem usage Increment Line Contents ================================================ 1255 101.6 MiB 0.0 MiB @profile 1256 def jinja(mode='render'): 1257 101.6 MiB 0.0 MiB if jtemplate: 1258 101.6 MiB 0.0 MiB j5 = time.time() 1259 101.6 MiB 0.0 MiB if mode == 'render': 1260 111.1 MiB 9.4 MiB js = jtemplate.render(jdata) 1261 111.1 MiB 0.0 MiB j5a = time.time() 1262 115.4 MiB 4.3 MiB js = js.encode('utf-8') \ 1263 115.4 MiB 0.0 MiB .translate(_translate_nop, 1264 106.7 MiB -8.7 MiB _invalid_control_chars) 1265 106.7 MiB 0.0 MiB j6 = time.time() 1266 106.7 MiB 0.0 MiB show_times('Jinja2', j2 - j1, 0, j5a - j5, j6 - j5a, 1267 106.7 MiB 0.0 MiB 'html') 1268 106.7 MiB 0.0 MiB return js
In summary, this means that for the big problematic pages, we can easily have a 10x speedup and more, by migrating to Jinja2, and this with a much lighter memory footprint. For smaller pages, the speed-up is between 5x to 10x as well.
Genshi to Jinja2 Migration
Some systematic comparison of the Genshi and Jinja2 template syntax can be seen in PortingFromGenshiToJinja#Changesinthetemplatesyntax (that page was modeled after the old PortingFromClearSilverToGenshi page).
See also PortingFromGenshiToJinja/Example for a full example presented side-by-side.
To facilitate the creation of error-free Jinja2 templates for HTML (or XML), we also wrote a utility called jinjachecker, which helps troubleshoot the most common nesting problems.