Version 18 (modified by 8 years ago) ( diff ) | ,
---|
Add Support for the Jinja2 Template Engine
We've decided some time ago to remove the legacy support for the ClearSilver template engine, for Trac 1.0 (r10570). Clearsilver had its share of inconveniences, enough that we decided to switch to the nicer Genshi template engine in 0.11, but to be honest ClearSilver was very fast and memory lenient. While we managed to keep Genshi memory usage somewhat in control (remember #6614?), the speed was never really adequate, especially for big changesets and for displaying source files over a few thousand lines of code (see TracDev/Performance#Genshi for details).
So one solution would be to switch once again, to a template engine that would combine the advantages of Genshi (pure Python, nice templates, flexible) and ClearSilver (speed!). Such a beast seems to exist now: Jinja2.
Several points remain to be clarified:
- what will be the upgrade path for plugins that came to rely on
ITemplateStreamFilter
s?- 127/898 plugins (14.1%) on trac-hacks.org use
filter_stream()
- → replacing ITemplateStreamFilter
- 127/898 plugins (14.1%) on trac-hacks.org use
- how to handle themeing? → see HtmlTemplates#Jinjaarchitecture
- should we rewrite tag builders or use lightweight string templates? → tag
Fragment
/Element
builder has been reimplemented - others?
See also this Trac-Dev discussion from 2010, which is still pertinent. Well, obviously we managed to release Genshi 0.6 since then, but the issue is a recurring one, see this recent (2016-01) Genshi question on Trac-Users.
Experimenting with Jinja2 (2.8)
Nothing like a few numbers to make a point ;-)
These are the timings for rendering r3871, with the diff options set to side-by-side, in place modifications, served by tracd on my development laptop. This generates a page weighing 11.5MB (Genshi) to 10.3MB (Jinja2) in size.
Genshi | Jinja2 | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
stream | blob | generate | stream (5) | stream (10) | stream (100) | stream (1000) | blob | |||||||||
1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | 1st | 2nd | |
TTFB | 16600 | 15670 | 25530 | 24460 | 2020 | 1160 | 2030 | 1160 | 2070 | 1170 | 2150 | 1230 | 2280 | 1230 | 3370 | 2450 |
CD | 16090 | 16050 | 387 | 1240 | 2820 | 2720 | 2730 | 2640 | 2730 | 2680 | 2470 | 2390 | 2350 | 2250 | 488 | 1060 |
Total | 32690 | 31720 | 25917 | 25700 | 4840 | 3880 | 4760 | 3800 | 4800 | 3850 | 4620 | 3620 | 4630 | 3480 | 3850 | 3510 |
Rdr | — | — | 23533 | 23273 | — | — | — | — | — | — | — | — | — | — | 1477 | 1263 |
Some explanations:
- Genshi (0.7 with speedups)
- stream means we return content via
Stream.serialize
and send chunks as we have them - blob means we first generate all the content in memory with
Stream.render
, then send it at once
- stream means we return content via
- Jinja2 (2.8 with speedups)
- generate means we use
Template.generate
and send chunks as we have them - stream means we use the
TemplateBuffer
wrapper on the above, which groups a few chunks (given by the number in parenthesis) together before we send them; for a chunk size of 100, we get the best compromise: still a very low TTFB and a reduced Content download time; actually the sweet spot is probably between 10 and 100, and will most certainly depend on the actual content (I just tested 75 which gives 1160/2430 for example) - blob means we first generate all the content in memory with
Template.render
- generate means we use
- both:
- 1st, is the time in ms for the first request, sent right after a server restart
- 2nd, is the time in ms for the second request, sent just after the first (usually the 3rd and subsequent requests would show the same results as this 2nd request)
We measure:
- TTFB (Time to first byte), as given by Chrome network panel in the developer window
- CD (Content download), idem
- Rdr (template rendering time), mostly significant for the "blob" method otherwise it also takes the network latency into account
All values are given in milliseconds.
Note that even if the total "blob" time seems better than the total "stream" one, the lower TTFB is nevertheless a major benefit for the streaming variant, as this means the secondary requests can start earlier (and in this case, finish before the main request).
In addition, while I didn't measure precisely the memory usage, Genshi made the python.exe process jump from 109MB to 239MB while rendering the request (blob). The memory seems to be freed afterwards (there were no concurrent requests). By contrast, with Jinja2 the memory spike was 106MB to 126MB.
In summary, this means that for the big problematic pages, we can easily have a 10x speedup and more, by migrating to Jinja2, and this with a much lighter memory footprint. For smaller pages, the speed-up is between 5x to 10x as well.
Genshi to Jinja2 Migration
Some systematic comparison of the Genshi and Jinja2 template syntax can be seen in PortingFromGenshiToJinja#Changesinthetemplatesyntax (modeled after the old PortingFromClearSilverToGenshi page).
See also PortingFromGenshiToJinja/Example for a full example presented side-by-side.
jinjachecker
Jinja2 is very helpful when it detects any kind of error, as you always end up with a meaningful backtrace. Nevertheless, it can be tedious to get the nesting of control structures right.
To help with that, there's the contrib/jinjachecker.py tool.
python contrib/jinjachecker.py trac/wiki/templates/jwiki_view.html
It first analyzes the Jinja2 control structures, and tries to provide some helpful diagnostics in case of nesting or stylistic errors. It also adds curly braces to the statements, so if you have an editor which can do matching of brace pairs, you can quickly spot the origin o a nesting problem.
Finally, while the indentation of the statements is free in Jinja2 templates, being consistent with it also helps to ensure a proper nesting.
# -- Jinja2 check for 'trac/wiki/templates/jwiki_view.html' 12 EXTENDS "jlayout.html" 18 {BLOCK title 19 {IF title: 21 }IF 22 }BLOCK title 25 {BLOCK head 26 SET modify_perm = 'WIKI_MODIFY' in perm(page.resource) 27 SET is_not_latest = page.exists and page.version != latest_version 31 {IF version or page.author == 'trac': 33 }IF 34 {IF modify_perm: 40 }IF 52 }BLOCK head 56 {BLOCK content 57 SET modify_perm = 'WIKI_MODIFY' in perm(page.resource) 58 SET create_perm = 'WIKI_CREATE' in perm(page.resource) 59 SET admin_perm = 'WIKI_ADMIN' in perm(page.resource) 60 SET is_not_latest = page.exists and page.version != latest_version 63 {IF version: 67 {WITH 68 SET version = page.version 69 SET author = authorinfo(page.author) 70 SET date = pretty_dateinfo(page.time) 71 SET hef = href.wiki(page.name, action='diff', version=page.version) 72 {TRANS version, author, date, href 77 }TRANS 78 }WITH 84 }IF 87 {IF page.exists: 91 SET last_modification = (page.comment and 97 {IF not version: 100 {WITH 101 SET href = href.wiki(page.name, action='diff', 103 SET date = pretty_dateinfo(page.time) 104 {TRANS href, last_modification, date 109 }TRANS 110 }WITH 116 }IF 117 ELSE: 119 {TRANS name = name_of(page.resource) 124 }TRANS 126 }IF 129 {WITH 130 SET alist = attachments 131 SET compact = True 132 SET foldable = True 133 INCLUDE "jlist_of_attachments.html" 134 }WITH 136 {WITH 137 SET delete_perm = 'WIKI_DELETE' in perm(page.resource) 138 SET rename_perm = 'WIKI_RENAME' in perm(page.resource) 139 {IF modify_perm or create_perm or delete_perm: 141 {IF modify_perm or create_perm: 145 {IF is_not_latest and modify_perm: 148 ELIF page.exists and modify_perm: 151 ELIF not page.exists and create_perm: 154 {IF templates: 161 {FOR t in sorted(templates): 165 }FOR 168 }IF 169 }IF 173 {IF page.exists: 174 {WITH alist = attachments 175 INCLUDE "jattach_file_form.html" 176 }WITH 177 }IF 178 }IF 180 {IF page.exists and rename_perm: 187 }IF 188 {IF page.exists and delete_perm: 193 {IF page.version == latest_version: 195 }IF 199 }IF 201 }IF 202 }WITH 204 {IF not page.exists and higher: 208 {FOR markup in higher: 210 }FOR 213 }IF 215 {IF not page.exists and related: 219 {FOR markup in related: 221 }FOR 224 }IF 230 }BLOCK content # -- Jinja2 OK
As a second step, jinjachecker removes the Jinja2 markup and performs a validation of the document, using lxml.
# -- HTML check for 'trac/wiki/templates/jwiki_view.html' 1 ... 13 14 <!DOCTYPE html> 15 <html> 16 <head> 17 <title> 18 19 20 ${title} ${ super() } 21 22 23 </title> 24 25 26 27 28 29 ${ super() } 30 ... 52 53 </head> 54 55 <body> 56 57 58 59 60 61 <div id="content" class="${classes('wiki', create=not page.exists)}"> 62 ... 213 214 215 </div> 216 217 ${ super() } 218 219 220 </body> 221 </html> # -- HTML OK