Switch to the Jinja2 Template Engine

for cboos.git@jinja2-trunk-r15379

Background

​Jinja2 is a ​template engine that combines the advantages of Genshi (pure Python, nice templates, flexible) and ClearSilver (speed!).

A bit of history first…

Trac started its life in 2005 as a very reactive web application, as the page content was generated with ClearSilver, a template engine written in C. However, as fast as it was, it had serious quirks, troubleshooting errors was quite difficult, installation on Windows was problematic, etc. Enters Genshi in 2006, which proved to be way more flexible, and more convenient to develop with. It was also way slower, but we were confident that this would eventually improve. Trac has been converted to use Genshi for the 0.11 release (2008), and we're using it ever since.

While we managed to keep Genshi memory usage somewhat in control (#6614), the speed was never really adequate, especially for big changesets and for displaying source files over a few thousand lines of code (see TracDev/Performance#Genshi for details). The maintenance of Genshi itself has often proved to be problematic to work with, and some issues remain unsolved (#11184).

Hence the proposal to look into an alternative template engine. Jinja2 had always looked like a very serious candidate. Kajiki was briefly considered, but while the performance was on par with Jinja2, it was not mature enough (see [7acf61ba/cboos.git]). As Jinja2 seemed to hold all its promises (see [7d4b6f84/cboos.git]), the migration effort started in this direction beginning of 2016.

Besides, the Jinja2 template engine has the peripheral benefit of being used by the Django community: ​trac-dev/KqWPQWuZ63k/GPfda0_PDgAJ.

Overview of activities and progress

There is an experimental branch which supports this proposal: cboos.git@jinja2. A mirror is available in ​github.

This branch has been reworked and prepared for merge on trunk, in cboos.git@jinja2-trunk-r15341.

Status of the branch (2017-01-15):

• TODO merge on trunk (#12639)
• DONE rebase on current trunk
• DONE support templates in plugins (e.g. SpamFilter plugin) - r15353
• ported all Genshi templates from trac/tracopt (10703 Jinja2 lines corresponding to 8059 Genshi lines)
• detailed porting guide: PortingFromGenshiToJinja
• DONE accesskey support (e.g. use ${accesskey("f")} in templates)
• DONE clarify upgrade path for plugins that came to rely on ITemplateStreamFilters? 127/898 plugins (14.1%) on trac-hacks.org use filter_stream() → see replacing ITemplateStreamFilter
• DONE clarify how to handle themeing? → see HtmlTemplates#Jinjaarchitecture
• DONE rewrite tag builders or use lightweight string templates? → tag Fragment/Element builder API has been reimplemented
• DONE hack ITemplateStreamFilter support for Jinja2 templates FIXME still a few glitches with that (CDATA gets messed up)
• DONE site.html replacement, for example try to reproduce t.e.o customizations → TracInterfaceCustomization
• TODO update l10n guide for plugins CookBook/PluginL10N

See also ​this Trac-Dev discussion from 2010, which is still pertinent. While we managed to release Genshi 0.6 since then, the issue is a recurring one, see this recent (2016-01) ​Genshi question on Trac-Users.

The topic is now (Feb / March 2016) again discussed on ​trac-dev.

Experimenting with Jinja2 (2.8)

Nothing like a few numbers to make a point.

These are the timings for rendering r3871 (don't try this one here, please), with the diff options set to side-by-side, in place modifications, served by tracd on my development laptop. This generates a page weighing from 11.5MB (Genshi) to 10.3MB (Jinja2) in size.

	Genshi				Jinja2
	stream		blob		generate		stream (5)		stream (10)		stream (100)		stream (1000)		blob
	1st	2nd	1st	2nd	1st	2nd	1st	2nd	1st	2nd	1st	2nd	1st	2nd	1st	2nd
TTFB	16600	15670	25530	24460	2020	1160	2030	1160	2070	1170	2150	1230	2280	1230	3370	2450
CD	16090	16050	387	1240	2820	2720	2730	2640	2730	2680	2470	2390	2350	2250	488	1060
Total	32690	31720	25917	25700	4840	3880	4760	3800	4800	3850	4620	3620	4630	3480	3850	3510
Rdr	—	—	23533	23273	—	—	—	—	—	—	—	—	—	—	1477	1263

Some explanations:

• Genshi (0.7 with speedups)
  • stream means we return content via Stream.serialize and send chunks as we have them
  • blob means we first generate all the content in memory with Stream.render, then send it at once
• Jinja2 (2.8 with ​speedups)
  • generate means we use Template.generate and send chunks as we have them
  • stream means we use the TemplateBuffer wrapper on the above, which groups a few chunks (given by the number in parentheses) together before we send them; for a chunk size of 100, we get the best compromise: still a very low TTFB and a reduced Content download time; actually the sweet spot is probably between 10 and 100, and will most certainly depend on the actual content (I just tested 75 which gives 1160/2430 for example)
  • blob means we first generate all the content in memory with Template.render
• both:
  • 1st, is the time in ms for the first request, sent right after a server restart
  • 2nd, is the time in ms for the second request, sent just after the first (usually the 3rd and subsequent requests would show the same results as this 2nd request)

We measure:

• TTFB (Time to first byte), as given by Chrome network panel in the developer window
• CD (Content download), idem
• Rdr (template rendering time), mostly significant for the "blob" method otherwise it also takes the network latency into account

All values are given in milliseconds.

Note that even if the total "blob" time seems better than the total "stream" one, the lower TTFB is nevertheless a major benefit for the streaming variant, as this means the secondary requests can start earlier. In this case, it can finish before the main request.

In addition, while I didn't measure precisely the memory usage, Genshi made the python.exe process jump from 109MB to 239MB while rendering the request (blob). The memory seems to be freed afterwards (there were no concurrent requests). By contrast, with Jinja2 the memory spike was 106MB to 126MB.

In another experiment, I used the ​memory_profiler on Windows, which provided the following results when rendering a big changeset, for a rendered page weighing 4.48MB (it was a side-by-side diff):

Genshi xhtml: 293.612 total (load=0.017, generate=0.017, filter=0.000, render=293.578)
Filename: d:\Trac\repos\trunk\trac\web\chrome.py

Line #    Mem usage    Increment   Line Contents
================================================
  1283    101.6 MiB      0.0 MiB               @profile
  1284                                         def genshi():
  1285    101.6 MiB      0.0 MiB                   buffer = StringIO()
  1286    101.6 MiB      0.0 MiB                   t5 = time.time()
  1287    101.6 MiB      0.0 MiB                   stream.render(method, doctype=doctype, out=buffer,
  1288    153.2 MiB     51.6 MiB                                 encoding='utf-8')
  1289    158.0 MiB      4.9 MiB                   gs = buffer.getvalue().translate(_translate_nop,
  1290    158.0 MiB      0.0 MiB                                                    _invalid_control_chars)
  1291    158.0 MiB      0.0 MiB                   t6 = time.time()
  1292    158.0 MiB      0.0 MiB                   show_times('Genshi', t2 - t1, t4 - t3, t5a - t4a, t6 - t5,
  1293    158.1 MiB      0.0 MiB                              method)
  1294    158.1 MiB      0.0 MiB                   return gs

vs.

Line #    Mem usage    Increment   Line Contents
================================================
  1255    101.6 MiB      0.0 MiB               @profile
  1256                                         def jinja(mode='render'):
  1257    101.6 MiB      0.0 MiB                   if jtemplate:
  1258    101.6 MiB      0.0 MiB                       j5 = time.time()
  1259    101.6 MiB      0.0 MiB                       if mode == 'render':
  1260    111.1 MiB      9.4 MiB                           js = jtemplate.render(jdata)
  1261    111.1 MiB      0.0 MiB                           j5a = time.time()
  1262    115.4 MiB      4.3 MiB                           js = js.encode('utf-8') \
  1263    115.4 MiB      0.0 MiB                                  .translate(_translate_nop,
  1264    106.7 MiB     -8.7 MiB                                             _invalid_control_chars)
  1265    106.7 MiB      0.0 MiB                           j6 = time.time()
  1266    106.7 MiB      0.0 MiB                           show_times('Jinja2', j2 - j1, 0, j5a - j5, j6 - j5a,
  1267    106.7 MiB      0.0 MiB                                      'html')
  1268    106.7 MiB      0.0 MiB                           return js

In summary, this means that for the big problematic pages, we can easily have a 10x speedup and more, by migrating to Jinja2, and this with a much lighter memory footprint. For smaller pages, the speed-up is between 5x to 10x as well.

Genshi to Jinja2 Migration

Some systematic comparison of the Genshi and Jinja2 template syntax can be seen in PortingFromGenshiToJinja#Changesinthetemplatesyntax (that page was modeled after the old PortingFromClearSilverToGenshi page).

See also PortingFromGenshiToJinja/Example for a full example presented side-by-side.

To facilitate the creation of error-free Jinja2 templates for HTML (or XML), we also wrote a utility called jinjachecker, which helps troubleshoot the most common nesting problems.