= Add Support for the Jinja2 Template Engine

We've decided some time ago to remove the legacy support for the ClearSilver template engine, for Trac 1.0 (r10570). Clearsilver had its share of inconveniences, enough that we decided to switch to the nicer [http://genshi.edgewall.org/ Genshi template engine] in 0.11, but to be honest ClearSilver was **very** fast and memory lenient. While we managed to keep Genshi memory usage somewhat in control (remember #6614?), the speed was never really adequate, especially for big changesets and for displaying source files over a few thousand lines of code (see TracDev/Performance#Genshi for details).

So one solution would be to switch once again, to a template engine that would combine the advantages of Genshi (pure Python, nice templates, flexible) and ClearSilver (speed!). Such a beast seems to exist now: **[http://jinja.pocoo.org/2/documentation/ Jinja2]**.


Several points remain to be clarified: 
 * what will be the upgrade path for plugins that came to rely on `IStreamFilter`s?
 * [#Themeing how to handle themeing?]
 * should we rewrite tag builders or use lightweight string templates?
 * others?

See also [googlegroups:trac-dev:fc8d8c0447140110 this Trac-Dev discussion] from 2010, which is still pertinent. Well, obviously we managed to release Genshi 0.6 since then, but the issue is a recurring one, see this recent (2016-01)
[gmessage:trac-users:PYqQ4UDRnl8/wg8lQzrGDAAJ Genshi question] on Trac-Users.


== Experimenting with Jinja2 (2.8)

Nothing like a few numbers to make a point ;-)

These are the timings for rendering !r3871, with the diff options set to side-by-side, in place modifications, served by tracd on my development laptop. This generates a page weighing 11.5MB (Genshi) to 10.3MB (Jinja2) in size.

||       ||||||||= Genshi                             ||||||||||||||||||||||||= Jinja2                                                                                     ||
||       ||||= stream          ||||= blob             ||||= generate    ||||= stream (5) ||||= stream (10)     ||||= stream (100)   ||||= stream (1000)||||= blob          ||
||       ||= //1st//||= 2nd    ||= //1st// ||=   2nd  ||=//1st//||= 2nd ||=//1st//||= 2nd  ||=//1st//||= 2nd   ||=//1st//||= 2nd    ||=//1st//||= 2nd  ||=//1st//||=   2nd ||
||= TTFB ||//16600//||**15670**|| //25530//||    24460||//2020//||  1160||//2030//||   1160||//2070//||    1170||//2150//|| **1230**||//2280//||   1230||//3370//||    2450||
||= CD   ||//16090//||**16050**||   //387//||     1240||//2820//||  2720||//2730//||   2640||//2730//||    2680||//2470//|| **2390**||//2350//||   2250|| //488//||    1060||
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
||= Total||//32690//||**31720**|| //25917//||    25700||//4840//||  3880||//4760//||   3800||//4800//||    3850||//4620//|| **3620**||//4630//||   3480||//3850//||    3510||
||= Rdr  ||       --||       --|| //23533//||**23273**||      --||    --||      --||     --||      --||      --||      --||       --||      --||     --||//1477//||**1263**||

Some explanations:
 - Genshi (0.7 with speedups)
   - ''stream'' means we return content via `Stream.serialize` and send chunks as we have them
   - ''blob'' means we first generate all the content in memory with `Stream.render`, then send it at once
 - Jinja2 (2.8 with [http://www.pocoo.org/projects/markupsafe/ speedups])
   - ''generate'' means we use `Template.generate` and send chunks as we have them
   - ''stream'' means we use the `TemplateBuffer` wrapper on the above, which groups a few chunks (given by the number in parenthesis) together before we send them;
     for a chunk size of **100**, we get the best compromise: still a very low TTFB and a reduced Content download time; actually the sweet spot is probably between
     10 and 100, and will most certainly depend on the actual content (I just tested 75 which gives 1160/2430 for example)
   - ''blob'' means we first generate all the content in memory with `Template.render`
 - both:
   - ''1st'', is the time in ms for the first request, sent right after a server restart
   - 2nd, is the time in ms for the second request, sent just after the first (usually the 3rd and subsequent requests would show the same results as this 2nd request)

We measure:
 - TTFB (Time to first byte), as given by Chrome network panel in the developer window
 - CD (Content download), idem
 - Rdr (template rendering time), mostly significant for the "blob" method otherwise it also takes the network latency into account

Note that even if the total "blob" time seems better than the total "stream" one, the lower TTFB is nevertheless a major benefit for the streaming variant, as this means the secondary requests can start earlier (and in this case, finish before the main request).

In addition, while I didn't measure precisely the memory usage, Genshi made the python.exe process jump from 109MB to 239MB while rendering the request (blob). The memory seems to be freed afterwards (there were no concurrent requests). By contrast, with Jinja2 the memory spike was 106MB to 126MB.

In summary, this means that for the big problematic pages, we can easily have a 10x speedup and more, by migrating to Jinja2,and this with a much lighter memory footprint.
For smaller pages, the speed-up is between 5x to 10x as well.


== Implementation details

=== Themeing
==== Genshi theme
I've never really tried the TH:ThemeEnginePlugin plugin, or alternatives, so I can't be sure if I got it right, but from what I can see in Trac's code base itself, the idea with Genshi-based themeing (and page architecture in general) was to have a dynamically loaded "theme" template page that would primarily be in charge of the main structure of all HTML pages.

Let's take the example of a simple "end user" page, the search.html page.
What happens is that:
 - search.html includes
   - layout.html, which includes
     - $chrome.theme (typically "theme.html", the default theme page that ships with Trac)

In more details, the 
[source:tags/trac-1.0.9/trac/search/templates/search.html search.html] is structured like this:
{{{#!html+genshi
<html>
  <xi:include href="layout.html" />
  <head>
    <title>Search</title>
    ...
  </head>
  <body>
    <div id="content" class="search">
      <h1>Search</h1>
      ...
    </div>
  </body>
</html>
}}}
 - it starts by including the layout.html page (`<xi:include href="layout.html" />`)
 - it then provides the `<head>` and `<body>` elements specific to that search page; 
   these elements will be processed by the `<py:match>`es filters defined so far, 
   in the order in which they have been included

The [source:tags/trac-1.0.9/trac/templates/layout.html  layout.html] 
page in turn is structure like this:
{{{#!html+genshi
<html>
  <py:match path="head"><head>
    <title py:with="title = list(select('title/text()'))">
      ${title} – ${project.name or 'Trac'}  <!-- e.g.  "Search - Trac" -->
    </title>
    ...
    ${select("*[local-name() != 'title']|text()|comment()")}
  </head></py:match>

  <py:match path="body"><body>

    ${select('*|text()|comment()')}         <!-- e.g. "<h1>Search</h1>"
                                                       ... -->
    ...
    <div id="altlinks" py:if="'alternate' in chrome.links">
      ...
    </div>
  </body></py:match>

  <xi:include href="$chrome.theme"><xi:fallback /></xi:include>
</html>      
}}}
 - it defines `<py:match>` filters for transforming the content provided 
   in `<head>` and `<body>` elements in the including template (search.html);
   the head filter adds some content to the `<title>` and prepends some content
   in the <head>, the body filter appends some content in  the <body>
 - it then **dynamically** includes some "theme" page, `$chrome.theme`

By default, this theme page will be our
[source:tags/trac-1.0.9/trac/templates/theme.html theme.html] template:
{{{#!xml
<html>
  <body>
    <div id="banner">
      ...
    </div>
    <div id="main">
      ... 
      ${select('*|text()|comment()')}       <!-- e.g. "<h1>Search</h1>"
                                                       ... 
                                                       <div id="altlinks" ... </div> -->
    </div>
    <div id="footer">
      ...
    </div>
  </body>
</html>
}}}
 - it defines a `<py:match>` filter on the `<body>` tag 
   (the one that will be produced by the previously applied filters, 
   i.e. the output of the `<py:match>` from the layout.html)
   and it will **embed** that element into some predefined HTML
   structure (inside a div element), and 
   it will prepend and append other divs around it:

So this dynamic theme template has the last say, and can theoretically re-order 
the content generated by the previous filters any way it likes, although in practice
it simply inserts the body content produced by previous steps inside a predefined 
structure (the `<div id="main">`).

An example of another theme page: [https://github.com/chevah/trac-bootstrap-theme/blob/master/templates/theme.html trac-bootstrap-theme's theme.html].



Note that this scheme can be extended to having more intermediate levels, 
for example what we have for the admin panels.

One such panel is
[source:tags/trac-1.0.9/trac/admin/templates/admin_basics.html admin_basics.html]:
{{{#!html+genshi
<html>
  <xi:include href="admin.html" />
  <head>
    <title>Basics</title>
  </head>

  <body>
    ...
  </body>
</html>
}}}
 - it starts by including the admin.html page
 - then it provides the `<head>` and `<body>` elements specific to that panel
    
The [source:tags/trac-1.0.9/trac/admin/templates/admin.html admin.html] page
is similar to the search.html page in that it includes the layout.html, but
it also first contains its own `<py:match>` templates to organize the content of
the admin panel which included it:
{{{#!html+genshi
<html>
  <py:match path="head" once="true"><head>
    <title>Administration: ${select('title/text()')}</title>
    ${select("*[local-name() != 'title']")}
  </head></py:match>

  <py:match path="body" once="true" buffer="false"><body>
    <div id="content" class="admin">
      <h1>Administration</h1>
      <div id="tabs">
      <div id="tabcontent">
        ${select("*|text()")}
        <br style="clear: right" />
      </div>
    </div>

  </body></py:match>

  <xi:include href="layout.html" />
</html>
}}}
 - defines `<py:match>` filters for `<head>` and `<body>` (of the including panel page), 
   which in turn will produce modified `<head>` and `<body>` elements
 - it then includes the layout.html page (see above)

Feel free to brush up your Genshi craft by reading ([G:GenshiTutorial#AddingaLayoutTemplate]),
as I just did ;-)


==== Jinja2 theme

Fortunately Jinja2 can do dynamic includes (or extends in this case) as well,
so the same idea can be transposed: have the end user page extend the layout page, 
then have the layout page extend whatever has been defined to be the theme page.

The differences with Genshi are subtle: while in both case the control of the
output is delegated to the more generic page, with Jinja2 the parent only controls
what it puts around //blocks//. It can put some default content in these blocks,
but the end user page has the final say about what to do with this default content, 
as it can reuse it inside its block (by calling `super()`) or not.

So let's transpose the example.
For as long as we have both template engines coexisting, 
we prefix the new Jinja2 templates with a `j`,
so we're now discussing:
 - jsearch.html, which extends
   - jlayout.html, which extends
     - jtheme.html (as this is our default theme page)

In more details, we describe what happens with the
[source:cboos.git/trac/search/templates/jsearch.html jsearch.html] page:
 - it starts by //extending// the jlayout.html page:
   {{{#!jinja
   # extends 'jlayout.html'
   }}}
   - the [source:cboos.git/trac/templates/jlayout.html jlayout.html] page:
     - first **dynamically** //extends// in turn some "theme" page
       {{{#!jinja
       # extends ('j' + chrome.theme)
       }}}
       - by default, this will be our 
         [source:cboos.git/trac/templates/jtheme.html jtheme.html] page:
         - it defines a `head` block inside of an otherwise empty `<head>` element;
           this means this a "slot" that will be filled by the content
           of the `head` block in the extending templates (in this case,
           jlayout.html)
         - it contains a `<body>` element; 
           as we want to replicate what the original theme.html did,
           what we want to achieve here is to provide a ''slot''
           at the place where we want to substitute in the content
           produced by the jlayout.html template:
           {{{#!html+jinja
           <body>
             # block body
             <div id="banner">
               ...
             </div>
             <div id="main">
               ... 
               # block content
               (here goes the content of the content block produced by layout.html)
               # endblock content
             </div>
             <div id="footer">
               ...
             </div>
             # block body
           </body>
           }}}
           I didn't name that inner block "body", as this could be confusing:
           we're not in control of the `<body>` element there, just of
           a fraction of it, the bottom part of the main div.
           //If// we want to be in complete control of the body in the extended 
           template, we could (as opposed to what happens in Genshi),
           we would simply have to redefine the `body` block which contains
           all of the default structure (possibly reusing the content of that 
           block by a call to `${ super() }`)
     - then the jlayout.html template defines the `head` and `content` blocks;
       it doesn't need to redefine the `body` block, as it's happy with what
       jtheme does with it;
       that `content` block is filled with some predefined, generic content,
       mostly the same stuff that could be found in the corresponding 
       `<py:match>` filters in layout.html; 
  - the end user template jsearch.html has to redefine the head and content
    blocks, and has to place a `${ super() }` expression in order to 
    insert the default content proposed by the extended template at 
    the right place

Depending how one looks at it, it seems this approach is even more flexible than
what we had in Genshi, as the end user template can decide which bits of the
parent template it wants or not ("bottom-up" control), something that was not 
readily doable with Genshi ("top-down" control).

I omitted the discussion of the title block (not even implemented yet) and the 
replacement for the `<xi:include href="site.html>` template 
(`# include site_head.html` and `# include site_body.html`).