[[PageOutline(2-5)]] = Porting Templates from Genshi to Jinja2 The following documentation is primarily targeted at plugin developers who wish to adapt their Genshi templates to the Jinja2 template engine that will be used in Trac [milestone:1.4]. == Overview We start we some examples, showing both the legacy Genshi templates and the new Jinja2 templates, highlighting their main differences. The second part of the document describes the Python code changes, from what you need to change to trigger the use of the Jinja2 renderer instead of the legacy Genshi renderer which still kicks in if nothing changes, to the new ways of generating content. Finally we go to great length to explain the most difficult part of the migration, how to replace the deprecated `ITemplateStreamFilter` interface which has no direct equivalent with Jinja2. In the last part of this document, we try to cover all the Genshi features used by Trac and present their Jinja2 equivalent. Whenever possible, we tried to minimize these differences by customizing the Jinja2 syntax. For example, we use `${...}` for variable expansion, like Genshi does, instead of `{{...}}`. Another aspect of our usage convention is that we favor [#ifthenelse line statements] over `{% ... %}`. So even someone familiar with the "default" Jinja2 syntax should glance through this document to see how "we" use Jinja2, as summarized in the table below. Note that Genshi will be supported concurrently with Jinja2 only for a short while, for the 1.3.x development period and for the 1.4-stable period. This support is removed in Trac [milestone:1.5.1]. If for some reason you're stuck to having to support Genshi templates, you'll have to stick to Trac 1.2.x or 1.3.1. But you really should make the transition effort as Jinja2 templates are 5-10x faster than their Genshi equivalent, for only a 1/5th of their cost in memory usage. == Examples! Before going into the details of the code changes involved and the precise differences in the template syntax between the two systems, let's see at a glance how the templates look like. === Standalone template Let's first take a simple full-contained example from the Trac source, the simple index.html / jindex.html templates. - Genshi [source:tags/trac-1.2/genshi/templates/index.html index.html]: {{{#!html+genshi Available Projects

Available Projects

}}} - Jinja [source:/tags/trac-1.4/trac/templates/index.html index.html]: {{{#!html+jinja ${_("Available Projects")}

${_("Available Projects")}

}}} In this small example, there's no common Trac layout used (as the index is a bit special). For how a "normal" template looks like, see for example [source:tags/trac-1.4/trac/templates/diff_view.html diff_view.html], another small template. The generic templates in [source:tags/trac-1.4/trac/templates trac/templates] are also interesting because we still have their Genshi equivalent close by, in [source:tags/trac-1.4/trac/templates/genshi trac/templates/genshi], so you can easily compare them if you're stuck during the migration of your own templates. Note that a Jinja2 .html template can usually be rendered directly in the browser, to have a rough idea about how it will look like: {{{#!div style="border: 2px solid #ccc; margin: 1em; padding: 0 2em" {{{#!html (html mode on purpose here!)

${_("Available Projects")}

}}} }}} Though there's absolutely no constraints on what text a Jinja2 template may contain, for templates that will produce HTML (or XML), it will be useful if the template is itself already '''a well-formed XML document'''. **Never** go back to the bad old habits from the ClearSilver time, were sometimes the logic in those templates took advantage of the lack of well-formedness constraints, e.g. by conditionally inserting end/start pairs of tags to split sequences. Such templates were hard to maintain, and you always have cleaner alternatives. The [JinjaChecker jinjachecker] tool should also help you maintain well-formed templates by stripping off Jinja2 expressions and line statements before attempting to XML validate the document ([pypi:lxml] should be installed for this feature). === "Standard" templates By "standard", we mean templates that follow the standard Trac layout, and even adapt to the currently selected theme. Instead of the Genshi way of including a template containing filters, the Jinja2 way follows an "object oriented" approach, with inheritance and overriders. Consider that some named sections (or "blocks") of the base template are similar to "methods", imagine that you only have to "subclass" this base template and "reimplement" the overridable methods with your specific content, and there you have it. More specifically, you'll have to "extend" the [source:/tags/trac-1.4/trac/templates/layout.html layout.html] template, and redefine the "head", and "content" blocks if needed. All the details are available in HtmlTemplates#Jinja2architecture, including a walkthrough for the specific example of the [source:tags/trac-1.4/trac/search/templates/search.html search.html] template. For the `search.html` example we focus on the //structure// of the templates, the //include// relationship and the decomposition in //blocks//. But we also have a complete conversion [./Example example], which displays the Genshi wiki_view.html template and the Jinja2 `wiki_view.html` template side-by-side, along with comments explaining the conversion choices. == Changes in the controllers == === Implementing the `IRequestHandler` interface #IRequestHandler With Genshi, the data for the template is basically a `dict`, which has to be returned by `process_request` at the same time as the template name. This hasn't changed with Jinja2. The `IRequestHandler.process_request` method has seen one, important, change: instead of returning a triple of the template, data, and content type, a simple pair of template and data must be returned. If the legacy return convention is used, this means that `'template.html'` is supposed to be a Genshi template: {{{#!py return 'template.html', data, None }}} (`None` here is interpreted to mean the default content type, i.e. `'text/html'`) The new return convention is simpler, and means that `'template.html'` is now supposed to be a Jinja2 template: {{{#!py return 'template.html', data }}} If a special content-type must be used, or if other variation on the generation of the content must be specified, this can now be done by passing a `dict`: {{{#!py return 'template.html', data, {'content_type': 'application/rss+xml'} }}} This has the advantage of supporting a few more keywords (see the API doc), and to be extensible with more metadata at little cost. Note that as long as we have to support the legacy Genshi templates, a `None` value passed as third argument won't be interpreted as an empty `dict`, but rather as an empty `content_type`. === Implementing `IAdminPanelProvider` and `IPreferencePanelProvider` #PanelProviders Plugins which implement custom admin or preference panels must follow the same conventions in `render_admin_panel` and `render_preference_panel` as the ones explained above for [#IRequestHandler IRequestHandler.process_request]: - a return value of `(template, data)` means that `template` is the name of a Jinja2 template - a return value of `(template, data, None)` means that `template` in this case is the name of a Genshi template The only "problem" is that in this specific case, the legacy API for the return value was also `(template, data)`. So in this case, the return value needs to be changed if the template remains a Genshi template. See for example [TH16217] (TH:FullBlogPlugin) and below in [#i18n] (SpamFilter). === Generating content ==== Rendering template "fragments" When one wants to directly render a template, the Chrome `render_template` can still be used, as before: {{{#!py return Chrome(self.env).render_template( req, 'query_results.html', data, None, fragment=True) }}} However, `render_template` prepares all the data needed to render a page in the full default layout. It also now consistently returns an output that is prepared to be sent back to the client. So if you need to embed the generated content in other generated content, this method is the best choice. `render_fragment` can be used instead. It returns a `Markup` string when generating output for the web (`text=False`) or an `unicode` string when generating plain text output (`text=True`). //See [source:tags/trac-1.4/trac/ticket/query.py#L1455 Ticket Query] macro (''table'' mode)// When the fragment needs to be sent to the client, there's still a better choice than `render_template`, it's `generate_fragment`, as it won't impose as much overhead on the data dictionary as `render_template`. It's best suited for responding to XHRs: {{{#!python if req.is_xhr: # render and return the content only stream = Chrome(self.env).generate_fragment( req, 'changeset_content.html', data) req.send(stream) }}} This automatically retrieves the `use_chunked_encoding` TracIni setting and uses it to return an iterable. In any case, the returned value can be sent directly from the `Request` object. There's even a lower-level public API in `Chrome` for generating content using Jinja2 templates, which provides even greater control. //See the [source:tags/trac-1.4/trac/ticket/notification.py#L311 rendering of the ticket change notification e-mail]// See the API documentation for further details. ==== The `tag` builder #tag Genshi provided a nice Python API for programmatically building (X)HTML elements. This consisted of the `Fragment`, `Element` and the `tag` builder, all from the `genshi.builder` module: {{{#!python from genshi import Element, Fragment, tag }}} This has now been replaced by an equivalent API which lives in `trac.util.html`, so the above import should be replaced with: {{{#!python from trac.util.html import Element, Fragment, tag }}} Note that the `html` symbol from `trac.util.html` which used to be a alias to `genshi.builder.tag` is now naturally an alias to `trac.util.html.tag`. One way to write "portable" code would be: {{{#!python from trac.util.html import html as tag }}} You can then use the `tag` builder API regardless of its origin. The behavior of the new `tag` builder is nearly the same as the old one, except that it has even more "knowledge" about the HTML format. For example, for the `class` attribute (or rather, `class_` as `class` is a reserved Python keyword), and for the `style` attribute, dicts can be given as parameters instead of plain strings. Other attributes, like `checked`, will be omitted when given a `False` value. As this special behavior could be unwanted when arbitrary XML must be generated instead of XHTML, another builder, `xml`, is now available. The `xml` builder can be used the same way as the `tag` builder, but when serialized, its only special behavior is to omit attributes which have the value `None`. ==== The `Markup` class Likewise, if you want to use the `Markup` class, you should write: {{{#!python from trac.util.html import Markup }}} In "old" versions of Trac, you'll get `genshi.core.Markup`, whereas now you'll get `markupsafe.Markup`: as we're using Jinja2, we're also making use of its direct dependency [PyPI:MarkupSafe]. ==== The `escape` function {{{#!python from trac.util.html import escape }}} Note that in a similar way to `Markup`, `escape` now also comes from `markupsafe`, with some slight adaptations, as `markupsafe.escape` always escapes the quotes, which is something we don't do by default. Hence always import `escape` from `trac.util.html`, never directly from `markupsafe` or Jinja2, unless you really know what you're doing. ==== Produce the correct content directly instead of relying on post-processing There were two post-processing steps performed by Trac using Genshi stream filters from which plugin writers did benefit, possibly unknowingly: 1. the addition of the `__FORM_TOKEN` hidden parameter to
elements, necessary for successful POST operations 2. accessibility key enabling/disabling As this no longer happen, it's now the responsibility of plugin writers to add this in their content. This is simple enough: {{{#!html+jinja ...
}}} This gets even simpler thanks to a default macro: {{{#!html+jinja
${jmacros.form_token_input()} ...
}}} The `jmacros` in the above corresponds to the [source:tags/trac-1.4/trac/templates/macros.html trac/templates/macros.html] default macros, and this file is included by default (in `layout.html`), so you don't have to bother to include it yourself, as long as your template extends `layout.html`. For the accessibility key, it's also quite simple: instead of hard-coding the key as an `accesskey="e"` attribute, simply use the `accesskey('e')` function call, it will know if it has to produce the attribute or not depending on the current user preferences. === Replacing the `ITemplateStreamFilter` interface #ReplacingITemplateStreamFilter One of the strengths of Genshi was its ability to transform the normal HTML content and, for example, to inject arbitrary content at any point in the HTML, thanks to the use of the `Transform` stream filter and its API. However, as elegant as it was, this feature was the main performance killer of Genshi, and Jinja2 doesn't propose an equivalent, for good reasons. With Jinja2, the content is produced in one step, with no possibility of post-processing. The only way left to alter the generated content is to **perform these modifications dynamically on client-side using JavaScript**. In February 2016, 127/898 plugins (14.1%) on trac-hacks.org made use of `filter_stream()` from the `ITemplateStreamFilter` interface. So this means this specific step of the migration, perhaps the less straightforward, will be of interest for most plugin developers. Note that though we guarantee some level of support for the `ITemplateStreamFilter` during the transition period, the new suggested way also works great with earlier versions of Trac (1.0 and 1.2, perhaps even 0.12), so there's really no reason to maintain both versions once you did the switch. One strong incentive for dropping the `ITemplateStreamFilter` usage in your code is that **by not doing so you kill all the performance benefits** brought by the switch to Jinja2. The support of `ITemplateStreamFilter` implies that we first render the page to HTML using Jinja2, then parse it back as an HTML stream and feed this stream to the Genshi filter, so that it can be transformed by these filters, and then finally rendered again(!). The steps for replacing `filter_stream()` are the following: 1. implement `ITemplateProvider` if you haven't done so already, as you'll need to provide a JavaScript file 2. implement `IRequestFilter` if you haven't done so already, as you'll need to add the