UnicodeEncodeError raised while rendering timeline rss when author of event is unknown and unicode

[Jun Omae]

I noticed UnicodeEncodeError raised while rendering timeline rss on production environment.

2020-02-12 18:42:48,520 ERROR: Genshi UnicodeEncodeError error while rendering template (unknown template location)
Traceback (most recent call last):
  File "/venv/lib/python2.7/site-packages/trac/web/chrome.py", line 1180, in iterable_content
    for chunk in stream.serialize(method, **kwargs):
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 241, in __call__
    for kind, data, pos in stream:
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 671, in __call__
    for kind, data, pos in stream:
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 776, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 596, in __call__
    for ev in stream:
  File "/venv/lib/python2.7/site-packages/genshi/core.py", line 292, in _ensure
    for event in stream:
  File "/venv/lib/python2.7/site-packages/genshi/core.py", line 292, in _ensure
    for event in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/base.py", line 639, in _include
    for event in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/base.py", line 579, in _flatten
    for kind, data, pos in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/directives.py", line 726, in __call__
    value = _eval_expr(expr, ctxt, vars)
  File "/venv/lib/python2.7/site-packages/genshi/template/base.py", line 289, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "/venv/lib/python2.7/site-packages/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "/venv/lib/python2.7/site-packages/trac/templates/author_or_creator.rss", line 16, in <Expression '?'>
    <py:with vars="author = email_map[author] if show_email_addresses and email_map and '@' not in author else author">
  File "/venv/lib/python2.7/site-packages/genshi/template/eval.py", line 338, in lookup_item
    val = getattr(obj, key, UNDEFINED)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

The issue occurs when email_map[author] expression is evaluated in genshi template and author is not existent in email_map dict.

How to reproduce same exception

>>> from genshi.template import MarkupTemplate
>>> t = MarkupTemplate('<x xmlns:py="http://genshi.edgewall.org/" py:with="v=d[k]" />')
>>> unicode(t.generate(d={'key': 42}, k=u'\u200b'*16))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python2.7/site-packages/genshi/core.py", line 251, in __unicode__
    return self.render(encoding=None)
  File "/venv/lib/python2.7/site-packages/genshi/core.py", line 184, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 57, in encode
    return _encode(''.join(list(iterator)))
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 241, in __call__
    for kind, data, pos in stream:
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 671, in __call__
    for kind, data, pos in stream:
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 776, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/venv/lib/python2.7/site-packages/genshi/output.py", line 596, in __call__
    for ev in stream:
  File "/venv/lib/python2.7/site-packages/genshi/core.py", line 274, in _ensure
    event = stream.next()
  File "/venv/lib/python2.7/site-packages/genshi/template/base.py", line 639, in _include
    for event in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/base.py", line 579, in _flatten
    for kind, data, pos in stream:
  File "/venv/lib/python2.7/site-packages/genshi/template/directives.py", line 726, in __call__
    value = _eval_expr(expr, ctxt, vars)
  File "/venv/lib/python2.7/site-packages/genshi/template/base.py", line 289, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "/venv/lib/python2.7/site-packages/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "<string>", line 1, in <Expression '?'>
  File "/venv/lib/python2.7/site-packages/genshi/template/eval.py", line 338, in lookup_item
    val = getattr(obj, key, UNDEFINED)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

Proposed changes

trac/templates/author_or_creator.rss

@@ -13 +13 @@
      py:with="email_map = value_of('email_map', None)" py:strip="">
   <py:if test="author">
     <!--! Try our best to retrieve an email address if wanted and possible -->
-    <py:with vars="author = email_map[author] if show_email_addresses and email_map and '@' not in author else author">
+    <py:with vars="author = (email_map.get(author, author)
+                             if show_email_addresses and email_map and '@' not in author
+                             else author)">
       <py:choose>
         <author py:when="show_email_addresses and '@' in author">${format_author(author)}</author>
         <dc:creator py:otherwise="">${format_author(author)}</dc:creator>

trac/timeline/tests/web_ui.py

@@ -120 +120 @@
 
         self.assertIn('prev', req.chrome['links'])
 
+    def test_render_rss_with_unicode_unknown_author(self):
+        self.env.config.set('trac', 'show_email_addresses', 'enabled')
+        self.env.known_users.append(('blah', 'Blah user', 'blah@example.org'))
+        req = MockRequest(self.env, path_info='/timeline',
+                          args={'format': 'rss'})
+        rv = TimelineModule(self.env).process_request(req)
+        self.assertEqual('timeline.rss', rv[0])
+        event = ('mock', datetime_now(utc), u'jöé', None)
+        rv[1]['events'] = [
+            {'kind': event[0], 'date': event[1], 'author': event[2],
+             'data': event[3], 'event': event, 'provider': None,
+             'dateuid': '42', 'render': lambda field, context: 'mock'},
+        ]
+        output = Chrome(self.env).render_template(req, rv[0], rv[1], rv[2])
+        self.assertIsNotNone(output)
+
 
 def suite():
     suite = unittest.TestSuite()

[Ryan J Ollos]

Please move back to 1.0.20 if you wish to fix for release 1.0.20 (#13294).

[Ryan J Ollos]

The change looks good to me. Feel free to move to 1.4.2 if you want to include the change in the upcoming release.

[Jun Omae]

Sorry for my delay.

Proposed fix is depended on Genshi. I'll push to 1.0-stable, also I'm going to confirm whether the same issue is existent on 1.4-stable.

[Jun Omae]

Fixed in [17413] and merged in [17414] (only tests for 1.2-stable).

1.4-stable doesn't have the issue. Also, 1.2-stable doesn't have it because email_map[author] is not evaluated on author_or_creator.css.

[Jun Omae]

The root cause is a Genshi issue. I've filed at ​https://github.com/edgewall/genshi/issues/28.