TypeError is raised while rendering wiki text on Genshi 0.6 with speedups

[Jun Omae]

Wiki including [[search:|""]] leads a TypeError on Genshi 0.6 with speedups. Root cause of this issue is genshi.core.escape(). I got the same issue on my production environment.

>>> import genshi
>>> genshi.__version__
'0.6'
>>> from genshi.core import Markup, escape
>>> Markup, escape
(<type 'genshi._speedups.Markup'>, <built-in method escape of type object at 0x7f5d42f5fb60>)
>>> escape('', False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode() argument 2 must be string, not bool

03:34:19 PM Trac[main] ERROR: Internal Server Error: <RequestWithSession "POST '/newticket'">, referrer 'http://27.local:3000/1.0-sqlite/newticket'
Traceback (most recent call last):
  File "<venv>/local/lib/python2.7/site-packages/trac/web/main.py", line 562, in _dispatch_request
    dispatcher.dispatch(req)
  File "<venv>/local/lib/python2.7/site-packages/trac/web/main.py", line 269, in dispatch
    iterable=chrome.use_chunked_encoding)
  File "<venv>/local/lib/python2.7/site-packages/trac/web/chrome.py", line 1115, in render_template
    encoding='utf-8')
  File "<venv>/local/lib/python2.7/site-packages/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "<venv>/local/lib/python2.7/site-packages/genshi/output.py", line 58, in encode
    for chunk in iterator:
  File "<venv>/local/lib/python2.7/site-packages/genshi/output.py", line 339, in __call__
    for kind, data, pos in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/output.py", line 826, in __call__
    for kind, data, pos in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/output.py", line 670, in __call__
    for kind, data, pos in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/output.py", line 771, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "<venv>/local/lib/python2.7/site-packages/genshi/output.py", line 586, in __call__
    for ev in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "<venv>/local/lib/python2.7/site-packages/trac/web/chrome.py", line 1310, in _generate
    for kind, data, pos in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "<venv>/local/lib/python2.7/site-packages/genshi/template/base.py", line 565, in _flatten
    result = _eval_expr(data, ctxt, vars)
  File "<venv>/local/lib/python2.7/site-packages/genshi/template/base.py", line 277, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "<venv>/local/lib/python2.7/site-packages/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "<venv>/local/lib/python2.7/site-packages/trac/ticket/templates/ticket_box.html", line 124, in <Expression u'wiki_to_html(context, ticket.description, escape_newlines=preserve_newlines)'>
    ${wiki_to_html(context, ticket.description, escape_newlines=preserve_newlines)}
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 1586, in format_to_html
    return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 1541, in generate
    escape_newlines)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 1323, in format
    result = re.sub(self.wikiparser.rules, self.replace, line)
  File "<venv>/lib/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 1241, in replace
    replacement = self.handle_match(fullmatch)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 1237, in handle_match
    return internal_handler(match, fullmatch)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 782, in _macrolink_formatter
    return self._lhref_formatter(match, fullmatch)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 618, in _lhref_formatter
    return self._make_lhref_link(match, fullmatch, rel, ns, target, label)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 653, in _make_lhref_link
    fullmatch)
  File "<venv>/local/lib/python2.7/site-packages/trac/wiki/formatter.py", line 664, in _make_link
    return resolver(self, ns, target, escape(label, False))
TypeError: unicode() argument 2 must be string, not bool

The genshi.core.escape() crashes when first argument is None or an empty string. This issue has been filed in ​genshi:#439 and fixed on Genshi 0.7. However, we recommend Genshi 0.6 due to #11184 and the version still has it….

Then, I think we could detect this issue and fix up it like this. Thoughts?

trac/util/html.py

@@ -291 +291 @@
 html = TransposingElementFactory(str.lower)
 
 
+try:
+    escape('', False)  # detecting genshi:#439 on Genshi 0.6 with speedups...
+except TypeError:
+    _escape = escape
+
+    def escape(text, quotes=True):
+        if text:
+            return _escape(text, quotes=quotes)
+        else:
+            return Markup(u'')
+
+
 def plaintext(text, keeplinebreaks=True):
     """Extract the text elements from (X)HTML content
 

[Ryan J Ollos]

That looks like a good workaround.

I noticed the error isn't seen when passing a keyword, so the following might be sufficient:

>>> try:
...   escape('', False)
... except TypeError:
...   _escape = escape
...   
...   def escape(text, quotes=True):
...     return _escape(text, quotes=quotes)
... 
>>> escape("''", False)
<Markup u"''">
>>> escape("''", True)
<Markup u"''">
>>> escape("", False)
<Markup u''>
>>> escape(None, True)
<Markup u'None'>
>>> escape(None, False)
<Markup u'None'>
>>> escape('"', True)
<Markup u'&#34;'>
>>> escape('"', False)
<Markup u'"'>

[Jun Omae]

Interested. However, escape(None, False) returns <Markup u''> on Genshi 0.7. I think work around should lead the same result for this, at least.

	0.6 without speedups	0.6 with speedups	0.7 without speedups	0.7 with speedups
escape(None, False)	<Markup u''>	TypeError	<Markup u''>	<Markup u''>
escape('', False)	<Markup u''>	TypeError	<Markup u''>	<Markup u''>
escape(0, False)	<Markup u''>	TypeError	<Markup u''>	<Markup u''>
escape(1, False)	AttributeError	<Markup u'1'>	AttributeError	<Markup u'1'>
escape(False, False)	<Markup u''>	TypeError	<Markup u''>	<Markup u''>
escape(True, False)	AttributeError	<Markup u'True'>	AttributeError	<Markup u'True'>
escape('"', False)	<Markup u'"'>	<Markup u'"'>	<Markup u'"'>	<Markup u'"'>
escape('"', True)	<Markup u'&#34;'>	<Markup u'&#34;'>	<Markup u'&#34;'>	<Markup u'&#34;'>

[Jun Omae]

Proposed changes in jomae.git@t12536.

[Ryan J Ollos]

The changes look good.

[Jun Omae]

Thanks for the review. I'll push it after granting commit right to me for 1.2-stable branch.

[Ryan J Ollos]

Okay, granted access rights just now.

[Jun Omae]

Thanks. Committed in [14967] and merged to 1.2-stable and trunk in [14968:14969].

[Christian Boos]

Replying to Jun Omae:

Interested. However, escape(None, False) returns <Markup u''> on Genshi 0.7.

Note how this also currently differs from escape in Trac/jinja2 branch:

	Trac jinja2
escape(None, False)	Markup(u'None')
escape('', False)	Markup(u'')
escape(0, False)	Markup(u'0')
escape(1, False)	Markup(u'1')
escape(False, False)	Markup(u'False')
escape(True, False)	Markup(u'True')
escape('"', False)	Markup(u'"')
escape('"', True)	Markup(u'"')

I somehow find that more consistent, but if we decide to favor backward compatibility, I could adapt it.