IWIkiSyntaxProvider extensions limited by Python regular expression limitation

[rkdecampo@…]

I have been using the RegexLinkInfo plugin successfully. However, if I configure 16 or more regexes, I get the error: AssertionError: sorry, but this version only supports 100 named groups. This is a limitation in python.

I did a little sleuthing and it appears that in parser.py, in _prepare_rules() (line 207), Trac concatenates all the regular expressions returned by IWikiSyntaxProvider implementations into one giant regular expression. Given the python limitation I noted that effectively limits the number of regular expressions available to IWikiSyntaxProvider implementations.

The log when I encountered the error:

2016-01-11 11:58:19,407 Trac[main] ERROR: Internal Server Error: <RequestWithSession "GET '/'">, referrer None
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/main.py", line 554, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/main.py", line 267, in dispatch
    iterable=chrome.use_chunked_encoding)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/chrome.py", line 1111, in render_template
    encoding='utf-8')
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 58, in encode
    for chunk in iterator:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 339, in __call__
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 826, in __call__
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 670, in __call__
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 771, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 586, in __call__
    for ev in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/chrome.py", line 1309, in _strip_accesskeys
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/chrome.py", line 1298, in _generate
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 565, in _flatten
    result = _eval_expr(data, ctxt, vars)
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 277, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/templates/wiki_view.html", line 58, in <Expression u'wiki_to_html(context, text)'>
    <div id="wikipage" class="trac-content" py:content="wiki_to_html(context, text)" />
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/formatter.py", line 1580, in format_to_html
    return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/formatter.py", line 1535, in generate
    escape_newlines)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/formatter.py", line 1317, in format
    result = re.sub(self.wikiparser.rules, self.replace, line)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/parser.py", line 178, in rules
    self._prepare_rules()
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/parser.py", line 207, in _prepare_rules
    rules = re.compile('(?:' + '|'.join(syntax) + ')', re.UNICODE)
  File "/usr/lib64/python2.6/re.py", line 190, in compile
    return _compile(pattern, flags)
  File "/usr/lib64/python2.6/re.py", line 243, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib64/python2.6/sre_compile.py", line 517, in compile
    "sorry, but this version only supports 100 named groups"
AssertionError: sorry, but this version only supports 100 named groups

[Ryan J Ollos]

Related to #10190.

[Jun Omae]

I guess the RegexLinkInfo plugin uses too much named captures in wiki syntax. The plugin should be fixed.

Also, where is the RegexLinkInfo plugin? ​Google search finds no results.

[rkdecampo@…]

Sorry, I should have said the RegexLinkPlugin, the url is ​RegexLinkPlugin.

If I am not mistaken (and I very well could be), in _prepare_links() in parser.py, Trac takes all the regexes returned by the IWIkiSyntaxProvider and transforms them using '(?P<i%d>%s)' % (i, regexp) causing each one to become a named group.

[Jun Omae]

The re instance has only 100 named groups at most. That plugin uses too many named groups like this if many regexs and urls are configured.

[regexlink]
regex1 = \b(?P<topdeskyymm>\d\d(?:0[1-9]|1[0-2])) (?P<topdesknr>\d{3})\b
url1 = http://topdesk/query=\g<topdeskyymm>%20\g<topdesknr>
regex2 = \bexample(?P<exampleid>\d+)\b
url2 = http://example.org/\g<exampleid>

The plugin should reduce uses of named groups.

regexlinkplugin/trunk/RegexLink/regex_link.py

@@ -58 +58 @@
     def get_wiki_syntax(self):
         """ IWikiSyntaxProvider method
         """
+        named_group_re = re.compile(r'\(\?P<[^>]+>')
         for regex_link in self.regex_links:
-            yield (regex_link.wiki_syntax_regex, (lambda rli:
-                lambda formatter, ns, match:
-                    formatter._make_ext_link(rli.replace_url(match), match.group(0)))(regex_link))
+            yield (named_group_re.sub('(?:', regex_link.wiki_syntax_regex),
+                   self._make_link(regex_link))
+
+    def _make_link(self, regex_link):
+        def fn(formatter, ns, match):
+            text = match.group(0)
+            match = re.match(regex_link.wiki_syntax_regex, text)
+            return formatter._make_ext_link(regex_link.replace_url(match), text)
+        return fn

[Jun Omae]

This is limitation of wiki parser by limitation of re module. Closing as a duplicate of #10190.

That plugin could reduce uses of named groups. See regexlink-reduce-uses-of-named-groups.diff​, which is verified with 36 regexs in regexlink section.

[Ryan J Ollos]

I'll test and apply the patch in ​th:#12641.