Edgewall Software
Modify

Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#12318 closed defect (duplicate)

IWIkiSyntaxProvider extensions limited by Python regular expression limitation

Reported by: rkdecampo@… Owned by:
Priority: normal Milestone:
Component: wiki system Version: 1.0.9
Severity: normal Keywords:
Cc: rkdecampo@… Branch:
Release Notes:
API Changes:
Internal Changes:

Description

I have been using the RegexLinkInfo plugin successfully. However, if I configure 16 or more regexes, I get the error: AssertionError: sorry, but this version only supports 100 named groups. This is a limitation in python.

I did a little sleuthing and it appears that in parser.py, in _prepare_rules() (line 207), Trac concatenates all the regular expressions returned by IWikiSyntaxProvider implementations into one giant regular expression. Given the python limitation I noted that effectively limits the number of regular expressions available to IWikiSyntaxProvider implementations.

The log when I encountered the error:

2016-01-11 11:58:19,407 Trac[main] ERROR: Internal Server Error: <RequestWithSession "GET '/'">, referrer None
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/main.py", line 554, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/main.py", line 267, in dispatch
    iterable=chrome.use_chunked_encoding)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/chrome.py", line 1111, in render_template
    encoding='utf-8')
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 58, in encode
    for chunk in iterator:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 339, in __call__
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 826, in __call__
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 670, in __call__
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 771, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/output.py", line 586, in __call__
    for ev in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/chrome.py", line 1309, in _strip_accesskeys
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/web/chrome.py", line 1298, in _generate
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 565, in _flatten
    result = _eval_expr(data, ctxt, vars)
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/base.py", line 277, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "/usr/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/templates/wiki_view.html", line 58, in <Expression u'wiki_to_html(context, text)'>
    <div id="wikipage" class="trac-content" py:content="wiki_to_html(context, text)" />
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/formatter.py", line 1580, in format_to_html
    return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/formatter.py", line 1535, in generate
    escape_newlines)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/formatter.py", line 1317, in format
    result = re.sub(self.wikiparser.rules, self.replace, line)
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/parser.py", line 178, in rules
    self._prepare_rules()
  File "/usr/lib/python2.6/site-packages/Trac-1.0.9-py2.6.egg/trac/wiki/parser.py", line 207, in _prepare_rules
    rules = re.compile('(?:' + '|'.join(syntax) + ')', re.UNICODE)
  File "/usr/lib64/python2.6/re.py", line 190, in compile
    return _compile(pattern, flags)
  File "/usr/lib64/python2.6/re.py", line 243, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib64/python2.6/sre_compile.py", line 517, in compile
    "sorry, but this version only supports 100 named groups"
AssertionError: sorry, but this version only supports 100 named groups

Attachments (1)

regexlink-reduce-uses-of-named-groups.diff (2.9 KB ) - added by Jun Omae 5 years ago.

Download all attachments as: .zip

Change History (8)

comment:1 by Ryan J Ollos, 5 years ago

Related to #10190.

comment:2 by Jun Omae, 5 years ago

I guess the RegexLinkInfo plugin uses too much named captures in wiki syntax. The plugin should be fixed.

Also, where is the RegexLinkInfo plugin? Google search finds no results.

comment:3 by rkdecampo@…, 5 years ago

Sorry, I should have said the RegexLinkPlugin, the url is RegexLinkPlugin.

If I am not mistaken (and I very well could be), in _prepare_links() in parser.py, Trac takes all the regexes returned by the IWIkiSyntaxProvider and transforms them using '(?P<i%d>%s)' % (i, regexp) causing each one to become a named group.

Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:4 by Raymond DeCampo <rkdecampo@…>, 5 years ago

Cc: rkdecampo@… added

comment:5 by Jun Omae, 5 years ago

The re instance has only 100 named groups at most. That plugin uses too many named groups like this if many regexs and urls are configured.

[regexlink]
regex1 = \b(?P<topdeskyymm>\d\d(?:0[1-9]|1[0-2])) (?P<topdesknr>\d{3})\b
url1 = http://topdesk/query=\g<topdeskyymm>%20\g<topdesknr>
regex2 = \bexample(?P<exampleid>\d+)\b
url2 = http://example.org/\g<exampleid>

The plugin should reduce uses of named groups.

  • regexlinkplugin/trunk/RegexLink/regex_link.py

    diff --git a/regexlinkplugin/trunk/RegexLink/regex_link.py b/regexlinkplugin/trunk/RegexLink/regex_link.py
    index 237fc9a..2b67f15 100644
    a b class RegexLinkSyntaxProvider(Component):  
    5858    def get_wiki_syntax(self):
    5959        """ IWikiSyntaxProvider method
    6060        """
     61        named_group_re = re.compile(r'\(\?P<[^>]+>')
    6162        for regex_link in self.regex_links:
    62             yield (regex_link.wiki_syntax_regex, (lambda rli:
    63                 lambda formatter, ns, match:
    64                     formatter._make_ext_link(rli.replace_url(match), match.group(0)))(regex_link))
     63            yield (named_group_re.sub('(?:', regex_link.wiki_syntax_regex),
     64                   self._make_link(regex_link))
     65
     66    def _make_link(self, regex_link):
     67        def fn(formatter, ns, match):
     68            text = match.group(0)
     69            match = re.match(regex_link.wiki_syntax_regex, text)
     70            return formatter._make_ext_link(regex_link.replace_url(match), text)
     71        return fn

comment:6 by Jun Omae, 5 years ago

Resolution: duplicate
Status: newclosed

This is limitation of wiki parser by limitation of re module. Closing as a duplicate of #10190.

That plugin could reduce uses of named groups. See regexlink-reduce-uses-of-named-groups.diff, which is verified with 36 regexs in regexlink section.

comment:7 by Ryan J Ollos, 5 years ago

I'll test and apply the patch in th:#12641.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The ticket will remain with no owner.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from (none) to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.