OverflowError: __len__() should return 0 <= outcome < 2**31

[jneff]

How to Reproduce

While doing a GET operation on /ticket/66, Trac issued an internal error.

(please provide additional details here)

Python Traceback
Most recent call last:

    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/ticket/templates/ticket_change.html", line 61, in <Expression u'wiki_to_html(context, change.comment, escape_newlines=preserve_newlines)'>
      Code fragment:
      Line	
      56	${wiki_to_html(context, text, escape_newlines=preserve_newlines)}
      57	</div>
      58	<div py:otherwise="" py:choose="" class="comment searchable" xml:space="preserve">
      59	<py:when test="show_history">${wiki_to_html(context, change.comment_history[int(cversion)].comment,
      60	escape_newlines=preserve_newlines)}</py:when>
      61	<py:otherwise>${wiki_to_html(context, change.comment, escape_newlines=preserve_newlines)}</py:otherwise>
      62	</div>
      63	</py:choose>
      64	</html>
      Local variables:
      Name	Value
      __data__ 	[{'show_editor': False}, {'comment_version': 0, 'show_editor': False, ...
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/util/compat.py", line 83, in newfunc
      Code fragment:
      Line	
      78	try:
      79	from functools import partial
      80	except ImportError:
      81	def partial(func_, *args, **kwargs):
      82	def newfunc(*fargs, **fkwargs):
      83	return func_(*(args + fargs), **dict(kwargs, **fkwargs))
      84	newfunc.func = func_
      85	newfunc.args = args
      86	newfunc.keywords = kwargs
      87	try:
      88	newfunc.__name__ = func_.__name__
      Local variables:
      Name	Value
      args 	(<trac.env.Environment object at 0x2b8daf26b410>,)
      fargs 	(<Context <Resource u'ticket:66'>>, u'SR #3-3217719731: system performance ...
      fkwargs 	{'escape_newlines': True}
      func_ 	<function format_to_html at 0x2b8dae3ddb18>
      kwargs 	{}
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1498, in format_to_html
      Code fragment:
      Line	
      1493	def format_to_html(env, context, wikidom, escape_newlines=None):
      1494	if not wikidom:
      1495	return Markup()
      1496	if escape_newlines is None:
      1497	escape_newlines = context.get_hint('preserve_newlines', False)
      1498	return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
      1499	 
      1500	def format_to_oneliner(env, context, wikidom, shorten=None):
      1501	if not wikidom:
      1502	return Markup()
      1503	if shorten is None:
      Local variables:
      Name	Value
      context 	<Context <Resource u'ticket:66'>>
      env 	<trac.env.Environment object at 0x2b8daf26b410>
      escape_newlines 	True
      wikidom 	u'SR #3-3217719731: system performance case was opend with SUN and ...
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1453, in generate
      Code fragment:
      Line	
      1448	newlines in the wikidom will be preserved if `escape_newlines` is set.
      1449	"""
      1450	# FIXME: compatibility code only for now
      1451	out = StringIO()
      1452	Formatter(self.env, self.context).format(self.wikidom, out,
      1453	escape_newlines)
      1454	return Markup(out.getvalue())
      1455	 
      1456	 
      1457	class InlineHtmlFormatter(object):
      1458	"""Format parsed wiki text to inline elements HTML.
      Local variables:
      Name	Value
      escape_newlines 	True
      out 	<StringIO.StringIO instance at 0x2b8daf5260e0>
      self 	<trac.wiki.formatter.HtmlFormatter object at 0x2b8daf522210>
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1240, in format
      Code fragment:
      Line	
      1235	 
      1236	self.in_list_item = False
      1237	self.in_quote = False
      1238	# Throw a bunch of regexps on the problem
      1239	self.line = line
      1240	result = re.sub(self.wikiparser.rules, self.replace, line)
      1241	 
      1242	if not self.in_list_item:
      1243	self.close_list()
      1244	 
      1245	if not self.in_quote:
      Local variables:
      Name	Value
      block_start_match 	None
      escape_newlines 	True
      line 	u'SR #3-3217719731: system performance case was opend with SUN and ...
      out 	<StringIO.StringIO instance at 0x2b8daf5260e0>
      self 	<trac.wiki.formatter.Formatter object at 0x2b8daf5220d0>
      text 	[u'SR #3-3217719731: system performance case was opend with SUN and ...
    * File "/usr/lib64/python2.4/sre.py", line 142, in sub
      Code fragment:
      Line	
      137	"""Return the string obtained by replacing the leftmost
      138	non-overlapping occurrences of the pattern in string by the
      139	replacement repl. repl can be either a string or a callable;
      140	if a callable, it's passed the match object and must return
      141	a replacement string to be used."""
      142	return _compile(pattern, 0).sub(repl, string, count)
      143	 
      144	def subn(pattern, repl, string, count=0):
      145	"""Return a 2-tuple containing (new_string, number).
      146	new_string is the string obtained by replacing the leftmost
      147	non-overlapping occurrences of the pattern in the source
      Local variables:
      Name	Value
      count 	0
      pattern 	<_sre.SRE_Pattern object at 0x2b8daf2c3740>
      repl 	<bound method Formatter.replace of <trac.wiki.formatter.Formatter object ...
      string 	u'SR #3-3217719731: system performance case was opend with SUN and ...
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1158, in replace
      Code fragment:
      Line	
      1153	internal_handler = getattr(self, '_%s_formatter' % itype)
      1154	return internal_handler(match, fullmatch)
      1155	 
      1156	def replace(self, fullmatch):
      1157	"""Replace one match with its corresponding expansion"""
      1158	replacement = self.handle_match(fullmatch)
      1159	if replacement:
      1160	return _markup_to_unicode(replacement)
      1161	 
      1162	_normalize_re = re.compile(r'[\v\f]', re.UNICODE)
      1163	 
      Local variables:
      Name	Value
      fullmatch 	<_sre.SRE_Match object at 0x2b8daf569630>
      self 	<trac.wiki.formatter.Formatter object at 0x2b8daf5220d0>
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1151, in handle_match
      Code fragment:
      Line	
      1146	# Check for preceding escape character '!'
      1147	if match[0] == '!':
      1148	return escape(match[1:])
      1149	if itype in self.wikiparser.external_handlers:
      1150	external_handler = self.wikiparser.external_handlers[itype]
      1151	return external_handler(self, match, fullmatch)
      1152	else:
      1153	internal_handler = getattr(self, '_%s_formatter' % itype)
      1154	return internal_handler(match, fullmatch)
      1155	 
      1156	def replace(self, fullmatch):
      Local variables:
      Name	Value
      external_handler 	<function <lambda> at 0x2b8daf4dab18>
      fullmatch 	<_sre.SRE_Match object at 0x2b8daf569630>
      itype 	u'i3'
      match 	u'#3-3217719731'
      self 	<trac.wiki.formatter.Formatter object at 0x2b8daf5220d0>
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/ticket/api.py", line 436, in <lambda>
      Code fragment:
      Line	
      431	# matches #... but not &#... (HTML entity)
      432	r"!?(?<!&)#"
      433	# optional intertrac shorthand #T... + digits
      434	r"(?P<it_ticket>%s)%s" % (WikiParser.INTERTRAC_SCHEME,
      435	Ranges.RE_STR),
      436	lambda x, y, z: self._format_link(x, 'ticket', y[1:], y, z))
      437	 
      438	def _format_link(self, formatter, ns, target, label, fullmatch=None):
      439	intertrac = formatter.shorthand_intertrac_helper(ns, target, label,
      440	fullmatch)
      441	if intertrac:
      Local variables:
      Name	Value
      self 	<trac.ticket.api.TicketSystem object at 0x2b8daf1fa990>
      x 	<trac.wiki.formatter.Formatter object at 0x2b8daf5220d0>
      y 	u'#3-3217719731'
      z 	<_sre.SRE_Match object at 0x2b8daf569630>
    * File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/ticket/api.py", line 446, in _format_link
      Code fragment:
      Line	
      441	if intertrac:
      442	return intertrac
      443	try:
      444	link, params, fragment = formatter.split_link(target)
      445	r = Ranges(link)
      446	if len(r) == 1:
      447	num = r.a
      448	ticket = formatter.resource('ticket', num)
      449	from trac.ticket.model import Ticket
      450	if Ticket.id_is_valid(num) and \
      451	'TICKET_VIEW' in formatter.perm(ticket):
      Local variables:
      Name	Value
      formatter 	<trac.wiki.formatter.Formatter object at 0x2b8daf5220d0>
      fragment 	''
      fullmatch 	<_sre.SRE_Match object at 0x2b8daf569630>
      intertrac 	None
      label 	u'#3-3217719731'
      link 	u'3-3217719731'
      ns 	'ticket'
      params 	''
      r 	<trac.util.Ranges object at 0x2b8daf522110>
      self 	<trac.ticket.api.TicketSystem object at 0x2b8daf1fa990>
      target 	u'3-3217719731'

File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/ticket/templates/ticket_change.html", line 61, in <Expression u'wiki_to_html(context, change.comment, escape_newlines=preserve_newlines)'>
  <py:otherwise>${wiki_to_html(context, change.comment, escape_newlines=preserve_newlines)}</py:otherwise>
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/util/compat.py", line 83, in newfunc
  return func_(*(args + fargs), **dict(kwargs, **fkwargs))
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1498, in format_to_html
  return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1453, in generate
  escape_newlines)
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1240, in format
  result = re.sub(self.wikiparser.rules, self.replace, line)
File "/usr/lib64/python2.4/sre.py", line 142, in sub
  return _compile(pattern, 0).sub(repl, string, count)
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1158, in replace
  replacement = self.handle_match(fullmatch)
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/wiki/formatter.py", line 1151, in handle_match
  return external_handler(self, match, fullmatch)
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/ticket/api.py", line 436, in <lambda>
  lambda x, y, z: self._format_link(x, 'ticket', y[1:], y, z))
File "/usr/lib/python2.4/site-packages/Trac-0.12.3dev_r10609-py2.4.egg/trac/ticket/api.py", line 446, in _format_link
  if len(r) == 1:

Request parameters:

{'id': u'66'}

User agent: #USER_AGENT#

System Information

Trac	0.12.3dev-r10609
Babel	0.9.5
CustomFieldAdmin	0.2.5
Genshi	0.6
mod_wsgi	3.2 (WSGIProcessGroup WSGIApplicationGroup %{GLOBAL})
MySQL	server: "5.0.77", client: "5.0.77", thread-safe: 1
MySQLdb	1.2.3
Pygments	0.9
Python	2.4.3 (#1, Jun 11 2009, 14:09:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)]
pytz	2006p
setuptools	0.6c12
Subversion	1.4.2 (r22196)
jQuery	#JQUERY#

Enabled Plugins

AutocompleteUsers	0.4.1
BasicThemeEditorPlugin	0.1-r163
BlackMagicTicketTweaks	0.12r1
siteupload	0.11dev
TracCustomFieldAdmin	0.2.5
TracDateField	1.0.1
TracDynamicFields	1.2.1
TracIniAdminPanel	0.8beta-r534

[Remy Blank]

This rings a bell. I'm pretty sure we had something similar already… Yes, but with revision ranges, in #9955. The second patch in that comment would have avoided this issue, so I'll probably apply it.

OT: We really, really need #7145 :)

[jneff]

I'm new to this - should I apply the second patch in #9955 myself and then the ticket should open?

[anonymous]

I made the second change as noted in #9955: ]# diff init.py init.py.ORIG

801,802d800
<           #Result must fit an int
<           return  min(self.b - self.a + 1, sys.maxint)

the first change is already present

I recycled httpd and tried to open ticket but same error…

[Remy Blank]

Replying to jneff:

I'm new to this - should I apply the second patch in #9955 myself and then the ticket should open?

Not necessarily, I was just recording my initial thoughts. It's strange though that the patch doesn't seem to fix the issue for you. I will try to reproduce the issue.

[Remy Blank]

The patch does fix the issue for me, and was applied in [10670].

[Christian Boos]

I wonder if we shouldn't rather fail creating a Range, when given invalid input?

[Remy Blank]

Replying to cboos:

I wonder if we shouldn't rather fail creating a Range, when given invalid input?

Those are really pathological cases, so it doesn't really matter if len() returns the wrong answer (especially considering it already returns an "approximation", i.e. the range from lowest to highest value, and not the number of items in the range). The input is actually valid, and the fact that __len__() must return an int is a Python implementation detail, one that ​may even be lifted one day.

[Christian Boos]

Well, not valid as far as ticket ids are concerned. But right, Ranges is not about tickets (only). We could pass Ticket.id_is_valid as a "validator" callback for Ranges?

[Remy Blank]

Is it really worth the trouble? The input used here (#3-3217719731) is most probably a mistake (or a text that was unintentionally interpreted as a ticket range), so the goal is only not to throw a 500 and allow fixing the error.

And even though the upper limit of that range is technically invalid for tickets (why do we have that limit on ticket IDs anyway?), the resulting query works fine. Let's leave it at that.

[Christian Boos]

I got the impression we were fixing a symptom rather than the cause, and that being consistent with ticket id validation (see r6832) could have been better.

But you're right, it would have helped but only "by accident" ;-) If we had a larger ticket max value, we would still have hit the len must return an int limitation.

[jneff]

Should I just reinstall 12.3dev in order to get your latest updates. I made the changes to my init.py but still am getting the same error noted above.

[jneff]

diff __init__.py __init__.py.ORIG
799,802c799,801
<         #if self.a is not None and self.b is not None:
<         #    return self.b - self.a + 1
<         #else:
<       if self.a is None or self.b is None:
---
>         if self.a is not None and self.b is not None:
>             return self.b - self.a + 1
>         else:
804,805d802
<       #Result must fit an int
<       return  min(self.b - self.a + 1, sys.maxint)

No change to the error in ticket #66 :-(

[jneff]

Ahh I see what happened:

SR #3-3217719731: system performance case was opened with SUN and...

The admin pasted the ticket from SUN for documenting the work - Trac is seeing the # as a ticket # it can't reference. How can I edit the Details to wrap the '#' in a block of ? Is it merely editing the details in the DB via SQL? And the changes mentioend above should have marked the # at the max range of an int right?

[Remy Blank]

Replying to jneff:

Should I just reinstall 12.3dev in order to get your latest updates.

That would be easiest, yes. And don't forget to restart your web server. After that, you should be able to edit the page again.

Trac is seeing the # as a ticket # it can't reference.

Not quite. It's seeing a reference to a range of tickets, where one end of the range doesn't fit an int.

And the changes mentioend above should have marked the # at the max range of an int right?

No, the link remains a link to a range of tickets. If you don't want that, wrap the reference in backquotes:

`SR #3-3217719731`: system performance case was opened with SUN and...

[anonymous]

Replying to rblank:

Replying to jneff:

Should I just reinstall 12.3dev in order to get your latest updates.

That would be easiest, yes. And don't forget to restart your web server. After that, you should be able to edit the page again.

Trac is seeing the # as a ticket # it can't reference.

Not quite. It's seeing a reference to a range of tickets, where one end of the range doesn't fit an int.

And the changes mentioend above should have marked the # at the max range of an int right?

No, the link remains a link to a range of tickets. If you don't want that, wrap the reference in backquotes:

`SR #3-3217719731`: system performance case was opened with SUN and...

That makes much more sense about the Range of tickets - however I'm still confused about the change you posted above - the change to the util.py is supposed to take the second number and if it is outside the range of an INT then it should return the MAX of an int?

SO instead of 3-3217719731 it would return 3-999999 ( assuming 999999 is the max of an int on my system?)

Thanks!

[Remy Blank]

Replying to anonymous:

however I'm still confused about the change you posted above - the change to the util.py is supposed to take the second number and if it is outside the range of an INT then it should return the MAX of an int?

No, the second number is truncated to sys.maxint only in the __len__() method, which means that len(range) gives a wrong answer. However, that expression is only used in a check len(range) > 1, so it won't change the representation of the link.

[thomas.moschny@…]

The patch is wrong.

The (inclusive) upper bound for len() is 2**31-1 for Python 2.4 (all platforms), and sys.maxsize for Python 2.5 and later, and not sys.maxint. That holds especially on 64bit platforms (but also on 32bit platforms the test is off by one.)

So correct would be:

try:
  return min(self.b - self.a + 1, sys.maxsize)
except AttributeError:
  return min(self.b - self.a + 1, 2**31-1)

Also, would be cool if sb could apply the patch then on the 0.11-stable branch.

[Remy Blank]

Mmh… On my system, sys.maxsize == sys.maxint == 2**31-1 (Python 2.6, WinXP 32-bit), so this wouldn't make a difference (and there's no off-by-one error). Also, sys.maxsize seems to be defined since Python 2.6, not 2.5. Finally, returning sys.maxint on a 64-bit system and Python 2.5 works.

So the only difference would be that the limit should be 2**31-1 even for 64-bit systems with Python 2.4. We could define a maxlen value in trac.util.compat, and set it to 2**31-1 on Python 2.4, sys.maxsize if available or else sys.maxint.

[thomas.moschny@…]

So, as the patch basically is a workaround anyway, wouldn't it be ok to always use 2**31-1?

[Remy Blank]

Replying to thomas.moschny@…:

So, as the patch basically is a workaround anyway, wouldn't it be ok to always use 2**31-1?

Yes, that's probably the simplest solution.

[Remy Blank]

Fixed in [10679]. I won't merge this to trunk, as 2.4 isn't supported anymore.