Edgewall Software

Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#9377 closed defect (fixed)

avoid "ValueError: timestamp out of range for platform time_t" errors

Reported by: syjeong Owned by: Remy Blank
Priority: normal Milestone: 0.12
Component: general Version: 0.12dev
Severity: normal Keywords: timestamp
Cc: hoff.st@…
Release Notes:
API Changes:


How to Reproduce

While doing a GET operation on /ticketcalendar, Trac issued an internal error.

(please provide additional details here)

Request parameters:


User agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.2 (KHTML, like Gecko) Chrome/5.0.342.9 Safari/533.2 ChromePlus/

System Information

Trac 0.12dev-r9719
Babel 1.0dev-r546
Docutils 0.6
Genshi 0.7dev-r1134
mod_python 3.3.1
Pygments 1.2.2
pysqlite 2.4.1
Python 2.6.5 (r265:79063, Apr 16 2010, 13:28:26)
[GCC 4.4.3]
pytz 2010b
setuptools 0.6
SQLite 3.6.22
Subversion 1.6.6 (r40053)
jQuery 1.4.2

Enabled Plugins

TracGanttCalendarPlugin 0.1

Python Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12dev_r9719-py2.6.egg/trac/web/main.py", line 513, in _dispatch_request
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12dev_r9719-py2.6.egg/trac/web/main.py", line 235, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.6/dist-packages/TracGanttCalendarPlugin-0.1-py2.6.egg/ganttcalendar/ticketcalendar.py", line 122, in process_request
    due_time = to_datetime(due, utc)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12dev_r9719-py2.6.egg/trac/util/datefmt.py", line 52, in to_datetime
    return datetime.fromtimestamp(t, tzinfo or localtz)
ValueError: timestamp out of range for platform time_t

Attachments (0)

Change History (11)

comment:1 Changed 8 years ago by Christian Boos

Resolution: cantfix
Status: newclosed

comment:2 Changed 8 years ago by Remy Blank

… probably related to the switch to microsecond timestamps (see #6466).

Should we try to make to_datetime() more resilient against that? For example, if a timestamp value is greater than the limit for datetime, interpret it as a microsecond timestamp?

comment:3 Changed 8 years ago by Christian Boos

Cc: hoff.st@… added
Keywords: timestamp added
Resolution: cantfix
Status: closedreopened
Summary: ValueError: timestamp out of range for platform time_tavoid "ValueError: timestamp out of range for platform time_t" errors

I suppose that this can't hurt. At most if it's not enough to ensure full compatibility with the plugins, they will break in some other place… not worse than what we see here.

comment:4 Changed 8 years ago by Remy Blank

Suggested patch:

  • trac/util/datefmt.py

    diff --git a/trac/util/datefmt.py b/trac/util/datefmt.py
    a b  
    4949    elif isinstance(t, date):
    5050        return (tzinfo or localtz).localize(datetime(t.year, t.month, t.day))
    5151    elif isinstance(t, (int, long, float)):
     52        if t > _max_ts: # Accept microsecond timestamps for 0.11 compatibility
     53            t = t / 1000000.0
    5254        return datetime.fromtimestamp(t, tzinfo or localtz)
    5355    raise TypeError('expecting datetime, int, long, float, or None; got %s' %
    5456                    type(t))
    407409utcmax = datetime.max.replace(tzinfo=utc)
    408410_epoc = datetime(1970, 1, 1, tzinfo=utc)
    409411_zero = timedelta(0)
     412_max_ts = (1 << 31) - 1
    411414localtz = LocalTimezone()
  • trac/util/tests/datefmt.py

    diff --git a/trac/util/tests/datefmt.py b/trac/util/tests/datefmt.py
    a b  
    6262        self.assertEqual(datefmt.to_datetime(23), expected)
    6363        self.assertEqual(datefmt.to_datetime(23L), expected)
    6464        self.assertEqual(datefmt.to_datetime(23.0), expected)
     66    def test_to_datetime_microsecond_timestamps(self):
     67        expected = datetime.datetime.fromtimestamp(2345.678912,
     68                                                   datefmt.localtz)
     69        self.assertEqual(datefmt.to_datetime(2345678912), expected)
     70        self.assertEqual(datefmt.to_datetime(2345678912L), expected)
     71        self.assertEqual(datefmt.to_datetime(2345678912.0), expected)
    6673    def test_to_datetime_can_convert_dates(self):
    6774        expected = datetime.datetime(2009, 5, 2, tzinfo=datefmt.localtz)

Ok to apply?

comment:5 Changed 8 years ago by Remy Blank

Ideally, we should also log a warning, but I suppose this would require passing the environment to to_datetime(), so it's probably not possible.

comment:6 Changed 8 years ago by Remy Blank

Milestone: 0.12
Resolution: fixed
Status: reopenedclosed

Slightly improved patch applied in [9789] (also handles negative microsecond timestamps).

comment:7 Changed 8 years ago by Remy Blank

Owner: set to Remy Blank

comment:8 Changed 8 years ago by Christian Boos

As we're talking about microseconds… what about doing * 0.000001 instead? :-)

$ python -m timeit -r 10 '123 / 1000000.0'
10000000 loops, best of 10: 0.0715 usec per loop

$ python -m timeit -r 10 '123 * 0.000001'
10000000 loops, best of 10: 0.0237 usec per loop

comment:9 Changed 8 years ago by Remy Blank

I don't see any reasonable reason why this would be the case, but numbers don't lie… Done in [9790].

comment:10 Changed 8 years ago by hasienda <hoff.st@…>

Maybe I'm still too far away from topics like code performance optimization, but a quick newbie research suggests, that there are differences between python versions too (http://bugs.python.org/issue4128). Division optimization was subject to testing of alternative algos for Python some time ago (http://bugs.python.org/issue3451).

My test results of Christian's timeit commands are 0.254 vs. 0.105 usec respectively (Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) from Debian 5.0 package on personal workstation idling around with AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ backed by 2x 3 GB DDR2/800 RAM in dualchannel-config). Division is clearly more costly.

While the issue for Trac is certainly not about big numbers it very well might be about many numbers processed in succession. What I got from the reading of second resource mentioned here, division involves recursions (while multiplication might be more straight forward). Thanks, that you took performance serious here.

Interesting, being a mathematic freak I never thought about implementation of basic arithmetic operations in computer programs like this before. Life-long learning with Trac…

comment:11 Changed 8 years ago by Remy Blank

The truth is, this optimization is probably totally irrelevant here:

  • We don't convert that many timestamps into datetime objects anyway.
  • Even if we did, the division is only executed if a microsecond timestamp is erroneously passed to to_datetime() instead of from_utimestamp().

The reasons I wrote above that I don't see why division should be slower than multiplication are the following:

  • The operation is performed on floats, so AFAIK they should be executed by the FPU.
  • Even if the execution time (on the FPU) of the multiplication is lower than that of the division, the time of the actual operation is probably at least one order of magnitude lower than the overhead of the Python virtual machine (instruction decoding, dispatching, …). So there must be something else in the VM that causes the difference, but I can't find a good reason why this should be the case.

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain Remy Blank.
The resolution will be deleted.
to The owner will be changed from Remy Blank to the specified user.

Add Comment

E-mail address and name can be saved in the Preferences .
Note: See TracTickets for help on using tickets.