Wrong arithmetic datetime with LocalTimezone if across a DST boundary

[Jun Omae]

If arithmetic datetime with LocalTimezone and timedelta across a DST boundary, the result have the lack of consistency.

$ TZ=Europe/Paris ~/venv/trac/0.12.4/bin/python
Python 2.4.3 (#1, Jun 18 2012, 08:55:31)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime, timedelta
>>> from trac.util.datefmt import to_datetime, localtz, to_timestamp, utc
>>> localtz
<LocalTimezone "CET" 1:00:00 "CEST" 2:00:00>
>>> t1 = datetime(2012, 3, 25, 1, 15, tzinfo=localtz)
>>> t2 = t1 + timedelta(hours=1)
>>> t1.isoformat()
'2012-03-25T01:15:00+01:00'
>>> t2.isoformat()
'2012-03-25T02:15:00+02:00'   # Expects 2012-03-25T02:15:00+01:00
                              #      or 2012-03-25T03:15:00+02:00
>>> t1 == t2
False
>>> t1 - t1.utcoffset() == t2 - t2.utcoffset()
True                          # Expects `False`
>>> to_timestamp(t2) - to_timestamp(t1)
0                             # Expects 3600

If arithmetic datetime with pytz across a DST boundary, the value of datetime.utcoffset() remains unchanged. tzinfo.normalize() method changes the utcoffset() value.

>>> from trac.util.datefmt import timezone
>>> tz = timezone('Europe/Paris')
>>> t1 = tz.normalize(tz.localize(datetime(2012, 3, 25, 1, 15)))
>>> t1
datetime.datetime(2012, 3, 25, 1, 15, tzinfo=<DstTzInfo 'Europe/Paris' CET+1:00:00 STD>)
>>> t2 = t1 + timedelta(hours=1)
>>> t1.isoformat()
'2012-03-25T01:15:00+01:00'
>>> t2.isoformat()
'2012-03-25T02:15:00+01:00'
>>> t3 = t2 + timedelta(hours=1)
>>> t3.isoformat()
'2012-03-25T03:15:00+01:00'
>>> tz.normalize(t2).isoformat()
'2012-03-25T03:15:00+02:00'

[Christian Boos]

On Windows, the first part works the way you expected:

>>> from datetime import datetime, timedelta
>>> from trac.util.datefmt import to_datetime, localtz, to_timestamp, utc
>>> localtz
<LocalTimezone "Romance Standard Time" 1:00:00 "Romance Daylight Time" 2:00:00>
>>> t1 = datetime(2012, 3, 25, 1, 15, tzinfo=localtz)
>>> t2 = t1 + timedelta(hours=1)
>>> t1.isoformat()
'2012-03-25T01:15:00+01:00'
>>> t2.isoformat()
'2012-03-25T02:15:00+01:00'
>>> t1 == t2
False
>>> t1 - t1.utcoffset() == t2 - t2.utcoffset()
False
>>> to_timestamp(t2) - to_timestamp(t1)
3600

However:

>>> t3 = t2 + timedelta(hours=1)
>>> t3.isoformat()
'2012-03-25T03:15:00+02:00'
>>> localtz.normalize(t2).isoformat()
'2012-03-25T02:15:00+01:00'
>>> localtz.normalize(t3).isoformat()
'2012-03-25T03:15:00+02:00'

So it also doesn't fully match what we get via pytz.

And it's therefore "buggy" as well, I think the criterion for correctness should definitely be localtz.normalize(t1 + timedelta(hours=1)).isoformat() → '2012-03-25T03:15:00+02:00'.

[Genie]

​In paris, Daylight Saving Time started on 25th March 2012 at 2.00am.

in case t1 == '2012-03-25T01:15:00+01:00'

t1 + 1 hours == '2012-03-25T03:15:00+02:00'

[Jun Omae]

I worked in repos:jomae.git:ticket10912/localtz. After the changes, LocalTimezone behaves like pytz.tzinfo. I confirmed on CentOS 5.8 and Windows XP sp3 with timezone in Paris. It needs more testing….

[Christian Boos]

[2ec1cf3e/jomae.git] could perhaps benefit from a LocalTimezone._localtime(dt) → localized time or None helper method, but other than that looks great! I'll test and let you know.

[Christian Boos]

I just tested [63b4287d/jomae.git], redoing the steps from comment:1:

>>> localtz.normalize(t1 + timedelta(hours=0)).isoformat()
'2012-03-25T01:15:00+01:00'
>>> localtz.normalize(t1 + timedelta(hours=1)).isoformat()
'2012-03-25T03:15:00+02:00'
>>> localtz.normalize(t1 + timedelta(hours=2)).isoformat()
'2012-03-25T03:15:00+02:00'
>>> localtz.normalize(t1 + timedelta(hours=3)).isoformat()
'2012-03-25T04:15:00+02:00'

… so while the result for +1 hour is now fine, there's no differences between +1 and +2 in the above? I don't think that's correct.

And indeed, pytz gives:

>>> tz.normalize(t1 + timedelta(hours=0)).isoformat()
'2012-03-25T01:15:00+01:00'
>>> tz.normalize(t1 + timedelta(hours=1)).isoformat()
'2012-03-25T03:15:00+02:00'
>>> tz.normalize(t1 + timedelta(hours=2)).isoformat()
'2012-03-25T04:15:00+02:00'
>>> tz.normalize(t1 + timedelta(hours=3)).isoformat()
'2012-03-25T05:15:00+02:00'

But other than that, the new code looks nice, so I think we're on the right track ;-)

[Jun Omae]

Thanks for the comments.

Replying to cboos:

I just tested [63b4287d/jomae.git], redoing the steps from comment:1:

>>> localtz.normalize(t1 + timedelta(hours=0)).isoformat()
'2012-03-25T01:15:00+01:00'
...
>>> localtz.normalize(t1 + timedelta(hours=2)).isoformat()
'2012-03-25T03:15:00+02:00'
...

… so while the result for +1 hour is now fine, there's no differences between +1 and +2 in the above? I don't think that's correct.

You're right. Ah, it seems difficult to fix without being localized, however, I'll try.

Also, time.mktime() is inconsistent with ambiguous times. The problem is caused by glibc. See ​time/mktime.c in glibc.git. I have yet to investigate the other C runtime.

$ PYTHONPATH=$PWD TZ=Europe/Paris ~/venv/py24/bin/python
Python 2.4.3 (#1, Jun 18 2012, 08:55:31)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(time.mktime((2001, 1, 1, 0, 0, 0, 0, 0, 0)))
(2001, 1, 1, 0, 0, 0, 0, 1, 0)
>>> time.localtime(time.mktime((2011, 10, 30, 2, 15, 0, 0, 0, -1)))
(2011, 10, 30, 2, 15, 0, 6, 303, 0)
>>> time.localtime(time.mktime((2001, 1, 1, 0, 0, 0, 0, 0, 1)))
(2000, 12, 31, 23, 0, 0, 6, 366, 0)
>>> time.localtime(time.mktime((2011, 10, 30, 2, 15, 0, 0, 0, -1)))
(2011, 10, 30, 2, 15, 0, 6, 303, 1)
>>>

[Jun Omae]

Replying to jomae:

… so while the result for +1 hour is now fine, there's no differences between +1 and +2 in the above? I don't think that's correct.

You're right. Ah, it seems difficult to fix without being localized, however, I'll try.

I worked in [04eaf8aa/jomae.git]. After the changes, if the datetime object is not localized, LocalTimezone.normalize do nothing.

>>> t1 = datetime(2012, 3, 25, 1, 15, tzinfo=localtz)
>>> localtz.normalize(t1 + timedelta(hours=0)).isoformat()
'2012-03-25T01:15:00+01:00'
>>> localtz.normalize(t1 + timedelta(hours=1)).isoformat()
'2012-03-25T02:15:00+01:00'
>>> localtz.normalize(t1 + timedelta(hours=2)).isoformat()
'2012-03-25T03:15:00+02:00'
>>> localtz.normalize(t1 + timedelta(hours=3)).isoformat()
'2012-03-25T04:15:00+02:00'

Use to_datetime (call internally tz.localize() and tz.normalize()) to retrieve normalized datetime objects, in the same manner as pytz.tzinfo.

>>> t1 = to_datetime(datetime(2012, 3, 25, 1, 15), localtz)
>>> to_datetime(t1 + timedelta(hours=0), localtz).isoformat()
'2012-03-25T01:15:00+01:00'
>>> to_datetime(t1 + timedelta(hours=1), localtz).isoformat()
'2012-03-25T03:15:00+02:00'
>>> to_datetime(t1 + timedelta(hours=2), localtz).isoformat()
'2012-03-25T04:15:00+02:00'
>>> to_datetime(t1 + timedelta(hours=3), localtz).isoformat()
'2012-03-25T05:15:00+02:00'

Also, time.mktime() is inconsistent with ambiguous times. The problem is caused by glibc. See ​time/mktime.c in glibc.git. I have yet to investigate the other C runtime. ...

That issue is fixed in [c2ac69c7/jomae.git]. I confirmed with Python 2.5-2.7 on Linux 64-bits, Python 2.7 on FreeBSD 64-bits and Python 2.4-2.7 on Windows XP (32-bits).

[Jun Omae]

The patch is applied in [11419-11421].