Three datefmt.to_datetime() tests fail when TZ=Europe/London

[Alex Willmerer <al.willmer@…>]

Three unit tests in source:/trunk/trac/util/tests/datefmt.py​ fail in latest trunk (r12624 on Ubuntu 13.10/Python 2.7.5, set to the UK timezone. trac.util.to_datetime(<n>) returns a datetime that is 1 hour ahead of that returned by datetime.fromtimestamp(<n>), where n is a POSIX timestamp.

In January 1970 ​the UK was 1 hour ahead of UTC. This was changed in October 1971, since then the UK has been on GMT (UTC+0) during winter months.

(venv)$ PYTHONPATH=. python trac/util/tests/datefmt.py
................F.FF...................................................................................
======================================================================
FAIL: test_to_datetime (__main__.DateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "trac/util/tests/datefmt.py", line 716, in test_to_datetime
    self.assertEqual(datefmt.to_datetime(23), expected)
AssertionError: datetime.datetime(1970, 1, 1, 2, 0, 23, tzinfo=<LocalTimezone "GMT" 0:00:00>) != datetime.datetime(1970, 1, 1, 1, 0, 23, tzinfo=<LocalTimezone "GMT" 0:00:00>)

======================================================================
FAIL: test_to_datetime_microsecond_negative_timestamps (__main__.DateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "trac/util/tests/datefmt.py", line 734, in test_to_datetime_microsecond_negative_timestamps
    self.assertEqual(datefmt.to_datetime(-2345678912), expected)
AssertionError: datetime.datetime(1970, 1, 1, 1, 20, 54, 321088, tzinfo=<LocalTimezone "GMT" 0:00:00>) != datetime.datetime(1970, 1, 1, 0, 20, 54, 321088, tzinfo=<LocalTimezone "GMT" 0:00:00>)

======================================================================
FAIL: test_to_datetime_microsecond_timestamps (__main__.DateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "trac/util/tests/datefmt.py", line 723, in test_to_datetime_microsecond_timestamps
    self.assertEqual(datefmt.to_datetime(2345678912), expected)
AssertionError: datetime.datetime(1970, 1, 1, 2, 39, 5, 678912, tzinfo=<LocalTimezone "GMT" 0:00:00>) != datetime.datetime(1970, 1, 1, 1, 39, 5, 678912, tzinfo=<LocalTimezone "GMT" 0:00:00>)

----------------------------------------------------------------------
Ran 103 tests in 0.355s

FAILED (failures=3)

Other timezones are not affected (checked by Jun Omae)

$ git status -sb
## 0.12-stable
$ TZ=Europe/London ~/venv/py24/bin/python setup.py test -s trac.util.tests.datefmt.suite ...
FAILED (failures=3)
$ TZ=GMT ~/venv/py24/bin/python setup.py test -s trac.util.tests.datefmt.suite ...
OK
$ TZ=Europe/Berlin ~/venv/py24/bin/python setup.py test -s trac.util.tests.datefmt.suite ...
OK

This was initially discussed in ​trac-dev:wzKUgAe38PU/hJEP8mwfRc4J.

[Jun Omae]

Proposed changes in jomae.git@t11563.

This issue occurs when the date/time has timezone which is different from time.timezone and time.altzone.

time	std	dst
now	GMT	BST (GMT+0100)
1968 - 1971	BST (GMT+0100)	n/a

[Jun Omae]

Improved unit tests in [500f3a78/jomae.git].

[Alex Willmer <al.willmer@…>]

All trac.util tests pass for me

(venv)~/src/d4/jomae$ git branch
* t11563
  trunk
(venv)~/src/d4/jomae$ python setup.py test -s trac.util.tests.suite
running test
running egg_info
writing requirements to Trac.egg-info/requires.txt
writing Trac.egg-info/PKG-INFO
writing top-level names to Trac.egg-info/top_level.txt
writing dependency_links to Trac.egg-info/dependency_links.txt
writing entry points to Trac.egg-info/entry_points.txt
reading manifest file 'Trac.egg-info/SOURCES.txt'
writing manifest file 'Trac.egg-info/SOURCES.txt'
running build_ext
test_existing (trac.util.tests.AtomicFileTestCase) ... ok
test_existing_open_for_reading (trac.util.tests.AtomicFileTestCase) ... ok
...
test_tracerror_with_tgettext (trac.util.tests.html.ToFragmentTestCase) ... ok
test_unicode (trac.util.tests.html.ToFragmentTestCase) ... ok

----------------------------------------------------------------------
Ran 169 tests in 0.587s

OK

[Jun Omae]

I noticed that another issue on DST boundaries and timezone changes, e.g. ​2011-03-27 02:00 and 2010-10-31 03:00 in Moscow. It's different between pytz's timezone and localtz in trac.util.datefmt.

>>> import os
>>> from datetime import datetime
>>> from trac.util.datefmt import to_datetime, get_timezone, localtz
>>> os.environ['TZ']
'Europe/Moscow'
>>> localtz
<LocalTimezone "MSK" 4:00:00 "MSK" 4:00:00>
>>> tz = get_timezone('Europe/Moscow')
>>>
>>> t = datetime(2011, 3, 27, 2)
>>> to_datetime(t, tz)
datetime.datetime(2011, 3, 27, 3, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+4:00:00 STD>)
>>> to_datetime(t, localtz)
datetime.datetime(2011, 3, 27, 1, 0, tzinfo=<LocalTimezone "MSK" 4:00:00>)
>>>
>>> t = datetime(2010, 10, 31, 3)
>>> to_datetime(t, tz)
datetime.datetime(2010, 10, 31, 3, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+3:00:00 STD>)
>>> to_datetime(t, localtz)
datetime.datetime(2010, 10, 31, 2, 0, tzinfo=<LocalTimezone "MSK" 4:00:00>)

I updated the branch jomae.git@t11563 and added more unit tests.

>>> t = datetime(2011, 3, 27, 2)
>>> to_datetime(t, tz)
datetime.datetime(2011, 3, 27, 3, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+4:00:00 STD>)
>>> to_datetime(t, localtz)
datetime.datetime(2011, 3, 27, 3, 0, tzinfo=<LocalTimezone "MSK" 4:00:00>)
>>>
>>> t = datetime(2010, 10, 31, 3)
>>> to_datetime(t, tz)
datetime.datetime(2010, 10, 31, 3, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+3:00:00 STD>)
>>> to_datetime(t, localtz)
datetime.datetime(2010, 10, 31, 3, 0, tzinfo=<LocalTimezone "UTC+03:00" 3:00:00>)

In the branch, all unit tests pass with Python 2.4-2.7 on i386 and x86-64.

I'll push it tonight.

[Jun Omae]

Committed in [12633] and merged in [12634-12635].

[Ryan J Ollos]

Proposed refactoring to address confusing line and unpythonic is True:

trac/util/datefmt.py

@@ -1022 +1022 @@
             if not std_correct:
                 raise ValueError('Non existent time "%s"' % dt)
         tt = None
-        if std_correct is dst_correct is True:
+        if std_correct and dst_correct:
             tt = local_tt[bool(is_dst)]
-        elif std_correct is True:
+        elif std_correct:
             tt = local_tt[0]
-        elif dst_correct is True:
+        elif dst_correct:
             tt = local_tt[1]
         if tt:
             utc_ts = to_timestamp(datetime(tzinfo=utc, *tt[:6]))

Unit tests pass for me after the change.

[anonymous]

comment:7 changes committed in [14732].