More flexible parse_date without Babel

[Jun Omae]

If Babel isn't installed, parse_date parses input using time.strptime. But strptime is a function from libc and not flexible. Especially, if CJK environment, it cannot parse formatted text using format_datetime.

>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, 'ja_JP.UTF8')
'ja_JP.UTF8'
>>> from trac.util.datefmt import parse_date, localtz, format_datetime, get_datetime_format_hint
>>> from datetime import datetime
>>> t = datetime.now(localtz)
>>> text = format_datetime(t, locale=None)
>>> print text
2013年11月15日 12時40分14秒
>>> parse_date(text, locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError>>>

Even if non-CJK environment, the function isn't flexible.

>>> setlocale(LC_ALL, 'en_US.UTF8')
'en_US.UTF8'
>>> get_datetime_format_hint(locale=None)
u'MM/DD/YYYY hh:mm:ss a'
>>> parse_date('11/15, 2013 13:10', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15, 2013 13:10" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('11/15/2013 13:10', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15/2013 13:10" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('11/15/2013 13:10:42', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15/2013 13:10:42" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('11/15/2013 13:10:42 PM', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15/2013 13:10:42 PM" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('Nov/15/2013 1:10:42 PM', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "Nov/15/2013 1:10:42 PM" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.

The parse_date function should be more flexible even without Babel.

[Jun Omae]

Proposed changes can be found in log:jomae.git:ticket11366. Verified with Python 2.5-2.7 on Linux and Python 2.6 on Windows XP for Japanese Edition.

[Jun Omae]

Committed in [12270] and merged into trunk in [12271].

[Ryan J Ollos]

I found an error when executing on Ubuntu,

======================================================================
ERROR: test_parse_date_libc (__main__.ParseDateWithoutBabelTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11342/teo-rjollos.git/trac/util/tests/datefmt.py", line 339, in test_parse_date_libc
    self._setlocale('zh_CN')
  File "/home/user/Workspace/t11342/teo-rjollos.git/trac/util/tests/datefmt.py", line 279, in _setlocale
    setlocale(LC_ALL, self.locales.get(locale, locale))
  File "/home/user/Workspace/t11342/lib/python2.7/locale.py", line 547, in setlocale
    return _setlocale(category, locale)
Error: unsupported locale setting

----------------------------------------------------------------------
Ran 100 tests in 0.377s

FAILED (errors=1)

It looks like I don't have the zh_CN and fr locales installed. Will this be a common problem?

$ locale -a
C
C.UTF-8
de_AT.utf8
de_BE.utf8
de_CH.utf8
de_DE.utf8
de_LI.utf8
de_LU.utf8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
ja_JP.utf8
POSIX

[Jun Omae]

It looks like I don't have the zh_CN and fr locales installed. Will this be a common problem?

Oh, that isn't good. I missed possibilities to configure locale data in debian and ubuntu. In CentOS, base system has many common locales and it rarely reconfigures locale data.

I'll change the unit tests to use only available locales.

[Jun Omae]

The fix can be found in log:jomae.git:ticket11366.1.

[Ryan J Ollos]

The changes work well in my environment where I don't have the zh_CN and fr locales. Thanks!

[Jun Omae]

Thanks for your verifying! Applied in [12280-12281] and merged into trunk in [12282].

[Christian Boos]

I had another issue with that test on Windows:

ERROR: test_parse_date_libc (trac.util.tests.datefmt.ParseDateWithoutBabelTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\1.0-stable\trac\util\tests\datefmt.py", line 288, in test_parse_date_libc
    datefmt.parse_date('Aug 28, 2010 1:45:56 PM', tz))
  File "c:\Trac\repos\1.0-stable\trac\util\datefmt.py", line 504, in parse_date
    raise TracError(msg, _('Invalid Date'))
TracError: <unprintable TracError object>

(with 2.5.4 or 2.6.5)

or even:

ERROR: test_get_console_locale (trac.admin.tests.console.TracadminTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\1.0-stable\trac\admin\tests\console.py", line 188, in _test_get_console_locale_with_babel
    self.assertEqual(Locale.default(), get_console_locale(None, None))
  File "c:\trac\dependencies\babel-0.9.x\babel\core.py", line 155, in default
    return cls(default_locale(category, aliases=aliases))
  File "c:\trac\dependencies\babel-0.9.x\babel\core.py", line 137, in __init__
    raise UnknownLocaleError(identifier)
UnknownLocaleError: unknown locale ''

======================================================================
ERROR: test_parse_date_libc (trac.util.tests.datefmt.ParseDateWithoutBabelTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\1.0-stable\trac\util\tests\datefmt.py", line 288, in test_parse_date_libc
    datefmt.parse_date('Aug 28, 2010 1:45:56 PM', tz))
  File "c:\Trac\repos\1.0-stable\trac\util\datefmt.py", line 504, in parse_date
    raise TracError(msg, _('Invalid Date'))
TracError: "Aug 28, 2010 1:45:56 PM" is an invalid date, or the date format is not known. Try "YYYY-MM-dd" or "YYYY-MM-DDThh:mm:ss\xb1hh:mm" instead.

(with 2.7.3)

I see that there's some Windows support in that test case, but maybe you tested with a mingw build of Python?

OTOH, I have:

>>> import locale
>>> locale.getlocale(locale.LC_TIME)
(None, None)
>>> locale.setlocale(locale.LC_ALL, 'English_United States')
'English_United States.1252'

[Jun Omae]

I see that there's some Windows support in that test case, but maybe you tested with a mingw build of Python?

Hmmm, I've tested with MSC build of Python on Windows XP. I'll test on Windows 7 x64 later.

C:> python
Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

C:> python trac\util\tests\datefmt.py
...............................................................................
----------------------------------------------------------------------
Ran 79 tests in 0.907s

OK

[Jun Omae]

All unit tests pass on Windows 7. However, reproduced after changing date formats in Control Panel.

It seems that MSVC locale system depends on date formats settings in user session.

Python 2.6.6 (r266:84297, Aug 24 2010, 18:13:38) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale, time
>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> time.strftime('%x %X')
'12/5/2013 1:39:47 AM'
...
(Changing "Short date" and "Long time" formats of "Region and Language"
 in "Control Panel")
...
>>> locale.setlocale(locale.LC_ALL, 'C')
'C'
>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> time.strftime('%x %X')
'2013-12-05 01:40:55'

I would like to skip ParseDateWithoutBabelTestCase because it's difficult to test it on Windows, log:jomae.git:ticket11366.2.

[Christian Boos]

Replying to jomae:

All unit tests pass on Windows 7. However, reproduced after changing date formats in Control Panel.

Oh, right on the spot!

>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> time.strftime('%x %X')
'2013-12-04 8:05:47 PM'

I totally forgot I made such a change… but I did ;-)

I would like to skip ParseDateWithoutBabelTestCase because it's difficult to test it on Windows, log:jomae.git:ticket11366.2.

Or what about a skip if we detect such a change?

time.strftime('%x %X', time.gmtime(123456)) != '01/02/70 10:17:36'

See log:cboos.git:ticket11366win.

[Jun Omae]

Or what about a skip if we detect such a change?

Good idea! I updated log:jomae.git:ticket11366win to fix some failures. I'll push it later. Thanks for the suggestions.

[Jun Omae]

Applied in [12289] and merge into trunk in [12290].

[Christian Boos]

Thanks!