Edgewall Software
Modify

Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#11366 closed enhancement (fixed)

More flexible parse_date without Babel

Reported by: Jun Omae Owned by: Jun Omae
Priority: normal Milestone: 1.0.2
Component: i18n Version: 1.0-stable
Severity: normal Keywords:
Cc:
Release Notes:

Make parse_date more flexible even if without Babel

API Changes:

Description

If Babel isn't installed, parse_date parses input using time.strptime. But strptime is a function from libc and not flexible. Especially, if CJK environment, it cannot parse formatted text using format_datetime.

>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, 'ja_JP.UTF8')
'ja_JP.UTF8'
>>> from trac.util.datefmt import parse_date, localtz, format_datetime, get_datetime_format_hint
>>> from datetime import datetime
>>> t = datetime.now(localtz)
>>> text = format_datetime(t, locale=None)
>>> print text
2013年11月15日 12時40分14秒
>>> parse_date(text, locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError>>>

Even if non-CJK environment, the function isn't flexible.

>>> setlocale(LC_ALL, 'en_US.UTF8')
'en_US.UTF8'
>>> get_datetime_format_hint(locale=None)
u'MM/DD/YYYY hh:mm:ss a'
>>> parse_date('11/15, 2013 13:10', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15, 2013 13:10" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('11/15/2013 13:10', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15/2013 13:10" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('11/15/2013 13:10:42', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15/2013 13:10:42" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('11/15/2013 13:10:42 PM', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "11/15/2013 13:10:42 PM" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.
>>> parse_date('Nov/15/2013 1:10:42 PM', locale=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jun66j5/venv/trac/1.0.1/lib/python2.5/site-packages/trac/util/datefmt.py", line 466, in parse_date
    date=text, hint=hint), _('Invalid Date'))
trac.core.TracError: "Nov/15/2013 1:10:42 PM" is an invalid date, or the date format is not known. Try "MM/DD/YYYY" instead.

The parse_date function should be more flexible even without Babel.

Attachments (0)

Change History (14)

comment:1 Changed 5 years ago by Jun Omae

Milestone: next-stable-1.0.x1.0.2
Owner: set to Jun Omae
Status: newassigned

Proposed changes can be found in log:jomae.git:ticket11366. Verified with Python 2.5-2.7 on Linux and Python 2.6 on Windows XP for Japanese Edition.

comment:2 Changed 5 years ago by Jun Omae

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Committed in [12270] and merged into trunk in [12271].

comment:3 Changed 5 years ago by Ryan J Ollos

I found an error when executing on Ubuntu,

======================================================================
ERROR: test_parse_date_libc (__main__.ParseDateWithoutBabelTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11342/teo-rjollos.git/trac/util/tests/datefmt.py", line 339, in test_parse_date_libc
    self._setlocale('zh_CN')
  File "/home/user/Workspace/t11342/teo-rjollos.git/trac/util/tests/datefmt.py", line 279, in _setlocale
    setlocale(LC_ALL, self.locales.get(locale, locale))
  File "/home/user/Workspace/t11342/lib/python2.7/locale.py", line 547, in setlocale
    return _setlocale(category, locale)
Error: unsupported locale setting

----------------------------------------------------------------------
Ran 100 tests in 0.377s

FAILED (errors=1)

It looks like I don't have the zh_CN and fr locales installed. Will this be a common problem?

$ locale -a
C
C.UTF-8
de_AT.utf8
de_BE.utf8
de_CH.utf8
de_DE.utf8
de_LI.utf8
de_LU.utf8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
ja_JP.utf8
POSIX
Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:4 in reply to:  3 Changed 5 years ago by Jun Omae

Resolution: fixed
Status: closedreopened

It looks like I don't have the zh_CN and fr locales installed. Will this be a common problem?

Oh, that isn't good. I missed possibilities to configure locale data in debian and ubuntu. In CentOS, base system has many common locales and it rarely reconfigures locale data.

I'll change the unit tests to use only available locales.

Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:5 Changed 5 years ago by Jun Omae

The fix can be found in log:jomae.git:ticket11366.1.

comment:6 Changed 5 years ago by Ryan J Ollos

The changes work well in my environment where I don't have the zh_CN and fr locales. Thanks!

comment:7 Changed 5 years ago by Jun Omae

Resolution: fixed
Status: reopenedclosed

Thanks for your verifying! Applied in [12280-12281] and merged into trunk in [12282].

comment:8 Changed 5 years ago by Christian Boos

Resolution: fixed
Status: closedreopened

I had another issue with that test on Windows:

ERROR: test_parse_date_libc (trac.util.tests.datefmt.ParseDateWithoutBabelTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\1.0-stable\trac\util\tests\datefmt.py", line 288, in test_parse_date_libc
    datefmt.parse_date('Aug 28, 2010 1:45:56 PM', tz))
  File "c:\Trac\repos\1.0-stable\trac\util\datefmt.py", line 504, in parse_date
    raise TracError(msg, _('Invalid Date'))
TracError: <unprintable TracError object>

(with 2.5.4 or 2.6.5)

or even:

ERROR: test_get_console_locale (trac.admin.tests.console.TracadminTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\1.0-stable\trac\admin\tests\console.py", line 188, in _test_get_console_locale_with_babel
    self.assertEqual(Locale.default(), get_console_locale(None, None))
  File "c:\trac\dependencies\babel-0.9.x\babel\core.py", line 155, in default
    return cls(default_locale(category, aliases=aliases))
  File "c:\trac\dependencies\babel-0.9.x\babel\core.py", line 137, in __init__
    raise UnknownLocaleError(identifier)
UnknownLocaleError: unknown locale ''

======================================================================
ERROR: test_parse_date_libc (trac.util.tests.datefmt.ParseDateWithoutBabelTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\1.0-stable\trac\util\tests\datefmt.py", line 288, in test_parse_date_libc
    datefmt.parse_date('Aug 28, 2010 1:45:56 PM', tz))
  File "c:\Trac\repos\1.0-stable\trac\util\datefmt.py", line 504, in parse_date
    raise TracError(msg, _('Invalid Date'))
TracError: "Aug 28, 2010 1:45:56 PM" is an invalid date, or the date format is not known. Try "YYYY-MM-dd" or "YYYY-MM-DDThh:mm:ss\xb1hh:mm" instead.

(with 2.7.3)

I see that there's some Windows support in that test case, but maybe you tested with a mingw build of Python?

OTOH, I have:

>>> import locale
>>> locale.getlocale(locale.LC_TIME)
(None, None)
>>> locale.setlocale(locale.LC_ALL, 'English_United States')
'English_United States.1252'

comment:9 in reply to:  8 Changed 5 years ago by Jun Omae

I see that there's some Windows support in that test case, but maybe you tested with a mingw build of Python?

Hmmm, I've tested with MSC build of Python on Windows XP. I'll test on Windows 7 x64 later.

C:> python
Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

C:> python trac\util\tests\datefmt.py
...............................................................................
----------------------------------------------------------------------
Ran 79 tests in 0.907s

OK

comment:10 Changed 5 years ago by Jun Omae

All unit tests pass on Windows 7. However, reproduced after changing date formats in Control Panel.

It seems that MSVC locale system depends on date formats settings in user session.

Python 2.6.6 (r266:84297, Aug 24 2010, 18:13:38) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale, time
>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> time.strftime('%x %X')
'12/5/2013 1:39:47 AM'
...
(Changing "Short date" and "Long time" formats of "Region and Language"
 in "Control Panel")
...
>>> locale.setlocale(locale.LC_ALL, 'C')
'C'
>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> time.strftime('%x %X')
'2013-12-05 01:40:55'

I would like to skip ParseDateWithoutBabelTestCase because it's difficult to test it on Windows, log:jomae.git:ticket11366.2.

Last edited 5 years ago by Jun Omae (previous) (diff)

comment:11 in reply to:  10 ; Changed 5 years ago by Christian Boos

Replying to jomae:

All unit tests pass on Windows 7. However, reproduced after changing date formats in Control Panel.

Oh, right on the spot!

>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> time.strftime('%x %X')
'2013-12-04 8:05:47 PM'

I totally forgot I made such a change… but I did ;-)

I would like to skip ParseDateWithoutBabelTestCase because it's difficult to test it on Windows, log:jomae.git:ticket11366.2.

Or what about a skip if we detect such a change?

time.strftime('%x %X', time.gmtime(123456)) != '01/02/70 10:17:36'

See log:cboos.git:ticket11366win.

comment:12 in reply to:  11 Changed 5 years ago by Jun Omae

Or what about a skip if we detect such a change?

Good idea! I updated log:jomae.git:ticket11366win to fix some failures. I'll push it later. Thanks for the suggestions.

comment:13 Changed 5 years ago by Jun Omae

Resolution: fixed
Status: reopenedclosed

Applied in [12289] and merge into trunk in [12290].

comment:14 Changed 5 years ago by Christian Boos

Thanks!

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.