Catalog compilation and test case failures with Babel >= 1.0

[Ryan J Ollos]

In the last day or so we've rocketed forward to Babel 1.2, now available on ​PyPI. I only noticed this just now due to some test case failures.

Compiling the catalogs results in:

(t11257)user@ubuntu:~/Workspace/t11257/teo-rjollos.git$ python setup.py compile_catalog -f
running compile_catalog
compiling catalog 'trac/locale/he/LC_MESSAGES/messages.po' to 'trac/locale/he/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/gl/LC_MESSAGES/messages.po' to 'trac/locale/gl/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/fa/LC_MESSAGES/messages.po' to 'trac/locale/fa/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/nb/LC_MESSAGES/messages.po' to 'trac/locale/nb/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/en_GB/LC_MESSAGES/messages.po' to 'trac/locale/en_GB/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/fi/LC_MESSAGES/messages.po' to 'trac/locale/fi/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/it/LC_MESSAGES/messages.po' to 'trac/locale/it/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/es_MX/LC_MESSAGES/messages.po' to 'trac/locale/es_MX/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/ja/LC_MESSAGES/messages.po' to 'trac/locale/ja/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/vi/LC_MESSAGES/messages.po' to 'trac/locale/vi/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/en_US/LC_MESSAGES/messages.po' to 'trac/locale/en_US/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/et/LC_MESSAGES/messages.po' to 'trac/locale/et/LC_MESSAGES/messages.mo'
error: trac/locale/ca/LC_MESSAGES/messages.po:668: unknown named placeholder u'resource'
error: trac/locale/ca/LC_MESSAGES/messages.po:704: unknown named placeholder u'subject'
error: trac/locale/ca/LC_MESSAGES/messages.po:709: unknown named placeholder u'subject'
error: trac/locale/ca/LC_MESSAGES/messages.po:736: unknown named placeholder u'page'
error: trac/locale/ca/LC_MESSAGES/messages.po:2722: unknown named placeholder u'revs'
error: trac/locale/ca/LC_MESSAGES/messages.po:2994: unknown named placeholder u'shortname'
error: trac/locale/ca/LC_MESSAGES/messages.po:4692: unknown named placeholder u'created'
error: trac/locale/ca/LC_MESSAGES/messages.po:4697: unknown named placeholder u'created'
error: trac/locale/ca/LC_MESSAGES/messages.po:5274: unknown named placeholder u'file'
error: trac/locale/ca/LC_MESSAGES/messages.po:5286: unknown named placeholder u'file'
error: trac/locale/ca/LC_MESSAGES/messages.po:5806: unknown named placeholder u'in_repo'
error: trac/locale/ca/LC_MESSAGES/messages.po:5811: unknown named placeholder u'in_repo'
error: trac/locale/ca/LC_MESSAGES/messages.po:6670: unknown named placeholder u'name'
error: trac/locale/ca/LC_MESSAGES/messages.po:6675: unknown named placeholder u'name'
error: trac/locale/ca/LC_MESSAGES/messages.po:6703: incompatible format for placeholder u'version': 'd' and 's' are not compatible
compiling catalog 'trac/locale/ca/LC_MESSAGES/messages.po' to 'trac/locale/ca/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/hu/LC_MESSAGES/messages.po' to 'trac/locale/hu/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/cs/LC_MESSAGES/messages.po' to 'trac/locale/cs/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/tr/LC_MESSAGES/messages.po' to 'trac/locale/tr/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/fr/LC_MESSAGES/messages.po' to 'trac/locale/fr/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/hy/LC_MESSAGES/messages.po' to 'trac/locale/hy/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/sv/LC_MESSAGES/messages.po' to 'trac/locale/sv/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/eo/LC_MESSAGES/messages.po' to 'trac/locale/eo/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/pt_BR/LC_MESSAGES/messages.po' to 'trac/locale/pt_BR/LC_MESSAGES/messages.mo'
Traceback (most recent call last):
  File "setup.py", line 162, in <module>
    **extra
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/messages/frontend.py", line 133, in run
    catalog = read_po(infile, locale)
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/messages/pofile.py", line 121, in read_po
    catalog = Catalog(locale=locale, domain=domain, charset=charset)
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/messages/catalog.py", line 247, in __init__
    locale = Locale.parse(locale)
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/core.py", line 289, in parse
    raise UnknownLocaleError(input_id)
babel.core.UnknownLocaleError: unknown locale 'zh_CN'

Running the tests cases with Python 2.6, including functional test cases, results in:

Doctest: trac.test.Mock ... ok

======================================================================
ERROR: test_i18n_date_hint (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11257/teo-rjollos.git/trac/util/tests/datefmt.py", line 830, in test_i18n_date_hint
    zh_CN = Locale.parse('zh_CN')
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/core.py", line 289, in parse
    raise UnknownLocaleError(input_id)
UnknownLocaleError: unknown locale 'zh_CN'

<-- 5 more failures due to unknown locale 'zh_CN' -->

======================================================================
FAIL: test_format_compatibility (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11257/teo-rjollos.git/trac/util/tests/datefmt.py", line 1026, in test_format_compatibility
    datefmt.format_datetime(t, '%x %X', tz, en_US))
AssertionError: 'Aug 28, 2010 1:45:56 PM' != u'Aug 28, 2010, 1:45:56 PM'

======================================================================
FAIL: test_i18n_format_datetime (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11257/teo-rjollos.git/trac/util/tests/datefmt.py", line 739, in test_i18n_format_datetime
    locale=en_US))
AssertionError: 'Aug 28, 2010 1:45:56 PM' != u'Aug 28, 2010, 1:45:56 PM'

----------------------------------------------------------------------
Ran 1346 tests in 204.396s

FAILED (failures=2, errors=8)

[Jun Omae]

Ah, that's so bad.

The former issue is caused that CLDR removed locale-aliases and Babel doesn't accept the aliases. See ​http://unicode.org/cldr/trac/ticket/3930 and ​http://unicode.org/cldr/trac/changeset/5903.

The latter issue is fixed in [11876].

[Jun Omae]

Another thing: if selecting zh_TW, i18n/l10n doesn't work.

In this case, Babel 1.x automatically expands to zh_Hant_TW. Trac expects zh_TW. See ​https://github.com/mitsuhiko/babel/issues/30.

>>> import babel
>>> from babel.core import Locale
>>> babel.__version__
'1.2'
>>> Locale.parse('zh_TW')
Locale('zh', territory='TW', script='Hant')
>>> str(Locale.parse('zh_TW'))
'zh_Hant_TW'

2013-07-28 21:21:46,629 Trac[main] DEBUG: Negotiated locale: zh_Hant_TW -> ja

[mitsuhiko]

Just a quick heads up: the zn_CN not resolving correctly will be fixed in 1.3 (should be soon). However the locale parsing result will expand to zn_Hans_CN similar to how zh_TW expands to zh_Hant_TW.

There is very little I can do about this right now due to how Babel was structured. The CLDR no longer carries the base data sets so implementing the likely subtags was the quickest way for me to "fix" the issue.

Later versions of Babel will hopefully allow the locale to retain being zh_CN and internally do the subtag resolving and find the most appropriate data without changing the behavior of the locale. Similarly locales should then be able to finally have variants assigned.

You can track the status on that here: ​https://github.com/mitsuhiko/babel/issues/30. Work has not started yet :(

[mitsuhiko]

1.3 is out now and fixes the problem with zh_CN not loading.

[Jun Omae]

Thanks, Armin!

Other failures caused by differences of date/time formats are fixed in [11892]. Yet another thing is fixed in [11893].

======================================================================
FAIL: test_i18n_parse_date_datetime_meridiem (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jun66j5/src/trac/edgewall/svn/branches/1.0-stable/trac/util/tests/datefmt.py", line 962, in test_i18n_parse_date_datetime_meridiem
    zh_CN))
AssertionError: datetime.datetime(2011, 2, 22, 0, 45, 56, tzinfo=<FixedOffset "GMT +2:00" 2:00:00>) != datetime.datetime(2011, 2, 22, 12, 45, 56, tzinfo=<FixedOffset "GMT +2:00" 2:00:00>)

----------------------------------------------------------------------
Ran 1391 tests in 93.843s

FAILED (failures=1)

This is caused by periods except am/pm which are retrieved from get_period_names() in Babel 1.0+. In particular, the names are same for the am and morning periods in zh_CN. The method returns only am and pm until Babel 0.9.x.

>>> for key, val in sorted(get_period_names(locale='de').items()): print key, val
...
afternoon Nachmittag
am vorm.
earlyMorning Morgen
evening Abend
morning Vormittag
night Nacht
noon Mittag
pm nachm.
>>> for key, val in sorted(get_period_names(locale='zh_CN').items()): print key, val
...
afternoon 下午
am 上午
earlyMorning 清晨
midDay 中午
morning 上午
night 晚上
noon 中午
pm 下午
weeHours 凌晨

[Jun Omae]

Merged [11876,11892-11893] into trunk in [11933].

[Jun Omae]

Three issues still are remaining with Babel 1.3 (zh-tw and zh-cn have the same issues).

1. If selecting Chineses (zh-tw) in preference panel, expanded locale id, e.g. zh_Hant_TW, will be stored in session data.
2. When user's locale is Chineses (zh-tw), /chrome/js/messages/zh_Hant_TW.js is added for javascript message catalog. It must be zh_TW.js.
3. minor issue, init_catalog -l zh_TW generates the following instead of Language-Team: zh_TW ....

   "Language-Team: zh_Hant_TW <LL@li.org>\n"
   

Proposed workaround for first and second can be found in log:jomae.git:ticket11258.

[Jun Omae]

4. Cannot save default language in Basic Settings admin panel (refs comment:13:ticket:11337). [ff5178d3/jomae.git] is the fix for it.

I'll push [3dba044f/jomae.git] and [ff5178d3/jomae.git] to 1.0-stable. Also, the first issue in comment:8 occurs on 0.12-stable with Babel 1.x. I'll backport [3dba044f/jomae.git] to 0.12-stable.

[Jun Omae]

I'll backport [3dba044f/jomae.git] to 0.12-stable.

[0e6b0872/jomae.git] for 0.12-stable. On 0.12-stable, hyphen is used as separator of locale identifier in session data. After [10178], the separator has been changed from hyphen to underscore.

[Ryan J Ollos]

I'm not sure if this has been mentioned elsewhere, but another issue that could cause trouble for our users is that Trac 1.0.x (and even 1.1.x) supports Python 2.5, but Babel doesn't support Python < 2.6 since ​version 1.0. We end up with a traceback that is sure to surface on the mailing list from time to time:

$ easy_install babel
Searching for babel
Reading http://pypi.python.org/simple/babel/
Best match: Babel 1.3
Downloading https://pypi.python.org/packages/source/B/Babel/Babel-1.3.tar.gz#md5=5264ceb02717843cbc9ffce8e6e06bdb
Processing Babel-1.3.tar.gz
Writing /tmp/easy_install-0wKSNr/Babel-1.3/setup.cfg
Running Babel-1.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0wKSNr/Babel-1.3/egg-dist-tmp-loGAxe
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
  File "build/bdist.linux-x86_64/egg/babel/messages/mofile.py", line 75
    if b':' in item:
          ^
SyntaxError: invalid syntax

build/bdist.linux-x86_64/egg/babel/messages/catalog.py:142: Warning: 'as' will become a reserved keyword in Python 2.6
  File "build/bdist.linux-x86_64/egg/babel/messages/catalog.py", line 142
    except TranslationError as e:
                             ^
SyntaxError: invalid syntax

build/bdist.linux-x86_64/egg/babel/messages/frontend.py:419: Warning: 'as' will become a reserved keyword in Python 2.6
  File "build/bdist.linux-x86_64/egg/babel/messages/frontend.py", line 419
    except UnknownLocaleError as e:
                               ^
SyntaxError: invalid syntax

build/bdist.linux-x86_64/egg/babel/support.py:545: Warning: 'with' will become a reserved keyword in Python 2.6
  File "build/bdist.linux-x86_64/egg/babel/support.py", line 545
    with open(filename, 'rb') as fp:
            ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/babel/util.py", line 43
    br'[ \t\f]* \# .* coding[=:][ \t]*([-\w.]+)', re.VERBOSE)
                                               ^
SyntaxError: invalid syntax

/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/support.py:545: Warning: 'with' will become a reserved keyword in Python 2.6
  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/support.py", line 545
    with open(filename, 'rb') as fp:
            ^
SyntaxError: invalid syntax

  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/util.py", line 43
    br'[ \t\f]* \# .* coding[=:][ \t]*([-\w.]+)', re.VERBOSE)
                                               ^
SyntaxError: invalid syntax

  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/mofile.py", line 75
    if b':' in item:
          ^
SyntaxError: invalid syntax

/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/catalog.py:142: Warning: 'as' will become a reserved keyword in Python 2.6
  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/catalog.py", line 142
    except TranslationError as e:
                             ^
SyntaxError: invalid syntax

/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/frontend.py:419: Warning: 'as' will become a reserved keyword in Python 2.6
  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/frontend.py", line 419
    except UnknownLocaleError as e:
                               ^
SyntaxError: invalid syntax

Adding Babel 1.3 to easy-install.pth file
Installing pybabel script to /home/user/Workspace/t11329/bin

Installed /home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg
Processing dependencies for babel
Finished processing dependencies for babel

[Jun Omae]

Replying to rjollos:

I'm not sure if this has been mentioned elsewhere, but another issue that could cause trouble for our users is that Trac 1.0.x (and even 1.1.x) supports Python 2.5, but Babel doesn't support Python < 2.6 since ​version 1.0. We end up with a traceback that is sure to surface on the mailing list from time to time:

I've tried to add 'Babel': ['Babel>=0.9.5,<1.0'] to extras_require in Trac's setup.py. However it didn't work probably cause Babel is optional.

At least, I think that Babel's setup.py should have the check of Python version like this.

setup.py

@@ -15 +15 @@
 from distutils.cmd import Command
 
 
+if sys.version_info < (2, 6):
+    print("Babel requires Python 2.6 or later")
+    sys.exit(1)
+if (3,) <= sys.version_info < (3, 3):
+    print("Babel requires Python 3.3 or later")
+    sys.exit(1)
+
+
 class import_cldr(Command):
     description = 'imports and converts the CLDR data'
     user_options = []

[Jun Omae]

2. When user's locale is Chineses (zh-tw), /chrome/js/messages/zh_Hant_TW.js is added for javascript message catalog. It must be zh_TW.js.

It's the last issue with Babel 1.3. Fixing the issue needs trick, locale.script = None, in [c5761471/jomae.git] which is not committed yet.

[Ryan J Ollos]

Replying to jomae:

At least, I think that Babel's setup.py should have the check of Python version like this.

Thanks, I opened a ​ticket.

[Jun Omae]

It's the last issue with Babel 1.3. Fixing the issue needs trick, locale.script = None, in [c5761471/jomae.git] which is not committed yet.

The trick is rebased with latest branches in log:jomae.git:ticket11258.1_0.12.6dev and log:jomae.git:ticket11258.1.

[Jun Omae]

The trick is rebased with latest branches in log:jomae.git:ticket11258.1_0.12.6dev and log:jomae.git:ticket11258.1.

Committed in [12363] into 0.12-stable and merged into 1.0-stable and trunk in [12364-12365].

[Robbie <robertvehse@…>]

Shouldn't this be re-milestoned to 0.12.6 then?

[Jun Omae]

extract_messages command with Babel 1.3 shows the following error, originally reported at bloodhound:comment:47:ticket:694.

$ PYTHONPATH=. ~/venv/py26-babel1/bin/python setup.py extract_messages
running extract_messages
extracting messages from tracopt/__init__.py
extracting messages from tracopt/mimeview/__init__.py
extracting messages from tracopt/mimeview/enscript.py
extracting messages from tracopt/mimeview/php.py
Traceback (most recent call last):
  File "setup.py", line 163, in <module>
    **extra
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/frontend.py", line 305, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/extract.py", line 163, in extract_from_dir
    strip_comment_tags):
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/extract.py", line 190, in extract_from_file
    strip_comment_tags))
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/extract.py", line 267, in extract
    for lineno, funcname, messages, comments in results:
  File "/home/jun66j5/src/trac/edgewall/git/trac/dist.py", line 182, in extract_python
    {'__builtins__':{}}, {})
  File "<string>", line 0
SyntaxError: encoding declaration in Unicode string

[Jun Omae]

extract_messages command with Babel 1.3 shows the following error, originally reported at bloodhound:comment:47:ticket:694.

Fixed in [12470] and merged to trunk in [12471].

[Ryan J Ollos]

Replying to Ryan J Ollos:

Replying to jomae:

At least, I think that Babel's setup.py should have the check of Python version like this.

Thanks, I opened a ​ticket.

Change has been committed to Babel master and will be included in Babel 3.0.

[Jun Omae]

Replying to jomae:

The trick is rebased with latest branches in log:jomae.git:ticket11258.1_0.12.6dev and log:jomae.git:ticket11258.1.

Committed in [12363] into 0.12-stable and merged into 1.0-stable and trunk in [12364-12365].

After ​https://github.com/python-babel/babel/pull/305, that trick wouldn't work and raise a ValueError.