Edgewall Software
Modify

Opened 5 years ago

Closed 5 years ago

Last modified 3 years ago

#11258 closed defect (fixed)

Catalog compilation and test case failures with Babel >= 1.0

Reported by: Ryan J Ollos Owned by: Jun Omae
Priority: normal Milestone: 0.12.6
Component: i18n Version: 1.0-stable
Severity: normal Keywords: babel10
Cc:
Release Notes:

Add support for Babel 1.0 and later.

API Changes:

Description

In the last day or so we've rocketed forward to Babel 1.2, now available on PyPI. I only noticed this just now due to some test case failures.

Compiling the catalogs results in:

(t11257)user@ubuntu:~/Workspace/t11257/teo-rjollos.git$ python setup.py compile_catalog -f
running compile_catalog
compiling catalog 'trac/locale/he/LC_MESSAGES/messages.po' to 'trac/locale/he/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/gl/LC_MESSAGES/messages.po' to 'trac/locale/gl/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/fa/LC_MESSAGES/messages.po' to 'trac/locale/fa/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/nb/LC_MESSAGES/messages.po' to 'trac/locale/nb/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/en_GB/LC_MESSAGES/messages.po' to 'trac/locale/en_GB/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/fi/LC_MESSAGES/messages.po' to 'trac/locale/fi/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/it/LC_MESSAGES/messages.po' to 'trac/locale/it/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/es_MX/LC_MESSAGES/messages.po' to 'trac/locale/es_MX/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/ja/LC_MESSAGES/messages.po' to 'trac/locale/ja/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/vi/LC_MESSAGES/messages.po' to 'trac/locale/vi/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/en_US/LC_MESSAGES/messages.po' to 'trac/locale/en_US/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/et/LC_MESSAGES/messages.po' to 'trac/locale/et/LC_MESSAGES/messages.mo'
error: trac/locale/ca/LC_MESSAGES/messages.po:668: unknown named placeholder u'resource'
error: trac/locale/ca/LC_MESSAGES/messages.po:704: unknown named placeholder u'subject'
error: trac/locale/ca/LC_MESSAGES/messages.po:709: unknown named placeholder u'subject'
error: trac/locale/ca/LC_MESSAGES/messages.po:736: unknown named placeholder u'page'
error: trac/locale/ca/LC_MESSAGES/messages.po:2722: unknown named placeholder u'revs'
error: trac/locale/ca/LC_MESSAGES/messages.po:2994: unknown named placeholder u'shortname'
error: trac/locale/ca/LC_MESSAGES/messages.po:4692: unknown named placeholder u'created'
error: trac/locale/ca/LC_MESSAGES/messages.po:4697: unknown named placeholder u'created'
error: trac/locale/ca/LC_MESSAGES/messages.po:5274: unknown named placeholder u'file'
error: trac/locale/ca/LC_MESSAGES/messages.po:5286: unknown named placeholder u'file'
error: trac/locale/ca/LC_MESSAGES/messages.po:5806: unknown named placeholder u'in_repo'
error: trac/locale/ca/LC_MESSAGES/messages.po:5811: unknown named placeholder u'in_repo'
error: trac/locale/ca/LC_MESSAGES/messages.po:6670: unknown named placeholder u'name'
error: trac/locale/ca/LC_MESSAGES/messages.po:6675: unknown named placeholder u'name'
error: trac/locale/ca/LC_MESSAGES/messages.po:6703: incompatible format for placeholder u'version': 'd' and 's' are not compatible
compiling catalog 'trac/locale/ca/LC_MESSAGES/messages.po' to 'trac/locale/ca/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/hu/LC_MESSAGES/messages.po' to 'trac/locale/hu/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/cs/LC_MESSAGES/messages.po' to 'trac/locale/cs/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/tr/LC_MESSAGES/messages.po' to 'trac/locale/tr/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/fr/LC_MESSAGES/messages.po' to 'trac/locale/fr/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/hy/LC_MESSAGES/messages.po' to 'trac/locale/hy/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/sv/LC_MESSAGES/messages.po' to 'trac/locale/sv/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/eo/LC_MESSAGES/messages.po' to 'trac/locale/eo/LC_MESSAGES/messages.mo'
compiling catalog 'trac/locale/pt_BR/LC_MESSAGES/messages.po' to 'trac/locale/pt_BR/LC_MESSAGES/messages.mo'
Traceback (most recent call last):
  File "setup.py", line 162, in <module>
    **extra
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/messages/frontend.py", line 133, in run
    catalog = read_po(infile, locale)
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/messages/pofile.py", line 121, in read_po
    catalog = Catalog(locale=locale, domain=domain, charset=charset)
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/messages/catalog.py", line 247, in __init__
    locale = Locale.parse(locale)
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/core.py", line 289, in parse
    raise UnknownLocaleError(input_id)
babel.core.UnknownLocaleError: unknown locale 'zh_CN'

Running the tests cases with Python 2.6, including functional test cases, results in:

Doctest: trac.test.Mock ... ok

======================================================================
ERROR: test_i18n_date_hint (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11257/teo-rjollos.git/trac/util/tests/datefmt.py", line 830, in test_i18n_date_hint
    zh_CN = Locale.parse('zh_CN')
  File "/home/user/Workspace/t11257/lib/python2.6/site-packages/Babel-1.2-py2.6.egg/babel/core.py", line 289, in parse
    raise UnknownLocaleError(input_id)
UnknownLocaleError: unknown locale 'zh_CN'

<-- 5 more failures due to unknown locale 'zh_CN' -->

======================================================================
FAIL: test_format_compatibility (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11257/teo-rjollos.git/trac/util/tests/datefmt.py", line 1026, in test_format_compatibility
    datefmt.format_datetime(t, '%x %X', tz, en_US))
AssertionError: 'Aug 28, 2010 1:45:56 PM' != u'Aug 28, 2010, 1:45:56 PM'

======================================================================
FAIL: test_i18n_format_datetime (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Workspace/t11257/teo-rjollos.git/trac/util/tests/datefmt.py", line 739, in test_i18n_format_datetime
    locale=en_US))
AssertionError: 'Aug 28, 2010 1:45:56 PM' != u'Aug 28, 2010, 1:45:56 PM'

----------------------------------------------------------------------
Ran 1346 tests in 204.396s

FAILED (failures=2, errors=8)

Attachments (0)

Change History (23)

comment:1 Changed 5 years ago by Jun Omae

Ah, that's so bad.

The former issue is caused that CLDR removed locale-aliases and Babel doesn't accept the aliases. See http://unicode.org/cldr/trac/ticket/3930 and http://unicode.org/cldr/trac/changeset/5903.

The latter issue is fixed in [11876].

comment:2 Changed 5 years ago by Jun Omae

Another thing: if selecting zh_TW, i18n/l10n doesn't work.

In this case, Babel 1.x automatically expands to zh_Hant_TW. Trac expects zh_TW. See https://github.com/mitsuhiko/babel/issues/30.

>>> import babel
>>> from babel.core import Locale
>>> babel.__version__
'1.2'
>>> Locale.parse('zh_TW')
Locale('zh', territory='TW', script='Hant')
>>> str(Locale.parse('zh_TW'))
'zh_Hant_TW'
2013-07-28 21:21:46,629 Trac[main] DEBUG: Negotiated locale: zh_Hant_TW -> ja
Last edited 5 years ago by Jun Omae (previous) (diff)

comment:3 Changed 5 years ago by mitsuhiko

Just a quick heads up: the zn_CN not resolving correctly will be fixed in 1.3 (should be soon). However the locale parsing result will expand to zn_Hans_CN similar to how zh_TW expands to zh_Hant_TW.

There is very little I can do about this right now due to how Babel was structured. The CLDR no longer carries the base data sets so implementing the likely subtags was the quickest way for me to "fix" the issue.

Later versions of Babel will hopefully allow the locale to retain being zh_CN and internally do the subtag resolving and find the most appropriate data without changing the behavior of the locale. Similarly locales should then be able to finally have variants assigned.

You can track the status on that here: https://github.com/mitsuhiko/babel/issues/30. Work has not started yet :(

comment:4 Changed 5 years ago by mitsuhiko

1.3 is out now and fixes the problem with zh_CN not loading.

comment:5 Changed 5 years ago by Jun Omae

Thanks, Armin!

Other failures caused by differences of date/time formats are fixed in [11892]. Yet another thing is fixed in [11893].

======================================================================
FAIL: test_i18n_parse_date_datetime_meridiem (trac.util.tests.datefmt.I18nDateFormatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jun66j5/src/trac/edgewall/svn/branches/1.0-stable/trac/util/tests/datefmt.py", line 962, in test_i18n_parse_date_datetime_meridiem
    zh_CN))
AssertionError: datetime.datetime(2011, 2, 22, 0, 45, 56, tzinfo=<FixedOffset "GMT +2:00" 2:00:00>) != datetime.datetime(2011, 2, 22, 12, 45, 56, tzinfo=<FixedOffset "GMT +2:00" 2:00:00>)

----------------------------------------------------------------------
Ran 1391 tests in 93.843s

FAILED (failures=1)

This is caused by periods except am/pm which are retrieved from get_period_names() in Babel 1.0+. In particular, the names are same for the am and morning periods in zh_CN. The method returns only am and pm until Babel 0.9.x.

>>> for key, val in sorted(get_period_names(locale='de').items()): print key, val
...
afternoon Nachmittag
am vorm.
earlyMorning Morgen
evening Abend
morning Vormittag
night Nacht
noon Mittag
pm nachm.
>>> for key, val in sorted(get_period_names(locale='zh_CN').items()): print key, val
...
afternoon 下午
am 上午
earlyMorning 清晨
midDay 中午
morning 上午
night 晚上
noon 中午
pm 下午
weeHours 凌晨

comment:6 Changed 5 years ago by Jun Omae

Owner: set to Jun Omae
Status: newassigned

Merged [11876,11892-11893] into trunk in [11933].

comment:7 Changed 5 years ago by Ryan J Ollos

Summary: Catalog compilation and test case failures with Babel 1.2Catalog compilation and test case failures with Babel >= 1.0

comment:8 Changed 5 years ago by Jun Omae

Three issues still are remaining with Babel 1.3 (zh-tw and zh-cn have the same issues).

  1. If selecting Chineses (zh-tw) in preference panel, expanded locale id, e.g. zh_Hant_TW, will be stored in session data.
  2. When user's locale is Chineses (zh-tw), /chrome/js/messages/zh_Hant_TW.js is added for javascript message catalog. It must be zh_TW.js.
  3. minor issue, init_catalog -l zh_TW generates the following instead of Language-Team: zh_TW ....
    "Language-Team: zh_Hant_TW <LL@li.org>\n"
    

Proposed workaround for first and second can be found in log:jomae.git:ticket11258.

comment:9 Changed 5 years ago by Jun Omae

  1. Cannot save default language in Basic Settings admin panel (refs comment:13:ticket:11337). [ff5178d3/jomae.git] is the fix for it.

I'll push [3dba044f/jomae.git] and [ff5178d3/jomae.git] to 1.0-stable. Also, the first issue in comment:8 occurs on 0.12-stable with Babel 1.x. I'll backport [3dba044f/jomae.git] to 0.12-stable.

comment:10 in reply to:  9 Changed 5 years ago by Jun Omae

I'll backport [3dba044f/jomae.git] to 0.12-stable.

[0e6b0872/jomae.git] for 0.12-stable. On 0.12-stable, hyphen is used as separator of locale identifier in session data. After [10178], the separator has been changed from hyphen to underscore.

comment:11 Changed 5 years ago by Ryan J Ollos

I'm not sure if this has been mentioned elsewhere, but another issue that could cause trouble for our users is that Trac 1.0.x (and even 1.1.x) supports Python 2.5, but Babel doesn't support Python < 2.6 since version 1.0. We end up with a traceback that is sure to surface on the mailing list from time to time:

$ easy_install babel
Searching for babel
Reading http://pypi.python.org/simple/babel/
Best match: Babel 1.3
Downloading https://pypi.python.org/packages/source/B/Babel/Babel-1.3.tar.gz#md5=5264ceb02717843cbc9ffce8e6e06bdb
Processing Babel-1.3.tar.gz
Writing /tmp/easy_install-0wKSNr/Babel-1.3/setup.cfg
Running Babel-1.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0wKSNr/Babel-1.3/egg-dist-tmp-loGAxe
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
  File "build/bdist.linux-x86_64/egg/babel/messages/mofile.py", line 75
    if b':' in item:
          ^
SyntaxError: invalid syntax

build/bdist.linux-x86_64/egg/babel/messages/catalog.py:142: Warning: 'as' will become a reserved keyword in Python 2.6
  File "build/bdist.linux-x86_64/egg/babel/messages/catalog.py", line 142
    except TranslationError as e:
                             ^
SyntaxError: invalid syntax

build/bdist.linux-x86_64/egg/babel/messages/frontend.py:419: Warning: 'as' will become a reserved keyword in Python 2.6
  File "build/bdist.linux-x86_64/egg/babel/messages/frontend.py", line 419
    except UnknownLocaleError as e:
                               ^
SyntaxError: invalid syntax

build/bdist.linux-x86_64/egg/babel/support.py:545: Warning: 'with' will become a reserved keyword in Python 2.6
  File "build/bdist.linux-x86_64/egg/babel/support.py", line 545
    with open(filename, 'rb') as fp:
            ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/babel/util.py", line 43
    br'[ \t\f]* \# .* coding[=:][ \t]*([-\w.]+)', re.VERBOSE)
                                               ^
SyntaxError: invalid syntax

/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/support.py:545: Warning: 'with' will become a reserved keyword in Python 2.6
  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/support.py", line 545
    with open(filename, 'rb') as fp:
            ^
SyntaxError: invalid syntax

  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/util.py", line 43
    br'[ \t\f]* \# .* coding[=:][ \t]*([-\w.]+)', re.VERBOSE)
                                               ^
SyntaxError: invalid syntax

  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/mofile.py", line 75
    if b':' in item:
          ^
SyntaxError: invalid syntax

/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/catalog.py:142: Warning: 'as' will become a reserved keyword in Python 2.6
  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/catalog.py", line 142
    except TranslationError as e:
                             ^
SyntaxError: invalid syntax

/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/frontend.py:419: Warning: 'as' will become a reserved keyword in Python 2.6
  File "/home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg/babel/messages/frontend.py", line 419
    except UnknownLocaleError as e:
                               ^
SyntaxError: invalid syntax

Adding Babel 1.3 to easy-install.pth file
Installing pybabel script to /home/user/Workspace/t11329/bin

Installed /home/user/Workspace/t11329/lib/python2.5/site-packages/Babel-1.3-py2.5.egg
Processing dependencies for babel
Finished processing dependencies for babel

comment:12 in reply to:  11 ; Changed 5 years ago by Jun Omae

Replying to rjollos:

I'm not sure if this has been mentioned elsewhere, but another issue that could cause trouble for our users is that Trac 1.0.x (and even 1.1.x) supports Python 2.5, but Babel doesn't support Python < 2.6 since version 1.0. We end up with a traceback that is sure to surface on the mailing list from time to time:

I've tried to add 'Babel': ['Babel>=0.9.5,<1.0'] to extras_require in Trac's setup.py. However it didn't work probably cause Babel is optional.

At least, I think that Babel's setup.py should have the check of Python version like this.

  • setup.py

    diff --git a/setup.py b/setup.py
    index 7a566ab..f721375 100755
    a b except ImportError:  
    1515from distutils.cmd import Command
    1616
    1717
     18if sys.version_info < (2, 6):
     19    print("Babel requires Python 2.6 or later")
     20    sys.exit(1)
     21if (3,) <= sys.version_info < (3, 3):
     22    print("Babel requires Python 3.3 or later")
     23    sys.exit(1)
     24
     25
    1826class import_cldr(Command):
    1927    description = 'imports and converts the CLDR data'
    2028    user_options = []

comment:13 Changed 5 years ago by Jun Omae

Keywords: babel10 added; babel removed

comment:14 in reply to:  8 ; Changed 5 years ago by Jun Omae

  1. When user's locale is Chineses (zh-tw), /chrome/js/messages/zh_Hant_TW.js is added for javascript message catalog. It must be zh_TW.js.

It's the last issue with Babel 1.3. Fixing the issue needs trick, locale.script = None, in [c5761471/jomae.git] which is not committed yet.

Last edited 5 years ago by Jun Omae (previous) (diff)

comment:15 in reply to:  12 ; Changed 5 years ago by Ryan J Ollos

Replying to jomae:

At least, I think that Babel's setup.py should have the check of Python version like this.

Thanks, I opened a ticket.

comment:16 in reply to:  14 ; Changed 5 years ago by Jun Omae

It's the last issue with Babel 1.3. Fixing the issue needs trick, locale.script = None, in [c5761471/jomae.git] which is not committed yet.

The trick is rebased with latest branches in log:jomae.git:ticket11258.1_0.12.6dev and log:jomae.git:ticket11258.1.

comment:17 in reply to:  16 ; Changed 5 years ago by Jun Omae

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

The trick is rebased with latest branches in log:jomae.git:ticket11258.1_0.12.6dev and log:jomae.git:ticket11258.1.

Committed in [12363] into 0.12-stable and merged into 1.0-stable and trunk in [12364-12365].

comment:18 Changed 5 years ago by Robbie <robertvehse@…>

Shouldn't this be re-milestoned to 0.12.6 then?

comment:19 Changed 5 years ago by Ryan J Ollos

Milestone: 1.0.20.12.6

comment:20 Changed 5 years ago by Jun Omae

Resolution: fixed
Status: closedreopened

extract_messages command with Babel 1.3 shows the following error, originally reported at bloodhound:comment:47:ticket:694.

$ PYTHONPATH=. ~/venv/py26-babel1/bin/python setup.py extract_messages
running extract_messages
extracting messages from tracopt/__init__.py
extracting messages from tracopt/mimeview/__init__.py
extracting messages from tracopt/mimeview/enscript.py
extracting messages from tracopt/mimeview/php.py
Traceback (most recent call last):
  File "setup.py", line 163, in <module>
    **extra
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/frontend.py", line 305, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/extract.py", line 163, in extract_from_dir
    strip_comment_tags):
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/extract.py", line 190, in extract_from_file
    strip_comment_tags))
  File "/home/jun66j5/venv/py26-babel1/lib/python2.6/site-packages/babel/messages/extract.py", line 267, in extract
    for lineno, funcname, messages, comments in results:
  File "/home/jun66j5/src/trac/edgewall/git/trac/dist.py", line 182, in extract_python
    {'__builtins__':{}}, {})
  File "<string>", line 0
SyntaxError: encoding declaration in Unicode string

comment:21 in reply to:  20 Changed 5 years ago by Jun Omae

Resolution: fixed
Status: reopenedclosed

extract_messages command with Babel 1.3 shows the following error, originally reported at bloodhound:comment:47:ticket:694.

Fixed in [12470] and merged to trunk in [12471].

comment:22 in reply to:  15 Changed 3 years ago by Ryan J Ollos

Replying to Ryan J Ollos:

Replying to jomae:

At least, I think that Babel's setup.py should have the check of Python version like this.

Thanks, I opened a ticket.

Change has been committed to Babel master and will be included in Babel 3.0.

comment:23 in reply to:  17 Changed 3 years ago by Jun Omae

Replying to jomae:

The trick is rebased with latest branches in log:jomae.git:ticket11258.1_0.12.6dev and log:jomae.git:ticket11258.1.

Committed in [12363] into 0.12-stable and merged into 1.0-stable and trunk in [12364-12365].

After https://github.com/python-babel/babel/pull/305, that trick wouldn't work and raise a ValueError.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.