checking catalog utility which supports genshi markups

[Jun Omae]

The translators for Trac can verify with msgfmt --check command (and make check-<locale>). However the command cannot show wrong genshi markups in the catalog.

It's pretty difficult to notice such markups for the translators without some utilities. The wrong markups bring the internal error at rendering templates, then it's so bad.

I think we should provide a utility which can check genshi markups in message catalog files. changeset:check-catalog-genshi-markups/jomae

jun66j5@gotanda:2163$ msgfmt --statistics --check trac/locale/hy/LC_MESSAGES/messages.po
1095 translated messages.

jun66j5@gotanda:2164$ PYTHONPATH=$PWD python2.4 setup.py check_catalog -l hy
running check_catalog
checking catalog trac/locale/hy/LC_MESSAGES/messages.po
trac/locale/hy/LC_MESSAGES/messages.po:608: a format specification for argument 'name', as in 'msgstr[0]', doesn't exist in 'msgid'
trac/locale/hy/LC_MESSAGES/messages.po:1175: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1175: a genshi markup for [2:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1485: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1649: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1704: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:2047: cannot parse message (exceptions.IndexError: list index out of range)
trac/locale/hy/LC_MESSAGES/messages.po:2182: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:2186: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3591: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3596: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3738: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3918: a genshi markup for [2:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:4563: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:5080: a format specification for argument 'rev', as in 'msgstr[0]', doesn't exist in 'msgid'
trac/locale/hy/LC_MESSAGES/messages.po:5414: a format specification for argument 'cmd', as in 'msgstr[0]', doesn't exist in 'msgid'
trac/locale/hy/LC_MESSAGES/messages.po:5663: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:5769: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:5800: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'

And trac/locale/ja/LC_MESSAGES/messages.po in Trac 0.12.3 has wrong genshi markup….

jun66j5@gotanda:2148$ PYTHONPATH=$PWD python2.4 setup.py check_catalog -l ja
running check_catalog
checking catalog trac/locale/ja/LC_MESSAGES/messages.po
trac/locale/ja/LC_MESSAGES/messages.po:3586: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr'

[Christian Boos]

Looks great and very useful.

Any possibility to use this for automatically preventing the compilation of a catalog if the check fails?

Have you investigated the alternative way to integrate those checks as Babel babel.checkers extensions? ([babel:r227])

[Christian Boos]

The last changes you did work great, however one must remember to run setup.py egg_info first ;-)

I verified that with [acc4bce9/jomae] we indeed catch errors like the one you now fixed in r10986:

$ make compile-ja
python setup.py compile_catalog -l ja \
    compile_catalog_js -l ja generate_messages_js -l ja
running compile_catalog
error: trac/locale\ja\LC_MESSAGES\messages.po:3586: genshi markups are unbalanced [1:]
catalog 'trac/locale\\ja\\LC_MESSAGES\\messages.po' contains errors, skipping
running compile_catalog_js
compiling catalog 'trac/locale\\ja\\LC_MESSAGES\\messages-js.po' to 'trac/locale\\ja\\LC_MESSAGES\\messages-js.mo'
running generate_messages_js
generating messages javascript 'trac/locale\\ja\\LC_MESSAGES\\messages-js.mo' to 'trac/htdocs/js/messages\\ja.js'

In the above, you also see the effect of the following change in Babel (skip a catalog if it contains problematic translations):

babel/messages/frontend.py

@@ -158 +158 @@
                 log.warn('catalog %r is marked as fuzzy, skipping', po_file)
                 continue
 
+            error = None
             for message, errors in catalog.check():
                 for error in errors:
                     log.error('error: %s:%d: %s', po_file, message.lineno,
                               error)
+            if error is not None:
+                log.warn('catalog %r contains errors, skipping', po_file)
+                continue
 
             log.info('compiling catalog %r to %r', po_file, mo_file)

[Jun Omae]

Thanks for your trying. The check_genshi_markup method as babel.checkers works fine for Trac and Trac plugins, except compile_catalog in Babel doesn't skip messages that have errors.

However the method has a bit of a problem. In a environment which Trac is installed, the method always checks its catalogs in a software which doesn't even use Genshi (e.g. Sphinx).

If the messages extractor in Genshi can dump a genshi-markup flag like python-format in message.flags, the method can check only messages that contains genshi-markup. As far as I know, it cannot.

[Christian Boos]

Replying to jomae:

If the messages extractor in Genshi can dump a genshi-markup flag like python-format in message.flags, the method can check only messages that contains genshi-markup. As far as I know, it cannot.

… yet? I'm sure the Genshi maintainers wouldn't mind a patch adding this ;-)

In the meantime, we can probably do some kind of ugly hack, like simply checking for a global (set while we're running our own setup.py).

[Christian Boos]

In the meantime, we can probably do some kind of ugly hack, like simply checking for a global (set while we're running our own setup.py)

… which wouldn't work when translating Trac plugins.

What could work is to move the babel.checkers entry point in a file of its own (e.g. trac.babel_checkers.py, and then only activate the check if trac.dist in sys.modules, as the latter must be used for both Trac core and plugins translations.

[Christian Boos]

What could work is to move the babel.checkers entry point in a file of its own (e.g. trac.babel_checkers.py, and then only activate the check if trac.dist in sys.modules, as the latter must be used for both Trac core and plugins translations.

I verified that this approach works (the patch against 0.12-stable is also simpler as trac/dist.py is not affected, all the code is in babel_checkers.py​). But of course having an all or nothing flag is not as nice as having such a genshi-markup flag which would allow to do the checks only for messages extracted by Genshi (I haven't investigated how to add such a flag yet).

But as there's really nothing in babel_checkers.py which is specific to Trac (except this coarse grained test for now), I wonder if such an extension wouldn't be better added to Genshi itself…

[Christian Boos]

self-contained babel.checkers extension (extracted from trac/dist.py in [acc4bce969e7/jomae])

[Christian Boos]

We should try to get this in, in one way or the other, before 1.0.

[Christian Boos]

Jun, as I really wish we can get this feature in for 1.0, I'm also OK with your initial approach!

[Jun Omae]

I just reworked now. I'll push them by tonight.

• [74acce96/jomae.git] for trunk
• [9210efca/jomae.git] for 0.12-stable

[Christian Boos]

Works well, it's OK for me!

[Jun Omae]

Thanks, Christian! Applied in [11230] and merged in [11231].