Edgewall Software
Modify

Opened 7 years ago

Closed 6 years ago

Last modified 6 years ago

#10569 closed enhancement (fixed)

checking catalog utility which supports genshi markups

Reported by: Jun Omae Owned by: Jun Omae
Priority: high Milestone: 0.12.4
Component: i18n Version: 0.12-stable
Severity: normal Keywords: genshi
Cc: hodgestar
Release Notes:

Added support for genshi markups for make check-*

API Changes:

Description

The translators for Trac can verify with msgfmt --check command (and make check-<locale>). However the command cannot show wrong genshi markups in the catalog.

It's pretty difficult to notice such markups for the translators without some utilities. The wrong markups bring the internal error at rendering templates, then it's so bad.

I think we should provide a utility which can check genshi markups in message catalog files. changeset:check-catalog-genshi-markups/jomae

jun66j5@gotanda:2163$ msgfmt --statistics --check trac/locale/hy/LC_MESSAGES/messages.po
1095 translated messages.

jun66j5@gotanda:2164$ PYTHONPATH=$PWD python2.4 setup.py check_catalog -l hy
running check_catalog
checking catalog trac/locale/hy/LC_MESSAGES/messages.po
trac/locale/hy/LC_MESSAGES/messages.po:608: a format specification for argument 'name', as in 'msgstr[0]', doesn't exist in 'msgid'
trac/locale/hy/LC_MESSAGES/messages.po:1175: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1175: a genshi markup for [2:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1485: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1649: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:1704: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:2047: cannot parse message (exceptions.IndexError: list index out of range)
trac/locale/hy/LC_MESSAGES/messages.po:2182: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:2186: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3591: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3596: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3738: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:3918: a genshi markup for [2:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:4563: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:5080: a format specification for argument 'rev', as in 'msgstr[0]', doesn't exist in 'msgid'
trac/locale/hy/LC_MESSAGES/messages.po:5414: a format specification for argument 'cmd', as in 'msgstr[0]', doesn't exist in 'msgid'
trac/locale/hy/LC_MESSAGES/messages.po:5663: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:5769: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'
trac/locale/hy/LC_MESSAGES/messages.po:5800: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr[0]'

And trac/locale/ja/LC_MESSAGES/messages.po in Trac 0.12.3 has wrong genshi markup….

jun66j5@gotanda:2148$ PYTHONPATH=$PWD python2.4 setup.py check_catalog -l ja
running check_catalog
checking catalog trac/locale/ja/LC_MESSAGES/messages.po
trac/locale/ja/LC_MESSAGES/messages.po:3586: a genshi markup for [1:], as in 'msgid', doesn't exist in 'msgstr'

Attachments (1)

babel_checkers.py (2.0 KB ) - added by Christian Boos 7 years ago.
self-contained babel.checkers extension (extracted from trac/dist.py in [acc4bce969e7/jomae])

Download all attachments as: .zip

Change History (13)

comment:1 Changed 7 years ago by Christian Boos

Looks great and very useful.

Any possibility to use this for automatically preventing the compilation of a catalog if the check fails?

Have you investigated the alternative way to integrate those checks as Babel babel.checkers extensions? (r227)

comment:2 Changed 7 years ago by Christian Boos

The last changes you did work great, however one must remember to run setup.py egg_info first ;-)

I verified that with [acc4bce9/jomae] we indeed catch errors like the one you now fixed in r10986:

$ make compile-ja
python setup.py compile_catalog -l ja \
    compile_catalog_js -l ja generate_messages_js -l ja
running compile_catalog
error: trac/locale\ja\LC_MESSAGES\messages.po:3586: genshi markups are unbalanced [1:]
catalog 'trac/locale\\ja\\LC_MESSAGES\\messages.po' contains errors, skipping
running compile_catalog_js
compiling catalog 'trac/locale\\ja\\LC_MESSAGES\\messages-js.po' to 'trac/locale\\ja\\LC_MESSAGES\\messages-js.mo'
running generate_messages_js
generating messages javascript 'trac/locale\\ja\\LC_MESSAGES\\messages-js.mo' to 'trac/htdocs/js/messages\\ja.js'

In the above, you also see the effect of the following change in Babel (skip a catalog if it contains problematic translations):

  • babel/messages/frontend.py

     
    158158                log.warn('catalog %r is marked as fuzzy, skipping', po_file)
    159159                continue
    160160
     161            error = None
    161162            for message, errors in catalog.check():
    162163                for error in errors:
    163164                    log.error('error: %s:%d: %s', po_file, message.lineno,
    164165                              error)
     166            if error is not None:
     167                log.warn('catalog %r contains errors, skipping', po_file)
     168                continue
    165169
    166170            log.info('compiling catalog %r to %r', po_file, mo_file)
Last edited 7 years ago by Christian Boos (previous) (diff)

comment:3 Changed 7 years ago by Jun Omae

Thanks for your trying. The check_genshi_markup method as babel.checkers works fine for Trac and Trac plugins, except compile_catalog in Babel doesn't skip messages that have errors.

However the method has a bit of a problem. In a environment which Trac is installed, the method always checks its catalogs in a software which doesn't even use Genshi (e.g. Sphinx).

If the messages extractor in Genshi can dump a genshi-markup flag like python-format in message.flags, the method can check only messages that contains genshi-markup. As far as I know, it cannot.

comment:4 in reply to:  3 ; Changed 7 years ago by Christian Boos

Replying to jomae:

If the messages extractor in Genshi can dump a genshi-markup flag like python-format in message.flags, the method can check only messages that contains genshi-markup. As far as I know, it cannot.

… yet? I'm sure the Genshi maintainers wouldn't mind a patch adding this ;-)

In the meantime, we can probably do some kind of ugly hack, like simply checking for a global (set while we're running our own setup.py).

comment:5 in reply to:  4 ; Changed 7 years ago by Christian Boos

In the meantime, we can probably do some kind of ugly hack, like simply checking for a global (set while we're running our own setup.py)

… which wouldn't work when translating Trac plugins.

What could work is to move the babel.checkers entry point in a file of its own (e.g. trac.babel_checkers.py, and then only activate the check if trac.dist in sys.modules, as the latter must be used for both Trac core and plugins translations.

comment:6 in reply to:  5 Changed 7 years ago by Christian Boos

Cc: hodgestar added
Keywords: genshi added

What could work is to move the babel.checkers entry point in a file of its own (e.g. trac.babel_checkers.py, and then only activate the check if trac.dist in sys.modules, as the latter must be used for both Trac core and plugins translations.

I verified that this approach works (the patch against 0.12-stable is also simpler as trac/dist.py is not affected, all the code is in babel_checkers.py). But of course having an all or nothing flag is not as nice as having such a genshi-markup flag which would allow to do the checks only for messages extracted by Genshi (I haven't investigated how to add such a flag yet).

But as there's really nothing in babel_checkers.py which is specific to Trac (except this coarse grained test for now), I wonder if such an extension wouldn't be better added to Genshi itself…

Changed 7 years ago by Christian Boos

Attachment: babel_checkers.py added

self-contained babel.checkers extension (extracted from trac/dist.py in [acc4bce969e7/jomae])

comment:7 Changed 6 years ago by Christian Boos

Priority: normalhigh

We should try to get this in, in one way or the other, before 1.0.

comment:8 Changed 6 years ago by Christian Boos

Jun, as I really wish we can get this feature in for 1.0, I'm also OK with your initial approach!

comment:9 Changed 6 years ago by Jun Omae

I just reworked now. I'll push them by tonight.

comment:10 Changed 6 years ago by Christian Boos

Works well, it's OK for me!

comment:11 Changed 6 years ago by Jun Omae

Release Notes: modified (diff)
Resolution: fixed
Status: newclosed

Thanks, Christian! Applied in [11230] and merged in [11231].

comment:12 Changed 6 years ago by Jun Omae

Owner: set to Jun Omae

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.