Translation Interfaces for javascript

[trac-ja@…]

There are some message strings in javascript:

• source:trunk/trac/htdocs/js/blame.js@4574#L85
• source:trunk/trac/htdocs/js/expand_dir.js@6099#L30​

I'd like to translate these strings, but there are no interfaces for I18N.

Do you have any ideas?

[trac-ja@…]

I tried to write a patch to use Genshi text template in javascript. But it conflicts with $ between jQuery and Genshi.

[trac-ja@…]

babel:#110, babel:r376

[Jonas Borgström]

A first prototype for i18n support for javascript

[Jonas Borgström]

I just attached a first attempt at i18n support for javascript.

Most of it is pretty straight forward and the babel javascript extractor works perfectly.

One problem that is still unsolved is how to reduce the size of the js message catalog. The current implementation includes the entire message catalog which isn't very practical. One option would be to store the js strings in a separate domain but I'm not sure the babel extractor currently can extract messages into more than one domain.

[Christian Boos]

We should use babel:source:tags/0.9.4/contrib/babel.js and a specific domain for .js files.

[Christian Boos]

We should try to get this for the 0.12 release…

[Carsten Klein <carsten.klein@…>]

please make all translations part of the server process, do not implement it as part of the client.

by that, make all javascripts, upon adding them to the response, also a genshi text template that then would be processed by genshi in order to resolve translation issues.

for plugins that will not use _(…) for translation in their javascript genshi plain text templates, the author of the plugin must be made responsible.

[anonymous]

Replying to Carsten Klein <carsten.klein@…>:

please make all translations part of the server process, do not implement it as part of the client.

by that, make all javascripts, upon adding them to the response, also a genshi text template that then would be processed by genshi in order to resolve translation issues.

for plugins that will not use _(…) for translation in their javascript genshi plain text templates, the author of the plugin must be made responsible.

I was initially skeptical about this approach, thinking that this goes against the existing Javascript support in Babel (comment:6), and that it could be too costly to have to serve the htdocs/js files dynamically…

But since then, we discussed the migration to Jinja2 and in this new context, I think your proposal makes perfectly sense: much simpler and still reasonably fast.

Given that there will be only ${_(...)} expressions in the template, I think it should even be possible to use the same templates indifferently with Jinja2 or Genshi, making a 0.12.x backport possible. But the primary target should be 0.13 or some later milestone.

[Christian Boos]

(comment above by me)

[Jun Omae]

I attached the patch that supports javascript localization for babel and trac. It works fine for me.

babel-l10n-js.diff​

New generate_messages_js distutils.command generates javascript files from compiled MO files. The command is used after compile_catalog command.

Generated javascript files load translations using [babel:source:trunk/contrib/babel.js trunk/contrib/babel.js in babel].

trac-l10n-js.diff​

setup.py generate_messages_js command generates trac/htdocs/js/messages/*.js from trac/locale/*/LC_MESSAGES/messages.mo.

trac/htdocs/js/babel.js and trac/htdocs/js/messages/<locale>.js are added in trac/web/chrome.py.

[Christian Boos]

Looks really promising! Thanks for working on this.

1. we should use a specific domain here; unless I'm mistaken, with the current patches, the messages/*.js will contain all the strings?
2. the babel-l10n patch should perhaps be re-targeted to Trac, we could also trigger the generate_messages_js step right after compile_catalog (r9638)
3. the add_javascript lines should be conditional, it's only needed if Babel is used

Let's see how it goes, eventually it can even make it for 0.12 ;-)

[Jun Omae]

1. we should use a specific domain here; unless I'm mistaken, with the current patches, the messages/*.js will contain all the strings?

generate_messages_js command has -D option that is specified the domain. messages/*.js contain all strings in locale/*/LC_MESSAGES/<domain>.mo. Should the file path be messages/<domain>-<locale>.js?

2. the babel-l10n patch should perhaps be re-targeted to Trac, we could also trigger the generate_messages_js step right after compile_catalog (r9638)

Ok. I will rework the patch for only trac. Because I expect that trac plugins use generate_messages_js command.

3. the add_javascript lines should be conditional, it's only needed if Babel is used

Oh…, understand.

[Christian Boos]

Replying to jomae:

1. we should use a specific domain here; unless I'm mistaken, with the current patches, the messages/*.js will contain all the strings?

generate_messages_js command has -D option that is specified the domain. messages/*.js contain all strings in locale/*/LC_MESSAGES/<domain>.mo. Should the file path be messages/<domain>-<locale>.js?

What I'd like to have is an extraction of the messages in *.js files in trac/locale/messages-js.pot, the corresponding catalogs in locale/*/LC_MESSAGES/messages-js.{po,mo} and the .js dictionary generated in js/messages/<locale>.js.

2. the babel-l10n patch should perhaps be re-targeted to Trac, we could also trigger the generate_messages_js step right after compile_catalog (r9638)

Ok. I will rework the patch for only trac. Because I expect that trac plugins use generate_messages_js command.

Yes, this ultimately belongs to Babel, but I don't want to depend on a new Babel release for Trac 0.12. But also submit your patch there, so that in a future version of Trac we could upgrade the requirements.

[Jun Omae]

patch for javascript l10n based on r9738

[Jun Omae]

Replying to cboos:

What I'd like to have is an extraction of the messages in *.js files in trac/locale/messages-js.pot, the corresponding catalogs in locale/*/LC_MESSAGES/messages-js.{po,mo} and the .js dictionary generated in js/messages/<locale>.js.

That's splendid! It's unacceptable that js/messages/<locale>.js are too large.

Yes, this ultimately belongs to Babel, but I don't want to depend on a new Babel release for Trac 0.12. But also submit your patch there, so that in a future version of Trac we could upgrade the requirements.

Agree. The patch for Babel is for the future.

I reworked the patch for javascript l10n - l10n-js-r9738.diff​.

1. Prepared the below commands for trac/locale/messages-js.pot.
   • extract_messages_js
   • init_catalog_js
   • compile_catalog_js
   • update_catalog_js
2. generate_messages_js commands is implemented in setup.py.
3. compile_catalog and generate_message_js commands are run before install_lib command like compile_catalog command.

I tested in python 2.6 and babel 0.9.5.

[hasienda <hoff.st@…>]

reminder: as soon as you come to a final implementation, please update ​http://trac.edgewall.org/wiki/CookBook/PluginL10N#i18nforJavascripts, thanks.

[Christian Boos]

I've now tested the patch, but the generate_messages_js command fails for me:

# PYTHONPATH=. python setup.py generate_messages_js  -l fr
running generate_messages_js
generating messages javascript 'trac/locale\\fr\\LC_MESSAGES\\messages-js.mo' to 'trac/htdocs/js/messages\\fr.js'
Traceback (most recent call last):
  File "setup.py", line 333, in <module>
    **extra
  File "C:\Dev\Python261\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Dev\Python261\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Dev\Python261\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "setup.py", line 108, in run
    t = Translations(infile, self.domain)
  File "c:\dev\python261\lib\site-packages\Babel-0.9.5-py2.6.egg\babel\support.py", line 283, in __init__
    gettext.GNUTranslations.__init__(self, fp=fileobj)
  File "C:\Dev\Python261\lib\gettext.py", line 180, in __init__
    self._parse(fp)
  File "C:\Dev\Python261\lib\gettext.py", line 287, in _parse
    tlen, toff = unpack(ii, buf[transidx:transidx+8])
struct.error: unpack requires a string argument of length 8

It happens on a file with all the translations, but also on a file freshly generated and compiled (init_catalog_js -l fr, compile_catalog_js -l fr), so the issue should be easy to reproduce.

Ok, turns out it was a missing 'b'inary spec while opening the file (the joy of Windows ;-) ).

Now it works perfectly! Great job, I'll integrate this tomorrow.

[Christian Boos]

Btw, I wanted to tweak the code a bit to re-use our trac.util.presentation.to_json, but thanks to the loading of trac/utils/__init__.py … this doesn't work (chokes on importing genshi.builder, via trac.util.translation and trac.util.text). So it looks like we have to keep this second implementation in place. Other ideas?

[Remy Blank]

Replying to cboos:

(chokes on importing genshi.builder, via trac.util.translation and trac.util.text)

What exception do you get?

[Christian Boos]

Oh, my bad, I used a Python version which had no Genshi installed. So yes, this could work.

[Christian Boos]

setup: remove to_json compatibility code and instead reuse the one from trac.util.presentation

[Christian Boos]

Patch committed in r9758 and several tweaks done in the follow up changesets.

In r9763, I moved most of the distutils related code in source:trunk/trac/util/dist.py​. There are two benefits:

• setup.py is getting even clearer than before (the r9638 changes are also moved there)
• the plugins can now easily reuse our special tweaks by importing either get_l10n_cmdclass or get_l10n_js_cmdclass from that module (depending whether they need to translate Javascript or not); it remains to be seen if more tweaks are needed in order to have js translation working for plugins, or if everything can be done with the config files (see _js sections in source:trunk/trac/setup.cfg for an example)

Note that in r9764, I adapted the Makefile L10N targets so that they take the messages-js.{po,pot} files into account.

Finally, as an added bonus, in r9766 I added an extractor in order to get a few messages defined in <scripts> tags. Quite mysteriously, it seems that the normal genshi:**.html extraction doesn't catch those messages redundantly… good!

It would be good to have some testing feedback on the changes, and a second check that I didn't forget any Javascript message. See also the updates in TracL10N@104#ForCommitters.

[Christian Boos]

… it seems that the normal genshi:**.html extraction doesn't catch those messages redundantly…

Of course, as the _(...) annotations in <script> tags are not contained within Genshi expressions, so they just go through the Genshi extractor unnoticed, as text.

[Christian Boos]

Replying to jomae:

(about babel-l10n-js.diff​) The patch for Babel is for the future.

Jun, do you intend to submit this to the Babel project? If so, don't forget to integrate r9762 in the patch.

[Jun Omae]

Replying to cboos:

Jun, do you intend to submit this to the Babel project? If so, don't forget to integrate r9762 in the patch.

No. No need to submit the patch, cause you have implemented get_l10n_cmdclass and get_l10n_js_cmdclass methods in trac/util/dist.py. The plugins for Trac 0.12+ should use these methods in the setup.py.

These get_*_cmdclass methods are excellent! Thanks.