= Adding i18n/l10n to Trac plugins ^(Trac >= 0.12)^ = == Intro and Motivation == Are you a user of Trac and do your work with it, nothing more? Well, you may ignore this page and go on reading another subpage of [wiki:CookBook CookBook]. Professional coders/translators, please skip to the actual cookbook content in '[#Requiredworkflow Required workflow]', since there can't be any news for you before that section. If you want to learn about translation for a plugin, that as you know already provides one/several message catalog/s, the section '[#Dotranslatorswork Do translators work]' and following parts are for you. Ultimately, all plugin maintainers and developers in general, who are facing requests and are willing to take care for growing demand of their plugin to speak same (foreign) language(s) as Trac >= 0.12 should just read on. == i18n, l10n, ... help! == In short '''i18n''' stands for '''`i`'''`nternationalizatio`'''`n`''' (count 18 more chars between i and n) and is defined as software design for programs with translation support. '''`l`'''`ocalisatio`'''`n`''' that is abbreviated as '''l10n''' could be seen as a follow-up process providing data for one or more locales. It is taking care of feature differences between the original/default (that is English is most cases including Trac) and a given locale as well. Such features are i.e. sentence structure including punctuation and formatting of numbers, date/time strings, and currencies. Once you did some ground work at the source (i18n), most remaining is proper translation work (l10n) putting more or less effort in preserving the sense of the original while looking as native locale as possible.^[#a1 1]^ '''NLS''' (National Language Support or Native Language Support) is meant to be the sum of both. And there are more related terms that we could safely skip for now.^[#a1 1], [#a2 2]^ == Background and concept of i18n/l10n support for Trac plugins == It begun with adding Babel to Trac. Some plugin maintainers created their own translation module inside each plugin separately. Growing amount of code redundancy and possibility of error within imperfect copies and variants of a translation module all that was certainly not a desirable situation. And Trac core maintainers took responsibility with adding functions dedicated to i18n/l10n support for Trac plugins. The evolution of this functions has been documented in [comment:11:ticket:7497 ticket 7497]. The final implementation as mentioned there in [comment:12:ticket:7497 comment 12] was introduced to Trac trunk in changeset r7705 and finally done with changeset r7714. Now adding the needed i18n/l10n helper functions is done by importing a set of functions from `trac/util/translation.py` and providing proper configuration for an additional translation layer ('domain') inside the plugin code. On plugin initialization the dedicated translation domain is created as well and corresponding catalog files holding translated messages are loaded into it. Whenever a translatable text is encountered during runtime inside plugin's code, i18n/l10n helper functions will try to get the corresponding translation from the message catalog of plugin's domain and fall back silently to Trac's main message catalog, if needed. The message catalog selection is done according to the locale setting. Valid settings are a combination of language and country code, optionally extended further by the character encoding used, i.e. to read like ‘de_DE.UTF-8’. Trac uses UTF-8 encoding internally, so there is not much to tell about that. 'C' is a special locale code since it disables all translations and programs use English texts as required by POSIX standard.^[#a3 3]^ {{{#!comment Both searches take the locale setting as a second request argument. Valid settings are a combination of language and country code, often extended further by the character encoding used, i.e. to read like ‘de_DE.UTF-8’. The encoding is of special relevance for languages that had an older encoding per default that was not sufficient for all common chars used by native speakers of that language. 'C' is a special locale code since it disables all translations and programs use English texts as required by POSIX standard. Character encoding is highly dependent on the underlying operation system then.^[#a3 3]^ // I'm not sure to what the above refers. Which search? What locale argument? I don't think the character encoding plays any role here (we deal with unicode internally, catalogs themselves are always encoded in UTF-8) - cboos Thanks for the hint on non-relevance for Trac since this has that uniform encoding. So I hope the current text is better. This might be deleted than. }}} First matching translation will replace the default text what by gettext convention is the same as the msgid, that is used, if all attempts fail to find an exact matching translation. == Required workflow == A walk-through... === Prepare plugin code === ==== Import i18n/l10n helper programs ==== Pick a reasonably unique name for the domain, e.g. ** 'foo' ** This will be the basename for the various translation catalog files (e.g. `/locale/fr/LC_MESSAGES/foo.po` for the French catalog). At run-time, the translation functions (typically `_(...)`) have to know in which catalog the translation will be found. Specifying the 'foo' domain in every such call would be tedious, that's why there's a facility for creating partially instantiated domain-aware translation functions, `domain_functions`. This helper function should be called at module load time, like this: {{{#!python from trac.util.translation import domain_functions _, tag_, N_, add_domain = \ domain_functions('foo', ('_', 'tag_', 'N_', 'add_domain')) }}} The translation functions which can be bound to a domain are: - `'_'`: extract and translate - `'ngettext'`: extract and translate (singular, plural, num) - `'tgettext'`, `'tag_'`: same as `'_'` but for Markup - `'tngettext'`, `'tagn_'`: same as `'ngettext'` but for Markup - `'gettext'`: translate only, don't extract - `'N_'`: extract only, don't translate - `'add_domain'`: register the catalog file for the bound domain To inform Trac about where the plugin's message catalogs can be found, you'll have to call the `add_domain` function obtained via `domain_functions` as shown above. One place to do this is in the `__init__` function of your plugin's main component, like this: {{{#!python def __init__(self): import pkg_resources # here or with the other imports # bind the 'foo' catalog to the specified locale directory locale_dir = pkg_resources.resource_filename(__name__, 'locale') add_domain(self.env.path, locale_dir) }}} assuming that folder `locale` will reside in the same folder as the file containing the code above, referred to as `` (as observable inside the Python egg after packaging). The i18n/l10n helper programs are available inside the plugin now, but if the plugin code contains several python script files and you encounter text for translation in one of them too, you need to import the functions from the main script, say its name is `api.py`, there: {{{#!python from api import _, tag_, N_ }}} ==== Preset configuration for i18n/l10n helper programs ==== Add some lines to `setup.cfg` or, if it doesn't exist by now, create it with the following content: {{{ [extract_messages] add_comments = TRANSLATOR: msgid_bugs_address = output_file = /locale/messages.pot # Note: specify as 'keywords' the functions for which the messages # should be extracted. This should match the list of functions # that you've listed in the `domain_functions()` call above. keywords = _ N_ tag_ # Other example: #keywords = _ ngettext:1,2 N_ tag_ width = 72 [init_catalog] input_file = /locale/messages.pot output_dir = /locale domain = foo [compile_catalog] directory = /locale domain = foo [update_catalog] input_file = /locale/messages.pot output_dir = /locale domain = foo }}} Replace `` as appropriate (i.e. the relative path to the folder containing the `locale` directory, for example `mytracplugin`). This will tell the i18n/l10n helper programs where to look for and store message catalog files. {{{#!comment resume review here... }}} In the `extract_messages` section there is just one more lines you may like to change: `msgid_bugs_address`. To allow for direct feedback regarding your i18n work add a valid e-mail address or a mailing list dedicated to translation issues there. The `add_comments` line simply lists the tags in the comments surrounding the calls to the translation functions in the source code that have to be propagated to the catalogs (see [Babel:wiki:Documentation/0.9/setup.html#extract-messages extract_messages] in Babel's documentation). So you will want to leave that one untouched. ==== Mark text for extraction ==== In python scripts you'll have to wrap text with the translation function `_()` to get it handled by translation helper programs. Some code, that was {{{ msg = 'This is a msg text.' }}} before, will read like {{{ msg = _("This is a msg text.") }}} afterwards. Note, that quoting of (i18n) message texts should really be done in double quotes. Single quotes are reserved for string constants (see commit note for r9751). This is a somewhat time consuming task depending on the size of the plugin's code. If you initially fail to find all desired texts you may notice this by missing them from the message catalog later and come back to this step again. If the plugin maintainer is unaware of your i18n work or unwilling to support it and he adds more message without the translation function call, remember that you have to do the wrapping of these new texts too. ==== Text extraction from Genshi templates ==== Message extraction for Genshi templates should be done auto-magically. However there is the markup `i18n:msg` available to ensure extraction even from less common tags. For a real-world example have a look at [changeset:9542/trunk/trac/ticket/templates/ Trac SVN changeset r9542] for marking previously undetected text in templates. See Genshi documentation on this topic, [http://genshi.edgewall.org/wiki/Documentation/0.6.x/i18n.html Internationalization and Localization]. //But there are not all issues settled with some special cases for message extraction from Genshi templates, i.e. see [g:ticket:385 Genshi ticket 385]. You should search for similar issues you may encounter while trying to handle plugin templates. ===== Extraction Babel only does extract from Python scripts by default. To extract messages from Genshi templates as well, you'll have to declare the needed extractors in `setup.py`: {{{#!diff diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -34,6 +35,21 @@ from setuptools import find_packages, setup +extra = {} +from trac.util.dist import get_l10n_cmdclass +cmdclass = get_l10n_cmdclass() +if cmdclass: # Yay, Babel is there, we've got something to do! + extra['cmdclass'] = cmdclass + extractors = [ + ('**.py', 'python', None), + ('**/templates/**.html', 'genshi', None), + ('**/templates/**.txt', 'genshi', { + 'template_class': 'genshi.template:TextTemplate' + }), + ] + extra['message_extractors'] = { + 'foo': extractors, + } +except ImportError: + pass + setup( name = 'foo', version = '0.12', @@ -53,6 +69,7 @@ 'templates/*.txt', 'htdocs/*.*', 'htdocs/css/*.*', + 'locale/*/LC_MESSAGES/*.mo', ] }, install_requires = [ @@ -96,4 +113,5 @@ ] }, test_suite = '.tests', + **extra ) }}} ==== i18n for Java scripts ==== {{{#!comment This is taken largely from studying code in http://trac-hacks.org/wiki/TracTicketTemplatePlugin. Some conclusions may be plain wrong, since I had not time to carefully investigate and verify this. Indeed, I suspect the final version of this section will be quite different... }}} Note 1: This section had some preliminary content. Please handle with care and contribute your/any better knowledge on the subject. And look at #6353 please, since it seems to prepare a official solution for Trac and Trac plugins reducing the code changes to almost the same level as for file types mentioned before.[[BR]] Note 2: Dedicated helper functions were introduced to Trac trunk recently (see r9758, '''r9763''' and r9764). Following content needs to be revised to make use of this new native support for Trac plugins as well. ==== Register message catalog files for packaging ==== To include the translated messages into the packaged plugin you need to add the path to the catalog files to `package_data` in the call for function `setup()` in `setup.py`: {{{#!diff diff -u a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -39,6 +39,7 @@ package_data = { : [ 'htdocs/css/*.css', + 'locale/*/LC_MESSAGES/*.mo', ], }, entry_points = { }}} ==== Announce new plugin version ==== The plugin will not work with any Trac version before 0.12dev, since import of the translation helper functions introduced for 0.12 will fail. It is possible to wrap the import with a '`try:`' and define dummy functions in a corresponding '`except ImportError:`' to allow the plugin to work with older versions of Trac, but there might already be a different version for 0.11 and 0.12, so this is not required in most cases. If it is strictly required for your plugin, have a look at `setup.py` of the [browser:plugins/0.12/mercurial-plugin/setup.py?rev=9133 Mercurial plugin] provided with Trac. In all other cases you'll just add a line like the following as another argument to the setup() function in plugin's `setup.py`: {{{ install_requires = ['Trac >= 0.12dev'], }}} All the work you did by now will go unnoticed, at least with regard to package naming. To help with identification of the new revision you should bump the plugin's version. This is done by changing the version/revision, typically in `setup.cfg` or `setup.py`. And you may wish to leave a note regarding your i18n work along the copyright notices as well. ==== Summing it up ==== Here's an example of the changes required to add i18n support to a plugin: [http://github.com/cboos/trachacks-hudsontracplugin/commit/1669cefa806614e62bd9ce5573750e30fa060408 HudsonTrac 0.12 branch]. Note: - it uses the old style for the `domain_functions` call. While this is still supported, new code should use the new style shown above, `domain_functions(domain, functions)` - it uses: `/locale/*.*, /locale/*/LC_MESSAGES/*.*` for the `package_data`. You should rather use only `/locale/*/LC_MESSAGES/*.*` (see r9638 and r9640) You'll find another example attached to this page. That is [th:wiki:SubticketsPlugin Sub-Tickets plugin] v0.1.0 and a [attachment:trac-subtickets-plugin_i18n-l10n.patch diff] containing all i18n/l10n related work to produce a German translation based on that source. === Do translators work === General advice from [wiki:TracL10N TracL10N] on making good translation for Trac applies here too. I.e. it's desirable to maintain a consistent wording across Trac and Trac plugins. Since this is going beyond the scope of aforementioned [wiki:TracL10N TracL10N], there might be the need for more coordination. Consider joining the [th:wiki:TracPluginTranslation Trac plugin l10n project], that utilizes [http://www.transifex.net/projects/p/Trac_Plugin-L10N/ Transifex] for uniform access to message catalogs for multiple plugins backed by a dedicated (Mercurial) [http://bitbucket.org/hasienda/trac_plugins-l10n message catalog repository] at Bitbucket.org. Trac has some language teams at Transifex as well, so this is a good chance for tight translator cooperation. For those who read previous parts, you do notice that we switch from talking about i18n to 'l10n' now, don't you? No source code mangling. All code below is no meant to become part of the plugin source but meant to be put to the command line. Switch to root directory of plugin's source, i.e.: {{{ cd /usr/src/trac_plugins/foo }}} Extract the messages that where marked for translation before, or on case of Genshi templates are exposed by other means: {{{ python ./setup.py extract_messages }}} The attentive reader will notice that the argument to `setup.py` has the same wording as a section in `setup.cfg`, that is not incidental. And this does apply to the following command lines as well. If you attempt to do improvements on existing message catalogs you'll update the one for your desired language: {{{ python ./setup.py update_catalog -l de_DE }}} If you omit the language selection argument `-l` and identifier string, existing catalogs of all languages will be updated, what is acceptably fast (just seconds) on current hardware. But if you happen to do all the i18n work before, the you know you there's nothing to update right now. Well, so now it's time to create the message catalog for your desired language: {{{ python ./setup.py init_catalog -l de_DE }}} As you may guess, there is not much to be done, if the helper programs don't know what language you'd like to work on, so the language selection argument `-l` and identifier string are mandatory here. Now fire up the editor of your choice. There are dedicated message catalog (.po) file editors that ensure for quick results as a beginner as well as make working on large message catalogs with few untranslated texts or translations marked 'fuzzy' much more convenient. See dedicated resources for details on choosing an editor program as well as for help on editing .po files.^[#a4 4], [#a5 5]^ If not already taken care for by your (PO) editor, the place to announce yourself as the last translator is after the default `TRANSLATOR:` label at top of the message catalog file. === Compile and use it === Compile the `messages.po` catalog file with your translations into a machine readable `messages.mo` file. {{{ python ./setup.py compile_catalog -f -l de_DE }}} The argument `-f` is needed to include even the msgid's marked 'fuzzy'. If you have prepared only one translated catalog the final language selection argument `-l` and identifier string are superfluous. But as soon as there are several other translations that you don't care, it will help to select just your work for compilation. Now you've used all four configuration sections in `setup.cfg`, that are dedicated to i18n/l10n helper programs. You could finish your work by packaging the plugin. Make the python egg as usual: {{{ python ./setup.py bdist_egg }}} Install the new egg and restart your web-server after you made sure to purge any former version of that plugin (without your latest work). === Advanced stuff === ==== About 'true' l10n ==== A human translator will/should do better than an automatic translation, since good l10n has more of a transcription than pure translation word by word. It's encouraging to see the raise of native words for such terms like changeset, ticket and repository in different languages. This will help Trac to not only fulfill its promise to help project teams focusing on their work but even extend its use to project management in general, where use of native language is much more common or even required in contrast to the traditional software development. Details on that matter tend to become religious, so let's stop here. == Related resources == See TracL10N and more specifically TracL10N#ForDevelopers, which contains general tips that are also valid for plugin translation. [=#a1 ^1^] http://en.wikipedia.org/wiki/Internationalization_and_localization - Internationalization and localization[[BR]] [=#a2 ^2^] http://en.wikipedia.org/w/index.php?title=Multilingualism§ion=18 - Multilingualism in computing[[BR]] [=#a3 ^3^] http://www.gnu.org/software/gettext/manual/gettext.html#Locale-Names - GNU 'gettext' utilities: Locale Names[[BR]] [=#a4 ^4^] http://www.gnu.org/software/gettext/manual/gettext.html#Editing - GNU 'gettext' utilities: Editing PO Files[[BR]] [=#a5 ^5^] http://techbase.kde.org/Localization/Concepts/PO_Odyssey - PO Odyssey in '!Localization/Concepts' section of KDE !TechBase