Release Trac 1.0.19

[Ryan J Ollos]

This ticket is used to coordinate the finalization and testing of the next stable version of Trac, 1.0.19.

The changes will be documented on 1.0/TracChangeLog.

[Ryan J Ollos]

I've closed out next-stable-1.0.x. We should aim for minimal changes on 1.0-stable, and tickets can be assigned to the next releases, currently 1.0.19 and 1.0.20.

[Jun Omae]

In log:jomae.git@t13183-tx, translations of all languages from Transifex are merged.

The merging task was performed after minor changing of contrib/merge_catalog.py and adding the following task to Makefile.cfg:

tx-merge-%:
	@$(foreach catalog,$(catalogs), \
	    touch $(catalog.po); \
	    mv $(catalog.po) $(catalog.po).orig; \
	)
	@tx pull --force --no-interactive -l $(*)
	@$(foreach catalog,$(catalogs), \
	    mv $(catalog.po) $(catalog.po).tx; \
	    mv $(catalog.po).orig $(catalog.po); \
	    python contrib/merge_catalog.py $(catalog) $(catalog.po).tx $(*); \
	    rm $(catalog.po).tx; \
	)

[Ryan J Ollos]

Replying to Jun Omae:

In log:jomae.git@t13183-tx, translations of all languages from Transifex are merged.

Thanks! I've long been wanting to make this part of the release process. I'd like to have a concise list of steps documented in TracDev/ReleaseChecklist.

The merging task was performed after minor changing of contrib/merge_catalog.py and adding the following task to Makefile.cfg:

Do you think the task should go in Makefile? It might also be useful to have a task tx-merge that operates for all languages.

Is the final step to push the catalogs back to Transifex?

Should update_catalog (make update) be a step we do for all translations after running tx-merge, before pushing back to Transifex?

Btw, I've been meaning to ask, after a make extraction, should I always run make update-en? It looks like that's what you've been doing.

After incorporating this into our release process, I'll post to trac-users to ask more users to get involved in translating on Transifex.

[Ryan J Ollos]

I'll make the release on Sunday 08-Sept if all the changes are ready.

[Jun Omae]

Replying to Ryan J Ollos:

Replying to Jun Omae:

In log:jomae.git@t13183-tx, translations of all languages from Transifex are merged.

Thanks! I've long been wanting to make this part of the release process. I'd like to have a concise list of steps documented in TracDev/ReleaseChecklist.

The merging task was performed after minor changing of contrib/merge_catalog.py and adding the following task to Makefile.cfg:

Do you think the task should go in Makefile? It might also be useful to have a task tx-merge that operates for all languages.

Agreed. Added tx-merge-% and tx-merge tasks in [27ee3eef1/jomae.git]. However, it would be good to specify locale=... parameter to tx-merge task to prevent merging for locales which have a committer (e.g. one of those locales is ja locale…). See wiki:TracL10N/Transifex#Translationteamcoordinator.

Is the final step to push the catalogs back to Transifex?

Yes. The catalogs on Transifex probably are not automatically updated even if catalogs in our repository are updated, excepted *.pot files.

Should update_catalog (make update) be a step we do for all translations after running tx-merge, before pushing back to Transifex?

Maybe, yes. However, it might be good to prevent marking as fuzzy to catalogs which have no fuzzy marker.

Btw, I've been meaning to ask, after a make extraction, should I always run make update-en? It looks like that's what you've been doing.

Currently, en_US catalogs are compiled, converted messages javascript and installed using pip install and setup.py install. Then, I consider the catalogs should have the same entries of *.pot files.

$ unzip -xp /dev/shm/Trac-1.0.18-py2-none-any.whl trac/locale/en_US/LC_MESSAGES/messages.mo | msgunfmt - | head -20
msgid ""
msgstr ""
"Project-Id-Version: Trac 0.12\n"
"Report-Msgid-Bugs-To: trac-dev@googlegroups.com\n"
"POT-Creation-Date: 2018-04-24 17:05+0000\n"
"PO-Revision-Date: 2010-07-19 23:05+0200\n"
"Last-Translator: Jeroen Ruigrok van der Werven <asmodai@in-nomine.org>\n"
"Language: en_US\n"
"Language-Team: en_US <trac-dev@googlegroups.com>\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.7.0\n"

msgid "%(items)s added"
msgid_plural "%(items)s added"
msgstr[0] "%(items)s added"
msgstr[1] "%(items)s added"

[Jun Omae]

Replying to Jun Omae:

The merging task was performed after minor changing of contrib/merge_catalog.py and adding the following task to Makefile.cfg:

BTW, I uses the following tasks on Makefile.cfg, which compares a original catalog with a modified catalog after sort messages and remove location comments by filtering with msgcat --no-wrap --no-location -s when make extraction update-{locale} is invoked.

ifndef vc
ifneq "$(wildcard $(CURDIR)/.svn)" ""
    vc := svn
else ifneq "$(wildcard $(CURDIR)/.git)" ""
    vc := git
else ifneq "$(wildcard $(CURDIR)/.hg)" ""
    vc := hg
endif
endif

define vc_cat-svn
    $(vc) cat -rBASE $(1)
endef
define vc_cat-git
    $(vc) show HEAD:$(1)
endef
define vc_cat-hg
    $(vc) cat -r. $(1)
endef
define vc_cat
    $(call vc_cat-$(vc),$(1))
endef

xdiff-pot:
        @$(foreach catalog,$(catalogs), \
            $(call vc_cat,$(catalog.pot)) \
                | msgcat --no-wrap --no-location -s -i - >$(catalog.pot).tmp; \
            msgcat --no-wrap --no-location -s -i - <$(catalog.pot) \
                | diff -u -L $(catalog.pot) -L $(catalog.pot) $(catalog.pot).tmp - || :; \
            rm -- $(catalog.pot).tmp; \
        )

xdiff-%:
        @$(foreach catalog,$(catalogs), \
            $(call vc_cat,$(catalog.po)) \
                | msgcat --no-wrap --no-location -s -i - >$(catalog.po).tmp; \
            msgcat --no-wrap --no-location -s -i - <$(catalog.po) \
                | diff -u -L $(catalog.po) -L $(catalog.po) $(catalog.po).tmp - || :; \
            rm -- $(catalog.po).tmp; \
        )

update-en_US-0:
        sed -i -n '0,/^"Generated-By: /p' trac/locale/en_US/LC_MESSAGES/*.po

update-en_US :: update-en_US-0

[Ryan J Ollos]

Replying to Jun Omae:

Agreed. Added tx-merge-% and tx-merge tasks in [27ee3eef1/jomae.git]. However, it would be good to specify locale=... parameter to tx-merge task to prevent merging for locales which have a committer (e.g. one of those locales is ja locale…). See wiki:TracL10N/Transifex#Translationteamcoordinator.

ja is the only locale for which I've seen any activity for years. Scanning trunk/trac/locale, the most recent update I see was 2015 r14343 (aside from a minor change to ko catalog by Christian). If I'm doing the step I could just skip ja by reverting those files after merging all catalogs.

Two other possibilities:

1. Merge and update all catalogs before release, and post them for review. However, are the catalogs overwritten on Transifex when doing a push? If that's the case, it seems like we want a short interval between pull and push, to minimize the possibility of overwriting.
2. Would you be willing to do merge and update all of the catalogs before each release? I'm expecting it's probably not very time consuming, but would be better to have someone familiar with the translation tools do the merge (rather than me, at least for the near term).

Should update_catalog (make update) be a step we do for all translations after running tx-merge, before pushing back to Transifex?

Maybe, yes. However, it might be good to prevent marking as fuzzy to catalogs which have no fuzzy marker.

To prevent catalogs being marked fuzzy, do we just remove the marker from the header like you've done in the merge script (target.fuzzy = False)? Would it make sense to have make update set fuzzy to False?

[Ryan J Ollos]

Replying to Jun Omae:

However, it would be good to specify locale=... parameter to tx-merge task to prevent merging for locales which have a committer (e.g. one of those locales is ja locale…). See wiki:TracL10N/Transifex#Translationteamcoordinator.

We should contact all coordinators, see who is still active, and update the list. I will put that on my todo list.

[Ryan J Ollos]

Replying to Jun Omae:

BTW, I uses the following tasks on Makefile.cfg, which compares a original catalog with a modified catalog after sort messages and remove location comments by filtering with msgcat --no-wrap --no-location -s when make extraction update-{locale} is invoked.

xdiff-pot and xdiff-% are very useful for generating accurate added/removed counts for commit log messages. Can we add those to Makefile?

I've experimented with update-en_US and it looks like it reverts the po file header to that of the pot file and removes strings that are no longer used (which get commented out by update), but I don't understand how that sed expression works.

[Jun Omae]

Replying to Ryan J Ollos:

Two other possibilities:

1. Merge and update all catalogs before release, and post them for review. However, are the catalogs overwritten on Transifex when doing a push? If that's the case, it seems like we want a short interval between pull and push, to minimize the possibility of overwriting.

Currently, translations ratios in the charts on milestone page are incorrect due to no pushes. However, I think no need to push catalogs to Transifex. The pushing is just only to refresh statistics charts.

The statistics charts should be correctly generated from *.pot and *.po files in source tree rather than catalogs on Transifex.

2. Would you be willing to do merge and update all of the catalogs before each release? I'm expecting it's probably not very time consuming, but would be better to have someone familiar with the translation tools do the merge (rather than me, at least for the near term).

Ok. I will merge the translations from Transifex on each release.

Should update_catalog (make update) be a step we do for all translations after running tx-merge, before pushing back to Transifex?

Maybe, yes. However, it might be good to prevent marking as fuzzy to catalogs which have no fuzzy marker.

To prevent catalogs being marked fuzzy, do we just remove the marker from the header like you've done in the merge script (target.fuzzy = False)? Would it make sense to have make update set fuzzy to False?

Ah, I misunderstood that *.po file is not compiled if the *.po file has any of msgid with fuzzy. Then, we should do make update after make tx-merge.

[Jun Omae]

Replying to Ryan J Ollos:

Replying to Jun Omae:

BTW, I uses the following tasks on Makefile.cfg, which compares a original catalog with a modified catalog after sort messages and remove location comments by filtering with msgcat --no-wrap --no-location -s when make extraction update-{locale} is invoked.

xdiff-pot and xdiff-% are very useful for generating accurate added/removed counts for commit log messages. Can we add those to Makefile?

Yeah. Added the tasks in [4abcf27f2/jomae.git]. The changes keep diff and diff-% tasks.

I've experimented with update-en_US and it looks like it reverts the po file header to that of the pot file and removes strings that are no longer used (which get commented out by update), but I don't understand how that sed expression works.

The task is invoked by the following steps:

1. update-en_US-0 task is invoked before update-en_US task by declare of update-en_US :: update-en_US-0.
2. update-en_US-0 task, the sed command removes all msgids and msgstrs from en_US *.po files, except po header.
3. update-en_US task merges all msgids from *.pot files to the *.po files.

update-% task comments out no longer used msgid and msgstr with #~, however the comments are not needed. Removed all msgs using sed before the updates.

[Ryan J Ollos]

Replying to Jun Omae:

Currently, translations ratios in the charts on milestone page are incorrect due to no pushes. However, I think no need to push catalogs to Transifex. The pushing is just only to refresh statistics charts.

If catalogs are updated before the push to Transifex, would that save some effort for Transifex committers? Also, the original intention for adding merge_catalog was to merge messages between Trac branches (e.g. 1.0-stable ↔ 1.2-stable). If that was part of the workflow, then it seems like it would be useful to push to those changes Transifex.

The statistics charts should be correctly generated from *.pot and *.po files in source tree rather than catalogs on Transifex.

If you are referring to these charts (wiki:TracL10N#Statistics) I'm not sure how we will create them from source tree catalogs since the charts are generated by Transifex.

[Jun Omae]

Replying to Ryan J Ollos:

1. Merge and update all catalogs before release, and post them for review. However, are the catalogs overwritten on Transifex when doing a push? If that's the case, it seems like we want a short interval between pull and push, to minimize the possibility of overwriting.

We could lock resources to prevent modifications from translators between merging and pushing while release procedure. See ​https://docs.transifex.com/projects/preventing-resource-edits#locking-a-resource.

[Ryan J Ollos]

Changes look good to me. If you have time to push the changes, I could make the release on Sunday 09/21.

[Jun Omae]

Thanks. The changes have been pushed.

• contrib/merge_catalog.py: commited in [17184], merged in [17185-17187]
• Makefile: commited in [17188], merged in [17189-17191]
• trac/locale/**/*.po: commited in [17192]

I re-merged from Transifex before the pushing and no merged messages.

[Ryan J Ollos]

It would probably be useful to add make help-l10n documentation for xdiff, xdiff-pot, xdiff-xy.

[Ryan J Ollos]

Replying to Ryan J Ollos:

It would probably be useful to add make help-l10n documentation for xdiff, xdiff-pot, xdiff-xy.

Proposed change:

Makefile

@@ -145 +145 @@
 
   update              update all the catalog files from the templates
   update-xy           update the catalogs for the xy locale only
+  [updateopts=...]    variable containing extra options for update (e.g. -N)
 
   compile             compile all the catalog files
   compile-xy          compile the catalogs for the xy locale only
@@ -163 +164 @@
   tx-merge            merge catalogs from Transifex for all locales
   tx-merge-xy         merge catalogs from Transifex for the xy locale only
 
-  diff                show relevant changes after an update for all catalogs
-  diff-xy             show relevant changes after an update for the xy locale
+  xdiff               show changes to message strings for all catalogs
+  xdiff-pot           show changes to message strings in the catalog template
+  xdiff-xy            show changes to message strings for the xy locale
+
+  diff                show summarized changes after an update for all catalogs
+  diff-xy             show summarized changes after an update for the xy locale
   [vc=...]            variable containing the version control command to use
 
   [locale=...]        variable for selecting a set of locales
 
-  [updateopts=...]    variable containing extra options for update (e.g. -N)
-
 endef
 export HELP_l10n

[Ryan J Ollos]

comment:18 changes committed in r17204, merged in [17205:17207].

I will try to get the release done tomorrow.