Deploy to PyPI from TravisCI

[Ryan J Ollos]

To speed creation of releases we can have the packages generated and ​deployed to PyPI on TravisCI.

In another ticket we'll need to investigate if the same can be done on AppVeyor.

We can then pull the packages from PyPI to the edgewall FTP site.

[Ryan J Ollos]

Another idea: we could run ​pypiserver on edgewall and deploy on every release. I wanted to deploy to testpypi on every release, but it won't work because packages cannot be overwritten.

[Ryan J Ollos]

Proposed changes for TravisCI in [4ec6d476c/rjollos.git] (minus the change to setup.cfg). It seems to be working correctly to test and deploy to ​testpypi. See ​build 181 and ​1.0.15dev4 on testpypi.

In theory we can just delete the server line, change the repository line, regenerate the encrypted password using the switch (-r edgewall/trac) and it will work on the next build that is tagged. A really frustrating issue I encountered was in encrypting the password. I was working from a clone with the origin,

origin	https://svn.edgewall.org/git/trac/mirror (fetch)
origin	https://svn.edgewall.org/git/trac/devs/rjollos (push)

but pushing to my fork on github. For that case, it's necessary to ​specify the repository:

$ travis encrypt <password> -r rjollos/trac

The documentation describes using travis encrypt --add deploy.password, but it annoyingly reformats the entire .travis.yml file.

Another tip I read: jobs is an alias for matrix, but both cannot be present in the file. That's why the matrix statement was merged with jobs. ​Build stages is a new feature. It would be nice if there were more recipes for Travis CI, I find it rather hard to work with.

Proposed next steps:

1. Commit the changes to 1.0-stable, keeping the server as testpypi, and just confirm that the deploy works correctly for release 1.0.15. If so, change server to point to pypi and merge to 1.2-stable and trunk.
2. Setup a pypiserver so we can have test packages on every build
3. It looks like we could ​deploy to a server, which would put the downloads on the Edgewall server, but another possibility could be to just run a script on the Edgewall server to pull from PyPI.
4. (If possible) Add a deploy to pypi steps to AppVeyor builds.

In retrospect, I should have started with trying to do the equivalent with AppVeyor since I work from OSX and generating builds for Windows is more of a pain-point for me.

[Ryan J Ollos]

Side note: travis lint is a useful tool to run after modifying .travis.yml, but it gives a few incorrect warnings:

$ travis lint
Warnings for .travis.yml:
[x] unexpected key jobs, dropping
[x] value for addons section is empty, dropping
[x] in addons section: unexpected key apt, dropping

There's an open ticket for the ​apt issue. Not recognizing the jobs key is a known issue, discussed ​here (there's a ​reply to that comment from one of the Travis CI devs).

[Ryan J Ollos]

Changes committed to 1.0-stable in r16032.

[Ryan J Ollos]

Looks like tags aren't getting pushed to GitHub.

[Ryan J Ollos]

Travis jobs are running on Ubuntu Precise (12.04).

hostname: i-0ed83f0-precise-production-2-worker-org-docker.travisci.net:064d8ef5-d124-48b6-bff5-0c3e7781cfd5

I did an experiment, running tests for trunk instead on Ubuntu ​Trusty (14.04): ​build 184.

• Trusty is in ​public beta.
• ​python-subversion is not yet available in packages, although the logs show that it gets installed. SVN tests are skipped with (no svn bindings).
• There's one failure:

  ERROR: runTest (trac.db.tests.functional.DatabaseBackupTestCase)
  
  Testing backup
  
  Traceback (most recent call last):
    File "/home/travis/build/rjollos/trac/trac/db/tests/functional.py", line 26, in runTest
      backup_file = env.backup()
    File "/home/travis/build/rjollos/trac/trac/env.py", line 716, in backup
      return DatabaseManager(self).backup(dest)
    File "/home/travis/build/rjollos/trac/trac/db/api.py", line 572, in backup
      return connector.backup(dest)
    File "/home/travis/build/rjollos/trac/trac/db/mysql_backend.py", line 280, in backup
      msg=to_unicode(errmsg.strip())))
  TracError: mysqldump failed: mysqldump: Got error: 1045: Access denied for user 'tracuser'@'localhost' (using password: NO) when trying to connect
  ----------------------------------------------------------------------
  Ran 205 tests in 357.456s
  FAILED (errors=1)
  make: *** [functional-test] Error 1
  

MySQL tests on Precise:

  Package        Version
  ------------------------------------------------------
  Python       : 2.7.3 (default, Dec 18 2014, 19:10:20) 
               : [GCC 4.6.3]
  Setuptools   : 12.0.5
  Jinja2       : 2.9.6
  Genshi       : 0.7 (with speedups)
  Babel        : 2.4.0
  sqlite3      : 2.6.0 (3.7.9)
  PySqlite     : not installed
  PyMySQL      : 0.7.11.None
  Psycopg2     : not installed
  SVN bindings : 1.6.17 (r1128011)
  Mercurial    : 2.0.2
  Pygments     : 2.2.0
  Textile      : 2.3.12
  Pytz         : 2017.2
  Docutils     : 0.13.1
  Twill        : 0.9
  LXML         : 3.8.0
  coverage     : not installed
  figleaf      : not installed

MySQL tests on Trusty:

  Package        Version
  ------------------------------------------------------
  Python       : 2.7.6 (default, Oct 26 2016, 20:30:19) 
               : [GCC 4.8.4]
  Setuptools   : 36.0.1
  Jinja2       : 2.9.6
  Genshi       : 0.7 (with speedups)
  Babel        : 2.4.0
  sqlite3      : 2.6.0 (3.8.2)
  PySqlite     : not installed
  PyMySQL      : 0.7.11.None
  Psycopg2     : not installed
  SVN bindings : not installed
  Mercurial    : not installed
  Pygments     : 2.2.0
  Textile      : 2.3.12
  Pytz         : 2017.2
  Docutils     : 0.13.1
  Twill        : 0.9
  LXML         : 3.8.0
  coverage     : not installed
  figleaf      : not installed

Subversion 1.8.8 is available on Precise.

If we can fix the issue, it might make sense to move the trunk to Trusty since it's closer to an environment that we support for the trunk: TracDev/ApiChanges/1.3. MySQL is 5.6.33 on Trusty, 5.5.41 on Precise.

We might want to echo MySQL and PostgreSQL versions in make status.

[Ryan J Ollos]

Testing deploy step in r16089 (removed tag in r16090). Hit an error:

HTTPError: 410 Client Error: Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at ​https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see ​https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see ​https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html) for url: ​https://testpypi.python.org/pypi

I'll test again after making modifications in my fork.

[Jun Omae]

The following settings is in .git/config on my working directory.

[remote "mirror"]
        url = http://svn.edgewall.org/git/trac/mirror
        fetch = +refs/heads/*:refs/remotes/mirror/*
        fetch = refs/remotes/tags/*:refs/tags/*
        fetch = refs/remotes/0.5-stable:refs/remotes/mirror/0.5-stable
        fetch = refs/remotes/0.6-stable:refs/remotes/mirror/0.6-stable
        fetch = refs/remotes/0.7-stable:refs/remotes/mirror/0.7-stable
        fetch = refs/remotes/0.8-stable:refs/remotes/mirror/0.8-stable
        fetch = refs/remotes/0.9-stable:refs/remotes/mirror/0.9-stable
        fetch = refs/remotes/0.10-stable:refs/remotes/mirror/0.10-stable
        fetch = refs/remotes/0.11-stable:refs/remotes/mirror/0.11-stable
        pushurl = no_push

However, git fetch mirror fails.

$ git fetch mirror
fatal: Cannot fetch both refs/remotes/tags/trac-1.0.16 and refs/tags/trac-1.0.16 to refs/tags/trac-1.0.16

It seems that trac-1.0.16 tags still exist in mirror repository. Could you please remove the tags from mirror repository?

$ git tag -l | grep -F 1.0.16
$ git ls-remote mirror | grep -F 1.0.16
9b803ccb48c43e216e283e1d4b2fed8dc35ba82c        refs/remotes/tags/trac-1.0.16
9b803ccb48c43e216e283e1d4b2fed8dc35ba82c        refs/tags/trac-1.0.16

[Jun Omae]

Mercurial mirror repository also has trac-1.0.16 tag in changeset:ab2dc11b8574/mirror.

[Ryan J Ollos]

Updated pypitest URL in r16091.

1. Deleted tag in svn repository (r16093)
2. Temporarily disabled pre-receive hook in Git mirror repository
3. Deleted tag:

   $ git push mirror :trac-1.0.16
   To https://svn.edgewall.org/git/trac/mirror
    - [deleted]             trac-1.0.16
   $ git push mirror :trac-1.0.16
   error: unable to delete 'trac-1.0.16': remote ref does not exist
   error: failed to push some refs to 'https://svn.edgewall.org/git/trac/mirror'
   

4. After pushing r16095 the tag has reappeared in the mirror repository

Am I missing something?

I didn't expect the tag to persist for more than a few hours, however it would have been a better idea to use trac-1.0.16dev as the tag name.

[Ryan J Ollos]

Latest attempt also failed:

Uploading distributions to ​https://test.pypi.org/legacy
Uploading Trac-1.0.16.dev0-py2-none-any.whl
RedirectDetected: "​https://test.pypi.org/legacy" attempted to redirect to "​https://test.pypi.org/legacy/" during upload. Aborting…

I'll make the minor adjustment and retry.

[Ryan J Ollos]

Finally, it seems to have worked:

Uploading distributions to ​https://test.pypi.org/legacy/
Uploading Trac-1.0.16.dev0-py2-none-any.whl
Uploading Trac-1.0.16.dev0.tar.gz

Releases are available at ​https://testpypi.python.org/pypi/Trac/1.0.16.dev0.

I'll try to cleanup the tags now.

[Ryan J Ollos]

I found another issue. The tar package has the locale dir, but it's missing from the wheel package.

[Jun Omae]

git-svn creates a remote branch in mirror Git repository for a tag in source Subversion repository. We create tags in mirror Git repository from the remote branches using fetch = refs/remotes/tags/*:refs/tags/*.

$ git ls-remote http://svn.edgewall.org/git/trac/mirror | grep -F 1.0.16
1dcb62395de01c86d749295dc5e60ec4fa205e0a        refs/remotes/tags/trac-1.0.16

We could remove remote branches using git branch -r -d NAME....

$ git branch -a | grep -F 1.0.16
  remotes/tags/trac-1.0.16
$ git branch -r -d tags/trac-1.0.16
Deleted remote branch tags/trac-1.0.16 (was 4afdfd60f).

In addition, it is needed to remove trac-1.0.16 from ​https://github.com/edgewall/trac.

[Ryan J Ollos]

I removed tags in Git repository and GitHub. However, it seems like I followed the same steps as previously so I'll look at whether they re-appear on the next commit.

I tried removing tags from mirror, but was unsuccessful: [12657/mirror].

$ hg tag --remove trac-1.0.16
$ hg push

[Ryan J Ollos]

Replying to Ryan J Ollos:

If we can fix the issue, it might make sense to move the trunk to Trusty since it's closer to an environment that we support for the trunk: TracDev/ApiChanges/1.3. MySQL is 5.6.33 on Trusty, 5.5.41 on Precise.

Trusty will ​soon be the default.

[Ryan J Ollos]

Additional configuration changes in r16122 for deploying to the Edgewall PyPI server. See ​build 256289210.

[Ryan J Ollos]

Replying to Ryan J Ollos:

I tried removing tags from mirror, but was unsuccessful: [12657/mirror].

$ hg tag --remove trac-1.0.16
$ hg push

Discussed elsewhere, the best choice at this point might be to just skip 1.0.16 and release 1.0.17.

comment:14 is the last item needing attention, then the TravisCI configuration can be merged to 1.2-stable and trunk.

[Ryan J Ollos]

Milestone renamed

[Ryan J Ollos]

Replying to Ryan J Ollos:

Discussed elsewhere, the best choice at this point might be to just skip 1.0.16 and release 1.0.17.

See comment:7:ticket:12839.

[Ryan J Ollos]

Removed skip_upload_docs in r16693 since true is ​now the default value.

[Ryan J Ollos]

Replying to Ryan J Ollos:

I found another issue. The tar package has the locale dir, but it's missing from the wheel package.

Comparing with a wheel built from my terminal, the message catalogs are compiled on Travis CI, but the .mo file are not copied to the build directory. The log on Travis CI is missing log messages like the following:

creating build/bdist.macosx-10.13-x86_64/wheel/trac/locale
creating build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl
creating build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl/LC_MESSAGES
copying build/lib/trac/locale/sl/LC_MESSAGES/tracini.mo -> build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl/LC_MESSAGES
copying build/lib/trac/locale/sl/LC_MESSAGES/messages.mo -> build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl/LC_MESSAGES

And the files are not added to the whl, missing log messages like the following:

adding 'trac/locale/sl/LC_MESSAGES/messages.mo'
adding 'trac/locale/sl/LC_MESSAGES/tracini.mo'

I have no idea of what the problem might be. The environment on Travis CI seems to be pretty much the same as my computer. The ​deploy script is fairly simple.

[Ryan J Ollos]

Oh, I think we may just need to run make compile. Testing the change …

[Ryan J Ollos]

comment:14 issue fixed in r16696. If deployment to test.pypi.org succeeds with the 1.0.17 release, we can change the configuration to deploy to pypi.org for release 1.0.18.

[Ryan J Ollos]

The 1.0.17 release was pushed to ​test.pypi. Needs testing.

[Ryan J Ollos]

Replying to Ryan J Ollos:

The 1.0.17 release was pushed to ​test.pypi. Needs testing.

I diff'ed the archives on test.pypi and pypi. The whl packages are the same. The sdist on pypi has a bunch of extra files that aren't needed, such as .po, THANKS, etc.

It looks like we should remove setuptools_subversion from the environment where packages are built. That package was in the environment when I created the 1.0.17 packages for uploading to pypi, but not in the Travis CI environment that created the packages for test.pypi.

[Ryan J Ollos]

Without setuptools_subversion in the environment in which the packages are built there is an error installing from the sdist package:

The sdist contains mo files, but no po files:

$ cd dist && tar xzvf Trac-1.3.3.tar.gz 2&> /dev/null
$ find Trac-1.3.3 -name "*.po" | head -4
$ find Trac-1.3.3 -name "*.mo" | head -4
Trac-1.3.3/trac/locale/sl/LC_MESSAGES/tracini.mo
Trac-1.3.3/trac/locale/sl/LC_MESSAGES/messages.mo
Trac-1.3.3/trac/locale/pl/LC_MESSAGES/tracini.mo
Trac-1.3.3/trac/locale/pl/LC_MESSAGES/messages.mo

This is what I'd expect, and the wheel similarly contains mo files and no po files.

Installing with pip from the sdist, a wheel package appears to be built and fails when trying to compile the message catalogs:

$ pip install dist/Trac-1.3.3.tar.gz
Processing ./dist/Trac-1.3.3.tar.gz
Requirement already satisfied: setuptools>=0.6 in /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/lib/python2.7/site-packages (from Trac==1.3.3) (40.0.0)
Requirement already satisfied: jinja2>=2.9.3 in /Users/rjollos/Documents/Workspace/trac-dev/trac-trunk/.eggs/Jinja2-2.10-py2.7.egg (from Trac==1.3.3) (2.10)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/rjollos/Documents/Workspace/trac-dev/trac-trunk/.eggs/MarkupSafe-1.0-py2.7-macosx-10.13-x86_64.egg (from jinja2>=2.9.3->Trac==1.3.3) (1.0)
Building wheels for collected packages: Trac
  Running setup.py bdist_wheel for Trac ... error
  Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-wheel-SoaYot --python-tag cp27:
  running bdist_wheel
  running build
  running compile_catalog_tracini
  error: no message catalogs found

  ----------------------------------------
  Failed building wheel for Trac
  Running setup.py clean for Trac
Failed to build Trac
Installing collected packages: Trac
  Running setup.py install for Trac ... error
    Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-SWD8dN/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac:
    running install
    running build
    running compile_catalog_tracini
    error: no message catalogs found

    ----------------------------------------
Command "/Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-SWD8dN/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac" failed with error code 1 in /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/

[Ryan J Ollos]

The sdist on test.pypi has the same problem as in comment:29:

$ pip install https://test-files.pythonhosted.org/packages/33/a6/a210399b48125f2af7fb23fb4bc93e4054cbe3b94e4ebf26aff483777f06/Trac-1.0.17.tar.gz
Collecting https://test-files.pythonhosted.org/packages/33/a6/a210399b48125f2af7fb23fb4bc93e4054cbe3b94e4ebf26aff483777f06/Trac-1.0.17.tar.gz
  Downloading https://test-files.pythonhosted.org/packages/33/a6/a210399b48125f2af7fb23fb4bc93e4054cbe3b94e4ebf26aff483777f06/Trac-1.0.17.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 626kB/s
Requirement already satisfied: setuptools>=0.6b1 in /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/lib/python2.7/site-packages (from Trac==1.0.17) (40.0.0)
Collecting Genshi>=0.6 (from Trac==1.0.17)
Building wheels for collected packages: Trac
  Running setup.py bdist_wheel for Trac ... error
  Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-wheel-cU1YjD --python-tag cp27:
  Genshi is needed by Trac setup, pre-installing
  running bdist_wheel
  running build
  running compile_catalog_tracini
  error: no message catalogs found

  ----------------------------------------
  Failed building wheel for Trac
  Running setup.py clean for Trac
Failed to build Trac
Installing collected packages: Genshi, Trac
  Running setup.py install for Trac ... error
    Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-ZLHuTc/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac:
    running install
    running build
    running compile_catalog_tracini
    error: no message catalogs found

    ----------------------------------------
Command "/Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-ZLHuTc/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac" failed with error code 1 in /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/

I would guess that some change to our packaging configuration is needed.

[Ryan J Ollos]

The wheel can be build from the sdist if the following commands are removed:

trac/dist.py

@@ -458 +458 @@
         # 'bdist_wininst' runs a 'build', so make the latter
         # run a 'compile_catalog' before 'build_py'
         class build(_build):
-            sub_commands = [('compile_catalog', None)] + _build.sub_commands
+            sub_commands = _build.sub_commands
 
         # 'bdist_egg' isn't that nice, all it does is an 'install_lib'
         class install_lib(_install_lib): # playing setuptools' own tricks ;-)
@@ -509 +509 @@
 
     def get_l10n_trac_cmdclass():
         build, _install_lib = get_command_overriders()
-        build.sub_commands.insert(0, ('generate_messages_js', None))
-        build.sub_commands.insert(0, ('compile_catalog_js', None))
-        build.sub_commands.insert(0, ('compile_catalog_tracini', None))
         class install_lib(_install_lib):
             def l10n_run(self):
-                self.run_command('compile_catalog_tracini')
-                self.run_command('compile_catalog_js')
-                self.run_command('generate_messages_js')
-                self.run_command('compile_catalog')
+                pass
         return {
             'build': build, 'install_lib': install_lib,
             'check_catalog': check_catalog,

Any idea of how we can conditionally run those commands only if the po files exist in the source?

[Ryan J Ollos]

How about something like this?:

setup.py

@@ -14 +14 @@
 
 import sys
 
+from pkg_resources import resource_exists
 from setuptools import setup, find_packages
 
 min_python = (2, 5)
@@ -29 +30 @@
 try:
     import babel
 
-    extractors = [
-        ('**.py',                'trac.dist:extract_python', None),
-        ('**/templates/**.html', 'genshi', None),
-        ('**/templates/**.txt',  'genshi',
-         {'template_class': 'genshi.template:NewTextTemplate'}),
-    ]
-    extra['message_extractors'] = {
-        'trac': extractors,
-        'tracopt': extractors,
-    }
-
-    from trac.dist import get_l10n_trac_cmdclass
-    extra['cmdclass'] = get_l10n_trac_cmdclass()
-
+    if resource_exists('trac', 'locale/en_US/LC_MESSAGES/messages.po'):
+        extractors = [
+            ('**.py',                'trac.dist:extract_python', None),
+            ('**/templates/**.html', 'genshi', None),
+            ('**/templates/**.txt',  'genshi',
+             {'template_class': 'genshi.template:NewTextTemplate'}),
+        ]
+        extra['message_extractors'] = {
+            'trac': extractors,
+            'tracopt': extractors,
+        }
+
+        from trac.dist import get_l10n_trac_cmdclass
+        extra['cmdclass'] = get_l10n_trac_cmdclass()
 except ImportError:
     pass
 

Or, might be able to avoid checking for a specific locale by looking for the pot file, since it won't be present in the sdist: resource_exists('trac', 'locale/messages.pot').

[Ryan J Ollos]

Replying to Ryan J Ollos:

comment:14 issue fixed in r16696.

Additional related changes in r16795.

[Jun Omae]

Replying to Ryan J Ollos:

The sdist contains mo files, but no po files:

[…]

This is what I'd expect, and the wheel similarly contains mo files and no po files.

I don't think that the sdist file expectably has no *.po files. I consider that *.po files are parts of source distribution and the sdist file should contain same files exported from our repository.

$ for i in ~/arc/Trac-1.*.tar.gz; do echo -e "$(basename $i):\t$(tar tzf $i | grep -c '\.po$') po files\t$(tar tzf $i | grep -c '\.mo$') mo files"; done
Trac-1.0.10.tar.gz:     96 po files     0 mo files
Trac-1.0.11.tar.gz:     96 po files     0 mo files
Trac-1.0.12.tar.gz:     96 po files     66 mo files
Trac-1.0.13.tar.gz:     96 po files     0 mo files
Trac-1.0.14.tar.gz:     96 po files     66 mo files
Trac-1.0.15.tar.gz:     96 po files     66 mo files
Trac-1.0.17.tar.gz:     96 po files     66 mo files
Trac-1.0.1.tar.gz:      96 po files     0 mo files
Trac-1.0.2.tar.gz:      96 po files     0 mo files
Trac-1.0.3.tar.gz:      96 po files     0 mo files
Trac-1.0.4.tar.gz:      96 po files     0 mo files
Trac-1.0.5.tar.gz:      96 po files     0 mo files
Trac-1.0.6.post2.tar.gz:        96 po files     0 mo files
Trac-1.0.7.tar.gz:      96 po files     0 mo files
Trac-1.0.8.tar.gz:      96 po files     66 mo files
Trac-1.0.9.tar.gz:      96 po files     0 mo files
Trac-1.0.tar.gz:        91 po files     0 mo files
Trac-1.2.1.tar.gz:      96 po files     66 mo files
Trac-1.2.2.tar.gz:      96 po files     66 mo files
Trac-1.2.3.tar.gz:      96 po files     64 mo files
Trac-1.2.tar.gz:        96 po files     0 mo files
Trac-1.3.1.tar.gz:      96 po files     68 mo files
Trac-1.3.2.tar.gz:      96 po files     66 mo files
Trac-1.3.3.tar.gz:      96 po files     66 mo files

Without setuptools_subversion in the environment in which the packages are built there is an error installing from the sdist package: …

I think we could use MANIFEST.in file. I'm unsure why the MANIFEST.in file has been removed in r4819. See also ​https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute.

new file MANIFEST.in

@@ +1 @@
+include .appveyor.yml .gitignore .hgeol .hgignore .travis.yml AUTHORS COPYING ChangeLog INSTALL MANIFEST.in Makefile Makefile.cfg.sample README RELEASE THANKS UPGRADE messages-js.cfg requirements-release.txt setup.cfg setup.py setup_wininst.bmp tracini.cfg
+recursive-include trac *.py *.html *.txt *.rss *.xml
+recursive-include tracopt *.py *.html *.txt *.rss *.xml
+recursive-include trac/locale *.po *.mo
+include trac/admin/templates/deploy_trac.*
+include trac/locale/*.pot
+include trac/templates/README
+include trac/tests/figleaf-exclude
+include trac/tests/functional/*.dtd
+include trac/tests/functional/*.ent
+include trac/ticket/workflows/*.ini
+include trac/wiki/default-pages/*
+include tracopt/versioncontrol/svn/tests/svnrepos.dump
+graft .tx
+graft contrib
+graft doc
+graft sample-plugins
+graft trac/htdocs
+prune .git
+prune .hg
+prune .svn

[Ryan J Ollos]

I've done some reading of the documentation, but I'm unsure. Should the sdist effectively be the same as a repository export? In which case, I would think the mo files should not be included because they are compiled artifacts. Or should the sdist be the minimal set of files needed to build a whl or egg archive?

I've posted the question at ​SO:53640308.

The ​docs say that .svn, .git and .hg are pruned by default, so it looks like we don't need to explicitly specify the prune of those directories.

[Ryan J Ollos]

Fixed failing deploy step in r16930.

[Ryan J Ollos]

Replying to Jun Omae:

I think we could use MANIFEST.in file. I'm unsure why the MANIFEST.in file has been removed in r4819. See also ​https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute.

I've been using ​setuptools_subversion: tags/trac-1.2.4/requirements-release.txt​. On Travis CI, we could use ​setuptools-git.

If we need to maintain MANIFEST.in, ​check-manifest could be useful.

[Ryan J Ollos]

Added setuptools-git to Travis CI configuration in r16987. The source release includes all of the files now.

[Ryan J Ollos]

First attempt to deploy on release failed with:

HTTPError: 400 Client Error: User 'edgewall-org' does not have a verified primary email address. Please add a verified primary email before attempting to upload to PyPI. See https://test.pypi.org/help/#verified-email for more information.for more information. for url: https://test.pypi.org/legacy/

Will try again for 1.0.19.

[Ryan J Ollos]

Merged changes to 1.2-stable in r16994, to trunk in r16995.

After testing is finished and we switch from test.pypi.org to pypi.org, we'll need to add an environment variable or conditional to skip deployment on trunk (developer stable versions don't get uploaded to PyPI).

[Ryan J Ollos]

Modified formatting of .travis.yml in r16996, merged in r16997, r16998.

[Ryan J Ollos]

I had intended to upload to Edgewall in the after_deploy step, but it seems the artifacts in dist are removed. Therefore, I reconsidered that it may be better to upload to the Edgewall server and then push to PyPI in the make-release.sh step (TracDev/ReleaseChecklist#Uploadpackages).

Uploading with cURL ​does not work ​as described in the docs, so I've switched to using SCP.

[Ryan J Ollos]

Proposed changes: [b328656cc/rjollos.git]

• DONE Remove t13064_upload_sftp branch from Edgewall repository on GitHub.
• Upload only on tags, unless we decide to upload for all branches.

[Ryan J Ollos]

comment:45 changes committed in r17006, merged in r17007, r17008.

See ​https://download.edgewall.org/trac/incoming.

[Ryan J Ollos]

Replying to Ryan J Ollos:

In another ticket we'll need to investigate if the same can be done on AppVeyor.

⇒ #13190.

[Jun Omae]

Replying to Ryan J Ollos:

Replying to Ryan J Ollos:

comment:14 issue fixed in r16696.

Additional related changes in r16795.

I just noticed…, why make compile is invoked only on deploy stage? I think we should compile catalogs on test stage also.

In 1.0-stable and 1.2-stable, catalogs are not compile on test stage at travis-ci. However, in 1.4-stable and trunk, catalogs are compile on test stage at travis-ci.

Several unit tests depend on existence of compiled catalogs.

$ git grep '\<get_available_locales(' mirror/1.2-stable -- 'trac*/**/tests/*.py'
mirror/1.2-stable:trac/admin/tests/console.py:        locales = get_available_locales()
mirror/1.2-stable:trac/admin/tests/functional.py:            if get_available_locales():
mirror/1.2-stable:trac/prefs/tests/functional.py:            if get_available_locales():
mirror/1.2-stable:trac/prefs/tests/functional.py:            if has_babel and get_available_locales():
mirror/1.2-stable:trac/prefs/tests/functional.py:        for second_locale_id in get_available_locales():
mirror/1.2-stable:trac/util/tests/translation.py:                      for locale in translation.get_available_locales()
mirror/1.2-stable:trac/util/tests/translation.py:                              set(translation.get_available_locales())) or \

[Ryan J Ollos]

make compile was extracted to a separate statement in r16696, then wrapped in conditional in r16795, as noted. I'll fix it.

[Ryan J Ollos]

Committed to 1.0-stable in r17163. Looks like catalogs are compiled on 1.2-stable:

• ​build 581050363
• branches/1.2-stable/.travis.yml@17077:117#L88​.

[Jun Omae]

Replying to Ryan J Ollos:

Committed to 1.0-stable in r17163. Looks like catalogs are compiled on 1.2-stable:

• ​build 581050363
• branches/1.2-stable/.travis.yml@17077:117#L88​.

Hmm, ​Branches page on travis-ci has something wrong. I checked ​Build #3819 on 1.2-stable branch, however the build page shows 1.0.19dev/l10n/ja: … as commit log.

As another possibility, git-svn maybe builds incorrect tree on svn merge commit.

[Ryan J Ollos]

Replying to Jun Omae:

As another possibility, git-svn maybe builds incorrect tree on svn merge commit.

On a few occasions I've seen Travis CI running the build for the wrong branch after a commit. I will keep an eye on it.

[Ryan J Ollos]

Replying to Ryan J Ollos:

Modified formatting of .travis.yml in r16996, merged in r16997, r16998.

r16998 removed print_version. That must have been an incorrect merge conflict resolution. Fixed by Jun in r17492.