Deploy to PyPI from TravisCI

[Ryan J Ollos]

To speed creation of releases we can have the packages generated and ​deployed to PyPI on TravisCI.

In another ticket we'll need to investigate if the same can be done on AppVeyor.

We can then pull the packages from PyPI to the edgewall FTP site.

[Ryan J Ollos]

Another idea: we could run ​pypiserver on edgewall and deploy on every release. I wanted to deploy to testpypi on every release, but it won't work because packages cannot be overwritten.

[Ryan J Ollos]

Proposed changes for TravisCI in [4ec6d476c/rjollos.git] (minus the change to setup.cfg). It seems to be working correctly to test and deploy to ​testpypi. See ​build 181 and ​1.0.15dev4 on testpypi.

In theory we can just delete the server line, change the repository line, regenerate the encrypted password using the switch (-r edgewall/trac) and it will work on the next build that is tagged. A really frustrating issue I encountered was in encrypting the password. I was working from a clone with the origin,

origin	https://svn.edgewall.org/git/trac/mirror (fetch)
origin	https://svn.edgewall.org/git/trac/devs/rjollos (push)

but pushing to my fork on github. For that case, it's necessary to ​specify the repository:

$ travis encrypt <password> -r rjollos/trac

The documentation describes using travis encrypt --add deploy.password, but it annoyingly reformats the entire .travis.yml file.

Another tip I read: jobs is an alias for matrix, but both cannot be present in the file. That's why the matrix statement was merged with jobs. ​Build stages is a new feature. It would be nice if there were more recipes for Travis CI, I find it rather hard to work with.

Proposed next steps:

1. Commit the changes to 1.0-stable, keeping the server as testpypi, and just confirm that the deploy works correctly for release 1.0.15. If so, change server to point to pypi and merge to 1.2-stable and trunk.
2. Setup a pypiserver so we can have test packages on every build
3. It looks like we could ​deploy to a server, which would put the downloads on the Edgewall server, but another possibility could be to just run a script on the Edgewall server to pull from PyPI.
4. (If possible) Add a deploy to pypi steps to AppVeyor builds.

In retrospect, I should have started with trying to do the equivalent with AppVeyor since I work from OSX and generating builds for Windows is more of a pain-point for me.

[Ryan J Ollos]

Side note: travis lint is a useful tool to run after modifying .travis.yml, but it gives a few incorrect warnings:

$ travis lint
Warnings for .travis.yml:
[x] unexpected key jobs, dropping
[x] value for addons section is empty, dropping
[x] in addons section: unexpected key apt, dropping

There's an open ticket for the ​apt issue. Not recognizing the jobs key is a known issue, discussed ​here (there's a ​reply to that comment from one of the Travis CI devs).

[Ryan J Ollos]

Changes committed to 1.0-stable in r16032.

[Ryan J Ollos]

Looks like tags aren't getting pushed to GitHub.

[Ryan J Ollos]

Travis jobs are running on Ubuntu Precise (12.04).

hostname: i-0ed83f0-precise-production-2-worker-org-docker.travisci.net:064d8ef5-d124-48b6-bff5-0c3e7781cfd5

I did an experiment, running tests for trunk instead on Ubuntu ​Trusty (14.04): ​build 184.

• Trusty is in ​public beta.
• ​python-subversion is not yet available in packages, although the logs show that it gets installed. SVN tests are skipped with (no svn bindings).
• There's one failure:

  ERROR: runTest (trac.db.tests.functional.DatabaseBackupTestCase)
  
  Testing backup
  
  Traceback (most recent call last):
    File "/home/travis/build/rjollos/trac/trac/db/tests/functional.py", line 26, in runTest
      backup_file = env.backup()
    File "/home/travis/build/rjollos/trac/trac/env.py", line 716, in backup
      return DatabaseManager(self).backup(dest)
    File "/home/travis/build/rjollos/trac/trac/db/api.py", line 572, in backup
      return connector.backup(dest)
    File "/home/travis/build/rjollos/trac/trac/db/mysql_backend.py", line 280, in backup
      msg=to_unicode(errmsg.strip())))
  TracError: mysqldump failed: mysqldump: Got error: 1045: Access denied for user 'tracuser'@'localhost' (using password: NO) when trying to connect
  ----------------------------------------------------------------------
  Ran 205 tests in 357.456s
  FAILED (errors=1)
  make: *** [functional-test] Error 1
  

MySQL tests on Precise:

  Package        Version
  ------------------------------------------------------
  Python       : 2.7.3 (default, Dec 18 2014, 19:10:20) 
               : [GCC 4.6.3]
  Setuptools   : 12.0.5
  Jinja2       : 2.9.6
  Genshi       : 0.7 (with speedups)
  Babel        : 2.4.0
  sqlite3      : 2.6.0 (3.7.9)
  PySqlite     : not installed
  PyMySQL      : 0.7.11.None
  Psycopg2     : not installed
  SVN bindings : 1.6.17 (r1128011)
  Mercurial    : 2.0.2
  Pygments     : 2.2.0
  Textile      : 2.3.12
  Pytz         : 2017.2
  Docutils     : 0.13.1
  Twill        : 0.9
  LXML         : 3.8.0
  coverage     : not installed
  figleaf      : not installed

MySQL tests on Trusty:

  Package        Version
  ------------------------------------------------------
  Python       : 2.7.6 (default, Oct 26 2016, 20:30:19) 
               : [GCC 4.8.4]
  Setuptools   : 36.0.1
  Jinja2       : 2.9.6
  Genshi       : 0.7 (with speedups)
  Babel        : 2.4.0
  sqlite3      : 2.6.0 (3.8.2)
  PySqlite     : not installed
  PyMySQL      : 0.7.11.None
  Psycopg2     : not installed
  SVN bindings : not installed
  Mercurial    : not installed
  Pygments     : 2.2.0
  Textile      : 2.3.12
  Pytz         : 2017.2
  Docutils     : 0.13.1
  Twill        : 0.9
  LXML         : 3.8.0
  coverage     : not installed
  figleaf      : not installed

Subversion 1.8.8 is available on Precise.

If we can fix the issue, it might make sense to move the trunk to Trusty since it's closer to an environment that we support for the trunk: TracDev/ApiChanges/1.3. MySQL is 5.6.33 on Trusty, 5.5.41 on Precise.

We might want to echo MySQL and PostgreSQL versions in make status.

[Ryan J Ollos]

Testing deploy step in r16089 (removed tag in r16090). Hit an error:

HTTPError: 410 Client Error: Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at ​https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see ​https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see ​https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html) for url: ​https://testpypi.python.org/pypi

I'll test again after making modifications in my fork.

[Jun Omae]

The following settings is in .git/config on my working directory.

[remote "mirror"]
        url = http://svn.edgewall.org/git/trac/mirror
        fetch = +refs/heads/*:refs/remotes/mirror/*
        fetch = refs/remotes/tags/*:refs/tags/*
        fetch = refs/remotes/0.5-stable:refs/remotes/mirror/0.5-stable
        fetch = refs/remotes/0.6-stable:refs/remotes/mirror/0.6-stable
        fetch = refs/remotes/0.7-stable:refs/remotes/mirror/0.7-stable
        fetch = refs/remotes/0.8-stable:refs/remotes/mirror/0.8-stable
        fetch = refs/remotes/0.9-stable:refs/remotes/mirror/0.9-stable
        fetch = refs/remotes/0.10-stable:refs/remotes/mirror/0.10-stable
        fetch = refs/remotes/0.11-stable:refs/remotes/mirror/0.11-stable
        pushurl = no_push

However, git fetch mirror fails.

$ git fetch mirror
fatal: Cannot fetch both refs/remotes/tags/trac-1.0.16 and refs/tags/trac-1.0.16 to refs/tags/trac-1.0.16

It seems that trac-1.0.16 tags still exist in mirror repository. Could you please remove the tags from mirror repository?

$ git tag -l | grep -F 1.0.16
$ git ls-remote mirror | grep -F 1.0.16
9b803ccb48c43e216e283e1d4b2fed8dc35ba82c        refs/remotes/tags/trac-1.0.16
9b803ccb48c43e216e283e1d4b2fed8dc35ba82c        refs/tags/trac-1.0.16

[Jun Omae]

Mercurial mirror repository also has trac-1.0.16 tag in changeset:ab2dc11b8574/mirror.

[Ryan J Ollos]

Updated pypitest URL in r16091.

1. Deleted tag in svn repository (r16093)
2. Temporarily disabled pre-receive hook in Git mirror repository
3. Deleted tag:

   $ git push mirror :trac-1.0.16
   To https://svn.edgewall.org/git/trac/mirror
    - [deleted]             trac-1.0.16
   $ git push mirror :trac-1.0.16
   error: unable to delete 'trac-1.0.16': remote ref does not exist
   error: failed to push some refs to 'https://svn.edgewall.org/git/trac/mirror'
   

4. After pushing r16095 the tag has reappeared in the mirror repository

Am I missing something?

I didn't expect the tag to persist for more than a few hours, however it would have been a better idea to use trac-1.0.16dev as the tag name.

[Ryan J Ollos]

Latest attempt also failed:

Uploading distributions to ​https://test.pypi.org/legacy
Uploading Trac-1.0.16.dev0-py2-none-any.whl
RedirectDetected: "​https://test.pypi.org/legacy" attempted to redirect to "​https://test.pypi.org/legacy/" during upload. Aborting…

I'll make the minor adjustment and retry.

[Ryan J Ollos]

Finally, it seems to have worked:

Uploading distributions to ​https://test.pypi.org/legacy/
Uploading Trac-1.0.16.dev0-py2-none-any.whl
Uploading Trac-1.0.16.dev0.tar.gz

Releases are available at ​https://testpypi.python.org/pypi/Trac/1.0.16.dev0.

I'll try to cleanup the tags now.

[Ryan J Ollos]

I found another issue. The tar package has the locale dir, but it's missing from the wheel package.

[Jun Omae]

git-svn creates a remote branch in mirror Git repository for a tag in source Subversion repository. We create tags in mirror Git repository from the remote branches using fetch = refs/remotes/tags/*:refs/tags/*.

$ git ls-remote http://svn.edgewall.org/git/trac/mirror | grep -F 1.0.16
1dcb62395de01c86d749295dc5e60ec4fa205e0a        refs/remotes/tags/trac-1.0.16

We could remove remote branches using git branch -r -d NAME....

$ git branch -a | grep -F 1.0.16
  remotes/tags/trac-1.0.16
$ git branch -r -d tags/trac-1.0.16
Deleted remote branch tags/trac-1.0.16 (was 4afdfd60f).

In addition, it is needed to remove trac-1.0.16 from ​https://github.com/edgewall/trac.

[Ryan J Ollos]

I removed tags in Git repository and GitHub. However, it seems like I followed the same steps as previously so I'll look at whether they re-appear on the next commit.

I tried removing tags from mirror, but was unsuccessful: [12657/mirror].

$ hg tag --remove trac-1.0.16
$ hg push

[Ryan J Ollos]

Replying to Ryan J Ollos:

If we can fix the issue, it might make sense to move the trunk to Trusty since it's closer to an environment that we support for the trunk: TracDev/ApiChanges/1.3. MySQL is 5.6.33 on Trusty, 5.5.41 on Precise.

Trusty will ​soon be the default.

[Ryan J Ollos]

Additional configuration changes in r16122 for deploying to the Edgewall PyPI server. See ​build 256289210.