Edgewall Software
Modify

Opened 7 years ago

Closed 5 years ago

Last modified 3 years ago

#12833 closed enhancement (fixed)

Deploy to PyPI from TravisCI

Reported by: Ryan J Ollos Owned by: Ryan J Ollos
Priority: normal Milestone: 1.0.19
Component: general Version:
Severity: normal Keywords: travis-ci
Cc: Branch:
Release Notes:

Added automated package deployments from Travis CI to the Edgewall server.

API Changes:
Internal Changes:

Description (last modified by Ryan J Ollos)

To speed creation of releases we can have the packages generated and deployed to PyPI on TravisCI.

In another ticket we'll need to investigate if the same can be done on AppVeyor.

We can then pull the packages from PyPI to the edgewall FTP site.

Attachments (0)

Change History (53)

comment:1 by Ryan J Ollos, 7 years ago

Owner: set to Ryan J Ollos
Status: newassigned

comment:2 by Ryan J Ollos, 7 years ago

Another idea: we could run pypiserver on edgewall and deploy on every release. I wanted to deploy to testpypi on every release, but it won't work because packages cannot be overwritten.

comment:3 by Ryan J Ollos, 7 years ago

Milestone: 1.0.151.0.16

Proposed changes for TravisCI in [4ec6d476c/rjollos.git] (minus the change to setup.cfg). It seems to be working correctly to test and deploy to testpypi. See build 181 and 1.0.15dev4 on testpypi.

In theory we can just delete the server line, change the repository line, regenerate the encrypted password using the switch (-r edgewall/trac) and it will work on the next build that is tagged. A really frustrating issue I encountered was in encrypting the password. I was working from a clone with the origin,

origin	https://svn.edgewall.org/git/trac/mirror (fetch)
origin	https://svn.edgewall.org/git/trac/devs/rjollos (push)

but pushing to my fork on github. For that case, it's necessary to specify the repository:

$ travis encrypt <password> -r rjollos/trac

The documentation describes using travis encrypt --add deploy.password, but it annoyingly reformats the entire .travis.yml file.

Another tip I read: jobs is an alias for matrix, but both cannot be present in the file. That's why the matrix statement was merged with jobs. Build stages is a new feature. It would be nice if there were more recipes for Travis CI, I find it rather hard to work with.

Proposed next steps:

  1. Commit the changes to 1.0-stable, keeping the server as testpypi, and just confirm that the deploy works correctly for release 1.0.15. If so, change server to point to pypi and merge to 1.2-stable and trunk.
  2. Setup a pypiserver so we can have test packages on every build
  3. It looks like we could deploy to a server, which would put the downloads on the Edgewall server, but another possibility could be to just run a script on the Edgewall server to pull from PyPI.
  4. AppVeyor.

In retrospect, I should have started with trying to do the equivalent with AppVeyor since I work from OSX and generating builds for Windows is more of a pain-point for me.

Version 2, edited 7 years ago by Ryan J Ollos (previous) (next) (diff)

comment:4 by Ryan J Ollos, 7 years ago

Side note: travis lint is a useful tool to run after modifying .travis.yml, but it gives a few incorrect warnings:

$ travis lint
Warnings for .travis.yml:
[x] unexpected key jobs, dropping
[x] value for addons section is empty, dropping
[x] in addons section: unexpected key apt, dropping

There's an open ticket for the apt issue. Not recognizing the jobs key is a known issue, discussed here (there's a reply to that comment from one of the Travis CI devs).

comment:5 by Ryan J Ollos, 7 years ago

Changes committed to 1.0-stable in r16032.

comment:6 by Ryan J Ollos, 7 years ago

Looks like tags aren't getting pushed to GitHub.

comment:7 by Ryan J Ollos, 7 years ago

Travis jobs are running on Ubuntu Precise (12.04).

hostname: i-0ed83f0-precise-production-2-worker-org-docker.travisci.net:064d8ef5-d124-48b6-bff5-0c3e7781cfd5

I did an experiment, running tests for trunk instead on Ubuntu Trusty (14.04): build 184.

  • Trusty is in public beta.
  • python-subversion is not yet available in packages, although the logs show that it gets installed. SVN tests are skipped with (no svn bindings).
  • There's one failure:
    ERROR: runTest (trac.db.tests.functional.DatabaseBackupTestCase)
    
    Testing backup
    
    Traceback (most recent call last):
      File "/home/travis/build/rjollos/trac/trac/db/tests/functional.py", line 26, in runTest
        backup_file = env.backup()
      File "/home/travis/build/rjollos/trac/trac/env.py", line 716, in backup
        return DatabaseManager(self).backup(dest)
      File "/home/travis/build/rjollos/trac/trac/db/api.py", line 572, in backup
        return connector.backup(dest)
      File "/home/travis/build/rjollos/trac/trac/db/mysql_backend.py", line 280, in backup
        msg=to_unicode(errmsg.strip())))
    TracError: mysqldump failed: mysqldump: Got error: 1045: Access denied for user 'tracuser'@'localhost' (using password: NO) when trying to connect
    ----------------------------------------------------------------------
    Ran 205 tests in 357.456s
    FAILED (errors=1)
    make: *** [functional-test] Error 1
    

MySQL tests on Precise:

  Package        Version
  ------------------------------------------------------
  Python       : 2.7.3 (default, Dec 18 2014, 19:10:20) 
               : [GCC 4.6.3]
  Setuptools   : 12.0.5
  Jinja2       : 2.9.6
  Genshi       : 0.7 (with speedups)
  Babel        : 2.4.0
  sqlite3      : 2.6.0 (3.7.9)
  PySqlite     : not installed
  PyMySQL      : 0.7.11.None
  Psycopg2     : not installed
  SVN bindings : 1.6.17 (r1128011)
  Mercurial    : 2.0.2
  Pygments     : 2.2.0
  Textile      : 2.3.12
  Pytz         : 2017.2
  Docutils     : 0.13.1
  Twill        : 0.9
  LXML         : 3.8.0
  coverage     : not installed
  figleaf      : not installed

MySQL tests on Trusty:

  Package        Version
  ------------------------------------------------------
  Python       : 2.7.6 (default, Oct 26 2016, 20:30:19) 
               : [GCC 4.8.4]
  Setuptools   : 36.0.1
  Jinja2       : 2.9.6
  Genshi       : 0.7 (with speedups)
  Babel        : 2.4.0
  sqlite3      : 2.6.0 (3.8.2)
  PySqlite     : not installed
  PyMySQL      : 0.7.11.None
  Psycopg2     : not installed
  SVN bindings : not installed
  Mercurial    : not installed
  Pygments     : 2.2.0
  Textile      : 2.3.12
  Pytz         : 2017.2
  Docutils     : 0.13.1
  Twill        : 0.9
  LXML         : 3.8.0
  coverage     : not installed
  figleaf      : not installed

Subversion 1.8.8 is available on Precise.

If we can fix the issue, it might make sense to move the trunk to Trusty since it's closer to an environment that we support for the trunk: TracDev/ApiChanges/1.3. MySQL is 5.6.33 on Trusty, 5.5.41 on Precise.

We might want to echo MySQL and PostgreSQL versions in make status.

comment:8 by Ryan J Ollos, 7 years ago

Testing deploy step in r16089 (removed tag in r16090). Hit an error:

HTTPError: 410 Client Error: Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html) for url: https://testpypi.python.org/pypi

I'll test again after making modifications in my fork.

comment:9 by Jun Omae, 7 years ago

The following settings is in .git/config on my working directory.

[remote "mirror"]
        url = http://svn.edgewall.org/git/trac/mirror
        fetch = +refs/heads/*:refs/remotes/mirror/*
        fetch = refs/remotes/tags/*:refs/tags/*
        fetch = refs/remotes/0.5-stable:refs/remotes/mirror/0.5-stable
        fetch = refs/remotes/0.6-stable:refs/remotes/mirror/0.6-stable
        fetch = refs/remotes/0.7-stable:refs/remotes/mirror/0.7-stable
        fetch = refs/remotes/0.8-stable:refs/remotes/mirror/0.8-stable
        fetch = refs/remotes/0.9-stable:refs/remotes/mirror/0.9-stable
        fetch = refs/remotes/0.10-stable:refs/remotes/mirror/0.10-stable
        fetch = refs/remotes/0.11-stable:refs/remotes/mirror/0.11-stable
        pushurl = no_push

However, git fetch mirror fails.

$ git fetch mirror
fatal: Cannot fetch both refs/remotes/tags/trac-1.0.16 and refs/tags/trac-1.0.16 to refs/tags/trac-1.0.16

It seems that trac-1.0.16 tags still exist in mirror repository. Could you please remove the tags from mirror repository?

$ git tag -l | grep -F 1.0.16
$ git ls-remote mirror | grep -F 1.0.16
9b803ccb48c43e216e283e1d4b2fed8dc35ba82c        refs/remotes/tags/trac-1.0.16
9b803ccb48c43e216e283e1d4b2fed8dc35ba82c        refs/tags/trac-1.0.16

comment:10 by Jun Omae, 7 years ago

Mercurial mirror repository also has trac-1.0.16 tag in changeset:ab2dc11b8574/mirror.

comment:11 by Ryan J Ollos, 7 years ago

Updated pypitest URL in r16091.

  1. Deleted tag in svn repository (r16093)
  2. Temporarily disabled pre-receive hook in Git mirror repository
  3. Deleted tag:
    $ git push mirror :trac-1.0.16
    To https://svn.edgewall.org/git/trac/mirror
     - [deleted]             trac-1.0.16
    $ git push mirror :trac-1.0.16
    error: unable to delete 'trac-1.0.16': remote ref does not exist
    error: failed to push some refs to 'https://svn.edgewall.org/git/trac/mirror'
    
  4. After pushing r16095 the tag has reappeared in the mirror repository

Am I missing something?

I didn't expect the tag to persist for more than a few hours, however it would have been a better idea to use trac-1.0.16dev as the tag name.

Last edited 7 years ago by Ryan J Ollos (previous) (diff)

comment:12 by Ryan J Ollos, 7 years ago

Latest attempt also failed:

Uploading distributions to https://test.pypi.org/legacy
Uploading Trac-1.0.16.dev0-py2-none-any.whl
RedirectDetected: "https://test.pypi.org/legacy" attempted to redirect to "https://test.pypi.org/legacy/" during upload. Aborting…

I'll make the minor adjustment and retry.

comment:13 by Ryan J Ollos, 7 years ago

Finally, it seems to have worked:

Uploading distributions to https://test.pypi.org/legacy/
Uploading Trac-1.0.16.dev0-py2-none-any.whl
Uploading Trac-1.0.16.dev0.tar.gz

Releases are available at https://testpypi.python.org/pypi/Trac/1.0.16.dev0.

I'll try to cleanup the tags now.

comment:14 by Ryan J Ollos, 7 years ago

I found another issue. The tar package has the locale dir, but it's missing from the wheel package.

Last edited 7 years ago by Ryan J Ollos (previous) (diff)

comment:15 by Jun Omae, 7 years ago

git-svn creates a remote branch in mirror Git repository for a tag in source Subversion repository. We create tags in mirror Git repository from the remote branches using fetch = refs/remotes/tags/*:refs/tags/*.

$ git ls-remote http://svn.edgewall.org/git/trac/mirror | grep -F 1.0.16
1dcb62395de01c86d749295dc5e60ec4fa205e0a        refs/remotes/tags/trac-1.0.16

We could remove remote branches using git branch -r -d NAME....

$ git branch -a | grep -F 1.0.16
  remotes/tags/trac-1.0.16
$ git branch -r -d tags/trac-1.0.16
Deleted remote branch tags/trac-1.0.16 (was 4afdfd60f).

In addition, it is needed to remove trac-1.0.16 from https://github.com/edgewall/trac.

comment:16 by Ryan J Ollos, 7 years ago

I removed tags in Git repository and GitHub. However, it seems like I followed the same steps as previously so I'll look at whether they re-appear on the next commit.

I tried removing tags from mirror, but was unsuccessful: [12657/mirror].

$ hg tag --remove trac-1.0.16
$ hg push

in reply to:  7 comment:17 by Ryan J Ollos, 7 years ago

Replying to Ryan J Ollos:

If we can fix the issue, it might make sense to move the trunk to Trusty since it's closer to an environment that we support for the trunk: TracDev/ApiChanges/1.3. MySQL is 5.6.33 on Trusty, 5.5.41 on Precise.

Trusty will soon be the default.

comment:18 by Ryan J Ollos, 7 years ago

Additional configuration changes in r16122 for deploying to the Edgewall PyPI server. See build 256289210.

comment:19 by Ryan J Ollos, 7 years ago

Description: modified (diff)
Summary: Deploy to PyPI from CIDeploy to PyPI from TravisCI

in reply to:  16 ; comment:20 by Ryan J Ollos, 7 years ago

Replying to Ryan J Ollos:

I tried removing tags from mirror, but was unsuccessful: [12657/mirror].

$ hg tag --remove trac-1.0.16
$ hg push

Discussed elsewhere, the best choice at this point might be to just skip 1.0.16 and release 1.0.17.

comment:14 is the last item needing attention, then the TravisCI configuration can be merged to 1.2-stable and trunk.

comment:21 by Ryan J Ollos, 6 years ago

Milestone: 1.0.161.0.17

Milestone renamed

in reply to:  20 comment:22 by Ryan J Ollos, 6 years ago

Replying to Ryan J Ollos:

Discussed elsewhere, the best choice at this point might be to just skip 1.0.16 and release 1.0.17.

See comment:7:ticket:12839.

comment:23 by Ryan J Ollos, 6 years ago

Removed skip_upload_docs in r16693 since true is now the default value.

in reply to:  14 comment:24 by Ryan J Ollos, 6 years ago

Replying to Ryan J Ollos:

I found another issue. The tar package has the locale dir, but it's missing from the wheel package.

Comparing with a wheel built from my terminal, the message catalogs are compiled on Travis CI, but the .mo file are not copied to the build directory. The log on Travis CI is missing log messages like the following:

creating build/bdist.macosx-10.13-x86_64/wheel/trac/locale
creating build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl
creating build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl/LC_MESSAGES
copying build/lib/trac/locale/sl/LC_MESSAGES/tracini.mo -> build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl/LC_MESSAGES
copying build/lib/trac/locale/sl/LC_MESSAGES/messages.mo -> build/bdist.macosx-10.13-x86_64/wheel/trac/locale/sl/LC_MESSAGES

And the files are not added to the whl, missing log messages like the following:

adding 'trac/locale/sl/LC_MESSAGES/messages.mo'
adding 'trac/locale/sl/LC_MESSAGES/tracini.mo'

I have no idea of what the problem might be. The environment on Travis CI seems to be pretty much the same as my computer. The deploy script is fairly simple.

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:25 by Ryan J Ollos, 6 years ago

Oh, I think we may just need to run make compile. Testing the change …

comment:26 by Ryan J Ollos, 6 years ago

Milestone: 1.0.171.0.18
Release Notes: modified (diff)

comment:14 issue fixed in r16696. If deployment to test.pypi.org succeeds with the 1.0.17 release, we can change the configuration to deploy to pypi.org for release 1.0.18.

comment:27 by Ryan J Ollos, 6 years ago

The 1.0.17 release was pushed to test.pypi. Needs testing.

in reply to:  27 comment:28 by Ryan J Ollos, 6 years ago

Replying to Ryan J Ollos:

The 1.0.17 release was pushed to test.pypi. Needs testing.

I diff'ed the archives on test.pypi and pypi. The whl packages are the same. The sdist on pypi has a bunch of extra files that aren't needed, such as .po, THANKS, etc.

It looks like we should remove setuptools_subversion from the environment where packages are built. That package was in the environment when I created the 1.0.17 packages for uploading to pypi, but not in the Travis CI environment that created the packages for test.pypi.

comment:29 by Ryan J Ollos, 6 years ago

Without setuptools_subversion in the environment in which the packages are built there is an error installing from the sdist package:

The sdist contains mo files, but no po files:

$ cd dist && tar xzvf Trac-1.3.3.tar.gz 2&> /dev/null
$ find Trac-1.3.3 -name "*.po" | head -4
$ find Trac-1.3.3 -name "*.mo" | head -4
Trac-1.3.3/trac/locale/sl/LC_MESSAGES/tracini.mo
Trac-1.3.3/trac/locale/sl/LC_MESSAGES/messages.mo
Trac-1.3.3/trac/locale/pl/LC_MESSAGES/tracini.mo
Trac-1.3.3/trac/locale/pl/LC_MESSAGES/messages.mo

This is what I'd expect, and the wheel similarly contains mo files and no po files.

Installing with pip from the sdist, a wheel package appears to be built and fails when trying to compile the message catalogs:

$ pip install dist/Trac-1.3.3.tar.gz
Processing ./dist/Trac-1.3.3.tar.gz
Requirement already satisfied: setuptools>=0.6 in /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/lib/python2.7/site-packages (from Trac==1.3.3) (40.0.0)
Requirement already satisfied: jinja2>=2.9.3 in /Users/rjollos/Documents/Workspace/trac-dev/trac-trunk/.eggs/Jinja2-2.10-py2.7.egg (from Trac==1.3.3) (2.10)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/rjollos/Documents/Workspace/trac-dev/trac-trunk/.eggs/MarkupSafe-1.0-py2.7-macosx-10.13-x86_64.egg (from jinja2>=2.9.3->Trac==1.3.3) (1.0)
Building wheels for collected packages: Trac
  Running setup.py bdist_wheel for Trac ... error
  Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-wheel-SoaYot --python-tag cp27:
  running bdist_wheel
  running build
  running compile_catalog_tracini
  error: no message catalogs found

  ----------------------------------------
  Failed building wheel for Trac
  Running setup.py clean for Trac
Failed to build Trac
Installing collected packages: Trac
  Running setup.py install for Trac ... error
    Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-SWD8dN/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac:
    running install
    running build
    running compile_catalog_tracini
    error: no message catalogs found

    ----------------------------------------
Command "/Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-SWD8dN/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac" failed with error code 1 in /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-Pd0vAq/
Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:30 by Ryan J Ollos, 6 years ago

The sdist on test.pypi has the same problem as in comment:29:

$ pip install https://test-files.pythonhosted.org/packages/33/a6/a210399b48125f2af7fb23fb4bc93e4054cbe3b94e4ebf26aff483777f06/Trac-1.0.17.tar.gz
Collecting https://test-files.pythonhosted.org/packages/33/a6/a210399b48125f2af7fb23fb4bc93e4054cbe3b94e4ebf26aff483777f06/Trac-1.0.17.tar.gz
  Downloading https://test-files.pythonhosted.org/packages/33/a6/a210399b48125f2af7fb23fb4bc93e4054cbe3b94e4ebf26aff483777f06/Trac-1.0.17.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 626kB/s
Requirement already satisfied: setuptools>=0.6b1 in /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/lib/python2.7/site-packages (from Trac==1.0.17) (40.0.0)
Collecting Genshi>=0.6 (from Trac==1.0.17)
Building wheels for collected packages: Trac
  Running setup.py bdist_wheel for Trac ... error
  Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-wheel-cU1YjD --python-tag cp27:
  Genshi is needed by Trac setup, pre-installing
  running bdist_wheel
  running build
  running compile_catalog_tracini
  error: no message catalogs found

  ----------------------------------------
  Failed building wheel for Trac
  Running setup.py clean for Trac
Failed to build Trac
Installing collected packages: Genshi, Trac
  Running setup.py install for Trac ... error
    Complete output from command /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-ZLHuTc/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac:
    running install
    running build
    running compile_catalog_tracini
    error: no message catalogs found

    ----------------------------------------
Command "/Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-record-ZLHuTc/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjollos/.pyenv/versions/2.7.14/envs/trac-trunk-release/include/site/python2.7/Trac" failed with error code 1 in /private/var/folders/qf/y5yt86vn54j_sy7dv1f8hb3r0000gn/T/pip-req-build-qSeM0e/

I would guess that some change to our packaging configuration is needed.

comment:31 by Ryan J Ollos, 6 years ago

The wheel can be build from the sdist if the following commands are removed:

  • trac/dist.py

    diff --git a/trac/dist.py b/trac/dist.py
    index ddb586e48..a0904dc85 100644
    a b try:  
    458458        # 'bdist_wininst' runs a 'build', so make the latter
    459459        # run a 'compile_catalog' before 'build_py'
    460460        class build(_build):
    461             sub_commands = [('compile_catalog', None)] + _build.sub_commands
     461            sub_commands = _build.sub_commands
    462462
    463463        # 'bdist_egg' isn't that nice, all it does is an 'install_lib'
    464464        class install_lib(_install_lib): # playing setuptools' own tricks ;-)
    try:  
    509509
    510510    def get_l10n_trac_cmdclass():
    511511        build, _install_lib = get_command_overriders()
    512         build.sub_commands.insert(0, ('generate_messages_js', None))
    513         build.sub_commands.insert(0, ('compile_catalog_js', None))
    514         build.sub_commands.insert(0, ('compile_catalog_tracini', None))
    515512        class install_lib(_install_lib):
    516513            def l10n_run(self):
    517                 self.run_command('compile_catalog_tracini')
    518                 self.run_command('compile_catalog_js')
    519                 self.run_command('generate_messages_js')
    520                 self.run_command('compile_catalog')
     514                pass
    521515        return {
    522516            'build': build, 'install_lib': install_lib,
    523517            'check_catalog': check_catalog,

Any idea of how we can conditionally run those commands only if the po files exist in the source?

comment:32 by Ryan J Ollos, 6 years ago

How about something like this?:

  • setup.py

    diff --git a/setup.py b/setup.py
    index cf12e87c5..7d1ec8d67 100755
    a b  
    1414
    1515import sys
    1616
     17from pkg_resources import resource_exists
    1718from setuptools import setup, find_packages
    1819
    1920min_python = (2, 5)
    extra = {}  
    2930try:
    3031    import babel
    3132
    32     extractors = [
    33         ('**.py',                'trac.dist:extract_python', None),
    34         ('**/templates/**.html', 'genshi', None),
    35         ('**/templates/**.txt',  'genshi',
    36          {'template_class': 'genshi.template:NewTextTemplate'}),
    37     ]
    38     extra['message_extractors'] = {
    39         'trac': extractors,
    40         'tracopt': extractors,
    41     }
    42 
    43     from trac.dist import get_l10n_trac_cmdclass
    44     extra['cmdclass'] = get_l10n_trac_cmdclass()
    45 
     33    if resource_exists('trac', 'locale/en_US/LC_MESSAGES/messages.po'):
     34        extractors = [
     35            ('**.py',                'trac.dist:extract_python', None),
     36            ('**/templates/**.html', 'genshi', None),
     37            ('**/templates/**.txt',  'genshi',
     38             {'template_class': 'genshi.template:NewTextTemplate'}),
     39        ]
     40        extra['message_extractors'] = {
     41            'trac': extractors,
     42            'tracopt': extractors,
     43        }
     44
     45        from trac.dist import get_l10n_trac_cmdclass
     46        extra['cmdclass'] = get_l10n_trac_cmdclass()
    4647except ImportError:
    4748    pass
    4849

Or, might be able to avoid checking for a specific locale by looking for the pot file, since it won't be present in the sdist: resource_exists('trac', 'locale/messages.pot').

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

in reply to:  26 ; comment:33 by Ryan J Ollos, 6 years ago

Replying to Ryan J Ollos:

comment:14 issue fixed in r16696.

Additional related changes in r16795.

in reply to:  29 ; comment:34 by Jun Omae, 6 years ago

Replying to Ryan J Ollos:

The sdist contains mo files, but no po files:

[…]

This is what I'd expect, and the wheel similarly contains mo files and no po files.

I don't think that the sdist file expectably has no *.po files. I consider that *.po files are parts of source distribution and the sdist file should contain same files exported from our repository.

$ for i in ~/arc/Trac-1.*.tar.gz; do echo -e "$(basename $i):\t$(tar tzf $i | grep -c '\.po$') po files\t$(tar tzf $i | grep -c '\.mo$') mo files"; done
Trac-1.0.10.tar.gz:     96 po files     0 mo files
Trac-1.0.11.tar.gz:     96 po files     0 mo files
Trac-1.0.12.tar.gz:     96 po files     66 mo files
Trac-1.0.13.tar.gz:     96 po files     0 mo files
Trac-1.0.14.tar.gz:     96 po files     66 mo files
Trac-1.0.15.tar.gz:     96 po files     66 mo files
Trac-1.0.17.tar.gz:     96 po files     66 mo files
Trac-1.0.1.tar.gz:      96 po files     0 mo files
Trac-1.0.2.tar.gz:      96 po files     0 mo files
Trac-1.0.3.tar.gz:      96 po files     0 mo files
Trac-1.0.4.tar.gz:      96 po files     0 mo files
Trac-1.0.5.tar.gz:      96 po files     0 mo files
Trac-1.0.6.post2.tar.gz:        96 po files     0 mo files
Trac-1.0.7.tar.gz:      96 po files     0 mo files
Trac-1.0.8.tar.gz:      96 po files     66 mo files
Trac-1.0.9.tar.gz:      96 po files     0 mo files
Trac-1.0.tar.gz:        91 po files     0 mo files
Trac-1.2.1.tar.gz:      96 po files     66 mo files
Trac-1.2.2.tar.gz:      96 po files     66 mo files
Trac-1.2.3.tar.gz:      96 po files     64 mo files
Trac-1.2.tar.gz:        96 po files     0 mo files
Trac-1.3.1.tar.gz:      96 po files     68 mo files
Trac-1.3.2.tar.gz:      96 po files     66 mo files
Trac-1.3.3.tar.gz:      96 po files     66 mo files

Without setuptools_subversion in the environment in which the packages are built there is an error installing from the sdist package: …

I think we could use MANIFEST.in file. I'm unsure why the MANIFEST.in file has been removed in r4819. See also https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute.

  • new file MANIFEST.in

    diff --git a/MANIFEST.in b/MANIFEST.in
    new file mode 100644
    index 000000000..d2905594b
    - +  
     1include .appveyor.yml .gitignore .hgeol .hgignore .travis.yml AUTHORS COPYING ChangeLog INSTALL MANIFEST.in Makefile Makefile.cfg.sample README RELEASE THANKS UPGRADE messages-js.cfg requirements-release.txt setup.cfg setup.py setup_wininst.bmp tracini.cfg
     2recursive-include trac *.py *.html *.txt *.rss *.xml
     3recursive-include tracopt *.py *.html *.txt *.rss *.xml
     4recursive-include trac/locale *.po *.mo
     5include trac/admin/templates/deploy_trac.*
     6include trac/locale/*.pot
     7include trac/templates/README
     8include trac/tests/figleaf-exclude
     9include trac/tests/functional/*.dtd
     10include trac/tests/functional/*.ent
     11include trac/ticket/workflows/*.ini
     12include trac/wiki/default-pages/*
     13include tracopt/versioncontrol/svn/tests/svnrepos.dump
     14graft .tx
     15graft contrib
     16graft doc
     17graft sample-plugins
     18graft trac/htdocs
     19prune .git
     20prune .hg
     21prune .svn

comment:35 by Ryan J Ollos, 6 years ago

Milestone: 1.0.181.0.19

comment:36 by Ryan J Ollos, 5 years ago

I've done some reading of the documentation, but I'm unsure. Should the sdist effectively be the same as a repository export? In which case, I would think the mo files should not be included because they are compiled artifacts. Or should the sdist be the minimal set of files needed to build a whl or egg archive?

I've posted the question at SO:53640308.

The docs say that .svn, .git and .hg are pruned by default, so it looks like we don't need to explicitly specify the prune of those directories.

Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:37 by Ryan J Ollos, 5 years ago

Keywords: travis-ci added

comment:38 by Ryan J Ollos, 5 years ago

Fixed failing deploy step in r16930.

in reply to:  34 comment:39 by Ryan J Ollos, 5 years ago

Replying to Jun Omae:

I think we could use MANIFEST.in file. I'm unsure why the MANIFEST.in file has been removed in r4819. See also https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute.

I've been using setuptools_subversion: tags/trac-1.2.4/requirements-release.txt. On Travis CI, we could use setuptools-git.

If we need to maintain MANIFEST.in, check-manifest could be useful.

Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:40 by Ryan J Ollos, 5 years ago

Added setuptools-git to Travis CI configuration in r16987. The source release includes all of the files now.

Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:41 by Ryan J Ollos, 5 years ago

First attempt to deploy on release failed with:

HTTPError: 400 Client Error: User 'edgewall-org' does not have a verified primary email address. Please add a verified primary email before attempting to upload to PyPI. See https://test.pypi.org/help/#verified-email for more information.for more information. for url: https://test.pypi.org/legacy/

Will try again for 1.0.19.

comment:42 by Ryan J Ollos, 5 years ago

Merged changes to 1.2-stable in r16994, to trunk in r16995.

After testing is finished and we switch from test.pypi.org to pypi.org, we'll need to add an environment variable or conditional to skip deployment on trunk (developer stable versions don't get uploaded to PyPI).

comment:43 by Ryan J Ollos, 5 years ago

Modified formatting of .travis.yml in r16996, merged in r16997, r16998.

comment:44 by Ryan J Ollos, 5 years ago

I had intended to upload to Edgewall in the after_deploy step, but it seems the artifacts in dist are removed. Therefore, I reconsidered that it may be better to upload to the Edgewall server and then push to PyPI in the make-release.sh step (TracDev/ReleaseChecklist#Uploadpackages).

Uploading with cURL does not work as described in the docs, so I've switched to using SCP.

comment:45 by Ryan J Ollos, 5 years ago

Proposed changes: [b328656cc/rjollos.git]

  • DONE Remove t13064_upload_sftp branch from Edgewall repository on GitHub.
  • Upload only on tags, unless we decide to upload for all branches.
Last edited 5 years ago by Ryan J Ollos (previous) (diff)

comment:46 by Ryan J Ollos, 5 years ago

Description: modified (diff)
Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

in reply to:  description comment:47 by Ryan J Ollos, 5 years ago

Replying to Ryan J Ollos:

In another ticket we'll need to investigate if the same can be done on AppVeyor.

#13190.

in reply to:  33 comment:48 by Jun Omae, 5 years ago

Replying to Ryan J Ollos:

Replying to Ryan J Ollos:

comment:14 issue fixed in r16696.

Additional related changes in r16795.

I just noticed…, why make compile is invoked only on deploy stage? I think we should compile catalogs on test stage also.

In 1.0-stable and 1.2-stable, catalogs are not compile on test stage at travis-ci. However, in 1.4-stable and trunk, catalogs are compile on test stage at travis-ci.

Several unit tests depend on existence of compiled catalogs.

$ git grep '\<get_available_locales(' mirror/1.2-stable -- 'trac*/**/tests/*.py'
mirror/1.2-stable:trac/admin/tests/console.py:        locales = get_available_locales()
mirror/1.2-stable:trac/admin/tests/functional.py:            if get_available_locales():
mirror/1.2-stable:trac/prefs/tests/functional.py:            if get_available_locales():
mirror/1.2-stable:trac/prefs/tests/functional.py:            if has_babel and get_available_locales():
mirror/1.2-stable:trac/prefs/tests/functional.py:        for second_locale_id in get_available_locales():
mirror/1.2-stable:trac/util/tests/translation.py:                      for locale in translation.get_available_locales()
mirror/1.2-stable:trac/util/tests/translation.py:                              set(translation.get_available_locales())) or \

comment:49 by Ryan J Ollos, 5 years ago

make compile was extracted to a separate statement in r16696, then wrapped in conditional in r16795, as noted. I'll fix it.

comment:50 by Ryan J Ollos, 5 years ago

Committed to 1.0-stable in r17163. Looks like catalogs are compiled on 1.2-stable:

in reply to:  50 ; comment:51 by Jun Omae, 5 years ago

Replying to Ryan J Ollos:

Committed to 1.0-stable in r17163. Looks like catalogs are compiled on 1.2-stable:

Hmm, Branches page on travis-ci has something wrong. I checked Build #3819 on 1.2-stable branch, however the build page shows 1.0.19dev/l10n/ja: … as commit log.

As another possibility, git-svn maybe builds incorrect tree on svn merge commit.

Last edited 5 years ago by Jun Omae (previous) (diff)

in reply to:  51 comment:52 by Ryan J Ollos, 5 years ago

Replying to Jun Omae:

As another possibility, git-svn maybe builds incorrect tree on svn merge commit.

On a few occasions I've seen Travis CI running the build for the wrong branch after a commit. I will keep an eye on it.

in reply to:  43 comment:53 by Ryan J Ollos, 3 years ago

Replying to Ryan J Ollos:

Modified formatting of .travis.yml in r16996, merged in r16997, r16998.

r16998 removed print_version. That must have been an incorrect merge conflict resolution. Fixed by Jun in r17492.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Ryan J Ollos.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Ryan J Ollos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.