TravisCI and AppVeyor build failures for Python 2.6 and OSX

[Ryan J Ollos]

There may be multiple problems at play here, but I suspect some failures are due to recent security changes to pypi and/or pip.

Here is a TravisCI failure on OSX with 1.2-stable. It seems that easy_install doesn't find virtualenv on pypi.

0.61s$ if [ "$TRAVIS_OS_NAME" = osx ]; then
    test -d $HOME/venv-lib || mkdir $HOME/venv-lib
    PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver \
      --index-url=https://pypi.python.org/simple/ -ZU -d $HOME/venv-lib virtualenv
    PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --no-download $HOME/venv-$pyver
    source $HOME/venv-$pyver/bin/activate
  fi
  
Creating /Users/travis/venv-lib/site.py
Searching for virtualenv
Reading https://pypi.python.org/simple/virtualenv/
Couldn't find index page for 'virtualenv' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or download links found for virtualenv
error: Could not find suitable distribution for Requirement.parse('virtualenv')
/usr/bin/python2.6: No module named virtualenv
/Users/travis/.travis/job_stages: line 62: /Users/travis/venv-2.6/bin/activate: No such file or directory

I'm going to try replacing the easy_install command with pip installed via get_pip.py.

[Jun Omae]

pypi.python.org currently requires TLSv1.2+.

$ { echo 'GET / HTTP/1.1'; echo 'Host: pypi.python.org'; echo; } | openssl s_client -quiet -tls1 -connect pypi.python.org:443
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
verify return:1
depth=0 businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Delaware, serialNumber = 3359300, C = US, ST = New Hampshire, L = Wolfeboro, O = Python Software Foundation, CN = www.python.org
verify return:1
HTTP/1.1 403 TLSv1.2+ is required
Server: Varnish
Retry-After: 0
Content-Type: text/plain; charset=UTF-8
Content-Length: 170
Accept-Ranges: bytes
Date: Tue, 10 Apr 2018 02:46:26 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-nrt6146-NRT
X-Cache: MISS
X-Cache-Hits: 0
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none

Support for TLSv1.0 has been removed, please upgrade to a TLSv1.2+ client. Please see https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html

However, Python 2.6 on Mac OS X uses openssl 0.9.8, which doesn't support TLS 1.2.

$ python2.6
Python 2.6.9 (unknown, Jul 14 2015, 19:46:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> import _ssl
>>> _ssl
<module '_ssl' from '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so'>
>>> ^D
$ otool -L /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so:
        /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

See also: ​https://status.python.org/incidents/hdx7w97m5hr8

[Ryan J Ollos]

The ​warning suggest possible solutions:

$ requires='Genshi==0.7 lxml twill==0.9.1'
  if [ "$build" != minimum ]; then requires="$requires Babel!=2.3.0,!=2.3.1 Pygments docutils textile pytz"; fi
  if [ "$tracdb" = postgres ]; then requires="$requires psycopg2"; fi
  if [ "$tracdb" = mysql ]; then requires="$requires MySQL-python"; fi
  pip install $requires
  python -c 'import sys, pkg_resources as p; p.require(sys.argv[1:])' $requires
  pip list
  
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting Genshi==0.7
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Could not find a version that satisfies the requirement Genshi==0.7 (from versions: )
No matching distribution found for Genshi==0.7
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/travis/venv-2.6/lib/python2.6/site-packages/pkg_resources/__init__.py", line 984, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/travis/venv-2.6/lib/python2.6/site-packages/pkg_resources/__init__.py", line 870, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pytz' distribution was not found and is required by the application
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
argparse (1.4.0)
pip (9.0.3)
setuptools (36.8.0)
wheel (0.29.0)
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning

Another solution might be to use ​pyenv.

[Jun Omae]

Work around is to download virtualenv tarball using curl command.

.travis.yml

@@ -51 +51 @@
   - |
     if [ "$TRAVIS_OS_NAME" = osx ]; then
       test -d $HOME/venv-lib || mkdir $HOME/venv-lib
-      PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver \
-        --index-url=https://pypi.python.org/simple/ -ZU -d $HOME/venv-lib virtualenv
+      if ! PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --help >/dev/null; then
+        curl -Lo /tmp/virtualenv-15.1.0.tar.gz https://github.com/pypa/virtualenv/archive/15.1.0.tar.gz
+        PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver -ZU -d $HOME/venv-lib /tmp/virtualenv-15.1.0.tar.gz
+      fi
       PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --no-download $HOME/venv-$pyver
       source $HOME/venv-$pyver/bin/activate
     fi

[Ryan J Ollos]

Work around is to download virtualenv tarball using curl command.

Hmm.. I tried similarly, but found that later pip commands would fail to install required packages.

[Ryan J Ollos]

​This change to use pyenv ​appears to be working:

@@ -101,10 +101,12 @@ before_install:
   - |
     if [ "$TRAVIS_OS_NAME" = osx ]; then
       test -d $HOME/venv-lib || mkdir $HOME/venv-lib
-      PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver \
-        --index-url=https://pypi.python.org/simple/ -ZU -d $HOME/venv-lib virtualenv
-      PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --no-download $HOME/venv-$pyver
-      source $HOME/venv-$pyver/bin/activate
+      brew install pyenv pyenv-virtualenv
+      eval "$(pyenv init -)"
+      pyver_found="$(pyenv install -l | grep -e $pyver'.[0-9]' | grep -v - | tail -1)"
+      pyenv install $pyver_found
+      pyenv virtualenv $pyver_found venv-$pyver
+      pyenv activate venv-$pyver
     fi
   - |
     case "$pyver" in

Is there a way we can cache the pyenv, pyenv-virtualenv, and python installations so that we only need to create a virtualenv on each run? I don't understand how the caching works on Travis CI.

[Ryan J Ollos]

I haven't tried making any configuration changes to address the ​failures on AppVeyor, but see that pip 9.0.1 is being used. Using 9.0.3 might be enough to fix the issue.

[Jun Omae]

Adding $HOME/.pyenv to cache directories. See also ​https://docs.travis-ci.com/user/caching#Arbitrary-directories.

diff --git a/.travis.yml b/.travis.yml
index 02fa53e14..da602dc67 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,6 +7,7 @@ addons:
 cache:
   directories:
     - "$HOME/.cache/pip"
+    - "$HOME/.pyenv"
     - "$HOME/venv-lib"
 matrix:
   include:

[Ryan J Ollos]

Thanks. I'm testing the cache change and another round of changes that will only install run pyenv install if the version isn't installed to $HOME/.pyenv.

[Ryan J Ollos]

Some findings:

• Initially I was using pyenv-virtualenv, but since we are caching .pyenv the virtualenv would be retained across builds. I reverted to creating the environment with virtualenv.
• Every pip install with Python 2.6.9 results in these warnings:

  /Users/travis/.pyenv/versions/2.6.9/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    SNIMissingWarning
  /Users/travis/.pyenv/versions/2.6.9/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    InsecurePlatformWarning
  

  The ​documentation suggests we could get rid of the warnings by pip install -U idna certifi "cryptography<2.2" pyOpenSSL. The install fails on my macOS (10.13.4 with openssl installed via brew) and I haven't investigated a fix yet. It's probably straightforward to fix, but might defer that since it works for now:

     clang -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/rjollos/.pyenv/versions/2.6.9/include/python2.6 -c build/temp.macosx-10.13-x86_64-2.6/_openssl.c -o build/temp.macosx-10.13-x86_64-2.6/build/temp.macosx-10.13-x86_64-2.6/_openssl.o -Wconversion -Wno-error=sign-conversion
     build/temp.macosx-10.13-x86_64-2.6/_openssl.c:493:10: fatal error: 'openssl/opensslv.h' file not found
     #include <openssl/opensslv.h>
              ^~~~~~~~~~~~~~~~~~~~
     1 error generated.
     error: command 'clang' failed with exit status 1
  
     ----------------------------------------
  

• The block if [ "$TRAVIS_OS_NAME" = osx ]; then takes about 150s when Python is not installed, and about 7s when the Python installation is cached.
• It might be possible to brew install svn, link the Subversion bindings to the virtualenv and run tests on Mac with Subversion for Python 2.7, which SVN 1.9 supports. I'll look at that later.
• Initially I included brew update and brew outdated pyenv || brew upgrade pyenv, however the brew update takes ~90s. The pyenv package ​is updated from homebrew when TravisCI updates its virtual machines, and I doubt we need to check for updates in the interim.
• We could look at using pyenv on Linux too since it would give us more control over the Python versions we are testing against. It might be possible to test against Python 2.5 on 1.0-stable, though I'm not sure that is worth the time to implement.

Proposed changes in [66d17a60/rjollos.git]. I'll rebase and test on 1.0-stable now.

[Ryan J Ollos]

Committed to 1.0-stable in r16473, merged in r16474, r16475.

[Ryan J Ollos]

AppVeyor is running Python 2.6.6, as that was the last 2.6.x release with an installer.

[Ryan J Ollos]

Minor refactoring in r16477, merged in r16480, r16481.

[Ryan J Ollos]

Replying to Ryan J Ollos:

AppVeyor is running Python 2.6.6, as that was the last 2.6.x release with an installer.

I don't have any simple ideas for fixing this, and don't want to spend too much time on it. If there are no better suggestions, I'll just remove the build configuration for Python 2.6.6.

[Jun Omae]

Replying to Ryan J Ollos:

Replying to Ryan J Ollos:

AppVeyor is running Python 2.6.6, as that was the last 2.6.x release with an installer.

I don't have any simple ideas for fixing this, and don't want to spend too much time on it. If there are no better suggestions, I'll just remove the build configuration for Python 2.6.6.

I tried to fix it with manual installation of pyopenssl and pip 9.0.3 but still failing.

• ​https://ci.appveyor.com/project/jun66j5/trac/build/t13006_appveyor.131
• ​https://github.com/jun66j5/trac/commit/305a9c02725dcda2a355eda8806884bc271a9351

I agree removing Python 2.6 from .appveyor.yml.

[Ryan J Ollos]

New ​TravisCI build failures on OSX, which I'm guessing are related to release of pip 10. Pip 10 ​dropped support for Python 2.6. The issue might be fixed in ​#1127, but not yet released (presumably will be pyenv 1.2.4).

Downloading Python-2.6.9.tgz...
-> https://www.python.org/ftp/python/2.6.9/Python-2.6.9.tgz
Installing Python-2.6.9...
patching file setup.py
Hunk #1 succeeded at 354 (offset 9 lines).
patching file ./configure
patching file ./Modules/readline.c
Hunk #1 succeeded at 199 (offset -7 lines).
Hunk #2 succeeded at 698 (offset -51 lines).
Hunk #3 succeeded at 808 (offset -51 lines).
Hunk #4 succeeded at 848 with fuzz 2 (offset -70 lines).
patching file ./setup.py
Hunk #1 succeeded at 1698 (offset 23 lines).
patching file ./Lib/ssl.py
patching file ./Modules/_ssl.c
Installing pip from https://bootstrap.pypa.io/get-pip.py...
error: failed to install pip via get-pip.py
BUILD FAILED (OS X 10.12.6 using python-build 20160602)
Inspect or clean up the working tree at /var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/python-build.20180420155034.2324
Results logged to /var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/python-build.20180420155034.2324.log
Last 10 log lines:
		/Users/travis/.pyenv/versions/2.6.9/share/man/man1/python.1
Traceback (most recent call last):
  File "get-pip.py", line 20649, in <module>
    main()
  File "get-pip.py", line 197, in main
    bootstrap(tmpdir=tmpdir)
  File "get-pip.py", line 82, in bootstrap
    import pip._internal
  File "/var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpkTM0Oy/pip.zip/pip/_internal/__init__.py", line 33, in <module>
AttributeError: 'module' object has no attribute 'OPENSSL_VERSION_NUMBER'
pyenv: version `2.6.9' not installed
/Users/travis/.travis/job_stages: line 64: pip: command not found
/usr/bin/python: No module named virtualenv
/Users/travis/.travis/job_stages: line 66: /Users/travis/venv-2.6.9/bin/activate: No such file or directory

[Ryan J Ollos]

Removed Python 2.6.6 from AppVeyor config in r16512, merged in r16513, r16514.

comment:16 build failures aren't yet seen on production branches, probably due to caching. Let's hope pyenv fixes the issue before it becomes a problem for us.

[Ryan J Ollos]

Not sure why ​test failures suddenly appear, but looks like we might need to call env.shutdown() for test cases in DatabaseFileTestCase.

[Jun Omae]

Replying to Jun Omae:

I tried to fix it with manual installation of pyopenssl and pip 9.0.3 but still failing.

• ​https://ci.appveyor.com/project/jun66j5/trac/build/t13006_appveyor.131
• ​https://github.com/jun66j5/trac/commit/305a9c02725dcda2a355eda8806884bc271a9351

I agree removing Python 2.6 from .appveyor.yml.

Python 2.6.6 for Windows uses openssl 0.9.8 which doesn't support TLSv1.2. So that, pypi is no longer available for Python 2.6.x for Windows since TLSv1.2+ is required.

C> strings.exe C:\python26\DLLs\_ssl.pyd | grep -i openssl
You need to read the OpenSSL FAQ, http://www.openssl.org/support/faq.html
openssl.cnf
openssl_conf
c:\loewis\26.amd64\openssl-0.9.8l\crypto\ec\ec2_smpt.c
.\crypto\ui\ui_openssl.c

[Ryan J Ollos]

Replying to Ryan J Ollos:

Not sure why ​test failures suddenly appear, but looks like we might need to call env.shutdown() for test cases in DatabaseFileTestCase.

Modified test case in r16515, merged in r16516, r16517.

[Ryan J Ollos]

Replying to Ryan J Ollos:

comment:16 build failures aren't yet seen on production branches, probably due to caching. Let's hope pyenv fixes the issue before it becomes a problem for us.

It looks like issue was fixed in ​pyenv 1.2.4. The ​build succeeds if we upgrade pyenv using Brew. However, the update of Brew adds about 100 seconds to the build. I'm looking at whether we can cache the installation of pyenv 1.2.4 and only call brew upgrade pyenv when the cache isn't populated.

[Ryan J Ollos]

comment:21 change applied to 1.0-stable in r16582, to 1.2-stable in r16583.

I've experimented with caching the Brew bottle and formula, and will follow-up with more info on that soon.

[Ryan J Ollos]

Replying to Ryan J Ollos:

I've experimented with caching the Brew bottle and formula, and will follow-up with more info on that soon.

Posted a length question in ​SO:50196723.

[Ryan J Ollos]

Following up on comment:24:ticket:12211, I tried:

• Adding builds@… as a member of the trac-builds@ google group.
• Reverting to an explicit email address line rather than an encrypted email address line (comment:10:ticket:12839).

No success so far. There doesn't seem to be a log associated with any build that shows the notification, so I'm not sure where to look for debug info.

[Ryan J Ollos]

Replying to Ryan J Ollos:

Replying to Ryan J Ollos:

I've experimented with caching the Brew bottle and formula, and will follow-up with more info on that soon.

Posted a length question in ​SO:50196723.

Changes associated with the SO question in [0113bcada/rjollos.git]. I'm not proposing to commit the changes, at least such time that until the consequences of caching homebrew-core are understood.

The alternate idea I propose in the SO question is to install pyenv from source. Another idea is to look at the Brew options to see if the Brew Formula for pyenv can be cached under $HOME.

[Ryan J Ollos]

Replying to Ryan J Ollos:

Another idea is to look at the Brew options to see if the Brew Formula for pyenv can be cached under $HOME.

Proposed changes in [cc1a37727/rjollos.git].

The updated pyenv bottle and formula are cached on the first build from a branch, or after the cache is deleted.

• ​First build after cache deletion
• ​Second build after cache deletion

[Ryan J Ollos]

comment:26 change committed in r16594, merged in r16595, r16596.

[Ryan J Ollos]

According to ​the docs,

Homebrew is installed and updated every time the virtual machines are updated.

I tested on a branch with the latest image, OSX Homebrew has pyenv version 1.2.4. Therefore I've rolled back r16594 in r16925, merged in r16926, r16927.