Edgewall Software
Modify

Opened 2 years ago

Closed 2 years ago

Last modified 15 months ago

#13006 closed enhancement (fixed)

TravisCI and AppVeyor build failures for Python 2.6 and OSX

Reported by: Ryan J Ollos Owned by: Ryan J Ollos
Priority: normal Milestone: 1.0.17
Component: general Version:
Severity: normal Keywords:
Cc: Branch:
Release Notes:

Fixed TravisCI test failures for Python 2.6.9 due to PyPI security changes. Removed Python 2.6.6 from AppVeyor test configuration.

API Changes:
Internal Changes:

Description

There may be multiple problems at play here, but I suspect some failures are due to recent security changes to pypi and/or pip.

Here is a TravisCI failure on OSX with 1.2-stable. It seems that easy_install doesn't find virtualenv on pypi.

0.61s$ if [ "$TRAVIS_OS_NAME" = osx ]; then
    test -d $HOME/venv-lib || mkdir $HOME/venv-lib
    PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver \
      --index-url=https://pypi.python.org/simple/ -ZU -d $HOME/venv-lib virtualenv
    PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --no-download $HOME/venv-$pyver
    source $HOME/venv-$pyver/bin/activate
  fi
  
Creating /Users/travis/venv-lib/site.py
Searching for virtualenv
Reading https://pypi.python.org/simple/virtualenv/
Couldn't find index page for 'virtualenv' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or download links found for virtualenv
error: Could not find suitable distribution for Requirement.parse('virtualenv')
/usr/bin/python2.6: No module named virtualenv
/Users/travis/.travis/job_stages: line 62: /Users/travis/venv-2.6/bin/activate: No such file or directory

I'm going to try replacing the easy_install command with pip installed via get_pip.py.

Attachments (0)

Change History (28)

comment:1 by Jun Omae, 2 years ago

pypi.python.org currently requires TLSv1.2+.

$ { echo 'GET / HTTP/1.1'; echo 'Host: pypi.python.org'; echo; } | openssl s_client -quiet -tls1 -connect pypi.python.org:443
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
verify return:1
depth=0 businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Delaware, serialNumber = 3359300, C = US, ST = New Hampshire, L = Wolfeboro, O = Python Software Foundation, CN = www.python.org
verify return:1
HTTP/1.1 403 TLSv1.2+ is required
Server: Varnish
Retry-After: 0
Content-Type: text/plain; charset=UTF-8
Content-Length: 170
Accept-Ranges: bytes
Date: Tue, 10 Apr 2018 02:46:26 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-nrt6146-NRT
X-Cache: MISS
X-Cache-Hits: 0
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none

Support for TLSv1.0 has been removed, please upgrade to a TLSv1.2+ client. Please see https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html

However, Python 2.6 on Mac OS X uses openssl 0.9.8, which doesn't support TLS 1.2.

$ python2.6
Python 2.6.9 (unknown, Jul 14 2015, 19:46:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> import _ssl
>>> _ssl
<module '_ssl' from '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so'>
>>> ^D
$ otool -L /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so:
        /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

See also: https://status.python.org/incidents/hdx7w97m5hr8

comment:2 by Ryan J Ollos, 2 years ago

The warning suggest possible solutions:

$ requires='Genshi==0.7 lxml twill==0.9.1'
  if [ "$build" != minimum ]; then requires="$requires Babel!=2.3.0,!=2.3.1 Pygments docutils textile pytz"; fi
  if [ "$tracdb" = postgres ]; then requires="$requires psycopg2"; fi
  if [ "$tracdb" = mysql ]; then requires="$requires MySQL-python"; fi
  pip install $requires
  python -c 'import sys, pkg_resources as p; p.require(sys.argv[1:])' $requires
  pip list
  
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting Genshi==0.7
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Could not find a version that satisfies the requirement Genshi==0.7 (from versions: )
No matching distribution found for Genshi==0.7
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/travis/venv-2.6/lib/python2.6/site-packages/pkg_resources/__init__.py", line 984, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/travis/venv-2.6/lib/python2.6/site-packages/pkg_resources/__init__.py", line 870, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pytz' distribution was not found and is required by the application
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
argparse (1.4.0)
pip (9.0.3)
setuptools (36.8.0)
wheel (0.29.0)
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/Users/travis/venv-2.6/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning

Another solution might be to use pyenv.

comment:3 by Jun Omae, 2 years ago

Work around is to download virtualenv tarball using curl command.

  • .travis.yml

    diff --git a/.travis.yml b/.travis.yml
    index 1b5c5ce52..aae432293 100644
    a b before_install:  
    5151  - |
    5252    if [ "$TRAVIS_OS_NAME" = osx ]; then
    5353      test -d $HOME/venv-lib || mkdir $HOME/venv-lib
    54       PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver \
    55         --index-url=https://pypi.python.org/simple/ -ZU -d $HOME/venv-lib virtualenv
     54      if ! PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --help >/dev/null; then
     55        curl -Lo /tmp/virtualenv-15.1.0.tar.gz https://github.com/pypa/virtualenv/archive/15.1.0.tar.gz
     56        PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver -ZU -d $HOME/venv-lib /tmp/virtualenv-15.1.0.tar.gz
     57      fi
    5658      PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --no-download $HOME/venv-$pyver
    5759      source $HOME/venv-$pyver/bin/activate
    5860    fi

comment:4 by Ryan J Ollos, 2 years ago

Work around is to download virtualenv tarball using curl command.

Hmm.. I tried similarly, but found that later pip commands would fail to install required packages.

comment:5 by Ryan J Ollos, 2 years ago

This change to use pyenv appears to be working:

@@ -101,10 +101,12 @@ before_install:
   - |
     if [ "$TRAVIS_OS_NAME" = osx ]; then
       test -d $HOME/venv-lib || mkdir $HOME/venv-lib
-      PYTHONPATH=$HOME/venv-lib /usr/bin/easy_install-$pyver \
-        --index-url=https://pypi.python.org/simple/ -ZU -d $HOME/venv-lib virtualenv
-      PYTHONPATH=$HOME/venv-lib /usr/bin/python$pyver -m virtualenv --no-download $HOME/venv-$pyver
-      source $HOME/venv-$pyver/bin/activate
+      brew install pyenv pyenv-virtualenv
+      eval "$(pyenv init -)"
+      pyver_found="$(pyenv install -l | grep -e $pyver'.[0-9]' | grep -v - | tail -1)"
+      pyenv install $pyver_found
+      pyenv virtualenv $pyver_found venv-$pyver
+      pyenv activate venv-$pyver
     fi
   - |
     case "$pyver" in

Is there a way we can cache the pyenv, pyenv-virtualenv, and python installations so that we only need to create a virtualenv on each run? I don't understand how the caching works on Travis CI.

comment:6 by Ryan J Ollos, 2 years ago

I haven't tried making any configuration changes to address the failures on AppVeyor, but see that pip 9.0.1 is being used. Using 9.0.3 might be enough to fix the issue.

comment:7 by Jun Omae, 2 years ago

Adding $HOME/.pyenv to cache directories. See also https://docs.travis-ci.com/user/caching#Arbitrary-directories.

diff --git a/.travis.yml b/.travis.yml
index 02fa53e14..da602dc67 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,6 +7,7 @@ addons:
 cache:
   directories:
     - "$HOME/.cache/pip"
+    - "$HOME/.pyenv"
     - "$HOME/venv-lib"
 matrix:
   include:

comment:8 by Ryan J Ollos, 2 years ago

Thanks. I'm testing the cache change and another round of changes that will only install run pyenv install if the version isn't installed to $HOME/.pyenv.

comment:9 by Ryan J Ollos, 2 years ago

Some findings:

  • Initially I was using pyenv-virtualenv, but since we are caching .pyenv the virtualenv would be retained across builds. I reverted to creating the environment with virtualenv.
  • Every pip install with Python 2.6.9 results in these warnings:
    /Users/travis/.pyenv/versions/2.6.9/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      SNIMissingWarning
    /Users/travis/.pyenv/versions/2.6.9/lib/python2.6/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecurePlatformWarning
    
    The documentation suggests we could get rid of the warnings by pip install -U idna certifi "cryptography<2.2" pyOpenSSL. The install fails on my macOS (10.13.4 with openssl installed via brew) and I haven't investigated a fix yet. It's probably straightforward to fix, but might defer that since it works for now:
       clang -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/rjollos/.pyenv/versions/2.6.9/include/python2.6 -c build/temp.macosx-10.13-x86_64-2.6/_openssl.c -o build/temp.macosx-10.13-x86_64-2.6/build/temp.macosx-10.13-x86_64-2.6/_openssl.o -Wconversion -Wno-error=sign-conversion
       build/temp.macosx-10.13-x86_64-2.6/_openssl.c:493:10: fatal error: 'openssl/opensslv.h' file not found
       #include <openssl/opensslv.h>
                ^~~~~~~~~~~~~~~~~~~~
       1 error generated.
       error: command 'clang' failed with exit status 1
    
       ----------------------------------------
    
  • The block if [ "$TRAVIS_OS_NAME" = osx ]; then takes about 150s when Python is not installed, and about 7s when the Python installation is cached.
  • It might be possible to brew install svn, link the Subversion bindings to the virtualenv and run tests on Mac with Subversion for Python 2.7, which SVN 1.9 supports. I'll look at that later.
  • Initially I included brew update and brew outdated pyenv || brew upgrade pyenv, however the brew update takes ~90s. The pyenv package is updated from homebrew when TravisCI updates its virtual machines, and I doubt we need to check for updates in the interim.
  • We could look at using pyenv on Linux too since it would give us more control over the Python versions we are testing against. It might be possible to test against Python 2.5 on 1.0-stable, though I'm not sure that is worth the time to implement.

Proposed changes in [66d17a60/rjollos.git]. I'll rebase and test on 1.0-stable now.

Last edited 2 years ago by Ryan J Ollos (previous) (diff)

comment:10 by Ryan J Ollos, 2 years ago

Committed to 1.0-stable in r16473, merged in r16474, r16475.

comment:11 by Ryan J Ollos, 2 years ago

AppVeyor is running Python 2.6.6, as that was the last 2.6.x release with an installer.

comment:12 by Ryan J Ollos, 2 years ago

Minor refactoring in r16477, merged in r16480, r16481.

in reply to:  11 ; comment:13 by Ryan J Ollos, 2 years ago

Replying to Ryan J Ollos:

AppVeyor is running Python 2.6.6, as that was the last 2.6.x release with an installer.

I don't have any simple ideas for fixing this, and don't want to spend too much time on it. If there are no better suggestions, I'll just remove the build configuration for Python 2.6.6.

in reply to:  13 ; comment:14 by Jun Omae, 2 years ago

Replying to Ryan J Ollos:

Replying to Ryan J Ollos:

AppVeyor is running Python 2.6.6, as that was the last 2.6.x release with an installer.

I don't have any simple ideas for fixing this, and don't want to spend too much time on it. If there are no better suggestions, I'll just remove the build configuration for Python 2.6.6.

I tried to fix it with manual installation of pyopenssl and pip 9.0.3 but still failing.

I agree removing Python 2.6 from .appveyor.yml.

comment:15 by Ryan J Ollos, 2 years ago

Owner: set to Ryan J Ollos
Release Notes: modified (diff)
Status: newassigned

comment:16 by Ryan J Ollos, 2 years ago

New TravisCI build failures on OSX, which I'm guessing are related to release of pip 10. Pip 10 dropped support for Python 2.6. The issue might be fixed in #1127, but not yet released (presumably will be pyenv 1.2.4).

Downloading Python-2.6.9.tgz...
-> https://www.python.org/ftp/python/2.6.9/Python-2.6.9.tgz
Installing Python-2.6.9...
patching file setup.py
Hunk #1 succeeded at 354 (offset 9 lines).
patching file ./configure
patching file ./Modules/readline.c
Hunk #1 succeeded at 199 (offset -7 lines).
Hunk #2 succeeded at 698 (offset -51 lines).
Hunk #3 succeeded at 808 (offset -51 lines).
Hunk #4 succeeded at 848 with fuzz 2 (offset -70 lines).
patching file ./setup.py
Hunk #1 succeeded at 1698 (offset 23 lines).
patching file ./Lib/ssl.py
patching file ./Modules/_ssl.c
Installing pip from https://bootstrap.pypa.io/get-pip.py...
error: failed to install pip via get-pip.py
BUILD FAILED (OS X 10.12.6 using python-build 20160602)
Inspect or clean up the working tree at /var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/python-build.20180420155034.2324
Results logged to /var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/python-build.20180420155034.2324.log
Last 10 log lines:
		/Users/travis/.pyenv/versions/2.6.9/share/man/man1/python.1
Traceback (most recent call last):
  File "get-pip.py", line 20649, in <module>
    main()
  File "get-pip.py", line 197, in main
    bootstrap(tmpdir=tmpdir)
  File "get-pip.py", line 82, in bootstrap
    import pip._internal
  File "/var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpkTM0Oy/pip.zip/pip/_internal/__init__.py", line 33, in <module>
AttributeError: 'module' object has no attribute 'OPENSSL_VERSION_NUMBER'
pyenv: version `2.6.9' not installed
/Users/travis/.travis/job_stages: line 64: pip: command not found
/usr/bin/python: No module named virtualenv
/Users/travis/.travis/job_stages: line 66: /Users/travis/venv-2.6.9/bin/activate: No such file or directory

comment:17 by Ryan J Ollos, 2 years ago

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Removed Python 2.6.6 from AppVeyor config in r16512, merged in r16513, r16514.

comment:16 build failures aren't yet seen on production branches, probably due to caching. Let's hope pyenv fixes the issue before it becomes a problem for us.

comment:18 by Ryan J Ollos, 2 years ago

Not sure why test failures suddenly appear, but looks like we might need to call env.shutdown() for test cases in DatabaseFileTestCase.

in reply to:  14 comment:19 by Jun Omae, 2 years ago

Replying to Jun Omae:

I tried to fix it with manual installation of pyopenssl and pip 9.0.3 but still failing.

I agree removing Python 2.6 from .appveyor.yml.

Python 2.6.6 for Windows uses openssl 0.9.8 which doesn't support TLSv1.2. So that, pypi is no longer available for Python 2.6.x for Windows since TLSv1.2+ is required.

C> strings.exe C:\python26\DLLs\_ssl.pyd | grep -i openssl
You need to read the OpenSSL FAQ, http://www.openssl.org/support/faq.html
openssl.cnf
openssl_conf
c:\loewis\26.amd64\openssl-0.9.8l\crypto\ec\ec2_smpt.c
.\crypto\ui\ui_openssl.c

in reply to:  18 comment:20 by Ryan J Ollos, 2 years ago

Replying to Ryan J Ollos:

Not sure why test failures suddenly appear, but looks like we might need to call env.shutdown() for test cases in DatabaseFileTestCase.

Modified test case in r16515, merged in r16516, r16517.

in reply to:  17 comment:21 by Ryan J Ollos, 2 years ago

Replying to Ryan J Ollos:

comment:16 build failures aren't yet seen on production branches, probably due to caching. Let's hope pyenv fixes the issue before it becomes a problem for us.

It looks like issue was fixed in pyenv 1.2.4. The build succeeds if we upgrade pyenv using Brew. However, the update of Brew adds about 100 seconds to the build. I'm looking at whether we can cache the installation of pyenv 1.2.4 and only call brew upgrade pyenv when the cache isn't populated.

comment:22 by Ryan J Ollos, 2 years ago

comment:21 change applied to 1.0-stable in r16582, to 1.2-stable in r16583.

I've experimented with caching the Brew bottle and formula, and will follow-up with more info on that soon.

in reply to:  22 ; comment:23 by Ryan J Ollos, 2 years ago

Replying to Ryan J Ollos:

I've experimented with caching the Brew bottle and formula, and will follow-up with more info on that soon.

Posted a length question in SO:50196723.

comment:24 by Ryan J Ollos, 2 years ago

Following up on comment:24:ticket:12211, I tried:

  • Adding builds@… as a member of the trac-builds@ google group.
  • Reverting to an explicit email address line rather than an encrypted email address line (comment:10:ticket:12839).

No success so far. There doesn't seem to be a log associated with any build that shows the notification, so I'm not sure where to look for debug info.

in reply to:  23 ; comment:25 by Ryan J Ollos, 2 years ago

Replying to Ryan J Ollos:

Replying to Ryan J Ollos:

I've experimented with caching the Brew bottle and formula, and will follow-up with more info on that soon.

Posted a length question in SO:50196723.

Changes associated with the SO question in [0113bcada/rjollos.git]. I'm not proposing to commit the changes, at least such time that until the consequences of caching homebrew-core are understood.

The alternate idea I propose in the SO question is to install pyenv from source. Another idea is to look at the Brew options to see if the Brew Formula for pyenv can be cached under $HOME.

Last edited 2 years ago by Ryan J Ollos (previous) (diff)

in reply to:  25 comment:26 by Ryan J Ollos, 2 years ago

Replying to Ryan J Ollos:

Another idea is to look at the Brew options to see if the Brew Formula for pyenv can be cached under $HOME.

Proposed changes in [cc1a37727/rjollos.git].

The updated pyenv bottle and formula are cached on the first build from a branch, or after the cache is deleted.

comment:27 by Ryan J Ollos, 2 years ago

comment:26 change committed in r16594, merged in r16595, r16596.

comment:28 by Ryan J Ollos, 15 months ago

According to the docs,

Homebrew is installed and updated every time the virtual machines are updated.

I tested on a branch with the latest image, OSX Homebrew has pyenv version 1.2.4. Therefore I've rolled back r16594 in r16925, merged in r16926, r16927.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Ryan J Ollos.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Ryan J Ollos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.