Functional test is failed in AppVeyor on Python 2.7.17

[Jun Omae]

======================================================================
ERROR: runTest (trac.tests.functional.testcases.RegressionTestTicket3663)
Regression test for non-UTF-8 PATH_INFO (#3663)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\trac\trac\tests\functional\testcases.py", line 301, in runTest
    self._tester.go_to_wiki(u'été'.encode('latin1'))
  File "C:\projects\trac\trac\tests\functional\tester.py", line 148, in go_to_wiki
    self.go_to_url(wiki_url)
  File "C:\projects\trac\trac\tests\functional\tester.py", line 117, in go_to_url
    tc.go(url)
  File "C:\Python27\lib\site-packages\twill\commands.py", line 112, in go
    browser.go(url)
  File "C:\Python27\lib\site-packages\twill\browser.py", line 113, in go
    self._journey('open', u)
  File "C:\Python27\lib\site-packages\twill\browser.py", line 523, in _journey
    r = func(*args, **kwargs)
  File "C:\Python27\lib\site-packages\twill\other_packages\_mechanize_dist\_mechanize.py", line 212, in open
    return self._mech_open(url, data)
  File "C:\Python27\lib\site-packages\twill\other_packages\_mechanize_dist\_mechanize.py", line 238, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "C:\Python27\lib\site-packages\twill\other_packages\_mechanize_dist\_opener.py", line 181, in open
    response = urlopen(self, req, data)
  File "C:\Python27\lib\urllib2.py", line 447, in _open
    '_open', req)
  File "C:\Python27\lib\urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\site-packages\twill\_browser.py", line 16, in http_open
    return self.do_open(wsgi_intercept.WSGI_HTTPConnection, req)
  File "C:\Python27\lib\site-packages\twill\other_packages\_mechanize_dist\_http.py", line 674, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "C:\Python27\lib\httplib.py", line 1069, in request
    self._send_request(method, url, body, headers)
  File "C:\Python27\lib\httplib.py", line 1103, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Python27\lib\httplib.py", line 940, in putrequest
    self._validate_path(url)
  File "C:\Python27\lib\httplib.py", line 1030, in _validate_path
    raise InvalidURL(msg)
InvalidURL: URL can't contain control characters. '/wiki/\xe9t\xe9' (found at least '\xe9')

See also:

• ​https://ci.appveyor.com/project/edgewall-org/trac/builds/29511471/job/oo37getxgxfh1g7d
• ​https://github.com/python/cpython/commit/f5b1abbb3b0083381925dcd5898ae6d019224826

[Ryan J Ollos]

Similar failures on ​Travis CI

======================================================================
ERROR: runTest (trac.tests.functional.testcases.RegressionTestTicket3663)
Regression test for non-UTF-8 PATH_INFO (#3663)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/build/edgewall/trac/trac/tests/functional/testcases.py", line 257, in runTest
    self._tester.go_to_wiki(u'été'.encode('latin1'))
  File "/Users/travis/build/edgewall/trac/trac/tests/functional/tester.py", line 141, in go_to_wiki
    self.go_to_url(wiki_url)
  File "/Users/travis/build/edgewall/trac/trac/tests/functional/tester.py", line 110, in go_to_url
    tc.go(url)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/commands.py", line 112, in go
    browser.go(url)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/browser.py", line 113, in go
    self._journey('open', u)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/browser.py", line 523, in _journey
    r = func(*args, **kwargs)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/other_packages/_mechanize_dist/_mechanize.py", line 212, in open
    return self._mech_open(url, data)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/other_packages/_mechanize_dist/_mechanize.py", line 238, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/other_packages/_mechanize_dist/_opener.py", line 181, in open
    response = urlopen(self, req, data)
  File "/Users/travis/.pyenv/versions/2.7.17/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/Users/travis/.pyenv/versions/2.7.17/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/_browser.py", line 16, in http_open
    return self.do_open(wsgi_intercept.WSGI_HTTPConnection, req)
  File "/Users/travis/venv-2.7.17/lib/python2.7/site-packages/twill/other_packages/_mechanize_dist/_http.py", line 674, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/Users/travis/.pyenv/versions/2.7.17/lib/python2.7/httplib.py", line 1069, in request
    self._send_request(method, url, body, headers)
  File "/Users/travis/.pyenv/versions/2.7.17/lib/python2.7/httplib.py", line 1103, in _send_request
    self.putrequest(method, url, **skips)
  File "/Users/travis/.pyenv/versions/2.7.17/lib/python2.7/httplib.py", line 940, in putrequest
    self._validate_path(url)
  File "/Users/travis/.pyenv/versions/2.7.17/lib/python2.7/httplib.py", line 1030, in _validate_path
    raise InvalidURL(msg)
InvalidURL: URL can't contain control characters. '/wiki/\xe9t\xe9' (found at least '\xe9')
----------------------------------------------------------------------
Ran 178 tests in 326.222s
FAILED (errors=1)
SKIP: versioncontrol/tests/functional.py (no svn bindings)
make: *** [functional-test] Error 1
The command "if [ "$TRAVIS_BUILD_STAGE_NAME" = Test ]; then make Trac.egg-info compile unit-test functional-test; fi" exited with 2.
cache.2
store build cache
Done. Your build exited with 1.

[Jun Omae]

The exception is raised from HTTPConnection._validate_path in httplib module. It is able to avoid to override _validate_path of HTTPConnection subclass but we cannot override the method because httplib is used from mechanize module bundled in twill module.

Instead, we could avoid the exception by monkey-patch of httplib._contains_disallowed_url_pchar_re.

Results:

• Python 2.7.17 on OS X: ​https://travis-ci.org/jun66j5/trac/jobs/628593118
• Python 2.7.12 on Ubuntu: ​https://travis-ci.org/jun66j5/trac/jobs/628593113

trac/tests/functional/testcases.py

@@ -237 +237 @@
 
         Verify that URLs not encoded with UTF-8 are reported as invalid.
         """
-        # invalid PATH_INFO
-        self._tester.go_to_wiki(u'été'.encode('latin1'))
-        tc.code(404)
-        tc.find('Invalid URL encoding')
-        # invalid SCRIPT_NAME
-        tc.go(u'été'.encode('latin1'))
-        tc.code(404)
-        tc.find('Invalid URL encoding')
+        import httplib
+        try:
+            # Work around for InvalidURL since Python 2.7.17 (#13233)
+            saved_re = \
+                httplib._contains_disallowed_url_pchar_re \
+                if hasattr(httplib, '_contains_disallowed_url_pchar_re') \
+                else None
+            if saved_re:
+                httplib._contains_disallowed_url_pchar_re = \
+                    re.compile(r'[\x00-\x20]')
+            # invalid PATH_INFO
+            self._tester.go_to_wiki(u'été'.encode('latin1'))
+            tc.code(404)
+            tc.find('Invalid URL encoding')
+            # invalid SCRIPT_NAME
+            tc.go(u'été'.encode('latin1'))
+            tc.code(404)
+            tc.find('Invalid URL encoding')
+        finally:
+            if saved_re:
+                httplib._contains_disallowed_url_pchar_re = saved_re
 
 
 class RegressionTestTicket6318(FunctionalTwillTestCaseSetup):

[Jun Omae]

Committed in [17223] and merged in [17224-17226].