Edgewall Software
Modify

Opened 9 years ago

Closed 3 years ago

Last modified 11 months ago

#12130 closed enhancement (fixed)

Add Python 3 compatibility

Reported by: Tim Graham <timograham@…> Owned by: Jun Omae
Priority: normal Milestone: 1.5.2
Component: general Version:
Severity: normal Keywords: python3
Cc: Jun Omae, gabriele.svelto@…, t17-0by@…, leho@…, archaerolog@…, andreas.haas@…, mbilderbeek@… Branch: 1.5-py3-tidylib.9
Release Notes:

Changed Python compatibility to 3.5+.

API Changes:
Internal Changes:

Description

As discussed on trac-dev, I'll implement Python 3 compatibility using the six library. There is also #10083 "Make Trac work with Python 3", but Ryan suggested a separate ticket to track my work.

Attachments (4)

2020-01-12-poc-v2-trac-trunk-patch-add-python3-support.patch.gz (123.1 KB ) - added by Rick van der Zwet <info@…> 4 years ago.
Python359_Jinja211.txt (39.7 KB ) - added by Ryan J Ollos 4 years ago.
Python 3.5.9 and Jinja2 2.11
Python359_Jinja2103.txt (8.9 KB ) - added by Ryan J Ollos 4 years ago.
Python 3.5.9 and Jinja2 2.10.3
Screen Shot 2020-12-17 at 23.14.05.jpg (19.4 KB ) - added by Ryan J Ollos 3 years ago.

Download all attachments as: .zip

Change History (128)

comment:1 by Tim Graham <timograham@…>, 9 years ago

Cleanups that don't require six (commits in this branch be merged ASAP): https://github.com/timgraham/trac/pull/1

Commits that require six (probably best to wait to merge this branch until the test suite is passing on Python 3): https://github.com/timgraham/trac/pull/2

comment:2 by Ryan J Ollos, 9 years ago

I have no direct experience with six, but it makes sense to me that we should use it. The main thing I would keep in mind is to organize the imports in a way that it will be easy to remove the dependency on six in the future, when we drop Python 2.7 support.

comment:3 by Ryan J Ollos, 9 years ago

Milestone: 1.2

comment:4 by Tim Graham <timograham@…>, 9 years ago

I didn't see anything explicit in the style guide, but the order of imports seems to be [builtins] [third-party] [internal] with import ... sorted before from ... import.

For example, would this be correct (under the third-party section):

import six
from genshi.builder import tag, Element
from six import StringIO

in reply to:  4 comment:5 by Ryan J Ollos, 9 years ago

Replying to Tim Graham <timograham@…>:

I didn't see anything explicit in the style guide, but the order of imports seems to be [builtins] [third-party] [internal] with import ... sorted before from ... import.

Yeah, we follow PEP-0008. Although you'll probably find a bit of code that doesn't comply.

For example, would this be correct (under the third-party section):

import six
from genshi.builder import tag, Element
from six import StringIO

That's how I'd order them, but I don't think we have been explicit about that anywhere and you'll see some variation in the codebase. I say, go with that unless anyone says differently in this ticket.

comment:6 by Jun Omae, 9 years ago

Cc: Jun Omae added

Updated https://github.com/jun66j5/trac/commits/python3 branch (rebased on current trunk and reworked with six module).

  • initenv works well.
  • tracd doesn't work.
    AttributeError: 'HTTPMessage' object has no attribute 'typeheader'
    
  • unit tests can run on Python 2.6, 2.7 and 3.3.
    Python: /home/jun66j5/venv/py33/bin/python
    
      Package        Version
      ------------------------------------------------------------------
      Python       : 3.3.6 (default, Jan 28 2015, 16:27:12)
      [GCC 4.6.3]
      Setuptools   : 3.5.1
      Genshi       : 0.8
      Babel        : not installed
      sqlite3      : 2.6.0
      PySqlite     : not installed
      MySQLdb      : not installed
      Psycopg2     : not installed
      SVN bindings : not installed
      Mercurial    : not installed
      Pygments     : 2.0.2
      Pytz         : 2015.4
      Docutils     : not installed
      Twill        : not installed
      LXML         : not installed
      coverage     : 3.7.1
      figleaf      : not installed
    
    ...
    
    FAILED (failures=182, errors=128)
    SKIP: utils/tests/datefmt.py (no babel installed)
    SKIP: tracopt/versioncontrol/svn/tests/svn_fs.py (no svn bindings)
    make: *** [unit-test] Error 1
    

comment:7 by Tim Graham <timograham@…>, 9 years ago

Should I abandon work on this then?

in reply to:  7 comment:8 by Ryan J Ollos, 9 years ago

Replying to Tim Graham <timograham@…>:

Should I abandon work on this then?

Or just base any future changes on the latest work that Jun has done.

comment:9 by Jun Omae, 9 years ago

Updated https://github.com/jun66j5/trac/commits/python3.

  • Unit tests:
    • FAILED (failures=2) on Python 3.3 with sqlite3 and psycopg2.
        Package        Version
        ------------------------------------------------------------------
        Python       : 3.3.6 (default, Jan 28 2015, 16:27:12)
        [GCC 4.6.3]
        Setuptools   : 3.5.1
        Genshi       : 0.8
        Babel        : not installed
        sqlite3      : 2.6.0
        PySqlite     : not installed
        MySQLdb      : not installed
        Psycopg2     : 2.6.1 (dt dec pq3 ext)
        SVN bindings : not installed
        Mercurial    : not installed
        Pygments     : 2.0.2
        Pytz         : 2015.4
        Docutils     : 0.12
        Twill        : not installed
        LXML         : 3.4.4
        coverage     : 3.7.1
        figleaf      : not installed
      ...
      Ran 1719 tests in 22.166s
      
      FAILED (failures=2)
      make: *** [unit-test] Error 1
      
    • Unit and functional tests pass on Python 2.6.
  • tracd works well with digest authentication.
  • Git repository probably works.

Known issues:

  • Babel leads RuntimeError: maximum recursion depth exceeded. I have no idea to fix it.
    ....
      File "/venv/py33/lib/python3.3/site-packages/babel/support.py", line 252, in __getattr__
        return getattr(self.value, name)
      File "/venv/py33/lib/python3.3/site-packages/babel/support.py", line 182, in value
        value = self._func(*self._args, **self._kwargs)
      File "trac/util/translation.py", line 218, in _ngettext
        trans = self.active.ungettext(singular, plural, num)
    RuntimeError: maximum recursion depth exceeded
    make: *** [unit-test] Error 1
    

in reply to:  9 comment:10 by Jun Omae, 9 years ago

Known issues:

  • Babel leads RuntimeError: maximum recursion depth exceeded. I have no idea to fix it.

I noticed that ugettext and ungettext methods have been removed from gettext.NullTranslations since Python 3. The issue has been fixed.

Last edited 9 years ago by Jun Omae (previous) (diff)

comment:11 by Ryan J Ollos, 9 years ago

Milestone: 1.21.3.1

comment:12 by jim.amberger@…, 8 years ago

I apologize for not being able to follow this conversation but may I inquire as to the status of trac on python3? Is there a working branch I can clone that basically works?

Thanks for your work on this.

comment:13 by Ryan J Ollos, 8 years ago

I doubt the branch discussed in comment:6 is suitable for use in production by even a very proficient Python developer. What's your motivation for running with Python 3?

comment:14 by Jun Omae, 8 years ago

Rebased https://github.com/jun66j5/trac/commits/python3 with current trunk (r14555). Unit tests pass with SQLite, PostgreSQL and MySQL (using pymysql).

in reply to:  14 ; comment:15 by Ryan J Ollos, 8 years ago

Replying to Jun Omae:

Rebased https://github.com/jun66j5/trac/commits/python3 with current trunk (r14555). Unit tests pass with SQLite, PostgreSQL and MySQL (using pymysql).

Nice. Do you have a general idea of when the changes would be committed? Do you think they are close enough that they might be pushed in milestone:1.3.1, or are there major outstanding issues?

in reply to:  13 comment:16 by jim.amberger@…, 8 years ago

My motivation is wanting to use mod-wsgi3 for everything I host rather than go through the configuration headaches of running python2 and python3 applications on the same machine/apache2 install. I will clone Jun Omae's repo in comment:14 and see what happens.

in reply to:  15 ; comment:17 by Jun Omae, 8 years ago

Replying to Ryan J Ollos:

Replying to Jun Omae:

Rebased https://github.com/jun66j5/trac/commits/python3 with current trunk (r14555). Unit tests pass with SQLite, PostgreSQL and MySQL (using pymysql).

Nice. Do you have a general idea of when the changes would be committed? Do you think they are close enough that they might be pushed in milestone:1.3.1, …

Yeah. I think we are ready to push those changes in 1.3.1.

General idea is that we could push with three steps:

  1. 64d9bd475...d1e22ca17: Add Python 3 compatibilities without six module while keeping unit tests pass on Python 2.
  2. 28a607ccc...c7c238121: Add six module to the dependencies. Add Python 3 compatibilities using six module while keeping unit tests pass on Python 2. As the result, it would be able to run unit tests on Python 3 with many failures.
  3. 493b9ccd9...e6d782ad3: Remove test failures on Python 2 and 3 until zero failures.

, or are there major outstanding issues?

There might be good solutions for the following incompatibilities.

  1. dict iterates items with different order between Python 2 and 3. It makes difficult to write unit tests between 2 and 3, for especially Genshi rendering. E.g. 74884b16d and 938118032.
  2. file instance with text mode on Python 3 reads and writes unicode strings. However, bytes strings on Python 2. We could use io.open for text mode on Python 2. In the branch, uses of open and fdopen with text mode are still remaining. E.g. f4e36082c.

in reply to:  17 comment:18 by Christian Boos, 8 years ago

Replying to Jun Omae:

Replying to Ryan J Ollos:

Replying to Jun Omae:

Rebased https://github.com/jun66j5/trac/commits/python3 with current trunk (r14555). Unit tests pass with SQLite, PostgreSQL and MySQL (using pymysql).

Nice. Do you have a general idea of when the changes would be committed? Do you think they are close enough that they might be pushed in milestone:1.3.1, …

Yeah. I think we are ready to push those changes in 1.3.1.

I think Ryan also has a long standing series mostly ready to push on 1.3.1 (#11901), probably that should land first as it will mostly remove stuff we won't have then to convert.

General idea is that we could push with three steps:

General impression, great work!

  1. 64d9bd475...d1e22ca17: Add Python 3 compatibilities without six module while keeping unit tests pass on Python 2.
  1. why not just:
    try:
        get_thread_id = threading.get_ident  # since Python 3.3
    except AttributeError:
        get_thread_id = threading._get_ident
    
  1. there's an import six in trac/versioncontrol/web_ui/log.py
  1. 28a607ccc...c7c238121: Add six module to the dependencies. Add Python 3 compatibilities using six module while keeping unit tests pass on Python 2. As the result, it would be able to run unit tests on Python 3 with many failures.
  • in trac/db/sqlite_backend.py, there's a six.iteritems but not import six

I wonder if it wouldn't be better looking to use from six import even for the iteritems etc.

I see you removed filter(None,...), but filter seems to be still there in 3.4, is it deprecated somehow? Also, by the way, which 3.x version are we targeting? 3.3?

  1. 493b9ccd9...e6d782ad3: Remove test failures on Python 2 and 3 until zero failures.
  • with the jinja2 branch, the attributes will be sorted, so no worries for the tests in this area.
  • lots of the __repr__ methods look alike (with the class name, some value and the value[value.startswith('u'):] trick); maybe we can come up with a helper function
  • all in all, it looks that it will require some practice to get the proper usage of bytes/b''/to_utf8/io.BytesIO vs. unicode/u''/to_unicode/io.StringIO, maybe some guidelines in our TracDev/CodingStyle will help (e.g. hashlib expects bytes, most of email also)

, or are there major outstanding issues?

There might be good solutions for the following incompatibilities.

  1. dict iterates items with different order between Python 2 and 3. It makes difficult to write unit tests between 2 and 3, for especially Genshi rendering. E.g. 74884b16d and 938118032.

Yes, see above, and [a5d0cf70/cboos.git].

  1. file instance with text mode on Python 3 reads and writes unicode strings. However, bytes strings on Python 2. We could use io.open for text mode on Python 2. In the branch, uses of open and fdopen with text mode are still remaining. E.g. f4e36082c.

comment:19 by Ryan J Ollos, 8 years ago

Ubuntu 16.04 is now packaging a fork of MySQL-python (TracDev/ApiChanges/1.3@19):

# apt search python-mysql
Sorting... Done
Full Text Search... Done
bibus/xenial,xenial 1.5.2-4 all
  bibliographic database

python-mysql.connector/xenial,xenial 2.0.4-1 all
  pure Python implementation of MySQL Client/Server protocol

python-mysqldb/xenial 1.3.7-1build2 amd64
  Python interface to MySQL

python-mysqldb-dbg/xenial 1.3.7-1build2 amd64
  Python interface to MySQL (debug extension)

We might want to add mysqlclient to our test matrix in 1.3.x, unless we plan to switch to supporting only pymysql.

comment:20 by Christian Boos, 7 years ago

Milestone: 1.3.11.3.3

I think we should address this after the Jinja2 merge (1.3.2):

  • less work to do with the tests w.r.t. dict ordering differences (comment:18)
  • no conversion needed for the Python code in the templates, as the expressions in Jinja2 are actually not Python, simply "Python-like" and obey to the same syntax regardless of the actual version of Python being used

comment:21 by Ryan J Ollos, 7 years ago

I'm working on rebasing Jun's changes on the latest trunk, which is non-trivial due to all the changes (#11901 in particular). I've started pushing some of the obvious changes that appear appropriate for Python 2.7: [15326-15335]. I'll post most of the rest, including all that depend on the six module, for additional review and testing.

Last edited 7 years ago by Ryan J Ollos (previous) (diff)

comment:22 by Christian Boos, 7 years ago

If I may, I know I've been somehow slow on rebasing my Jinja2 changes, but now that I've pushed my CSS changes and mostly finished with the t.e.o "clean-up", I'm going to tackle that now. And there are some parts of the code that have changed a lot (e.g. trac/util/html.py), or are no longer necessary (all the code in the Genshi template). So maybe at least don't bother fixing the Genshi templates, as the Python code there will just go away.

OTOH, the changes you committed so far all look nice to have!

Last edited 7 years ago by Christian Boos (previous) (diff)

comment:23 by Ryan J Ollos, 7 years ago

Okay, I'll only push changes to py files, and also defer the changes that depend on six.

comment:24 by Ryan J Ollos, 7 years ago

Additional changes in [15336-15337].

in reply to:  21 comment:25 by Ryan J Ollos, 7 years ago

Replying to Ryan J Ollos:

I'm working on rebasing Jun's changes on the latest trunk, which is non-trivial due to all the changes (#11901 in particular).

Now rebasing on the latest trunk with Jinja2, so a few more days.

comment:26 by Ryan J Ollos, 7 years ago

The get_all_columns refactoring from r15332 was applied to 1.2-stable in r15590, merged to trunk in r15591.

comment:27 by Ryan J Ollos, 7 years ago

Additional changes in [15804:15813].

comment:28 by Ryan J Ollos, 7 years ago

I've rebased the changes in log:rjollos.git:python3.2. There are a few test failures for Python 2.7 (due to [93029820/rjollos.git]). There are many test failures with Python 3.5+. Feel free to fork and propose additional changes. I'll keep rebasing this against the trunk and post rebased changes every few weeks or so (like in #11901).

I've squashes the original changes into two primary changesets:

  1. [21dabbf6/rjollos.git]: refactoring that doesn't depend on six
  2. [352d3af3/rjollos.git]: changes that use six

If we can get all the tests passing, I was thinking to push (1) much sooner than (2) (see also comment:17). I could keep rebasing (2) to be sure the tests pass, and push that in milestone 1.3.3 or 1.3.4.

We should update the AppVeyor and TravisCI configs in the branch to run tests with pymysql and Python 3.5, 3.6. I'm unsure what we are going to do for SVN bindings in Python3 (comment:32:ticket:10083 and comment:36:ticket:10083). It also appears there is no twill for Python3, which may necessitate work on #11988, long overdue anyway.

Last edited 7 years ago by Ryan J Ollos (previous) (diff)

comment:29 by Ryan J Ollos, 7 years ago

#10083 closed as a duplicate, but some the ticket has some useful information.

in reply to:  29 comment:30 by ilewismsl, 7 years ago

Replying to Ryan J Ollos:

#10083 closed as a duplicate, but some the ticket has some useful information.

Not sure whether I am missing something, but your note here says that you closed #10083 as duplicate, but it is still open.

comment:31 by Gabriele Svelto <gabriele.svelto@…>, 7 years ago

Cc: gabriele.svelto@… added

comment:32 by Ryan J Ollos, 7 years ago

Regression in r15812 reported and fixed in #12800.

comment:33 by Ryan J Ollos, 7 years ago

Addition of pymysql will be considered in #12821. I'll post a rebase of comment:28 changes after changes in #12821 are committed.

Last edited 7 years ago by Ryan J Ollos (previous) (diff)

in reply to:  33 comment:34 by Ryan J Ollos, 7 years ago

Replying to Ryan J Ollos:

Addition of pymysql will be considered in #12821. I'll post a rebase of comment:28 changes after changes in #12821 are committed.

log:rjollos.git:t12130_python3. Test passing on Python 2.7. Needs work for Python 3.5+.

comment:35 by Ryan J Ollos, 7 years ago

Milestone: 1.3.31.5.1

comment:36 by Ryan J Ollos, 7 years ago

comment:37 by Thomas Lae <t17-0by@…>, 7 years ago

Cc: t17-0by@… added

comment:38 by Tim Graham <timograham@…>, 6 years ago

DONE PR#12 to remove usage of Python's deprecated dircache module (removed in Python 3).

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:39 by Tim Graham <timograham@…>, 6 years ago

DONE PR#15 to remove use of method argument unpacking.

DONE PR#16 to remove ur string literals.

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:40 by Tim Graham <timograham@…>, 6 years ago

DONE PR#17 to make some exception catching Python 3 compatible.

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

in reply to:  38 ; comment:41 by Jun Omae, 6 years ago

Replying to Tim Graham <timograham@…>:

PR to remove usage of Python's deprecated dircache module (removed in Python 3).

Thanks. However, the same changes are already in [644892e1e/rjollos.git] of rjollos.git@t12130_python3.1 branch.

Last edited 6 years ago by Jun Omae (previous) (diff)

in reply to:  40 comment:42 by Jun Omae, 6 years ago

Replying to Tim Graham <timograham@…>:

PR#17 to make some exception catching Python 3 compatible.

Fixed in [16608-16609] on 1.2-stable and trunk. Thanks.

in reply to:  41 ; comment:43 by Tim Graham <timograham@…>, 6 years ago

Replying to Jun Omae:

Replying to Tim Graham <timograham@…>:

PR to remove usage of Python's deprecated dircache module (removed in Python 3).

Thanks. However, the same changes are already in [644892e1e/rjollos.git] of rjollos.git@t12130_python3.1 branch.

I saw that, but I think my PR is simpler. It filters out directories in the first loop rather than keeping them around and filtering them out later.

I'm trying to offer smaller commits that can be more easily reviewed and merged rather than one huge patch.

in reply to:  43 comment:44 by Ryan J Ollos, 6 years ago

Replying to Tim Graham <timograham@…>:

I'm trying to offer smaller commits that can be more easily reviewed and merged rather than one huge patch.

Thank you, that is much appreciated! I will rebase my t12130_python3 branch soon. It needs a lot of work still.

comment:45 by Ryan J Ollos, 6 years ago

Removed use of unpack argument in r16611 (comment:39).

Replaced use of ur prefix in r16622 (comment:39).

Replaced use of dircache in r16612 (comment:38).

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:46 by Tim Graham <timograham@…>, 6 years ago

DONE PR #18 to make numeric literals compatible with Python 3.

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:47 by Ryan J Ollos, 6 years ago

Made numeric literals compatible with Python3 in r16658 (comment:46).

comment:48 by Ryan J Ollos, 6 years ago

Removed PermissionError inheritance from StandardError in r16675. Edited TracDev/Exceptions@21.

Last edited 6 years ago by Ryan J Ollos (previous) (diff)

comment:49 by Jun Omae, 5 years ago

async is a reserved word in Python 3.7:

$ make python=37 status

Python: /venv/py37/bin/python

Traceback (most recent call last):
  File "contrib/make_status.py", line 14, in <module>
    from trac.util.text import print_table, printout
  File "/src/tracdev/git/trac/util/__init__.py", line 45, in <module>
    from trac.util.datefmt import time_now, to_datetime, to_timestamp, utc
  File "/src/tracdev/git/trac/util/datefmt.py", line 56, in <module>
    from trac.util.translation import _, ngettext
  File "/src/tracdev/git/trac/util/translation.py", line 23, in <module>
    from trac.util.html import tag
  File "/src/tracdev/git/trac/util/html.py", line 229
    async=None, autofocus=None, autoplay=None, checked=None, controls=None,
        ^
SyntaxError: invalid syntax
Makefile:92: recipe for target 'status' failed

comment:50 by Christian Boos, 5 years ago

Branch: log:rjollos.git@t12130_python3

comment:51 by Jun Omae, 5 years ago

Branch: log:rjollos.git@t12130_python3log:rjollos.git@t12130_python3.1

in reply to:  49 comment:52 by Christian Boos, 5 years ago

Replying to Jun Omae:

async is a reserved word in Python 3.7:

Fixed in r16846.

comment:53 by Jun Omae, 5 years ago

Recently, Subversion swig-py3 branch is active. I worked to make compatible with Python 2 and 3 using the swig-py3 branch, in https://github.com/jun66j5/trac/tree/python3+swig-py3+r1853738.

comment:54 by anonymous, 5 years ago

Mercurial would also be interesting: https://www.mercurial-scm.org/wiki/Python3

comment:55 by anonymous, 5 years ago

Mercurial 5.0 is scheduled for May 1 as a Python 3 beta release.

in reply to:  55 comment:56 by Jun Omae, 5 years ago

Mercurial 5.0 is released and has beta support for Python 3.

comment:57 by anonymous, 5 years ago

Hi, Fedora Trac rpm maintainer here. How is this coming? Python2 is EOL in 2020 and won't be an option in Fedora in a release or two.

comment:58 by gwync@…, 5 years ago

Bah. Commenting with my email address. Need coffee. :)

comment:59 by Ryan J Ollos, 5 years ago

I've proposed a cut-over to Python 3 for Trac 1.6: gmessage:trac-dev:cgfUUFY1CVo/n4m0cxl1BgAJ. So far, there have been no objections raised. If that's the case, I will pursue that route once Trac 1.4 is released.

comment:60 by gwync@…, 5 years ago

Hopefully there's at least a test release I can use before Fedora 32 is GA, otherwise I might have to drop Trac, which I'd like to avoid, since I use it. :)

comment:61 by Ryan J Ollos, 5 years ago

Milestone: 1.5.1next-dev-1.5.x

in reply to:  53 ; comment:62 by Jun Omae, 4 years ago

Subversion allows to build Python 3 bindings, since https://svn.apache.org/viewvc?view=revision&revision=1869354.

comment:63 by debacle@…, 4 years ago

On 2019-10-14 11:41, Martin wrote on trac-users@…:

Hi,

just a heads-up for users of Trac on Debian:

Debian is in the process of removing all Python 2 libraries. As a result, all applications based on Python 2 need either be ported to Python 3 or get removed very soon.

See: https://lists.debian.org/debian-python/2019/07/msg00080.html and https://pythonclock.org/

I'll try to get 1.2.5 into Debian 910 as an update, but versions greater than 1.2.x will only be in Debian, when Trac supports Python 3.

Cheers

in reply to:  62 comment:64 by Ryan J Ollos, 4 years ago

Replying to Jun Omae:

Subversion allows to build Python 3 bindings, since https://svn.apache.org/viewvc?view=revision&revision=1869354.

Also discussed in 1.14 Release Notes.

comment:65 by M M, 4 years ago

Is anything happening here? I am considering switching to a different ticketing system.

comment:66 by dkg@…, 4 years ago

I'd also like to know more about the status of trac with python3, as trac was removed from debian today: https://tracker.debian.org/news/1092730/removed-123dfsg-1-from-unstable/

comment:67 by lkraav <leho@…>, 4 years ago

Cc: leho@… added

comment:68 by Rick van der Zwet <info@…>, 4 years ago

Good question, I am also puzzled, since FreeBSD is also dropping python2 support soonish, causing trac to be removed over here as well. I tried to make a proof-of-concept patch to port trac to python3 and after a while got to a 'hello-world' state.

How-ever since I am not a trac developer, I have no glue who is coordinating this effort.

by Rick van der Zwet <info@…>, 4 years ago

comment:69 by gena bug <archaerolog@…>, 4 years ago

Cc: archaerolog@… added

comment:70 by Jun Omae, 4 years ago

Proposed changes in log:jomae.git@1.5-py3 (build https://travis-ci.org/jun66j5/trac/builds/639889731).

  • Python 3.5+ required
  • Python 2.7 dropped
    • six is not used
  • Subversion trunk (1.14dev) is required if needed
  • Unit tests:
    • Database backends:
      • SQLite: All tests pass with Python 3.6 through 3.8
      • PostgreSQL: All tests pass with Python 3.8
      • MySQL: All tests except 1 failure pass with Python 3.8
  • Functional tests not working because twill doesn't support Python 3
  • Confirmed tracd and mod_wsgi working fine

Git repository: https://svn.edgewall.org/git/trac/devs/jomae

(I'm still using 1.0-stable. I need 1.0-stable supported both Python 2.7 and 3.6+ for me….)

in reply to:  70 comment:71 by Jun Omae, 4 years ago

Proposed changes in log:jomae.git@1.5-py3 (build https://travis-ci.org/jun66j5/trac/builds/639889731).

Todo:

  • Add building Subversion trunk to .travis.yml
  • Switch to Python 3 from Python 2 in .appveyor.yml

comment:72 by Ryan J Ollos, 4 years ago

Nice! I will do some testing.

comment:73 by Peter Suter, 4 years ago

This is great!

Is it too early / distracting to post about problems encountered with this?

Using Python 3.8 browsing mostly works without problems (wiki, timeline, roadmap, ticket, attachment, query, reports, search, preferences, about), but browsing to the Admin tab failed:

16:06:06 Trac[main] DEBUG: Rendering response with template admin_basics.html
16:06:06 Trac[chrome] ERROR: Jinja2 TypeError error while rendering XML/HTML template
Traceback (most recent call last):
  File "trac-1.5.1.dev0-py3.8.egg\trac\web\chrome.py", line 1565, in iterable_content
    for chunk in stream:
  File "jinja2-2.10.3-py3.8.egg\jinja2\environment.py", line 1271, in __next__
    return self._next()
  File "jinja2-2.10.3-py3.8.egg\jinja2\environment.py", line 1248, in _buffered_generator
    c = next(self._gen)
  File "jinja2-2.10.3-py3.8.egg\jinja2\environment.py", line 1045, in generate
    yield self.environment.handle_exception(exc_info, True)
  File "jinja2-2.10.3-py3.8.egg\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "jinja2-2.10.3-py3.8.egg\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\admin\templates\admin_basics.html", line 11, in <module>
    # extends 'admin.html'
  File "jinja2-2.10.3-py3.8.egg\jinja2\environment.py", line 1039, in generate
    for event in self.root_render_func(self.new_context(vars)):
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\admin\templates\admin_basics.html", line 14, in root
    <head>
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\admin\templates\admin.html", line 16, in root
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\templates\layout.html", line 18, in root
    <title>
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\templates\theme.html", line 16, in root
    # block head
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\templates\theme.html", line 141, in block_body
    </div>
  File "trac-1.5.1.dev0-py3.8.egg-tmp\trac\admin\templates\admin.html", line 52, in block_content
    <!--      <div id="tabcontent">
  File "jinja2-2.10.3-py3.8.egg\jinja2\asyncfilters.py", line 45, in wrapper
    return normal_filter(*args, **kwargs)
  File "jinja2-2.10.3-py3.8.egg\jinja2\filters.py", line 852, in do_groupby
    in groupby(sorted(value, key=expr), expr)]
TypeError: '<' not supported between instances of 'dict' and 'dict'

comment:74 by Peter Suter, 4 years ago

(I also tested a small set of plugins, and luckily so far it seems that some worked out of the box and some needed only relatively few simple fixes.)

in reply to:  73 comment:75 by Jun Omae, 4 years ago

Replying to Peter Suter:

This is great!

Is it too early / distracting to post about problems encountered with this?

Using Python 3.8 browsing mostly works without problems (wiki, timeline, roadmap, ticket, attachment, query, reports, search, preferences, about), but browsing to the Admin tab failed:

Thanks for the testing! I missed the issue because functional tests doesn't work. Fix it in log:jomae.git@1.5-py3.

See also results of unit tests:

We need switch to someone from twill to restore functional tests on Python 3 (#11988).

comment:76 by Ryan J Ollos, 4 years ago

With Python 3.8.1 on OSX (reproducible with other Python versions also), 2 tests failures. It seems the QP encoded message is not decoded.

$ python -m unittest trac.ticket.tests.notification.AttachmentNotificationTestCase.test_author_is_obfuscated
F
======================================================================
FAIL: test_author_is_obfuscated (trac.ticket.tests.notification.AttachmentNotificationTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/notification.py", line 1737, in test_author_is_obfuscated
    self.assertIn('Changes (by user@…)', body)
AssertionError: 'Changes (by user@…)' not found in '--===============9162979103760298556==\nMIME-Version: 1.0\nContent-Type: text/plain; charset="utf-8"\nContent-Transfer-Encoding: quoted-printable\n\n#1: Ticket summary\n---------------------+-------------------------\n  Reporter:  user@=E2=80=A6  |      Owner:  < default >\n      Type:  defect  |     Status:\n  Priority:  major   |  Milestone:\n Component:          |    Version:\nResolution:          |   Keywords:\n---------------------+-------------------------\nChanges (by user@=E2=80=A6):\n\n
...
----------------------------------------------------------------------
Ran 1 test in 0.305s

FAILED (failures=1)
$ python -m unittest trac.ticket.tests.notification.AttachmentNotificationTestCase.test_author_full_name
F
======================================================================
FAIL: test_author_full_name (trac.ticket.tests.notification.AttachmentNotificationTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/notification.py", line 1759, in test_author_full_name
    self.assertIn('Changes (by Thę Ußęr)', body)
AssertionError: 'Changes (by Thę Ußęr)' not found in '--===============1073304943626721172==\nMIME-Version: 1.0\nContent-Type: text/plain; charset="utf-8"\nContent-Transfer-Encoding: quoted-printable\n\n#1: Ticket summary\n-----------------------+-------------------------\n  Reporter:  Th=C4=99 U=C3=9F=C4=99r  |      Owner:  < default >\n      Type:  defect    |     Status:\n  Priority:  major     |  Milestone:\n Component:            |    Version:\nResolution:            |   Keywords:\n-----------------------+-------------------------\nChanges (by Th=C4=99 U=C3=9F=C4=99r):\n\n
...
----------------------------------------------------------------------
Ran 1 test in 0.305s

FAILED (failures=1)

comment:77 by Ryan J Ollos, 4 years ago

I removed u prefix in [839475f83/rjollos.git]. So far I don't have Subversion 1.14dev installed, so I haven't executed the Subversion test cases.

in reply to:  76 ; comment:78 by Jun Omae, 4 years ago

Replying to Ryan J Ollos:

With Python 3.8.1 on OSX (reproducible with other Python versions also), 2 tests failures. It seems the QP encoded message is not decoded.

Reproduced it with Python 3.8.1 and 3.7.6 on my OSX environment. The test expects 7bit or 8bit in Content-Transfer-Encoding header. Also, project url is typically TracTest <http://example.org/trac.cgi> in the test but TracTest <http://example.org/trac.cgi/%3CContext%20%7B\'range\'%3A%20%... is generated. That is so weird.

======================================================================
FAIL: test_author_full_name (trac.ticket.tests.notification.AttachmentNotificationTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/groove/src/trac.git/trac/ticket/tests/notification.py", line 1759, in test_author_full_name
    self.assertIn('Changes (by Thę Ußęr)', body)
AssertionError: 'Changes (by Thę Ußęr)' not found in '--===============853248259
9417674180==\nMIME-Version: 1.0\nContent-Type: text/plain; charset="utf-8"\nCont
ent-Transfer-Encoding: quoted-printable\n\n#1: Ticket summary\n-----------------
------+-------------------------\n  Reporter:  Th=C4=99 U=C3=9F=C4=99r  |      O
wner:  < default >\n      Type:  defect    |     Status:\n  Priority:  major
 |  Milestone:\n Component:            |    Version:\nResolution:            |
 Keywords:\n-----------------------+-------------------------\nChanges (by Th=C4
=99 U=C3=9F=C4=99r):\n\n * Attachment "foo.txt" added.\n\n The attachment descri
ption\n--=20\nTicket URL: <http://example.org/trac.cgi/ticket/1>\nTracTest <http
://example.org/trac.cgi/%3CContext%20%7B\'range\'%3A%20%3Cclass%20\'range\'%3E%2
C%20\'dict\'%3A%20%3Cclass%20\'dict\'%3E%2C%20\'lipsum\'%3A%20%3Cfunction%20gene
rate_lorem_ipsum%20at%200x10bcd1280%3E%2C%20\'cycler\'%3A%20%3Cclass%20\'jinja2.
utils.Cycler\'%3E%2C%20\'joiner\'%3A%20%3Cclass%20\'jinja2.utils.Joiner\'%3E%2C%
20\'namespace\'%3A%20%3Cclass%20\'jinja2.utils.Namespace\'%3E%2C%20\'_\'%3A%20%3
...
3A%20\'http%3A/example.org/trac.cgi/ticket/1\'%7D%2C%20\'changes_body\'%3A%20\'%
20*%20Attachment%20%22foo.txt%22%20added.%5Cn%5Cn%20The%20attachment%20descripti
on\'%2C%20\'changes_descr\'%3A%20\'\'%2C%20=\n\'change\'%3A%20%7B\'author\'%3A%2
0\'Th%C4%99%20U%C3%9F%C4%99r\'%7D%7D%20of%20\'ticket_notify_email.txt\'%3E>\nMy
example project\n\n--===============8532482599417674180==--\n'

in reply to:  78 ; comment:79 by Jun Omae, 4 years ago

Replying to Jun Omae:

Reproduced it with Python 3.8.1 and 3.7.6 on my OSX environment.

I noticed that the tests fail with Jinja2 2.11.0 on Linux and OSX but pass with Jinja2 2.10.3. It seems to be due to Jinja2 2.11.0.

Last edited 4 years ago by Jun Omae (previous) (diff)

by Ryan J Ollos, 4 years ago

Attachment: Python359_Jinja211.txt added

Python 3.5.9 and Jinja2 2.11

by Ryan J Ollos, 4 years ago

Attachment: Python359_Jinja2103.txt added

Python 3.5.9 and Jinja2 2.10.3

in reply to:  79 comment:80 by Ryan J Ollos, 4 years ago

Replying to Jun Omae:

I noticed that the tests fail with Jinja2 2.11.0 on Linux and OSX but pass with Jinja2 2.10.3. It seems to be due to Jinja2 2.11.0.

Confirmed here as well, no test failures for Jinja 2.10.3 for Python 3.6.10, 3.7.6 3.8.1.

With Python 3.5.9 and Jinja2 2.10.3, I see a 2 test failures: Python359_Jinja2103.txt

With Python 3.5.9 and Jinja2 2.11, I see a 11 failures: Python359_Jinja211.txt

comment:81 by Jun Omae, 4 years ago

Investigating the behavior, the issue occurs when passing callable instance with __getattr__ method to Jinja2 template (e.g. Href instance).

#13244

in reply to:  77 ; comment:82 by Ryan J Ollos, 4 years ago

Replying to Ryan J Ollos:

I removed u prefix in [839475f83/rjollos.git]. So far I don't have Subversion 1.14dev installed, so I haven't executed the Subversion test cases.

I created a local build of Subversion trunk(1.14dev) @ r1873233 and the Subversion tests pass on OSX.

Minor change to correctly print SVN version with make status:

  • contrib/make_status.py

    diff --git a/contrib/make_status.py b/contrib/make_status.py
    index 5cccff9fb..763e3ac7c 100755
    a b def _svn_version():  
    1818    from svn import core
    1919    version = (core.SVN_VER_MAJOR, core.SVN_VER_MINOR,
    2020               core.SVN_VER_MICRO)
    21     return '%d.%d.%d' % version + core.SVN_VER_TAG
     21    return '%d.%d.%d' % version + str(core.SVN_VER_TAG, 'utf-8')
    2222
    2323PACKAGES = [
    2424    ("Python",            'sys.version'),

in reply to:  82 ; comment:83 by Jun Omae, 4 years ago

Replying to Ryan J Ollos:

Minor change to correctly print SVN version with make status: #!diff...

Yes. I noticed it while adding build steps for Subversion Python bindings on Travis CI: [32585416b/jomae.git].

I think we should catch ImportError in resolve_accessor():

@@ -70,8 +70,10 @@ def resolve_accessor(accessor):
         if attr.endswith('()'):
             version = version()
         return version
-    except Exception:
+    except ImportError:
         return None
+    except Exception as e:
+        return 'unknown ({})'.format(e)

 def shift(prefix, block):
     return '\n'.join(prefix + line for line in block.split('\n') if line)

Also, the unit tests pass with Subversion on Linux and OSX in Travis CI: https://travis-ci.org/jun66j5/trac/builds/646253285

in reply to:  83 ; comment:84 by Jun Omae, 4 years ago

Replying to Jun Omae:

Also, the unit tests pass with Subversion on Linux and OSX in Travis CI: https://travis-ci.org/jun66j5/trac/builds/646253285

Minor issue is Deploy stage is too slow even unless repository is edgewall/trac.

in reply to:  84 comment:85 by Jun Omae, 4 years ago

Replying to Jun Omae:

Minor issue is Deploy stage is too slow even unless repository is edgewall/trac.

Fixed in [8bd167278/jomae.git] which installs only Jinja2 and Babel on deploy stage: Build #211

comment:86 by Ryan J Ollos, 4 years ago

I haven't looked closely at the issue yet, but make jinja yields:

Traceback (most recent call last):
  File "contrib/jinjachecker.py", line 445, in <module>
    main(sys.argv[1:])
  File "contrib/jinjachecker.py", line 96, in main
    status += analyze(template, only, quiet)
  File "contrib/jinjachecker.py", line 111, in analyze
    line_statements, html, html_hints = scan(lines)
  File "contrib/jinjachecker.py", line 185, in scan
    linenum, line = lines.next()
AttributeError: 'enumerate' object has no attribute 'next'
make: *** [jinja] Error 1

in reply to:  86 ; comment:87 by Jun Omae, 4 years ago

Replying to Ryan J Ollos:

I haven't looked closely at the issue yet, but make jinja yields:

Fixed in [adb666962/jomae.git]. Confirmed no errors on jinjachecker.

comment:88 by Ryan J Ollos, 4 years ago

Milestone: next-dev-1.5.x1.5.2

comment:89 by Andreas <andreas.haas@…>, 4 years ago

Cc: andreas.haas@… added

comment:90 by Ryan J Ollos, 4 years ago

Rebased on trunk in log:rjollos.git:1.5-py3-tidylib.2. All tests pass with Python 3.8.2 on OSX, with the exception of two failures noted in comment:12:ticket:11988. I'm investigating a possible workaround.

I'll test with Python 3.5 - 3.7 in coming days.

The tests are taking a long time to run on Travis CI, resulting in build job timeouts.

Other than minor things like that, which can be fixed in next several days, it seems more than good enough to push to trunk after 1.5.1 is released (comment:24:ticket:11988). Do you want to push the work?

If there are no blockers, I could go ahead with that next weekend, after releasing 1.5.1, but please go ahead with the push if you think it's ready, and also since your name should really be on the commit. I was thinking all the changes could be made in a single commit on trunk. Or perhaps separate out the pytidylib change if that can be backported to 1.4-stable. However, at this point I have very little interest in 1.4-stable and favor making the Python 3 work available to users as soon as practical.

Thanks for this massive amount of work. I doubt I could have done it!

comment:91 by Ryan J Ollos, 4 years ago

We should update minimum required versions:

  • Babel 2.2 added support for Python 3.4 and 3.5.
  • psycopg2: Unclear what the minimum version is, but 2.5 added support for Python 3.3, so that may be the minimum required.
  • Textile: Unclear from change log, 2.3 may be a good minimum.
  • docutils: Unclear from change log, but ≥ 14 might be a good choice.
  • setuptools: ≥ 30? At least if we drop < 5.7 we can remove some compatibility code: warn_setuptools_issue.

comment:92 by Ryan J Ollos, 4 years ago

pypi:html2rest doesn't support Python 3. I've experimented with using pypandoc as a replacement. I'll try that again in #12835.

Last edited 4 years ago by Ryan J Ollos (previous) (diff)

comment:93 by Ryan J Ollos, 4 years ago

In the release checklist, reindent is used to fix indentation. pypi:reindent doesn't support Python 3. It appears it's just packaged reindent.py from the Python Core.

Instead, we could use pypi:autopep8. This is also a good opportunity to configure some PEP8 rules that can be checked (with pypi:pycodestyle or similar) and fixed with autopep8. Then, the tools can be used when creating patches to ensure proper formatting. It looks like we'll have to exclude some of the indentation rules (E12x) to maintain our (informal) coding practices. I'll post some examples later.

#13304.

Last edited 4 years ago by Ryan J Ollos (previous) (diff)

comment:94 by Ryan J Ollos, 4 years ago

Latest in log:rjollos.git:1.5-py3-tidylib.4.

  • DONE Need to solve slow selenium test execution on OSX that is resulting in timeouts running on Travis CI.
  • TODO Add tidylib / pytidylib to AppVeyor.
  • TODO Use pytidylib for HTML validation in jinjachecker? (comment:26:ticket:11988)
  • TODO Test failures (comment:29:ticket:11988).
Last edited 3 years ago by Ryan J Ollos (previous) (diff)

comment:95 by Ryan J Ollos, 4 years ago

Subversion 1.14 released, which is needed for using Subversion with the Py3 branch.

comment:96 by Ryan J Ollos, 4 years ago

Summary: Add Python 3 compatibility using sixAdd Python 3 compatibility

in reply to:  91 comment:97 by Ryan J Ollos, 4 years ago

Replying to Ryan J Ollos:

#13310

in reply to:  93 comment:98 by anonymous, 4 years ago

Replying to Ryan J Ollos:

In the release checklist, reindent is used to fix indentation. pypi:reindent doesn't support Python 3. It appears it's just packaged reindent.py from the Python Core.

Related question in SO:62272780.

comment:99 by Ryan J Ollos, 4 years ago

Several related issues have been fixed, but the 4 items in comment:94 still need to be addressed. I've been focused on addressing the test failures and functional test execution timeouts on TravisCI and AppVeyor. Latest is log:rjollos.git:1.5-py3-tidylib.7.

If we could get all the tests passing on TravisCI and AppVeyor I'd favor pushing the work to the trunk before addressing the last set of issues.

comment:100 by Ryan J Ollos, 4 years ago

I saved a Brew versioned formula for Subversion 1.13 in my tap repository. See SO:62822638/121694.

If using Brew on OSX, Subversion 1.13 is needed with Trac < 1.5.2. Subversion 1.14 is linked to Python3.

Last edited 4 years ago by Ryan J Ollos (previous) (diff)

comment:101 by Ryan J Ollos, 4 years ago

I haven't had time to investigate lately, but will try to find some time soon. The comment:99 issues are the blockers to getting this on trunk and releasing 1.5.2. If anyone else can investigate, that would be great.

comment:102 by Manuel Bilderbeek <mbilderbeek@…>, 4 years ago

Cc: mbilderbeek@… added

comment:103 by Christopher Head <bugs@…>, 3 years ago

Cc: bugs@… added

comment:104 by Ryan J Ollos, 3 years ago

I've commented out the platforms for which the tests time-out. After I get a successful test run, I'm going to create the release, probably on Saturday.

  • .travis.yml

    diff --git a/.travis.yml b/.travis.yml
    index 2103b79f03..970b074bfe 100644
    a b jobs:  
    219219  - {os: linux, python: '3.8', env: 'tracdb=postgresql', services: postgresql}
    220220  - {os: linux, python: '3.8', env: 'tracdb=mysql', services: mysql}
    221221  - {os: osx, language: generic, env: 'pyver=3.8 tracdb= build=minimum'}
    222   - {os: osx, language: generic, env: 'pyver=3.8 tracdb='}
    223   - {os: osx, language: generic, env: 'pyver=3.8 tracdb=sqlite'}
    224   - {os: osx, language: generic, env: 'pyver=3.8 tracdb=postgresql'}
     222    #- {os: osx, language: generic, env: 'pyver=3.8 tracdb='}
     223    #- {os: osx, language: generic, env: 'pyver=3.8 tracdb=sqlite'}
     224    #- {os: osx, language: generic, env: 'pyver=3.8 tracdb=postgresql'}
    225225  - stage: deploy
    226226    before_script: skip
    227227    script: skip

Latest changes in log:rjollos.git:1.5-py3-tidylib.8. I'd be happy to integrate any suggestions and improvements before then.

I'm planning to push in a single squashed commit. Feedback welcome.

comment:105 by Jun Omae, 3 years ago

I noticed we could reduce the time of builds using binary packages of homebrew, and the timeout is gone. It is needed to switch osx_image to xcode12.2.

comment:106 by Ryan J Ollos, 3 years ago

That's great. Thanks! I'm running the tests again after rebasing on trunk. If all tests pass I think we could push to trunk: log:rjollos.git:1.5-py3-tidylib.9.

comment:107 by Ryan J Ollos, 3 years ago

Functional tests don't run on AppVeyor. Also, AppVeyor is still using SVN 1.8, but needs to use 1.14 or later.

comment:108 by Ryan J Ollos, 3 years ago

The issue may be related to this behavior. On macOS and Windows make functional-test succeeds:

$ make functional-test
 It looks like you don't have a Makefile.cfg file yet.
 You can get started by doing `cp Makefile.cfg.sample Makefile.cfg'
 and then adapt it to your environment.

Python: /Users/rjollos/.pyenv/shims/python
...
writing manifest file 'Trac.egg-info/SOURCES.txt'
python  -m unittest  trac.tests.functional.test_suite
SKIP: versioncontrol/tests/functional.py (no svn bindings)
initenv /Users/rjollos/Documents/Workspace/trac-dev/trac-py3/testenv/trac sqlite:db/trac.db --config=/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/testenv/config.ini
................

python -m unittest trac.tests.functional.test_suite does not:

$ python  -m unittest  trac.tests.functional.test_suite
SKIP: versioncontrol/tests/functional.py (no svn bindings)
initenv /Users/rjollos/Documents/Workspace/trac-dev/trac-py3/testenv/trac sqlite:db/trac.db --config=/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/testenv/config.ini
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/admin/console.py", line 29, in <module>
    from trac import __version__ as TRAC_VERSION
ModuleNotFoundError: No module named 'trac'

Traceback (most recent call last):
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/main.py", line 101, in __init__
    self.runTests()
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/runner.py", line 176, in run
    test(result)
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/suite.py", line 122, in run
    test(result)
  File "/Users/rjollos/.pyenv/versions/3.8.6/lib/python3.8/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/test.py", line 265, in run
    self.setUp()
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/tests/functional/__init__.py", line 136, in setUp
    self._testenv = self.env_class(env_path, server_port, baseurl)
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/tests/functional/testenv.py", line 80, in __init__
    self.create()
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/tests/functional/testenv.py", line 155, in create
    self._tracadmin('initenv', self.tracdir, self.dburi,
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/tests/functional/testenv.py", line 274, in _tracadmin
    raise Exception("Failed while running trac-admin with arguments "
Exception: Failed while running trac-admin with arguments ('initenv', '/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/testenv/trac', 'sqlite:db/trac.db', '--config=/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/testenv/config.ini').
Exitcode: 1
None

The problem seems to be in rjollos.git/trac/tests/functional/testenv.py@1.5-py3-tidylib.9:266#L263, and is essentially:

$ /Users/rjollos/.pyenv/versions/trac-py386/bin/python /Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/admin/console.py
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/trac-py3/trac/admin/console.py", line 29, in <module>
    from trac import __version__ as TRAC_VERSION
ModuleNotFoundError: No module named 'trac'

Maybe we should use -m trac.admin.console instead of a path?

comment:109 by Ryan J Ollos, 3 years ago

by Ryan J Ollos, 3 years ago

comment:110 by Ryan J Ollos, 3 years ago

On the test vs Test for TRAVIS_BUILD_STAGE_NAME, according to the docs, adding version: ~> 1.0 should enable build config validation, which should result in the variable being "test" rather than "Test". However, this build doesn't show validation being enabled. I have the repository setting enabled which does result in build config enabled. See also comment:19:ticket:13257.

I posted this to the forums.

Last edited 3 years ago by Ryan J Ollos (previous) (diff)

in reply to:  109 ; comment:111 by Ryan J Ollos, 3 years ago

Replying to Ryan J Ollos:

Testing [48680dbd65/rjollos.git].

That doesn't fix the AppVeyor issue, but does allow running using python setup.py egg_info; python -m unittest trac.tests.functional.test_suite. I hadn't noticed that fails from a clean checkout because usually I run python setup.py develop first.

AppVeyor appears to hang at the start of functional tests, however I watched the output of unit-tests and the output is buffered. Meaning, the following is shown for 2 minutes:

SKIP_FUNCTIONAL_TESTS=1 python  -m unittest  trac.test.test_suite

Then the results appear suddenly:

SKIP_FUNCTIONAL_TESTS=1 python  -m unittest  trac.test.test_suite
make.exe : ..........................................................................................................................................................................................................................
.....................................................................................................................................................................................................................................
.....................................................................................................................................................................................................................................
.....................................................................................................................................................................................................................................
.....................................................................................................................................................................................................................................
..................................................................................................................s................................................s.................................................................
............................................................................ssss.....................................................................................................................................................
.....................................................................................................................................................................................................................................
.....................................................................................................................................................................................................................................
.....................................................................................................................................................................................................................................
.........................................................................................................................................................sss......................................................
At C:\projects\trac-6f5m6\contrib\appveyor.ps1:360 char:9
+         & make.exe $goal 2>&1 | Tee-Object -Variable make
+         ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (.................................:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
----------------------------------------------------------------------
Ran 2489 tests in 107.945s
OK (skipped=9)
SKIP: validation of HTML output in functional tests (no tidylib installed)

So more likely that one of the functional tests is hanging.

comment:112 by Ryan J Ollos, 3 years ago

On my Windows 10 VM, the functional test execution hangs at trac.timeline.tests.functional.RegressionTestTicket12946. Looks like we are running out of words on Windows:

>>> from trac.tests.contentgen import random_unique_camel
>>> for i, w in enumerate(range(0, 101)):
...   print(i, random_unique_camel())
...
0 FourTen
1 SixFour
...
97 TenEight
98 FourSix
99 EightThree

We should probably just grab 1k words and put them in a text file for the fallback (SO:49524775).

Last edited 3 years ago by Ryan J Ollos (previous) (diff)

comment:113 by Ryan J Ollos, 3 years ago

Branch: log:rjollos.git@t12130_python3.11.5-py3-tidylib.9

comment:114 by Ryan J Ollos, 3 years ago

Proposed fix in [6593e3d4/rjollos.git]. Words generated on macOS with:

>>> from trac.tests.contentgen import random_word
>>> with open("trac/tests/random_words.txt", "w") as f:
...     f.write(' '.join(random_word() for _ in range(0, 1000)))
Last edited 3 years ago by Ryan J Ollos (previous) (diff)

in reply to:  111 ; comment:115 by Jun Omae, 3 years ago

AppVeyor appears to hang at the start of functional tests, however I watched the output of unit-tests and the output is buffered. Meaning, the following is shown for 2 minutes:

So more likely that one of the functional tests is hanging.

How about using verbose output by testopts=-v?

@@ -359,7 +359,8 @@ function Trac-Tests {
         Write-Host "make $goal"

         Add-AppveyorTest -Name $name -Outcome Running
-        & make.exe $goal 2>&1 | Tee-Object -Variable make
+        # Enable verbose output to avoid to seem hanging job on appveyor
+        & make.exe testopts=-v $goal 2>&1 | Tee-Object -Variable make

         # Determine outcome Passed or Failed
         $outcome = 'Passed'

in reply to:  114 comment:116 by Jun Omae, 3 years ago

Proposed fix in [6593e3d4/rjollos.git]. Words generated on macOS with:

Looks good to me.

Last edited 3 years ago by Ryan J Ollos (previous) (diff)

in reply to:  115 comment:117 by Ryan J Ollos, 3 years ago

Replying to Jun Omae:

How about using verbose output by testopts=-v?

Thanks! I will add it.

comment:118 by Ryan J Ollos, 3 years ago

Release Notes: modified (diff)
Resolution: fixed
Status: newclosed

Changes committed to trunk in r17483.

comment:119 by Ryan J Ollos, 3 years ago

Owner: set to Jun Omae

in reply to:  87 comment:120 by Ryan J Ollos, 3 years ago

Replying to Jun Omae:

Replying to Ryan J Ollos:

I haven't looked closely at the issue yet, but make jinja yields:

Fixed in [adb666962/jomae.git]. Confirmed no errors on jinjachecker.

I didn't adapt changes from #13307. Fixed in r17484.

comment:121 by Jun Omae, 3 years ago

Many thanks, Ryan!

  • TODO Use tarball of Subversion 1.14.0 instead of trunk on linux build on travis-ci
    • Another possibility, we could use packages for subversion 1.14 and python bindings via ppa. However I cannot find it. We could build the packages and publish via ppa for the testing on travis-ci.
  • TODO Use Subversion 1.14 Python bindings for Python 3 on Windows build on appveyor
    • However, it is unable to find pre-built binaries for the Python bindings for Windows.
    • Also, another issue, Subversion 1.14.0 Python bindings with Python 3.8+ on Windows cannot work. The issue has been fixed in 1.14.x branch.

comment:122 by Ted S, 3 years ago

If this is fixed, then the TracInstall needs to be updated. It still says "​Python, version ≥ 2.7 and < 3.0" and I'd like to install Trac with Python3 for a university course I'm developing for Spring Semester. (everything needs to be set up in advance, so I'm starting to prepare now).

comment:123 by anonymous, 3 years ago

TracInstall should say using Python3 is now possible with 1.5/TracInstall but with unstable version caveats.

Last edited 3 years ago by Ryan J Ollos (previous) (diff)

comment:124 by bugs@…, 18 months ago

Cc: bugs@… removed

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.