Support PyPy

[kallisti5]

I was able to get trac running under PyPy in ​https://foss.heptapod.net/pypy/pypy/-/issues/3528

Originally it was crashing on ticket queries, but switching psycopg2 to psycopg2cffi via this patch below resolved the crashing:

trac/db/postgres_backend.py

@@ -31 +31 @@
 from trac.util.text import empty, exception_to_unicode, to_unicode
 from trac.util.translation import _
 
+# Attempt to use psycopg2cffi in compatibility mode if available
+# this enables usage of psycopg2cffi under pypy
+try:
+    from psycopg2cffi import compat
+    compat.register()
+except ImportError:
+    pass
+
 try:
     import psycopg2 as psycopg
     import psycopg2.extensions

[Jun Omae]

Trac depends on __del__ behavior of CPython. Many failures should exist in unit and functional tests on PyPy, even if applying the changes.

See ​Differences between PyPy and CPython — PyPy documentation.

[kallisti5]

Hm. We haven't seen any issues and haven't had any reports of issues. I was looking to trac for some guidance, but the PyPy wiki page just explains what PyPy is.

Trac 1.4.x Performance under python2 has been pretty lackluster lately, the PyPy change gave us a badly needed boost.

As a side note, the trac wiki pages are getting filled with spam. https://trac.edgewall.org/demo-1.4

[Jun Omae]

The existence of PyPy wiki page doesn't mean Trac supporting pypy.

Trac doesn't currently support pypy due to depending __del__ behavior.

unit-test in trunk with pypy3.7

Python: /home/jun66j5/venv/pypy3/bin/python

  Package        Version
  ----------------------------------------------------------------------------
  Python       : 3.7.10 (7.3.5+dfsg-1~ppa1~ubuntu16.04, May 23 2021, 14:57:28)
               : [PyPy 7.3.5 with GCC 5.4.0 20160609]
  Setuptools   : 42.0.2
  Pip          : 19.3.1
  Wheel        : 0.33.6
  Jinja2       : 3.0.1
  Babel        : 2.9.1
  sqlite3      : 2.6.0 (3.11.0)
  PySqlite3    : not installed
  PyMySQL      : 1.0.2
  Psycopg2     : not installed
  SVN bindings : not installed
  Mercurial    : not installed
  Pygments     : 2.9.0
  Textile      : 4.0.1
  Pytz         : 2021.1
  Docutils     : 0.17.1
  Selenium     : 3.141.0
  PyTidyLib    : 0.3.2 (5.6.0 libtidy.so.5deb1)
  LXML         : not installed
  coverage     : 5.5

Variables:
  PATH=/home/jun66j5/venv/pypy3/bin:/home/jun66j5/bin:/home/jun66j5/.gem/ruby/2.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
  PYTHONPATH=.
  TRAC_TEST_DB_URI=
  server-options= -p 3000 -a '*,/home/jun66j5/src/trac-htdigest.txt,auth'  -e

External dependencies:
  Git version: git version 2.29.0
  Subversion version: 1.9.3

python  setup.py egg_info
running egg_info
writing Trac.egg-info/PKG-INFO
writing dependency_links to Trac.egg-info/dependency_links.txt
writing entry points to Trac.egg-info/entry_points.txt
writing requirements to Trac.egg-info/requires.txt
writing top-level names to Trac.egg-info/top_level.txt
reading manifest file 'Trac.egg-info/SOURCES.txt'
writing manifest file 'Trac.egg-info/SOURCES.txt'
grep -q '^min_python = (2, [0-5])' setup.py || ( \
    ! grep -r --include='*.py' '\<import *with_statement\>' trac tracopt contrib && \
    ! grep -r --include='*.py' '\<except *.*, *[A-Za-z0-9_]\+:' trac tracopt contrib \
)
SKIP_FUNCTIONAL_TESTS=1 python  -m unittest  trac.test.test_suite
SKIP: tracopt/versioncontrol/svn/tests/svn_fs.py (no svn bindings)
..........................................................................................................................................................................................................................................................................Exception ignored in: <bound method PooledConnection.__del__ of <trac.db.pool.PooledConnection object at 0x0000000004b0fe50>>
Traceback (most recent call last):
  File "/home/jun66j5/src/tracdev/git/trac/db/pool.py", line 52, in __del__
    self.close()
  File "/home/jun66j5/src/tracdev/git/trac/db/pool.py", line 49, in close
    self._pool._return_cnx(cnx, self._key, self._tid)
  File "/home/jun66j5/src/tracdev/git/trac/db/pool.py", line 162, in _return_cnx
    assert (tid, key) in self._active
AssertionError
Exception ignored in: <bound method PooledConnection.__del__ of <trac.db.pool.PooledConnection object at 0x0000000004b0fef8>>
Traceback (most recent call last):
  File "/home/jun66j5/src/tracdev/git/trac/db/pool.py", line 52, in __del__
    self.close()
  File "/home/jun66j5/src/tracdev/git/trac/db/pool.py", line 49, in close
    self._pool._return_cnx(cnx, self._key, self._tid)
  File "/home/jun66j5/src/tracdev/git/trac/db/pool.py", line 162, in _return_cnx
    assert (tid, key) in self._active
AssertionError
<snip>
----------------------------------------------------------------------
Ran 2542 tests in 84.334s

FAILED (failures=4, skipped=3)
Makefile:477: recipe for target 'unit-test' failed
make: *** [unit-test] Error 1

unit-test in 1.4-stable with pypy2.7

$ make python=pypy unit-test

Python: /home/jun66j5/venv/pypy/bin/python

  Package        Version
  ----------------------------------------------------------------------------
  Python       : 2.7.18 (7.3.5+dfsg-1~ppa1~ubuntu16.04, May 23 2021, 14:34:53)
               : [PyPy 7.3.5 with GCC 5.4.0 20160609]
  Setuptools   : 42.0.2
  Pip          : 19.3.1
  Wheel        : 0.33.6
  Jinja2       : 2.11.3
  Genshi       : 0.7.5 (without speedups)
  Babel        : 2.9.1
  sqlite3      : 2.6.0 (3.11.0)
  PySqlite     : not installed
  PyMySQL      : 0.10.1
  Psycopg2     : not installed
  SVN bindings : not installed
  Mercurial    : not installed
  Pygments     : 2.5.2
  Textile      : 3.0.4
  Pytz         : 2021.1
  Docutils     : 0.17.1
  Twill        : not installed
  LXML         : not installed
  coverage     : 5.5

Variables:
  PATH=/home/jun66j5/venv/pypy/bin:/home/jun66j5/bin:/home/jun66j5/.gem/ruby/2.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
  PYTHONPATH=.
  TRAC_TEST_DB_URI=
  server-options= -p 3000 -a '*,/home/jun66j5/src/trac-htdigest.txt,auth'  -e

External dependencies:
  Git version: git version 2.29.0
  Subversion version: 1.9.3

python  setup.py egg_info
running egg_info
writing Trac.egg-info/PKG-INFO
writing dependency_links to Trac.egg-info/dependency_links.txt
writing entry points to Trac.egg-info/entry_points.txt
writing requirements to Trac.egg-info/requires.txt
writing top-level names to Trac.egg-info/top_level.txt
reading manifest file 'Trac.egg-info/SOURCES.txt'
writing manifest file 'Trac.egg-info/SOURCES.txt'
grep -q '^min_python = (2, [0-5])' setup.py || ( \
    ! grep -r --include='*.py' '\<import *with_statement\>' trac tracopt contrib && \
    ! grep -r --include='*.py' '\<except *.*, *[A-Za-z0-9_]\+:' trac tracopt contrib \
)
SKIP_FUNCTIONAL_TESTS=1 python  -m unittest  trac.test.test_suite
SKIP: tracopt/versioncontrol/svn/tests/svn_fs.py (no svn bindings)
......................................................................................trac/config.py:184: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if name not in self._sections:
trac/config.py:185: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  self._sections[name] = Section(self, name)
trac/config.py:186: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return self._sections[name]
................................................................................................................Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x000000000647a7c8> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x000000000647a838> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x000000000439c058> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x000000000439c0c8> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x00000000032c7398> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x00000000032c7408> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x0000000005559210> ignored
Exception AssertionError: AssertionError() in method __del__ of <trac.db.pool.PooledConnection object at 0x0000000005559280> ignored
....................................................F.............F.F........................................................................................................................................................................................................E.............................................................................................................................................................................................................................................................................................................................F.........................................................FFFFFFF......................F.FF..FF....FF.............................................FFF.F.F.F................................................................................F..................................................................................................................................F.F..............................................................................................................................................................................................................FF...............................................................F............FF...........................................................F..................F.......................................................................................................................................F.....F.F......F..F...........................F...........................................................................................................................................................F....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F..................................F............F.....................................

<snip>

Ran 2539 tests in 86.631s

FAILED (failures=44, errors=1)
Makefile:477: recipe for target 'unit-test' failed
make: *** [unit-test] Error 1

[figaro]

Replying to kallisti5:

Trac 1.4.x Performance under python2 has been pretty lackluster lately, the PyPy change gave us a badly needed boost.

If Trac performance is your chief concern, then please have a look at TracPerformance. It lists a few options to look into. Also optimizing your web server for compressing traffic and caching files is recommended.

[anonymous]

If Trac performance is your chief concern, then please have a look at TracPerformance. It lists a few options to look into. Also optimizing your web server for compressing traffic and caching files is recommended.

Our most common issue is the persistent 100% cpu issue. It occurs daily on 1.4.x (even with minimal to zero load per the http request logs)

After switching from python2 to pypy, the "tracd stuck at 100% cpu" issue has not reoccurred.

[Jun Omae]

If Trac performance is your chief concern, then please have a look at TracPerformance. It lists a few options to look into. Also optimizing your web server for compressing traffic and caching files is recommended.

Our most common issue is the persistent 100% cpu issue. It occurs daily on 1.4.x (even with minimal to zero load per the http request logs)

After switching from python2 to pypy, the "tracd stuck at 100% cpu" issue has not reoccurred.

I think that is probably an issue HTTPServer library and/or CPython. Please create new ticket with the details of the hang (how to reproduce it if possible). At least, I haven't get the same issue with tracd on Linux.

Also, we don't recommend using tracd in production environment. Instead, try mod-wsgi.