make extraction fails with TypeError

[Ryan J Ollos]

$ make status

Python: /Users/rjollos/.pyenv/shims/python

/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
  Package        Version
  --------------------------------------------------------------------------------------
  Python       : 3.10.3 (main, Mar 22 2022, 21:01:41) [Clang 13.1.6 (clang-1316.0.21.2)]
  Setuptools   : 62.1.0
  Pip          : 22.0.4
  Wheel        : 0.37.1
  Jinja2       : 3.0.3
  Babel        : 2.9.1
  sqlite3      : 2.6.0 (3.36.0)
  PySqlite3    : not installed
  PyMySQL      : 1.0.2
  Psycopg2     : 2.9.3 (dt dec pq3 ext lo64)
  SVN bindings : 1.15.0 (under development)
  Mercurial    : 6.1
  Pygments     : 2.11.2
  Textile      : 4.0.2
  Pytz         : 2022.1
  Docutils     : 0.17.1
  Selenium     : 4.1.3
  PyTidyLib    : 0.3.2 (5.8.0 /usr/local/lib/libtidy.dylib)
  LXML         : 4.8.0
  coverage     : 6.3.2

Variables:
  PATH=/Users/rjollos/Documents/Workspace/trac-dev/bin:/Users/rjollos/Documents/Workspace/trac-dev/subversion-trunk/local/bin:/usr/local/Cellar/tidy-html5/5.6.0/bin:/Users/rjollos/.pyenv/shims:/usr/local/opt/sqlite/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/usr/local/munki:/opt/X11/bin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/Users/rjollos/.pyenv/bin
  PYTHONPATH=.
  TRAC_TEST_DB_URI=
  server-options= -p 8000 -a '*,/Users/rjollos/Documents/Workspace/trac-dev/tracenvs/htdigest.trac,trac' -r -e /Users/rjollos/Documents/Workspace/trac-dev/tracenvs

External dependencies:
  Git version: git version 2.35.1
  Subversion version: 1.15.0-dev

$ make extraction
python  setup.py  extract_messages  extract_messages_js  extract_messages_tracini
running extract_messages
extracting messages from tracopt/__init__.py
extracting messages from tracopt/perm/__init__.py
extracting messages from tracopt/perm/authz_policy.py
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/teo-rjollos.git/setup.py", line 42, in <module>
    setup(**extra)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/frontend.py", line 481, in run
    for filename, lineno, message, comments, context in extracted:
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 149, in extract_from_dir
    for message_tuple in check_and_call_extract_file(
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    for message_tuple in extract_from_file(
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 243, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 325, in extract
    for lineno, funcname, messages, comments in results:
  File "/Users/rjollos/Documents/Workspace/trac-dev/teo-rjollos.git/trac/dist.py", line 148, in extract_python
    for _ in tokens:
  File "/Users/rjollos/.pyenv/versions/3.10.3/lib/python3.10/tokenize.py", line 498, in _tokenize
    if line[pos] in '#\r\n':           # skip comments or blank lines
TypeError: 'in <string>' requires string as left operand, not int
make: *** [extraction] Error 1

[Jun Omae]

The exception goes away by the following patch:

trac/dist.py

@@ -143 +143 @@
         if 'cleandoc_keywords' in options:
             cleandoc_keywords.update(options['cleandoc_keywords'])
 
-        tokens = generate_tokens(fileobj.readline)
+        tokens = generate_tokens(lambda: fileobj.readline().decode(encoding))
         tok = value = None
         for _ in tokens:
             prev_tok, prev_value = tok, value
@@ -166 +166 @@
                 continue
             elif call_stack == -1 and tok == COMMENT:
                 # Strip the comment token from the line
-                value = value.decode(encoding)[1:].strip()
+                value = value[1:].strip()
                 if in_translator_comments and \
                         translator_comments[-1][0] == lineno - 1:
                     # We're already inside a translator comment, continue

However, I get new exception from Jinja2.

extracting messages from trac/admin/templates/admin.html (extensions="jinja2.ext.do, jinja2.ext.with_", variable_start_string="${", variable_end_string="}", line_statement_prefix="#", line_comment_prefix="##", trim_blocks="yes", lstrip_blocks="yes", newstyle_gettext="yes", silent="no")
Traceback (most recent call last):
  File "/home/jun66j5/src/tracdev/git/setup.py", line 42, in <module>
    setup(**extra)
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/frontend.py", line 481, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 149, in extract_from_dir
    for message_tuple in check_and_call_extract_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    for message_tuple in extract_from_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 243, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 325, in extract
    for lineno, funcname, messages, comments in results:
  File "/home/jun66j5/src/tracdev/git/trac/dist.py", line 296, in extract_html
    for m in extractor(fileobj, keywords, comment_tags, options):
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/jinja2/ext.py", line 808, in babel_extract
    extensions[import_string(extension_name)] = None
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/jinja2/utils.py", line 149, in import_string
    return getattr(__import__(module, None, None, [obj]), obj)
AttributeError: module 'jinja2.ext' has no attribute 'with_'
make: *** [Makefile:208: extract] Error 1

Removing temporally jinja2.ext.with_ from messages.cfg because it has been removed since Jinja2 3 (r17543), the error is fixed.

messages.cfg

@@ -6 +6 @@
 [python: **.py]
 
 [html: **/templates/**.html]
-extensions = jinja2.ext.do, jinja2.ext.with_
+extensions = jinja2.ext.do
 variable_start_string = ${
 variable_end_string = }
 line_statement_prefix = #
@@ -17 +17 @@
 silent = no
 
 [text: **/templates/**.txt]
-extensions = jinja2.ext.do, jinja2.ext.with_
+extensions = jinja2.ext.do
 variable_start_string = ${
 variable_end_string = }
 line_statement_prefix = #

…, Third exception is raised while extract_messages_js. It seems that is a Babel issue.

running extract_messages_js
extracting messages from trac/admin/templates/admin.html
Traceback (most recent call last):
  File "/home/jun66j5/src/tracdev/git/setup.py", line 42, in <module>
    setup(**extra)
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/frontend.py", line 481, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 149, in extract_from_dir
    for message_tuple in check_and_call_extract_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    for message_tuple in extract_from_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 243, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 325, in extract
    for lineno, funcname, messages, comments in results:
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 539, in extract_javascript
    fileobj.read().decode(encoding),
AttributeError: 'str' object has no attribute 'decode'
make: *** [Makefile:208: extract] Error 1

[Jun Omae]

…, Third exception is raised while extract_messages_js. It seems that is a Babel issue.

I noticed that is due to extract_javascript_script in trac/dist.py:

@@ -274,7 +274,8 @@ try:
         extractor = ScriptExtractor(out)
         extractor.feed(str(fileobj.read(), 'utf-8'))
         extractor.close()
-        out.seek(0)
+        # extract_javascript expects a binary file object
+        out = io.BytesIO(out.getvalue().encode('utf-8'))
         return extract_javascript(out, keywords, comment_tags, options)

[Jun Omae]

Proposed changes in [33e96570b/jomae.git].

In the changes, extensions in messages.cfg is moved to trac/dist.py in order to support Jinja2 2.x and 3.x.

[Ryan J Ollos]

I tested in my environment and those changes work well. Thanks!

[Jun Omae]

Thanks for the testing. I'm going to push the changes.

[Jun Omae]

Committed in [17594].