Edgewall Software
Modify

Opened 4 months ago

Closed 4 months ago

#13479 closed defect (fixed)

make extraction fails with TypeError

Reported by: Ryan J Ollos Owned by: Jun Omae
Priority: normal Milestone: 1.5.4
Component: i18n Version:
Severity: normal Keywords:
Cc: Branch:
Release Notes:

Fixed message extraction with Python3, Babel and Jinja2 3+.

API Changes:
Internal Changes:

Description

$ make status

Python: /Users/rjollos/.pyenv/shims/python

/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
  Package        Version
  --------------------------------------------------------------------------------------
  Python       : 3.10.3 (main, Mar 22 2022, 21:01:41) [Clang 13.1.6 (clang-1316.0.21.2)]
  Setuptools   : 62.1.0
  Pip          : 22.0.4
  Wheel        : 0.37.1
  Jinja2       : 3.0.3
  Babel        : 2.9.1
  sqlite3      : 2.6.0 (3.36.0)
  PySqlite3    : not installed
  PyMySQL      : 1.0.2
  Psycopg2     : 2.9.3 (dt dec pq3 ext lo64)
  SVN bindings : 1.15.0 (under development)
  Mercurial    : 6.1
  Pygments     : 2.11.2
  Textile      : 4.0.2
  Pytz         : 2022.1
  Docutils     : 0.17.1
  Selenium     : 4.1.3
  PyTidyLib    : 0.3.2 (5.8.0 /usr/local/lib/libtidy.dylib)
  LXML         : 4.8.0
  coverage     : 6.3.2

Variables:
  PATH=/Users/rjollos/Documents/Workspace/trac-dev/bin:/Users/rjollos/Documents/Workspace/trac-dev/subversion-trunk/local/bin:/usr/local/Cellar/tidy-html5/5.6.0/bin:/Users/rjollos/.pyenv/shims:/usr/local/opt/sqlite/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/usr/local/munki:/opt/X11/bin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/Users/rjollos/.pyenv/bin
  PYTHONPATH=.
  TRAC_TEST_DB_URI=
  server-options= -p 8000 -a '*,/Users/rjollos/Documents/Workspace/trac-dev/tracenvs/htdigest.trac,trac' -r -e /Users/rjollos/Documents/Workspace/trac-dev/tracenvs

External dependencies:
  Git version: git version 2.35.1
  Subversion version: 1.15.0-dev
$ make extraction
python  setup.py  extract_messages  extract_messages_js  extract_messages_tracini
running extract_messages
extracting messages from tracopt/__init__.py
extracting messages from tracopt/perm/__init__.py
extracting messages from tracopt/perm/authz_policy.py
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/teo-rjollos.git/setup.py", line 42, in <module>
    setup(**extra)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/frontend.py", line 481, in run
    for filename, lineno, message, comments, context in extracted:
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 149, in extract_from_dir
    for message_tuple in check_and_call_extract_file(
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    for message_tuple in extract_from_file(
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 243, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
  File "/Users/rjollos/.pyenv/versions/trac-py310/lib/python3.10/site-packages/babel/messages/extract.py", line 325, in extract
    for lineno, funcname, messages, comments in results:
  File "/Users/rjollos/Documents/Workspace/trac-dev/teo-rjollos.git/trac/dist.py", line 148, in extract_python
    for _ in tokens:
  File "/Users/rjollos/.pyenv/versions/3.10.3/lib/python3.10/tokenize.py", line 498, in _tokenize
    if line[pos] in '#\r\n':           # skip comments or blank lines
TypeError: 'in <string>' requires string as left operand, not int
make: *** [extraction] Error 1

Attachments (0)

Change History (6)

comment:1 by Jun Omae, 4 months ago

The exception goes away by the following patch:

  • trac/dist.py

    diff --git a/trac/dist.py b/trac/dist.py
    index a9d6364bf..c085df00e 100644
    a b try:  
    143143        if 'cleandoc_keywords' in options:
    144144            cleandoc_keywords.update(options['cleandoc_keywords'])
    145145
    146         tokens = generate_tokens(fileobj.readline)
     146        tokens = generate_tokens(lambda: fileobj.readline().decode(encoding))
    147147        tok = value = None
    148148        for _ in tokens:
    149149            prev_tok, prev_value = tok, value
    try:  
    166166                continue
    167167            elif call_stack == -1 and tok == COMMENT:
    168168                # Strip the comment token from the line
    169                 value = value.decode(encoding)[1:].strip()
     169                value = value[1:].strip()
    170170                if in_translator_comments and \
    171171                        translator_comments[-1][0] == lineno - 1:
    172172                    # We're already inside a translator comment, continue

However, I get new exception from Jinja2.

extracting messages from trac/admin/templates/admin.html (extensions="jinja2.ext.do, jinja2.ext.with_", variable_start_string="${", variable_end_string="}", line_statement_prefix="#", line_comment_prefix="##", trim_blocks="yes", lstrip_blocks="yes", newstyle_gettext="yes", silent="no")
Traceback (most recent call last):
  File "/home/jun66j5/src/tracdev/git/setup.py", line 42, in <module>
    setup(**extra)
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/frontend.py", line 481, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 149, in extract_from_dir
    for message_tuple in check_and_call_extract_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    for message_tuple in extract_from_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 243, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 325, in extract
    for lineno, funcname, messages, comments in results:
  File "/home/jun66j5/src/tracdev/git/trac/dist.py", line 296, in extract_html
    for m in extractor(fileobj, keywords, comment_tags, options):
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/jinja2/ext.py", line 808, in babel_extract
    extensions[import_string(extension_name)] = None
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/jinja2/utils.py", line 149, in import_string
    return getattr(__import__(module, None, None, [obj]), obj)
AttributeError: module 'jinja2.ext' has no attribute 'with_'
make: *** [Makefile:208: extract] Error 1

Removing temporally jinja2.ext.with_ from messages.cfg because it has been removed since Jinja2 3 (r17543), the error is fixed.

  • messages.cfg

    diff --git a/messages.cfg b/messages.cfg
    index 0ae9d5b47..4b2d47928 100644
    a b  
    66[python: **.py]
    77
    88[html: **/templates/**.html]
    9 extensions = jinja2.ext.do, jinja2.ext.with_
     9extensions = jinja2.ext.do
    1010variable_start_string = ${
    1111variable_end_string = }
    1212line_statement_prefix = #
    newstyle_gettext = yes  
    1717silent = no
    1818
    1919[text: **/templates/**.txt]
    20 extensions = jinja2.ext.do, jinja2.ext.with_
     20extensions = jinja2.ext.do
    2121variable_start_string = ${
    2222variable_end_string = }
    2323line_statement_prefix = #

…, Third exception is raised while extract_messages_js. It seems that is a Babel issue.

running extract_messages_js
extracting messages from trac/admin/templates/admin.html
Traceback (most recent call last):
  File "/home/jun66j5/src/tracdev/git/setup.py", line 42, in <module>
    setup(**extra)
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/frontend.py", line 481, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 149, in extract_from_dir
    for message_tuple in check_and_call_extract_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    for message_tuple in extract_from_file(
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 243, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 325, in extract
    for lineno, funcname, messages, comments in results:
  File "/home/jun66j5/venv/py39/lib/python3.9/site-packages/babel/messages/extract.py", line 539, in extract_javascript
    fileobj.read().decode(encoding),
AttributeError: 'str' object has no attribute 'decode'
make: *** [Makefile:208: extract] Error 1

in reply to:  1 comment:2 by Jun Omae, 4 months ago

…, Third exception is raised while extract_messages_js. It seems that is a Babel issue.

I noticed that is due to extract_javascript_script in trac/dist.py:

@@ -274,7 +274,8 @@ try:
         extractor = ScriptExtractor(out)
         extractor.feed(str(fileobj.read(), 'utf-8'))
         extractor.close()
-        out.seek(0)
+        # extract_javascript expects a binary file object
+        out = io.BytesIO(out.getvalue().encode('utf-8'))
         return extract_javascript(out, keywords, comment_tags, options)


comment:3 by Jun Omae, 4 months ago

Proposed changes in [33e96570b/jomae.git].

In the changes, extensions in messages.cfg is moved to trac/dist.py in order to support Jinja2 2.x and 3.x.

comment:4 by Ryan J Ollos, 4 months ago

I tested in my environment and those changes work well. Thanks!

comment:5 by Jun Omae, 4 months ago

Owner: set to Jun Omae
Status: newassigned

Thanks for the testing. I'm going to push the changes.

comment:6 by Jun Omae, 4 months ago

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Committed in [17594].

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.