convert_db fails if both spamfilter and tags plugin are enabled

[btbn@…]

This is a rather specific case, but I think I have it tracked down to some degree. What is happening is this:

During the migration process, trac calls the "upgrade" admin command on the new environment, after initially creating it empty, so that plugins can create their databases.

At some point during this process, this function ends up called: https://trac.edgewall.org/browser/plugins/trunk/spam-filter/tracspamfilter/filters/trapfield.py#L116

However, it fails with:

17:36:10 Trac[env] ERROR: Component <Component tracspamfilter.filters.trapfield.TrapFieldFilterStrategy> failed with
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 382, in component_guard
    yield
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 750, in needs_upgrade
    if participant.environment_needs_upgrade():
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tracspamfilter/filters/trapfield.py", line 119, in environment_needs_upgrade
    chrome.jenv.add_extension(self._trap_field_ext)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'add_extension'
Error: Unable to check for upgrade of tracspamfilter.filters.trapfield.TrapFieldFilterStrategy: AttributeError: 'NoneType' object has no attribute 'add_extension'

Upon further digging, why jenv is None, I removed the exception handlers around it, and got the following exception:

17:43:29 Trac[env] ERROR: Component <Component tracspamfilter.filters.trapfield.TrapFieldFilterStrategy> failed with
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 382, in component_guard
    yield
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 750, in needs_upgrade
    if participant.environment_needs_upgrade():
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tracspamfilter/filters/trapfield.py", line 116, in environment_needs_upgrade
    chrome.load_template(os.devnull)  # to set chrome.jenv property
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/web/chrome.py", line 1316, in load_template
    jinja2_dirs = self.get_all_templates_dirs()
  File "/usr/local/lib/python3.13/site-packages/trac/web/chrome.py", line 772, in get_all_templates_dirs
    for provider in self.template_providers:
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 96, in extensions
    components = [component.compmgr[cls] for cls in classes]
                  ~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 238, in __getitem__
    component = cls(self)
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 158, in __call__
    self.__init__()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/wiki.py", line 101, in __init__
    self.tag_system = TagSystem(self.env)
                      ~~~~~~~~~^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 158, in __call__
    self.__init__()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/api.py", line 249, in __init__
    self._populate_provider_map()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/api.py", line 465, in _populate_provider_map
    for provider in self.tag_providers)
                    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 96, in extensions
    components = [component.compmgr[cls] for cls in classes]
                  ~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 238, in __getitem__
    component = cls(self)
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 158, in __call__
    self.__init__()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/ticket.py", line 53, in __init__
    self._fetch_tkt_tags()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/ticket.py", line 215, in _fetch_tkt_tags
    rw_cursor.execute("""
    ~~~~~~~~~~~~~~~~~^^^^
        DELETE FROM tags
        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
        """ % (db.cast('tags.name', 'int'), ignore),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        (self.realm,))
        ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/db/util.py", line 73, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedTable: relation "tags" does not exist
LINE 2:                 DELETE FROM tags
                                    ^
UndefinedTable: relation "tags" does not exist
LINE 2:                 DELETE FROM tags
                                    ^

Error: Unable to check for upgrade of tracspamfilter.filters.trapfield.TrapFieldFilterStrategy: UndefinedTable: relation "tags" does not exist
LINE 2:                 DELETE FROM tags
                                    ^

So basically, upgrading/initializing the spamfilter plugin requires the tags plugin to be already upgraded/initialized, but since this is happening during migration/initialization, it is not.

Not sure how to properly fix this. I guess one way would be to ensure the tags plugin is initialized first on its own, before any other plugins are? Or maybe even generally initialize the plugins one by one during migration, and call upgrade after each one?

[Timo R. <btbn@…>]

Found a workaround to successfully migrate: Simply added a "catch Exception: pass" clause to this try-Block: ​https://trac-hacks.org/browser/tagsplugin/trunk/tractags/ticket.py#L52

With that in place, the migration completes successfully, and the hack can be removed again.

[Jun Omae]

The root cause is that TicketTagProvider in ​th:TagsPlugin fetches tags records before creating tags table.

[Jun Omae]

I think we could use IRequestFilter.post_process_request instead of IEnvironmentSetupParticipant.environment_needs_upgrade(), in order to avoid the exception being raised from component initialization in other plugins.

tracspamfilter/filters/trapfield.py

@@ -14 +14 @@
 import os
 
 import jinja2.ext
+import jinja2.exceptions
 
-from trac.api import IEnvironmentSetupParticipant
 from trac.config import IntOption, Option
 from trac.core import Component, implements
 from trac.util import lazy
 from trac.util.text import javascript_quote, shorten_line
+from trac.web.api import IRequestFilter
 from trac.web.chrome import Chrome
 
 from tracspamfilter.api import IFilterStrategy, N_
@@ -28 +29 @@
 class TrapFieldFilterStrategy(Component):
     """Spam filter using a hidden trap field.
     """
-    implements(IFilterStrategy, IEnvironmentSetupParticipant)
+    implements(IFilterStrategy, IRequestFilter)
 
     karma_points = IntOption('spam-filter', 'trap_karma', '10',
         """By how many points a trap reject impacts the overall karma of
@@ -104 +105 @@
            '<input type="text" name="%s" value="" />' % (self.name) + \
            '<input type="hidden" name="%s" value="" />' % (self.name_hidden) + \
            '</div>'
-    # IEnvironmentSetupParticipant interface
 
-    def environment_created(self):
-        pass
+    # IRequestFilter interface
 
-    def environment_needs_upgrade(self):
-        if self.karma_points > 0:
-            chrome = Chrome(self.env)
-            try:
-                chrome.load_template(os.devnull)  # to set chrome.jenv property
-            except:
-                pass
-            chrome.jenv.add_extension(self._trap_field_ext)
-        return False
+    def pre_process_request(self, req, handler):
+        return handler
 
-    def upgrade_environment(self):
-        pass
+    def post_process_request(self, req, template, data, metadata):
+        if template in self.anchors:
+            # Add trapfield extension before rendering the template
+            self._setup_trap_field_ext()
+        return template, data, metadata
 
     # Internal methods
 
+    def _setup_trap_field_ext(self):
+        chrome = Chrome(self.env)
+        if chrome.jenv is None:
+            try:
+                # to initialize chrome.jenv attribute
+                chrome.load_template(os.devnull)
+            except jinja2.exceptions.TemplateNotFound:
+                pass
+            except:
+                self.log.warning('Exception caught while letting Chrome.jenv '
+                                 'attribute initialize', exc_info=True)
+                return
+        chrome.jenv.add_extension(self._trap_field_ext)
+
     @lazy
     def _trap_field_ext(self):
 

[Jun Omae]

Replying to Jun Omae:

The root cause is that TicketTagProvider in ​th:TagsPlugin fetches tags records before creating tags table.

I've filed it at ​th:#14360.

[Jun Omae]

At least, it might be good to log the exception rather than TemplateNotFound and check the jenv attribute being not None.

tracspamfilter/filters/trapfield.py

@@ -14 +14 @@
 import os
 
 import jinja2.ext
+import jinja2.exceptions
 
 from trac.api import IEnvironmentSetupParticipant
 from trac.config import IntOption, Option
@@ -114 +115 @@
             chrome = Chrome(self.env)
             try:
                 chrome.load_template(os.devnull)  # to set chrome.jenv property
+            except jinja2.exceptions.TemplateNotFound:
+                pass
             except:
-                pass
-            chrome.jenv.add_extension(self._trap_field_ext)
+                self.log.warning('Exception caught while letting Chrome.jenv '
+                                 'attribute initialize', exc_info=True)
+            if chrome.jenv is not None:
+                chrome.jenv.add_extension(self._trap_field_ext)
         return False
 
     def upgrade_environment(self):

[Jun Omae]

Replying to Jun Omae:

Replying to Jun Omae:

The root cause is that TicketTagProvider in ​th:TagsPlugin fetches tags records before creating tags table.

I've filed it at ​th:#14360.

I just committed the workaround of ​th:TagsPlugin in ​th:r18665.