Edgewall Software
Modify

Opened 4 weeks ago

Closed 3 weeks ago

#13804 closed defect (fixed)

convert_db fails if both spamfilter and tags plugin are enabled

Reported by: btbn@… Owned by: Dirk Stöcker
Priority: normal Milestone: plugin - spam-filter
Component: plugin/spamfilter Version: 1.6
Severity: normal Keywords:
Cc: btbn@… Branch:
Release Notes:
API Changes:
Internal Changes:

Description

This is a rather specific case, but I think I have it tracked down to some degree. What is happening is this:

During the migration process, trac calls the "upgrade" admin command on the new environment, after initially creating it empty, so that plugins can create their databases.

At some point during this process, this function ends up called: https://trac.edgewall.org/browser/plugins/trunk/spam-filter/tracspamfilter/filters/trapfield.py#L116

However, it fails with:

17:36:10 Trac[env] ERROR: Component <Component tracspamfilter.filters.trapfield.TrapFieldFilterStrategy> failed with
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 382, in component_guard
    yield
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 750, in needs_upgrade
    if participant.environment_needs_upgrade():
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tracspamfilter/filters/trapfield.py", line 119, in environment_needs_upgrade
    chrome.jenv.add_extension(self._trap_field_ext)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'add_extension'
Error: Unable to check for upgrade of tracspamfilter.filters.trapfield.TrapFieldFilterStrategy: AttributeError: 'NoneType' object has no attribute 'add_extension'

Upon further digging, why jenv is None, I removed the exception handlers around it, and got the following exception:

17:43:29 Trac[env] ERROR: Component <Component tracspamfilter.filters.trapfield.TrapFieldFilterStrategy> failed with
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 382, in component_guard
    yield
  File "/usr/local/lib/python3.13/site-packages/trac/env.py", line 750, in needs_upgrade
    if participant.environment_needs_upgrade():
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tracspamfilter/filters/trapfield.py", line 116, in environment_needs_upgrade
    chrome.load_template(os.devnull)  # to set chrome.jenv property
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/web/chrome.py", line 1316, in load_template
    jinja2_dirs = self.get_all_templates_dirs()
  File "/usr/local/lib/python3.13/site-packages/trac/web/chrome.py", line 772, in get_all_templates_dirs
    for provider in self.template_providers:
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 96, in extensions
    components = [component.compmgr[cls] for cls in classes]
                  ~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 238, in __getitem__
    component = cls(self)
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 158, in __call__
    self.__init__()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/wiki.py", line 101, in __init__
    self.tag_system = TagSystem(self.env)
                      ~~~~~~~~~^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 158, in __call__
    self.__init__()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/api.py", line 249, in __init__
    self._populate_provider_map()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/api.py", line 465, in _populate_provider_map
    for provider in self.tag_providers)
                    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 96, in extensions
    components = [component.compmgr[cls] for cls in classes]
                  ~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 238, in __getitem__
    component = cls(self)
  File "/usr/local/lib/python3.13/site-packages/trac/core.py", line 158, in __call__
    self.__init__()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/ticket.py", line 53, in __init__
    self._fetch_tkt_tags()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/tractags/ticket.py", line 215, in _fetch_tkt_tags
    rw_cursor.execute("""
    ~~~~~~~~~~~~~~~~~^^^^
        DELETE FROM tags
        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
        """ % (db.cast('tags.name', 'int'), ignore),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        (self.realm,))
        ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/trac/db/util.py", line 73, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedTable: relation "tags" does not exist
LINE 2:                 DELETE FROM tags
                                    ^
UndefinedTable: relation "tags" does not exist
LINE 2:                 DELETE FROM tags
                                    ^

Error: Unable to check for upgrade of tracspamfilter.filters.trapfield.TrapFieldFilterStrategy: UndefinedTable: relation "tags" does not exist
LINE 2:                 DELETE FROM tags
                                    ^

So basically, upgrading/initializing the spamfilter plugin requires the tags plugin to be already upgraded/initialized, but since this is happening during migration/initialization, it is not.

Not sure how to properly fix this. I guess one way would be to ensure the tags plugin is initialized first on its own, before any other plugins are? Or maybe even generally initialize the plugins one by one during migration, and call upgrade after each one?

Attachments (0)

Change History (7)

comment:1 by Timo R. <btbn@…>, 4 weeks ago

Cc: btbn@… added

Found a workaround to successfully migrate: Simply added a "catch Exception: pass" clause to this try-Block: https://trac-hacks.org/browser/tagsplugin/trunk/tractags/ticket.py#L52

With that in place, the migration completes successfully, and the hack can be removed again.

comment:2 by Jun Omae, 4 weeks ago

The root cause is that TicketTagProvider in th:TagsPlugin fetches tags records before creating tags table.

comment:3 by Jun Omae, 4 weeks ago

Component: admin/consoleplugin/spamfilter
Milestone: plugin - spam-filter
Owner: set to Dirk Stöcker

I think we could use IRequestFilter.post_process_request instead of IEnvironmentSetupParticipant.environment_needs_upgrade(), in order to avoid the exception being raised from component initialization in other plugins.

  • tracspamfilter/filters/trapfield.py

     
    1414import os
    1515
    1616import jinja2.ext
     17import jinja2.exceptions
    1718
    18 from trac.api import IEnvironmentSetupParticipant
    1919from trac.config import IntOption, Option
    2020from trac.core import Component, implements
    2121from trac.util import lazy
    2222from trac.util.text import javascript_quote, shorten_line
     23from trac.web.api import IRequestFilter
    2324from trac.web.chrome import Chrome
    2425
    2526from tracspamfilter.api import IFilterStrategy, N_
     
    2829class TrapFieldFilterStrategy(Component):
    2930    """Spam filter using a hidden trap field.
    3031    """
    31     implements(IFilterStrategy, IEnvironmentSetupParticipant)
     32    implements(IFilterStrategy, IRequestFilter)
    3233
    3334    karma_points = IntOption('spam-filter', 'trap_karma', '10',
    3435        """By how many points a trap reject impacts the overall karma of
     
    104105           '<input type="text" name="%s" value="" />' % (self.name) + \
    105106           '<input type="hidden" name="%s" value="" />' % (self.name_hidden) + \
    106107           '</div>'
    107     # IEnvironmentSetupParticipant interface
    108108
    109     def environment_created(self):
    110         pass
     109    # IRequestFilter interface
    111110
    112     def environment_needs_upgrade(self):
    113         if self.karma_points > 0:
    114             chrome = Chrome(self.env)
    115             try:
    116                 chrome.load_template(os.devnull)  # to set chrome.jenv property
    117             except:
    118                 pass
    119             chrome.jenv.add_extension(self._trap_field_ext)
    120         return False
     111    def pre_process_request(self, req, handler):
     112        return handler
    121113
    122     def upgrade_environment(self):
    123         pass
     114    def post_process_request(self, req, template, data, metadata):
     115        if template in self.anchors:
     116            # Add trapfield extension before rendering the template
     117            self._setup_trap_field_ext()
     118        return template, data, metadata
    124119
    125120    # Internal methods
    126121
     122    def _setup_trap_field_ext(self):
     123        chrome = Chrome(self.env)
     124        if chrome.jenv is None:
     125            try:
     126                # to initialize chrome.jenv attribute
     127                chrome.load_template(os.devnull)
     128            except jinja2.exceptions.TemplateNotFound:
     129                pass
     130            except:
     131                self.log.warning('Exception caught while letting Chrome.jenv '
     132                                 'attribute initialize', exc_info=True)
     133                return
     134        chrome.jenv.add_extension(self._trap_field_ext)
     135
    127136    @lazy
    128137    def _trap_field_ext(self):
    129138
Last edited 4 weeks ago by Jun Omae (previous) (diff)

in reply to:  2 ; comment:4 by Jun Omae, 4 weeks ago

Replying to Jun Omae:

The root cause is that TicketTagProvider in th:TagsPlugin fetches tags records before creating tags table.

I've filed it at th:#14360.

comment:5 by Jun Omae, 4 weeks ago

At least, it might be good to log the exception rather than TemplateNotFound and check the jenv attribute being not None.

  • tracspamfilter/filters/trapfield.py

     
    1414import os
    1515
    1616import jinja2.ext
     17import jinja2.exceptions
    1718
    1819from trac.api import IEnvironmentSetupParticipant
    1920from trac.config import IntOption, Option
     
    114115            chrome = Chrome(self.env)
    115116            try:
    116117                chrome.load_template(os.devnull)  # to set chrome.jenv property
     118            except jinja2.exceptions.TemplateNotFound:
     119                pass
    117120            except:
    118                 pass
    119             chrome.jenv.add_extension(self._trap_field_ext)
     121                self.log.warning('Exception caught while letting Chrome.jenv '
     122                                 'attribute initialize', exc_info=True)
     123            if chrome.jenv is not None:
     124                chrome.jenv.add_extension(self._trap_field_ext)
    120125        return False
    121126
    122127    def upgrade_environment(self):

in reply to:  4 comment:6 by Jun Omae, 4 weeks ago

Replying to Jun Omae:

Replying to Jun Omae:

The root cause is that TicketTagProvider in th:TagsPlugin fetches tags records before creating tags table.

I've filed it at th:#14360.

I just committed the workaround of th:TagsPlugin in th:r18665.

comment:7 by Dirk Stöcker, 3 weeks ago

Resolution: fixed
Status: newclosed

See r17871.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Dirk Stöcker.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Dirk Stöcker to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.