environment_created in plugins never be called on initenv using --inherit option

[Jun Omae]

I created the plugin that uses IEnvironmentSetupParticipant.environment_created, and created a project using $ENV initenv --inherit=$HOME/var/trac-inherit.ini. The plugin is enabled in trac-inherit.ini.

I expect that environment_created in the plugin is called. However, the method isn't called with 0.12rc1 (r9839).

The method is called with 0.11-stable (r9726), and I expect it!

$HOME/var/trac-inherit.ini:

[components]
tracenvsetup.* = enabled

[logging]
log_file = trac.log
log_level = DEBUG
log_type = file

plugin source - tracenvsetup/__init__.py

from trac.core import implements, Component
from trac.env import IEnvironmentSetupParticipant

class SampleEnvironmentSetupParticipant(Component):

    implements(IEnvironmentSetupParticipant)

    # IEnvironmentSetupParticipant methods

    def environment_created(self):
        # if raise the exception, call it via initenv
        raise ValueError("call environment_created")

    def environment_needs_upgrade(self, db):
        self.log.debug("the existing environment requires an upgrade for sample plugin.")
        return True

    def upgrade_environment(self, db):
        self.log.debug("upgrading existing environment for sample plugin.")
        pass

In 0.11-stable:

$HOME/bin/trac-admin $HOME/var/trac/0.11-stable-envsetup initenv --inherit=$HOME/var/trac-inherit.ini

Creating and Initializing Project
Initenv for '/Users/jun66j5/var/trac/0.11-stable-envsetup' failed.
 Failed to create environment.
call environment_created
Traceback (most recent call last):
  File "/Users/jun66j5/lib/python2.6/site-packages/Trac-0.11.7.1dev_r9726-py2.6.egg/trac/admin/console.py", line 583, in do_initenv
    options=options)
  File "/Users/jun66j5/lib/python2.6/site-packages/Trac-0.11.7.1dev_r9726-py2.6.egg/trac/env.py", line 213, in __init__
    setup_participant.environment_created()
  File "/Users/jun66j5/lib/python2.6/site-packages/TracEnvSetupSample-0.1-py2.6.egg/tracenvsetup/__init__.py", line 13, in environment_created
    raise ValueError("call environment_created")
ValueError: call environment_created

[Remy Blank]

There's something weird in Environment.__init__(): plugins are loaded before the environment (and config file and all) is created, so env.shared_plugins_dir is always empty at that point (I assume this should allow creating database backend plugins, and ensure that they are loaded before creating the database). Is your plugin installed globally or in a shared plugin folder? How is it packaged (egg, extracted)? Then again, this code hasn't changed from 0.11 to 0.12, so I don't see how this could be the cause.

Other than that, I don't see what could cause the difference. I would need to step through the creation process with a debugger to find out what happens.

[Jun Omae]

Replying to rblank:

Is your plugin installed globally or in a shared plugin folder?

The directory that my plugin installed is on PYTHONPATH.

export PYTHONPATH=$HOME/lib/python2.6/site-packages
python setup.py install --prefix=$HOME
...

Trac is installed on the same directory.

How is it packaged (egg, extracted)?

extracted.

Other than that, I don't see what could cause the difference. I would need to step through the creation process with a debugger to find out what happens.

Ok, I'll try it.

[Jun Omae]

I created the patch.

The comoponents settings are cached when calling Environment.setup_config (trac/env.py:207​). At this time the inherit file isn't used.

When setup_participants is called (trac/env.py:226​) the cache is used and the components settings in inherit file are ignored.

trac/env.py

@@ -402 +402 @@
             self.config.set(section, name, value)
         self.config.save()
         # Full reload to get 'inherit' working
+        delattr(self, '_rules')
         self.config.parse_if_needed(force=True)
 
         # Create the database

Stacktrace on calling Environment.setup_config

Breakpoint 2 at /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/env.py:269
(Pdb) c
> /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/env.py(269)_component_rules()
-> return self._rules
(Pdb) w
  /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/bdb.py(368)run()
-> exec cmd in globals, locals
  <string>(1)<module>()
  /Users/jun66j5/bin/trac-admin(9)<module>()
-> load_entry_point('Trac==0.12rc1', 'console_scripts', 'trac-admin')()
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/admin/console.py(543)run()
-> return admin.onecmd(command)
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/admin/console.py(107)onecmd()
-> rv = cmd.Cmd.onecmd(self, line) or 0
  /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cmd.py(219)onecmd()
-> return func(arg)
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/admin/console.py(413)do_initenv()
-> options=options)
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/env.py(207)__init__()
-> self.setup_config(load_defaults=create)
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/env.py(435)setup_config()
-> for section, default_options in self.config.defaults(self).items():
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/config.py(157)defaults()
-> for (section, key), option in Option.get_registry(compmgr).items():
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/config.py(519)get_registry()
-> return dict(each for each in Option.registry.items()
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/config.py(521)<genexpr>()
-> or compmgr.is_enabled(components[each[1]]))
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/core.py(222)is_enabled()
-> self.enabled[cls] = self.is_component_enabled(cls)
  /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/env.py(302)is_component_enabled()
-> rules = self._component_rules
> /Users/jun66j5/lib/python2.6/site-packages/Trac-0.12rc1-py2.6.egg/trac/env.py(269)_component_rules()
-> return self._rules

[Remy Blank]

Excellent detective work! I'll apply the patch tonight.

[Christian Boos]

Ok, the delattr(self, '_rules') looks a bit scary, but create is not supposed to be called in a multi-threaded setup, only from initenv so I suppose this is safe.

[Mikael Relbe]

Thank you for finding this, which explains ​TH:ticket:7203#comment:3 (I thought I was in deep water there… :)

[Remy Blank]

I have applied jomae's patch in [9847] as a temporary workaround, as it fixes the issue. However, the way environment creation is implemented could use some refactoring. I'm working on it, and it also fixes this issue, but we should probably delay this until 0.12.1. Patch following shortly.

[Remy Blank]

Refactored environment creation.

[Remy Blank]

In the current state, a "preliminary" configuration is loaded first, which doesn't take the [inherit] file option into account. Then, logging is setup, plugins are loaded, and only then is the environment created and the final configuration loaded. As a consequence, logging uses the default configuration, instead of an inherited configuration. Also, plugins located in a shared plugins folder set in an inherited configuration are not loaded.

The patch above refactors the creation of the environment, with the following advantages:

• The configuration doesn't need to be reloaded, and the cache for the [components] section doesn't need to be cleared. So it fixes this issue.
• Logging is set up correctly from the start.
• Plugins located in a shared plugins folder that is configured in a shared configuration are loaded.

All tests pass, and environment creation with and without --inherit= works.

Thoughts?

[Christian Boos]

Looks good, but comment:8 is not that clear about "plugins located in a shared plugins folder set in an inherited configuration": are they loaded or not? Point 3. seem to say they are, contradicting the first sentence, unless "shared configuration" means something different than "inherited configuration". /me confused…

In any case, I'd very much prefer to postpone this change to 0.12.1.

[Remy Blank]

Let me try again ;)

• Shared plugins folder: folder defined with the [inherit] plugins_dir option.
• Inherited configuration: file defined with the [inherit] file option.

In the current state, plugins located in a shared plugins folder that is defined in an inherited configuration are not loaded during creation, and hence, their IEnvironmentSetupParticipants are not called. With the patch, they are. Plugins installed globally already work as expected.

common.ini:

[inherit]
plugins_dir = /path/to/shared/plugins

Environment creation:

trac-admin env initenv --inherit=/path/to/common.ini

[Christian Boos]

Replying to rblank:

Let me try again ;)

We should put that in the docs. TracEnvironment? That way, we can also clearly document the current limitation and, after the fix, document the behavior change.

[Remy Blank]

Ok, moving over to 0.12.1.

[Christian Boos]

It was not obvious to add the bit of information from comment:10 somewhere, without several other clarifications in various places, but I think it's OK now, with TracEnvironment@42.

[Remy Blank]

9416-environment-creation-r9848.patch​ applied in [9940].