Plugins loaded from site-packages in preference to plugins/ directories

[Dave Abrahams <dave@…>]

Once you've "installed a plugin" in the usual way (all directions say "setup.py install"), it goes into site-packages, and putting a newer version of the plugin in your trac's plugins/ directory makes no difference. That's bad for flexibility; consider that several tracs may run on the same machine. One might want to set up a new trac instance for testing, with a new version of some plugin.

It's a sad fact of Python that "uninstalling" anything placed in site-packages is a nontrivial and undocumented operation that may involve combing through .pth files for references to the package, so there are plenty of good reasons to try to use the plugins/ directory. It would be nice if setandard plugin installation was able to place plugins there, but that's another story. In the meantime, please allow plugins directories to override what's in site-packages.

[osimons]

I don't think this will accomplish much actually, based on the fact that as long as you run the various projects inside the same interpreter the plugin will only load once - for all projects. This is basic Python - there can only be one myplugin namespace, one entry in sys.modules pointing to some code. Regardless of where it is loaded first time (globally or inside a project).

If you want to test things, then do what others generally do:

• use something like virtualenv.py to set up a second interpreter based off the first that can override anything you like.
• fiddle with sys.path so that you in you init script, PYTHON_PATH setting or other valid alternative depending on frontend, set it so that you insert your new-module-path first in that list.
• rename the plugin namespace and entry points (theplugin2), and disable the global plugin and enable the custom new version in one project only.
• and other variations…

Now on to why changing the load order will be both troublesome and give unwanted results:

1. When Python/Trac loads initially it will only know the globally installed modules. To load the new plugin, Trac will have to defer initial global loading, and then set off on loading once the first request is made (or Environment() is first instantiated) - preferring project if it exists. As we depend on setuptools that won't be easy as it looks up extention points and will load the code to serve us - providing us with available plugins, including much of the Trac code itself that is also based on the same archtecture. Unloading running code to load some newly discovered code is really non-trivial.
2. Say that we managed to 'hack' this load order somehow. Then say you have 5 projects, and one project contains an updated plugin. Things will then work fine if you can always make sure that the first request won't hit one of the 4 other projects - if so, it would have to load the global one. If those 5 projects all had different versions of the plugin, each project would in fact run the version loaded from the project where the first request was made - whatever that was. And, the plugin will need to be loaded even though it isn't enabled for a particular project as we need to know the namespace/plugin name to actually check to see if it is enabled.
3. Support. There could be an endless line of support issues resulting from indeterminate plugin loading. You've disabled an old version in one project, and plonked a new version inside another - leaving a globally installed module as-is (seeing it is so hard to uninstall). Something is not working. What is wrong? Please help… One location that is preferred over any other, is a way to bring order into this possible chaos.

I really can't see the project looking to solve this perceived 'defect'. I think it works great.

[Dave Abrahams <dave@…>]

Fair enough. Do you think it's worth documenting all or some of this? Especially in light of the "one plugin per interpreter instance" issue, it seems likely that anyone using the plugins/ directory is misinterpreting its meaning. Until you know some of the implementation details described above, it seems very reasonable to assume that each Trac instance's own plugins/ directory is separately bound to that instance.

[osimons]

Good point - the docs may be a bit thin on this…

Please review TracPlugins and see if it makes sense. Changes welcome.

I'll leave it for review for 24 hours, and get it committed to the 0.11 docs in repos as a last minute addition.

[osimons]

For my to-do list.

[Dave Abrahams <dave@…>]

Well, I don't think you used "Trac Instance" in the usual way, so it's really confusing. Normally I think docs mean what you call "project" when they say "trac instance." I think you should say something like "instance of the python interpreter," and explicitly discuss "multi-project setups" (but check to make sure that's the term used when TRAC_ENV_PARENT_DIR is introduced). Thanks.

[Christian Boos]

TracPlugins@45 is fine for me.

[osimons]

Page updated in repos as [6316]. Closing.

[Dave Abrahams <dave@…>]

Replying to osimons:

Good point - the docs may be a bit thin on this…

Please review TracPlugins and see if it makes sense. Changes welcome.

I'll leave it for review for 24 hours, and get it committed to the 0.11 docs in repos as a last minute addition.

This looks really good to me, but there's one important thing missing: the way this works needs to be known by ​http://trac.edgewall.org/wiki/TracPlugins#ForaSingleProject, or you basically set people up to make this mistake and trawl for answers later. I would add something like this:

Note that in a multi-project setup, a pool of Python interpreter instances will be dynamically allocated to projects based on need, and since plugins occupy a place in Python's module system, the first version of any given plugin to be loaded will be used for all the projects. In other words, you cannot use different versions of a single plugin in two projects of a multi-project setup. It may be safer to install plugins for all projects (see below) and then enable them selectively on a project-by-project basis.

I would also like to see a note in the section on installing for all projects that explains how to deinstall plugins, and how not using easy-install et. al may save lots of pain in the long run.

[osimons]

Replying to Dave Abrahams <dave@boost-consulting.com>:

This looks really good to me, but there's one important thing missing…

Thanks for the text, I've updated the page and also added some information on uninstalling plugins.

For the future, if you have things to add or make more precise in the wiki, just update the page directly. No need to file (or reopen) tickets unless it concerns major or possibly controversial changes to the content of pages that are part of the default documentation. Changes are migrated to the repos in an ad-hoc manner - they'll get there sooner or later :-)