Remove hard dependency on pkg_resources

[anatoly techtonik <techtonik@…>]

Trac (unlike other Python apps) heavily relies on packages that pkg_resources can find, and pkg_resources can't find packages already available in sys.path:

>>> import sys, os, pkg_resources
>>> pkg_resources.get_distribution('genshi')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 298, in get_distribution
    if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 177, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 552, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: genshi
>>> import imp
>>> imp.find_module('genshi')
(None, 'genshi', ('', '', 5))
>>> import genshi
>>> genshi.__version__
'0.6.1'
>>>

[anatoly techtonik <techtonik@…>]

By abstracting the logic pkg_resources provides into separate component, it will be easier to get rid of one more dependency and provide more flexible Trac installation scenarios.

[anatoly techtonik <techtonik@…>]

Patch to execute trac-admin without installation of Trac as a package.

trac/admin/console.py

@@ -35 +35 @@
 from trac.wiki.admin import WikiAdmin
 from trac.wiki.macros import WikiMacroBase
 
-TRAC_VERSION = pkg_resources.get_distribution('Trac').version
+TRAC_VERSION = VERSION
 rl_completion_suppress_append = None
 LANG = os.environ.get('LANG')

[Remy Blank]

The thing is, Trac requires setuptools, which provides pkg_resources. We don't want to get rid of the dependency on setuptools (adding it was a conscious decision). So I don't see what we can gain by not using pkg_resources or abstracting it.

Sure, this means your plugins must also require setuptools and be installed properly. That's hardly a tough constraint. If you run packages like Genshi from sources, you simply need to install them in development mode:

python setup.py develop

And of course, the same advice as in #10418: use virtualenv, it will make your life much easier.

Then again, maybe I'm missing something. Can you give us a concrete use case where the setuptools constraint is unacceptable, and where using a virtualenv isn't possible?

[anatoly techtonik <techtonik@…>]

Replying to rblank:

Sure, this means your plugins must also require setuptools and be installed properly. That's hardly a tough constraint.

That's not a constraint for a core developers, but considering that these developers always ask for patches, it will be good for them to consider the position of occasional administrator who run into a problem with live Trac and need a solution ASAP.

If you run packages like Genshi from sources, you simply need to install them in development mode:

python setup.py develop

I've completely forgot about 'develop'. Unfortunately I don't have time and patience to reread the docs I've already studied. Do I need that for every plugin? What if my plugin doesn't include setup.py? There are many questions that you don't really want to care about when your Trac is in trouble.

And of course, the same advice as in #10418: use virtualenv, it will make your life much easier.

Do you know how to make FastCGI script use virtualenv? Me not. Another problem is that I am not always root to install virtualenv on machines where Trac needs to be troubleshooted.

Then again, maybe I'm missing something. Can you give us a concrete use case where the setuptools constraint is unacceptable, and where using a virtualenv isn't possible?

The scenario I am currently facing is the bug ​http://trac-hacks.org/ticket/8545 for which I need to insert a lot of log statements into various parts of code. I can not do this on server, because installation is not versioned and I won't be able to rollback it if something goes wrong (needless to say about the speed of development in remote terminal).

So, I need to get working Trac on my machine in less than 15 minutes. Patch it, test and upload to server. I've created script to bootstrap Trac - at ​https://gist.github.com/1292939 - but it doesn't even create test environment - 'sqlite' can not be found even though is importable. I think it's because sqlite connectot is a component that is loaded by pkg_resources, and that's a big problem.

[anatoly techtonik <techtonik@…>]

Note that I am not sure that sqlite component is not loaded, but because of pkg_resources I don't have any control over loading process and can't debug it.

[Remy Blank]

Replying to anatoly techtonik <techtonik@…>:

Unfortunately I don't have time and patience to reread the docs I've already studied.

I have to admit that I'm not very inclined to optimize Trac for people who don't have the time and patience to read some docs. Especially if said docs would make these people's lives easier.

Do you know how to make FastCGI script use virtualenv? Me not. Another problem is that I am not always root to install virtualenv on machines where Trac needs to be troubleshooted.

That's another good reason to ​read the docs. The whole point of virtualenv is that you don't need to be root, and you can isolate your install from the packages installed in the system. You can install from sources if you want, and therefore make changes in the install without affecting system packages.

So, I need to get working Trac on my machine in less than 15 minutes. Patch it, test and upload to server. I've created script to bootstrap Trac - at ​https://gist.github.com/1292939 - but it doesn't even create test environment - 'sqlite' can not be found even though is importable.

This may be simpler to script in bash, and using virtualenv would indeed make your life much simpler. The following should install Trac, Genshi and any number of plugins from sources (untested, but you get the idea):

install_from_sources() {
    pushd
    svn co "$2" "$1"
    cd "$1"
    python setup.py develop -N
    popd
}

# Create virtualenv
virtualenv --distribute venv
. venv/bin/activate

# Install as egg when sources aren't needed
easy_install pytz
easy_install pygments
easy_install docutils

# Install from sources
install_from_sources genshi http://svn.edgewall.org/repos/genshi/branches/stable/0.6.x/
install_from_sources trac http://svn.edgewall.org/repos/trac/branches/0.12-stable/
install_from_sources accountmanager https://trac-hacks.org/svn/accountmanagerplugin/trunk/

# Create environment
trac-admin env initenv "My test project" sqlite:db/trac.db

No root required (you could even install virtualenv locally, and run it with python -m). To use this installation from a FCGI script, activate the virtualenv as explained in the page linked above. To use it from a shell (i.e. to run trac-admin), activate the environment by running:

. venv/bin/activate

You can edit the checked-out sources, and the changes will be picked up at the next server restart. virtualenv does all the sys.path magic, so you don't have to do it (wrong) yourself.

So, no, I still see no reason to drop pkg_resources.

[anatoly techtonik <techtonik@…>]

Replying to rblank:

Replying to anatoly techtonik <techtonik@…>:

Unfortunately I don't have time and patience to reread the docs I've already studied.

I have to admit that I'm not very inclined to optimize Trac for people who don't have the time and patience to read some docs. Especially if said docs would make these people's lives easier.

…Our mission is to help developers write great software while staying out of the way. Trac should impose as little as possible on a team's established development process and policies…

There is one little thing wrong here. Trac needs customization, because at least AccountManager plugin is the thing that almost everybody installs. While part of its mission is to stay out of the way, in reality people often need to ​debug various problems with Trac plugins and customization. And while you can pretend that plugins have nothing to do with Trac itself, it is still Trac core where a lot of stuff is imposed on the way plugins and customizations are developed.

Do you know how to make FastCGI script use virtualenv? Me not. Another problem is that I am not always root to install virtualenv on machines where Trac needs to be troubleshooted.

That's another good reason to ​read the docs. The whole point of virtualenv is that you don't need to be root, and you can isolate your install from the packages installed in the system. You can install from sources if you want, and therefore make changes in the install without affecting system packages.

I know what virtualenv is. My increased activity in this tracker is caused by dumb trac-admin . upgrade after installing new version of bitten plugin without proper source activate. It appeared that there was global version of Trac, which suddenly upgraded all environment to a new version. After that nobody could login.

People are imperfect and don't like to read the docs, so if it is possible to have a workaround for this defect - it will save a lot of time - both for you and for others.

So, I need to get working Trac on my machine in less than 15 minutes. Patch it, test and upload to server. I've created script to bootstrap Trac - at ​https://gist.github.com/1292939 - but it doesn't even create test environment - 'sqlite' can not be found even though is importable.

This may be simpler to script in bash, and using virtualenv would indeed make your life much simpler. The following should install Trac, Genshi and any number of plugins from sources (untested, but you get the idea):

There is only one problem - one of my development machines is Windows, and I am more proficient in writing batch files than bash, and if we speak about batch I prefer Python. But I got the idea.

No root required (you could even install virtualenv locally, and run it with python -m).

With my script neither root, nor virtualenv is required. =)

You can edit the checked-out sources, and the changes will be picked up at the next server restart. virtualenv does all the sys.path magic, so you don't have to do it (wrong) yourself.

Just don't forget to activate your terminal every time, or it may happen that you'll be messing with wrong sources. ,)

So, no, I still see no reason to drop pkg_resources.

I believe there is substitution of concepts from your side. It is not 'removal of pkg_resources` - it is removal of 'hard dependency' on pkg_resources, so that alternative import (component discovery) schemes are also possible.