bootstrap script for Trac development

[anatoly techtonik <techtonik@…>]

After some experience with AppEngine launcher, I can not stand to re-construct the environment required for Trac each time I do a new clone anymore, so I've compiled a simple script to do the bootstrapping for development

Hope you'll find it useful - tested on 0.12.2

​https://bitbucket.org/techtonik/trac-bootstrap-script/src

[Christian Boos]

Do you imply that Trac runs on the AppEngine? Interesting.

[Remy Blank]

As I have already mentioned elsewhere, a reasonable development environment for Trac should be done with virtualenv, and not with import magic requiring patches to Trac core. It may work for you, but it's certainly not to be recommended in general.

I wouldn't be against providing a "development set-up" script in contrib, but your approach is far too fragile.

As for Trac on AppEngine, the first thing we would need is a SQL database (AFAIK, the database engine in AppEngine isn't SQL).

[anatoly techtonik <techtonik@…>]

Replying to cboos:

Do you imply that Trac runs on the AppEngine? Interesting.

No. I just think App Engine Launcher is awesome.

Replying to rblank:

As I have already mentioned elsewhere, a reasonable development environment for Trac should be done with virtualenv, and not with import magic requiring patches to Trac core. It may work for you, but it's certainly not to be recommended in general.

Please, be more specific. Importing modules by hand with priority of current checkout path in sys.path guarantees that you're executing Trac from this specific working copy. Can you guarantee that with setuptools? Patches are required to neutralize setuptools magic and bring back standard Python import behavior.

I wouldn't be against providing a "development set-up" script in contrib, but your approach is far too fragile.

Please be more specific instead of throwing vague points like 'not recommended' and 'too fragile'. Provide some arguments.

As for Trac on AppEngine, the first thing we would need is a SQL database (AFAIK, the database engine in AppEngine isn't SQL).

​http://googleappengine.blogspot.com/2011/10/google-cloud-sql-your-database-in-cloud.html

[Remy Blank]

Replying to anatoly techtonik <techtonik@…>:

Please, be more specific. Importing modules by hand with priority of current checkout path in sys.path guarantees that you're executing Trac from this specific working copy. Can you guarantee that with setuptools? Patches are required to neutralize setuptools magic and bring back standard Python import behavior.

You are doing it wrong. See #10419.

​http://googleappengine.blogspot.com/2011/10/google-cloud-sql-your-database-in-cloud.html

Heh, I wasn't sure anymore if this was already public information. I have to admit that I thought of Trac when I first heard about it ;)

[anatoly techtonik <techtonik@…>]

Replying to rblank:

Replying to anatoly techtonik <techtonik@…>:

Please, be more specific. Importing modules by hand with priority of current checkout path in sys.path guarantees that you're executing Trac from this specific working copy. Can you guarantee that with setuptools? Patches are required to neutralize setuptools magic and bring back standard Python import behavior.

You are doing it wrong. See #10419.

Yet another meaningless comment. Can I ask you to repeat or copy-paste arguments, because it is really hard to track things you want to say through the links to various comments that were probably used in another context. So, the question is:

Does created virtualenv guarantee that if you're execute Trac server from the current checkout - the sources that are executed are the same sources from the same checkout?

[Remy Blank]

Replying to anatoly techtonik <techtonik@…>:

Yet another meaningless comment.

It's actually much more relevant that you think. You know, I'm going to stop wasting time on this. I suggested several times to read the documentation of virtualenv, because I know it will solve your problem. I provided a complete script that does exactly what you want, at least as far as I can understand. If you don't want to read the doc and / or try the script, that's fine by me, but there's no need to post insulting comments.

Does created virtualenv guarantee that if you're execute Trac server from the current checkout - the sources that are executed are the same sources from the same checkout?

Yes, if you install into the virtualenv with python setup.py develop.

[anatoly techtonik <techtonik@…>]

Replying to rblank:

Replying to anatoly techtonik <techtonik@…>:

Yet another meaningless comment.

It's actually much more relevant that you think. You know, I'm going to stop wasting time on this. I suggested several times to read the documentation of virtualenv, because I know it will solve your problem

Sorry if it was insulting. The point is - virtualenv doesn't seems to solve my problem. See below.

Does created virtualenv guarantee that if you're execute Trac server from the current checkout - the sources that are executed are the same sources from the same checkout?

Yes, if you install into the virtualenv with python setup.py develop.

And how many clones can I run from the same virtualenv? I am not sure, but I guess only one - the last one installed with python setup.py develop. I usually maintain multiple clones - one for each patch in process. So if I want to factor out some big patch into a different branch (clone) - I need to recreate this virtualenv, deactivate, activate, and run server from new copy (and be extra careful about what and where from I execute).

With my script I just execute it from the corresponding clone and doesn't care about which python setup.py develop was called last.

[Christian Boos]

I had a look at your bootstrap.py … well, as this pretty much tries to undo a lot of things we do in Trac w.r.t. setuptools, so I don't see how it could make sense to integrate this in Trac itself.

If you really think this approach has some use, then you should continue to maintain this script externally, otherwise there are plenty of other ways to develop with Trac within the existing framework.

For example, I personally don't even use virtualenv and just add PYTHONPATH=. in front of the command line (see also the source:trunk/Makefile​). This works well enough for me, you just have to be careful to use a Python installation (or virtualenv) in which no Trac has been installed.

[anatoly techtonik <techtonik@…>]

Replying to cboos:

I had a look at your bootstrap.py … well, as this pretty much tries to undo a lot of things we do in Trac w.r.t. setuptools, so I don't see how it could make sense to integrate this in Trac itself.

The script is a crude hack - a demonstration that Trac can work without setuptools. No wonder it doesn't make sense to you. However, it solves a very important user story - makes Trac development setup and testing easier. Sophisticate development setup is the primary reason I don't submit patches for Trac anymore.

If you really think this approach has some use, then you should continue to maintain this script externally, otherwise there are plenty of other ways to develop with Trac within the existing framework.

I know only one way - trac setup.py develop from here - TracDev/DevelopmentEnvironmentSetup

For example, I personally don't even use virtualenv

Shame on you. =)

and just add PYTHONPATH=. in front of the command line (see also the source:trunk/Makefile​). This works well enough for me, you just have to be careful to use a Python installation (or virtualenv) in which no Trac has been installed.

you just have to be careful to use a Python installation (or virtualenv) in which //no// Trac has been installed - you won't believe, but all this stuff had started after several wasted hours (days and maybe even weeks) thanks to damn setuptools magic that was loading wrong version of the code right after the first package instruction. I couldn't trace what setuptools was doing, but I believe that was the time I have to learn about metaclasses and Trac interface metamagic. It didn't help at that time, so wish it could be easier for other people.

[anatoly techtonik <techtonik@…>]

So, the proposal is to think if setuptools loader can be made optional (or replaceable) for scripts like mine without having to maintain patches for every Trac version.

[lkraav <leho@…>]

Hmmm I looked at the bootstrap script and it made my head spin a little. :) Then I re-read the thread here and I think comment:7 is the most understandable description of the problem for me. One I can definitely feel is an issue. Separate virtualenvs for different feature clones that each need a complete plugin, python modules, etc setup is definitely bleh in my book.

Whether anatoly's script is the answer is above me right now. Either way I'll keep this ticket in the back of my mind, I have a feeling I'm going to have to work it out for my own scenario sooner or later. I hope anatoly's not feeling his effort has been in vain and he'll keep submitting whatever he thinks is good in the future, whether accepted in core or not. Thought diversity ftw.

[Christian Boos]

Just in case I wasn't clear enough (comment:10): using multiple source code checkouts of Trac from a single Python installation (or alternatively from a single virtualenv) is supported right now.

For example, consider the following organization:

~/Trac
  |_ testenv
  |_ trunk-feature1
  | |_ Trac.egg-info
  | |_ trac
  | |_ tracopt
  |_ trunk-feature2
    |_ Trac.egg-info
    |_ trac
    |_ tracopt
~/Python
  |_ bin
  |_ Lib
    |_ site-packages
       (no trac here!)

To work on a 3rd feature …

$ cd ~/Trac
$ svn co http://svn.edgewall.org/trac/trunk trunk-feature3
$ cd trunk-feature3
$ ~/Python/bin/python setup.py egg_info

You need to do the above once per checkout. The presence of the Trac.egg-info directory and its content is all what is required for setuptools to work.

$ PYTHONPATH=. ~/Python/bin/python trac/admin/console.py ../testenv permission list
...
$ PYTHONPATH=. ~/Python/bin/python trac/web/standalone.py -p 9999 ../testenv

And it's even simpler if you use the Makefile.

No need to change anything in Trac, really. While having one checkout for each feature is perhaps not the easiest way to organize your development, it's possible to do so and it's quite close to the one checkout per development line model that I use (i.e. one for 0.12-stable, one for trunk).

[lkraav <leho@…>]

Hmmm, interesting. I think git new-workdir trac.git trac-feature1.git feature1 is going to work very well with that.

[anatoly techtonik <techtonik@…>]

The development environment still depends on system state, and that suxx, because it is easy to miss the fact that Trac is installed and run python setup.py develop as advised in development documentation. With 95% probability anyone starting to use this PYTHONPATH method will sooner or later run into the conflict with installed version.

This can be avoided with development bootstrap script, where can be cleaned of hacks by implementing support for alternative plugin/component discovery interfaces in Trac.

[Ryan J Ollos` <ryano@…>]

I have a pretty ugly shell script which installs Trac in a virtualenv. I create a new environment for every ticket I work on. It takes about 30 seconds to run, then the following commands and I'm off and running to dev for ticket #10433.

./install_tracdev_env t10433 trac-trunk genshi-0.6.0
cd t10433
. bin/active
. start_tracd

There have been many days when I've been active on trac-hacks and created 15-20 new dev envs per day, pushed the patches and then deleted all of them.

Anyway, I'm hoping to improve the script soon to support multiple DB backends, run Trac from Apache, etc …

[Christian Boos]

Replying to anatoly techtonik <techtonik@…>:

The development environment still depends on system state, and that suxx, because it is easy to miss the fact that Trac is installed and run python setup.py develop as advised in development documentation.

I'll grant you that the current TracDev/DevelopmentEnvironmentSetup is dedicated to the one checkout, one virtualenv approach, which doesn't mix well with the PYTHONPATH=. method described above. That can be fixed, along with hints about the more sophisticated ways to maintain different patch sets in a single checkout (Mercurial's MQ, branches or bookmarks, Git topic branches). TODO

With 95% probability anyone starting to use this PYTHONPATH method will sooner or later run into the conflict with installed version.

If one can find a way to detect such a situation and give an appropriate warning at startup (or even abort), I'm interested. OPEN

This can be avoided with development bootstrap script, where can be cleaned of hacks by implementing support for alternative plugin/component discovery interfaces in Trac.

This is not going to happen.

[osimons]

I no longer use virtualenv for development either, instead I use make with a custom additions to Makefile.cfg that handle my typical needs - most notably:

• make testenv (and make clean-testenv) that hooks into functional tests to just create the complete test project with repository infrastructure.
• make python to run a python code or start interactive shell.
• make trac-admin for interactive admin mode.

With PYTHONPATH and all other variables defined in Makefile.cfg, everything runs in a predictable way - both my custom commands, and the make server and various test commands defined in the main Makefile.

Using Mercurial and MQ patch queues, I can swtich and change code as needed in-place and just use make commands to test in any way I need.

I've made simplified Makefile and Makefile.cfg for each of my plugins too, with adjustments such as enabling plugins and adding settings as part of the relevant make command.

[Ryan J Ollos <ryano@…>]

If you don't mind sharing one of your Makefile.cfg files, I'd be really interested to take a look. I've been looking for something more configurable for each plugin than my dumb bash script allows and had previously been considering adding and reading from a config.ini in the bash script. I suspect your solution is better.

[lkraav <leho@…>]

I must say, seeing some screencasts of how exactly you guys do however you do what you do with some commentary, would be awesomely helpful. Trying to wrap my head around all these different approaches is making my head spin twice as fast as anatoly's script alone did. A tiny bit of ​Screencast-o-matic time perhaps, anyone?

[Christian Boos]

Replying to osimons:

• make testenv (and make clean-testenv) that hooks into functional tests to just create the complete test project with repository infrastructure.
• make python to run a python code or start interactive shell.
• make trac-admin for interactive admin mode.

These would be valuable additions. I thought about adding the last two myself ;-)

To people wondering why it can be better to use e.g. make trac-admin rather than PYTHONPATH=. python trac/admin/console.py: you can quickly reuse the other settings you made in Makefile.cfg, like specifying which Python version to use (python=25), which db backend, etc.

[osimons]

A simple Makefile.cfg adding custom settings and commands.

[osimons]

I've added my attachment:Makefile.cfg​ as an example. Its sets paths that I can adjust, adds other settings needed, and defines the custom commands I mentioned.

It does not use the advanced python and db configuration possibilities from Makefile.cfg.sample shipped with Trac, as I really never got my head wrapped around how it is supposed to work… Nor have I spent time making my config cross-plattform. No doubt Christian can transform my file into something that works for all - but understood by few ;-)

[Christian Boos]

Replying to cboos:

Replying to osimons:

• make testenv (and make clean-testenv) that hooks into functional tests to just create the complete test project with repository infrastructure.
• make python to run a python code or start interactive shell.
• make trac-admin for interactive admin mode.

These would be valuable additions. I thought about adding the last two myself ;-)

I added these two in r12167, starting from the 0.12-stable branch.

[Christian Boos]

Replying to osimons:

It does not use the advanced python and db configuration possibilities from Makefile.cfg.sample shipped with Trac, as I really never got my head wrapped around how it is supposed to work…

Oh, sorry I missed this… Well, it's really simple. (Hopefully) better explanations added in r12168.

Nor have I spent time making my config cross-plattform. No doubt Christian can transform my file into something that works for all - but understood by few ;-)

Hm, nice try but it didn't happen (yet) ;-)

[Ryan J Ollos]

Moved ticket assigned to next-stable-1.0.x since maintenance of 1.0.x is coming to a close. Please move the ticket back if it's critical to fix on 1.0.x.

[Ryan J Ollos]

Milestone renamed