improve support for setuptools/distribute VCS plugins

[benjamin.a.lau@…]

Subversion 1.7 is still not correctly supported by setuptools or even distribute. But it's not just svn, to be properly supported the other VCS need setuptools plugins (see comment:3).

Unfortunately they don't all yet support the tag_svn_revision = true feature, so we would first need to extend distribute for that…

Original Report

Getting IndexError: list index out of range when trying to install Trac 0.13 on OSX

When I try to install try (either via easy_install or from sources) I'm getting the error below.

I think I've managed to track down the root cause of the problem, but I'm unsure of how to go about fixing it. I think the problem here is that newer versions of SVN (I've got 1.7.4 installed on my machine) have some sort of new format and setuptools doesn't know how to deal with it. (I'm running through a debugging scenario while I write this…)

So I tried with a copy of svn 1.6 that I had on my machine as well… and it doesn't have this problem. It looks like the entire format of the .svn/entries field has been changed for 1.7 and that setup tools will need to be modified to know about this. It seems like ​other people have also been having this issue, but it can't be super widespread or there'd would be a ticket for this already I suspect.

Error:

Traceback (most recent call last):
  File "/usr/bin/easy_install-2.7", line 10, in <module>
    load_entry_point('setuptools==0.6c12dev-r85381', 'console_scripts', 'easy_install')()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 1712, in main
    with_ei_usage(lambda:
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 1700, in with_ei_usage
    return f()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 1716, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 211, in run
    self.easy_install(spec, not self.no_deps)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 446, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 476, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 930, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/sandbox.py", line 62, in run_setup
    lambda: execfile(
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/sandbox.py", line 105, in run
    return func()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/sandbox.py", line 64, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 162, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/bdist_egg.py", line 94, in finalize_options
    ei_cmd = self.ei_cmd = self.get_finalized_command("egg_info")
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/egg_info.py", line 85, in finalize_options
    self.vtags = self.tags()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/egg_info.py", line 185, in tags
    ):  version += '-r%s' % self.get_svn_revision()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/egg_info.py", line 233, in get_svn_revision
    dirurl = data[0][3]
IndexError: list index out of range

Workaround

pip install distribute

(you won't get an error, but won't get the revision number either)

[Christian Boos]

It's a well known recurring issue:

• ​http://bugs.python.org/setuptools/issue79 (the original setuptools project)
• ​https://bitbucket.org/tarek/distribute/issue/252/subversion-17-introduces-new-storage

(and there was a ticket here already ;-) #10479)

But given that there's no sign of progress there, we should probably try to cope with it at our level.

[Remy Blank]

Replying to cboos:

But given that there's no sign of progress there, we should probably try to cope with it at our level.

Yes, especially since it's a recurring issue that is never fixed quickly upstream. IIUC, we use setuptools' SVN support to avoid having to write our own MANIFEST.in, and to get the SVN revision. We should use a hand-written MANIFEST.in, and get the revision from the command-line svn (although I'm not sure how we can feed it to setuptools). Both will also help for the day we migrate to Mercurial or Git.

Or we could write an alternative SVN support using the command-line svn tool exclusively, and feed it upstream. Or even generic support for arbitrary, configurable command-lines, so that we can support different VC backends.

[Christian Boos]

I was simply thinking about doing the workaround mentioned in #10479 programmatically. That is, if there's no .svn/entries file, just ensure the option [egg_info] tag_svn_revision is set to false. It must be possible to do something like that…

As for the pluggable VCS support in setuptools/distribute, this already exists:

• ​pypi:setuptools-git, ​pypi:setuptools_hg, ​pypi:setuptools_bzr, ​pypi:setuptools_mtn, ​pypi:setuptools_darcs
• there's even a … ​pypi:setuptools_subversion which claims to support 1.7 (not tested)

Nevertheless, the workaround I suggested above would still be useful for the vast majority of unsuspecting users (I didn't know about that setuptools_subversion plugin until now). We could perhaps just refine that workaround so that it keeps tag_svn_revision = true if that setuptools_subversion plugin is present.

[Christian Boos]

Replying to cboos:

I was simply thinking about doing the workaround mentioned in #10479 programmatically. That is, if there's no .svn/entries file, just ensure the option [egg_info] tag_svn_revision is set to false. It must be possible to do something like that…

Oh, that would avoid the problem with the version but not with the MANIFEST.in…

So what we could do is to detect which VCS is being used and only proceed if the appropriate plugin is installed, abort and tell the user to install it otherwise.

[Remy Blank]

Replying to cboos:

As for the pluggable VCS support in setuptools/distribute, this already exists:

• ​pypi:setuptools-git, ​pypi:setuptools_hg, ​pypi:setuptools_bzr, ​pypi:setuptools_mtn, ​pypi:setuptools_darcs

Yes, I know about those. My suggestion was to implement an alternative SVN support module that use the command-line tool (I didn't know about setuptools_subversion, maybe that's what it does) or even a generic setuptools_vc where you define a command-line for each operation (get files, get revision, etc.) and capture its output.

So what we could do is to detect which VCS is being used and only proceed if the appropriate plugin is installed, abort and tell the user to install it otherwise.

Or hand-write MANIFEST.in. Would that be so bad? It probably wouldn't even have to be changed that often, as it supports wildcards.

[Christian Boos]

Replying to rblank:

Replying to cboos:

As for the pluggable VCS support in setuptools/distribute, this already exists:

• ​pypi:setuptools-git, ​pypi:setuptools_hg, ​pypi:setuptools_bzr, ​pypi:setuptools_mtn, ​pypi:setuptools_darcs

Yes, I know about those. My suggestion was to implement an alternative SVN support module that use the command-line tool (I didn't know about setuptools_subversion, maybe that's what it does)

Yes, that's what setuptools_subversion does and it works with a svn 1.7 checkout. I've also tested the setuptools-git and setuptools_hg, they both work OK.

Note that with all of (even setuptools_subversion) we end up with a -r0. They indeed only implement the [setuptools.file_finders] entry point.

or even a generic setuptools_vc where you define a command-line for each operation (get files, get revision, etc.) and capture its output.

Well, those plugins indeed look similar, but refactoring them sounds more a job for setuptools/distribute rather than for us… and in the end you still have to install such a setuptools_vc plugin and adapt it (by configuration or with a sub-plugin?) for the VCS you use. So for the end user, there would be no real simplification over doing easy_install setuptools_<your_VCS>.

So what we could do is to detect which VCS is being used and only proceed if the appropriate plugin is installed, abort and tell the user to install it otherwise.

Or hand-write MANIFEST.in. Would that be so bad? It probably wouldn't even have to be changed that often, as it supports wildcards.

It wouldn't be that bad, yes.

Though there are a few things unclear for me with this:

• wouldn't that be redundant with package_data and then how come package_data is not sufficient in the first place?
• is MANIFEST.in "future" safe? There's a huge ​Deprecate MANIFEST.in thread started by Tarek on distutils-sig, though it's not clear for me what the outcome was (on one hand I see there's still a MANIFEST.in in Distribute itself, OTOH I saw that ​The MANIFEST.in is definitely gone in distutils2.)
• related interesting reading: ​http://wiki.python.org/moin/Distutils/ManifestPluginSystem

The alternative I've been testing in the meantime is to ask the user to install the proper setuptools plugin if it's needed.

[Christian Boos]

setup: if not using a Subversion ⇐ 1.7 checkout, require appropriate setuptools plugin

[Christian Boos]

Note that I'm not sure that by using setuptools_subversion the problem reported here will be avoided, as I don't get a backtrace when trying (and failing) to get the svn version with a Subversion 1.7 checkout, only a warning, and the same warning is still emitted when I'm using the plugin.

So that must be yet another problem with setuptools (Mac? version specific?).

Your setuptools==0.6c12dev-r85381 seems to be coming from PEAK's original setuptools. Could you try either a more recent version from there (​http://svn.python.org/projects/sandbox/branches/setuptools-0.6/) or simply install ​pypi:distribute?

[anonymous]

So I tried installing distribute and tested it out. You don't get an error, but you do end up with a -r0 instead of the actual revision. But at least it builds. So that's kind of like progress.

[benjamin.a.lau@…]

Oh sorry… I got the warning about the unrecognized format:

unrecognized .svn/entries format; skipping .

I just didn't see it hiding what back at the beginning.

[anonymous]

I just encountered this issue when doing some upgrades on my Trac installation, that has had subversion 1.7 for a while. After a bit of digging, I discovered that a newer version of setuptools (1.2 or higher) fixed this problem.

For details, see ​https://pypi.python.org/pypi/setuptools#changes

[Jun Omae]

Thanks for information. I confirmed with Python 2.4, setuptools 1.4.2 and Subversion 1.7.6. That works well. (setuptools 2.0 dropped support for Python 2.4 and 2.5).

$ ~/venv/py24/bin/easy_install -Z setuptools==1.4.2
$ ~/venv/py24/bin/easy_install --version
setuptools 1.4.2
$ svn --version | head -1
svn, version 1.7.6 (r1370777)
$ svn co http://svn.edgewall.com/repos/trac/branches/0.12-stable
$ cd 0.12-stable
$ ~/venv/py24/bin/python setup.py egg_info
$ grep '^Version:' Trac.egg-info/PKG-INFO
Version: 0.12.6dev-r12540

[anonymous]

​Setuptools 10.0 removed svn support and now always silently generated -r0 versions.

[Peter Suter]

Replying to anonymous:

​Setuptools 10.0 removed svn support and now always silently generated -r0 versions.

This and other problems suggest tag_svn_revision is falling out of favor and not worth the trouble.