Edgewall Software
Modify

Opened 3 years ago

Last modified 9 months ago

#10658 new defect

improve support for setuptools/distribute VCS plugins

Reported by: benjamin.a.lau@… Owned by: cboos
Priority: normal Milestone: not applicable
Component: general Version: 0.13dev
Severity: major Keywords: setuptools distribute
Cc: benjamin.a.lau@…
Release Notes:
API Changes:

Description (last modified by cboos)

Subversion 1.7 is still not correctly supported by setuptools or even distribute. But it's not just svn, to be properly supported the other VCS need setuptools plugins (see comment:3).

Unfortunately they don't all yet support the tag_svn_revision = true feature, so we would first need to extend distribute for that…

Original Report

Getting IndexError: list index out of range when trying to install Trac 0.13 on OSX

When I try to install try (either via easy_install or from sources) I'm getting the error below.

I think I've managed to track down the root cause of the problem, but I'm unsure of how to go about fixing it. I think the problem here is that newer versions of SVN (I've got 1.7.4 installed on my machine) have some sort of new format and setuptools doesn't know how to deal with it. (I'm running through a debugging scenario while I write this…)

So I tried with a copy of svn 1.6 that I had on my machine as well… and it doesn't have this problem. It looks like the entire format of the .svn/entries field has been changed for 1.7 and that setup tools will need to be modified to know about this. It seems like other people have also been having this issue, but it can't be super widespread or there'd would be a ticket for this already I suspect.

Error:

Traceback (most recent call last):
  File "/usr/bin/easy_install-2.7", line 10, in <module>
    load_entry_point('setuptools==0.6c12dev-r85381', 'console_scripts', 'easy_install')()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 1712, in main
    with_ei_usage(lambda:
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 1700, in with_ei_usage
    return f()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 1716, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 211, in run
    self.easy_install(spec, not self.no_deps)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 446, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 476, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 930, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/sandbox.py", line 62, in run_setup
    lambda: execfile(
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/sandbox.py", line 105, in run
    return func()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/sandbox.py", line 64, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 162, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/bdist_egg.py", line 94, in finalize_options
    ei_cmd = self.ei_cmd = self.get_finalized_command("egg_info")
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/egg_info.py", line 85, in finalize_options
    self.vtags = self.tags()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/egg_info.py", line 185, in tags
    ):  version += '-r%s' % self.get_svn_revision()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/egg_info.py", line 233, in get_svn_revision
    dirurl = data[0][3]
IndexError: list index out of range

Workaround

pip install distribute

(you won't get an error, but won't get the revision number either)

Attachments (1)

t10658-require-setuptools-plugin-if-needed-r11028.patch (2.1 KB) - added by cboos 3 years ago.
setup: if not using a Subversion ⇐ 1.7 checkout, require appropriate setuptools plugin

Download all attachments as: .zip

Change History (15)

comment:1 follow-up: Changed 3 years ago by cboos

It's a well known recurring issue:

(and there was a ticket here already ;-) #10479)

But given that there's no sign of progress there, we should probably try to cope with it at our level.

comment:2 in reply to: ↑ 1 Changed 3 years ago by rblank

Replying to cboos:

But given that there's no sign of progress there, we should probably try to cope with it at our level.

Yes, especially since it's a recurring issue that is never fixed quickly upstream. IIUC, we use setuptools' SVN support to avoid having to write our own MANIFEST.in, and to get the SVN revision. We should use a hand-written MANIFEST.in, and get the revision from the command-line svn (although I'm not sure how we can feed it to setuptools). Both will also help for the day we migrate to Mercurial or Git.

Or we could write an alternative SVN support using the command-line svn tool exclusively, and feed it upstream. Or even generic support for arbitrary, configurable command-lines, so that we can support different VC backends.

comment:3 follow-ups: Changed 3 years ago by cboos

I was simply thinking about doing the workaround mentioned in #10479 programmatically. That is, if there's no .svn/entries file, just ensure the option [egg_info] tag_svn_revision is set to false. It must be possible to do something like that…

As for the pluggable VCS support in setuptools/distribute, this already exists:

Nevertheless, the workaround I suggested above would still be useful for the vast majority of unsuspecting users (I didn't know about that setuptools_subversion plugin until now). We could perhaps just refine that workaround so that it keeps tag_svn_revision = true if that setuptools_subversion plugin is present.

comment:4 in reply to: ↑ 3 Changed 3 years ago by cboos

Replying to cboos:

I was simply thinking about doing the workaround mentioned in #10479 programmatically. That is, if there's no .svn/entries file, just ensure the option [egg_info] tag_svn_revision is set to false. It must be possible to do something like that…

Oh, that would avoid the problem with the version but not with the MANIFEST.in…

So what we could do is to detect which VCS is being used and only proceed if the appropriate plugin is installed, abort and tell the user to install it otherwise.

comment:5 in reply to: ↑ 3 ; follow-up: Changed 3 years ago by rblank

Replying to cboos:

As for the pluggable VCS support in setuptools/distribute, this already exists:

Yes, I know about those. My suggestion was to implement an alternative SVN support module that use the command-line tool (I didn't know about setuptools_subversion, maybe that's what it does) or even a generic setuptools_vc where you define a command-line for each operation (get files, get revision, etc.) and capture its output.

So what we could do is to detect which VCS is being used and only proceed if the appropriate plugin is installed, abort and tell the user to install it otherwise.

Or hand-write MANIFEST.in. Would that be so bad? It probably wouldn't even have to be changed that often, as it supports wildcards.

comment:6 in reply to: ↑ 5 Changed 3 years ago by cboos

Replying to rblank:

Replying to cboos:

As for the pluggable VCS support in setuptools/distribute, this already exists:

Yes, I know about those. My suggestion was to implement an alternative SVN support module that use the command-line tool (I didn't know about setuptools_subversion, maybe that's what it does)

Yes, that's what setuptools_subversion does and it works with a svn 1.7 checkout. I've also tested the setuptools-git and setuptools_hg, they both work OK.

Note that with all of (even setuptools_subversion) we end up with a -r0. They indeed only implement the [setuptools.file_finders] entry point.

or even a generic setuptools_vc where you define a command-line for each operation (get files, get revision, etc.) and capture its output.

Well, those plugins indeed look similar, but refactoring them sounds more a job for setuptools/distribute rather than for us… and in the end you still have to install such a setuptools_vc plugin and adapt it (by configuration or with a sub-plugin?) for the VCS you use. So for the end user, there would be no real simplification over doing easy_install setuptools_<your_VCS>.

So what we could do is to detect which VCS is being used and only proceed if the appropriate plugin is installed, abort and tell the user to install it otherwise.

Or hand-write MANIFEST.in. Would that be so bad? It probably wouldn't even have to be changed that often, as it supports wildcards.

It wouldn't be that bad, yes.

Though there are a few things unclear for me with this:

The alternative I've been testing in the meantime is to ask the user to install the proper setuptools plugin if it's needed.

Changed 3 years ago by cboos

setup: if not using a Subversion ⇐ 1.7 checkout, require appropriate setuptools plugin

comment:7 Changed 3 years ago by cboos

Note that I'm not sure that by using setuptools_subversion the problem reported here will be avoided, as I don't get a backtrace when trying (and failing) to get the svn version with a Subversion 1.7 checkout, only a warning, and the same warning is still emitted when I'm using the plugin.

So that must be yet another problem with setuptools (Mac? version specific?).

Your setuptools==0.6c12dev-r85381 seems to be coming from PEAK's original setuptools. Could you try either a more recent version from there (http://svn.python.org/projects/sandbox/branches/setuptools-0.6/) or simply install pypi:distribute?

comment:8 Changed 3 years ago by anonymous

So I tried installing distribute and tested it out. You don't get an error, but you do end up with a -r0 instead of the actual revision. But at least it builds. So that's kind of like progress.

comment:9 Changed 3 years ago by benjamin.a.lau@…

Oh sorry… I got the warning about the unrecognized format:

unrecognized .svn/entries format; skipping .

I just didn't see it hiding what back at the beginning.

comment:10 Changed 2 years ago by Ryan J Ollos <ryano@…>

  • Keywords setuptools added

comment:11 Changed 2 years ago by cboos

  • Description modified (diff)
  • Keywords distribute added
  • Milestone set to not applicable
  • Owner set to cboos
  • Severity changed from normal to major

comment:12 Changed 2 years ago by cboos

  • Description modified (diff)
  • Summary changed from Getting IndexError: list index out of range when trying to install Trac 0.13 on OSX to improve support for setuptools/distribute VCS plugins

comment:13 Changed 9 months ago by anonymous

I just encountered this issue when doing some upgrades on my Trac installation, that has had subversion 1.7 for a while. After a bit of digging, I discovered that a newer version of setuptools (1.2 or higher) fixed this problem.

For details, see https://pypi.python.org/pypi/setuptools#changes

comment:14 Changed 9 months ago by jomae

Thanks for information. I confirmed with Python 2.4, setuptools 1.4.2 and Subversion 1.7.6. That works well. (setuptools 2.0 dropped support for Python 2.4 and 2.5).

$ ~/venv/py24/bin/easy_install -Z setuptools==1.4.2
$ ~/venv/py24/bin/easy_install --version
setuptools 1.4.2
$ svn --version | head -1
svn, version 1.7.6 (r1370777)
$ svn co http://svn.edgewall.com/repos/trac/branches/0.12-stable
$ cd 0.12-stable
$ ~/venv/py24/bin/python setup.py egg_info
$ grep '^Version:' Trac.egg-info/PKG-INFO
Version: 0.12.6dev-r12540

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain cboos.
as The resolution will be set. Next status will be 'closed'.
The owner will be changed from cboos to anonymous. Next status will be 'assigned'.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.