Edgewall Software
Modify

Opened 16 years ago

Closed 15 years ago

Last modified 12 years ago

#6467 closed defect (wontfix)

Cannot allocate memory: '/usr/lib64/python2.5' after httpd restart

Reported by: ken@… Owned by: Christopher Lenz
Priority: normal Milestone:
Component: web frontend/mod_python Version: 0.11b2
Severity: normal Keywords: setuptools
Cc: tkalix@…, dfraser Branch:
Release Notes:
API Changes:
Internal Changes:

Description (last modified by Matthew Good)

OS: Fedora 8 x86_64 SELinux CPU: Core 2 Duo 2.66 MEM: 4GB

Everytime I restart my apache server I am unable to access Trac. If I refresh the page 10 times, then on the 10th refresh it works perfectly. The exact same error message is displayed those 10 times in the log:

[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] mod_python (pid=24035, interpreter='trac.mydomain.com', phase='PythonHandler', handler='trac.web.modpython_frontend'): Application error
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] ServerName: 'trac.mydomain.com'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] DocumentRoot: '/var/www/trac'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] URI: '/'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Location: '/'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Directory: None
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Filename: '/var/www/trac/'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] PathInfo: ''
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Traceback (most recent call last):
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib64/python2.5/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch\n    default=default_handler, arg=req, silent=hlist.silent)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib64/python2.5/site-packages/mod_python/importer.py", line 1229, in _process_target\n    result = _execute_target(config, req, object, arg)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib64/python2.5/site-packages/mod_python/importer.py", line 1128, in _execute_target\n    result = object(arg)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/modpython_frontend.py", line 87, in handler\n    gateway.run(dispatch_request)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 391, in dispatch_request\n    env = _open_environment(env_path, run_once=run_once)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 58, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/env.py", line 462, in open_environment\n    env = Environment(env_path)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/env.py", line 141, in __init__\n    from trac.loader import load_components
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/loader.py", line 28, in <module>\n    import pkg_resources
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 2552, in <module>\n    working_set = WorkingSet()
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 384, in __init__\n    self.add_entry(entry)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 399, in add_entry\n    for dist in find_distributions(entry, True):
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 1659, in find_on_path\n    for entry in os.listdir(path_item):
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] OSError: [Errno 12] Cannot allocate memory: '/usr/lib64/python2.5'

Attachments (0)

Change History (19)

comment:1 by Matthew Good, 16 years ago

Description: modified (diff)

formatting description

comment:2 by sid, 16 years ago

This does not appear to be a Trac bug to me.

One possibility (raised here link) is mod_python is causing the issue.

comment:3 by Christian Boos, 16 years ago

Milestone: not applicable

mod_python is known to be a memory hog… Maybe you start too many server processes, and all of them get loaded with mod_python and the dependencies, which can amount to a quite big total. Things only start to work once enough process have died and therefore freed the memory for the others. Ok, this is only a theory, but you could try to reduce memory usage following those tips or switching to wiki:TracModWSGI.

comment:4 by anonymous, 16 years ago

I have the very same problem after upgrading to Fedora 8. As written previously I just keep trying for about a dozen times and then it worked, and now it works every single time I try it, even after a restart of Apache.

comment:5 by anonymous, 16 years ago

Not restricted to Fedora 8. We see this on Fedora 7, 1G Sunfire 2100 (2.6.23.8-34.fc7). Throttled down to a minimal ServerLimit with no obvious reduction in error occurrence. Per earlier commentator, it will eventually load Trac after about 10 attempts. We suspect either a 64 bit or Python 2.5 issue… No such problems installing Trac on our 32-bit Fedora….

comment:6 by anonymous, 16 years ago

I am also experiencing the same problem on my Fedora 8 x86_64. I have 2GB memory in my machine.

comment:7 by Christian Boos, 16 years ago

Keywords: setuptools added
Milestone: 0.11.1
Priority: lownormal
Severity: minornormal

Hm, actually this is quite weird, it's not exactly an out-of-memory condition:

...   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 1659, in find_on_path\n    for entry in os.listdir(path_item):
... OSError: [Errno 12] Cannot allocate memory: '/usr/lib64/python2.5'

Might be a setuptools issue. Which setuptools version is used? Does upgrading setuptools makes a difference?

comment:8 by mishamanulis@…, 16 years ago

Try running

service httpd reload

I was having the same problem and I restarted the server using reload vs restart and that worked for me.

Could trac and/or mod_python not be cleaning up after themselves correctly?

comment:9 by ken@…, 16 years ago

Version: 0.10.40.11b2
X86_64 Core 2 Duo machine w/1G ram
Fedora 7 x86_64 with updates as of 2008Apr22
Trac 0.11b2 ( 2008Apr22 easy_install Trac==0.11b2 )
  Not Trac v0.10 available from repository)
Genshi v0.4 ( 2008Apr22 easy_install Genshi)
Python 2.5 r25:51908 2007Nov06
  Not sure which version of easy_install
Subversion 1.4.4 r25188 2007Jul26
Apache 2.2.8 2008Jan24
Not using virtual domains, but using multiple projects

Still happening. Here's what I've found out so far:

# /etc/init.d/httpd restart
  - Hit 'Reload' until 'Cannot allocate memory' error
    gets cycled through all of the background processes,
    then everything works fine

However, if you do this:

# /etc/init.d/httpd restart (immediately followed by)
# /etc/init.d/httpd reload
  - Works fine the first time

comment:10 by Ken Roberts <ken@…>, 16 years ago

Also using mod_DAV_svn and mod_python - didn't have the problem when using tracd to verify I installed it correctly.

Forgot to add, I had no authorization setup - this was just a testing run from a local machine

comment:11 by Christian Boos, 16 years ago

Milestone: 0.11.1not applicable

Well, this seems to be a mod_python quirk on this platform.

  • do you use a version of mod_python recent enough (at least version 3.3.1?)
  • looking again at the description of the ticket, I see this:
    mod_python (pid=24035, interpreter='trac.mydomain.com',
    
    It looks like you didn't set the PythonInterpreter to main_interpreter (see TracModPython#Subversionissues), maybe that will help.

comment:12 by Ken <ken@…>, 16 years ago

cboos: You mean this section in /etc/httpd/conf.d/trac.conf?
( btw - I'm not the same ken that started this thread :)

<Location /trac>
   SetHandler mod_python
   PythonInterpreter main_interpreter
   PythonHandler trac.web.modpython_frontend
   PythonOption TracEnvParentDir /var/www/html/trac
   PythonOption TracUriRoot /trac
   PythonDebug on
</Location>

comment:13 by anonymous, 16 years ago

I'm the ken that changed the milestone from 10 to 0.11b2, but the issue appears to still be here (unless the original ken and I have the same version of Apache+mod_python).
Since I used Fedora 7 repository, I'm not sure of the version of mod_python that was compiled into the apache that's installed on my machine.

comment:14 by Torsten <tkalix@…>, 16 years ago

Cc: tkalix@… added

Hello!

I have searched a bit and found a work-around using Apache and fcgi (see TracFastCgi) it seems to run fine. At least my production server is running fine for a day now ;-) I have also tried to debug the problem and used mod_python, mod_wsgi and different settings. But it just did not work.

comment:15 by felix.schwarz@…, 15 years ago

Maybe I'm wrong but this looks to me like a bug in Python's os.listdir which is fixed in Fedora since August 2008:

References:

comment:16 by Christian Boos, 15 years ago

Yes, it seems to be the case. As the above os.listdir bug happens on one of the directories from sys.path, you should check your PATH environment variable to see if it doesn't contain a folder which has a huge number of entries (like /tmp or /var/tmp).

in reply to:  16 comment:17 by dfraser, 15 years ago

Resolution: wontfix
Status: newclosed

Replying to cboos:

Yes, it seems to be the case. As the above os.listdir bug happens on one of the directories from sys.path, you should check your PATH environment variable to see if it doesn't contain a folder which has a huge number of entries (like /tmp or /var/tmp).

So this is basically an upstream bug, there is a fix available - closing unless somebody wants to propose how to work around this in trac…

(I've tested this fix on my machine by applying the patch manually and rebuilding python, and it's now working merrily…)

comment:18 by dfraser, 15 years ago

Cc: dfraser added

comment:19 by Christian Boos, 12 years ago

Milestone: not applicable

(clearing report:35)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christopher Lenz.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Christopher Lenz to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.