Cannot allocate memory: '/usr/lib64/python2.5' after httpd restart

[ken@…]

OS: Fedora 8 x86_64 SELinux CPU: Core 2 Duo 2.66 MEM: 4GB

Everytime I restart my apache server I am unable to access Trac. If I refresh the page 10 times, then on the 10th refresh it works perfectly. The exact same error message is displayed those 10 times in the log:

[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] mod_python (pid=24035, interpreter='trac.mydomain.com', phase='PythonHandler', handler='trac.web.modpython_frontend'): Application error
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] ServerName: 'trac.mydomain.com'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] DocumentRoot: '/var/www/trac'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] URI: '/'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Location: '/'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Directory: None
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Filename: '/var/www/trac/'
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] PathInfo: ''
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] Traceback (most recent call last):
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib64/python2.5/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch\n    default=default_handler, arg=req, silent=hlist.silent)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib64/python2.5/site-packages/mod_python/importer.py", line 1229, in _process_target\n    result = _execute_target(config, req, object, arg)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib64/python2.5/site-packages/mod_python/importer.py", line 1128, in _execute_target\n    result = object(arg)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/modpython_frontend.py", line 87, in handler\n    gateway.run(dispatch_request)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 391, in dispatch_request\n    env = _open_environment(env_path, run_once=run_once)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 58, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/env.py", line 462, in open_environment\n    env = Environment(env_path)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/env.py", line 141, in __init__\n    from trac.loader import load_components
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/trac/loader.py", line 28, in <module>\n    import pkg_resources
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 2552, in <module>\n    working_set = WorkingSet()
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 384, in __init__\n    self.add_entry(entry)
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 399, in add_entry\n    for dist in find_distributions(entry, True):
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143]   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 1659, in find_on_path\n    for entry in os.listdir(path_item):
[Sat Dec 08 11:50:59 2007] [error] [client 74.64.54.143] OSError: [Errno 12] Cannot allocate memory: '/usr/lib64/python2.5'

[Matthew Good]

formatting description

[sid]

This does not appear to be a Trac bug to me.

One possibility (raised here ​link) is mod_python is causing the issue.

[Christian Boos]

mod_python is known to be a memory hog… Maybe you start too many server processes, and all of them get loaded with mod_python and the dependencies, which can amount to a quite big total. Things only start to work once enough process have died and therefore freed the memory for the others. Ok, this is only a theory, but you could try to reduce memory usage following those ​tips or switching to wiki:TracModWSGI.

[anonymous]

I have the very same problem after upgrading to Fedora 8. As written previously I just keep trying for about a dozen times and then it worked, and now it works every single time I try it, even after a restart of Apache.

[anonymous]

Not restricted to Fedora 8. We see this on Fedora 7, 1G Sunfire 2100 (2.6.23.8-34.fc7). Throttled down to a minimal ServerLimit with no obvious reduction in error occurrence. Per earlier commentator, it will eventually load Trac after about 10 attempts. We suspect either a 64 bit or Python 2.5 issue… No such problems installing Trac on our 32-bit Fedora….

[anonymous]

I am also experiencing the same problem on my Fedora 8 x86_64. I have 2GB memory in my machine.

[Christian Boos]

Hm, actually this is quite weird, it's not exactly an out-of-memory condition:

...   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 1659, in find_on_path\n    for entry in os.listdir(path_item):
... OSError: [Errno 12] Cannot allocate memory: '/usr/lib64/python2.5'

Might be a setuptools issue. Which setuptools version is used? Does upgrading setuptools makes a difference?

[mishamanulis@…]

Try running

service httpd reload

I was having the same problem and I restarted the server using reload vs restart and that worked for me.

Could trac and/or mod_python not be cleaning up after themselves correctly?

[ken@…]

X86_64 Core 2 Duo machine w/1G ram
Fedora 7 x86_64 with updates as of 2008Apr22
Trac 0.11b2 ( 2008Apr22 easy_install Trac==0.11b2 )
  Not Trac v0.10 available from repository)
Genshi v0.4 ( 2008Apr22 easy_install Genshi)
Python 2.5 r25:51908 2007Nov06
  Not sure which version of easy_install
Subversion 1.4.4 r25188 2007Jul26
Apache 2.2.8 2008Jan24
Not using virtual domains, but using multiple projects

Still happening. Here's what I've found out so far:

# /etc/init.d/httpd restart
  - Hit 'Reload' until 'Cannot allocate memory' error
    gets cycled through all of the background processes,
    then everything works fine

However, if you do this:

# /etc/init.d/httpd restart (immediately followed by)
# /etc/init.d/httpd reload
  - Works fine the first time

[Ken Roberts <ken@…>]

Also using mod_DAV_svn and mod_python - didn't have the problem when using tracd to verify I installed it correctly.

Forgot to add, I had no authorization setup - this was just a testing run from a local machine

[Christian Boos]

Well, this seems to be a mod_python quirk on this platform.

• do you use a version of mod_python recent enough (at least version 3.3.1?)
• looking again at the description of the ticket, I see this:

  mod_python (pid=24035, interpreter='trac.mydomain.com',
  

  It looks like you didn't set the PythonInterpreter to main_interpreter (see TracModPython#Subversionissues), maybe that will help.

[Ken <ken@…>]

cboos: You mean this section in /etc/httpd/conf.d/trac.conf?
( btw - I'm not the same ken that started this thread :)

<Location /trac>
   SetHandler mod_python
   PythonInterpreter main_interpreter
   PythonHandler trac.web.modpython_frontend
   PythonOption TracEnvParentDir /var/www/html/trac
   PythonOption TracUriRoot /trac
   PythonDebug on
</Location>

[anonymous]

I'm the ken that changed the milestone from 10 to 0.11b2, but the issue appears to still be here (unless the original ken and I have the same version of Apache+mod_python).
Since I used Fedora 7 repository, I'm not sure of the version of mod_python that was compiled into the apache that's installed on my machine.

[Torsten <tkalix@…>]

Hello!

I have searched a bit and found a work-around using Apache and fcgi (see TracFastCgi) it seems to run fine. At least my production server is running fine for a day now ;-) I have also tried to debug the problem and used mod_python, mod_wsgi and different settings. But it just did not work.

[felix.schwarz@…]

Maybe I'm wrong but this looks to me like a bug in Python's os.listdir which is fixed in Fedora since August 2008:

References:

• ​Red Hat Bugzilla #451494
• ​http://bugs.python.org/issue3115
• ​http://bugs.python.org/issue1608818

[Christian Boos]

Yes, it seems to be the case. As the above os.listdir bug happens on one of the directories from sys.path, you should check your PATH environment variable to see if it doesn't contain a folder which has a huge number of entries (like /tmp or /var/tmp).

[dfraser]

Replying to cboos:

Yes, it seems to be the case. As the above os.listdir bug happens on one of the directories from sys.path, you should check your PATH environment variable to see if it doesn't contain a folder which has a huge number of entries (like /tmp or /var/tmp).

So this is basically an upstream bug, there is a fix available - closing unless somebody wants to propose how to work around this in trac…

(I've tested this fix on my machine by applying the patch manually and rebuilding python, and it's now working merrily…)

[Christian Boos]

(clearing report:35)