Edgewall Software
Modify

Opened 10 years ago

Closed 9 years ago

Last modified 9 years ago

#5782 closed defect (fixed)

Browse source don't deallocate memory

Reported by: didley@… Owned by: Christian Boos
Priority: high Milestone: 0.11
Component: version control/browser Version: devel
Severity: major Keywords: svn pool memory
Cc:
Release Notes:
API Changes:

Description

If I browse my repository with trac and if there a folder with files to list, trac takes memory for this listing. After this trac don't deallocate the memory. If I list the files again trac allocate memory again without to deallocate the memory before. I have always to restart tracd to deallocate the memory.

First of all I don't understand why trac allocate memory for a listing of files. And why don't deallocate trac this memory.

Have I anything to configure to use the standalone server?

Attachments (0)

Change History (19)

comment:1 Changed 10 years ago by Emmanuel Blot

Milestone: 0.11

This is the very nature of Python to manage the memory.

To answer your question:

I don't understand why trac allocate memory for a listing of files

Whatever the system, even a basic C call, you need to allocate memory to list the content of a directory. Moreover:

  • Trac needs to access the Subversion repository to get the "list of files" for a revision, and the SVN API requires to allocate some memory to perform this kind of operation - as many other SVN calls.
  • Trac needs memory to render the list of files (along with their properties) as HTML
  • As with any other request, Trac - or the underlying Python VM - needs to allocate some memory for temp objects that are used to handle and process the client request

There is no guarantee (nor need) that the memory is released as soon as it not used anymore. The Python garbage collector collects and releases unused memory blocks when the VM needs more memory.

You need to perform several requests to check how tracd works: the memory allocation will grow to some level, and then will stabilize even when other web requests are served. The amount of memory can grow high, but this is the normal behviour.

Propose to close as worksforme, except if you observe that the memory allocation keeps growing and never stabilizes.

comment:2 Changed 10 years ago by Emmanuel Blot

As an example: on my machine, tracd eats up memory at a high rate up to 160 or 170MB, then successive calls to the browser show a nearly constant memory allocation level: some calls increase a bit the allocated memory, some other calls show a small decrease of memory usage.

comment:3 Changed 10 years ago by didley@…

You are right! All works fine if I use folders with round about fifty files in it. No file is bigger than 16KB. The memory is stabilizing

If I use a folder with 1600 files and no one is bigger than 16 KB the memory grows from 0.2 GB to 1.5 GB. Tracd gives the logging

Exception happened during processing of request from ('192.168.2.88', 2252)
Traceback (most recent call last):
  File "D:\Programme\Python25\lib\SocketServer.py", line 463, in process_request_thread
    self.finish_request(request, client_address)
  File "D:\Programme\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "D:\Programme\Python25\lib\SocketServer.py", line 521, in __init__
    self.handle()
  File "D:\Programme\Python25\lib\BaseHTTPServer.py", line 316, in handle
    self.handle_one_request()
  File "d:\programme\python25\lib\site-packages\Trac-0.11dev_r5555-py2.5.egg\trac\web\wsgi.py", line
 174, in handle_one_request
    gateway.run(self.server.application)
  File "d:\programme\python25\lib\site-packages\Trac-0.11dev_r5555-py2.5.egg\trac\web\wsgi.py", line
 97, in run
    self._write('')
  File "d:\programme\python25\lib\site-packages\Trac-0.11dev_r5555-py2.5.egg\trac\web\wsgi.py", line
 195, in _write
    self.handler.send_response(int(status[:3]))
  File "D:\Programme\Python25\lib\BaseHTTPServer.py", line 370, in send_response
    self.send_header('Server', self.version_string())
  File "D:\Programme\Python25\lib\BaseHTTPServer.py", line 376, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
  File "D:\Programme\Python25\lib\socket.py", line 261, in write
    self.flush()
  File "D:\Programme\Python25\lib\socket.py", line 248, in flush
    self._sock.sendall(buffer)
error: (10053, 'Software caused connection abort')

But tracd gives the request after a minute back. Now I change for example to a wiki page the response needs a long time. And now I go back to the folder with these much files. The memory grows up to 2.2 GB and tracd gives the logging.

Exception happened during processing of request from ('192.168.2.88', 2744)
Traceback (most recent call last):
  File "D:\Programme\Python25\lib\SocketServer.py", line 222, in handle_request
    self.process_request(request, client_address)
  File "D:\Programme\Python25\lib\SocketServer.py", line 476, in process_request
    t.start()
  File "D:\Programme\Python25\lib\threading.py", line 434, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread

The OS gives the message 'Runtime error abnormal program termination python.exe'.

After this I stopped tracd but the python.exe is running as process. So I have to kill these process.

comment:4 Changed 10 years ago by didley@…

Other errormessage to the same topic
Trac detected an internal error:
Programming Error: library routine called out of sequence

File "d:\programme\python25\lib\site-packages\Trac-0.11dev .5.egg\trac\web\main.py", line 406, in dispatch_request dispatcher.dispatch(req) File "d:\programme\python25\lib\site-packages\Trac-0.11dev-py2.5.egg\trac\web\main.py", line 216, in dispatch resp = chosen_handler.process_request(req) File "d:\programme\python25\lib\site-packages\Trac-0.11dev-py2.5.egg\trac\versioncontrol\web_ui\browser.py", line 361, in process_request 'dir': node.isdir and self._render_dir(req, repos, node, rev), File "d:\programme\python25\lib\site-packages\Trac-0.11dev-py2.5.egg\trac\versioncontrol\web_ui\browser.py", line 384, in _render_dir changes = get_changes(repos, [i.rev for i in entries]) File "d:\programme\python25\lib\site-packages\Trac-0.11dev-py2.5.egg\trac\versioncontrol\web_ui\util.py", line 36, in get_changes changeset = repos.get_changeset(rev) File "d:\programme\python25\lib\site-packages\Trac-0.11dev-py2.5.egg\trac\versioncontrol\cache.py", line 53, in get_changeset self.db, self.authz) File "d:\programme\python25\lib\site-packages\Trac-0.11dev-py2.5.egg\trac\versioncontrol\cache.py", line 257, in __init__ row = cursor.fetchone()

comment:5 Changed 10 years ago by Christian Boos

Milestone: 1.0

comment:6 Changed 10 years ago by osimons

Milestone: 1.00.11

Is this not related to current 0.11dev memory issues in #6614? I'm setting it for 0.11 milestone to get it on the radar. cboos, feel free to change as you please.

comment:7 Changed 10 years ago by Christian Boos

Thanks. You're right, this seems to be related.

+ there's an interesting test case:

If I use a folder with 1600 files and no one is bigger than 16 KB the memory grows from 0.2 GB to 1.5 GB.

comment:8 in reply to:  7 Changed 10 years ago by anonymous

Replying to cboos:

Thanks. You're right, this seems to be related.

+ there's an interesting test case:

If I use a folder with 1600 files and no one is bigger than 16 KB the memory grows from 0.2 GB to 1.5 GB.

I (pierreroth64@…) am the reporter of #6614 and it is exactly the same test case I faced when reporting this ticket : my folder contains a large list of short source code files. Tickets #6614 and #5782 seems to be very close ! Hope that helps..

comment:9 Changed 10 years ago by Christian Boos

Keywords: svn pool memory added
Priority: normalhigh
Severity: normalmajor

comment:10 Changed 9 years ago by techtonik <techtonik@…>

The same problem with "out of memory" when browsing source. Sometimes it even fails to display Trac-styled error page and prints standard backtrace. Is it possible to measure and display memory usage after each successful query?

comment:11 Changed 9 years ago by Christian Boos

Status: newassigned

Committed a small bunch of optimization for directory browsing, in [6633:6635]. Feedback very much appreciated.

comment:12 Changed 9 years ago by Christian Boos

Resolution: fixed
Status: assignedclosed

I would have very much appreciated direct feedback from didley, to see how the memory usage he reported in comment:3 has now improved.

But based on feedback from Pierre (comment:8) and my own measures, this seems to have improved enough: fixed in [6633:6635] and by other memory related improvements.

Of course, listing big directories also share the general issues associated with rendering costly resources (see #6614 and #G190). Here, like for the reports, reducing the quantity of entries to display by introducing paging support is also something we can consider in future releases (see #6101 for that specific enhancement).

comment:13 in reply to:  12 Changed 9 years ago by didley@…

Replying to cboos:

I would have very much appreciated direct feedback from didley, to see how the memory usage he reported in comment:3 has now improved.

But based on feedback from Pierre (comment:8) and my own measures, this seems to have improved enough: fixed in [6633:6635] and by other memory related improvements.

Of course, listing big directories also share the general issues associated with rendering costly resources (see #6614 and #G190). Here, like for the reports, reducing the quantity of entries to display by introducing paging support is also something we can consider in future releases (see #6101 for that specific enhancement).

Hi,

I willgive you feedbackthis weekend.

didley

comment:14 Changed 9 years ago by didley@…

I tried to use the r6750 of Trac. Most of my Plugins won't work with it. So I can't give you feedback. I need more time.

comment:15 Changed 9 years ago by Christian Boos

I think we could still try to optimize this a bit more.

On my trunk/BIG test folder which contains 10k entries, r6634 reduced the memory usage from 2Gb down to 64Mb (I actually had to kill the process, without r6634). So now we're doing fine memory wise, but the timings are still awful: generating the list of entries takes 700s. The rendering takes an additional 73s.

comment:16 Changed 9 years ago by Christian Boos

With the [6787] change, going through the get_entries() for 10k entries now only takes about 8s, down from 700s! Rendering still takes about the same 70s, so that's now the bottleneck.

comment:17 Changed 9 years ago by didley@…

My test for the memory usage:
To List 684 Files needs 32 sec and memory grows from 460912 KB to 479724 KB
To select one file of it needs 20 sec and memory goes down to 467356 KB
Leave Browser Memory goes down to 466184 KB

number of Filelist timememoryselect one file timememoryleave browser memory
68432 s+ 19 MB20 s- 12 MB- 1.3 MB
94950 s+ 20 MB30 s- 12 MB- 5 MB
1614105 s+ 33 MB85 s- 31 MB+ 1 MB


Conclusion:
The memory usage is ok but the handling of the files is frustrating. We develop with 5000 files so it's not workable for us.
We remain with CVS. Thanx for your work.

didley

comment:18 Changed 9 years ago by Tim Hatch

didley,

Can I inquire about your methods for that last table? 20s seems like a long time for rendering a single file. The last time I saw that, it was actually the startup delay for mod_python.

Does that go down with repeated requests? Also is your repository local or mounted using something that is perhaps adding latency (nfs?) What is the time for rendering an empty directory?

comment:19 Changed 9 years ago by didley@…

I forgot to say the table above contains data when I use a browser (IE) on the server directly. That was a mistake. Now I did make the test again on a client with Firefox also. The times are much better than before but not usable.

trialsBrowsernumber of Filelist timeselect one file time
1IE161461 s20 s
2IE161462 s23 s
1Firefox161445 s40 s
2Firefox161437 s36 s

The select "one file time" will also be needed when I change to a other folder which contains no files.

  • Does that go down with repeated requests?

See table above. It seems not.

  • Also is your repository local …?

It is on the same Server like trac.

  • What is the time for rendering an empty directory?

0.5 s

didley

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christian Boos.
The resolution will be deleted.
to The owner will be changed from Christian Boos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .

 
Note: See TracTickets for help on using tickets.