Edgewall Software
Modify

Opened 7 years ago

Last modified 3 years ago

#6101 new defect

browsing folder with too many files should use paging

Reported by: alex.theissen Owned by: cboos
Priority: lowest Milestone: next-major-releases
Component: version control/browser Version: devel
Severity: minor Keywords: memory paging
Cc: xjakub@…, peter.westwood@…
Release Notes:
API Changes:

Description

From time to time and everytime when I try to open a folder which contains many files at the "Browse source" component I get this memory error:

Traceback (most recent call last):
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/api.py", line 342, in send_error
    'text/html')
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/chrome.py", line 601, in render_template
    return stream.render(method, doctype=doctype)
  File "build/bdist.linux-i686/egg/genshi/core.py", line 154, in render
    return encode(generator, method=method, encoding=encoding)
  File "build/bdist.linux-i686/egg/genshi/output.py", line 45, in encode
    output = u''.join(list(iterator))
  File "build/bdist.linux-i686/egg/genshi/output.py", line 274, in __call__
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/output.py", line 519, in __call__
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/output.py", line 679, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "build/bdist.linux-i686/egg/genshi/output.py", line 459, in __call__
    for ev in stream:
  File "build/bdist.linux-i686/egg/genshi/core.py", line 212, in _ensure
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/core.py", line 212, in _ensure
    for event in stream:
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/chrome.py", line 631, in _strip_accesskeys
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/core.py", line 212, in _ensure
    for event in stream:
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/chrome.py", line 620, in _generate
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 252, in _include
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 322, in _match
    content = list(self._include(content, ctxt))
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 252, in _include
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 297, in _match
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 286, in _strip
    event = stream.next()
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 242, in _exec
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/base.py", line 421, in _eval
    result = data.evaluate(ctxt)
  File "build/bdist.linux-i686/egg/genshi/template/eval.py", line 136, in evaluate
    return eval(self.code, _globals, {'data': data})
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/templates/error.html", line 161, in <Expression u'shorten_line(repr(value))'>
    <td><code>${shorten_line(repr(value))}</code></td>
MemoryError

This error only occurs when I have some data in the associated repository.

Attachments (0)

Change History (9)

comment:1 Changed 7 years ago by cboos

  • Component changed from general to browser
  • Keywords memory paging added
  • Milestone changed from 0.11 to 1.0
  • Owner changed from jonas to cboos
  • Summary changed from Memory Error to Memory Error when browsing folder with too many files

Define "many" ;-)

I guess we're doing fine if it's not in the thousands. Otherwise, I think it's true that we'd need paging of the results.

comment:2 Changed 6 years ago by cboos

  • Summary changed from Memory Error when browsing folder with too many files to browsing folder with too many files should use paging

The present ticket could have been a duplicate of #5782, but as memory usage has now improved and browsing directories with thousand of entries is now practical, I've closed #5782 and will dedicate this one to paging support in the browser view.

comment:3 Changed 5 years ago by cboos

#7841 was closed as duplicate.

comment:4 Changed 5 years ago by xjakub@…

  • Cc xjakub@… added

As displaying a source directory with 6162 takes currently almost 6 minutes, the paging function would be really helpful.

comment:5 Changed 4 years ago by peter.westwood@…

This is a major issue for us on the WordPress? Plugins trac as it makes the browser unusable at a top level.

http://plugins.trac.wordpress.org/

Would anyone be will to work with us to help us implement a solution to this as a plugin or patch for trac?

comment:6 Changed 4 years ago by cboos

  • Milestone changed from 1.0 to unscheduled

Milestone 1.0 deleted

comment:7 Changed 4 years ago by Carsten Klein <carsten.klein@…>

I think that the main problem here is the general use of the EagerCursor.

Perhaps for cases where there is lots of data to be expected from the database, a non eager cursor would be better.

comment:8 Changed 4 years ago by cboos

  • Milestone changed from triaging to next-major-0.1X
  • Priority changed from normal to lowest
  • Severity changed from normal to minor

Patch welcomed.

It could do something like this:

  • there should first be a call to Node.get_entries_count()
  • use a page size of 500 entries, which should make the change unobtrusive in most situations
  • if more entries available than page size, paginate (see the search or the report module for hints)

comment:9 Changed 3 years ago by peter.westwood@…

  • Cc peter.westwood@… added

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as new The owner will remain cboos.
as The resolution will be set. Next status will be 'closed'.
The owner will be changed from cboos to anonymous. Next status will be 'assigned'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.