browsing folder with too many files should use paging

[alex.theissen]

From time to time and everytime when I try to open a folder which contains many files at the "Browse source" component I get this memory error:

Traceback (most recent call last):
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/api.py", line 342, in send_error
    'text/html')
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/chrome.py", line 601, in render_template
    return stream.render(method, doctype=doctype)
  File "build/bdist.linux-i686/egg/genshi/core.py", line 154, in render
    return encode(generator, method=method, encoding=encoding)
  File "build/bdist.linux-i686/egg/genshi/output.py", line 45, in encode
    output = u''.join(list(iterator))
  File "build/bdist.linux-i686/egg/genshi/output.py", line 274, in __call__
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/output.py", line 519, in __call__
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/output.py", line 679, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "build/bdist.linux-i686/egg/genshi/output.py", line 459, in __call__
    for ev in stream:
  File "build/bdist.linux-i686/egg/genshi/core.py", line 212, in _ensure
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/core.py", line 212, in _ensure
    for event in stream:
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/chrome.py", line 631, in _strip_accesskeys
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/core.py", line 212, in _ensure
    for event in stream:
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/web/chrome.py", line 620, in _generate
    for kind, data, pos in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 252, in _include
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 322, in _match
    content = list(self._include(content, ctxt))
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 252, in _include
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 297, in _match
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 286, in _strip
    event = stream.next()
  File "build/bdist.linux-i686/egg/genshi/template/markup.py", line 242, in _exec
    for event in stream:
  File "build/bdist.linux-i686/egg/genshi/template/base.py", line 421, in _eval
    result = data.evaluate(ctxt)
  File "build/bdist.linux-i686/egg/genshi/template/eval.py", line 136, in evaluate
    return eval(self.code, _globals, {'data': data})
  File "/usr/local/python2.5/lib/python2.5/site-packages/Trac-0.11dev_r6038-py2.5.egg/trac/templates/error.html", line 161, in <Expression u'shorten_line(repr(value))'>
    <td><code>${shorten_line(repr(value))}</code></td>
MemoryError

This error only occurs when I have some data in the associated repository.

[Christian Boos]

Define "many" ;-)

I guess we're doing fine if it's not in the thousands. Otherwise, I think it's true that we'd need paging of the results.

[Christian Boos]

The present ticket could have been a duplicate of #5782, but as memory usage has now improved and browsing directories with thousand of entries is now practical, I've closed #5782 and will dedicate this one to paging support in the browser view.

[Christian Boos]

#7841 was closed as duplicate.

[xjakub@…]

As displaying a source directory with 6162 takes currently almost 6 minutes, the paging function would be really helpful.

[peter.westwood@…]

This is a major issue for us on the WordPress Plugins trac as it makes the browser unusable at a top level.

​http://plugins.trac.wordpress.org/

Would anyone be will to work with us to help us implement a solution to this as a plugin or patch for trac?

[Christian Boos]

Milestone 1.0 deleted

[Carsten Klein <carsten.klein@…>]

I think that the main problem here is the general use of the EagerCursor.

Perhaps for cases where there is lots of data to be expected from the database, a non eager cursor would be better.

[Christian Boos]

Patch welcomed.

It could do something like this:

• there should first be a call to Node.get_entries_count()
• use a page size of 500 entries, which should make the change unobtrusive in most situations
• if more entries available than page size, paginate (see the search or the report module for hints)