#5167 closed defect (worksforme)
MemoryError -- Changeset displaying 20+ large HTML files
| Reported by: | Owned by: | Christian Boos | |
|---|---|---|---|
| Priority: | normal | Milestone: | |
| Component: | version control/changeset view | Version: | 0.10.3 |
| Severity: | normal | Keywords: | memory needinfo |
| Cc: | ilias@… | Branch: | |
| Release Notes: | |||
| API Changes: | |||
| Internal Changes: | |||
Description
I have a public repository. Feel free to check it out and click around.
http://trac.genaud.net/index.fcgi/changeset/35
This changeset query (and many others) fail with the following message:
Trac detected an internal error:
....
Python Traceback
Traceback (most recent call last):
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/web/main.py", line 387, in dispatch_request
dispatcher.dispatch(req)
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/web/main.py", line 237, in dispatch
resp = chosen_handler.process_request(req)
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 254, in process_request
diff_args, diff_options)
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 502, in _render_html
diffs = _content_changes(old_node, new_node)
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 472, in _content_changes
ignore_space_changes='-b' in options)
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/versioncontrol/diff.py", line 198, in hdf_diff
line = escape(line, quotes=False).replace('\0', '<del>') \
File "/home/aegsvn/packages/lib/python2.3/site-packages/trac/util/html.py", line 112, in escape
text = text.replace('&', '&') \
MemoryError
Attachments (0)
Change History (10)
comment:1 by , 18 years ago
follow-up: 3 comment:2 by , 18 years ago
Hi cboos,
This message always occurs the first time:
(Not Found, "No handler matched request to /internal_error.html")
I've found if I reload the page I get the stack trace — a very helpful feature! There seems to be some check like "if this page has just been requested and it failed both times, print a message with a stack trace directing user to the Trac ticket page".
Anyway, I grok that the generated page would be enormous, and am not personally concerned, but there really *MUST* be a pre-check: if the difference is greater than some limit the page should be broken up… for example, just listing the changed files so that the user can select the individual file changeset. If there is a single file with enormous changes who knows maybe just: "sorry this change was simply too massive".
comment:3 by , 18 years ago
Replying to alex@genaud.net:
Anyway, I grok that the generated page would be enormous, and am not personally concerned, but there really *MUST* be a pre-check
This is already implemented, see TracIni#changeset-section as cboos already mentioned: when too many files are involved or the output diff is too large, Trac only shows up the list of changed files, with links to individual changesets.
comment:4 by , 18 years ago
I also wonder if the defaults values for the max_… settings shouldn't be adapted for 0.11. What is the memory available for Trac on this machine, and can you monitor the memory usage while rendering this changeset?
comment:5 by , 18 years ago
| Keywords: | memory needinfo added |
|---|---|
| Version: | → 0.10.3 |
Some more testing on my side:
- retrieving the diff text for that rev. 35 gives a 4Mbytes file.
- attaching and rendering that file in a Trac using tracd on windows gives an increase from the base 20 Mbytes to 86 Mbytes memory usage (86 Mbytes at peak)
- a second rendering of the same file makes the memory usage jump to 104 MBytes (141 peak) and after that it stabilizes at 110 Mbytes (141 peak) and the memory seems to be reused
Those numbers were obtained with 0.11. Re-doing the same test with 0.10-stable, I get a stabilization at 60 Mbytes (90 peak).
So while the memory usage seems to be quite costly, this should nevertheless still be ok on today's machines.
I'm still interested to get some information about your memory usage before closing the ticket.
comment:6 by , 18 years ago
Hi, I am using a shared host (dreamhost). I'd be happy to help you collect more information, if you wanna contact me directly (alex at genaud.net). Here's a cut-and-paste top at random intervals:
top - 01:02:06 up 12:37, 4 users, load average: 9.19, 4.29, 2.45 Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 8.2% user, 27.5% system, 9.8% nice, 54.6% idle Mem: 3615716k total, 3104388k used, 511328k free, 28048k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1377088k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15520 aegsvn 11 0 22812 22m 4224 D 9.2 0.6 0:00.78 trac.fcgi 14467 aegsvn 9 0 22812 22m 4224 S 0.3 0.6 0:00.01 trac.fcgi 12909 aegsvn 9 0 22812 22m 4224 S 0.0 0.6 0:00.61 trac.fcgi 11774 aegsvn 9 0 22812 22m 4224 S 0.0 0.6 0:00.33 trac.fcgi === top - 01:02:32 up 12:37, 4 users, load average: 6.81, 4.12, 2.45 Tasks: 8 total, 2 running, 6 sleeping, 0 stopped, 0 zombie Cpu(s): 7.5% user, 4.4% system, 17.2% nice, 71.0% idle Mem: 3615716k total, 3148968k used, 466748k free, 28212k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1386636k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15520 aegsvn 16 0 41424 40m 4224 R 99.9 1.1 0:22.81 trac.fcgi 11774 aegsvn 8 0 41424 40m 4224 S 0.0 1.1 0:00.33 trac.fcgi 17633 aegsvn 9 0 12208 11m 4072 S 0.0 0.3 0:00.18 trac.fcgi 13723 aegsvn 8 0 12208 11m 4072 S 0.0 0.3 0:00.00 trac.fcgi 6250 aegsvn 9 0 12608 12m 4060 S 0.0 0.3 0:00.19 trac.fcgi 22734 aegsvn 8 0 12608 12m 4060 S 0.0 0.3 0:00.00 trac.fcgi === top - 01:02:44 up 12:37, 4 users, load average: 5.74, 4.01, 2.44 Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 29.0% user, 3.4% system, 13.2% nice, 54.4% idle Mem: 3615716k total, 3112804k used, 502912k free, 28244k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1392272k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17633 aegsvn 9 0 13644 13m 4136 S 0.0 0.4 0:00.18 trac.fcgi 13723 aegsvn 8 0 13644 13m 4136 S 0.0 0.4 0:00.00 trac.fcgi 6250 aegsvn 9 0 13176 12m 4176 S 0.0 0.4 0:00.19 trac.fcgi 22734 aegsvn 8 0 13176 12m 4176 S 0.0 0.4 0:00.00 trac.fcgi === top - 01:03:08 up 12:38, 4 users, load average: 4.65, 3.88, 2.43 Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 7.7% user, 1.9% system, 9.0% nice, 81.4% idle Mem: 3615716k total, 3105652k used, 510064k free, 28316k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1395276k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17633 aegsvn 9 0 13712 13m 4144 S 0.0 0.4 0:00.18 trac.fcgi 13723 aegsvn 8 0 13712 13m 4144 S 0.0 0.4 0:00.00 trac.fcgi 6250 aegsvn 9 0 14052 13m 4176 S 0.0 0.4 0:00.20 trac.fcgi 22734 aegsvn 7 0 14052 13m 4176 S 0.0 0.4 0:00.00 trac.fcgi === <FINISHED with the HTTP request about here> === top - 01:03:17 up 12:38, 4 users, load average: 4.08, 3.79, 2.41 Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 3.3% user, 3.3% system, 10.7% nice, 82.8% idle Mem: 3615716k total, 3105948k used, 509768k free, 28356k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1395896k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17633 aegsvn 9 0 13988 13m 4188 S 0.0 0.4 0:00.18 trac.fcgi 13723 aegsvn 4 0 13988 13m 4188 S 0.0 0.4 0:00.00 trac.fcgi 6250 aegsvn 9 0 14068 13m 4176 S 0.0 0.4 0:00.20 trac.fcgi 22734 aegsvn 7 0 14068 13m 4176 S 0.0 0.4 0:00.00 trac.fcgi === top - 01:04:02 up 12:38, 4 users, load average: 2.45, 3.39, 2.34 Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 4.3% user, 6.3% system, 9.1% nice, 80.3% idle Mem: 3615716k total, 3115724k used, 499992k free, 28540k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1401612k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17633 aegsvn 9 0 14052 13m 4188 S 0.3 0.4 0:00.19 trac.fcgi 13723 aegsvn 8 0 14052 13m 4188 S 0.0 0.4 0:00.01 trac.fcgi 6250 aegsvn 9 0 14080 13m 4180 S 0.0 0.4 0:00.20 trac.fcgi 22734 aegsvn 9 0 14080 13m 4180 S 0.0 0.4 0:00.01 trac.fcgi === top - 01:11:27 up 12:46, 4 users, load average: 1.19, 1.92, 2.04 Tasks: 4 total, 1 running, 3 sleeping, 0 stopped, 0 zombie Cpu(s): 10.0% user, 7.8% system, 9.9% nice, 72.3% idle Mem: 3615716k total, 3190384k used, 425332k free, 30120k buffers Swap: 6313512k total, 20340k used, 6293172k free, 1460632k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17633 aegsvn 10 0 15412 15m 4192 S 0.7 0.4 0:00.25 trac.fcgi 13723 aegsvn 8 0 15412 15m 4192 S 0.0 0.4 0:00.02 trac.fcgi
comment:7 by , 18 years ago
Well, nothing terrible here in the memory usage. I wonder though if there isn't some kind of rlimit set by the hosting provider (see #4885). Can you check that?
comment:9 by , 17 years ago
| Resolution: | → worksforme |
|---|---|
| Status: | new → closed |
Just retested your web site (listed in the description) and it looked okay. Seems like you've resolved this. Closing.
comment:10 by , 17 years ago
| Cc: | added |
|---|
Well, I just went to this changeset, and it failed mid-processing as well, with a different error (Not Found, "No handler matched request to /internal_error.html"). I think this is a "simple" out-of-memory condition, which has nothing to do with the
text.replaceline. You can prevent this to happen by tweaking the default settings for viewing changesets (see TracIni#changeset-section).In the future we might produce the diffs in a streamed way, instead of building all the data in memory first. I'm not 100% sure if this is possible though, due to all the conversions from stream to list that may happen during the processing and rendering of templates. cmlenz could better comment on this than me.