Version controls diffs of large text files kill the system

[anonymous]

We have some large text files (they are some data files we deal with) and when you try to view "last revision" or a revision with them it bombs (i.e. the service's cpu usage goes to 100%, and things come to a screeching halt).

Looks like the issue is in: versioncontrol/web_ui/changeset.py

The _content_changes method tries to do a diff of anything that isn't a binary. This is fine for most text files, unless they are big (like even 5 to 10 megs seems a bit much for the diffing method used). Anyway, I simply added the following to _content_changes:

if (len(old_content) > 100000 or len(new_content) > 100000):

return None

(Sorry if it isn't proper pep8)

It looks like you have a max_diff_bytes but that only effects the diff results. Ideally you would want a threshold before you have the diff results.

[cboos]

Incidentally I faced the very same problem this morning…

When I didn't get any response in the browser, I logged on the server and saw the httpd process stuck at 100% CPU. With gdb, I couldn't figure out the problem… because there was no error, just intensive processing in order to generate a diff for an XML file that grew from 70k lines to 100k lines, as I found out after waiting long enough ;-)

[anonymous]

Same problem here. Changeset view fails for an update to a huge sql dump. The suggested patch works very well for me. I suggest to include the fix in the next minor update.

[cboos]

We indeed need another threshold, like max_bytes_for_diff. And it probably should take into account the cumulative size, otherwise you'd still have a problem with 10 1MB files if your threshold is at 2MB …