Edgewall Software
Modify

Opened 18 months ago

Closed 4 months ago

Last modified 4 months ago

#13038 closed defect (fixed)

Frequent MemoryErrors in log file

Reported by: Ryan J Ollos Owned by: Ryan J Ollos
Priority: normal Milestone: 1.3.6
Component: version control/log view Version: 1.3dev
Severity: normal Keywords: jinja2
Cc: Branch:
Release Notes:
  • Improved error reporting.
  • Error is not sent when the response has already started.
  • [trac] use_chunked_encoding is enabled by default and the error page is never sent as iterable content.
API Changes:

Description

How to Reproduce

While doing a GET operation on /log/rjollos.git/contrib, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{'path': u'/rjollos.git/contrib',
 u'rev': u'0ea0b82bc66cfcc0a636ebaa614cf17ae7ae9c2a'}

User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36

System Information

Trac 1.3.3.dev0
Babel 2.3.4
dnspython 1.15.0
Docutils 0.13.1
Genshi 0.7 (with speedups)
GIT 2.11.0
Jinja2 2.9.5
Mercurial 4.0
mod_wsgi 4.5.11 (WSGIProcessGroup trac WSGIApplicationGroup %{GLOBAL})
Pillow 4.0.0
PostgreSQL server: 9.6.7, client: 9.6.7
psycopg2 2.6.2
Pygments 2.2.0
Python 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
pytz 2016.7
setuptools 39.0.1
SpamBayes 1.1b1
Subversion 1.9.5 (r1770682)
jQuery 1.12.4
jQuery UI 1.12.1
jQuery Timepicker 1.6.3

Enabled Plugins

Interface Customization

shared-htdocs
shared-templates
site-htdocs
site-templates site.html, site_footer.html, site_head.html, site_header.html, site_leftbox.html

Python Traceback

Traceback (most recent call last):
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/main.py", line 664, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/main.py", line 270, in dispatch
    method=method)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1428, in render_template
    fragment, iterable, method)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1520, in _render_jinja_template
    iterable)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1633, in generate_template_stream
    bytes = template.render(data).encode('utf-8')
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/versioncontrol/templates/revisionlog.html", line 12, in top-level template code
    # extends 'layout.html'
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/theme.html", line 22, in top-level template code
    # block body
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/theme.html", line 128, in block "body"
    # block content
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/versioncontrol/templates/revisionlog.html", line 255, in block "content"
    branch}</span>
MemoryError

There are many instances of this traceback in the logs. There is one other common traceback that also results in MemoryError:

[pid 26408 140365895468800] 2018-06-05 05:31:44,191 Trac[main] ERROR: [144.76.64.79] Internal Server Error: <RequestWithSession "GET '/log/rjollos.git/trac/locale/messages.pot?rev=08d6651b83373d11a70c9d328882900109b99eeb'">, referrer None
Traceback (most recent call last):
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/main.py", line 664, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/main.py", line 270, in dispatch
    method=method)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1428, in render_template
    fragment, iterable, method)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1520, in _render_jinja_template
    iterable)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1633, in generate_template_stream
    bytes = template.render(data).encode('utf-8')
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 774, in handle_exception
    traceback = _make_traceback(exc_info, source_hint)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/debug.py", line 140, in make_traceback
    return translate_exception(exc_info, initial_skip)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/debug.py", line 193, in translate_exception
    reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 1005, in render
    return concat(self.root_render_func(self.new_context(vars)))

Attachments (0)

Change History (23)

comment:1 by Ryan J Ollos, 18 months ago

Issue can be reproduced by navigating here. I'm guessing it is due to all of the branch annotations, because issue is not seen if number of log entries is limited to 10.

Last edited 18 months ago by Ryan J Ollos (previous) (diff)

comment:2 by Jun Omae, 18 months ago

Work around enabling [trac] use_chunked_encoding reduces memory usage:

[trac] use_chunked_encoding = disabled

Line #    Mem usage    Increment   Line Contents
================================================
   653     49.3 MiB     49.3 MiB   @profile(stream=sys.stderr)
   654                             def _dispatch_request(req, env, env_error):
   655     49.3 MiB      0.0 MiB       resp = []
   656
   657                                 # fixup env.abs_href if `[trac] base_url` was not specified
   658     49.3 MiB      0.0 MiB       if env and not env.abs_href.base:
   659     49.3 MiB      0.0 MiB           env.abs_href = req.abs_href
   660
   661     49.3 MiB      0.0 MiB       try:
   662     49.3 MiB      0.0 MiB           if not env and env_error:
   663                                         raise HTTPInternalServerError(env_error)
   664     49.3 MiB      0.0 MiB           dispatcher = RequestDispatcher(env)
   665     49.3 MiB      0.0 MiB           try:
   666     49.3 MiB      0.0 MiB               dispatcher.set_default_callbacks(req)
   667    124.3 MiB     75.0 MiB               dispatcher.dispatch(req)
   668    124.3 MiB      0.0 MiB           except RequestDone as req_done:
   669    124.3 MiB      0.0 MiB               resp = req_done.iterable
   670    124.3 MiB      0.0 MiB           resp = resp or req._response or []
   671                                 except HTTPException as e:
   672                                     _send_user_error(req, env, e)
   673                                 except Exception:
   674                                     send_internal_error(env, req, sys.exc_info())
   675    124.3 MiB      0.0 MiB       return resp

[trac] use_chunked_encoding = enabled

Line #    Mem usage    Increment   Line Contents
================================================
   653     49.5 MiB     49.5 MiB   @profile(stream=sys.stderr)
   654                             def _dispatch_request(req, env, env_error):
   655     49.5 MiB      0.0 MiB       resp = []
   656
   657                                 # fixup env.abs_href if `[trac] base_url` was not specified
   658     49.5 MiB      0.0 MiB       if env and not env.abs_href.base:
   659     49.5 MiB      0.0 MiB           env.abs_href = req.abs_href
   660
   661     49.5 MiB      0.0 MiB       try:
   662     49.5 MiB      0.0 MiB           if not env and env_error:
   663                                         raise HTTPInternalServerError(env_error)
   664     49.5 MiB      0.0 MiB           dispatcher = RequestDispatcher(env)
   665     49.5 MiB      0.0 MiB           try:
   666     49.5 MiB      0.0 MiB               dispatcher.set_default_callbacks(req)
   667     78.4 MiB     28.9 MiB               dispatcher.dispatch(req)
   668     78.4 MiB      0.0 MiB           except RequestDone as req_done:
   669     78.4 MiB      0.0 MiB               resp = req_done.iterable
   670     78.4 MiB      0.0 MiB           resp = resp or req._response or []
   671                                 except HTTPException as e:
   672                                     _send_user_error(req, env, e)
   673                                 except Exception:
   674                                     send_internal_error(env, req, sys.exc_info())
   675     78.4 MiB      0.0 MiB       return resp

comment:3 by Jun Omae, 18 months ago

I consider that we should use Template.stream() or Template.generate() rather than Template.render() because rendering content can be large.

  • trac/web/chrome.py

    diff --git a/trac/web/chrome.py b/trac/web/chrome.py
    index 933f3d3d9..6330f8675 100644
    a b class Chrome(Component):  
    16251625        *text* parameter.
    16261626
    16271627        """
     1628        stream = template.stream(data)
     1629        stream.enable_buffering(75)  # buffer_size
    16281630        if iterable or iterable is None and self.use_chunked_encoding:
    1629             stream = template.stream(data)
    1630             stream.enable_buffering(75) # buffer_size
    16311631            return self._iterable_jinja_content(stream, text)
    16321632        else:
    1633             bytes = template.render(data).encode('utf-8')
    1634             if not text:
    1635                 bytes = valid_html_bytes(bytes)
    1636             return bytes
     1633            if text:
     1634                def generate():
     1635                    for chunk in stream:
     1636                        yield chunk.encode('utf-8')
     1637            else:
     1638                def generate():
     1639                    for chunk in stream:
     1640                        yield valid_html_bytes(chunk.encode('utf-8'))
     1641            return b''.join(generate())
    16371642
    16381643    def render_template_string(self, template, data, text=False):
    16391644        """Renders the template as an unicode or Markup string.

After the patch:

Line #    Mem usage    Increment   Line Contents
================================================
   653     49.4 MiB     49.4 MiB   @profile(stream=sys.stderr)
   654                             def _dispatch_request(req, env, env_error):
   655     49.4 MiB      0.0 MiB       resp = []
   656
   657                                 # fixup env.abs_href if `[trac] base_url` was not specified
   658     49.4 MiB      0.0 MiB       if env and not env.abs_href.base:
   659     49.4 MiB      0.0 MiB           env.abs_href = req.abs_href
   660
   661     49.4 MiB      0.0 MiB       try:
   662     49.4 MiB      0.0 MiB           if not env and env_error:
   663                                         raise HTTPInternalServerError(env_error)
   664     49.4 MiB      0.0 MiB           dispatcher = RequestDispatcher(env)
   665     49.4 MiB      0.0 MiB           try:
   666     49.4 MiB      0.0 MiB               dispatcher.set_default_callbacks(req)
   667     86.8 MiB     37.4 MiB               dispatcher.dispatch(req)
   668     86.8 MiB      0.0 MiB           except RequestDone as req_done:
   669     86.8 MiB      0.0 MiB               resp = req_done.iterable
   670     86.8 MiB      0.0 MiB           resp = resp or req._response or []
   671                                 except HTTPException as e:
   672                                     _send_user_error(req, env, e)
   673                                 except Exception:
   674                                     send_internal_error(env, req, sys.exc_info())
   675     86.8 MiB      0.0 MiB       return resp

in reply to:  2 comment:4 by Ryan J Ollos, 18 months ago

Replying to Jun Omae:

Work around enabling [trac] use_chunked_encoding reduces memory usage:

Thanks, I made the change and it appears to resolve the issue.

comment:5 by Ryan J Ollos, 18 months ago

Variations of this traceback have been appearing in the logs:

[pid 29239 140326419699456] 2018-06-14 04:42:44,020 Trac[chrome] ERROR: Jinja2 HTTPBadRequest error while rendering XML/HTML template 
Traceback (most recent call last):
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/chrome.py", line 1686, in _iterable_jinja_content
    for chunk in stream:
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 1271, in __next__
    return self._next()
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 1248, in _buffered_generator
    c = next(self._gen)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 1045, in generate
    yield self.environment.handle_exception(exc_info, True)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/error.html", line 94, in top-level template code
    <input type="submit" name="create" value="${_('Create')}" />
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/theme.html", line 22, in top-level template code
    # block body
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/templates/theme.html", line 165, in block "body"
    # include 'site_footer.html' ignore missing
  File "/var/trac/trac/templates/site_footer.html", line 37, in top-level template code
    # if req.environ.PATH_INFO == '/newticket' and (not 'preview' in req.args):
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/jinja2/environment.py", line 430, in getattr
    return getattr(obj, attribute)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/api.py", line 598, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/api.py", line 581, in <lambda>
    'args': lambda req: arg_list_to_args(req.arg_list),
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/api.py", line 598, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/api.py", line 1015, in _parse_arg_list
    msg=exception_to_unicode(e)))
Last edited 17 months ago by Ryan J Ollos (previous) (diff)

comment:6 by Ryan J Ollos, 18 months ago

Here is another URL that leads to MemoryError, even with use_chunked_encoding = enabled. We are running TracMercurial 1.0.0.8 on t.e.o.

[pid 9665 140326419486464] 2018-06-27 06:48:49,936 Trac[main] ERROR: [185.219.132.223] Internal Server Error: <RequestWithSession "GET '/log/mercurial-plugin/?format=changelog&rev=3e5b5b476b105f4d1dd577e4eebbbb92d617e0b4&limit=100&mode=stop_on_copy'">, referrer 'http://trac.edgewall.org/log/mercurial-plugin/?rev=0.12'
Traceback (most recent call last):
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/main.py", line 664, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/web/main.py", line 249, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/trac/versioncontrol/web_ui/log.py", line 271, in process_request
    for cpath, kind, chg, bpath, brev in changeset.get_changes():
  File "/usr/local/virtualenv/1.3dev/lib/python2.7/site-packages/tracext/hg/backend.py", line 1342, in get_changes
    edits = [p for p in parents if str_file in p.manifest()]
  File "/usr/lib/python2.7/dist-packages/mercurial/context.py", line 180, in manifest
    return self._manifest
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 770, in __get__
    result = self.func(obj)
  File "/usr/lib/python2.7/dist-packages/mercurial/context.py", line 531, in _manifest
    return self._repo.manifestlog[self._changeset.manifest].read()
  File "/usr/lib/python2.7/dist-packages/mercurial/manifest.py", line 1316, in read
    self._data = manifestdict(text)
  File "/usr/lib/python2.7/dist-packages/mercurial/manifest.py", line 414, in __init__
    self._lm = _lazymanifest(data)
MemoryError: 

comment:7 by Ryan J Ollos, 17 months ago

I took a closer look at the traceback in comment:5. The issue can be reproduced with the footer that utilizes req.args and a request to /newticket?preview=1&name=%FF.

Example site_footer.html:

<div>
    # if 'preview' not in req.args:
    This is the footer, only present if not newticket preview.
    # endif
</div>

What if we just re-raise HTTPError?:

diff --git a/trac/web/chrome.py b/trac/web/chrome.py
index 6330f8675..ac7f0d8e1 100644
--- a/trac/web/chrome.py
+++ b/trac/web/chrome.py
@@ -66,7 +66,8 @@ from trac.util.datefmt import (
     get_first_week_day_jquery_ui, get_timepicker_separator_jquery_ui,
     get_period_names_jquery_ui, localtz)
 from trac.util.translation import _, get_available_locales
-from trac.web.api import IRequestHandler, ITemplateStreamFilter, HTTPNotFound
+from trac.web.api import IRequestHandler, ITemplateStreamFilter, \
+                         HTTPException, HTTPNotFound
 from trac.web.href import Href
 from trac.wiki import IWikiSyntaxProvider
 from trac.wiki.formatter import format_to, format_to_html, format_to_oneliner
@@ -1690,6 +1691,8 @@ class Chrome(Component):
             else:
                 for chunk in stream:
                     yield valid_html_bytes(chunk.encode('utf-8'))
+        except HTTPException:
+            raise
         except Exception as e:
             self.log.error('Jinja2 %s error while rendering %s template %s',
                            e.__class__.__name__,

For this case, the log message would be the same as when [trac] use_chunked_encoding = disabled

04:41:53 Trac[main] WARNING: [127.0.0.1] HTTPBadRequest: 400 Bad Request (Invalid encoding in form data: UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte), <RequestWithSession "GET '/newticket?preview=1&name=%FF'">, referrer None

The difference is, when using chunked encoding the browser is blank, when not using chunked encoding the traceback is shown in the browser.

in reply to:  7 ; comment:8 by Jun Omae, 17 months ago

What if we just re-raise HTTPError?:

+        except HTTPException:
+            raise

For this case, the log message would be the same as when [trac] use_chunked_encoding = disabled

04:41:53 Trac[main] WARNING: [127.0.0.1] HTTPBadRequest: 400 Bad Request (Invalid encoding in form data: UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte), <RequestWithSession "GET '/newticket?preview=1&name=%FF'">, referrer None

Sounds good. In addition, it might be good to catch TracBaseError.

+        except (HTTPException, TracBaseError):
+            raise

Also the same issue probably exists in 1.0-stable and 1.2-stable.

The difference is, when using chunked encoding the browser is blank, when not using chunked encoding the traceback is shown in the browser.

Another considering, Trac sends http headers and the error as response body even if response is started when use_chunked_encoding is enabled and an error is raised. Then, I consider we should send the error unless response is started.

  • trac/web/api.py

    diff --git a/trac/web/api.py b/trac/web/api.py
    index 87f5813fa..428335357 100644
    a b class Request(object):  
    661661        if user is not None:
    662662            return to_unicode(user)
    663663
     664    @property
     665    def response_started(self):
     666        return self._write is not None
     667
    664668    @property
    665669    def scheme(self):
    666670        """The scheme of the request URL"""
  • trac/web/main.py

    diff --git a/trac/web/main.py b/trac/web/main.py
    index 897f12117..b20c19c33 100644
    a b def dispatch_request(environ, start_response):  
    644644            resp = req_done.iterable
    645645        resp = resp or req._response or []
    646646    except HTTPException as e:
    647         _send_user_error(req, env, e)
     647        resp = ()
     648        if not req.response_started:
     649            _send_user_error(req, env, e)
    648650    except Exception:
    649         send_internal_error(env, req, sys.exc_info())
     651        resp = ()
     652        if not req.response_started:
     653            send_internal_error(env, req, sys.exc_info())
    650654    finally:
    651655        translation.deactivate()
    652656        if env and not run_once:
Last edited 17 months ago by Ryan J Ollos (previous) (diff)

in reply to:  8 comment:9 by Ryan J Ollos, 17 months ago

Replying to Jun Omae:

Sounds good. In addition, it might be good to catch TracBaseError.

+        except (HTTPException, TracBaseError):
+            raise

Good idea. And since HTTPException inherits from TracBaseError (TracDev/Exceptions#CustomTracExceptions), it looks sufficient to just trap TracBaseError.

Also the same issue probably exists in 1.0-stable and 1.2-stable.

I'll test.

comment:10 by Ryan J Ollos, 17 months ago

Milestone: 1.3.31.3.4
Owner: set to Ryan J Ollos
Status: newassigned

Committed comment:3 change in r16704. Other changes will be committed in milestone:1.3.4.

comment:11 by Ryan J Ollos, 17 months ago

Would there be any downside to setting use_chunked_encoding enabled by default on the trunk?

  • trac/web/chrome.py

    diff --git a/trac/web/chrome.py b/trac/web/chrome.py
    index 6330f8675..c87e192e1 100644
    a b class Chrome(Component):  
    615615        displaying relative format and 'absolute' for displaying absolute
    616616        format. (''since 1.0'')""")
    617617
    618     use_chunked_encoding = BoolOption('trac', 'use_chunked_encoding', 'false',
     618    use_chunked_encoding = BoolOption('trac', 'use_chunked_encoding', 'true',
    619619        """If enabled, send contents as chunked encoding in HTTP/1.1.
    620620        Otherwise, send contents with `Content-Length` header after entire of
    621621        the contents are rendered. (''since 1.0.6'')""")

comment:12 by Jun Omae, 17 months ago

The downside of use_chunked_encoding option is that sending content is aborted by error. Therefore, reporting form on error and detail of the error would not be shown.

Anther idea to reduce the memory usage is to store content generated by Jinja2 to temporary file if size of the content is exceeded a threshold value (configurable), rather than using memory to store entire of the content.

in reply to:  12 comment:13 by Ryan J Ollos, 17 months ago

Replying to Jun Omae:

The downside of use_chunked_encoding option is that sending content is aborted by error. Therefore, reporting form on error and detail of the error would not be shown.

I don't know if it's a good idea, but it appears we could get around that by never sending error page as iterable content (same patch as in comment:description:ticket:13039):

  • trac/web/main.py

    diff --git a/trac/web/main.py b/trac/web/main.py
    index cd183f964..8de8f49f5 100644
    a b def _send_error(req, exc_info, template='error.html', content_type='text/html',  
    669669                status=500, env=None, data={}):
    670670    if env:
    671671        add_stylesheet(req, 'common/css/code.css')
    672         metadata = {'content_type': 'text/html'}
     672        metadata = {'content_type': 'text/html', 'iterable': False}
    673673        try:
    674674            content = Chrome(env).render_template(req, template,
    675675                                                  data, metadata)

comment:14 by Ryan J Ollos, 15 months ago

Milestone: 1.3.41.3.5

comment:15 by Jun Omae, 11 months ago

The macro which uses template to render the content eats huge memory even if use_chunked_encoding = enabled, e.g. TicketQuery macro.

comment:16 by Christian Boos, 11 months ago

Well, today we had that a lot.

I think there's a memory leak somewhere. When the wsgi process hits 612M it won't grow above that and will start to systematically generate MemoryErrors for about any new request coming in. And the CPU tops at 100% as a side-effect, as the response time being quite short, these processes get all the incoming requests and keep rejecting them very quickly and bots on the other end are happy to retry immediately afterwards.

Not good… maybe as an interim measure we could start to exit from the mod_wsgi server after we sent an error report due to a MemoryError?

comment:17 by Ryan J Ollos, 5 months ago

Milestone: 1.3.51.3.6

comment:18 by Ryan J Ollos, 4 months ago

Release Notes: modified (diff)

Proposed changes in log:rjollos.git:t13038_memory_error. Should [cb3908f52/rjollos.git] be applied to 1.0-stable and 1.2-stable?

comment:19 by Ryan J Ollos, 4 months ago

Proposed changes for 1.0 and 1.2: [3e50da148/rjollos.git].

in reply to:  19 comment:20 by Ryan J Ollos, 4 months ago

Replying to Ryan J Ollos:

Proposed changes for 1.0 and 1.2: [3e50da148/rjollos.git].

Changes committed to 1.0-stable in r17055, merged in r17056, r17057.

comment:21 by Ryan J Ollos, 4 months ago

Resolution: fixed
Status: assignedclosed
  • Reraise TracErrorBase in r17058.
  • Use chunked encoding by default in r17059.

in reply to:  16 comment:22 by Ryan J Ollos, 4 months ago

Replying to Christian Boos:

Not good… maybe as an interim measure we could start to exit from the mod_wsgi server after we sent an error report due to a MemoryError?

Sounds interesting. Do you have a proposed patch we can test? I'm not very familiar with that part of the codebase.

comment:23 by Ryan J Ollos, 4 months ago

Summary: MemoryError:Frequent MemoryErrors in log file

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Ryan J Ollos.
The resolution will be deleted. Next status will be 'reopened'.
to as closed The owner will be changed from Ryan J Ollos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.