Edgewall Software

Opened 7 years ago

Closed 6 years ago

Last modified 6 years ago

#12017 closed defect (fixed)

UnicodeDecodeError in log file

Reported by: Ryan J Ollos Owned by: Jun Omae
Priority: normal Milestone: 1.0.7
Component: general Version:
Severity: normal Keywords:
Cc: Branch:
Release Notes:

UnicodeDecodeError due to invalid encoded byte sequence in request query string is trapped and logged as an HTTPBadRequest, avoiding a traceback in the log file.

API Changes:
Internal Changes:

Description (last modified by Jun Omae)

Running the latest SpamFilter plugin [13942], I find the following errors in the log:

2015-04-01 23:15:33,851 Trac[main] ERROR: Internal Server Error:
Traceback (most recent call last):
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/main.py", line 513, in _dispatch_request
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/main.py", line 187, in dispatch
    self._pre_process_request(req, chosen_handler)
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/main.py", line 334, in _pre_process_request
    chosen_handler = filter_.pre_process_request(req, chosen_handler)
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/TracSpamFilter-1.0.6dev_r13942-py2.6.egg/tracspamfilter/captcha/api.py", line 167, in pre_process_request
    elif req.args.get('captcha_response', "") == exp:
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/api.py", line 338, in <lambda>
    'args': lambda req: arg_list_to_args(req.arg_list),
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.5-py2.6.egg/trac/web/api.py", line 704, in _parse_arg_list
    value = unicode(value.value, 'utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 37: invalid start byte

SpamFilter plugin should at least trap the UnicodeDecodeErrors so that tracebacks don't result in the log.

Attachments (0)

Change History (8)

comment:1 by Jun Omae, 7 years ago

Component: plugin/spamfiltergeneral
Description: modified (diff)

No. That is not a SpamFilter issue. We can raise the same error on any components, e.g. /wiki?name=%FF, /query?name=%FF.

  • trac/web/api.py

    diff --git a/trac/web/api.py b/trac/web/api.py
    index 3944020..0e62804 100644
    a b class Request(object):  
    702702            self.environ['QUERY_STRING'] = qs_on_post
    704704        args = []
    705         for value in fs.list or ():
    706             name = value.name
    707             if not value.filename:
    708                 value = unicode(value.value, 'utf-8')
    709             args.append((name, value))
     705        try:
     706            for value in fs.list or ():
     707                name = value.name
     708                if not value.filename:
     709                    value = unicode(value.value, 'utf-8')
     710                args.append((name, value))
     711        except UnicodeDecodeError, e:
     712            raise HTTPBadRequest(_("Invalid encoding in form data: %(msg)s",
     713                                   msg=exception_to_unicode(e)))
    710714        return args
    712716    def _parse_cookies(self):

After the patch, the following without traceback would be logged by invalid enoding.

2015-04-13 13:35:57,989 Trac[main] WARNING: [] HTTPBadRequest: 400 Bad Request (Invalid encoding in form data: UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: unexpected code byte)

I noticed that /wiki?%FF=xxxx doesn't lead the same error. I think we should decode the names to unicode with utf-8 encoding.

-            name = value.name
+            name = unicode(value.name, 'utf-8')
Last edited 7 years ago by Jun Omae (previous) (diff)

comment:2 by Ryan J Ollos, 7 years ago

Milestone: next-stable-1.0.x

comment:3 by Ryan J Ollos, 6 years ago

Milestone: next-stable-1.0.x1.0.7
Owner: Dirk Stöcker removed

comment:4 by Ryan J Ollos, 6 years ago

I added unit tests in log:rjollos.git:t12017. When merging to the trunk I'll replace except UnicodeDecodeError, e: with except UnicodeDecodeError as e:. Would you mind reviewing?

comment:5 by Jun Omae, 6 years ago

The proposed changes look good to me.

comment:6 by Ryan J Ollos, 6 years ago

Release Notes: modified (diff)
Resolution: fixed
Status: newclosed

Committed to 1.0-stable in [14095], merged to trunk in [14096].

comment:7 by Ryan J Ollos, 6 years ago

Owner: set to Jun Omae

comment:8 by Ryan J Ollos, 6 years ago

Release Notes: modified (diff)

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain Jun Omae.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Jun Omae to the specified user.

Add Comment

E-mail address and name can be saved in the Preferences .
Note: See TracTickets for help on using tickets.