TypeError: coercing to Unicode: need string or buffer, NoneType found

[Ryan J Ollos]

2015-12-24 02:53:30,390 Trac[main] ERROR: Internal Server Error: <RequestWithSession "POST '/'">, referrer 'http://bitten.edgewall.org'
Traceback (most recent call last):
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/main.py", line 554, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/main.py", line 237, in dispatch
    req.args.get('__FORM_TOKEN') != req.form_token:
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 338, in <lambda>
    'args': lambda req: arg_list_to_args(req.arg_list),
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 732, in _parse_arg_list
    name = unicode(value.name, 'utf-8')
TypeError: coercing to Unicode: need string or buffer, NoneType found

[Ryan J Ollos]

I'm unsure which form submit leads to the error. The only POST form at ​http://bitten.edgewall.org/ is the logout form.

According to the documentation None can occur as a field name. We may want to account for that: tags/trac-1.0.9/trac/web/api.py@:725,732#L705​.

Maybe the following patch would handle the issue:

trac/web/api.py

@@ -728 +728 @@
 
         args = []
         for value in fs.list or ():
+            name = None
             try:
-                name = unicode(value.name, 'utf-8')
+                if value.name is not None:
+                    name = unicode(value.name, 'utf-8')
                 if not value.filename:
                     value = unicode(value.value, 'utf-8')
             except UnicodeDecodeError, e:

Or would we want to use an empty string as the name when value.name is None?

[Jun Omae]

That issue can be reproduced with form-data with unnamed value. It's needed to retrieve the value from req.arg_list instead of req.args. However, I don't think that is valid request.

trac/web/api.py

@@ -729 +729 @@
         args = []
         for value in fs.list or ():
             try:
-                name = unicode(value.name, 'utf-8')
+                name = value.name
+                if name is not None:
+                    name = unicode(name, 'utf-8')
                 if not value.filename:
                     value = unicode(value.value, 'utf-8')
             except UnicodeDecodeError, e:

trac/web/tests/api.py

@@ -331 +331 @@
         req = Request(environ, None)
         self.assertRaises(HTTPBadRequest, lambda: req.arg_list)
 
+    def test_post_with_unnamed_value(self):
+        boundary = '_BOUNDARY_'
+        data = (
+            '--%(boundary)s\r\n'
+            'Content-Disposition: form-data; name="foo"\r\n'
+            '\r\n'
+            'named value\r\n'
+            '--%(boundary)s\r\n'
+            'Content-Disposition: form-data; name=""\r\n'
+            '\r\n'
+            'name is empty\r\n'
+            '--%(boundary)s\r\n'
+            'Content-Disposition: form-data\r\n'
+            '\r\n'
+            'unnamed value\r\n'
+            '--%(boundary)s--\r\n'
+        )
+        data %= {'boundary': boundary}
+        content_type = 'multipart/form-data; boundary="%s"' % boundary
+        environ = self._make_environ(method='POST',
+                                     **{'wsgi.input': StringIO(data),
+                                        'CONTENT_LENGTH': str(len(data)),
+                                        'CONTENT_TYPE': content_type})
+        req = Request(environ, None)
+        self.assertEqual('named value', req.args['foo'])
+        self.assertEqual([('foo', 'named value'), ('', 'name is empty'),
+                          (None, 'unnamed value')],
+                         req.arg_list)
+
 
 class RequestSendFileTestCase(unittest.TestCase):
 

[Ryan J Ollos]

Here's another issue which touches similar areas of the code:

2016-01-13 20:39:12,290 Trac[main] ERROR: Internal Server Error: <RequestWithSession "POST '/attachment/wiki/Documentation/configure.html/'">, referrer 'http://bitten.edgewall.org/attachment/wiki/Documentation/configure.html/?action=new'
Traceback (most recent call last):
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/main.py", line 554, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/main.py", line 203, in dispatch
    if handler.match_request(req):
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/attachment.py", line 481, in match_request
    req.args['realm'] = realm
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 338, in <lambda>
    'args': lambda req: arg_list_to_args(req.arg_list),
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 725, in _parse_arg_list
    fs = _FieldStorage(fp, environ=self.environ, keep_blank_values=True)
  File "/usr/lib/python2.7/cgi.py", line 508, in __init__
    self.read_multi(environ, keep_blank_values, strict_parsing)
  File "/usr/local/virtualenv/1.0-stable/lib/python2.7/site-packages/trac/web/api.py", line 208, in read_multi
    cgi.FieldStorage.read_multi(self, *args, **kwargs)
  File "/usr/lib/python2.7/cgi.py", line 637, in read_multi
    environ, keep_blank_values, strict_parsing)
  File "/usr/lib/python2.7/cgi.py", line 510, in __init__
    self.read_single()
  File "/usr/lib/python2.7/cgi.py", line 647, in read_single
    self.read_lines()
  File "/usr/lib/python2.7/cgi.py", line 669, in read_lines
    self.read_lines_to_outerboundary()
  File "/usr/lib/python2.7/cgi.py", line 697, in read_lines_to_outerboundary
    line = self.fp.readline(1<<16)
IOError: request data read error

[Ryan J Ollos]

Replying to Jun Omae:

That issue can be reproduced with form-data with unnamed value. It's needed to retrieve the value from req.arg_list instead of req.args. However, I don't think that is valid request.

Is that a patch that you think we should consider committing, or is it just a proof of concept to demonstrate?

[Ryan J Ollos]

I'll push this change after #12403 is committed.

[Ryan J Ollos]

Committed to 1.0-stable [14622], merged to trunk in [14623].