Sending POST requests to trac-0.10 server makes the server error with UnicodeDecodeError

[www_yllapito@…]

Traceback (most recent call last):
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 356, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 206, in dispatch
    populate_hdf(req.hdf, self.env, req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 120, in populate_hdf
    hdf['args.%s' % arg] = req.args[arg]
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 195, in __setitem__
    self.set_value(name, value, True)
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 251, in set_value
    add_value(name, value)
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 238, in add_value
    set_unicode(prefix, escape(value))
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 212, in set_unicode
    self.hdf.setValue(prefix.encode('utf-8'), value.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 22: ordinal not in range(128)

We are developing a plugin that takes input from a POST form. The relevant html code:

                        <tr class="Jälkiruoka">
                                <th scope="row">Jälkiruoka</th>
                                
                                        <td>
                                                <input type="text" name="ruoat_[Maanantai][Jälkiruoka]"/> 
                                        </td>
                                
                                        <td>
                                                <input type="text" name="ruoat_[Tiistai][Jälkiruoka]"/> 
                                        </td>
                                
                                        <td>

                                                <input type="text" name="ruoat_[Keskiviikko][Jälkiruoka]"/> 
                                        </td>
                                
                                        <td>
                                                <input type="text" name="ruoat_[Torstai][Jälkiruoka]"/> 
                                        </td>
                                
                                        <td>
                                                <input type="text" name="ruoat_[Perjantai][Jälkiruoka]"/> 
                                        </td>
                                
                        </tr>

we use the integrated tracd as the server atm

[Christian Boos]

self.hdf.setValue(prefix.encode('utf-8'), value.encode('utf-8'))

This is not the appropriate way to do it. While it might still work that way (testing if prefix is an unicode object, then only in this case try to encode it), it's much safer to use the HDFWrapper API, which has been strengthened to allow mixed str/unicode usage (see source:tags/trac-0.10/trac/web/clearsilver.py​):

req.hdf[prefix] = value

In general, you can find good advice (I hope ;) ) about usage of Unicode within Trac 0.10 in TracDev/UnicodeGuidelines.

[anonymous]

Replying to cboos:

self.hdf.setValue(prefix.encode('utf-8'), value.encode('utf-8'))

This is not the appropriate way to do it. While it might still work that way (testing if prefix is an unicode object, then only in this case try to encode it), it's much safer to use the HDFWrapper API, which has been strengthened to allow mixed str/unicode usage (see source:tags/trac-0.10/trac/web/clearsilver.py​):

Yeah, but that is not our code but upstream trac code from trac/web/clearsilver.py

[Christian Boos]

Replying to anonymous:

Replying to cboos:

self.hdf.setValue(prefix.encode('utf-8'), value.encode('utf-8'))

… Yeah, but that is not our code but upstream trac code from trac/web/clearsilver.py

Oops, you're right of course.

[Christian Boos]

Can you try this patch?

Index: trac/web/clearsilver.py
===================================================================
--- trac/web/clearsilver.py	(revision 3910)
+++ trac/web/clearsilver.py	(working copy)
@@ -249,7 +249,7 @@
                     set_unicode(prefix, value)
             elif isinstance(value, dict):
                 for k in value.keys():
-                    add_value('%s.%s' % (prefix, k), value[k])
+                    add_value('%s.%s' % (prefix, to_unicode(k)), value[k])
             else:
                 if hasattr(value, '__iter__') or \
                         isinstance(value, (list, tuple)):

But if you're targeting 0.10 and not 0.9, you should really use unicode strings internally, not UTF-8 encoded str strings.

[Christian Boos]

I'm going to assume the previous patch worked. If not, please reopen the ticket.

Patch applied in r3923.

[anonymous]

Sorry, I am not available to test during the weekends. No I still keep getting following:

Traceback (most recent call last):
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 356, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 206, in dispatch
    populate_hdf(req.hdf, self.env, req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 120, in populate_hdf
    hdf['args.%s' % arg] = req.args[arg]
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 195, in __setitem__
    self.set_value(name, value, True)
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 251, in set_value
    add_value(name, value)
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 238, in add_value
    set_unicode(prefix, escape(value))
  File "/usr/local/lib/python2.4/site-packages/trac/web/clearsilver.py", line 212, in set_unicode
    self.hdf.setValue(prefix.encode('utf8'), value.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

This is because the line you changed is not called in this case. This solves it for me but did not test if it works if the template is not utf-8. You can test this with the following template code:

<div id="content" class="helloworld">
 <form method="post" action="/milfi_dev/helloworld">
        <input type="text" name="Jäätää" />
</form>
<?cs each:item = args ?>
  <?cs name:item ?> - <h1> <?cs var:item ?></h1><br>
<?cs /each ?>
</div>

diff -ur trac-0.10.old/trac/web/clearsilver.py trac-0.10/trac/web/clearsilver.py
--- trac-0.10.old/trac/web/clearsilver.py       2006-10-16 09:19:31.000000000 +0300
+++ trac-0.10/trac/web/clearsilver.py   2006-10-16 09:39:35.000000000 +0300
@@ -208,7 +208,7 @@
         Add data to the HDF dataset.
         """
         def set_unicode(prefix, value):
-            self.hdf.setValue(prefix.encode('utf-8'), value.encode('utf-8'))
+            self.hdf.setValue(prefix, value.encode('utf-8'))
         def set_str(prefix, value):
             self.hdf.setValue(prefix.encode('utf-8'), str(value))

[www_yllapito@…]

Any news on my patch? Seems 0.10.1 and 0.10.2 were released but this bug is still open.

[Christian Boos]

Yes, because there are a few other places than the one you spotted that needs to be fixed, so it's not a straightforward fix.

[Christian Boos]

Well, given that the support for ClearSilver is being dropped, I see more risks than benefits in trying to fix this area of the code, in 0.10-stable…

If someone is hitting this very special problem, there are patches here to workaround the issue.

Proposing a wontfix but if someone else wants to take the responsibility of this part of the code, feel free to reopen and reassign.