UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 43: ordinal not in range(128)

[bhack]

How to Reproduce

While doing a GET operation on /report/6, Trac issued an internal error.

This was created when i have a non ascii character in the sqlite db. I.e. à é ì ò in a ticket description or summary

Request parameters:

{'id': u'6'}

User agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:11.0) Gecko/20100101 Firefox/11.0

System Information

Trac	0.12.3
Babel	0.9.6
Docutils	0.7
Genshi	0.6
GIT	1.7.9
mod_wsgi	3.3 (WSGIProcessGroup kforge WSGIApplicationGroup %{GLOBAL})
Pygments	1.3.1
pysqlite	2.4.1
Python	2.6.6 (r266:84292, Dec 26 2010, 22:48:11) [GCC 4.4.5]
pytz	2010b
RPC	1.1.2
setuptools	0.6
SQLite	3.7.3
Subversion	1.6.12 (r955767)
jQuery	1.4.4

Enabled Plugins

TracGit	0.12.0.5dev
TracXMLRPC	1.1.2

Python Traceback

Traceback (most recent call last):
  File "/var/lib/cvkforge/lib/python2.6/site-packages/Trac-0.12.3-py2.6.egg/trac/web/main.py", line 522, in _dispatch_request
    dispatcher.dispatch(req)
  File "/var/lib/cvkforge/lib/python2.6/site-packages/Trac-0.12.3-py2.6.egg/trac/web/main.py", line 243, in dispatch
    resp = chosen_handler.process_request(req)
  File "/var/lib/cvkforge/lib/python2.6/site-packages/Trac-0.12.3-py2.6.egg/trac/ticket/report.py", line 118, in process_request
    template, data, content_type = self._render_view(req, id)
  File "/var/lib/cvkforge/lib/python2.6/site-packages/Trac-0.12.3-py2.6.egg/trac/ticket/report.py", line 498, in _render_view
    value = cell_value(result[col_idx])
  File "/var/lib/cvkforge/lib/python2.6/site-packages/Trac-0.12.3-py2.6.egg/trac/ticket/report.py", line 49, in cell_value
    return v is 0 and '0' or v and unicode(v) or ''
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 43: ordinal not in range(128)

[Remy Blank]

Weird. At the location of the exception, v should already be a unicode string, so calling unicode() on it shouldn't have any effect. You seem to be reading byte strings from the database.

Your PySqlite version seems somewhat old. Would it be possible for you to upgrade to 2.5.6 and try again? OTOH, 2.4.1 is the version bundled with Python 2.6.6, so this may not help.

Or do you have the possibility to add a print statement showing repr(v) before the statement generating the exception, so that we can see the type of "v"?

[bhack]

I've tried to upgrade all the python environment with new version of component but i got the same error when i have in à é ò character in ticket summary or description.

i.e when value in sqlite db is "funzionalità" v become "funzionalit\xc3\xa0"

This is the new environment:

System Information

Trac	0.12.3
Babel	0.9.6 (translations unavailable)
Docutils	0.7
Genshi	0.6
GIT	1.7.9
mod_wsgi	3.3 (WSGIProcessGroup kforge WSGIApplicationGroup %{GLOBAL})
Pygments	1.5
pysqlite	2.6.3
Python	2.7.2+ (default, Nov 30 2011, 22:33:30) [GCC 4.6.2]
pytz	2010b
RPC	1.1.2-r11439
setuptools	0.6
SQLite	3.7.3
Subversion	1.6.17 (r1128011)
jQuery	1.4.4

Enabled Plugins

TracGit	0.12.0.5dev
TracXMLRPC	1.1.2-r11439

Python Traceback

Traceback (most recent call last):
  File "/var/lib/cvkforge/lib/python2.7/site-packages/Trac-0.12.3-py2.7.egg/trac/web/main.py", line 522, in _dispatch_request
    dispatcher.dispatch(req)
  File "/var/lib/cvkforge/lib/python2.7/site-packages/Trac-0.12.3-py2.7.egg/trac/web/main.py", line 243, in dispatch
    resp = chosen_handler.process_request(req)
  File "/var/lib/cvkforge/lib/python2.7/site-packages/Trac-0.12.3-py2.7.egg/trac/ticket/report.py", line 118, in process_request
    template, data, content_type = self._render_view(req, id)
  File "/var/lib/cvkforge/lib/python2.7/site-packages/Trac-0.12.3-py2.7.egg/trac/ticket/report.py", line 498, in _render_view
    value = cell_value(result[col_idx])
  File "/var/lib/cvkforge/lib/python2.7/site-packages/Trac-0.12.3-py2.7.egg/trac/ticket/report.py", line 49, in cell_value
    return v is 0 and '0' or v and unicode(v) or ''
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 43: ordinal not in range(128)

[Remy Blank]

I'm unable to reproduce the issue. If I create a new ticket with the subject "funzionalità", I'm able to view the report {6} without any issues. I assume that your report {6} is still the stock-provided report. If not, please post your report here.

Did you create that ticket using the Trac web interface? Or did you use an XML-RPC call or possibly a DB import? Can you try and see what's actually stored in the database, using something like:

>>> import sqlite3
>>> cnx = sqlite3.connect('env/db/trac.db')
>>> c = cnx.cursor()
>>> c.execute("select summary from ticket where id=58")
<sqlite3.Cursor object at 0xb77cd2f0>
>>> c.fetchall()
[(u'funzionalit\xe0',)]

Mmh, interesting, you mentioned above that v had the value 'funzionalit\xc3\xa0', when it should actually be u'funzionalit\xe0' (note the leading u, and the change in the string content). So it looks like you are getting bytestrings from SQLite instead of Unicode strings. Have you compiled SQLite or its Python bindings yourself, or installed from the official Ubuntu repositories? In the former case, you may have mis-configured something.

[bhack]

From this python code snipset i got u'funzionalit\xe0'

But on error on code fragment i got Local Variable: Name Value v funzionalit\xc3\xa0

I tried to update with sql and from interface but i got the same error. When i submit via interface i got: Genshi UnicodeDecodeError error while rendering template '/var/lib/cvkforge/lib/python2.7/site-packages/Trac-0.12.3-py2.7.egg/trac/templates/layout.html', line 12, char -1

Instead preview render fine!

And on timeline: Genshi UnicodeDecodeError error while rendering template (unknown template location)

I have official Sqlite debian version and official python bindings from packages. Trac was installed trought easy_install.

I also got error if i use unicode character inside a WikiPage.

I think that the problem is when trac find unicode character in the sqlite db (in ticket, wiki etc)

[Remy Blank]

So the data is stored correctly in the database, and can be retrieved correctly, but when Trac does it, it mangles the data.

I'm afraid I'm at my wits' end, and the only way forward I can see is to debug the data retrieval path from database to output generation, either by single-stepping or by adding lots of print statements. This looks very much like an InstallationIssue, as it's working fine for everyone else, and there doesn't seem to be anything fundamentally wrong in Trac. But maybe in certain cases, with certain encodings, we don't handle things properly. Who knows.

At this point, I would suggest asking on the MailingList if anyone else has seen such a symptom. Also, if you are familiar with Python, you could add print statements along the data retrieval path for report data, starting from the location where the data is retrieved from the database (everything should be in trac.ticket.report), and try to find out where the data is corrupted (hint: you should get Unicode strings out of the database, and they should remain Unicode strings up to the location where you see the error).

[bhack]

While debbugin i've tried to run with gdb and tracd standalone. Seems that works fine with the same db content. I run tracd from the same virtual environment of trac wsgi but with wsgi i still have this problem. What could be different between wsgi and tracd run?

[anonymous]

investigated more and I suppose that the problem is in kforge trac.wsgi deployment:

    from kforge.handlers.modwsgi import WsgiAccessControlHandler
    from trac.web.main import dispatch_request
    access_control = WsgiAccessControlHandler(dispatch_request)
    return access_control(environ, start_response)
    #return dispatch_request(environ, start_response)

If i use return dispatch_request(environ, start_response) instead of retun access_control all works fine but i miss kforge autentication for trac. What can alter this access_control method to create this kind of problem with unicode?

[Remy Blank]

Replying to anonymous:

If i use return dispatch_request(environ, start_response) instead of retun access_control all works fine but i miss kforge autentication for trac. What can alter this access_control method to create this kind of problem with unicode?

This pretty much points to kforge as the source of the issue, so you will have to ask the kforge maintainers for help on this. I won't be able to help you, unfortunately.

[Christian Boos]

Replying to anonymous:

What can alter this access_control method to create this kind of problem with unicode?

No idea either, but don't forget to drop us a note when you'll get the answer from the kforge maintainers!

What is KForge anyway?

…

Ok, so I was curious and found out the following:

• in ​/kforge/handlers/modwsgi.py, that WsgiAccessControlHandler adds TracTickets to the WSGI pipeline whose job seems to update "KForge tickets" from Trac tickets
• this is done here: ​src/kforge/plugin/trac/command.py, we can see that they use some high-level db code (self.tracProject.tickets.all.findDomainObjects), which corresponds to their ​domainmodel package, which in turn is based on SQLObject; the latter does this for its db connections:

  conn.text_factory = str # Convert text data to str, not unicode
  

  (​in sqlobject/sqlite/sqliteconnection.py)

So those "KForge tickets" are clearly containing UTF-8 str values, mirroring the unicode values contained in Trac tickets. What I miss in that picture is how those KForge tickets and UTF-8 values "go back" to the Trac layers while displaying a report…

But at least what is clear is that it's definitely not a Trac issue ;-)

[bhack]

Hello, only as info the bug seems resolved on kforge trunk version 0.20.

Tanks