proposal for handling utf-errors (e.g. after migration to trac 0.10)

[fg.reinisch@…]

After migration from 0.95 to 0.10 in our trac-evaluation system some problems occured:

some tickets (usually containing some german "Umlaute" like "ä,ö,ü" lead to error messages when rendering tickets or timeline (when tickets were chosen).

Proposal: what ever the reason might be for that (even if some bad guy hacks the contents in the database) the system could behave a little bit mor tolerant. Just replace the not readable/translatable character(s) with (an) e.g. asterisk(s) and possibly add a warning, but trac please should go on rendering tickets and timeline.

k.r. Franz

PS. Trac is very promising !

example:

Oops… Trac detected an internal error: If you think this really should work and you can reproduce it, you should consider reporting this problem to the Trac team. 
Go to http://trac.edgewall.org/ and create a new ticket where you describe the problem, how to reproduce it. Don't forget to include the Python traceback found below. 

TracGuide — The Trac User and Administration Guide 
Python Traceback Traceback (most recent call last): 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\web\main.py", line 356, in dispatch_request 

dispatcher.dispatch(req) 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\web\main.py", line 224, in dispatch 

resp = chosen_handler.process_request(req) 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\ticket\web_ui.py", line 302, in process_request 

get_reporter_id(req, 'author')) 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\ticket\web_ui.py", line 615, in _insert_ticket_data 

for change in self.grouped_changelog_entries(ticket, db): 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\ticket\web_ui.py", line 662, in grouped_changelog_entries 

changelog = ticket.get_changelog(when=when, db=db) 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\ticket\model.py", line 299, in get_changelog 

for t, author, field, oldvalue, newvalue, permanent in cursor: 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\db\util.py", line 40, in iter 

row = self.cursor.fetchone() 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\db\sqlite_backend.py", line 73, in fetchone 

return row and self._convert_row(row) or None 

File "D:\PROGRA~1\Python23\Lib\site-packages\trac\db\sqlite_backend.py", line 69, in _convert_row 

return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0] 

'''UnicodeDecodeError?: 'utf8' codec can't decode byte 0x81 in position 102: unexpected code byte'''

[Christian Boos]

Can you try this?

trac/db/sqlite_backend.py

@@ -68 +68 @@
 
     class SQLiteUnicodeCursor(sqlite.Cursor):
         def _convert_row(self, row):
-            return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
+            return tuple([(isinstance(v, str) and [v.decode('utf-8', 'replace')] or [v])[0]
                           for v in row])
         def fetchone(self):
             row = sqlite.Cursor.fetchone(self)

[Christian Boos]

Also, I'd be interested to know if you still have those errors when using a more recent PySqlite version (as you appear to be running pysqlite 1).

[Jonas Borgström]

I think this sounds like a bad idea. All text stored in sqlite databases should really be utf-8 encoded, and will be if Trac was used to store the data. So any non utf-8 encoded text must have been inserted by an other application. And that application should be fixed, not Trac. It should really not be our responsibility to clean up after misbehaving applications incorrectly modifying our database.

[Christian Boos]

#3945 was marked as duplicate.

As I said there, the patch in comment:1 can be used by those wanting to have Trac to be tolerant when it retrieves non utf-8 encoded data from the database, but this will not be Trac's official line, as explained in comment:3.