Trac SEARCH does not work at all

[testinggirl@…]

Trying to do a simple search on the word "completion" at http://www.aptana.com/trac/search

Trac is throwing all sorts of errors. See http://www.aptana.com/trac/search?q=completion&noquickjump=1&ticket=on&wiki=on

This is really frustrating. Hope you can fix this show-stopper bug very soon.

Here's a copy of all the errors on that page:

Oops…
Trac detected an internal error:

If you think this really should work and you can reproduce it, you should consider reporting this problem to the Trac team.

Go to http://trac.edgewall.org/ and create a new ticket where you describe the problem, how to reproduce it. Don't forget to include the Python traceback found below.

TracGuide — The Trac User and Administration Guide
Python Traceback

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 356, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 224, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/Search.py", line 181, in process_request
    results += list(source.get_search_results(req, terms, filters))
  File "/usr/lib/python2.3/site-packages/trac/ticket/api.py", line 265, in get_search_results
    for summary, desc, author, keywords, tid, date, status in cursor:
  File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 40, in __iter__
    row = self.cursor.fetchone()
  File "/usr/lib/python2.3/site-packages/trac/db/sqlite_backend.py", line 73, in fetchone
    return row and self._convert_row(row) or None
  File "/usr/lib/python2.3/site-packages/trac/db/sqlite_backend.py", line 69, in _convert_row
    return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa3 in position 5277: unexpected code byte

[mgood]

cleaning up description formatting

[mgood]

You appear to have non-UTF8 data in your tickets. This is one, though it seems like there may be more: http://www.aptana.com/trac/ticket/1341

You can use "sqlite" or "sqlite3" (depending on the version you have installed) to dump some information about that ticket:

$ sqlite3 db/trac.db
SQLite version 3.2.8
Enter ".help" for instructions
sqlite> select * from ticket where id = 1341;
sqlite> select * from ticket_change where ticket = 1341;

A couple questions:

• Did these problems occur before or after you upgraded to Trac 0.10?
• Did you import tickets from another system into Trac?
• Do you have any scripts updating/inserting ticket data into your Trac database?
• What plugins are installed?

I suspect that this data is coming from somewhere other than the core Trac ticket system, since it uses unicode throughout, so it's possibly another script or plugin that's inserting data that has not been correctly (utf8) encoded.

[mgood]

It appears that the character "0xa3" is the "£" character in the "latin1" encoding if that helps in determining where that data may be coming from.

[cboos]

Maybe we could be more tolerant in this situation, e.g.

trac/db/sqlite_backend.py

@@ -68 +68 @@
 
     class SQLiteUnicodeCursor(sqlite.Cursor):
         def _convert_row(self, row):
-            return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
+            return tuple([(isinstance(v, str) and [v.decode('utf-8', 'replace')] or [v])[0]
                           for v in row])
         def fetchone(self):
             row = sqlite.Cursor.fetchone(self)

Another possibility would be to switch to pysqlite2, as the above corresponds to pysqlite1 code.

[cboos]

Replying to cboos:

Maybe we could be more tolerant in this situation, e.g. ...

Elsewhere (#4037), jonas ruled against that approach. The patch above can be used by anyone interested in keeping their wrongly encoded content... use at your own risk, which is mainly the risk to not detect wrongly encoded data, and see a few spurious '?' characters here and there ;)