UnicodeDecodeError in SpamFilter - akismet filter

[Christian Boos]

While testing adding image attachments using IExplorer, as anonymous:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 314, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 199, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/attachment.py", line 341, in process_request
    self._do_save(req, attachment)
  File "/usr/lib/python2.3/site-packages/trac/attachment.py", line 434, in _do_save
    for field, message in manipulator.validate_attachment(req, attachment):
  File "build/bdist.linux-i686/egg/tracspamfilter/adapters.py", line 102, in validate_attachment
  File "build/bdist.linux-i686/egg/tracspamfilter/api.py", line 70, in test
  File "build/bdist.linux-i686/egg/tracspamfilter/filters/akismet.py", line 59, in test
  File "build/bdist.linux-i686/egg/tracspamfilter/filters/akismet.py", line 90, in _check_comment
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

Btw. we need a SpamFilter component

[Christian Boos]

The comment was simply "test".

Same operation when logged in worked fine.

[Christian Boos]

Replying to cboos:

Same operation when logged in worked fine.

Hum, of course, no spam check for authenticated users, stupid :)

[anarcat@…]

I have a similar problem in the timeline here.

Ticket changes event provider (TicketModule) failed:

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 12-14: invalid data

[anarcat]

here is a more complete backtrace.

 'utf8' codec can't decode bytes in position 26-28: invalid data, referer: http://dev.alternc.org/trac/alternc/ticket/129
 , referer: http://dev.alternc.org/trac/alternc/ticket/129
 Traceback (most recent call last):, referer: http://dev.alternc.org/trac/alternc/ticket/129
   File "/usr/lib/cgi-bin/trac.cgi", line 20, in ?, referer: http://dev.alternc.org/trac/alternc/ticket/129
     cgi_frontend.run(), referer: http://dev.alternc.org/trac/alternc/ticket/129
   File "/usr/lib/python2.3/site-packages/trac/web/cgi_frontend.py", line 68, in run, referer: http://dev.alternc.org/trac/alternc/ticket/129
     gateway.run(dispatch_request), referer: http://dev.alternc.org/trac/alternc/ticket/129
   File "/usr/lib/python2.3/site-packages/trac/web/wsgi.py", line 87, in run, referer: http://dev.alternc.org/trac/alternc/ticket/129
     response = application(self.environ, self._start_response), referer: http://dev.alternc.org/trac/alternc/ticket/129
   File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 308, in dispatch_request, referer: http://dev.alternc.org/trac/alternc/ticket/129
     req = Request(environ, start_response), referer: http://dev.alternc.org/trac/alternc/ticket/129
   File "/usr/lib/python2.3/site-packages/trac/web/api.py", line 142, in __init__, referer: http://dev.alternc.org/trac/alternc/ticket/129
     self.args = self._parse_args(), referer: http://dev.alternc.org/trac/alternc/ticket/129
   File "/usr/lib/python2.3/site-packages/trac/web/api.py", line 166, in _parse_args, referer: http://dev.alternc.org/trac/alternc/ticket/129
     value = unicode(value.value, 'utf-8'), referer: http://dev.alternc.org/trac/alternc/ticket/129
 UnicodeDecodeError: 'utf8' codec can't decode bytes in position 26-28: invalid data, referer: http://dev.alternc.org/trac/alternc/ticket/129

[anarcat]

actually, i now have some doubts that this backtrace is really related to my issue, since i fixed that call in api.py to have a 'replace' argument and the problem still occurs. The occurence in the log also doesn't concur with a web invocation. So that's not it.

Also note that the akismet plugin is off.

Here's what i have in my debug log:

2006-07-06 09:17:47,669 Trac[loader] DEBUG: Loading egg plugin spamfilter.regex from /usr/lib/python2.3/site-packages/TracSpamFilter-0.1dev_r3465-py2.3.egg
2006-07-06 09:17:47,671 Trac[loader] DEBUG: Loading egg plugin spamfilter from /usr/lib/python2.3/site-packages/TracSpamFilter-0.1dev_r3465-py2.3.egg
2006-07-06 09:17:47,672 Trac[loader] DEBUG: Loading egg plugin spamfilter.ip_blacklist from /usr/lib/python2.3/site-packages/TracSpamFilter-0.1dev_r3465-py2.3.egg
2006-07-06 09:17:47,680 Trac[loader] WARNING: Cannot load plugin spamfilter.ip_blacklist because it requires "dnspython>=1.3.5"
2006-07-06 09:17:47,680 Trac[loader] DEBUG: Loading egg plugin spamfilter.adapters from /usr/lib/python2.3/site-packages/TracSpamFilter-0.1dev_r3465-py2.3.egg
2006-07-06 09:17:47,681 Trac[loader] DEBUG: Loading egg plugin spamfilter.akismet from /usr/lib/python2.3/site-packages/TracSpamFilter-0.1dev_r3465-py2.3.egg
2006-07-06 09:17:47,878 Trac[api] DEBUG: Updating wiki page index
2006-07-06 09:17:48,096 Trac[main] WARNING: 500 Internal Error (Ticket changes  event provider (<tt>TicketModule</tt>) failed:<br /><br />UnicodeDecodeError: 'utf8' codec can't decode bytes in position 12-14: invalid data<p>You may want to see the other kind of events from the <a href="/trac/alternc/timeline?wiki=on&amp;wiki=on&amp;milestone=on&amp;changeset=on">Timeline</

this is a bit annoying since there's no available backtrace…

[Christian Boos]

Actually, when you go to ​http://dev.alternc.org/trac/alternc/ticket/129, you get the following stack trace:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 314, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 199, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/ticket/web_ui.py", line 255, in process_request
    ticket = Ticket(self.env, id, db=db)
  File "/usr/lib/python2.3/site-packages/trac/ticket/model.py", line 39, in __init__
    self._fetch_ticket(tkt_id, db)
  File "/usr/lib/python2.3/site-packages/trac/ticket/model.py", line 86, in _fetch_ticket
    row = cursor.fetchone()
  File "/usr/lib/python2.3/site-packages/trac/db/sqlite_backend.py", line 64, in fetchone
    return row and self._convert_row(row) or None
  File "/usr/lib/python2.3/site-packages/trac/db/sqlite_backend.py", line 60, in _convert_row
    return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 26-28: invalid data

which suggests that you have some non-UTF8 string in your database.

Try to see manually what you have there, i.e.

$ sqlite3 db/trac.db 'select * from ticket where id=129'

Note also that this problem has nothing to do with this ticket ;)

[Christian Boos]

So the problem was not really related to the Akismet filter, rather to the Attachment filter adapter.

Fixed by [3558:3559].