Edgewall Software
Modify

Opened 9 years ago

Closed 8 years ago

Last modified 6 years ago

#2905 closed defect (fixed)

UnicodeDecodeError

Reported by: anonymous Owned by: cboos
Priority: normal Milestone: 0.10
Component: general Version: 0.10.2
Severity: normal Keywords: UnicodeDecodeError unicode utf8 pysqlite
Cc: m@…, johannes.rudolph@…
Release Notes:
API Changes:

Description

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 148, in dispatch
    populate_hdf(req.hdf, self.env, req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 68, in populate_hdf
    hdf['trac'] = {
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, util.escape(value))
  File "/usr/lib/python2.4/site-packages/trac/util/html.py", line 95, in escape
    text = unicode(text).replace('&', '&') \
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 4: ordinal not in range(128)

I use mod_python for trac-devel -r 3025. This is my apache setting.

SetHandler mod_python
PythonHandler trac.web.modpython_frontend
PythonOption TracEnv /my/TracRepo
PythonOption TracLocale "ko_KR.utf8"
SetEnv PYTHON_EGG_CACHE /my/EggCache

If i delete "PythonOption TracLocale "ko_KR.utf8"", The error message is not occured.

Attachments (4)

trac_2905.diff (1.4 KB) - added by cmlenz 9 years ago.
Patch
trac_2905_2.diff (792 bytes) - added by cmlenz 9 years ago.
Patch for the login name issue
explicit_unicode_diff.patch (3.9 KB) - added by cboos 9 years ago.
Patch on top of r3091 for doing the version control related diffs using unicode strings
another_2905_fix.diff (1.6 KB) - added by cboos 9 years ago.
Patch on r3107 — fixes the report error for me.

Download all attachments as: .zip

Change History (56)

comment:1 Changed 9 years ago by rbrown@…

This happens to me also, and rolling back to r3022 makes the problem go away.

We generally use the en_GB locale, however removing references to locale in the apache config did not cure the problem for us.

comment:2 Changed 9 years ago by cmlenz

  • Milestone changed from 0.9.5 to 0.10
  • Owner changed from jonas to cmlenz
  • Status changed from new to assigned

This is a problem in our date formatting

Changed 9 years ago by cmlenz

Patch

comment:3 Changed 9 years ago by cmlenz

Can you try the attached patch and let me know whether it fixes the issue?

comment:4 Changed 9 years ago by tdkim@…

Thanks for patch. I tested Trac -r 3032(head) and -r 3030.

Before appling the patch.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 148, in dispatch
    populate_hdf(req.hdf, self.env, req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 68, in populate_hdf
    hdf['trac'] = {
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, markup.escape(value))
  File "/usr/lib/python2.4/site-packages/trac/util/markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 4: ordinal not in range(128)

After appling the patch.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 162, in dispatch
    chrome.populate_hdf(req, chosen_handler)
  File "/usr/lib/python2.4/site-packages/trac/web/chrome.py", line 262, in populate_hdf
    for category, name, text in contributor.get_navigation_items(req):
  File "/usr/lib/python2.4/site-packages/trac/web/auth.py", line 70, in get_navigation_items
    yield ('metanav', 'login',
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 4: ordinal not in range(128)

Whether I switched off the "PythonOption TracLocale "ko_KR.utf8"" or not, The second error is occured.

I think that this is a new problem about UnicodeDecodeError.

comment:5 Changed 9 years ago by cmlenz

I guess this is related to your login name containing non-ASCII characters, right?

comment:6 Changed 9 years ago by cmlenz

Previously posted patch committed in [3035], which should fix the issue with date/time formatting.

Changed 9 years ago by cmlenz

Patch for the login name issue

comment:7 Changed 9 years ago by cmlenz

I've attached another patch that should hopefully help with non-ASCII login names. Can you please try that again? Thanks.

comment:8 Changed 9 years ago by tdkim@…

Thanks for your patch. But I use ASCII login names(AlphaNumeric?).

I tested it for trac -r 3040.

Before the patch, the same error is occured at auth.py. After the patch, the error message is only

'NoneType' object has no attribute 'decode'

Whether I switched off the "PythonOption TracLocale "ko_KR.utf8"" or not, the same error is occured.

comment:9 Changed 9 years ago by tdkim@…

Please ignore my previous posting. I need more testing for the problem.

After testing, I will post the test result. Thank you…

comment:10 Changed 9 years ago by tdkim@…

The test result using trac-3040 with the patch.

'NoneType' object has no attribute 'decode'

The test result using trac-3040 without the patch.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 170, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/wiki/web_ui.py", line 109, in process_request
    self._render_history(req, db, page)
  File "/usr/lib/python2.4/site-packages/trac/wiki/web_ui.py", line 343, in _render_history
    req.hdf['wiki.history'] = history
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 233, in add_value
    add_value('%s.%d' % (prefix, idx), item)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, markup.escape(value))
  File "/usr/lib/python2.4/site-packages/trac/util/markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 2: ordinal not in range(128)

comment:11 Changed 9 years ago by Markus Tacker <m@…>

  • Cc m@… added
  • Keywords unicode utf8 added

comment:12 Changed 9 years ago by anonymous

  • Cc peter.bruin@… added

It seems like this is happening in many modules. I also have this with changesets:

Traceback (most recent call last):
  File "c:\python23\Lib\site-packages\trac\web\main.py", line 290, in dispatch_request
    dispatcher.dispatch(req)
  File "c:\python23\Lib\site-packages\trac\web\main.py", line 170, in dispatch
    resp = chosen_handler.process_request(req)
  File "c:\python23\Lib\site-packages\trac\versioncontrol\web_ui\changeset.py", line 218, in process_request
    diff_args, diff_options)
  File "c:\python23\Lib\site-packages\trac\versioncontrol\web_ui\changeset.py", line 456, in _render_html
    diffs = _content_changes(old_node, new_node)
  File "c:\python23\Lib\site-packages\trac\versioncontrol\web_ui\changeset.py", line 428, in _content_changes
    ignore_space_changes='-b' in options)
  File "C:\Python23\lib\site-packages\trac\versioncontrol\diff.py", line 188, in hdf_diff
    line = space_re.sub(htmlify, escape(line, quotes=False))
  File "c:\python23\Lib\site-packages\trac\util\markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

comment:13 Changed 9 years ago by Markus Tacker <m@…>

  • Cc changed from m@tacker.org;peter.bruin@boschman.com.sg to m@tacker.org, peter.bruin@boschman.com.sg

Use comma+space in cc field

comment:14 Changed 9 years ago by PBruin

Thanks for the correction.

I rolled back to r3022 and the problem is gone. 0Xc3 is indeed a character that appears in one of the text files that was changed in the changeset that caused the Oops.

comment:15 Changed 9 years ago by cmlenz

  • Resolution set to fixed
  • Status changed from assigned to closed

This should be fixed in [3084].

comment:16 Changed 9 years ago by Markus Tacker <m@…>

  • Resolution fixed deleted
  • Status changed from closed to reopened

Sorry, the error still persists.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 175, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 86, in process_request
    resp = self._render_view(req, db, id)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 254, in _render_view
    req.hdf['%s.real' % prefix] = col[0]
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, markup.escape(value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1: ordinal not in range(128)

comment:17 Changed 9 years ago by Johannes Rudolph <johannes_rudolph@…>

  • Cc johannes.rudolph@… added

I'm getting this error:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 175, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 241, in process_request
    diff_args, diff_options)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 479, in _render_html
    diffs = _content_changes(old_node, new_node)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 451, in _content_changes
    ignore_space_changes='-b' in options)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/diff.py", line 198, in hdf_diff
    line = escape(line, quotes=False).replace('\0', '<del>') \
  File "/usr/lib/python2.3/site-packages/trac/util/markup.py", line 114, in escape
    text = text.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid data

comment:18 Changed 9 years ago by cboos

I can confirm that, for me at least, [3084] fixes the issue.

However, there's problably a better way to avoid the problem, as in this situation we try to explicitely convert both file contenst to utf8 (in _content_changes(old_node, new_node)).

As we already discussed, we should try to take into account the file encoding before comparing them. I'll see what I can do in this area.

comment:19 Changed 9 years ago by cboos

  • Priority changed from normal to high

Please try out the following patch, it works for me even without r3084:

Index: changeset.py
===================================================================
--- changeset.py	(revision 3090)
+++ changeset.py	(working copy)
@@ -420,16 +420,21 @@
             The list is empty when no differences between comparable files
             are detected, but the return value is None for non-comparable files.
             """
-            data = old_node.get_content().read()
-            if is_binary(data):
+            old_content = old_node.get_content().read()
+            if is_binary(old_content):
                 return None
-            old_content = mimeview.to_utf8(data, old_node.content_type)
 
-            data = new_node.get_content().read()
-            if is_binary(data):
+            new_content = new_node.get_content().read()
+            if is_binary(new_content):
                 return None
-            new_content = mimeview.to_utf8(data, new_node.content_type)
 
+            old_cset = mimeview.get_charset(old_content, old_node.content_type)
+            new_cset = mimeview.get_charset(new_content, new_node.content_type)
+            # character sets should be 'iso-8859-15' at the very least
+            assert old_cset and new_cset
+            old_content = unicode(old_content, old_cset)
+            new_content = unicode(new_content, new_cset)
+
             if old_content != new_content:
                 context = 3
                 options = diff_options[1]

I'll follow-up with a more complete patch, which will also take care of the textual diff view.

Changed 9 years ago by cboos

Patch on top of r3091 for doing the version control related diffs using unicode strings

comment:20 Changed 9 years ago by Johannes Rudolph <johannes_rudolph@…>

OK, the small patch did certainly help. But I have some strange non-deterministic behaviour, sometimes it works without patch, sometimes it doesn't. My working copy seems to be somewhat corrupted…

comment:21 Changed 9 years ago by cboos

… but with the patch, does it always work or do you still have errors?

comment:22 Changed 9 years ago by Johannes Rudolph <johannes_rudolph@…>

Yes, that was the only version which worked all times.

comment:23 Changed 9 years ago by athomas

I think this should also be applied to util.markup.Fragment:

  • trac/util/markup.py

     
    317324                yield escape(child, quotes=False) 
    318325 
    319326    def __str__(self): 
    320         return Markup(''.join(self.serialize())) 
     327        return Markup(''.join(self.serialize())).encode('utf-8') 
    321328 
    322329    def __add__(self, other): 
    323330        return Fragment()[self, other] 

comment:24 Changed 9 years ago by cboos

  • Owner changed from cmlenz to cboos
  • Status changed from reopened to new

Well, Alec, I don't think so.

We have to convert unicode strings to plain str with a specified encoding only at clearly defined times, like when sending text to the browser (convert to utf8) or sending a generated mail (convert to the configured encoding), because we can't switch back and forth between encoded strings and unicode as the charset used for the encoding is not remembered.

If you make an object's __str__ return an UTF-8 encoded string, next time you'll call unicode on that, you'll most likely get an exception:

>>> class txt(object):
...    def __str__(self):
...      return u'été'.encode('utf-8')
...
>>> str(txt())
'\xc3\xa9t\xc3\xa9'
>>> unicode(txt())
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

with 'ascii' replaced by whatever is your sys.getdefaultencoding().

As for the patch above, I'll complement it with some fallback in the style of trac.util.to_utf8, in case there's still an UnicodeError exception raised. This can happen if a wrong charset has been associated to the file, using the svn:mime-type property.

comment:25 Changed 9 years ago by cboos

The issues with TracChangeset and TracBrowser modules are fixed by r3092. It should be noted that with that fix, r3084 is not needed, so I wonder if we should revert it in or not, since assuming an 'utf-8' encoding could lead to or mask other errors…

I also saw that Markus reported errors with the report module, which are probably still there, so I'll leave this ticket opened.

comment:26 Changed 9 years ago by Markus Tacker <m@…>

Right. In r3092 the error still persists.

comment:27 Changed 9 years ago by PBruin

I confirm that the problems I had with TracChangeset are resolved with r3092. Thanks.

comment:28 Changed 9 years ago by athomas

Christian, that makes sense. However, I've been testing with the PEC data and #2868 causes an exception at source:sandbox/workflow/trac/web/clearsilver.py on line 235. markup.Markup is treated explicitly, but markup.Element is not, which I believe it should be. My previous patch fixed this.

I'm seeing this in WorkFlow though, so I could be not encoding something correctly, but my thought was that if it's already got explicit handling for Markup it probably should for Element?

Traceback (most recent call last):
  File "/home/athomas/projects/trac/workflow/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/home/athomas/projects/trac/workflow/trac/web/main.py", line 175, in dispatch
    resp = chosen_handler.process_request(req)
  File "/home/athomas/projects/trac/workflow/trac/ticket/web_ui.py", line 188, in process_request
    self._insert_ticket_data(req, db, ticket, reporter_id, fields)
  File "/home/athomas/projects/trac/workflow/trac/ticket/web_ui.py", line 469, in _insert_ticket_data
    req.hdf['ticket.%s' % field['name']] = field
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 235, in add_value
    self.hdf.setValue(prefix, str(value))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 232-237: ordinal not in range(128)

comment:29 Changed 9 years ago by cboos

As for a special treatment for Element (Fragment even), I'm all for it, but it should rather be done like this:

  • clearsilver.py

     
    213213                self.hdf.setValue(prefix, str(int(value))) 
    214214            elif isinstance(value, markup.Markup): 
    215215                self.hdf.setValue(prefix, value.encode('utf-8')) 
     216            elif isinstance(value, markup.Fragment): 
     217                self.hdf.setValue(prefix, unicode(value).encode('utf-8')) 
    216218            elif isinstance(value, str): 
    217219                if escape: 
    218220                    self.hdf.setValue(prefix, markup.escape(value)) 

comment:30 Changed 9 years ago by athomas

Beautiful! Works perfectly.

comment:31 Changed 9 years ago by tdkim@…

I have the problem. I tested it in Trac -r 3104.

Traceback (most recent call last):
 File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch\n    result = object(req)
 File "/usr/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 86, in handler\n    gateway.run(dispatch_request)
 File "/usr/lib/python2.4/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
 File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 311, in dispatch_request\n    req.hdf['error'] = {
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__\n    self.set_value(name, value, True)
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 238, in set_value\n    add_value(name, value)
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 230, in add_value\n    add_value('%s.%s' % (prefix, k), value[k])
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 220, in add_value\n    self.hdf.setValue(prefix, markup.escape(value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\\uc774' in position 0: ordinal not in range(128)

If I apply the patch, there is no problem.

  • trunk/trac/web/clearsilver.py

     
    217217                self.hdf.setValue(prefix, unicode(value).encode('utf-8')) 
    218218            elif isinstance(value, str): 
    219219                if escape: 
    220                     self.hdf.setValue(prefix, markup.escape(value)) 
     220                    self.hdf.setValue(prefix, markup.escape(value).encode('utf-8')) 
    221221                else: 
    222222                    self.hdf.setValue(prefix, value) 
    223223            elif isinstance(value, unicode): 

Please, confirm the patch.

comment:32 Changed 9 years ago by cboos

Thanks for the patch, applied in r3105.

comment:33 Changed 9 years ago by cboos

  • Resolution set to fixed
  • Status changed from new to closed

The latest changeset (r3107) hopefully fixes all the remaining issues (it does fix the report/CSV issue, at least).

I've also sent a mail to Trac-Dev in order to discuss r3084.

comment:34 Changed 9 years ago by Markus Tacker <m@…>

  • Resolution fixed deleted
  • Status changed from closed to reopened

I still have the error with a report.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 300, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 176, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 86, in process_request
    resp = self._render_view(req, db, id)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 335, in _render_view
    prefix = 'report.items.%d.%s' % (row_idx, unicode(column))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

This is the report entry from the db

11||Termine|SELECT

   id AS ticket,
   strftime('%d.%m.%Y', d.value) AS Fälligkeit,
   round((julianday(d.value) - julianday('now')))||' Tage' AS verbleibend,
   summary AS Beschreibung,
   summary AS _summary,
   description AS _description_,
   strftime('%s', d.value) AS _changetime,
   p.value AS __color__

  FROM ticket t, enum p

  LEFT OUTER JOIN ticket_custom d ON
       (t.id=d.ticket AND d.name='due')

  WHERE status IN ('new', 'assigned', 'reopened')
  AND t.type = 'Termin'
  AND p.name = t.priority AND p.type = 'priority'
  AND d.value != ''
  AND d.value > date('now')

  ORDER BY d.value ASC, p.value|Zeigt alle Termine nach ihrem [wiki:TicketFaelligkeit Datum] sortiert an.

Changed 9 years ago by cboos

Patch on r3107 — fixes the report error for me.

comment:35 Changed 9 years ago by cboos

Thanks for giving us report entry. While testing it, I could reproduce the error, but I had a slightly different error than you.

Can you try the following patch: attachment:another_2905_fix.diff

First, there's a fix for cursor.description which used to give back an utf-8 encoded string (fixed pysqlite2 only so far).

Then, the "Fälligkeit" value was used in the HDF name, a situation rather unusual…

comment:36 Changed 9 years ago by Markus Tacker <m@…>

Yeah, the patch attachment:another_2905_fix.diff works.

comment:37 Changed 9 years ago by cboos

  • Keywords pysqlite added

Ok, I applied a slightly different fix in r3109. Can you test it too, please?

comment:38 Changed 9 years ago by Markus Tacker <m@…>

r3109 works too. Thanks a lot.

comment:39 Changed 9 years ago by Markus Tacker <m@…>

Uh. Found another one using the TicketQuery Macro.

Error: Macro TicketQuery(type=Termin&status=new|assigned|reopened) failed
'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

comment:40 Changed 9 years ago by Markus Tacker <m@…>

And a third on in ticket view

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 300, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 176, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/web_ui.py", line 222, in process_request
    ticket = Ticket(self.env, id, db=db)
  File "/usr/lib/python2.4/site-packages/trac/ticket/model.py", line 35, in __init__
    self.fields = TicketSystem(self.env).get_ticket_fields()
  File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 99, in get_ticket_fields
    for field in self.get_custom_fields():
  File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 118, in get_custom_fields
    field = {
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 835, in wiki_to_oneliner
    OneLinerFormatter(env, absurls, db).format(wikitext, out, shorten)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 765, in format
    result = re.sub(self.wiki.rules, self.replace, result)
  File "/usr/lib/python2.4/sre.py", line 142, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 610, in replace
    return getattr(self, '_' + itype + '_formatter')(match, fullmatch)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 342, in _lhref_formatter
    return self._make_link(ns, target, match, label)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 349, in _make_link
    escape(label, False)))
  File "/usr/lib/python2.4/site-packages/trac/util/markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

comment:41 Changed 9 years ago by cboos

Markus, the error just above is related to #2972. There's a patch there which I believe will work.

For the error with the TicketQuery macro, I've not been able to reproduce it, but it's probably the same cause as the other one.

comment:42 Changed 9 years ago by Markus Tacker <m@…>

Right, the patch in #2972 fixed both errors. Thanks.

comment:43 Changed 9 years ago by anonymous

  • Cc peter.bruin@… removed

Removing myself from cc list.

comment:44 Changed 9 years ago by cboos

  • Resolution set to fixed
  • Status changed from reopened to closed

I think that as of r3141, there are no errors remaining (particularly thanks to r3137, which should fix the error with TicketQuery above)

comment:45 Changed 9 years ago by tdkim@…

  • Resolution fixed deleted
  • Status changed from closed to reopened

I used trac-r3213(dev) of trunk.

When I set the due date to the schedule of milestone, the problem is occured.

I use "2006년 03월 30일" as due date(YYYY년 MM월 DD일). This is a korean date/time format.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 196, in dispatch
    raise HTTPInternalError(e.message)
  File "/usr/lib/python2.4/site-packages/trac/web/api.py", line 46, in __call__
    Exception.__init__(self, '%s %s (%s)' % (self.code, self.reason,
UnicodeEncodeError: 'ascii' codec can't encode character u'\ub144' in position 4: ordinal not in range(128)

comment:46 Changed 9 years ago by cboos

Yes, I could reproduce this too.

Can you please upgrade to r3215 and try if that fix works for you?

comment:47 Changed 9 years ago by tdkim@…

Thank you.

I tested the date input of "timeline", "due" and "completed" in r3215. The UnicodeEncodeError was fixed.

But new error is occured.

Internal Error
2006년 04월 30일 is not a known date format.

I think that it is similar to #2182

comment:48 Changed 9 years ago by cboos

  • Resolution set to fixed
  • Status changed from reopened to closed

Yes, that's another issue then, cmlenz just scheduled it for 0.11, to be part of our i18n effort.

I'll close this one for now (once more :) ).

comment:49 Changed 8 years ago by freesoitj@…

  • Priority changed from high to normal
  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Version changed from devel to 0.10.2

I am sorry to reopen this ticket but I have an issue with UnicodeDecodeError too with 0.10.2 on Fedora Core 6 (x86-64). I am french and I have used non ascii names for component (at least I suppose I have since I can not see them anymore).

When I upgraded , quite a long time ago, new ticket and roadmap stopped working :

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 387, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 238, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/web_ui.py", line 129, in process_request
    ticket = Ticket(self.env, db=db)
  File "/usr/lib/python2.4/site-packages/trac/ticket/model.py", line 37, in __init__
    self.fields = TicketSystem(self.env).get_ticket_fields()
  File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 123, in get_ticket_fields
    options = [val.name for val in cls.select(self.env, db=db)]
  File "/usr/lib/python2.4/site-packages/trac/ticket/model.py", line 541, in select
    for name, owner, description in cursor:
  File "/usr/lib/python2.4/site-packages/trac/db/util.py", line 40, in __iter__
    row = self.cursor.fetchone()
  File "/usr/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line 73, in fetchone
    return row and self._convert_row(row) or None
  File "/usr/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line 69, in _convert_row
    return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
  File "/usr/lib64/python2.4/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: invalid data

When I go in WebAdmin, I can not edit components and when I use trac-admin, it is the same…

Trac [/var/trac/xxx]> component list
Command failed: 'utf8' codec can't decode bytes in position 1-2: invalid data
Trac [/var/trac/xxx]> 

Do you have any idea how I can recover?

comment:50 Changed 8 years ago by cboos

  • Milestone set to 0.10
  • Resolution set to fixed
  • Status changed from reopened to closed

(Sorry for the delay)

You have to do some manipulations at the SQLite db level. Delete and replace the faulty components, rename their occurrences in ticket and ticket_change tables.

We already discussed about being tolerant with wrongly encoded strings in the database, and jonas vetoed using replacement chars (the rationale being that those errors shouldn't be silently ignored and must be fixed at the db level).

(restoring the fixed status for the 0.10 milestone)

comment:51 Changed 6 years ago by anonymous

  • Component changed from general to plugin/spamfilter

comment:52 Changed 6 years ago by anonymous

  • Component changed from plugin/spamfilter to general

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed The owner will remain cboos.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from cboos to the specified user.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.