cant post the letter ö (among others)

[Bjorn Leví]

Traceback (most recent call last):
  File "C:\Python23\Lib\site-packages\trac\web\main.py", line 315, in dispatch_request
    dispatcher.dispatch(req)
  File "C:\Python23\Lib\site-packages\trac\web\main.py", line 200, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\tractags\web_ui.py", line 64, in process_request
  File "C:\Python23\lib\site-packages\trac\wiki\web_ui.py", line 133, in process_request
    self._render_view(req, db, page)
  File "C:\Python23\lib\site-packages\trac\wiki\web_ui.py", line 460, in _render_view
    req.hdf['wiki'] = {
  File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 994, in wiki_to_html
    Formatter(env, req, absurls, db).format(wikitext, out, escape_newlines)
  File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 820, in format
    result = re.sub(self.wiki.rules, self.replace, line)
  File "C:\Python23\lib\sre.py", line 143, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 768, in replace
    return to_unicode(replacement)
  File "C:\Python23\Lib\site-packages\trac\util\text.py", line 57, in to_unicode
    return unicode(text)
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 313, in __str__
    return Markup(''.join(self.serialize()))
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 452, in serialize
    for part in Fragment.serialize(self):
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 308, in serialize
    yield unicode(child)
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 313, in __str__
    return Markup(''.join(self.serialize()))
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 452, in serialize
    for part in Fragment.serialize(self):
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 310, in serialize
    yield escape(child, quotes=False)
  File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)

[Christian Boos]

It appears you're running trunk, not 0.9.6.

Also, it appears that the error was triggered by the ​TracHacks:TagsPlugin. From what I understand looking at the stack trace, that plugin is wrapping a str which contains utf-8 encoded data in a Fragment or Element object (here the str is: '\xc3\xb6', which corresponds to u'ö'.decode('utf-8')).

I always hesitated to use to_unicode(text) instead of unicode(text) in Markup.escape, because it helped to catch a lot of bad uses of non-unicode strings in the core, but I think that increased compatibility with 0.9.3+ plugins is a good reason to do so, now that the unicode issues in the core are gone.

Can you please try the following fix:

Index: trac/util/html.py
===================================================================
--- trac/util/html.py	(revision 3605)
+++ trac/util/html.py	(working copy)
@@ -21,6 +21,8 @@
 from StringIO import StringIO
 import sys
 
+from trac.util.text import to_unicode
+
 __all__ = ['escape', 'unescape', 'html']
 
 _EMPTY_TAGS = frozenset(['br', 'hr', 'img', 'input'])
@@ -104,7 +106,7 @@
         """
         if isinstance(text, (cls, Element)):
             return text
-        text = unicode(text)
+        text = to_unicode(text)
         if not text:
             return cls()
         text = text.replace('&', '&') \

(Note that the patch is on html.py, because markup.py has been moved lately; it should apply on markup.py however, or you could upgrade to the latest)

[Christian Boos]

Ok, the above fix was applied in r3609, one more reason to upgrade to the latest rev ;)