Cannot render (or upgrade) some wiki pages.

[tjb@…]

When trying to access the TracReports wiki page, we see the following error

Python Traceback

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 300, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 176, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/wiki/web_ui.py", line 117, in process_request
    self._render_view(req, db, page)
  File "/usr/lib/python2.3/site-packages/trac/wiki/web_ui.py", line 364, in _render_view
    req.hdf['wiki.page_html'] = wiki_to_html(page.text, self.env, req)
  File "/usr/lib/python2.3/site-packages/trac/wiki/formatter.py", line 826, in wiki_to_html
    return Markup(out.getvalue())
  File "/usr/lib/python2.3/StringIO.py", line 203, in getvalue
    self.buf += ''.join(self.buflist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 63: ordinal not in range(128)

searchHighlight()

This is apparently another unicode problem, and is consistent across our trac installations. It also occurs on some other wiki pages.

At the same time, trac-admin wiki upgrade fails when trying to upgrade TracReports, and gets no further.

It reports an error when trying to decode a character - I'll look for the details and attache them when I can.

Our current trac version is 3130, and we are running on Suse Enterprise Linux 9, Apache 2.0.49, and mod-python.

[cboos]

I think I know why. From StringIO.py:

The StringIO object can accept either Unicode or 8-bit strings, but mixing the two may take some care. If both are used, 8-bit strings that cannot be interpreted as 7-bit ASCII (that use the 8th bit) will cause a UnicodeError? to be raised when getvalue() is called.

You certainly have a macro which still produces output as str objects.

Could you try the following patch?

Index: formatter.py
===================================================================
--- formatter.py	(revision 3118)
+++ formatter.py	(working copy)
@@ -26,7 +26,7 @@
 from trac.core import *
 from trac.mimeview import *
 from trac.wiki.api import WikiSystem, IWikiChangeListener, IWikiMacroProvider
-from trac.util import shorten_line
+from trac.util import shorten_line, to_unicode
 from trac.util.markup import escape, Markup, Element, html
 
 __all__ = ['wiki_to_html', 'wiki_to_oneliner', 'wiki_to_outline', 'Formatter' ]
@@ -91,7 +91,8 @@
             if self.name in list(macro_provider.get_macros()):
                 self.env.log.debug('Executing Wiki macro %s by provider %s'
                                    % (self.name, macro_provider))
-                return macro_provider.render_macro(req, self.name, text)
+                return to_unicode(macro_provider.render_macro(req, self.name,
+                                                              text))
 
     def _mimeview_processor(self, req, text):
         return Mimeview(self.env).render(req, self.name, text)

[pacopablo]

I ran into this error too. You patch fixed it cboos. However, your last change [3138] broke format_datetime. You forgot to specify the module name when calling getlocale. Here is the patch

Index: util/__init__.py
===================================================================
--- util/__init__.py    (revision 3138)
+++ util/__init__.py    (working copy)
@@ -208,7 +208,7 @@
             t = time.localtime(int(t))
 
     text = time.strftime(format, t)
-    lc_time_encoding = sys.platform != 'win32' and getlocale(locale.LC_TIME)[1]
+    lc_time_encoding = sys.platform != 'win32' and locale.getlocale(locale.LC_TIME)[1]
     encoding = lc_time_encoding or locale.getpreferredencoding()
     return unicode(text, encoding, 'replace')

[tjb@…]

I updated to [3139], and applied both of the above patches.
The result is still the same, with an identical traceback to the original.
Running trac-admin wiki upgrade still fails as well.

[cboos]

Please upgrade to [3141] and try again...

[tjb@…]

I upgraded to 3141, and had effectively no change. Some of the line numbers have changed, but the stack trace is the same.

I am leaving work in 30 minutes, and won't be back for 7 days - therefore cannot do any more testing in the short term. Sorry - I'll check the ticket from time to time at home.

[cboos]

Would be nice to post at least the stack trace, if you can.

[tjb@…]

Here it is

Python Traceback

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 301, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 177, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/wiki/web_ui.py", line 117, in process_request
    self._render_view(req, db, page)
  File "/usr/lib/python2.3/site-packages/trac/wiki/web_ui.py", line 364, in _render_view
    req.hdf['wiki.page_html'] = wiki_to_html(page.text, self.env, req)
  File "/usr/lib/python2.3/site-packages/trac/wiki/formatter.py", line 827, in wiki_to_html
    return Markup(out.getvalue())
  File "/usr/lib/python2.3/StringIO.py", line 203, in getvalue
    self.buf += ''.join(self.buflist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 63: ordinal not in range(128)

[tjb@…]

I updated to [3222] today, tried the wiki upgrade, and it works. The wiki pages display normally as well. I don't know what might have changed to fix this over the last week, my skim of the timeline showed nothing obvious.

The return to_unicode(...) patch is still in place.

I will leave it to you to close this if you are happy - thanks.

[cboos]

I'm quite sure it was [3141] which fixed it... maybe a wrong install step on your part at that time. Well, never mind, now it works :)

The patch adding to_unicode() in _macro_processor is not necessary anymore, as this is now done at a later stage (in process()). You should revert it.

I think we can close this one now.

[anonymous]

Hi guys,

I'm still seeing this in the latest 0.11dev subversion code.

Any ideas?

I also get UnicodeDecodeErrors? in my wiki nder some of the help pages. Perhaps my setup is wrong but I can't for the life of me think how.

See my comments under {3908} for more details including my version info.

Cheers, Matt

[anonymous]

Err, of course I meant to link to the ticket number: #3908

[anonymous]

Got this error when trying to access the TracReports wiki page

[cboos]

Replying to anonymous:

Hi guys, ... See my comments under {3908} for more details including my version info.

So no need to keep this one reopened as well. Restoring status for 0.10.