restructuredtext renderer confused by utf-8 BOM

[anonymous]

If there is a "utf-8 Byte Order Marker" at the front of an .rst file (the bytes 0xEFBBBF), then trac misrenders the first line of the file, by counting the length of that line incorrectly. (For example, if the length of a "======" line differs from the length of the subsequent title and following "======" line, then it won't be turned into a headline.) Trac should probably check for a byte order marker and remove it before rendering.

[Christian Boos]

Isn't that more a problem with docutils itself?

• which version of docutils do you use?
• what encoding is Trac using when reading that file? (if you use svn and have no charset specified in the svn:mime-type property, then check your [trac] default_charset setting; if using Mercurial, then check [hg] encoding)

[zooko@…]

I would have thought it was a problem with docutils, but docutils doesn't misparse the same file:

zooko@tahoe-lafs:~/playground/tahoe-lafs/1.8.3/docs$ hexdump -C known_issues.rst | head -4
00000000  ef bb bf 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 3d 3d 0a  |...============.|
00000010  4b 6e 6f 77 6e 20 49 73  73 75 65 73 0a 3d 3d 3d  |Known Issues.===|
00000020  3d 3d 3d 3d 3d 3d 3d 3d  3d 0a 0a 2a 20 60 4f 76  |=========..* `Ov|
00000030  65 72 76 69 65 77 60 5f  0a 2a 20 60 49 73 73 75  |erview`_.* `Issu|
zooko@tahoe-lafs:~/playground/tahoe-lafs/1.8.3/docs$ rst2html known_issues.rst > ~/public_html/known_issues.html
zooko@tahoe-lafs:~/playground/tahoe-lafs/1.8.3/docs$ elinks -dump ~/public_html/known_issues.html | head -4
                                  Known Issues

     * [1]Overview
     * Issues in Tahoe-LAFS v1.8.2, released 2011-01-30
zooko@tahoe-lafs:~/playground/tahoe-lafs/1.8.3/docs$ rst2html --version
rst2html (Docutils 0.6 [release], Python 2.6.5, on linux2)

In contrast, the trac rendering to html of the same file on the same machine starts with:

============ Known Issues ============

    Overview

(Visible here: ​https://tahoe-lafs.org/trac/tahoe-lafs/browser/1.8.3/docs/known_issues.rst )

Which suggests that trac doesn't realize that those first three lines are actually the same length as one another. If I insert a couple of blank lines before the first "============" line, then the mis-rendering from trac stops happening.

Now about the encoding, isn't the point of having a BOM in the file that then the parser can know what encoding the file is in even if it isn't told out-of-band? But anyway, even if it isn't told out-of-band, or if it is told the incorrect encoding out-of-band, a good parser should probably (?) pop off any leading UTf-8 BOM no matter what, before parsing.

Anyway, in answer to your question, I'm using darcs, [trac]default_charset = utf-8. I'm not 100% sure that means the trac darcs plugin is giving the right encoding to the trac rst renderer. Lele Gaifax (author of trac darcs) would probably be able to tell us for sure.

Thanks for your help!

[Christian Boos]

Ok, docutils first removes the BOM but only if it had to try to decode the str itself. If it is given unicode, it takes it wholesale and then doesn't know what to do with the BOM.

So at the very least we should remove the BOM as well in this case… and maybe even in our content_to_unicode utility function:

trac/mimeview/api.py

@@ -451 +451 @@
         return None
 
 def content_to_unicode(env, content, mimetype):
-    """Retrieve an `unicode` object from a `content` to be previewed"""
+    """Retrieve an `unicode` object from a `content` to be previewed.
+
+    In case the raw content had an unicode BOM, we remove it.
+    """
     mimeview = Mimeview(env)
     if hasattr(content, 'read'):
         content = content.read(mimeview.max_preview_size)
-    return mimeview.to_unicode(content, mimetype)
+    u = mimeview.to_unicode(content, mimetype)
+    if u and u[0] == u'\ufeff':
+        u = u[1:]
+    return u
 
 
 class IHTMLPreviewRenderer(Interface):

The fix will be for 0.12.4 though, unless people give this a lot of testing ;-)

[lkraav <leho@…>]

While we're riding the Trac UTF8 BOM mindshare train, perhaps #10441 could get a comment?

[Christian Boos]

Fix applied in r10967.