whitespace not preserved in `{{{ }}}` blocks

[Matthew Good]

There's a test failure which appears to be due to whitespace not being preserved in the {{{ }}} blocks. This is possibly an issue in Genshi, though I thought it was already fixed to preserve whitespace inside <pre> elements.

======================================================================
FAIL: Test Unfinished code blocks
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matt/Projects/trac/trunk/trac/wiki/tests/formatter.py", line 129, in test
    raise AssertionError( # See below for details
AssertionError:
--------------- expected:
<p>
Block
</p>
<pre class="wiki">number one

and block
{{{
number two
 }}
}}}
</pre>
--------------- actual:
<p>
Block
</p>
<pre class="wiki">number one
and block
{{{
number two
 }}
}}}
</pre>
---------------


/home/matt/Projects/trac/trunk/trac/wiki/tests/wiki-tests.txt:413: "Unfinished code blocks" (default flavor)

[Christian Boos]

That only happens with Python 2.5, AFAICT. At least, all test succeed with ActivePython 2.4.2, using ​[genshi 347].

I'll re-test with 2.5 on win64 later today.

[Manuzhai]

I see the same problem with Python 2.4.3 on Gentoo Linux (x86).

[Christian Boos]

Ok, tried with Python 2.4.2 on Linux (x64) and I have the issue too (latest trunk of both).

[Manuzhai]

That would make it (at least partly) an OS/architecture-related problem? Could have something to do with differing newlines? \r\n vs. \n, that sort of thing.

[Christian Boos]

Right, on Windows, the usage of \r\n masks the problem.

The problem is that the <pre> blocks, as output by the Wiki formatter, are seen as chunks of Markup by Genshi (he, nice to have different names now ;) ). As such, the genshi.output.WhitespaceFilter will simplify the whitespace and collapse the lines. The default regexp for line collapsing is '\n{2,}', that's why the problem was not visible for me on Windows, due to the usage of os.linesep throughout trac.wiki.Formatter as the line separator.

See the __call__ method at ​genshi:source:trunk/genshi/output.py@351#L475.

That's one situation where it would make sense to generate XML serialization events for Genshi, when rendering a parse tree to XHTML.

[Christian Boos]

Maybe an intermediate solution would be to use the builder for the <pre> blocks.

[Manuzhai]

This should get some love. It's the last failing test; the buildbots would be much nicer when this didn't always fail.

[Christian Boos]

… don't forget to revert r4608 when testing this issue ;)

[Emmanuel Blot]

Increasing a bit the priority: many people are using {{{ }}} to copy n' paste existing documents in the wiki - bypassing WikiFormatting - and all their documents have become pretty hard to read without the empty lines ;-(

[Christian Boos]

Well, it's probably about time to think about this one if we want to have it fixed for 0.11, you're right ;)

Related to #4431.

[Emmanuel Blot]

Seems to be related to ​What is Genshi doing with the whitespace in my markup template

[Christian Boos]

This is one of the main annoyance with 0.11, bumping up the priority.

[Christopher Lenz]

This should be fixed in [5586]. As I noted in the log message, a proper fix can only be achieved once the wiki formatter has been updated to emit event streams.

[Emmanuel Blot]

Thanks a million, Chris.

I think the fix is missing here, as the wiki preview keeps removing whitespace:

Index: wiki/templates/wiki_edit.html
===================================================================
--- wiki/templates/wiki_edit.html       (revision 5586)
+++ wiki/templates/wiki_edit.html       (working copy)
@@ -54,7 +54,7 @@
             <div py:when="diff" class="diff">
               <xi:include href="diff_div.html" py:with="no_id=True" />
             </div>
-            <div py:otherwise="" class="wikipage">${wiki_to_html(context, page.text)}</div>
+            <div py:otherwise="" class="wikipage" xml:space="preserve">${wiki_to_html(context, page.text)}</div>
           </fieldset>
         </py:when>
         <py:when test="'collision'">

[Christopher Lenz]

You're right. Should be fixed in [5587:5588].