encoding related test case failure on Mac OS X

[Christian Boos]

As reported by eblot in #3331:

======================================================================
FAIL: test_milestone_add_utf8_ok (trac.scripts.tests.admin.TracadminTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/eblot/Sources/Svn/edgewall.com/trunk/trac/scripts/tests/admin.py", line 860, in test_milestone_add_utf8_ok
    self.assertEqual(self.expected_results[test_name], output)
AssertionError: u'\nName        Due         Completed\n---------------------------------\n\xe9tat_final  2004-01-11\nmilestone1\nmilestone2\nmilestone3\nmilestone4\n\n' != u'\nName        Due         Completed\n---------------------------------\n?tat_final  2004-01-11\nmilestone1\nmilestone2\nmilestone3\nmilestone4\n\n'

======================================================================
FAIL: test_implicit_lossless (trac.util.tests.text.ToUnicodeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/eblot/Sources/Svn/edgewall.com/trunk/trac/util/tests/text.py", line 40, in test_implicit_lossless
    self.assertEquals(u'\xc3', uc)
AssertionError: u'\xc3' != u'\u221a'

----------------------------------------------------------------------
Ran 580 tests in 31.179s

[Emmanuel Blot]

As cmlenz reported on IRC yesterday evening, the first "bug" is due to the default encoding of the terminal test scripts are running in. If the terminal locale is configured to use UTF-8, the first issue disappears. The second one remains however.

[Christian Boos]

Also, the third parameter to to_unicode (i.e. lossy) was meant to help while converting raw content, in the mimeview pipeline. I think it turned out we ain't gonna need it, after all ;) So this will probably be removed for 0.10.

[Christian Boos]

Replying to eblot:

… the first "bug" is due to the default encoding of the terminal test scripts are running in. If the terminal locale is configured to use UTF-8, the first issue disappears. …

Can you try the following patch?

Index: trac/scripts/tests/admin.py
===================================================================
--- trac/scripts/tests/admin.py	(revision 3566)
+++ trac/scripts/tests/admin.py	(working copy)
@@ -118,7 +118,7 @@
             _err = sys.stderr
             _out = sys.stdout
             sys.stderr = sys.stdout = out = StringIO()
-            setattr(out, 'encoding', _out.encoding) # fake output encoding
+            setattr(out, 'encoding', 'utf-8') # fake output encoding
             retval = None
             try:
                 retval = self._admin.onecmd(cmd)
@@ -126,7 +126,7 @@
                 pass
             value = out.getvalue()
             if isinstance(value, str): # reverse what print_listing did
-                value = value.decode(_out.encoding)
+                value = value.decode('utf-8')
             if strip_trailing_space:
                 return retval, STRIP_TRAILING_SPACE.sub('', value)
             else:

[Christian Boos]

Replying to cboos:

Also, the third parameter to to_unicode (i.e. lossy) … will probably be removed for 0.10.

Did that in r3569.

[Emmanuel Blot]

Replying to cboos:

Can you try the following patch?

Yes, it fixes the first issue, when the Terminal is not configured for a custom encoding, i.e. when LANG and LC_ALL are not defined in the environment.

[Christian Boos]

Replying to eblot:

Yes, it fixes the first issue, when the Terminal is not configured for a custom encoding, i.e. when LANG and LC_ALL are not defined in the environment.

Fine then, I've committed that patch as r3570. But now the test should work in all cases, regardless of the LANG or LC_ALL settings, right?

[Emmanuel Blot]

But now the test should work in all cases, regardless of the LANG or LC_ALL settings, right?

I'll try to run a couple of test sessions w/ various encoding this evening.

(btw, the ticket reply feature is really chaotic with Safari, I can't find what's wrong: I've been able to reply 5 minutes ago on this very ticket, I can't with this last comment)

[Christian Boos]

Replying to eblot:

(btw, the ticket reply feature is really chaotic with Safari, I can't find what's wrong: I've been able to reply 5 minutes ago on this very ticket, I can't with this last comment)

Time for yet another ticket ;) (a 0.11/0.10.x ticket, though)

[Emmanuel Blot]

Time for yet another ticket ;) (a 0.11/0.10.x ticket, though)

It already exists, I re-opened it: #3327

Back to this ticket: I'm not sure to understand why you have closed this ticket: if the patch makes the first issue disappear, it does not solve the second one.

[Christian Boos]

Well, the second one should be gone with r3569, no? In any case, add here any related encoding issue you'll find this evening … and if none, then please close the ticket again.

[Emmanuel Blot]

Works perfectly,whatever the terminal locale settings. Thanks a lot. Both issues have been resolved.