Unicode initenv problem

[fred.stober@…]

Hi!

I've had a big problem with umlauts in my svn repository. I've just converted my cvs repository to svn using cvs2svn and used trac-admin to import the repository using initenv. But something went wrong during the import. I've always got this error trying to access the timeline or using the browser to view directories with files with had an umlaut in their log message:

OperationalError: Could not decode to UTF-8 column 'message' with text …

I've had to apply this patch:

/trac/versioncontrol/svn_fs.py
643c635
<         message = self._get_prop(core.SVN_PROP_REVISION_LOG)
>         message = to_unicode(self._get_prop(core.SVN_PROP_REVISION_LOG))

(Usiing Gentoo ebuild www-apps/trac-0.10.1)

And now everything works fine!

Keep up the good work!

[Jonas Borgström]

Subversion always stores filenames and log messages as utf-8 internally. So the proposed patch should never be needed. This was probably caused by a missing or incorrect '—encoding' parameter when running the cvs2svn command. '—encoding=iso-8859-1' will probably do the trick.

[Christian Boos]

Replying to jonas:

Subversion always stores filenames and log messages as utf-8 internally. So the proposed patch should never be needed.

No, but a similar one which transforms the UTF-8 string to unicode should. Currently, this (last?) use of UTF-8 goes unnoticed thanks to the cache.

[Christian Boos]

Different patch applied in r4394. As jonas said, we should assume that Subversion gives us proper UTF-8 encoded strings, so we use here _from_svn() instead of the more error tolerant to_unicode().

[eli.carter@…]

This needs to be back-ported to the 0.10 line.

[Christian Boos]

Replying to eli.carter@commprove.com:
As discussed on #trac, I'm waiting for some feedback (the repr() of the message or author which is failing), but I'm quite sure that it can well be that Subversion does not always give back properly UTF-8 encoded strings (e.g. repositories created from cvs-import, like in your case). We shouldn't abort in this situation but be more robust (using to_unicode(text, 'utf-8') which will assume UTF-8 but use replacement characters if invalid UTF-8 data is found).

i.e. (patch on 0.10-stable)

svn_fs.py

@@ -635 +635 @@
         self.pool = Pool(pool)
         message = self._get_prop(core.SVN_PROP_REVISION_LOG)
         author = self._get_prop(core.SVN_PROP_REVISION_AUTHOR)
+        message = to_unicode(message, 'utf-8')
+        author = to_unicode(author, 'utf-8')
         date = self._get_prop(core.SVN_PROP_REVISION_DATE)
         if date:
             date = core.svn_time_from_cstring(date, self.pool()) / 1000000

[Christian Boos]

PS: Note that this will require a resync

[eli.carter@…]

Replying to cboos:

Replying to eli.carter@commprove.com:
As discussed on #trac, I'm waiting for some feedback (the repr() of the message or author which is failing),

The relevant portion of one of the log messages is:

'Struttura modificata per uniformit\xfc\xb0\x80\x80\x81\xa0 ...'

And the resync gave this error:

Command failed: 'utf8' codec can't decode bytes in position 34-39: unsupported Unicode code range

[eli.carter@…]

The patch shown above for 0.10-stable works for me. trunk will need to change _from_svn(...) to to_unicode(..., 'utf-8') as well.

[Christian Boos]

Ok, will do. Thanks for the feedback!

[Christian Boos]

Applied in r4491 (trunk) and r4492 (0.10-stable).