CSV export of tickets has wrong encoding

[jho]

When I download a CSV version of a ticket report, all the Umlaut characters (ö,ü,ä, etc.) are displayed incorrectly (i.e.: instead of ä I get ä)

[Christian Boos]

The export is currently done using UTF-8 and it's true that Excel for one doesn't like that…

Given that Excel expects UTF16-BE (according to some sources, like ​http://www.herongyang.com/unicode/utf.html, not yet verified), it might be worth changing if other tools like OpenOffice also understand that encoding.

[Thijs Triemstra <lists@…>]

Doesn't look like OpenOffice (3.2.1) supports utf16-be..

[Thijs Triemstra <lists@…>]

Replying to cboos:

The export is currently done using UTF-8 and it's true that Excel for one doesn't like that…

Given that Excel expects UTF16-BE (according to some sources, like ​http://www.herongyang.com/unicode/utf.html, not yet verified), it might be worth changing if other tools like OpenOffice also understand that encoding.

Unless I'm reading that article incorrectly, but it states that "MS Word and Excel don't support this [UTF-16BE] encoding". That same guy wrote ​this article that states "This section provides a tutorial example to prove that Excel can not open a UTF-16BE text file. Its Text Import Wizard only supports UTF-7 and UTF-8 encodings."

OpenOffice also only support UTF-7 and 8 (and the other encodings that excel also supports) so it looks like this should be fixed, or can be easily fixed.

[Thijs Triemstra <lists@…>]

I tested this with OpenOffice and the .csv export below and everything works as expected. Since this was reported against 0.10.4 I expect that this issue has been fixed in the meantime so I'm closing it as worksforme. Please re-open if you can still reproduce it (or if I'm not allowed to close tickets hehe).

__color__,ticket,summary,component,version,milestone,type,owner,status,created,_changetime,_description,_reporter
3,1,Title ticket 1,component1,,,defect,somebody,new,2010-09-29T00:10:30+02:00,2010-10-04T00:43:13+02:00,"[[TracAdminHelp(upgrade)]]

2,sdsadsa
dsadsdsadasdas
",admin
3,2,Title ticket 2,component1,,,defect,somebody,new,2010-09-29T00:10:36+02:00,2010-10-04T00:27:34+02:00,Description ticket 2.,admin
3,3,Title ticküät 3,component1,,,defect,somebody,new,2010-09-29T00:10:43+02:00,2010-10-04T19:37:02+02:00,Descriptiön ticket 3.,admin

[Remy Blank]

Wasn't the question rather whether Excel can read UTF-8 or not?

[Jun Omae]

csv export with utf-8 bom patch (r10225)

[Jun Omae]

I have the same problem and CJK users also. Microsoft Excel read csv files with Shift_JIS in the Japanese version of Windows.

Replying to rblank:

Wasn't the question rather whether Excel can read UTF-8 or not?

Excel cannot read UTF-8 however it can read UTF-8 BOM. I tested with Excel 2003 and 2007 for Japanese version.

t6656-utf8-bom-r10225.diff​ - CSV export with UTF-8 BOM

[Christian Boos]

How do other tools react with this patch? In particular OpenOffice?

[Jun Omae]

OpenOffice 3.2 Calc can read UTF-8 w/o BOM and UTF-8 w/ BOM. It works fine.

[Christian Boos]

Perfect! I'll apply the patch, then.

[Christian Boos]

Patch applied in r10226.

Could you please next time write a commit log message in your patches? They will be perfect, then ;-)

[Christian Boos]

… and run the functional tests ;-)

Follow-up fixes in r10227.

[anonymous]

you'd have to tell the OpenOffice to import it as Unicode anyways though, alternate solution is a UTF-16LE, but that is troublesome because the csv writer that only support utf-8 and you need a not so nice hack for that, so probably this is good enough.