trac-admin copystatic raises unicode decode error

[ilias@…]

P:\>trac-admin intra copystatic  tstatic
Copying resources from:
  trac.web.chrome.Chrome
    p:\own\dev\infra\trac\trac\htdocs
    P:\intra\htdocs
Traceback (most recent call last):
  File "C:\prg\py24\Scripts\trac-admin-script.py", line 7, in ?
    sys.exit(
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 1222, in run
    return admin.onecmd(command)
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 102, in onecmd
    rv = cmd.Cmd.onecmd(self, line) or 0
  File "C:\prg\py24\lib\cmd.py", line 219, in onecmd
    return func(arg)
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 1172, in do_copystatic
    copytree(source, dest)
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 64, in copytree
    copytree(srcname, dstname, symlinks, skip)
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 64, in copytree
    copytree(srcname, dstname, symlinks, skip)
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 64, in copytree
    copytree(srcname, dstname, symlinks, skip)
  File "P:\own\dev\infra\trac\trac\admin\console.py", line 58, in copytree
    dstname = os.path.join(dst, name)
  File "C:\prg\py24\lib\ntpath.py", line 102, in join
    path += "\\" + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 1: ordinal not in range(128)

[Ernesto <ernestohdez87@…>]

hacker@ubuntu1:~/Desktop/todo/Trac-0.11rc1$ trac-admin help Traceback (most recent call last):

File "/usr/bin/trac-admin", line 8, in <module>

load_entry_point('Trac==0.11rc1', 'console_scripts', 'trac-admin')()

File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 277, in load_entry_point

return get_distribution(dist).load_entry_point(group, name)

File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 2179, in load_entry_point

return ep.load()

File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 1912, in load

entry = import(self.module_name, globals(),globals(), __name__)

File "/usr/lib/python2.5/site-packages/Trac-0.11rc1-py2.5.egg/trac/admin/console.py", line 34, in <module>

from trac.ticket.model import *

File "/usr/lib/python2.5/site-packages/Trac-0.11rc1-py2.5.egg/trac/ticket/init.py", line 1, in <module>

from trac.ticket.api import *

File "/usr/lib/python2.5/site-packages/Trac-0.11rc1-py2.5.egg/trac/ticket/api.py", line 31, in <module>

from trac.wiki import IWikiSyntaxProvider, WikiParser

File "/usr/lib/python2.5/site-packages/Trac-0.11rc1-py2.5.egg/trac/wiki/init.py", line 1, in <module>

from trac.wiki.api import *

File "/usr/lib/python2.5/site-packages/Trac-0.11rc1-py2.5.egg/trac/wiki/api.py", line 36, in <module>

from trac.wiki.parser import WikiParser

ImportError: No module named parser hacker@ubuntu1:~/Desktop/todo/Trac-0.11rc1$

[osimons]

I got a very similar error doing trac-admin ... hotcopy which uses the same copytree() method.

Could you try the following patch and confirm that it solves it? Make sure to double-check the result for any non-ascii filenames and paths you have…

trac/admin/console.py

@@ -54 +54 @@
     os.mkdir(dst)
     errors = []
     for name in names:
-        srcname = os.path.join(src, name)
+        srcname = to_unicode(os.path.join(src, name))
         if srcname in skip:
             continue
-        dstname = os.path.join(dst, name)
+        dstname = to_unicode(os.path.join(dst, name))
         try:
             if symlinks and os.path.islink(srcname):
                 linkto = os.readlink(srcname)

(As for Ernesto, that traceback of yours is completely unrelated to the issue of this ticket…)

[anonymous]

Replying to osimons:

I got a very similar error doing trac-admin ... hotcopy which uses the same copytree() method.

Could you try the following patch and confirm that it solves it? Make sure to double-check the result for any non-ascii filenames and paths you have…

Sorry, I am in a non-coding phase currently, so i cannot make this test.

I missed to write the following: I'v used greek characters ας the project τιτλε, that was the only utf-8 if I remember right.

You could use this as a project title:

Ελληνικά

and reproduce the error, apply the patch, test again as you like.

cu!

[ilias@…]

anonymous was me

[Christian Boos]

I think os.path normally knows how to handle unicode paths.

I couldn't reproduce the bug (neither for deploy nor for hotcopy) even when using a non-ascii target directory (C:/TEMP/testé).

Simon, could you give me either a precise reproduction recipe or more details debug information? (os.path should normally be able to cope with unicode paths).

[ilias@…]

Replying to cboos:

Simon, could you give me either a precise reproduction recipe or more details debug information? (os.path should normally be able to cope with unicode paths).

see comment:4

the bug raised simply by using an utf-8 project title:

You could use this as a project title:

Ελληνικά

[Christian Boos]

Replying to ilias@lazaridis.com:

the bug raised simply by using an utf-8 project title:

I doubt this, because since #6535 my test project name is Trac Dével (ü) Σχέδιο.

[Christian Boos]

Replying to cboos:

I think os.path normally knows how to handle unicode paths.

I couldn't reproduce the bug (neither for deploy nor for hotcopy) even when using a non-ascii target directory (C:/TEMP/testé).

Actually, it only worked by luck, since the target directory after the copy appears to be named: "C:/TEMP/testΘ". So there's indeed an encoding issue with the target path.

[Christian Boos]

Tricky … my cmd.exe shell has cp437:

> chcp
Active code page: 437

> python - C:\TEMP\testé
>>> import sys
>>> sys.stdin.encoding
'cp437'

so far so good, python recognized that, and handles strings entered from that shell the expected way:

>>> test = r'C:\TEMP\testé'
>>> test.decode('cp437')
u'C:\\TEMP\\test\xe9'

Now, the funny thing. Remember that python - C:\TEMP\testé command line?

>>> sys.argv[1]
'C:\\TEMP\\test\xe9'

Right, that's latin-1 apparently.

Therefore our code in admin/console.py (which does a line = to_unicode(line, sys.stdin.encoding)) is based on the wrong assumption that the strings in sys.argv are encoded using the stdin encoding. Assumption which is not true on Windows when using cmd.exe at least.

But when parsing the command line arguments

[Christian Boos]

See related ​pythonbug:2128.

[Christian Boos]

So it seems that sys.argv strings are actually encoded in the locale encoding (on Windows)

trac/admin/console.py

@@ -99 +99 @@
         """`line` may be a `str` or an `unicode` object"""
         try:
             if isinstance(line, str):
-                line = to_unicode(line, sys.stdin.encoding)
+                line = to_unicode(line, locale.getpreferredencoding())
             line = line.replace('\\', '\\\\')
             rv = cmd.Cmd.onecmd(self, line) or 0
         except SystemExit:

Would be worth testing on other platforms (Linux, MacOS, …)

[osimons]

Did some more testing, and my traceback actually happens in copytree() when testing if srcname in skip for a file in project htdocs directory. Here is a very simple illustration of what I think happens at my end:

>>> f = os.listdir('.')[-1] # only need one file for testing
>>> f
'\xc3\xa6\xc3\xb8a\xcc\x8a.txt'
>>> type(f)
<type 'str'>
>>> print f
æøå.txt
>>> l = ['hi.txt']
>>> f in l
False
>>> l = [u'hi.txt', 'hi.txt'] # one item is unicode
>>> f in l
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position...

The output from os.listdir() are plain strings. A slightly more focused patch is therefore:

trac/admin/console.py

@@ -50 +50 @@
     Added a `skip` parameter consisting of absolute paths
     which we don't want to copy.
     """
-    names = os.listdir(src)
+    names = [to_unicode(f) for f in os.listdir(src)]
     os.mkdir(dst)
     errors = []
     for name in names:

My testing is on OSX, and the result of sys.stdin.encoding is 'UTF-8'. Looking at console encoding I have a feeling that it is a slightly different issue as no console input/output are directly involved - see also #7303. In the example above, the console outputs the filename just fine when printing.

[osimons]

Oh, should add this to the description of my test environment as well:

>>> locale.getpreferredencoding()
'mac-roman'

[osimons]

Here is an old (still open) Python bug with an interesting discussion about unicode and filenames; ​767645.

>>> os.path.supports_unicode_filenames
False

[Christian Boos]

Replying to osimons:

Did some more testing, and my traceback actually happens in copytree() when testing if srcname in skip

IIUC, srcname is a str and skip contains a unicode object, right? What does print repr((srcname,skip)) shows before the error triggers?

Actually, I think that for having a robust copytree, we shouldn't bother with unicode at all. We don't have control over the files present in the tree and in the general case, there could be file names consisting in byte sequences which don't correspond to the current encoding (be it sys.getfilesystemencoding(), sys.getdefaultencoding() or locale.getpreferredencoding()).

[Christian Boos]

About that other issue:

Replying to cboos:

So it seems that sys.argv strings are actually encoded in the locale encoding (on Windows) … Would be worth testing on other platforms (Linux, MacOS, …)

Some indications that this should be safe on Linux: ​encoding of sys.argv ? on python-list.

What about MacOS/X?

[osimons]

Replying to cboos:

IIUC, srcname is a str and skip contains a unicode object, right? What does print repr((srcname,skip)) shows before the error triggers?

('/...../project1/htdocs/CR1blA\xcc\x8a\xe2\x80\x99_neg.png',
[u'/...../project1/db/trac.db-journal', u'/...../project1/db/trac.db-stmtjrnl'])

(Paths shortened by me.)

[Christian Boos]

Okay: hotcopy does a copytree(self.__env.path, dest, ... skip), where self.__env.path is a str (sys.argv[0] didn't go through the line = line.to_unicode(...) conversion) and skip is computed using db_path, taken from the config file and hence is unicode.

We also need to fix dest, otherwise the os.path.join(dst, name) will fail when seeing '.../CR1blA\xcc\x8a\xe2\x80\x99_neg.png'.

In order to convert that dest to a str, we first need to decode it properly, and for that I just realized that the patch in comment:12 is only needed in batch mode and won't work in interactive mode, in which case using sys.stdin.encoding was the correct thing to do!

[Christian Boos]

fix command line conversion to unicode, in non-interactive mode (see comment:12)

[Christian Boos]

fix copytree so that it can cope with any filename it finds on its way

[Christian Boos]

Please try out the attached patches (they are Mercurial patches, so they need to be applied using -p1).

[osimons]

Replying to cboos:

Please try out the attached patches (they are Mercurial patches, so they need to be applied using -p1).

Patches looks good. At least my hotcopy now works without problems.

[Christian Boos]

Ok, thanks. Could the reporter confirm the fix as well?

[ilias@…]

Replying to cboos:

Ok, thanks. Could the reporter confirm the fix as well?

sorry, i cannot recreate the environment that cause the error at this moment.

will do the test when i update the installation to the latest version (in a few weeks)

[Christian Boos]

Replying to ilias@lazaridis.com:

Replying to cboos:

Ok, thanks. Could the reporter confirm the fix as well?

will do the test when i update the installation to the latest version (in a few weeks)

No need Ilias, I think I have now figured out what you meant in comment:7: it's when the environment path contains non-ascii characters, it fails right away. Because you said "utf-8 project title", this made me think about #6535 and I got confused.

So actually a trac-admin C:\TEMP\testé1 hotcopy C:\TEMP\testé2 command also fails for me, even with the above patches. I'll investigate further.

[ilias@…]

btw, I'm missing copystatic form trac-admin, it didn't list in the help, too. Version:

P:\local>trac-admin
trac-admin - The Trac Administration Console 0.11b1

[Christian Boos]

That command was renamed a few times (staticcopy, copystatic and now deploy), see log:trunk/trac/admin/console.py@6894. Note that 0.11 is out since a month now (0.11b1 is way obsolete - there has been b2, rc1 and rc2 since then).

[Christian Boos]

Ok, so I've applied attachment:6677-fix-argv-encoding.patch​ (r7360) and attachment:6677-fix-copytree.patch​ (r7361).

The error you get for a command like trac-admin C:\TEMP\testé1 hotcopy C:\TEMP\testé2 can't be fixed easily, as the implication would be that a non-ascii environment path would be supported in Trac, which is not the case for now. Adding that support is beyond the scope of that ticket, so for now I simply added an explicit error message in this situation (r7363).