support non-ascii filenames in mercurial vcs backend

[anonymous]

This buf was reported in #8018, #7799, and #7160 but solution on #7160:

os.environHGENCODING = "UTF-8" not solves error.

We solve the error with this patch:

When you want to open dir in browse source with a file or child file with an accented charcater (ó) the error shows:

Most recent call last: 
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 360, in process_request
  'dir': node.isdir and self._render_dir(req, repos, node, rev),
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 406, in _render_dir
  entries = [entry(n) for n in node.get_entries()]
File "/var/trac/scrum.pxgo.es/norprevencion_openerp/plugins/TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py", line 697, in get_entriesFile "/var/trac/scrum.pxgo.es/norprevencion_openerp/plugins/TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py", line 605, in findnode

Patch:

In TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py method findnode:

for f in self.repos.repo.changectx(rev).files():
    f = unicode(f, 'utf-8', 'replace')  

#convert to unicode with replace and utf-8 encode and the name of file process correctly.

Then the dir works if the file is another dir else the error shows in other part of code:

Most recent call last: 
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 360, in process_request
  'dir': node.isdir and self._render_dir(req, repos, node, rev),
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 406, in _render_dir
  entries = [entry(n) for n in node.get_entries()]
File "/var/trac/scrum.pxgo.es/norprevencion_openerp/plugins/TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py", line 691, in get_entriesFile "/usr/local/lib/python2.5/posixpath.py", line 65, in join
  path += '/' + b

Patch:

In TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py method get_entries the same:

for entry in self.entries:
    entry = unicode(entry, 'utf-8', 'replace')

#convert the file's name to unicode with replace and encode in utf-8

Then the dir with file works but when you want to open the file and error shows:

Most recent call last: 
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 373, in process_request
  href=req.href.changeset(node.rev, node.created_path)))
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/href.py", line 163, in <lambda>
  self._derived[name] = lambda *args, **kw: self(name, *args, **kw)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/href.py", line 148, in __call__
  if arg != None])

This bug raise on Trac, i don't know if in new versions it solves, occurs beacause teh name of file have a accented character and in url not escapes the accented character.

[Christian Boos]

Any chance for me to get a clone of this repository (or better, a small sample one that will produce similar errors)?

[anonymous]

Replying to cboos:

Any chance for me to get a clone of this repository (or better, a small sample one that will produce similar errors)?

Hello,

Only you must have a repository when a file with an accented name and do commit. eg: código.jpg. When you want to view the repository in trac, in directory under is accented file teh error shows, and you can fix it. Then when you open teh dir and the dir shows teh namne of fiel the error shows too, you can fix it.

Thanks

[Andreas Schäfer <gentryx@…>]

Hi,

I get the same error, but had no success with the solution proposed above. Please have a the following tar balls:

• ​http://gentryx.de/~gentryx/trac-hg.tbz Mercurial (version 8864112d4f57, live ebuild) repository which triggers the error)
• ​http://gentryx.de/~gentryx/trac-unicode-bug-test-repo-hg.tbz Trac instance which produces said error.

Any help on this would be highly appreciated. :-)

[Andreas Schäfer <gentryx@…>]

BTW, I don't see this error with the GIT plugin, if that is of any help.

[Christian Boos]

Nom that doesn't help ;-)

I'm working on a fix, but it'll take me a few more days.

[Christian Boos]

minimal fix - not very well tested

[Christian Boos]

Just in case you'd like to do some testing, here's what I've got so far.

[Andreas Schäfer <gentryx@…>]

Sorry for lagging, didn't watch the ticket as closely as I should have. Looks like the patch works just fine, many thanks. :-)

[Christian Boos]

#7058 and #7799 were closed as duplicate of this ticket.

[richard.liao.i@…]

This patch not work for me, with gbk filename. I have

• set filename_encoding = gbk
• patch os.environHGENCODING = 'utf-8'

When browsing changeset, raise

2010-01-19 17:16:26,025 Trac[main] ERROR: 'ascii' codec can't decode byte 0xb2 in position 25: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/main.py", line 462, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/main.py", line 213, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/versioncontrol/web_ui/changeset.py", line 324, in process_request
    self._render_html(req, repos, chgset, restricted, xhr, data)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/versioncontrol/web_ui/changeset.py", line 587, in _render_html
    'new': new_node and node_info(new_node, annotated),
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/versioncontrol/web_ui/changeset.py", line 452, in node_info
    None),
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/href.py", line 163, in <lambda>
    self._derived[name] = lambda *args, **kw: self(name, *args, **kw)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/href.py", line 148, in __call__
    if arg != None])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb2 in position 25: ordinal not in range(128)

[Christian Boos]

#9631 was closed as duplicate of this ticket. It also had a partial patch.

What is needed is some comprehensive from/to_hg_path conversion…

[anonymous]

anything new?

now i apply this patch to TracMercurial-0.12.0.23dev_r9953-py2.6 the error is : NameError: global name 'options' is not defined

and then i changed this line: self.filename_encoding = options.get('filename_encoding') to: self.filename_encoding = 'utf-8'

and then the error is :

Traceback (most recent call last):
  File "\Python26\lib\site-packages\trac\web\api.py", line 436, in send_error
    data, 'text/html')
  File "\Python26\lib\site-packages\trac\web\chrome.py", line 861, in render_template
    location=location))
TracError: <unprintable TracError object>

i'm sure the non-ascii filenames in mercurial is in utf-8 (confirmed by hg serve ) everything is ok in hg serve web interface why so difficult to display it in tracmercurial

[Christian Boos]

Replying to anonymous:

anything new?

Thanks for the heads up, I'll try to fix this, this evening. Would you mind in the meantime provide me with a sample repository I could use for testing? (or simply the URL for cloning yours in case it's a public repo).

[anonymous]

try this: hg clone ​https://bardiku@bitbucket.org/bardiku/testtrachgutf8filename

[anonymous]

tracmercurial can list the non-ascii filenames but can not when diaplaying its content.

[anonymous]

Replying to cboos:

Thanks for the heads up, I'll try to fix this, this evening.

good news. such a long time. good luck!

[Christian Boos]

Please update to source:plugins/0.12/mercurial-plugin@10491​ and give it a try.

I'm not closing yet as further heavy changes are expected (as the heavy number of FIXME shows…) and it's likely that there are or will be a few glitches. Please report them here as you find them.

[anonymous]

it works. great job.

[Christian Boos]

#10000 was one such glitch.

Now with the even bigger change of r10496, I imagine there could be some more… maybe better open a new ticket in such case anyway.

As the issues with the unicode paths should be all fixed now, I'm closing this ticket.