Edgewall Software
Modify

Opened 14 years ago

Closed 12 years ago

#8538 closed defect (fixed)

support non-ascii filenames in mercurial vcs backend

Reported by: anonymous Owned by: Christian Boos
Priority: highest Milestone: plugin - mercurial
Component: plugin/mercurial Version: 0.11.4
Severity: major Keywords: unicode
Cc: Branch:
Release Notes:
API Changes:
Internal Changes:

Description (last modified by Christian Boos)

This buf was reported in #8018, #7799, and #7160 but solution on #7160:

os.environHGENCODING = "UTF-8" not solves error.

We solve the error with this patch:

When you want to open dir in browse source with a file or child file with an accented charcater (ó) the error shows:

Most recent call last: 
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 360, in process_request
  'dir': node.isdir and self._render_dir(req, repos, node, rev),
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 406, in _render_dir
  entries = [entry(n) for n in node.get_entries()]
File "/var/trac/scrum.pxgo.es/norprevencion_openerp/plugins/TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py", line 697, in get_entriesFile "/var/trac/scrum.pxgo.es/norprevencion_openerp/plugins/TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py", line 605, in findnode

Patch:

In TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py method findnode:

for f in self.repos.repo.changectx(rev).files():
    f = unicode(f, 'utf-8', 'replace')  

#convert to unicode with replace and utf-8 encode and the name of file process correctly.

Then the dir works if the file is another dir else the error shows in other part of code:

Most recent call last: 
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 360, in process_request
  'dir': node.isdir and self._render_dir(req, repos, node, rev),
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 406, in _render_dir
  entries = [entry(n) for n in node.get_entries()]
File "/var/trac/scrum.pxgo.es/norprevencion_openerp/plugins/TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py", line 691, in get_entriesFile "/usr/local/lib/python2.5/posixpath.py", line 65, in join
  path += '/' + b

Patch:

In TracMercurial-0.11.0.7-py2.5.egg/tracext/hg/backend.py method get_entries the same:

for entry in self.entries:
    entry = unicode(entry, 'utf-8', 'replace')

#convert the file's name to unicode with replace and encode in utf-8

Then the dir with file works but when you want to open the file and error shows:

Most recent call last: 
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/versioncontrol/web_ui/browser.py", line 373, in process_request
  href=req.href.changeset(node.rev, node.created_path)))
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/href.py", line 163, in <lambda>
  self._derived[name] = lambda *args, **kw: self(name, *args, **kw)
File "/usr/local/lib/python2.5/site-packages/Trac-0.11.4-py2.5.egg/trac/web/href.py", line 148, in __call__
  if arg != None])

This bug raise on Trac, i don't know if in new versions it solves, occurs beacause teh name of file have a accented character and in url not escapes the accented character.

Attachments (1)

filename_encoding-mercurial-0.11-r7849.diff (3.6 KB ) - added by Christian Boos 13 years ago.
minimal fix - not very well tested

Download all attachments as: .zip

Change History (20)

comment:1 by Christian Boos, 14 years ago

Description: modified (diff)
Keywords: unicode added
Milestone: 0.11.6not applicable

Any chance for me to get a clone of this repository (or better, a small sample one that will produce similar errors)?

in reply to:  1 comment:2 by anonymous, 14 years ago

Replying to cboos:

Any chance for me to get a clone of this repository (or better, a small sample one that will produce similar errors)?

Hello,

Only you must have a repository when a file with an accented name and do commit. eg: código.jpg. When you want to view the repository in trac, in directory under is accented file teh error shows, and you can fix it. Then when you open teh dir and the dir shows teh namne of fiel the error shows too, you can fix it.

Thanks

comment:3 by Andreas Schäfer <gentryx@…>, 13 years ago

Hi,

I get the same error, but had no success with the solution proposed above. Please have a the following tar balls:

Any help on this would be highly appreciated. :-)

comment:4 by Andreas Schäfer <gentryx@…>, 13 years ago

BTW, I don't see this error with the GIT plugin, if that is of any help.

comment:5 by Christian Boos, 13 years ago

Status: newassigned

Nom that doesn't help ;-)

I'm working on a fix, but it'll take me a few more days.

by Christian Boos, 13 years ago

minimal fix - not very well tested

comment:6 by Christian Boos, 13 years ago

Just in case you'd like to do some testing, here's what I've got so far.

comment:7 by Andreas Schäfer <gentryx@…>, 13 years ago

Sorry for lagging, didn't watch the ticket as closely as I should have. Looks like the patch works just fine, many thanks. :-)

comment:8 by Christian Boos, 13 years ago

Milestone: not applicablemercurial-plugin
Priority: normalhigh
Severity: normalmajor

comment:9 by Christian Boos, 13 years ago

Summary: UnicodeDecodeError on mercurial-plugin 0.11support non-ascii filenames in mercurial vcs backend

#7058 and #7799 were closed as duplicate of this ticket.

comment:10 by richard.liao.i@…, 13 years ago

This patch not work for me, with gbk filename. I have

  • set filename_encoding = gbk
  • patch os.environHGENCODING = 'utf-8'

When browsing changeset, raise

2010-01-19 17:16:26,025 Trac[main] ERROR: 'ascii' codec can't decode byte 0xb2 in position 25: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/main.py", line 462, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/main.py", line 213, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/versioncontrol/web_ui/changeset.py", line 324, in process_request
    self._render_html(req, repos, chgset, restricted, xhr, data)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/versioncontrol/web_ui/changeset.py", line 587, in _render_html
    'new': new_node and node_info(new_node, annotated),
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/versioncontrol/web_ui/changeset.py", line 452, in node_info
    None),
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/href.py", line 163, in <lambda>
    self._derived[name] = lambda *args, **kw: self(name, *args, **kw)
  File "/usr/local/lib/python2.5/site-packages/Trac-0.12dev_r7678-py2.5.egg/trac/web/href.py", line 148, in __call__
    if arg != None])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb2 in position 25: ordinal not in range(128)

comment:11 by Christian Boos, 12 years ago

#9631 was closed as duplicate of this ticket. It also had a partial patch.

What is needed is some comprehensive from/to_hg_path conversion…

comment:12 by anonymous, 12 years ago

anything new?

now i apply this patch to TracMercurial-0.12.0.23dev_r9953-py2.6 the error is : NameError: global name 'options' is not defined

and then i changed this line: self.filename_encoding = options.get('filename_encoding') to: self.filename_encoding = 'utf-8'

and then the error is :

Traceback (most recent call last):
  File "\Python26\lib\site-packages\trac\web\api.py", line 436, in send_error
    data, 'text/html')
  File "\Python26\lib\site-packages\trac\web\chrome.py", line 861, in render_template
    location=location))
TracError: <unprintable TracError object>

i'm sure the non-ascii filenames in mercurial is in utf-8 (confirmed by hg serve ) everything is ok in hg serve web interface why so difficult to display it in tracmercurial

Last edited 12 years ago by Christian Boos (previous) (diff)

in reply to:  12 ; comment:13 by Christian Boos, 12 years ago

Priority: highhighest

Replying to anonymous:

anything new?

Thanks for the heads up, I'll try to fix this, this evening. Would you mind in the meantime provide me with a sample repository I could use for testing? (or simply the URL for cloning yours in case it's a public repo).

comment:15 by anonymous, 12 years ago

tracmercurial can list the non-ascii filenames but can not when diaplaying its content.

in reply to:  13 comment:16 by anonymous, 12 years ago

Replying to cboos:

Thanks for the heads up, I'll try to fix this, this evening.

good news. such a long time. good luck!

comment:17 by Christian Boos, 12 years ago

Please update to source:plugins/0.12/mercurial-plugin@10491 and give it a try.

I'm not closing yet as further heavy changes are expected (as the heavy number of FIXME shows…) and it's likely that there are or will be a few glitches. Please report them here as you find them.

comment:18 by anonymous, 12 years ago

it works. great job.

comment:19 by Christian Boos, 12 years ago

Resolution: fixed
Status: assignedclosed

#10000 was one such glitch.

Now with the even bigger change of r10496, I imagine there could be some more… maybe better open a new ticket in such case anyway.

As the issues with the unicode paths should be all fixed now, I'm closing this ticket.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christian Boos.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Christian Boos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.