Edgewall Software
Modify

Opened 15 years ago

Closed 14 years ago

Last modified 13 years ago

#1239 closed defect (fixed)

Milestone with national language characters don't work

Reported by: Norbert Unterberg <nepo@…> Owned by: Christopher Lenz
Priority: normal Milestone: 0.9
Component: roadmap Version: devel
Severity: normal Keywords:
Cc: nepo@… Branch:
Release Notes:
API Changes:

Description

When entering a new milestone from the Roadmap view, putting any national language character into the milestone name does not work:

  • Add New Milestone
  • Name of the milestone: Täst
  • Trac reports:
    Invalid Milestone Number
    Milestone T�st does not exist. 
    

Attachments (1)

1239.diff (858 bytes ) - added by Christopher Lenz 14 years ago.
Always transcode PATH_INFO as UTF-8

Download all attachments as: .zip

Change History (21)

comment:1 by Shun-ichi Goto <gotoh@…>, 15 years ago

I'm using Japanese characters as milestone title/contents string with HEAD version of Trac. Also it's no problem for your word "Täst". I tested clean installed trac.

In aditional, in some months ago, with old version of Trac, I experienced same trouble but I forgot that's why and how resolved it.

comment:2 by Norbert Unterberg <nepo@…>, 15 years ago

My platform is Apache 2.0.52 on a German Windows XP. Maybe it is a OS, language or configuration issue?

I tried a (hopefully) clean install of trunk (revision 1301).

comment:3 by Norbert Unterberg <nepo@…>, 14 years ago

Milestone: 0.9

Ok, in the current trunk "Täst" works, but "Teßt" still fails. It seems to be a unicode encoding issue.

Ticket #1653 seems to be a duplicate of this issue.

comment:4 by Christopher Lenz, 14 years ago

#2094 has been marked as duplicate of this ticket.

comment:5 by Christopher Lenz, 14 years ago

Both “Täst” and “Teßt” work for me (CGI, Apache, Mac OS X).

comment:6 by markus, 14 years ago

"Teßt" doesn't work for me either (CGI, IIS5, Windows 2000, PySQLite 2.0.4). Seems like this issue is platform dependent.

comment:7 by Christopher Lenz, 14 years ago

Status: newassigned

What does the URL for such milestones look like in the HTML source? Is it properly URL-encoded?

For example, Trac generates the URL /trac.cgi/milestone/Te%C3%9Ft here for a milestone named “Teßt”.

comment:8 by markus, 14 years ago

Yep, the link in the HTML source is /trac.cgi/milestone/Te%C3%9Ft.

trac.ini looks like:

Trac[main] ERROR: Milestone Teßt does not exist.
Traceback (most recent call last):
  File "D:\Programme\Python23\Lib\site-packages\trac\web\cgi_frontend.py", line 132, in run
    dispatch_request(req.path_info.replace(os.getenv('SCRIPT_NAME'),''), req, env)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\roadmap.py", line 338, in process_request
    milestone = Milestone(self.env, req.args.get('id'), db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 504, in __init__
    self._fetch(name, db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 520, in _fetch
    'Invalid Milestone Name')
TracError: Milestone Teßt does not exist.

comment:9 by Christian Boos, 14 years ago

Both “Täst” and “Teßt” work for me, with either tracd or Apache mod_python 3.1.3, Windows XP, but not for trac.cgi:

When trying to go to an already existing Teßt milestone from the Roadmap view, it will use the URL: http://cblaptop/cgi-bin/trac.cgi/milestone/Te%C3%9Ft and this triggers the following TracError: "Milestone Te�?t does not exist." (but no traceback).

Then, I delete that milestone either with tracd or mod_python, and try to recreate it with trac.cgi: same TracError (still no traceback) but the milestone has been correctly created, as it can be seen from the Roadmap view, accessed from trac.cgi.

Also, when I type directly http://cblaptop:8002/devel/milestone/Teßt in the browser, it gets replaced by http://cblaptop:8002/devel/milestone/Te%DFt and this time the TracError message is "Milestone Te?t does not exist." (also no traceback).

comment:10 by Christopher Lenz, 14 years ago

So, it seems like this problem has only been reproduced when running as CGI under Windows as of yet.

The difference between Te%C3%9Ft and Te%DFt is that the first is encoded in UTF-8 (which requires two bytes for a "ß"), while the second is encoded using some 8-bit charset, probably ISO-8859-15.

I don't understand why this doesn't work. The CGI receives the PATH_INFO in the URL-encoded form. It is then "unquoted" using Python's urllib, basically recreating the byte string. The resulting string is then simply assumed to be UTF-8 by Trac.

comment:11 by Emmanuel Blot, 14 years ago

Does the urllib deal with the actual encoding, i.e. with the "Content-Encoding" header of the client's request ? Maybe it could help if Trac were specifying the "Accept-Charset" HTTP header to the client, to ensure that the client sends the form with the expected charset ?

comment:12 by markus, 14 years ago

I also tried entering ../milestone/Teßt directly into the browser's address bar some days ago. The results are the same:

  • It gets replaced by milestone/Te%DFt
  • Trac shows me the following error:
Invalid Milestone Name
Milestone Te�t does not exist. 
  • Traceback in trac.log:
Trac[main] ERROR: Milestone Teßt does not exist.
Traceback (most recent call last):
  File "D:\Programme\Python23\Lib\site-packages\trac\web\cgi_frontend.py", line 132, in run
    dispatch_request(req.path_info.replace(os.getenv('SCRIPT_NAME'),''), req, env)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\roadmap.py", line 338, in process_request
    milestone = Milestone(self.env, req.args.get('id'), db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 504, in __init__
    self._fetch(name, db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 520, in _fetch
    'Invalid Milestone Name')
TracError: Milestone Teßt does not exist.

How can I help finding that bug?

comment:13 by Christopher Lenz, 14 years ago

eblot, there's no form involved, it's just a link. That link is an URL-encoded UTF-8 string which gets sent to the server as part of the Request URL, and then gets passed to the CGI as the content of the PATH_INFO variable. I don't think the Content-Type is actually used for parsing decoding the Request URL. For urllib it's just an URL-encoded stream of octets.

comment:14 by markus, 14 years ago

This patch makes milestones like Teßt work with trac.cgi. Not sure why, though…

  • trac/ticket/model.py

     
    510510            self.description = ''
    511511
    512512    def _fetch(self, name, db=None):
     513        from trac.util import to_utf8
    513514        if not db:
    514515            db = self.env.get_db_cnx()
    515516        cursor = db.cursor()
    516517        cursor.execute("SELECT name,due,completed,description "
    517                        "FROM milestone WHERE name=%s", (name,))
     518                       "FROM milestone WHERE name=%s", (to_utf8(name),))
    518519        row = cursor.fetchone()
    519520        if not row:
    520521            raise TracError('Milestone %s does not exist.' % name,

comment:15 by Christopher Lenz, 14 years ago

Markus, does this apply to Te%C3%9Ft or Te%DFt?

by Christopher Lenz, 14 years ago

Attachment: 1239.diff added

Always transcode PATH_INFO as UTF-8

comment:16 by Christopher Lenz, 14 years ago

Can you please test with the patch I just attached?

comment:17 by markus, 14 years ago

It applies to both Te%C3%9Ft and Te%DFt! I already tested that.

Your patch looks good. AFAICT it works with all milestones mentioned here in this ticket.

comment:18 by Christopher Lenz, 14 years ago

Resolution: fixed
Status: assignedclosed

Okay, I've applied the patch in [2417].

comment:19 by anonymous, 14 years ago

Resolution: fixed
Status: closedreopened

I have a milestone called 'Some things 100% done' and it doesn't work.

When I access the following URL http://trac/INFOinvest/milestone/Demandas%20de%20maladireta%20da%20FIRB%20100%25%20atendidas

I get

Milestone Demandas de maladireta da FIRB 100 tendidas does not exist.

It just strips off the % character from the name of the milestone!

comment:20 by Emmanuel Blot, 14 years ago

Resolution: fixed
Status: reopenedclosed

This is not the same issue.
Please fill in a new bug when the bug summary does not match the your issue.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christopher Lenz.
The resolution will be deleted. Next status will be 'reopened'.
to as closed The owner will be changed from Christopher Lenz to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.