Milestone with national language characters don't work

[Norbert Unterberg <nepo@…>]

When entering a new milestone from the Roadmap view, putting any national language character into the milestone name does not work:

• Add New Milestone
• Name of the milestone: Täst
• Trac reports:

  Invalid Milestone Number
  Milestone T�st does not exist. 
  

[Shun-ichi Goto <gotoh@…>]

I'm using Japanese characters as milestone title/contents string with HEAD version of Trac. Also it's no problem for your word "Täst". I tested clean installed trac.

In aditional, in some months ago, with old version of Trac, I experienced same trouble but I forgot that's why and how resolved it.

[Norbert Unterberg <nepo@…>]

My platform is Apache 2.0.52 on a German Windows XP. Maybe it is a OS, language or configuration issue?

I tried a (hopefully) clean install of trunk (revision 1301).

[Norbert Unterberg <nepo@…>]

Ok, in the current trunk "Täst" works, but "Teßt" still fails. It seems to be a unicode encoding issue.

Ticket #1653 seems to be a duplicate of this issue.

[cmlenz]

#2094 has been marked as duplicate of this ticket.

[cmlenz]

Both “Täst” and “Teßt” work for me (CGI, Apache, Mac OS X).

[markus]

"Teßt" doesn't work for me either (CGI, IIS5, Windows 2000, PySQLite 2.0.4). Seems like this issue is platform dependent.

[cmlenz]

What does the URL for such milestones look like in the HTML source? Is it properly URL-encoded?

For example, Trac generates the URL /trac.cgi/milestone/Te%C3%9Ft here for a milestone named “Teßt”.

[markus]

Yep, the link in the HTML source is /trac.cgi/milestone/Te%C3%9Ft.

trac.ini looks like:

Trac[main] ERROR: Milestone Teßt does not exist.
Traceback (most recent call last):
  File "D:\Programme\Python23\Lib\site-packages\trac\web\cgi_frontend.py", line 132, in run
    dispatch_request(req.path_info.replace(os.getenv('SCRIPT_NAME'),''), req, env)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\roadmap.py", line 338, in process_request
    milestone = Milestone(self.env, req.args.get('id'), db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 504, in __init__
    self._fetch(name, db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 520, in _fetch
    'Invalid Milestone Name')
TracError: Milestone Teßt does not exist.

[cboos]

Both “Täst” and “Teßt” work for me, with either tracd or Apache mod_python 3.1.3, Windows XP, but not for trac.cgi:

When trying to go to an already existing Teßt milestone from the Roadmap view, it will use the URL: http://cblaptop/cgi-bin/trac.cgi/milestone/Te%C3%9Ft and this triggers the following TracError: "Milestone Te�?t does not exist." (but no traceback).

Then, I delete that milestone either with tracd or mod_python, and try to recreate it with trac.cgi: same TracError (still no traceback) but the milestone has been correctly created, as it can be seen from the Roadmap view, accessed from trac.cgi.

Also, when I type directly http://cblaptop:8002/devel/milestone/Teßt in the browser, it gets replaced by http://cblaptop:8002/devel/milestone/Te%DFt and this time the TracError message is "Milestone Te?t does not exist." (also no traceback).

[cmlenz]

So, it seems like this problem has only been reproduced when running as CGI under Windows as of yet.

The difference between Te%C3%9Ft and Te%DFt is that the first is encoded in UTF-8 (which requires two bytes for a "ß"), while the second is encoded using some 8-bit charset, probably ISO-8859-15.

I don't understand why this doesn't work. The CGI receives the PATH_INFO in the URL-encoded form. It is then "unquoted" using Python's urllib, basically recreating the byte string. The resulting string is then simply assumed to be UTF-8 by Trac.

[eblot]

Does the urllib deal with the actual encoding, i.e. with the "Content-Encoding" header of the client's request ? Maybe it could help if Trac were specifying the "Accept-Charset" HTTP header to the client, to ensure that the client sends the form with the expected charset ?

[markus]

I also tried entering ../milestone/Teßt directly into the browser's address bar some days ago. The results are the same:

- It gets replaced by milestone/Te%DFt
- Trac shows me the following error:

Invalid Milestone Name
Milestone Te�t does not exist. 

- Traceback in trac.log:

Trac[main] ERROR: Milestone Teßt does not exist.
Traceback (most recent call last):
  File "D:\Programme\Python23\Lib\site-packages\trac\web\cgi_frontend.py", line 132, in run
    dispatch_request(req.path_info.replace(os.getenv('SCRIPT_NAME'),''), req, env)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\web\main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\roadmap.py", line 338, in process_request
    milestone = Milestone(self.env, req.args.get('id'), db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 504, in __init__
    self._fetch(name, db)
  File "D:\Programme\Python23\Lib\site-packages\trac\ticket\model.py", line 520, in _fetch
    'Invalid Milestone Name')
TracError: Milestone Teßt does not exist.

How can I help finding that bug?

[cmlenz]

eblot, there's no form involved, it's just a link. That link is an URL-encoded UTF-8 string which gets sent to the server as part of the Request URL, and then gets passed to the CGI as the content of the PATH_INFO variable. I don't think the Content-Type is actually used for parsing decoding the Request URL. For urllib it's just an URL-encoded stream of octets.

[markus]

This patch makes milestones like Teßt work with trac.cgi. Not sure why, though...

trac/ticket/model.py

@@ -510 +510 @@
             self.description = ''
 
     def _fetch(self, name, db=None):
+        from trac.util import to_utf8
         if not db:
             db = self.env.get_db_cnx()
         cursor = db.cursor()
         cursor.execute("SELECT name,due,completed,description "
-                       "FROM milestone WHERE name=%s", (name,))
+                       "FROM milestone WHERE name=%s", (to_utf8(name),))
         row = cursor.fetchone()
         if not row:
             raise TracError('Milestone %s does not exist.' % name,

[cmlenz]

Markus, does this apply to Te%C3%9Ft or Te%DFt?

[cmlenz]

Always transcode PATH_INFO as UTF-8

[cmlenz]

Can you please test with the patch I just attached?

[markus]

It applies to both Te%C3%9Ft and Te%DFt! I already tested that.

Your patch looks good. AFAICT it works with all milestones mentioned here in this ticket.

[cmlenz]

Okay, I've applied the patch in [2417].

[anonymous]

I have a milestone called 'Some things 100% done' and it doesn't work.

When I access the following URL http://trac/INFOinvest/milestone/Demandas%20de%20maladireta%20da%20FIRB%20100%25%20atendidas

I get

Milestone Demandas de maladireta da FIRB 100 tendidas does not exist.

It just strips off the % character from the name of the milestone!

[eblot]

This is not the same issue.
Please fill in a new bug when the bug summary does not match the your issue.