UnicodeEncodeError in trac/web/_fcgi.py", line 1212

[gavin.kou@…]

this error caused 500 error the click the attachment in ticket.

source:/branches/1.6-stable/trac/web/_fcgi.py#L1212​

fixed the issue by added 'ignore' in encode method.

2024-01-06 10:19:57,084 Trac[main] ERROR: [127.0.0.1] Internal Server Error: <RequestWithSession "GET '/attachment/ticket/8/2023è\x92\x99æ\x96°å\x86¬ä»¤è\x90¥_代ç\x90\x86ä»·.png'">, referrer 'https://trac.foo.bar/ticket/8'
Traceback (most recent call last):
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/main.py", line 609, in dispatch_request
    dispatcher.dispatch(req)
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/main.py", line 301, in dispatch
    raise e
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/main.py", line 266, in dispatch
    req.send(output, content_type or 'text/html')
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/api.py", line 875, in send
    self._send(content, content_type, status)
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/api.py", line 1011, in _send
    self.write(content)
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/api.py", line 977, in write
    self._write(b''.join(buf))
  File "/u0/app/trac/lib64/python3.11/site-packages/trac/web/_fcgi.py", line 1212, in write
    req.stdout.write(_str_join(generator()).encode('iso-8859-1'))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 25-26: ordinal not in range(256)

[Jun Omae]

Unable to reproduce it.

1. Create new ticket
2. Attach 2023蒙新冬令营_代理价.png to the created ticket
3. Click the attachment link and download link in attachments area
   • It works fine

Please provide more information. Also, are you using non-ascii characters in username?

Request and response headers

$ curl -s -v -o /dev/null 'http://127.0.0.1:3000/trac/attachment/ticket/22/2023%E8%92%99%E6%96%B0%E5%86%AC%E4%BB%A4%E8%90%A5_%E4%BB%A3%E7%90%86%E4%BB%B7.png'
*   Trying 127.0.0.1:3000...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /trac/attachment/ticket/22/2023%E8%92%99%E6%96%B0%E5%86%AC%E4%BB%A4%E8%90%A5_%E4%BB%A3%E7%90%86%E4%BB%B7.png HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 Ok
< Date: Sun, 07 Jan 2024 00:59:55 GMT
< Server: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 SVN/1.13.0
< Cache-Control: must-revalidate
< Expires: Fri, 01 Jan 1999 00:00:00 GMT
< Set-Cookie: trac_form_token=100da4f5f95c660e3a38a3d6; HttpOnly; Path=/trac
< Set-Cookie: trac_session=0dfdef1f495a05b018e39431; expires=Sat, 06 Apr 2024 00:59:55 GMT; HttpOnly; Path=/trac
< ETag: W/"anonymous/Sun, 07 Jan 2024 00:34:17 GMT/False"
< Transfer-Encoding: chunked
< Content-Type: text/html;charset=utf-8
<
{ [7143 bytes data]
* Connection #0 to host 127.0.0.1 left intact

$ curl -s -v -o /dev/null 'http://127.0.0.1:3000/trac/raw-attachment/ticket/22/2023%E8%92%99%E6%96%B0%E5%86%AC%E4%BB%A4%E8%90%A5_%E4%BB%A3%E7%90%86%E4%BB%B7.png'
*   Trying 127.0.0.1:3000...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /trac/raw-attachment/ticket/22/2023%E8%92%99%E6%96%B0%E5%86%AC%E4%BB%A4%E8%90%A5_%E4%BB%A3%E7%90%86%E4%BB%B7.png HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 Ok
< Date: Sun, 07 Jan 2024 01:00:01 GMT
< Server: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 SVN/1.13.0
< Content-Disposition: attachment
< Set-Cookie: trac_session=365e7bac7277b25146747bea; expires=Sat, 06 Apr 2024 01:00:01 GMT; HttpOnly; Path=/trac
< ETag: W/"anonymous/Sun, 07 Jan 2024 00:34:17 GMT/False"
< Content-Length: 12134
< Last-Modified: Sun, 07 Jan 2024 00:34:17 GMT
< Content-Type: image/png; charset=iso-8859-15
<
{ [12134 bytes data]
* Connection #0 to host 127.0.0.1 left intact

[Jun Omae]

Replying to gavin.kou@…:

fixed the issue by added 'ignore' in encode method.

Instead, try the following patch:

trac/web/_fcgi.py

@@ -1209 +1209 @@
                     for header in headers:
                         yield '%s: %s\r\n' % header
                     yield '\r\n'
-                req.stdout.write(_str_join(generator()).encode('iso-8859-1'))
+                req.stdout.write(_str_join(generator()).encode('utf-8'))
 
             req.stdout.write(data)
             req.stdout.flush()

[anonymous]

thanks so much for quick fixing.

that's a attachment with chinese characters, as right as you commented in #comment:2

the trac 1.6 is running with Python 3.11.2 behind Nginx with fastcgi interface.

i have tried Python 3.5 as well , which mentioned in wiki:TracDownload#Trac16StableRelease , this UnicodeEncodeError occured too.

will verify the changing .encode('iso-8859-1', 'ignore') to .encode('utf-8').

keep posted.

[gavin.kou@…]

Replying to Jun Omae:

verified, this patch works!

thanks, so much.

$ cat -n "/u0/app/trac/lib64/python3.11/site-packages/trac/web/_fcgi.py" | grep 1212 -A 3 -B 3
  1209                      for header in headers:
  1210                          yield '%s: %s\r\n' % header
  1211                      yield '\r\n'
  1212                  #req.stdout.write(_str_join(generator()).encode('iso-8859-1','ignore'))
  1213                  req.stdout.write(_str_join(generator()).encode('utf-8'))
  1214
  1215              req.stdout.write(data)

$ 

Replying to gavin.kou@…:

fixed the issue by added 'ignore' in encode method.

Instead, try the following patch:

trac/web/_fcgi.py

@@ -1209 +1209 @@
                     for header in headers:
                         yield '%s: %s\r\n' % header
                     yield '\r\n'
-                req.stdout.write(_str_join(generator()).encode('iso-8859-1'))
+                req.stdout.write(_str_join(generator()).encode('utf-8'))
 
             req.stdout.write(data)
             req.stdout.flush()

[Jun Omae]

Okay. Found how to reproduce it.

1. Log in with username which has non-ascii characters.
2. Visit attachment page, as the result, an exception raised

tracd

The same UnicodeEncodeError is raised while sending response headers.

Traceback (most recent call last):
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 609, in dispatch_request
    dispatcher.dispatch(req)
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 301, in dispatch
    raise e
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 266, in dispatch
    req.send(output, content_type or 'text/html')
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/api.py", line 875, in send
    self._send(content, content_type, status)
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/api.py", line 1011, in _send
    self.write(content)
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/api.py", line 977, in write
    self._write(b''.join(buf))
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/wsgi.py", line 259, in _write
    self.handler.send_header(name, value)
  File "/usr/lib/python3.11/http/server.py", line 526, in send_header
    ("%s: %s\r\n" % (keyword, value)).encode('latin-1', 'strict'))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 9-11: ordinal not in range(256)

mod_wsgi

The mod_wsgi raises ValueError if the response headers have non latin-1 characters.

Traceback (most recent call last):
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 609, in dispatch_request
    dispatcher.dispatch(req)
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 301, in dispatch
    raise e
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 266, in dispatch
    req.send(output, content_type or 'text/html')
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/api.py", line 875, in send
    self._send(content, content_type, status)
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/api.py", line 1008, in _send
    self.end_headers(exc_info)
  File "/venv/trac/1.6/lib/python3.11/site-packages/trac/web/api.py", line 799, in end_headers
    self._write = self._start_response(self._status, self._outheaders,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: unicode object contains non latin-1 characters

[Jun Omae]

Proposed changes in [a7e892a08/jomae.git]. After the changes, only ascii characters for the ETag header are used even if logged in with username contains non-ascii characters.

[Jun Omae]

Also, I consider that it might be good to check the status and headers do not contain latin-1 characters in WSGIGateway._start_response like mod_wsgi.

trac/web/wsgi.py

@@ -16 +16 @@
 
 from abc import ABCMeta, abstractmethod
 import errno
+import re
 import sys
 from http.server import HTTPServer, BaseHTTPRequestHandler
 from socketserver import ThreadingMixIn
@@ -131 +132 @@
 
     def _start_response(self, status, headers, exc_info=None):
         """Callback for starting a HTTP response."""
+        ctrl_re = re.compile(r'[\x00-\x1f\x7f]')
+
+        def check_header(item, label):
+            if not isinstance(item, str):
+                raise TypeError('Expected str instance in %s' % label)
+            try:
+                item.encode('latin-1')
+            except UnicodeEncodeError:
+                raise ValueError('Non latin-1 characters are used in %s' %
+                                 label) from None
+            if ctrl_re.search(item):
+                raise ValueError('Control characters are used in %s' % label)
+
+        check_header(status, 'status')
+        for name, value in headers:
+            check_header(name, 'headers')
+            check_header(value, 'headers')
+
         if exc_info:
             try:
                 if self.headers_sent: # Re-raise original exception

[Jun Omae]

Fixed in [17786:17787] and merged in [17788].