Logging non-ascii text to file failed on running with mod_wsgi

[Jun Omae]

apache2.services typically run with C locale. The log output to the file with mod_wsgi using ascii encoding, so logging non-ascii string will fail.

The following is logged in error.log of apache2 when the error occurs:

[Tue May 21 08:38:27.959113 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] --- Logging error ---
[Tue May 21 08:38:27.970746 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] Traceback (most recent call last):
[Tue May 21 08:38:27.970949 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/home/jun66j5/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 609, in dispatch_request
[Tue May 21 08:38:27.970961 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     dispatcher.dispatch(req)
[Tue May 21 08:38:27.970970 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/home/jun66j5/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 301, in dispatch
[Tue May 21 08:38:27.970974 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     raise e
[Tue May 21 08:38:27.970981 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/home/jun66j5/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 247, in dispatch
[Tue May 21 08:38:27.970985 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     resp = chosen_handler.process_request(req)
[Tue May 21 08:38:27.970995 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Tue May 21 08:38:27.971006 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/home/jun66j5/venv/trac/1.6/lib/python3.11/site-packages/trac/admin/web_ui.py", line 83, in process_request
[Tue May 21 08:38:27.971012 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     raise HTTPNotFound(_("No administration panels available"))
[Tue May 21 08:38:27.971042 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] abc.HTTPNotFound: 404 Not Found (Keine Administrations-Seiten verf\xc3\xbcgbar)
[Tue May 21 08:38:27.971095 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]
[Tue May 21 08:38:27.971103 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] During handling of the above exception, another exception occurred:
[Tue May 21 08:38:27.971108 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]
[Tue May 21 08:38:27.971120 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] Traceback (most recent call last):
[Tue May 21 08:38:27.971175 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/usr/lib/python3.11/logging/__init__.py", line 1113, in emit
[Tue May 21 08:38:27.971184 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     stream.write(msg + self.terminator)
[Tue May 21 08:38:27.971197 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] UnicodeEncodeError: 'ascii' codec can't encode character '\\xfc' in position 123: ordinal not in range(128)
[Tue May 21 08:38:27.971211 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] Call stack:
[Tue May 21 08:38:27.971286 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/home/jun66j5/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 613, in dispatch_request
[Tue May 21 08:38:27.971294 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     _send_user_error(req, env, e)
[Tue May 21 08:38:27.971301 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]   File "/home/jun66j5/venv/trac/1.6/lib/python3.11/site-packages/trac/web/main.py", line 674, in _send_user_error
[Tue May 21 08:38:27.971307 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286]     env.log.warning('[%s] %s, %r, referrer %r',
[Tue May 21 08:38:27.971324 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] Message: '[%s] %s, %r, referrer %r'
[Tue May 21 08:38:27.971330 2024] [wsgi:error] [pid 1908845:tid 140306708121344] [remote 127.0.0.1:61286] Arguments: ('127.0.0.1', 'HTTPNotFound: 404 Not Found (Keine Administrations-Seiten verf\xc3\xbcgbar)', <RequestWithSession "GET '/admin/x'">, None)

I consider this is an issue of mod_wsgi because The file encoding is utf-8 if locale is not specified since Python 3.7 (​pep:540).

$ for i in python{2.7,3.{5,6,7,8,9,1{0,1,2,3}}}; do echo "$i: $(env - $i -c 'with open("/dev/null") as f: print(f.encoding)')"; done
python2.7: None
python3.5: ANSI_X3.4-1968
python3.6: ANSI_X3.4-1968
python3.7: UTF-8
python3.8: UTF-8
python3.9: UTF-8
python3.10: UTF-8
python3.11: utf-8
python3.12: utf-8
python3.13: utf-8

Workaround is passing locale=C.UTF-8 parameter to WSGIDaemonProcess.

Patch is to pass encoding='utf-8' parameter to FileHandler in logger_handler_factory. We expect utf-8 encoding is used to log to a file regardless of user's locale.

trac/log.py

@@ -40 +40 @@
     logger = logging.getLogger(logid)
     logtype = logtype.lower()
     if logtype == 'file':
-        hdlr = logging.FileHandler(logfile)
+        hdlr = logging.FileHandler(logfile, encoding='utf-8')
     elif logtype in ('eventlog', 'winlog', 'nteventlog'):
         # Requires win32 extensions
         hdlr = logging.handlers.NTEventLogHandler(logid,

[Jun Omae]

Fixed in [17804] and merged in [17805].