Some Unicode characters causing errors

[emeitner@…]

Creating ticket with Unicode emoji character in description results in:

 Trac detected an internal error:

DataError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8E\\x85' for column `trac15`.`ticket_change`.`newvalue` at row 1")

Python Traceback

Most recent call last:

File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/web/main.py", line 610, in dispatch_request          
  dispatcher.dispatch(req)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/web/main.py", line 302, in dispatch          
  raise e
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/web/main.py", line 248, in dispatch          
  resp = chosen_handler.process_request(req)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/ticket/web_ui.py", line 162, in process_request          
  return self._process_newticket_request(req)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/ticket/web_ui.py", line 513, in _process_newticket_request          
  self._do_create(req, ticket, action)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/ticket/web_ui.py", line 1333, in _do_create          
  ticket.insert()
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/ticket/model.py", line 305, in insert          
  cursor.execute("INSERT INTO ticket (%s) VALUES (%s)"
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/db/util.py", line 73, in execute          
  return self.cursor.execute(sql_escape_percent(sql), args)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/trac/db/mysql_backend.py", line 50, in execute          
  return super().execute(query, args)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/cursors.py", line 153, in execute          
  result = self._query(query)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/cursors.py", line 322, in _query          
  conn.query(q)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/connections.py", line 558, in query          
  self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/connections.py", line 822, in _read_query_result          
  result.read()
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/connections.py", line 1200, in read          
  first_packet = self.connection._read_packet()
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/connections.py", line 772, in _read_packet          
  packet.raise_for_error()
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/protocol.py", line 221, in raise_for_error          
  err.raise_mysql_exception(self._data)
File "/var/www/trac1.5/venv/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception          
  raise errorclass(errno, errval)

System Information

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0

Package	Version
Trac 	1.5.4
Jinja2 	3.1.2
mod_wsgi 	4.6.8 (WSGIProcessGroup trac WSGIApplicationGroup %{GLOBAL})
MySQL 	server: "5.5.5-10.3.38-MariaDB-0ubuntu0.20.04.1-log", client: "1.4.6", thread-safe: True
pymysql 	1.1.0
Python 	3.8.10 (default, May 26 2023, 14:05:08) [GCC 9.4.0]
setuptools 	44.0.0
jQuery	3.6.0
jQuery UI	1.13.0
jQuery Timepicker	1.6.3

Installed Plugins

Name 	Version 	Location
autocompleteusers 	N/A 	/var/www/trac1.5/proj/plugins/autocompleteusers.py
CustomNotifications 	N/A 	/var/www/trac1.5/proj/plugins/CustomNotifications.py
UserStats 	1.0 (r16991 ) 	/var/www/trac1.5/proj/plugins/UserStats.py 

The database encoding/collation seems to be correct:

SELECT DEFAULT_CHARACTER_SET_NAME,DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA 
WHERE schema_name = "trac15"

Returns: utf8, utf8_bin

SELECT TABLE_COLLATION
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = "trac15" and TABLE_NAME='ticket';

Returns: utf8_bin

[Jun Omae]

That is not a defect. You're using \xF0\x9F\x8E\x85 (UTF-8 4-bytes character, U+1F385) for utf8 database. You should use utf8mb4 and utf8mb4_bin.