|Version 11 (modified by 14 years ago) ( diff ),|
Trac and Unicode: Development Guidelines
Since Trac 0.10, Trac uses
unicode strings internally.
This document aims at clarifying what the implications of this change are.
Unicode Mini Tutorial
In Python, they are two kinds of string types, both subclasses of
unicodeis a string type in which each character is an Unicode code point.
All common string operations (len, slicing, etc.) will operate on those code points. i.e. "real" character boundaries, in any language.
stris a string type in which each character is a byte.
The string operations will operate on those bytes, and byte boundaries don't correspond to character boundaries in many common encodings.
unicode provides a real representation of textual data: once you're in
unicode, you know that your text data can contain any kind of multilingual characters, and that you can safely manipulate it the expected way.
On the other hand, a
str object can be used to contain anything, binary data, or some text using any conceivable encoding. But if it's supposed to contain text, it is crucial to know which encoding was used. That knowledge must be known or inferred from somewhere, which is not always a trivial thing to do.
In summary, it is not manipulating
unicode object which is problematic (it is not), but how to go from the "wild" side (
str) to the "safe" side (
unicode)… Going from
str is usually less problematic, because you can always control what kind of encoding you want to use for serializing your Unicode data.
How does all the above look like in practice? Let's take an example (from ):
u"ndré Le"is an Unicode object containing the following sequence of Unicode code points:
>>> ["U-%04x" % ord(x) for x in u"ndré Le"] ['U-006e', 'U-0064', 'U-0072', 'U-00e9', 'U-0020', 'U-004c', 'U-0065']
- From there, you can easily transform that to a
As we said above, we have to freedom to choose the encoding:
- UTF-8: it's a variable length encoding which is widely understood,
and in which any code point can be represented:
>>> u"ndré Le".encode('utf-8') 'ndr\xc3\xa9 Le'
- iso-8859-15: it's a fixed length encoding, which is commonly used
in European countries. It happens that the
unicodesequence we are interested in can be mapped to a sequence of bytes in this encoding.
>>> u"ndré Le".encode('iso-8859-15') 'ndr\xe9 Le'
- ascii: it is a very "poor" encoding, as there are only 128 unicode
code points (those in the U-0000 to U-007e range) that can be mapped to
ascii. Therefore, trying to encode our sample sequence will fail,
as it contains one code point outside of this range (U-00e9).
>>> u"ndré Le".encode('ascii') Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 3: ordinal not in range(128)It should be noted that this is also the error one would get by doing a coercion to
stron that unicode object, because the system encoding is usually
>>> str(u"ndré Le") Traceback (...): # same as above >>> sys.getdefaultencoding() 'ascii'Lastly, there are ways to force a conversion to succeed, even if there's no way to encode some of the original unicode characters in the targeted charset. One possible way is to use replacement characters:
>>> u"ndré Le".encode('ascii', 'replace') 'ndr? Le'
- UTF-8: it's a variable length encoding which is widely understood, and in which any code point can be represented:
- Now, you might wonder how to get a
unicodeobject in the first place, starting from a string.
Well, from the above it should be obvious that it's absolutely necessary to know what is the encoding used in the
strobject, as either
'ndr\xc3\xa9 Le'could be decoded into the same unicode string
u"ndré Le"(as a matter of fact, it is as important as knowing if that stream of bytes has been gzipped or ROT13-ed…)
- Assuming we know the encoding of the
strobject, getting an
unicodeobject out of it is trivial:
>>> unicode('ndr\xc3\xa9 Le', 'utf-8') u'ndr\xe9 Le' >>> unicode('ndr\xe9 Le', 'iso-8859-15') u'ndr\xe9 Le'The above can be rewritten using the
>>> 'ndr\xc3\xa9 Le'.decode('utf-8') u'ndr\xe9 Le' >>> 'ndr\xe9 Le'.decode('iso-8859-15') u'ndr\xe9 Le'
- But what happens if we do a bad guess?
>>> unicode('ndr\xc3\xa9 Le', 'iso-8859-15') u'ndr\xc3\xa9 Le'No errors here, but the unicode string now contains garbage
(NB: as we have seen above, 'iso-8859-15' is a fixed-byte encoding with a mapping defined for all the 0..255 range, so decoding any input assuming such an encoding will always succeed).
>>> unicode('ndr\xe9 Le', 'utf-8') Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3-5: invalid dataHere, we clearly see that not all sequence of bytes can be interpreted as UTF-8…
- What happens if we don't provide an encoding at all?
>>> unicode('ndr\xe9 Le') Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 3: ordinal not in range(128) >>> 'ndr\xe9 Le'.decode() Traceback (...) # same as aboveThis is very symmetrical to the encoding situation: the
sys.getdefaultencoding()is used (usually 'ascii') when no encoding is explicitely given.
- Now, as with the encoding situation, there are ways to force the encoding
process to succeed, even if we are wrong about the charset used by our
- One possibility would be to use replacement characters:
>>> unicode('ndr\xe9 Le', 'utf-8', 'replace') u'ndr\ufffde'
- The other one would be to choose an encoding guaranteed to succeed (as iso-8859-1 or iso-8859-15, see above).
- One possibility would be to use replacement characters:
- Assuming we know the encoding of the
This was a very rough mini-tutorial on the question, I hope it's enough for getting in the general mood needed to read the rest of the guidelines…
Of course, there are a lot of more in-depth tutorials on Unicode in general and Python/Unicode in particular available on the Web:
-  http://www.egenix.com/files/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf
-  http://www.amk.ca/python/howto/unicode
-  http://www.python.org/dev/peps/pep-0100
Now we can move to the specifics of Trac programming…
Trac utilities for Unicode
In order to handle the unicode related issues in a cohesive way,
there are a few utility functions that can be used, but this
is mainly our swiss-army knife
to_unicode function was designed with flexibility and
robustness in mind: Calling
to_unicode() on anything should
The use cases are as follows:
- given any arbitrary object
x, one could use
to_unicode(x)as one would use
unicode(x)to convert it to an unicode string
- given a
s, which might be a text but for which we have no idea what was the encoding used, one can use
to_unicode(s)to convert it to an
unicodeobject in a safe way.
Actually, a decoding using 'utf-8' will be attempted first, and if this fails, a decoding using the
locale.getpreferredencoding()will be done, in replacement mode.
- given a
s, for which we think we know what is the encoding
encused, we can do
to_unicode(s, enc)to try to decode it using the
encencoding, in replacement mode.
A practical advantage of using
unicode(s, enc, 'replace')is that our first form will revert to the use case 2, should
So, you may ask, if the above works in all situations, where should you
- you could use
unicode(x)when you know for sure that x is anything but a
strcontaining bytes in the 128..255 range;
It should be noted that
to_unicode(x)simply does a
unicode(x)call for anything which is not a
strobject, so there's virtually no performance penalty in using
to_unicodeinstead (in particular, there's no exception handler set in this case).
unicode(buf, encoding)when you know for sure what the encoding is. You will have a performance gain here over
to_unicode, as no exception handler will be set. Of course, the downside is that you will get an
UnicodeDecodeErrorexception if your assumption was wrong. Therefore, use this if you want to catch errors in this situation.
The Mimeview component
The Mimeview component is the place where we collect some intelligence about the MIME types and charsets auto-detection.
Most of the time, when we manipulate file content, we only have partial information about the nature of the data actually contained in those files.
This is true whether the file is located in the filesystem, in a version control repository or is streamed by the web browser (file upload).
The Mimeview component tries to associate a MIME type to a file content, based on the filename or, if that's not enough on the file's content itself. During this process, the charset used by the file might be inferred as well.
The API is quite simple:
Mimeview.get_mimetype(self, filename, content)
guess the MIME type from the
filenameor eventually from the
Mimeview.get_charset(self, content, mimetype=None)
guess the charset from the
contentor from the
mimetypemight convey charset information as well)
Mimeview.to_unicode(self, content, mimetype=None, charset=None)
to_unicodeutility and eventually guess the charset if needed
Note that the Mimeview API is currently behing overhauled and will most probably change in the next releases. See #3332.
Trac boundaries for Unicode Data
Most of the time, within Trac we assume that we are manipulating
But there are places where we need to deal with raw
str objects, and therefore
we must know what to do, either when encoding to or when decoding from
Each database connector should configure its database driver
so that the
Cursor objects are able to accept and will return
unicode objects. This sometimes involve writing a wrapper class
for the original Cursor class. See for example
SQLiteUnicodeCursor, for pysqlite1.
When reading from the console, we assume the text is encoded
When writing to the console, we assume that the
should be used.
The logging API seems to handle
unicode objects just fine.
Whenever a file is read or written, some care should be taken about the content.
Usually, when writing text data, we will choose to encode it using
When reading, it is context dependent: there are situations were we know for sure
the data in the file is encoded using
We therefore usually do a
to_unicode(filecontent, 'utf-8') in these situations.
There's an additional complexity here in that the filenames are also possibly
using non-ascii characters. In Python, it should be safe to provide
objects for all the
os filesystem related functions.
More information about how Python deals with Python at system boundaries can be found here: http://kofoto.rosdahl.net/wiki/UnicodeInPython.
This is dependent on the backend.
In Subversion, there are clear rules about the pathnames used
by the SVN Bindings for Python: those should be UTF-8 encoded
unicode pathnames should be 'utf-8' encoded before
being passed to the bindings, and pathnames returned by
the bindings should be decoded using 'utf-8' before being
returned to callers of the
As noted above when talking about file contents, the node content
can contain any kind of data, including binary data and therefore
Node.get_content().read() returns a
Depending on the backend, some hints about the nature of the
content (and eventually about the charset used if the content
is text) can be given by the
The Mimeview component can be used in order to use those hints in a streamlined way.
Generating content with ClearSilver templates
The main "source" of generated text from Trac is the ClearSilver template engine.
The ClearSilver engine doesn't accept
unicode objects, so those are
converted to UTF-8 encoded
str objects just before being inserted in the "HDF"
(the data structure used by the template engine to fill in the templates).
This is done automatically by our
HDFWrapper class, so anywhere else
in the code one can safely associate unicode values to entries in
The body of those templates (the
.cs files) must also use the UTF-8 encoding.
The Web interface
The information in the
Request object (
req) is converted to
from 'UTF-8' encoded strings.
The data sent out is generally converted to 'UTF-8' as well
(like the headers), except if some charset information has
been explicitely set in the
If this is the case, that encoding is used.
Interaction with plugins
Whenever Trac gets data from plugins, it must try to cope
str objects. Those might be 0.9 pre-unicode plugins
which have not been migrated fully to 0.10 and beyond.
Feel free to correct me, ask questions, etc.; this is a Wiki. :)
Q: When dealing with plugins that weren't designed to be unicode friendly and used unicode in favour of to_unicode, what parts of the plugin should be updated, what should use to_unicode ? —JamesMills
A: There shouldn't be any reason to replace a working call to
unicode() by a call to
to_unicode(), unless you specified the encoding, like in:
ustring = unicode(data_from_trac, 'utf-8')
The above doesn't work if
data_from_trac is actually an unicode object (you'd get
TypeError: decoding Unicode is not supported).
In this case, either don't use
unicode at all (0.10 and above only plugins) or replace it by
to_unicode (0.9 and 0.10 plugins).