Error when login name has non-ascii characters

[kloining@…]

"Everything" is set to UTF-8, but trac returns error error when login name has non-ascii characters.

raceback (most recent call last):
  File "/var/lib/python-support/python2.4/trac/web/main.py", line 406, in dispatch_request
    dispatcher.dispatch(req)
  File "/var/lib/python-support/python2.4/trac/web/main.py", line 183, in dispatch
    req.perm = PermissionCache(self.env, req.authname)
  File "/var/lib/python-support/python2.4/trac/perm.py", line 263, in __init__
    self.perms = PermissionSystem(env).get_user_permissions(username)
  File "/var/lib/python-support/python2.4/trac/perm.py", line 227, in get_user_permissions
    for perm in self.store.get_user_permissions(username):
  File "/var/lib/python-support/python2.4/trac/perm.py", line 117, in get_user_permissions
    if user in subjects:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)

[Christian Boos]

• What version of Trac are you using?
• Which web front-end do you use?

[kloining@…]

I'm using version 0.10.4 "from" Debian Lenny(testing).

Is this the thing you mean about web front-end?: default_handler = WikiModule I've changed nothing since install.

[Emmanuel Blot]

Replying to kloining@powadezign.no:

Is this the thing you mean about web front-end?:

One of the following:

• mod_python
• tracd
• cgi /fastcgi

[Emmanuel Blot]

I meant, one among the following web front-ends

[Kristian Løining <kloining@…>]

Replying to eblot:

I meant, one among the following web front-ends

Ah. I first started with CGI, and are now using mod_python. Both returned same error.

[hans@…]

I just want to confirm this.

I'm using Trac 0.10.4, Apache 2.2.8, mod_python

Apache uses authnz_ldap to authenticate users from a w2k3 active directory server. The login-username gets replaced by the real name of the user, by using the ​AuthLDAPRemoteUserAttribute displayName option in authnz_ldap. (That's really nice feature by the way!)

Trac breaks with the same error/traceback as the OP posted above, when the real name contains non-7bits-ascii characters.

[osimons]

#7714 closed as duplicate.

[Christian Boos]

#7325 closed as duplicate. There it mentions that the problem happens with tracd on windows, so it should be pretty easy for me to reproduce ;-)

[Christian Boos]

… as a reminder that I should try to reproduce/fix this.

[Christian Boos]

So yes, it's somewhat reproducible. For me however, it broke in a slightly different way.

The first error was:

8:08:18 PM Trac[main] ERROR: Internal Server Error:
Traceback (most recent call last):
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\main.py", line 435, in _dispatch_request
    dispatcher.dispatch(req)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\main.py", line 205, in dispatch
    resp = chosen_handler.process_request(req)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\auth.py", line 101, in process_request
    self._do_login(req)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\auth.py", line 141, in _do_login
    req.remote_addr, int(time.time())))
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\db\util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\db\sqlite_backend.py", line 58, in execute
    args or [])
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\db\sqlite_backend.py", line 50, in _rollback_on_error
    return function(self, *args, **kwargs)
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings
(like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

(found in the log)

Then, when trying to render the above in the error page, a secondary error was triggered, shown as a text page in the browser:

Traceback (most recent call last):
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\api.py", line 367, in send_error
    'text/html')
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\chrome.py", line 728, in render_template
    if not req.session or not int(req.session.get('accesskeys', 0)):
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\api.py", line 194, in __getattr__
    value = self.callbacks[name](self)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\main.py", line 267, in _get_session
    return Session(self.env, req)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\session.py", line 156, in __init__
    self.promote_session(sid)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\web\session.py", line 214, in promote_session
    "WHERE sid=%s OR sid=%s ", (sid, self.req.authname))
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\db\util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\db\sqlite_backend.py", line 58, in execute
    args or [])
  File "C:\Workspace\src\trac\repos\0.11-stable\trac\db\sqlite_backend.py", line 50, in _rollback_on_error
    return function(self, *args, **kwargs)
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Off-Topic Note: what happens in the above is a bit awkward, we're trying to promote and save a session even here, in the case of an internal error.

But anyway, this boils down to the fact that throughout, req.authname is a str, not an unicode object. All is fine when it's only ASCII, of course.

Now, the questions:

• I've typed in the browser testé as the username, but Trac got that as 'test\xe9', i.e. latin1 encoded. I've no idea what decides this choice of encoding, hints welcomed.
• On the server-side (tracd here), we're loading the password files as byte streams as well (for basic and htdigest authentifaction). So for the login to succeed, the username had to be latin1 encoded in that file as well.

So it looks like there are several hurdles on the way… to be continued.

[tdwebste@…]

Error resync to a git repository using containing non-ascii file names

A related problem with trac-admin /path rsync. This is blocker for me. I am quite happy to accept a work around until after this problem is resolved. I have 6 repos with ascii encoded filenames which work fine. Unfortunately my main repo does include some unicoded filenames.

The problem appears with trac.ini containing cached_repository = true

changing cached_repository = false

allows the wiki pages to be accessable.

When the trac reports.

 Trac detected an internal error:

ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

resync also fails

trac-admin /var/trac/genist resync
Resyncing repository history... 
Traceback (most recent call last):
  File "/usr/bin/trac-admin", line 8, in <module>
    load_entry_point('Trac==0.11.1', 'console_scripts', 'trac-admin')()
  File "/usr/lib/python2.5/site-packages/trac/admin/console.py", line 1295, in run
    return admin.onecmd(command)
  File "/usr/lib/python2.5/site-packages/trac/admin/console.py", line 123, in onecmd
    rv = cmd.Cmd.onecmd(self, line) or 0
  File "/usr/lib/python2.5/cmd.py", line 219, in onecmd
    return func(arg)
  File "/usr/lib/python2.5/site-packages/trac/admin/console.py", line 665, in do_resync
    repos = env.get_repository().sync(self._resync_feedback)
  File "/usr/lib/python2.5/site-packages/trac/versioncontrol/cache.py", line 211, in sync
    path, kind, action, bpath, brev))
  File "/usr/lib/python2.5/site-packages/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/lib/python2.5/site-packages/trac/db/sqlite_backend.py", line 58, in execute
    args or [])
  File "/usr/lib/python2.5/site-packages/trac/db/sqlite_backend.py", line 50, in _rollback_on_error
    return function(self, *args, **kwargs)
pysqlite2.dbapi2.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

[Christian Boos]

Replying to tdwebste@…:

Error resync to a git repository using containing non-ascii file names

This has nothing to do with this ticket. Please report your issue to the ​TracHacks:GitPlugin maintainer.

[Christian Boos]

#7287 was closed as duplicate.

[anonymous]

I have plain ASCII names of allowed users in htpasswd (I use apache + python module install of Trac, with SVN access).

And yet, in few places, Trac fails with:

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/trac/web/api.py", line 339, in send_error
    'text/html')
  File "/usr/lib/python2.5/site-packages/trac/web/chrome.py", line 715, in render_template
    return stream.render(method, doctype=doctype)
  File "/var/lib/python-support/python2.5/genshi/core.py", line 179, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "/var/lib/python-support/python2.5/genshi/output.py", line 60, in encode
    return _encode(u''.join(list(iterator)))
  File "/var/lib/python-support/python2.5/genshi/output.py", line 311, in __call__
    for kind, data, pos in stream:
  File "/var/lib/python-support/python2.5/genshi/output.py", line 753, in __call__
    for kind, data, pos in stream:
  File "/var/lib/python-support/python2.5/genshi/output.py", line 592, in __call__
    for kind, data, pos in stream:
  File "/var/lib/python-support/python2.5/genshi/output.py", line 707, in __call__
    text = mjoin(textbuf, escape_quotes=False)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 646: ordinal not in range(128)

So far I see this in administration, when I try to:

• set default Milestone (but del/add works)
• set default Component (but del/add works)
• save logging settings

[Christian Boos]

Milestone 1.0 deleted

[carl@…]

I've been able to reproduce same issue using trac 0.12.1 and with agilo plugin (only noticed in their team planning for sprint view):

ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.


File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/web/main.py", line 511, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/web/main.py", line 237, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/api/view.py", line 155, in process_request
  return self._call_filters_and_handler(req, handler)
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/api/view.py", line 143, in _call_filters_and_handler
  data = handler(req) or {}
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/scrum/team/web_ui.py", line 212, in do_post
  tmc = self.controller.tmm_manager.get(name=m).calendar
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/core/model.py", line 1128, in get
  m = self.model(self.env, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/scrum/team/model.py", line 301, in __init__
  super(TeamMember, self).__init__(env, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/core/model.py", line 524, in __init__
  self._load(db=db)
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/core/model.py", line 708, in _load
  safe_execute(cursor, self.SQL_S + filter, keys)
File "/usr/local/lib/python2.6/dist-packages/binary_agilo-1.3.2_PRO-py2.6.egg/agilo/core/model.py", line 118, in safe_execute
  return cursor.execute(sql, args)
File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/util.py", line 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/sqlite_backend.py", line 78, in execute
  result = PyFormatCursor.execute(self, *args)
File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/sqlite_backend.py", line 56, in execute
  args or [])
File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
  return function(self, *args, **kwargs)

I've not investigate it to decide if the error is related to agilo or trac though.

[Christian Boos]

It's a known issue, belonging to Trac. As the milestone says, we currently don't have any plan to fix this (a.k.a. PatchWelcome, though I suspect it's a tough one - it has to cover all kinds of authentication, all kinds of web front-ends, etc.).

[Christian Boos]

#8425 was closed as duplicate.

The fix might well be a simple matter of using a server side setting ([trac] authname_encoding, defaulting to "utf-8") specifying the encoding that is used by the web front end, and then require the administrator to be consistent with this when saving authentication information (htdigest files etc.).

[anonymous]

Setting authname_encoding = utf-8 in [trac] does not help.

[Christian Boos]

Sorry if that was not clear, comment:20 was only a suggestion about how to implement the fix (and that would probably not be all there's to it). So the issue is still open and there's not yet an authname_encoding setting. PatchWelcome!

[123kash@…]

Patch that force using unicode for user names

[Jun Omae]

#10935 was closed as duplicate.

[Christian Boos]

Replying to jomae:

#10935 was closed as duplicate.

… and claims it's a one line diff. So we might as well try it one of these days (i.e. a simple to_unicode()).

[Christian Boos]

Should be fixed by r11752.

It was a bit more involved than just a one liner, especially the functional test part.

Also, in real life conditions, whether or not this feature will actually work now depends on a large part on external factors: which browser is used, which authentication method (and how the authentication data is saved), which web server, etc.

For example in my tests with tracd and digest auth, I managed to make it work by saving the username in utf-8 and computing the hash on that, but only with Chrome and Opera, not with Safari, FF or IE10. IE10 sends back the username in latin1, so no wonder, but FF sends it back in utf-8 so the error must be elsewhere in that case (well, it's ​known not to work anyway). And even for Chrome, though it "workedforme", there's still ​an open issue about this, so YMMV.

I think we did what we could at the Trac level.

[Christian Boos]

… and closing.

[Christian Boos]

… and reopening

Traceback (most recent call last):
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/web/api.py", line 514, in send_error
    data, 'text/html')
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/web/chrome.py", line 976, in render_template
    message = Markup(req.session.pop('chrome.%s.%d'
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/web/api.py", line 316, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/web/main.py", line 268, in _get_session
    return Session(self.env, req)
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/web/session.py", line 210, in __init__
    self.promote_session(sid)
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/web/session.py", line 269, in promote_session
    (sid, self.req.authname))]
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/db/util.py", line 121, in execute
    cursor.execute(query, params)
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/db/util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/db/sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/db/sqlite_backend.py", line 56, in execute
    args or [])
  File "/usr/local/virtualenv/1.1dev/lib/python2.6/site-packages/Trac-1.1.2dev_r11755-py2.6.egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

[Christian Boos]

Ah, ah, stupid me, I didn't merge yet on trunk ;-)

Done so in r11756.

[Jun Omae]

#11716 was closed as a duplicate.

[lars@…]

Running Trac 1.0.1 on Debian Wheezy, I'm still bitten by this ancient problem. I would like to add Subversion and Trac to an existing Drupal site with more than 2.500 users. Subversion works fine, but Trac crashes for all users having Danish characters in their names - like myself.

I cannot ask 2.500+ users to change their usernames just to satisfy Trac, so I will have to dump Trac if a solution cannot be found.

Regards, Lars

[Ryan J Ollos]

Most likely you have an InstallationIssue. Please ask on the MailingList and include the following info:

• Errors from the "crash" obtained from TracLogging.
• Describe what "crash" means (traceback in browser window?).
• List any plugins you have installed such as ​th:AccountManagerPlugin.
• Describe how you've configured authentication.

[Robbie <robertvehse@…>]

Replying to lars@…:

Running Trac 1.0.1 […]

The milestone of this ticket would suggest the issue was fixed with version 1.0.2 so why not update?

[Ryan J Ollos]

Replying to Robbie <robertvehse@…>:

Replying to lars@…:

Running Trac 1.0.1 […]

The milestone of this ticket would suggest the issue was fixed with version 1.0.2 so why not update?

Good catch. I completely overlooked the milestone of this ticket!