UTF-8 encoding in SVN trouble

[jerome@…]

Hi,

I get the followimg errors messages while trying to pointing trac to my SVN repository : Traceback (most recent call last):

 File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 379, in dispatch_request
   dispatcher.dispatch(req)
 File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 230, in dispatch
   resp = chosen_handler.process_request(req)
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 131, in process_request
   self._render_directory(req, repos, node, rev)
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 156, in _render_directory
   changes = get_changes(self.env, repos, [i['rev'] for i in info])
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/util.py", line 37, in get_changes
   changeset = repos.get_changeset(rev)
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 41, in get_changeset
   self.sync()
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 95, in sync
   "VALUES (%s,%s,%s,%s)", (str(current_rev),
 File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute
   return self.cursor.execute(sql_escape_percent(sql), args)
 File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute
   return self.cursor.execute(sql_escape_percent(sql), args)
ProgrammingError: invalid byte sequence for encoding "UTF8": 0x80

I believe the issue is related to some character encoding issues within something in the repository. I've also noticed that when the error occurs, a transaction is held open to the database. That causes intermittent other errors when hitting other pages after someone has hit the browse page.

Any ideas about that ? Cheers

[Christian Boos]

Please upgrade to latest source:branches/0.10-stable​, or at least apply the r4235 diff.

This will solve the intermittent other errors and if you uncomment the print statements added in that changeset, we'll be able to see the problematic bytes…

[anonymous]

Hum OK. Actually, the problem is now solved. I've installed an SQLite-based Trac instance in order to test my SVN repository on it (I was using Postgresql install) and it's working very well like that. Then I'm going to use my SQLite-based instance.

Thanks for your comment. By the way, I'm discovering Trac and I find it really great ! Thanks a lot.

[Christian Boos]

So it's a PostgreSQL issue… switching to SQLite doesn't mean the problem is fixed, it works for you and that's great but we should nevertheless try to fix the issue if we can ;)

It would be very helpful on your part to apply the r4235 patch, uncomment the print statements and drop the output here… after that of course, you're free to go on with SQLite!

[Jonas Borgström]

Besides testing the things cboos suggested above it would also help if you could say which postgresql python module you used (psycopg2 or pypgsql).

[jerome@…]

Hi,

Well, I've updated db/util.py and uncommented the print comment lines.

Traceback (most recent call last):

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 379, in dispatch_request

dispatcher.dispatch(req)

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 230, in dispatch

resp = chosen_handler.process_request(req)

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 131, in process_request

self._render_directory(req, repos, node, rev)

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 156, in _render_directory

changes = get_changes(self.env, repos, [irev for i in info])

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/util.py", line 37, in get_changes

changeset = repos.get_changeset(rev)

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 41, in get_changeset

self.sync()

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 95, in sync

"VALUES (%s,%s,%s,%s)", (str(current_rev),

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute

print 'execute', repr(sql)

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute

print 'execute', repr(sql)

ProgrammingError: invalid byte sequence for encoding "UTF8": 0x80

Notice that when I refresh the page or go to other pages, I get an other error message :

Traceback (most recent call last):

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 379, in dispatch_request

dispatcher.dispatch(req)

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 190, in dispatch

req.perm = PermissionCache(self.env, req.authname)

File "/usr/lib/python2.3/site-packages/trac/perm.py", line 267, in init

self.perms = PermissionSystem(env).get_user_permissions(username)

File "/usr/lib/python2.3/site-packages/trac/perm.py", line 231, in get_user_permissions

for perm in self.store.get_user_permissions(username):

File "/usr/lib/python2.3/site-packages/trac/perm.py", line 111, in get_user_permissions

cursor.execute("SELECT username,action FROM permission")

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 48, in execute

if args:

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 48, in execute

if args:

ProgrammingError: current transaction is aborted, commands ignored until end of transaction block

To answer to jonas, note I'm using psycopg2.

Hope it could help.

[Christian Boos]

Thanks! Some quick notes:

For the first error: it seems really weird that the SQL query string from source:tags/trac-0.10.2/trac/versioncontrol/cache.py@#L95​ would trigger an unicode error, plus, how could that trigger a Programming exception? Something seems to be wrong in the stacktrace itself.

For the second error, perhaps we should do something similar at the postrgresql level than what we do at the pysqlite level, i.e. do a rollback on the connection after an exception.

[Christian Boos]

Starting a new collection of bugs ;)

[anonymous]

Hi everyone … I have been looking into this, too, Using version 10.1 and pypgsql. It looks like trac is using unicode for all of its strings, but my postres is utf-8 so this clashes. However, I am not sure how to fix this.

[torsten]

Hi again I have done some more reseach and found that svn_fs relies on getting UTF-8 strings for the log-message. So for a re-sync of my svn-repository I go the above error-message. Resolution is shown in #4321

[Christian Boos]

Great news, thanks!

This explanation also seems likely to apply to the original report, so I agree with the duplicate resolution.

[sid]

#5655 was marked as duplicate of this ticket.