Edgewall Software
Modify

Opened 16 years ago

Closed 16 years ago

Last modified 15 years ago

#4128 closed defect (duplicate)

UTF-8 encoding in SVN trouble

Reported by: jerome@… Owned by: Christian Boos
Priority: normal Milestone:
Component: general Version: 0.10
Severity: normal Keywords: svn unicode
Cc: Branch:
Release Notes:
API Changes:
Internal Changes:

Description (last modified by Emmanuel Blot)

Hi,

I get the followimg errors messages while trying to pointing trac to my SVN repository : Traceback (most recent call last):

 File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 379, in dispatch_request
   dispatcher.dispatch(req)
 File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 230, in dispatch
   resp = chosen_handler.process_request(req)
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 131, in process_request
   self._render_directory(req, repos, node, rev)
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 156, in _render_directory
   changes = get_changes(self.env, repos, [i['rev'] for i in info])
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/util.py", line 37, in get_changes
   changeset = repos.get_changeset(rev)
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 41, in get_changeset
   self.sync()
 File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 95, in sync
   "VALUES (%s,%s,%s,%s)", (str(current_rev),
 File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute
   return self.cursor.execute(sql_escape_percent(sql), args)
 File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute
   return self.cursor.execute(sql_escape_percent(sql), args)
ProgrammingError: invalid byte sequence for encoding "UTF8": 0x80

I believe the issue is related to some character encoding issues within something in the repository. I've also noticed that when the error occurs, a transaction is held open to the database. That causes intermittent other errors when hitting other pages after someone has hit the browse page.

Any ideas about that ? Cheers

Attachments (0)

Change History (12)

comment:1 by Emmanuel Blot, 16 years ago

Description: modified (diff)

comment:2 by Christian Boos, 16 years ago

Keywords: svn unicode needinfo added
Milestone: 0.10.2
Owner: changed from Jonas Borgström to Christian Boos
Priority: normalhigh

Please upgrade to latest source:branches/0.10-stable, or at least apply the r4235 diff.

This will solve the intermittent other errors and if you uncomment the print statements added in that changeset, we'll be able to see the problematic bytes…

comment:3 by anonymous, 16 years ago

Resolution: fixed
Status: newclosed

Hum OK. Actually, the problem is now solved. I've installed an SQLite-based Trac instance in order to test my SVN repository on it (I was using Postgresql install) and it's working very well like that. Then I'm going to use my SQLite-based instance.

Thanks for your comment. By the way, I'm discovering Trac and I find it really great ! Thanks a lot.

comment:4 by Christian Boos, 16 years ago

Priority: highnormal
Resolution: fixed
Status: closedreopened

So it's a PostgreSQL issue… switching to SQLite doesn't mean the problem is fixed, it works for you and that's great but we should nevertheless try to fix the issue if we can ;)

It would be very helpful on your part to apply the r4235 patch, uncomment the print statements and drop the output here… after that of course, you're free to go on with SQLite!

comment:5 by Jonas Borgström, 16 years ago

Besides testing the things cboos suggested above it would also help if you could say which postgresql python module you used (psycopg2 or pypgsql).

comment:6 by jerome@…, 16 years ago

Hi,

Well, I've updated db/util.py and uncommented the print comment lines.

Traceback (most recent call last):

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 379, in dispatch_request

dispatcher.dispatch(req)

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 230, in dispatch

resp = chosen_handler.process_request(req)

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 131, in process_request

self._render_directory(req, repos, node, rev)

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/browser.py", line 156, in _render_directory

changes = get_changes(self.env, repos, [irev for i in info])

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/util.py", line 37, in get_changes

changeset = repos.get_changeset(rev)

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 41, in get_changeset

self.sync()

File "/usr/lib/python2.3/site-packages/trac/versioncontrol/cache.py", line 95, in sync

"VALUES (%s,%s,%s,%s)", (str(current_rev),

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute

print 'execute', repr(sql)

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute

print 'execute', repr(sql)

ProgrammingError: invalid byte sequence for encoding "UTF8": 0x80

Notice that when I refresh the page or go to other pages, I get an other error message :

Traceback (most recent call last):

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 379, in dispatch_request

dispatcher.dispatch(req)

File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 190, in dispatch

req.perm = PermissionCache(self.env, req.authname)

File "/usr/lib/python2.3/site-packages/trac/perm.py", line 267, in init

self.perms = PermissionSystem(env).get_user_permissions(username)

File "/usr/lib/python2.3/site-packages/trac/perm.py", line 231, in get_user_permissions

for perm in self.store.get_user_permissions(username):

File "/usr/lib/python2.3/site-packages/trac/perm.py", line 111, in get_user_permissions

cursor.execute("SELECT username,action FROM permission")

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 48, in execute

if args:

File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 48, in execute

if args:

ProgrammingError: current transaction is aborted, commands ignored until end of transaction block

To answer to jonas, note I'm using psycopg2.

Hope it could help.

comment:7 by Christian Boos, 16 years ago

Thanks! Some quick notes:

For the first error: it seems really weird that the SQL query string from source:tags/trac-0.10.2/trac/versioncontrol/cache.py@#L95 would trigger an unicode error, plus, how could that trigger a Programming exception? Something seems to be wrong in the stacktrace itself.

For the second error, perhaps we should do something similar at the postrgresql level than what we do at the pysqlite level, i.e. do a rollback on the connection after an exception.

comment:8 by Christian Boos, 16 years ago

Keywords: weird added

Starting a new collection of bugs ;)

comment:9 by anonymous, 16 years ago

Hi everyone … I have been looking into this, too, Using version 10.1 and pypgsql. It looks like trac is using unicode for all of its strings, but my postres is utf-8 so this clashes. However, I am not sure how to fix this.

comment:10 by torsten, 16 years ago

Resolution: duplicate
Status: reopenedclosed

Hi again I have done some more reseach and found that svn_fs relies on getting UTF-8 strings for the log-message. So for a re-sync of my svn-repository I go the above error-message. Resolution is shown in #4321

comment:11 by Christian Boos, 16 years ago

Keywords: weird needinfo removed
Milestone: 0.10.4

Great news, thanks!

This explanation also seems likely to apply to the original report, so I agree with the duplicate resolution.

comment:12 by sid, 15 years ago

#5655 was marked as duplicate of this ticket.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christian Boos.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Christian Boos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.