Edgewall Software
Modify

Opened 8 years ago

Closed 8 years ago

#10548 closed task (fixed)

Release Trac 0.12.3

Reported by: Christian Boos Owned by:
Priority: high Milestone: 0.12.3
Component: general Version: 0.12-stable
Severity: normal Keywords: release
Cc: Branch:
Release Notes:
API Changes:

Description

a.k.a. the yearly bug fix release ;-)

This task ticket is used to coordinate the finalization and testing of the next stable version of Trac, 0.12.3.

We're entering a one week period of testing for the rc1, tagged in [10948]. Packages can be downloaded from TracDownload#LatestReleaseCandidate.

If everything goes well, the release of 0.12.3 will happen the 5th of February, 2012.

Attachments (0)

Change History (22)

comment:1 by Remy Blank, 8 years ago

And a note to translators: I forgot to re-extract the messages prior to the string freeze, so there are a few new strings in rc1. Please feel free to update your translations until the final 0.12.3 release. Apologies for the mistake.

in reply to:  1 comment:2 by Mikael Relbe, 8 years ago

Replying to rblank:

And a note to translators: I forgot to re-extract the messages prior to the string freeze, so there are a few new strings in rc1. Please feel free to update your translations until the final 0.12.3 release. Apologies for the mistake.

No worries, uk-en and sv 100% done (r10950:10951)

comment:3 by Christian Boos, 8 years ago

Small glitch noticed during testing on Linux (Python 2.6, mysql 5.0.67, mysql-python 1.2.2)

test_props_format_wrap_bothsides_unicode (trac.ticket.tests.notification.NotificationTestCase) ... /home/cboos/trac/trac-0.12.3rc1/trac/db/util.py:65: Warning: Incorrect string value: '\xF0\xA0\x80\x8B\xE9\x85...' for column 'component' at row 1
  return self.cursor.execute(sql_escape_percent(sql), args)
ok

i.e. the test pass, but the warning is a bit scary, as this string doesn't seem to be UTF-8 encoded.

in reply to:  3 ; comment:4 by Jun Omae, 8 years ago

Replying to cboos:

i.e. the test pass, but the warning is a bit scary, as this string doesn't seem to be UTF-8 encoded.

I get the same warning now.

\xF0\xA0\x80\x8B (U+2000B) is valid UTF-8 bytes sequence. The same bytes exist in trac.util.tests.text.WrapTestCase.test_wrap_ambiwidth_double, the test confirms working on both UCS2/UCS4 builds of Python. But mysql 5.0 and 5.1 don't support 4-byte utf-8.

I'll replace the bytes with 3-byte utf-8 (UCS2) in test_props_format_wrap_bothsides_unicode, later.

comment:5 by Christian Boos, 8 years ago

On Windows with MariaDB 5.3 and MySQLdb 1.2.3:

  • same warning as above
  • 5 errors, all similar:
    ======================================================================
    ERROR: test_delete_all_comments (trac.ticket.tests.model.TicketCommentDeleteTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "c:\Trac\repos\0.12-stable\trac\ticket\tests\model.py", line 665, in test_delete_all_comments
        ticket.delete_change(4)
      File "c:\Trac\repos\0.12-stable\trac\ticket\model.py", line 442, in delete_change
        @self.env.with_transaction()
      File "c:\Trac\repos\0.12-stable\trac\db\api.py", line 77, in transaction_wrapper
        fn(ldb)
      File "c:\Trac\repos\0.12-stable\trac\ticket\model.py", line 504, in do_delete
        """, (self.id, self.id, self.id))
      File "c:\Trac\repos\0.12-stable\trac\db\util.py", line 65, in execute
        return self.cursor.execute(sql_escape_percent(sql), args)
      File "C:\Dev\Python272\lib\site-packages\MySQLdb\cursors.py", line 174, in execute
        self.errorhandler(self, exc, value)
      File "C:\Dev\Python272\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (1093, "You can't specify target table 'ticket' for update in FROM clause")
    

The 1093 error seems to be prevalent in most versions of MySQL, from at least 4.1 to 5.6.

There are several workarounds documented for this problem (e.g. http://stackoverflow.com/a/45498), however I just see that such a workaround was already attempted in r10803 and apparently it must work for some configurations (like in comment:3).

So it might well be a MariaDB-only issue…

in reply to:  5 comment:6 by Christian Boos, 8 years ago

Follow-up to comment:5:

On Windows with MariaDB 5.3 and MySQLdb 1.2.3:

  • same warning as above
  • 5 errors, all similar:
    ======================================================================
    ERROR: test_delete_all_comments (trac.ticket.tests.model.TicketCommentDeleteTestCase)
    ...
    OperationalError: (1093, "You can't specify target table 'ticket' for update in FROM clause")
    

So it might well be a MariaDB-only issue…

The following seems to do the trick:

  • trac/ticket/model.py

    diff -r e4b61ba781b1 trac/ticket/model.py
    a b  
    498498                    SELECT time FROM ticket_change WHERE ticket=%s
    499499                    UNION
    500500                    SELECT time FROM (
    501                         SELECT time FROM ticket WHERE id=%s) AS t
     501                        SELECT time FROM ticket WHERE id=%s LIMIT 1) AS t
    502502                    ORDER BY time DESC LIMIT 1)
    503503                WHERE id=%s
    504504                """, (self.id, self.id, self.id))

Also, the UTF-8 issue is actually not a warning in this setup, but a plain error:

OperationalError: (1366, "Incorrect string value: '\\xF0\\xA0\\x80\\x8B\\xE9\\x85...' for column 'component' at row 1")

in reply to:  5 ; comment:7 by Christian Boos, 8 years ago

Another follow-up to comment:5:

On Windows with MariaDB 5.3 and MySQLdb 1.2.3:

And one failure with the functional tests:

======================================================================
ERROR: runTest (trac.db.tests.functional.DatabaseBackupTestCase)
Testing backup
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\0.12-stable\trac\db\tests\functional.py", line 12, in runTest
    backup_file = env.backup()
  File "c:\Trac\repos\0.12-stable\trac\env.py", line 493, in backup
    return DatabaseManager(self).backup(dest)
  File "c:\Trac\repos\0.12-stable\trac\db\api.py", line 180, in backup
    return connector.backup(dest)
  File "c:\Trac\repos\0.12-stable\trac\db\mysql_backend.py", line 203, in backup
    path=self.pg_dump_path,
AttributeError: 'MySQLConnector' object has no attribute 'pg_dump_path'

pg_dump???

in reply to:  7 comment:8 by Christian Boos, 8 years ago

  • trac/db/mysql_backend.py

    diff -r e4b61ba781b1 trac/db/mysql_backend.py
    a b  
    200200            p = Popen(args, env=environ, stderr=PIPE, close_fds=close_fds)
    201201        except OSError, e:
    202202            raise TracError(_("Unable to run %(path)s: %(msg)s",
    203                               path=self.pg_dump_path,
     203                              path=self.mysqldump_path,
    204204                              msg=exception_to_unicode(e)))
    205205        errmsg = p.communicate()[1]
    206206        if p.returncode != 0:

Then test pass after fixing the PATH to locate the mysqldump.exe binary. Good to have misconfigured environments at times, to test the error reporting code paths ;-)

in reply to:  4 ; comment:9 by Christian Boos, 8 years ago

Replying to jomae:

Replying to cboos:

i.e. the test pass, but the warning is a bit scary, as this string doesn't seem to be UTF-8 encoded.

I get the same warning now.

\xF0\xA0\x80\x8B (U+2000B) is valid UTF-8 bytes sequence. The same bytes exist in trac.util.tests.text.WrapTestCase.test_wrap_ambiwidth_double, the test confirms working on both UCS2/UCS4 builds of Python. But mysql 5.0 and 5.1 don't support 4-byte utf-8.

Hold on. If you can easily enter it in the browser (and it looks like: "𠀈" from list of CJK unified ideographs) and Python can handle it… then we will hit an error when saving the data (comment:6) (verified: when saving this in a Wiki page, we trigger an internal error). We probably need to fix this, unless using these characters or others which MySQL doesn't support is very very unusual.

I'll replace the bytes with 3-byte utf-8 (UCS2) in test_props_format_wrap_bothsides_unicode, later.

in reply to:  9 ; comment:10 by Jun Omae, 8 years ago

Replying to cboos:

Hold on. If you can easily enter it in the browser (and it looks like: "𠀈" from list of CJK unified ideographs) and Python can handle it… then we will hit an error when saving the data (comment:6) (verified: when saving this in a Wiki page, we trigger an internal error). We probably need to fix this, unless using these characters or others which MySQL doesn't support is very very unusual.

The character is 𠀋 (U+2000B) (not U+20008) ;-)

In Japan, using these characters probably is very unusual, not very very unusual. And, for example, Emoticon characters encode as 4-bytes UTF-8. The Emoticon characters is able to use in iOS, Gmail, etc. I think using UCS4 characters increase in the future.

in reply to:  10 ; comment:11 by Christian Boos, 8 years ago

I experimented with a MySQL database which was created using utf8mb4, which is available with recent versions of MySQL (≥ 5.5.3). But MySQLdb 1.2.3 I'm apparently not able to take benefit of this, as I end up with the same error (after changing node_rev.path key length to be 188… which is another story).

So either we keep the current internal error, or, if we find it to be too cryptic, we could show it as:

  • trac/db/mysql_backend.py

    diff -r e4b61ba781b1 trac/db/mysql_backend.py
    a b  
    229231        if hasattr(cnx, 'encoders'):
    230232            # 'encoders' undocumented but present since 1.2.1 (r422)
    231233            cnx.encoders[Markup] = cnx.encoders[types.UnicodeType]
     234        if hasattr(cnx, 'errorhandler'):
     235            defaulterrorhandler = cnx.errorhandler
     236            def mb4errorhandler(cursor, errorclass, errorvalue):
     237                if cursor is not None and \
     238                        isinstance(errorvalue, MySQLdb.OperationalError) and \
     239                        errorvalue.args[0] == 1366:
     240                    raise ValueError("4-bytes UTF-8 characters not supported "
     241                                     "by MySQL (%s)" % errorvalue)
     242                return defaulterrorhandler(cursor, errorclass, errorvalue)
     243            cnx.errorhandler = mb4errorhandler
    232244        ConnectionWrapper.__init__(self, cnx, log)
    233245        self._is_closed = False

Raising a TracError instead wouldn't work. For example in a wiki page save, we have a except TracError statement upon which we only show a warning (even without any specifics about the actual error, which is not good), so triggering an internal error is somehow needed.

in reply to:  11 ; comment:12 by Jun Omae, 8 years ago

Replying to cboos:

I experimented with a MySQL database which was created using utf8mb4, which is available with recent versions of MySQL (≥ 5.5.3). But MySQLdb 1.2.3 I'm apparently not able to take benefit of this, as I end up with the same error (after changing node_rev.path key length to be 188… which is another story).

MySQL utf8mb4 issue has been reported on #9766. And, I worked at changeset:t9766-mysql-utf8mb4/jomae which is tested with MySQL 5.0.77 and 5.5.20.

So either we keep the current internal error, or, if we find it to be too cryptic, we could show it as: #!diff....

I think that we should keep the current internal error. Detecting the error and raising a exception is very good when a user submits the wiki text, a user can remove these characters and resubmit. But, when these characters in a repository, trac-admin repository sync and trac-admin changeset added will probably stop and it's difficult to remove from the repository.

BTW, I tried your patch, however, errorhandler does not work with MySQLdb 1.2.2, 1.2.3 and MySQL 5.0.77, 5.5.20. I don't know why….

in reply to:  12 ; comment:13 by Christian Boos, 8 years ago

Replying to jomae:

Replying to cboos:

I experimented with a MySQL database which was created using utf8mb4, which is available with recent versions of MySQL (≥ 5.5.3). But MySQLdb 1.2.3 I'm apparently not able to take benefit of this, as I end up with the same error (after changing node_rev.path key length to be 188… which is another story).

MySQL utf8mb4 issue has been reported on #9766. And, I worked at changeset:t9766-mysql-utf8mb4/jomae which is tested with MySQL 5.0.77 and 5.5.20.

Great job! I confirm this works with MySQL 5.5.20 on Windows 7, with MySQLdb 1.2.3.

However, the trick seems to be the SET NAMES rather than the execute() overrider which seems to be unnecessary. In my tests, everything still worked when commenting it out.

So either we keep the current internal error, or, if we find it to be too cryptic, we could show it as: #!diff....

I think that we should keep the current internal error. Detecting the error and raising a exception is very good when a user submits the wiki text, a user can remove these characters and resubmit. But, when these characters in a repository, trac-admin repository sync and trac-admin changeset added will probably stop and it's difficult to remove from the repository.

Agreed, we should keep the exception. My idea was just to make it a bit more explicit, but I guess a note in MySqlDb will also do.

BTW, I tried your patch, however, errorhandler does not work with MySQLdb 1.2.2, 1.2.3 and MySQL 5.0.77, 5.5.20. I don't know why….

No idea, but probably not worth investigating either, at this point ;-)

in reply to:  13 ; comment:14 by Jun Omae, 8 years ago

Replying to cboos:

However, the trick seems to be the SET NAMES rather than the execute() overrider which seems to be unnecessary. In my tests, everything still worked when commenting it out.

The trick is necessary on utf8mb4 database. Fetching and storing UCS4 characters without the trick bring broken characters that are replaced with "?" character. For example, you please try to paste and save the character 𠀋 to a wiki page.

in reply to:  14 comment:15 by Christian Boos, 8 years ago

Replying to jomae:

Replying to cboos:

However, the trick seems to be the SET NAMES rather than the execute() overrider which seems to be unnecessary. In my tests, everything still worked when commenting it out.

The trick is necessary on utf8mb4 database. Fetching and storing UCS4 characters without the trick bring broken characters that are replaced with "?" character. For example, you please try to paste and save the character 𠀋 to a wiki page.

Well, that's what I did when I said it works… I now double checked, and I can assure you it really works even if the execute() does nothing more than a print (to be sure the correct code is being executed) and a redirect to the original method ;-)

The 𠀋 character is saved in the page and correctly displayed back when reloading the page.

Doing a trac-admin testenv/trac wiki export WikiStart | less shows:

= <F0><A0><80><8B> Trac !^M
...

The MySQL Workbench UI indeed shows only a '?', but that's probably because that tool can't deal with utf8mb4 (when executing SET NAMES 'utf8mb4' in the workbench, the next query reports Error Code: 0 Server sent uknown charsetnr. Please report(sic)).

comment:16 by Jun Omae, 8 years ago

Hmm, I confirmed with the following environments, again.

  • Database:
    • MySQL 5.5.20 on CentOS 5.6
    • MySQL 5.5.20 on Windows XP
  • Trac:
    • MySQLdb 1.2.2 on Windows XP
    • MySQLdb 1.2.3 on CentOS 5.6

Trac without the trick breaks these characters on each environment.

  • Without the trick.
    jun66j5@gotanda:1083$ PYTHONPATH=$PWD python trac/admin/console.py ~/var/trac/mysql-utf8mb4 wiki export utf8mb4 | od -c
    0000000   T   h   e       ?       c   h   a   r   a   c   t   e   r  \n
    0000020
    
  • With the trick.
    jun66j5@gotanda:1079$ PYTHONPATH=$PWD python trac/admin/console.py ~/var/trac/mysql-utf8mb4 wiki export utf8mb4 | od -c
    0000000   T   h   e     360 240 200 213       c   h   a   r   a   c   t
    0000020   e   r  \n
    0000023
    

It seems that MySQLdb on utf8mb4 database needs the trick.

jun66j5@gotanda:1063$ python
Python 2.4.3 (#1, Sep 21 2011, 20:06:00)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> conn = MySQLdb.connect(db='trac', host='192.168.11.16', port=3306, user='tracuser', passwd='password', charset='utf8')
>>> cursor = conn.cursor()
>>> cursor.execute("SHOW VARIABLES LIKE '%char%'")
8L
>>> for row in cursor: print row
...
(u'character_set_client', u'utf8')
(u'character_set_connection', u'utf8')
(u'character_set_database', u'utf8mb4')
(u'character_set_filesystem', u'binary')
(u'character_set_results', u'utf8')
(u'character_set_server', u'utf8mb4')
(u'character_set_system', u'utf8')
(u'character_sets_dir', u'/usr/share/mysql/charsets/')
>>> cursor.execute("SELECT text, hex(text) FROM wiki WHERE name='utf8mb4'")
1L
>>> for row in cursor: print row
...
('The ? character', '54686520F0A0808B20636861726163746572')
>>>
>>>
>>>
>>> conn.query('SET NAMES utf8mb4')
>>> conn.store_result()
>>> cursor.execute("SHOW VARIABLES LIKE '%char%'")
8L
>>> for row in cursor: print row
...
(u'character_set_client', u'utf8mb4')
(u'character_set_connection', u'utf8mb4')
(u'character_set_database', u'utf8mb4')
(u'character_set_filesystem', u'binary')
(u'character_set_results', u'utf8mb4')
(u'character_set_server', u'utf8mb4')
(u'character_set_system', u'utf8')
(u'character_sets_dir', u'/usr/share/mysql/charsets/')
>>> cursor.execute("SELECT text, hex(text) FROM wiki WHERE name='utf8mb4'")
1L
>>> for row in cursor: print row
...
('The \xf0\xa0\x80\x8b character', '54686520F0A0808B20636861726163746572')
>>>

comment:17 by Christian Boos, 8 years ago

Jun, I'm afraid there's a misunderstanding here ;-)

I was simply saying that SET NAMES utf8mb4 alone was necessary and enough to make it work.

From my experiments, I found that there was no need for overriding MySQLUnicodeCursor.execute as you did (maybe that was a left over from an initial attempt?).

in reply to:  17 comment:18 by Jun Omae, 8 years ago

Replying to cboos:

From my experiments, I found that there was no need for overriding MySQLUnicodeCursor.execute as you did (maybe that was a left over from an initial attempt?).

Oh…. You're right! Sorry, I'm wrong.

Updated my branch, the method has been removed in 7957/jomae.

comment:19 by Christian Boos, 8 years ago

Perfect! Please commit.

I'll commit my own fixes (comment:6, comment:8) a bit later and then we'll be ready for the release.

comment:20 by Jun Omae, 8 years ago

Ok, committed in [10961].

comment:21 by Christian Boos, 8 years ago

Release tagged in [10963].

comment:22 by Christian Boos, 8 years ago

Resolution: fixed
Status: newclosed

0.12.3 packages are now available (TracDownload@136).

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The ticket will remain with no owner.
The resolution will be deleted. Next status will be 'reopened'.
to as closed The owner will be changed from (none) to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.