Release Trac 0.12.3

[Christian Boos]

a.k.a. the yearly bug fix release ;-)

This task ticket is used to coordinate the finalization and testing of the next stable version of Trac, 0.12.3.

We're entering a one week period of testing for the rc1, tagged in [10948]. Packages can be downloaded from TracDownload#LatestReleaseCandidate.

If everything goes well, the release of 0.12.3 will happen the 5th of February, 2012.

[Remy Blank]

And a note to translators: I forgot to re-extract the messages prior to the string freeze, so there are a few new strings in rc1. Please feel free to update your translations until the final 0.12.3 release. Apologies for the mistake.

[Mikael Relbe]

Replying to rblank:

And a note to translators: I forgot to re-extract the messages prior to the string freeze, so there are a few new strings in rc1. Please feel free to update your translations until the final 0.12.3 release. Apologies for the mistake.

No worries, uk-en and sv 100% done (r10950:10951)

[Christian Boos]

Small glitch noticed during testing on Linux (Python 2.6, mysql 5.0.67, mysql-python 1.2.2)

test_props_format_wrap_bothsides_unicode (trac.ticket.tests.notification.NotificationTestCase) ... /home/cboos/trac/trac-0.12.3rc1/trac/db/util.py:65: Warning: Incorrect string value: '\xF0\xA0\x80\x8B\xE9\x85...' for column 'component' at row 1
  return self.cursor.execute(sql_escape_percent(sql), args)
ok

i.e. the test pass, but the warning is a bit scary, as this string doesn't seem to be UTF-8 encoded.

[Jun Omae]

Replying to cboos:

i.e. the test pass, but the warning is a bit scary, as this string doesn't seem to be UTF-8 encoded.

I get the same warning now.

\xF0\xA0\x80\x8B (U+2000B) is valid UTF-8 bytes sequence. The same bytes exist in trac.util.tests.text.WrapTestCase.test_wrap_ambiwidth_double, the test confirms working on both UCS2/UCS4 builds of Python. But mysql 5.0 and 5.1 don't support 4-byte utf-8.

I'll replace the bytes with 3-byte utf-8 (UCS2) in test_props_format_wrap_bothsides_unicode, later.

[Christian Boos]

On Windows with MariaDB 5.3 and MySQLdb 1.2.3:

• same warning as above
• 5 errors, all similar:

  ======================================================================
  ERROR: test_delete_all_comments (trac.ticket.tests.model.TicketCommentDeleteTestCase)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "c:\Trac\repos\0.12-stable\trac\ticket\tests\model.py", line 665, in test_delete_all_comments
      ticket.delete_change(4)
    File "c:\Trac\repos\0.12-stable\trac\ticket\model.py", line 442, in delete_change
      @self.env.with_transaction()
    File "c:\Trac\repos\0.12-stable\trac\db\api.py", line 77, in transaction_wrapper
      fn(ldb)
    File "c:\Trac\repos\0.12-stable\trac\ticket\model.py", line 504, in do_delete
      """, (self.id, self.id, self.id))
    File "c:\Trac\repos\0.12-stable\trac\db\util.py", line 65, in execute
      return self.cursor.execute(sql_escape_percent(sql), args)
    File "C:\Dev\Python272\lib\site-packages\MySQLdb\cursors.py", line 174, in execute
      self.errorhandler(self, exc, value)
    File "C:\Dev\Python272\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
      raise errorclass, errorvalue
  OperationalError: (1093, "You can't specify target table 'ticket' for update in FROM clause")
  

The 1093 error seems to be prevalent in most versions of MySQL, from at least ​4.1 to ​5.6.

There are several workarounds documented for this problem (e.g. ​http://stackoverflow.com/a/45498), however I just see that such a workaround was already attempted in r10803 and apparently it must work for some configurations (like in comment:3).

So it might well be a MariaDB-only issue…

[Christian Boos]

Follow-up to comment:5:

On Windows with MariaDB 5.3 and MySQLdb 1.2.3:

• same warning as above
• 5 errors, all similar:

  ======================================================================
  ERROR: test_delete_all_comments (trac.ticket.tests.model.TicketCommentDeleteTestCase)
  ...
  OperationalError: (1093, "You can't specify target table 'ticket' for update in FROM clause")
  

So it might well be a MariaDB-only issue…

The following seems to do the trick:

trac/ticket/model.py

@@ -498 +498 @@
                     SELECT time FROM ticket_change WHERE ticket=%s
                     UNION
                     SELECT time FROM (
-                        SELECT time FROM ticket WHERE id=%s) AS t
+                        SELECT time FROM ticket WHERE id=%s LIMIT 1) AS t
                     ORDER BY time DESC LIMIT 1)
                 WHERE id=%s
                 """, (self.id, self.id, self.id))

Also, the UTF-8 issue is actually not a warning in this setup, but a plain error:

OperationalError: (1366, "Incorrect string value: '\\xF0\\xA0\\x80\\x8B\\xE9\\x85...' for column 'component' at row 1")

[Christian Boos]

Another follow-up to comment:5:

On Windows with MariaDB 5.3 and MySQLdb 1.2.3:

And one failure with the functional tests:

======================================================================
ERROR: runTest (trac.db.tests.functional.DatabaseBackupTestCase)
Testing backup
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Trac\repos\0.12-stable\trac\db\tests\functional.py", line 12, in runTest
    backup_file = env.backup()
  File "c:\Trac\repos\0.12-stable\trac\env.py", line 493, in backup
    return DatabaseManager(self).backup(dest)
  File "c:\Trac\repos\0.12-stable\trac\db\api.py", line 180, in backup
    return connector.backup(dest)
  File "c:\Trac\repos\0.12-stable\trac\db\mysql_backend.py", line 203, in backup
    path=self.pg_dump_path,
AttributeError: 'MySQLConnector' object has no attribute 'pg_dump_path'

pg_dump???

[Christian Boos]

trac/db/mysql_backend.py

@@ -200 +200 @@
             p = Popen(args, env=environ, stderr=PIPE, close_fds=close_fds)
         except OSError, e:
             raise TracError(_("Unable to run %(path)s: %(msg)s",
-                              path=self.pg_dump_path,
+                              path=self.mysqldump_path,
                               msg=exception_to_unicode(e)))
         errmsg = p.communicate()[1]
         if p.returncode != 0:

Then test pass after fixing the PATH to locate the mysqldump.exe binary. Good to have misconfigured environments at times, to test the error reporting code paths ;-)

[Christian Boos]

Replying to jomae:

Replying to cboos:

i.e. the test pass, but the warning is a bit scary, as this string doesn't seem to be UTF-8 encoded.

I get the same warning now.

\xF0\xA0\x80\x8B (U+2000B) is valid UTF-8 bytes sequence. The same bytes exist in trac.util.tests.text.WrapTestCase.test_wrap_ambiwidth_double, the test confirms working on both UCS2/UCS4 builds of Python. But mysql 5.0 and 5.1 don't support 4-byte utf-8.

Hold on. If you can easily enter it in the browser (and it looks like: "𠀈" from ​list of CJK unified ideographs) and Python can handle it… then we will hit an error when saving the data (comment:6) (verified: when saving this in a Wiki page, we trigger an internal error). We probably need to fix this, unless using these characters or others which MySQL doesn't support is very very unusual.

I'll replace the bytes with 3-byte utf-8 (UCS2) in test_props_format_wrap_bothsides_unicode, later.

[Jun Omae]

Replying to cboos:

Hold on. If you can easily enter it in the browser (and it looks like: "𠀈" from ​list of CJK unified ideographs) and Python can handle it… then we will hit an error when saving the data (comment:6) (verified: when saving this in a Wiki page, we trigger an internal error). We probably need to fix this, unless using these characters or others which MySQL doesn't support is very very unusual.

The character is ​𠀋 (U+2000B) (not ​U+20008) ;-)

In Japan, using these characters probably is very unusual, not very very unusual. And, for example, ​Emoticon characters encode as 4-bytes UTF-8. The Emoticon characters is able to use in iOS, Gmail, etc. I think using UCS4 characters increase in the future.

[Christian Boos]

I experimented with a MySQL database which was created using ​utf8mb4, which is available with recent versions of MySQL (≥ 5.5.3). But MySQLdb 1.2.3 I'm apparently not able to take benefit of this, as I end up with the same error (after changing node_rev.path key length to be 188… which is another story).

So either we keep the current internal error, or, if we find it to be too cryptic, we could show it as:

trac/db/mysql_backend.py

@@ -229 +231 @@
         if hasattr(cnx, 'encoders'):
             # 'encoders' undocumented but present since 1.2.1 (r422)
             cnx.encoders[Markup] = cnx.encoders[types.UnicodeType]
+        if hasattr(cnx, 'errorhandler'):
+            defaulterrorhandler = cnx.errorhandler
+            def mb4errorhandler(cursor, errorclass, errorvalue):
+                if cursor is not None and \
+                        isinstance(errorvalue, MySQLdb.OperationalError) and \
+                        errorvalue.args[0] == 1366:
+                    raise ValueError("4-bytes UTF-8 characters not supported "
+                                     "by MySQL (%s)" % errorvalue)
+                return defaulterrorhandler(cursor, errorclass, errorvalue)
+            cnx.errorhandler = mb4errorhandler
         ConnectionWrapper.__init__(self, cnx, log)
         self._is_closed = False

Raising a TracError instead wouldn't work. For example in a wiki page save, we have a except TracError statement upon which we only show a warning (even without any specifics about the actual error, which is not good), so triggering an internal error is somehow needed.

[Jun Omae]

Replying to cboos:

I experimented with a MySQL database which was created using ​utf8mb4, which is available with recent versions of MySQL (≥ 5.5.3). But MySQLdb 1.2.3 I'm apparently not able to take benefit of this, as I end up with the same error (after changing node_rev.path key length to be 188… which is another story).

MySQL utf8mb4 issue has been reported on #9766. And, I worked at changeset:t9766-mysql-utf8mb4/jomae which is tested with MySQL 5.0.77 and 5.5.20.

So either we keep the current internal error, or, if we find it to be too cryptic, we could show it as: #!diff....

I think that we should keep the current internal error. Detecting the error and raising a exception is very good when a user submits the wiki text, a user can remove these characters and resubmit. But, when these characters in a repository, trac-admin repository sync and trac-admin changeset added will probably stop and it's difficult to remove from the repository.

BTW, I tried your patch, however, errorhandler does not work with MySQLdb 1.2.2, 1.2.3 and MySQL 5.0.77, 5.5.20. I don't know why….

[Christian Boos]

Replying to jomae:

Replying to cboos:

I experimented with a MySQL database which was created using ​utf8mb4, which is available with recent versions of MySQL (≥ 5.5.3). But MySQLdb 1.2.3 I'm apparently not able to take benefit of this, as I end up with the same error (after changing node_rev.path key length to be 188… which is another story).

MySQL utf8mb4 issue has been reported on #9766. And, I worked at changeset:t9766-mysql-utf8mb4/jomae which is tested with MySQL 5.0.77 and 5.5.20.

Great job! I confirm this works with MySQL 5.5.20 on Windows 7, with MySQLdb 1.2.3.

However, the trick seems to be the SET NAMES rather than the execute() overrider which seems to be unnecessary. In my tests, everything still worked when commenting it out.

So either we keep the current internal error, or, if we find it to be too cryptic, we could show it as: #!diff....

I think that we should keep the current internal error. Detecting the error and raising a exception is very good when a user submits the wiki text, a user can remove these characters and resubmit. But, when these characters in a repository, trac-admin repository sync and trac-admin changeset added will probably stop and it's difficult to remove from the repository.

Agreed, we should keep the exception. My idea was just to make it a bit more explicit, but I guess a note in MySqlDb will also do.

BTW, I tried your patch, however, errorhandler does not work with MySQLdb 1.2.2, 1.2.3 and MySQL 5.0.77, 5.5.20. I don't know why….

No idea, but probably not worth investigating either, at this point ;-)

[Jun Omae]

Replying to cboos:

However, the trick seems to be the SET NAMES rather than the execute() overrider which seems to be unnecessary. In my tests, everything still worked when commenting it out.

The trick is necessary on utf8mb4 database. Fetching and storing UCS4 characters without the trick bring broken characters that are replaced with "?" character. For example, you please try to paste and save the character 𠀋 to a wiki page.

[Christian Boos]

Replying to jomae:

Replying to cboos:

However, the trick seems to be the SET NAMES rather than the execute() overrider which seems to be unnecessary. In my tests, everything still worked when commenting it out.

The trick is necessary on utf8mb4 database. Fetching and storing UCS4 characters without the trick bring broken characters that are replaced with "?" character. For example, you please try to paste and save the character 𠀋 to a wiki page.

Well, that's what I did when I said it works… I now double checked, and I can assure you it really works even if the execute() does nothing more than a print (to be sure the correct code is being executed) and a redirect to the original method ;-)

The 𠀋 character is saved in the page and correctly displayed back when reloading the page.

Doing a trac-admin testenv/trac wiki export WikiStart | less shows:

= <F0><A0><80><8B> Trac !^M
...

The MySQL Workbench UI indeed shows only a '?', but that's probably because that tool can't deal with utf8mb4 (when executing SET NAMES 'utf8mb4' in the workbench, the next query reports Error Code: 0 Server sent uknown charsetnr. Please report(sic)).

[Jun Omae]

Hmm, I confirmed with the following environments, again.

• Database:
  • MySQL 5.5.20 on CentOS 5.6
  • MySQL 5.5.20 on Windows XP
• Trac:
  • MySQLdb 1.2.2 on Windows XP
  • MySQLdb 1.2.3 on CentOS 5.6

Trac without the trick breaks these characters on each environment.

• Without the trick.

  jun66j5@gotanda:1083$ PYTHONPATH=$PWD python trac/admin/console.py ~/var/trac/mysql-utf8mb4 wiki export utf8mb4 | od -c
  0000000   T   h   e       ?       c   h   a   r   a   c   t   e   r  \n
  0000020
  

• With the trick.

  jun66j5@gotanda:1079$ PYTHONPATH=$PWD python trac/admin/console.py ~/var/trac/mysql-utf8mb4 wiki export utf8mb4 | od -c
  0000000   T   h   e     360 240 200 213       c   h   a   r   a   c   t
  0000020   e   r  \n
  0000023
  

It seems that MySQLdb on utf8mb4 database needs the trick.

jun66j5@gotanda:1063$ python
Python 2.4.3 (#1, Sep 21 2011, 20:06:00)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> conn = MySQLdb.connect(db='trac', host='192.168.11.16', port=3306, user='tracuser', passwd='password', charset='utf8')
>>> cursor = conn.cursor()
>>> cursor.execute("SHOW VARIABLES LIKE '%char%'")
8L
>>> for row in cursor: print row
...
(u'character_set_client', u'utf8')
(u'character_set_connection', u'utf8')
(u'character_set_database', u'utf8mb4')
(u'character_set_filesystem', u'binary')
(u'character_set_results', u'utf8')
(u'character_set_server', u'utf8mb4')
(u'character_set_system', u'utf8')
(u'character_sets_dir', u'/usr/share/mysql/charsets/')
>>> cursor.execute("SELECT text, hex(text) FROM wiki WHERE name='utf8mb4'")
1L
>>> for row in cursor: print row
...
('The ? character', '54686520F0A0808B20636861726163746572')
>>>
>>>
>>>
>>> conn.query('SET NAMES utf8mb4')
>>> conn.store_result()
>>> cursor.execute("SHOW VARIABLES LIKE '%char%'")
8L
>>> for row in cursor: print row
...
(u'character_set_client', u'utf8mb4')
(u'character_set_connection', u'utf8mb4')
(u'character_set_database', u'utf8mb4')
(u'character_set_filesystem', u'binary')
(u'character_set_results', u'utf8mb4')
(u'character_set_server', u'utf8mb4')
(u'character_set_system', u'utf8')
(u'character_sets_dir', u'/usr/share/mysql/charsets/')
>>> cursor.execute("SELECT text, hex(text) FROM wiki WHERE name='utf8mb4'")
1L
>>> for row in cursor: print row
...
('The \xf0\xa0\x80\x8b character', '54686520F0A0808B20636861726163746572')
>>>

[Christian Boos]

Jun, I'm afraid there's a misunderstanding here ;-)

I was simply saying that SET NAMES utf8mb4 alone was necessary and enough to make it work.

From my experiments, I found that there was no need for overriding MySQLUnicodeCursor.execute as you did (maybe that was a left over from an initial attempt?).

[Jun Omae]

Replying to cboos:

From my experiments, I found that there was no need for overriding MySQLUnicodeCursor.execute as you did (maybe that was a left over from an initial attempt?).

Oh…. You're right! Sorry, I'm wrong.

Updated my branch, the method has been removed in 7957/jomae.

[Christian Boos]

Perfect! Please commit.

I'll commit my own fixes (comment:6, comment:8) a bit later and then we'll be ready for the release.

[Jun Omae]

Ok, committed in [10961].

[Christian Boos]

Release tagged in [10963].

[Christian Boos]

0.12.3 packages are now available (TracDownload@136).