Release Trac 0.12.2

[Christian Boos]

This task ticket is used to coordinate the finalization and testing of the next stable version of Trac, 0.12.2.

We're entering a one week period of testing for the rc1, tagged in [10465]. Packages can be downloaded from TracDownload#LatestReleaseCandidate.

If everything goes well, the release of 0.12.2 will happen the 29th of January, 2011.

Testing

If you'd like to contribute, please fill an entry in the table in TracDev/ReleaseTesting, detailing the configuration(s) you will test, and report the status of your testing in this ticket.

But you can also report glitches found with the rc1 directly here. If the problem is serious enough, you can also directly create a new ticket against 0.12.2.

[Christian Boos]

0.12.2rc1 deployed here for demo-0.12 and [Babel:]. I'm also testing locally and if all seems fine, I'll tag and release the rc1 a bit later today. In the meantime, feedback on r10464 is welcome.

[Christian Boos]

0.12.2rc1 version tagged and corresponding packages uploaded.

[Christian Boos]

I noticed an issue on Windows with tracd.exe installed using Trac-0.12.2rc1.win32.exe into either Python 2.4.4 or Python 2.5.4: if you interrupt it with CTRL+C, the tracd.exe process stops and you can interact with the console again, but its child process python.exe continues to run in the background and serves requests.

This can end up being quite confusing, especially coupled with the fact that on Windows you can start again the same tracd.exe on the same port without getting a port in use error. For example:

• start tracd.exe with an invalid environment path → no projects available
• interrupt → tracd.exe dies, you're back on the console
• restart tracd.exe with corrected path → still no projects available!

This is because the python.exe child process started by the first tracd.exe was still running and producing logging output on the console like it would the second server…

This doesn't happen with either Python 2.6.5 or Python 2.7.

[Remy Blank]

All tests pass here on a number of combinations of OS (Linux, OS X, Windows XP, Windows Vista, Windows 7), Python versions (2.4, 2.5, 2.6, 2.7) and database backends (SQLite, PostgreSQL, MySQL).

Release package installed on two production servers, and everything looks fine.

Created new environments on Linux/2.4/SQLite, OS X/2.6/SQLite and Windows 7/2.7/SQLite and played with them. Nothing to report.

[Jun Omae]

Replying to cboos:

I noticed an issue on Windows with tracd.exe installed using Trac-0.12.2rc1.win32.exe into either Python 2.4.4 or Python 2.5.4: if you interrupt it with CTRL+C, the tracd.exe process stops and you can interact with the console again, but its child process python.exe continues to run in the background and serves requests.

I have the same issue on Windows XP using Python 2.5.4. It seems that tracd.exe stops with CTRL-C, however, it still accepts requests.

It seems that CTRL-C cannot raise KeyboardInterrupt during calling socket.accept() on Windows.

This doesn't happen with either Python 2.6.5 or Python 2.7.

SocketServer.BaseServer.serve_forever() has changed from Python 2.6+. The method polls using select.select(). tracd.exe can stop with the polling on Windows.

The following patch needs more testing. I tested on Windows XP with Python 2.5.4 and Python 2.6.6.

trac/web/standalone.py

@@ -21 +21 @@
 
 import pkg_resources
 import os
+import select
 import sys
 from SocketServer import ThreadingMixIn
 
@@ -92 +93 @@
 class TracHTTPServer(ThreadingMixIn, WSGIServer):
     daemon_threads = True
 
+    poll_interval = 0.5
+
     def __init__(self, server_address, application, env_parent_dir, env_paths,
                  use_http_11=False):
         request_handlers = (TracHTTPRequestHandler, TracHTTP11RequestHandler)
         WSGIServer.__init__(self, server_address, application,
                             request_handler=request_handlers[bool(use_http_11)])
 
+    def serve(self):
+        while 1:
+            r, w, e = select.select([self], [], [], self.poll_interval)
+            if self in r:
+                self.handle_request()
 
+
 class TracHTTPRequestHandler(WSGIRequestHandler):
 
     server_version = 'tracd/' + VERSION
@@ -255 +264 @@
                 print 'Serving on http://%s:%s/%s' % (addr, port, base_path)
             if options.http11:
                 print 'Using HTTP/1.1 protocol version'
-            httpd.serve_forever()
+            httpd.serve()
     elif options.protocol in ('scgi', 'ajp', 'fcgi'):
         def serve():
             server_cls = __import__('flup.server.%s' % options.protocol,

[Christian Boos]

Replying to jomae:

… The following patch needs more testing. I tested on Windows XP with Python 2.5.4 and Python 2.6.6.

I've tested it heavily with Python 2.7 yesterday (without tracd.exe), and a bit today with Python 2.4.4 and tracd.exe. No real issues yesterday (sometimes strange backtraces, but nothing we haven't seen before), and with 2.4 the problem described in comment:4 is gone.

If we can test this on Linux as well, I'm OK to let it go in 0.12-stable before the 0.12.2 release.

Other than that, I think we're ready for the release.

[Thijs Triemstra]

Not sure if this one's expected but one test fails on ubuntu 10.10, python 2.7.1:

$ python -m trac.test --skip-functional-tests
SKIP: versioncontrol/tests/svn_fs.py (no svn bindings)
....................................................................................................................................................................................................................................................F..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
======================================================================
FAIL: test_from_suffix_using_mimetypes (trac.mimeview.tests.api.GetMimeTypeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "trac/mimeview/tests/api.py", line 39, in test_from_suffix_using_mimetypes
    self.assertEqual(image_png, get_mimetype('doc/trac_logo.png', None))
AssertionError: 'image/x-png' != 'image/png'

----------------------------------------------------------------------
Ran 979 tests in 18.001s

FAILED (failures=1)

[Remy Blank]

Replying to cboos:

If we can test this on Linux as well, I'm OK to let it go in 0.12-stable before the 0.12.2 release.

The SocketServer module has changed from 2.5 to 2.6, in the way Jun suggests in the patch. I would prefer keeping the new implementation in 2.6, and only "patch" 2.5 and lower, by overriding serve_forever() conditionally. This makes it clear that it's a workaround that can be removed once we drop support for 2.6.

trac/web/standalone.py

@@ -21 +21 @@
 
 import pkg_resources
 import os
+import select
 import sys
 from SocketServer import ThreadingMixIn
 
@@ -98 +99 @@
         WSGIServer.__init__(self, server_address, application,
                             request_handler=request_handlers[bool(use_http_11)])
 
+    if sys.version_info < (2, 6):
+        def serve_forever(self, poll_interval=0.5):
+            while True:
+                r, w, e = select.select([self], [], [], poll_interval)
+                if self in r:
+                    self.handle_request()
+
 
 class TracHTTPRequestHandler(WSGIRequestHandler):
 

All unit and functional tests pass with this patch on Linux with 2.4.6, 2.5.4 and 2.6.6. With 2.7.1, I get a few failures (including the one reported by Thijs), which I'm currently investigating.

[Christian Boos]

Replying to thijstriemstra:

Not sure if this one's expected but one test fails on ubuntu 10.10, python 2.7.1:

FAIL: test_from_suffix_using_mimetypes (trac.mimeview.tests.api.GetMimeTypeTestCase)
...
AssertionError: 'image/x-png' != 'image/png'

Yeah, not expected, but not surprising either, see #3858. At that time, I we picked image/png as I we figured that would be more "stable". Tough luck.

(Edit: that was actually Simon, not me… could have been me ;-) )

[Remy Blank]

Replying to cboos:

Yeah, not expected, but not surprising either, see #3858. At that time, I picked image/png as I figured that would be more "stable". Tough luck.

It's even more complicated: on Linux, the mimetypes module returns "image/png" for the tested filename with 2.6 and 2.7, whereas on Windows 2.7 returns "image/x-png". Can we just accept both?

trac/mimeview/tests/api.py

@@ -33 +33 @@
         self.assertEqual('text/plain', get_mimetype('README.txt', None))
         
     def test_from_suffix_using_mimetypes(self):
-        image_png = 'image/png'
-        if sys.version_info >= (2, 7):
-            image_png = 'image/x-png'
-        self.assertEqual(image_png, get_mimetype('doc/trac_logo.png', None))
+        accepted = ('image/png', 'image/x-png')
+        self.assert_(get_mimetype('doc/trac_logo.png', None) in accepted)
         
     def test_from_content_using_CONTENT_RE(self):
         self.assertEqual('text/x-python',

[Christian Boos]

Sigh. Is there nothing that is set in stone in this area? .jpg -> image/jpg perhaps? Oh, well. Perhaps just use assertTrue instead of assert_ then.

[Remy Blank]

Replying to rblank:

It's even more complicated: on Linux, the mimetypes module returns "image/png" for the tested filename with 2.6 and 2.7, whereas on Windows 2.7 returns "image/x-png".

Since 2.7, the mimetypes module now adds MIME types read from the Windows registry, so that's where the "image/x-png" comes from. Patch applied in [10503].

[Remy Blank]

As for the remaining 6 functional test failures with 2.7, they seem to be due to twill. More precisely, it seems that redirects to URLs with a fragment (e.g. http://127.0.0.1/ticket/40#comment:2) loose the fragment.

[Christian Boos]

All the functional tests pass for me, with Python 2.7 and twill 0.9 on Windows 7. Did you change your version of twill since comment:5?

[Remy Blank]

Replying to cboos:

All the functional tests pass for me, with Python 2.7 and twill 0.9 on Windows 7. Did you change your version of twill since comment:5?

No, 2.7 on Linux wasn't one of the combinations I tested. It was 2.7 on OS X Windows. I'm still trying to figure out what happens, but I'm not getting anywhere.

[Remy Blank]

Ok, so it's definitely an issue with twill on 2.7:

Python 2.6.6 (r266:84292, Dec 10 2010, 11:03:53) 
[GCC 4.4.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twill.commands as tc
>>> tc.go("http://trac.edgewall.org/#frag")
==> at http://trac.edgewall.org/#frag
'http://trac.edgewall.org/#frag'

but:

Python 2.7.1 (r271:86832, Dec 10 2010, 11:19:19) 
[GCC 4.4.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twill.commands as tc
>>> tc.go("http://trac.edgewall.org/#frag")
==> at http://trac.edgewall.org/
'http://trac.edgewall.org/'

Note the missing fragment on 2.7.1. With 2.7 on Windows, it works:

Python 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import twill.commands as tc
>>> tc.go("http://trac.edgewall.org/#frag")
==> at http://trac.edgewall.org/#frag
'http://trac.edgewall.org/#frag'

But not with 2.7.1 on Windows:

Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import twill.commands as tc
>>> tc.go("http://trac.edgewall.org/#frag")
==> at http://trac.edgewall.org/
'http://trac.edgewall.org/'

So it's an issue introduced with 2.7.1. Stay tuned…

[Remy Blank]

Looks very much related to this entry in the changelog of 2.7.1:

Fix ​Issue8280 - urllib2's Request method will remove fragements in the url. This is how it is supposed to work, wget and curl do the same. Previous behavior was wrong.

It seems like we should remove the fragment when we verify the current URL:

trac/tests/functional/tester.py

@@ -147 +147 @@
         tc.formvalue('propertyform', 'comment', comment)
         tc.submit("submit")
         # Verify we're where we're supposed to be.
-        tc.url(self.url + '/ticket/%s#comment:.*' % ticketid)
+        # The fragment is present up to 2.7, see:
+        # http://trac.edgewall.org/ticket/9990#comment:18
+        tc.url(self.url + '/ticket/%s(?:#comment:.*)?$' % ticketid)
         return comment
 
     def attach_file_to_ticket(self, ticketid, data=None, tempfilename=None,

With this fix, all tests pass on 2.7.1 as well.

[Jun Omae]

Replying to rblank:

The SocketServer module has changed from 2.5 to 2.6, in the way Jun suggests in the patch. I would prefer keeping the new implementation in 2.6, and only "patch" 2.5 and lower, by overriding serve_forever() conditionally. This makes it clear that it's a workaround that can be removed once we drop support for 2.6.

OK. I think that your patch is better than my patch. I applied your patch in r10504. I verified all unit and functional tests on Windows with 2.4.4, 2.5.4, 2.6.6 and 2.7.1, on Linux with 2.4.3, 2.5.2, 2.6.4 and 2.7.0.

[Christian Boos]

Replying to jomae:

I verified all unit and functional tests on Windows with 2.4.4, 2.5.4, 2.6.6 and 2.7.1, on Linux with 2.4.3, 2.5.2, 2.6.4 and 2.7.0.

Incidentally I just tested r10504 + comment:18 patch on Linux with 2.7.1, all OK as well.

[Remy Blank]

Great, thanks for testing. Patch applied in [10505].

[Christian Boos]

Ok, so branches/0.12-stable@10506​ looks final to me. Do you think it's worth testing a bit more or could I proceed and package it this evening?

[Remy Blank]

Nothing else from my side, so I vote for packaging. Except if you want to wait a bit more for translations.

Speaking of translations, I wonder if it wouldn't be better to move translations to a separate package, so people could install Trac from an official release, and install translations from SVN, or more frequent releases? Requiring all translators to update their translation during the one-week string freeze doesn't seem to work too well…

[Christian Boos]

Well, in this specific case as there were very little changes between 0.12.1 and 0.12.2, I thought that a week would be enough (using the rc1 test period for the freeze).

But the idea is worth digging.

[Thijs Triemstra]

Tests run fine now. I also suggest to add .twill-history to svn:ignore.

[Christian Boos]

Replying to thijstriemstra:

Tests run fine now. I also suggest to add .twill-history to svn:ignore.

… or disable writing it in the first place:

trac/tests/functional/better_twill.py

@@ -41 +41 @@
     # #5497).  Therefore we turn it off here.
     twill.commands.config('use_tidy', '0')
 
+    # We don't want a .twill-history file
+    twill.shell.readline = None
+
     # We use a transparent proxy to access the global browser object through
     # twill.get_browser(), as the browser can be destroyed by browser_reset()
     # (see #7472).

(untested)

[Thijs Triemstra]

Replying to cboos:

… or disable writing it in the first place:

That patch worked here, file's gone.

[Remy Blank]

Replying to cboos:

… or disable writing it in the first place:

Weird, I can't find a trace of that file anywhere. When is it created?

[Christian Boos]

Replying to rblank:

Weird, I can't find a trace of that file anywhere. When is it created?

Same for me, I've never seen that file, but apparently if you have readline it can get created (see twill/shell.py).

[Christian Boos]

Replying to cboos:

Replying to rblank:

Weird, I can't find a trace of that file anywhere. When is it created?

… as for "when", never seems to be the answer. I suppose Thijs experimented with twill in interactive mode? No need for that patch then, of course.

[Thijs Triemstra]

Replying to cboos:

Replying to cboos:

Replying to rblank:

Weird, I can't find a trace of that file anywhere. When is it created?

… as for "when", never seems to be the answer. I suppose Thijs experimented with twill in interactive mode? No need for that patch then, of course.

Twill was installed into stock python 2.6.6 on ubuntu 10.10 using easy_install, then I simply ran:

PYTHONPATH=. python2.6 ./trac/tests/functional/__init__.py

and svn status showed the file.

[Christian Boos]

Hm, weird. Can you please try to patch your twill/shell.py like that:

./twill/shell.py

@@ -237 +237 @@
     def do_EOF(self, *args):
         "Exit on CTRL-D"
         if readline:
+            import traceback
+            debug = open("/tmp/debug-twill.txt", "w")
+            traceback.print_stack(None, None, debug)
+            debug.close()
             readline.write_history_file('.twill-history')
 
         raise SystemExit()

[Christian Boos]

The Trac 0.12.2 packages are available, enjoy!

Thanks to all for having tested.

(Thijs, despite my closing of the ticket, I'm interested to continue to troubleshoot the problem you had with .twill-history, please redo the functional tests once you've patched twill, the /tmp/debug-twill.txt should tell us how you got to print that file)

[Thijs Triemstra]

I'll check it out cboos.

I received this error with 0.12-stable on python 2.6.6, probably needs a new ticket?

Detected modification of /home/thijs/workspaces/opensource/trac-stable-0.12/trac/versioncontrol/web_ui/changeset.py, restarting.
Unhandled exception in thread started by <function serve at 0x18acaa0>
Traceback (most recent call last):
  File "/home/thijs/workspaces/opensource/trac-stable-0.12/trac/web/standalone.py", line 266, in serve
    httpd.serve_forever()
  File "/usr/lib/python2.6/SocketServer.py", line 224, in serve_forever
    r, w, e = select.select([self], [], [], poll_interval)
AttributeError: 'NoneType' object has no attribute 'select'
Server starting in PID 11448.
Serving on http://localhost:8000/
Using HTTP/1.1 protocol version

[Christian Boos]

Replying to thijstriemstra:

I'll check it out cboos.

Ok, thanks!

I received this error with 0.12-stable on python 2.6.6, probably needs a new ticket?

  File "/usr/lib/python2.6/SocketServer.py", line 224, in serve_forever
    r, w, e = select.select([self], [], [], poll_interval)
AttributeError: 'NoneType' object has no attribute 'select'

This was one of the weird errors I referred to in comment:7 ;-) You can find other references to this error here and there (#3957). It's something we never really understood, my guess it's that some code gets executed once the builtin select module has already been destroyed.

Some googling confirms this:

When Python begins to shutdown it takes each module and sets each variable in the global namespace to None. If a thread has not terminated before the interpreter terminates then the thread tries to use a global variable which has been set to None. (​http://bugs.python.org/issue1722344#msg62078)

And I just see you were already aware of this issue ;-) I'll look at this later in more depth, but I'm not sure we can do anything about this problem. Also, it's harmless, the process is exiting anyway at this point, so for now I would say there's no need to create a dedicated ticket.

[Christian Boos]

Incidentally when browsing the code for tempfile (#9925), I saw the following in ​http://svn.python.org/view/python/trunk/Lib/tempfile.py?revision=73744:

class _TemporaryFileWrapper
#...
        # Cache the unlinker so we don't get spurious errors at
        # shutdown when the module-level "os" is None'd out.  Note
        # that this must be referenced as self.unlink, because the
        # name TemporaryFileWrapper may also get None'd out before
        # __del__ is called.
        unlink = _os.unlink

So we could just do something similar for select.select.

[Remy Blank]

Replying to cboos:

So we could just do something similar for select.select.

Too bad BaseServer.serve_forever() is outside of our control, at least for Python ≥2.6. Maybe report it to the Python folks with a reference to _TemporaryFileWrapper?

[Jun Omae]

Replying to rblank:

Replying to cboos:

So we could just do something similar for select.select.

Too bad BaseServer.serve_forever() is outside of our control, at least for Python ≥2.6. Maybe report it to the Python folks with a reference to _TemporaryFileWrapper?

I think that we should wait for the TracHTTPServer.serve_forever() termination before the python interpreter shutdowns.

At Python 2.6+, we can wait using BaseServer.shutdown(). At Python ⇐ 2.5, the method doesn't exist, however we can define TracHTTPServer.shutdown() like TracHTTPServer.serve_forever().

t9990-tracd-shutdown.diff​.

[Christian Boos]

Replying to jomae:

At Python 2.6+, we can wait using BaseServer.shutdown(). At Python ⇐ 2.5, the method doesn't exist, however we can define TracHTTPServer.shutdown() like TracHTTPServer.serve_forever().

t9990-tracd-shutdown.diff​.

Looks great: I'm testing & trying to make sure I understand, this can take a while ;-)

[Christian Boos]

Btw, looks like we forgot to send the release announcement, last week… so I just did it now. Any taker for the last 2 other steps in TracDev/ReleaseChecklist (Pypi + Freshmeat)?

[Remy Blank]

Isn't the Pypi update a simple:

python setup.py upload

that requires the release packages to be present? Or do we add the new entry by hand? In the latter case, I can do the update. I'm not sure I have access to the Freshmeat account, though.

[Thijs Triemstra]

Replying to cboos:

(Thijs, despite my closing of the ticket, I'm interested to continue to troubleshoot the problem you had with .twill-history, please redo the functional tests once you've patched twill, the /tmp/debug-twill.txt should tell us how you got to print that file)

I tried it last night with exactly the same setup afaik but I couldn't reproduce it :-/ weird, but oh well.