tracd crashes on connection resets

[mr_seaworthy@…]

We are using tracd on Windows 2003 SP1 and most of the times it is running fine. Lately more and more users are browsing the websites and every now and then the daemon crashes.

Some of these crashes can be reproduced by for instance pressing the reload button in the website a couple of times (try to make it reload the page while the last reload was not finished yet).

----------------------------------------
Exception happened during processing of request from ('myip', 3104)
Traceback (most recent call last):
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python23\lib\SocketServer.py", line 463, in process_request_thread
  File "C:\Python23\lib\SocketServer.py", line 521, in __init__
    self.finish_request(request, client_address)
    self.handle()
  File "C:\Python23\lib\SocketServer.py", line 254, in finish_request
  File "C:\Python23\lib\BaseHTTPServer.py", line 324, in handle
    self.RequestHandlerClass(request, client_address, self)
    self.handle_one_request()
  File "C:\Python23\lib\SocketServer.py", line 521, in __init__
  File "C:\Python23\lib\BaseHTTPServer.py", line 318, in handle_one_request
    self.handle()
    method()
  File "C:\Python23\lib\BaseHTTPServer.py", line 324, in handle
  File "C:\Python23\lib\site-packages\trac\web\standalone.py", line 264, in do_G
ET
    self.handle_one_request()
    self._do_trac_req()
  File "C:\Python23\lib\BaseHTTPServer.py", line 318, in handle_one_request
  File "C:\Python23\lib\site-packages\trac\web\standalone.py", line 308, in _do_
trac_req
    method()
    dispatch_request(path_info, req, env)
  File "C:\Python23\lib\site-packages\trac\web\standalone.py", line 264, in do_G
ET
  File "C:\Python23\Lib\site-packages\trac\web\main.py", line 139, in dispatch_r
equest
    self._do_trac_req()
    dispatcher.dispatch(req)
  File "C:\Python23\lib\site-packages\trac\web\standalone.py", line 308, in _do_
trac_req
  File "C:\Python23\Lib\site-packages\trac\web\main.py", line 107, in dispatch
    dispatch_request(path_info, req, env)
    resp = chosen_handler.process_request(req)
  File "C:\Python23\Lib\site-packages\trac\web\main.py", line 139, in dispatch_r
equest
  File "C:\Python23\Lib\site-packages\trac\attachment.py", line 275, in process_
request
    dispatcher.dispatch(req)
    self._render_view(req, attachment)
  File "C:\Python23\Lib\site-packages\trac\attachment.py", line 435, in _render_
view
  File "C:\Python23\Lib\site-packages\trac\web\main.py", line 107, in dispatch
    mime_type + ';charset=' + charset)
  File "C:\Python23\Lib\site-packages\trac\web\api.py", line 202, in send_file
    resp = chosen_handler.process_request(req)
  File "C:\Python23\Lib\site-packages\trac\attachment.py", line 275, in process_
request
    self._render_view(req, attachment)
    self.write(data)
  File "C:\Python23\Lib\site-packages\trac\attachment.py", line 435, in _render_
view
  File "C:\Python23\lib\site-packages\trac\web\standalone.py", line 357, in writ
e
    mime_type + ';charset=' + charset)
    self.__handler.wfile.write(data)
  File "C:\Python23\Lib\site-packages\trac\web\api.py", line 202, in send_file
    self.write(data)
  File "C:\Python23\lib\socket.py", line 254, in write
    self.flush()
  File "C:\Python23\lib\site-packages\trac\web\standalone.py", line 357, in writ
e
  File "C:\Python23\lib\socket.py", line 241, in flush
    self.__handler.wfile.write(data)
    self._sock.sendall(buffer)
  File "C:\Python23\lib\socket.py", line 254, in write
error: (10054, 'Connection reset by peer')
    self.flush()
----------------------------------------
  File "C:\Python23\lib\socket.py", line 241, in flush
    self._sock.sendall(buffer)
error: (10054, 'Connection reset by peer')
----------------------------------------

Sometimes it is an error like the following:

----------------------------------------
Exception happened during processing of request from ('otherip', 4003)
Traceback (most recent call last):
  File "C:\Python23\lib\SocketServer.py", line 463, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python23\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python23\lib\SocketServer.py", line 521, in __init__
    self.handle()
  File "C:\Python23\lib\BaseHTTPServer.py", line 324, in handle
    self.handle_one_request()
  File "C:\Python23\lib\BaseHTTPServer.py", line 307, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python23\lib\socket.py", line 338, in readline
    data = self._sock.recv(self._rbufsize)
error: (10054, 'Connection reset by peer')
----------------------------------------

We upgraded to 0.9.3 recently but still have this problem (we started at 0.8.4 and had the same problem then, and in all versions in between).

[Christian Boos]

When you say tracd "crash", you actually means it only produces the above error in the log, or do you really mean that the server process dies (segfaults?)

If it's only about the errors, it's not that problematic. This error is caused by the clients (i.e. browsers) interrupting their requests, either because the user presses the Stop button, or … hits Reload in rapid succession!

We should however filter out such errors… We already do it for error code 32 (EPIPE on Unix), 10053 (WSAECONNABORTED on Windows), so we should do it for 10054 (WSAECONNRESET on Windows) as well.

[mr_seaworthy@…]

When i say tracd crashed i mean the proces dies. Most of the times it will continue, but some of the times the proces really dies.

We normally use a service for starting and stopping the daemon but for testing i now start it from a command prompt. When it crashes i get the prompt back.

[mr_seaworthy@…]

Small addition. Most of the times when it crashes a couple of requests are served in between the connection reset message and the crash.

[Christian Boos]

Ah… different story, then.

Anyway, I was not able to trigger 10054 errors by playing with the browser: only 10053 errors (those one are already intercepted and logged as "Trac[standalone] INFO: Lost connection to client: localhost"

10054 are perhaps due do an other software (firewall/antivirus?) which intercepts the connections and force a close. Maybe it's also the same thing which is killing tracd? I'm a bit clueless here, especially given that tracd seems to exit cleanly (no segfault message?)

tracd on Windows 2003 SP1: have you tried on another windows system, XP for instance (that's what I use, and I never got a 10054 error).

[anonymous]

About the segfaults… it triggered me to check the eventlogs.

I see the following entry: Faulting application python.exe, version 0.0.0.0, faulting module _sqlite.pyd, version 0.0.0.0, fault address 0x00025e11

But i only see this once today and the daemon crashes some 4 times already. So it might not be related.

[Christian Boos]

Which PySqlite version are you using? Can you try with either 2.0.5 or 2.0.6?

[mr_seaworthy@…]

After posting the last messages i figured it maybe had something to do with PySqlite so i upgrade it to the lastest versie 2.1.2. Since then the daemon hasn't crashed yet so we will have to see if it remains steady today.

[anonymous]

So far i cannot reproduce it by reloading the page again and again like i could reprodure it before. Was their a special reason i should have tried 2.0.5 or 2.0.6 instead of 2.1.2?

[Christian Boos]

Yes, 2.0.5 is the most robust version so far (2.0.6 has the same code, it's only a different packaging).

2.1.x has some known issues, which however don't seem to affect Trac. I'm currently testing 2.1.3 [pysqlite 207], which seems to be also pretty stable.

But once it stabilizes, 2.1.x will be the preferred version to use with Trac (it features an improved handling of concurrent writes and transparent statement caching).

[mr_seaworthy@…]

The daemon is still running without crashing, so i think it is safe to say this ticket can be closed.

Thanks for the help.

[Christian Boos]

I'd like to know what was the version of pysqlite which was crashing.

[mr_seaworthy@…]

It was with 2.0.3.

[Christian Boos]

Thanks! I'm close this one, now and I'll update PySqlite with the info.