Submitting a ticket is very slow

[as_barrus@…]

Hello Trac developers,

I use trac for my small development team. The wiki is responsive, as is looking up tickets and searching, etc. Browsing the subversion source is fast. But sumbitting any ticket is very slow. Creating a new ticket takes about 30 seconds to respond after I press the "submit ticket" button. Likewise, when updating a ticket, it takes about 30 seconds for trac to answer that the update was successful. I'm running trac 0.9.5. The trac server is a dual 2.0 Ghz machine with a four-member SCSI RAID-5 stripe. The load average is 0.01. Trac runs against an Sqlite database. I admit I don't know much about how that part works, but I suspect it could be where the problem lies.

It's getting bad enough that I'm considering reinstalling on a new server just to see if the problem goes away magically; or point it to a mysql or postgress db - just for troubleshooting. But solving it the correct way would be a lot nicer for me!

Attached is a screenshot from the Tamper FF plugin to show the load times for various pages. I am happy to provide any additional follow up data to help me solve this problem. If this is the wrong way to request trac help, then please redirect me. Thank you! Trac is great and I want to continue using it!

[as_barrus@…]

Tamper FF plugin showing load times from trac server

[Alec Thomas]

This is almost certainly related to notification. When Trac attempts delivery of notification mails it will block until the mail server has taken delivery, so things like spam filtering, RBL lookups, DNS timeouts, etc. will all slow down notification and consequently, the web interface.

[Christian Boos]

Yes, but I don't see why this make the ticket invalid. If those delays are somewhat inevitable, I see no reason why they should interfere with the ticket creation/change, as it's only a notification, not something that could invalidate the submission.

Why not address the problem by making the notification asynchronous?

[Alec Thomas]

Frankly, my view is that if you're seeing a delay of 30 seconds from an MTA, then it's misconfigured.

All mail servers that I'm aware of can be configured to skip these checks if accessed from the Trac server's IP address, so adding the complexity to Trac seems unnecessary. Still, if you think it's necessary…

[Alec Thomas]

Another thing to consider, if you go ahead, is warding against fork bombs caused by rapid ticket creation/updates. MTA's generally already have this facility built in of course, but you'd have to replicate that logic in Trac.

[Christian Boos]

Frankly, my view is that if you're seeing a delay of 30 seconds from an MTA, then it's misconfigured.

I agree, and I'm aware that one problem with my proposal is that it could make the detection of such a misconfiguration a bit harder.

On the implementation p.o.v, I was thinking of queueing a notification "event" and have a dedicated notifier thread, which would simply do the notification of those events, sequentially.

If that thread detects abusive delays, it could log a warning or even an error.

[Emmanuel Blot]

Why not address the problem by making the notification asynchronous?

I would say "keep it simple". Or "keep keeping it simple" ;-)

Introducing background tasks in Trac seems to be quite overkill, and would probably lead to create a new bunch of hard-to-solve issues, to fix up the issue at the wrong place (in Trac, rather than at the MTA level).

I don't see the point to introduce serialization in email notifications, whereas the MTAs are already able to receive concurrent notifications.

Trac is not a MTA debugging agent. To detect long latency issues, it's far easier (and safer…) to retrieve the time before the SMTP connection and after the notification, and log a warning message when the time difference reaches an upper limit.

[Christopher Lenz]

I strongly agree with Manu. Let's close this again.

[ghazel@…]

We have run in to slow ticket posting as well, and have found it very difficult to debug. This appears to us as a problem with Trac.

Please, at least log about slow notification submissions.

I would rather the ticket be created and mail be delivered later or not at all. Blocking the browser creates opportunities for the ticket to be lost, including the user cancelling out of frustration. Imagine if sending an email waited until it was submitted to the MTA.

[Jonas Borgström]

I'm closing this as fixed now (r7322, r7323). Notification mails are still sent synchronously but slow MTAs are now detected and logged making troubleshooting easier.

[edwardrf@…]

I think one point missing here is in many cases, Trac is hosted in a local environment (like in my office) which does not have a working local MTA, notifications sent out relies on external SMTP server (like gmail). The connection to an external smtp server with authentication can vary in performance, but usually results in a 2~10 seconds wait which drastically reduce the response speed of Trac.

I totally agree the "keep it simple" principal, but it does not mean problems should not be fixed if it takes a little bit more effort. I here by proposing another solution, apart from treading and sending queue).

The program can output and flush the page content to the browser first, then execute the email sending code

Please do not laugh at me if this is not implementable or hard to achieve with the current trac architecture, I am not very familiar with python nor familiar with Trac source code.

[Remy Blank]

Replying to edwardrf@…:

I think one point missing here is in many cases, Trac is hosted in a local environment (like in my office) which does not have a working local MTA, notifications sent out relies on external SMTP server (like gmail).

Sending through an external SMTP server doesn't prevent you from having a working local MTA. For example, my setup has a local Postfix server accepting outgoing mail, and sending it all through my provider's SMTP server. Postfix does the queuing.

Your idea isn't ridiculous at all, it's probably how we would implement it if this was an issue with Trac. But it really isn't, and we'd rather leave the queuing to the MTA.

[edwardrf@…]

Replying to rblank:

Sending through an external SMTP server doesn't prevent you from having a working local MTA. For example, my setup has a local Postfix server accepting outgoing mail, and sending it all through my provider's SMTP server. Postfix does the queuing.

Hi rblank,

Thanks for your prompt reply. I am impressed by the trac team for such a prompt reply. yes, you are totally right, we do have a local running MTA, however, our office is on dynamic IP (like most of the small offices do) which results all email sent out from the local MTA rejected by other MTAs. (And to configure a dynamic IP address to use a relay is not a simple task)

From the engineering point of view, yes, Trac does not have a problem as it is not designed to solve email delivering issues. However, from the users' point of view, you have to agree it is a usability issue which is really hard to find out the cause (took me 1 week to find out the email sending was the killer, and 2 weeks to arrive at this thread). And I believe I am not the only one affected.

I am a engineer too, i totally understand what you feel about this tread. It is quite similar to when clients comeback with requests that totally defeats the design principles of the system. But years of experiences taught me that if the client comes back again and again with the same issue, they are feeling the pain. (As i am right now)

Please do help out, it is really needed. Could you please:

1. Is there an existing plugin or patch to deal with this issue?
2. How can I contribute if I want to help solve this issue?
3. Is there anyone already working on this issue?

Again, Thanks for your reply.

Regards Edward

[Christian Boos]

You may want to try out the ​use_threaded_delivery option of the ​TH:AnnouncerPlugin.

[Remy Blank]

Replying to edwardrf@…:

yes, you are totally right, we do have a local running MTA, however, our office is on dynamic IP (like most of the small offices do) which results all email sent out from the local MTA rejected by other MTAs. (And to configure a dynamic IP address to use a relay is not a simple task)

I did understand that, and I have the same IP configuration here, but this still shouldn't be a problem. I have my local MTA send all its messages to the SMTP server of my ISP (through an authenticated account), which then forwards it on to their destination. So the dynamic IP is only ever seen by my ISP, who will accept the messages because I'm on its network.

[anonymous]

Replying to rblank:

I did understand that, and I have the same IP configuration here, but this still shouldn't be a problem. I have my local MTA send all its messages to the SMTP server of my ISP (through an authenticated account), which then forwards it on to their destination. So the dynamic IP is only ever seen by my ISP, who will accept the messages because I'm on its network.

I am aware of that option too. But the configuration process to modify a local SMTP server to send using only one account is complicated and it varies with different server and very little documented.

So your bottom line is no matter how time wasting and tedious it is for a normal user to use a solution on the MTA side could be, it does not worth the little effort to create a solution which may or may not affect the "cleanness" of your design/code?

I wouldn't mind to learn more about python and contribute to this project, and apparently that's the only choice left.

[edwardrf@…]

Replying to cboos:

You may want to try out the ​use_threaded_delivery option of the ​TH:AnnouncerPlugin.

Thanks cboos, i tried the announcer plugin, had some problem with it before, i'll look into that again.

[Christian Boos]

Replying to edwardrf@…:

Replying to cboos:

You may want to try out the ​use_threaded_delivery option of the ​TH:AnnouncerPlugin.

Thanks cboos, i tried the announcer plugin, had some problem with it before, i'll look into that again.

Even if you intend to patch Trac core instead of using that plugin, the source code for the use_threaded_delivery option is a good inspiration for what you'll need to do. I see that their ​DeliveryThread is however recreated each time the environment gets reloaded, which is not good. But well, as we don't have yet a proper support for environment life-cycle (#5010) we can't blame them ;-)

Be aware that in some server setups, the serving process might be killed right after the request has been sent, regardless of the fact there might be other threads still running… In such case, asynchronous e-mail delivery by using a thread wouldn't work, of course.

[Remy Blank]

Replying to anonymous:

I am aware of that option too. But the configuration process to modify a local SMTP server to send using only one account is complicated and it varies with different server and very little documented.

That's simply not true. For example, with Postfix, it's as simple as setting:

relayhost = [mail.example.com]

I'm sure other MTAs have similar options.

So your bottom line is no matter how time wasting and tedious it is for a normal user to use a solution on the MTA side could be, it does not worth the little effort to create a solution which may or may not affect the "cleanness" of your design/code?

You may have other services running on your server that also send e-mails (cron? Other web services?). Should all of them re-implement a threaded e-mail delivery subsystem, simply because you don't want to configure your MTA properly? You aren't wasting your time configuring your MTA, you are learning something new. Once you have learned it, you will configure it correctly every time, and not waste any time anymore.

The bottom line is this: I want to fix the root cause of the issue, not work around it. And here, the root cause is the MTA, not Trac.

It's a somewhat different situation in the Announcer plugin. IIUC, it sends many more individual e-mails per action (e.g. when submitting a form), possibly even encrypting them. Even with a correctly configured MTA, this takes too much time. Then it makes sense to implement a threaded delivery system. Once Announcer gets merged into Trac (if that's still the plan), Trac will get threaded delivery as well. I wouldn't implement it in the meantime, though.

[edwardrf@…]

Replying to rblank:

That's simply not true. For example, with Postfix, it's as simple as setting:

relayhost = [mail.example.com]

I'm sure other MTAs have similar options.

This is great info. If we can list out solutions like this for common MTAs maybe a lot of confusion could be avoided. Maybe list it on trac performance page (​http://trac.edgewall.org/wiki/TracPerformance) would be way more helpful.

You may have other services running on your server that also send e-mails (cron? Other web services?). Should all of them re-implement a threaded e-mail delivery subsystem, simply because you don't want to configure your MTA properly? You aren't wasting your time configuring your MTA, you are learning something new. Once you have learned it, you will configure it correctly every time, and not waste any time anymore.

All other email sending services does not affect my users' experience. Cron can send out emails however slow it want to be, no one is going to complain, but if every click on a browser is going to take 5 seconds of pure waiting time, that is really disruptive and annoying. I have hudson setup as a CI tool, it does not have the email sending blockage problem as trac does.

The bottom line is this: I want to fix the root cause of the issue, not work around it. And here, the root cause is the MTA, not Trac.

I really appreciate (in fact admire is more appropriate) the way you stand by your priciples, it is really needed for big projects. However, I want to point out 2 things:

1. if MTA miss configuration is such a common problem, list out the common solution would me way more helpful than constantly restate it is not trac's problem.
2. If remote SMTP is always gonna be a chock point, should SMTP email sending be coded in the first place? (Ya, this sound stupid, but I don't feel i have received enough warning about the sending speed given when I choose that option, and it took me a week to notice it is not my virtual machine's problem, not the python running too slow, memory was not the limit, but the email sending is slowing everyone down)

BTW, whoever is facing the same problem please take a look at the relayhost setup below, it worked for me.

​http://www.freelock.com/kb/postfix-relayhost ​http://wingloon.com/2008/06/21/setup-relay-host-port-and-smtp-authentication-client-in-postfix/

Regards Edward