alternate notification submission (non-SMTP) using sendmail program

[vivek@…]

It would be helpful to have a notification mechanism that uses /usr/sbin/sendmail to inject mail instead of using SMTP.

The server on which I run trac has no SMTP server, so I have trac pointing to our main SMTP server at the office. That server locked up yesterday causing trac notifications to stall the trac web pages until they timed out the SMTP connection. Those notices were also lost. If we had submission via /usr/sbin/sendmail, the local nullmailer queue would deal with retries. This also allows us to centralize the mail server routing on the server in one place (the nullmailer configuration), rather than on every application.

I'd prefer not to run a full SMTP server on the trac box for administrative reasons, but if I have to, I have to, I suppose.

[Emmanuel Blot]

I don't think it is worth adding another dependency to the Trac core to circumvent buggy environments.

Maybe this could be implemented as a plugin - although there is no extension point at the moment to bypass the default notification system.

You can customize trac/notification.py to use a local tool instead of a TCP connection.

Another solution would be to install a SMTP server on your Trac server that only listen on localhost (127.0.0.1), and forward requests to your main server.

I'm not sure to understand the point about email routing (?)

[Emmanuel Blot]

I would recommend to close this ticket as wontfix.

[nick+trac@…]

For what it's worth, I'd also like this feature for the same reasons the ticket originator mentioned. It's simpler, easier to administrate, more performant, and fails more gracefully. I would say that an application with a dependency on an SMTP server when it has sendmail available might be considered the "buggy environment", but I suppose that is a matter of opinion. ;-)

Also, I don't think that this is a dependency that would really matter in Trac core. If someone doesn't have sendmail in their $PATH, then they should configure Trac to use SMTP. However, if someone were to create a plugin, I'd use that.

At any rate, I can solve the bulk of the problem by simply pointing the config to localhost. So this isn't urgent. But please don't throw away this ticket as wontfix.

[Emmanuel Blot]

Replying to nick+trac@ekenosen.net:

I would say that an application with a dependency on an SMTP server when it has sendmail available might be considered the "buggy environment", but I suppose that is a matter of opinion. ;-)

I would say the exact opposite: spawning an external process - which is OS-dependent - on each notification event, whereas a standard library than comes with Python is available would have been a weird choice.

I'd really see this feature implemented as a plugin rather than be integrated in the core.

[Christian Boos]

So maybe one day someone will contribute a patch for this (a plugin would first need some e-mail sending abstraction changes in the NotifyEmail class).

[Emmanuel Blot]

Replying to cboos:

So maybe one day someone will contribute a patch for this (a plugin would first need some e-mail sending abstraction changes in the NotifyEmail class).

Yes, I guess that some kind of notification API is required to implement alternate notification scheme here.

[anonymous]

While I grant that having raw SMTP client embedded in a cross-platform application that needs to send email makes sense, the ability to have an alternative mechanism of calling an external program (which just takes the message on its STDIN) is also quite valuable. In my case outbound port 25 is blocked by firewall policy and I have to use an alternative (non-SMTP) mechanism to get mail injected to the outside world (you really don't want to know the grody details). Yes, the specific program will be machine-specific, but the *interface* to the program is simple and portable.

Unfortunately, I'm not a Python programmer, so the patch I'm attaching probably does not quite work, but it should be close enough that one who is familiar with the language can clean it up easily…

[Emmanuel Blot]

The proposed patch being platform-dependent (use of sendmail), I don't think it can get integrated in Trac core as-is.

BTW, another alternative might be to duplicate the notification test code that comes along with Trac and re-use the mini SMTP server to serve Trac notification requests and redirect them to the desired tool.

A notification API to allow Trac notification plugins would nevertheless be the best option.

[anonymous]

"use of sendmail" is just the freaking default. It is *any* external program that can take a rfc822 message on stdin and deal with it! On *any* platform that supports popen (which is part of the python core).

[anonymous]

This is actually a security issue. Circumventing 'sendmail' prevents the local mail system from identifying the sender UID (you cannot identify the sender UID over a TCP socket, not even on the local host).

Since you may want to limit any action that could be performed by an intruder who has obtained the webserver UID (through some arbitrary vulnerability in some arbitrary webapp), you definitely want to identify, log, and eventually filter emails sent by the webserver user.

Avoiding the 'sendmail' interface to the local mail system is a major design flaw that sabotages efforts to limit the potential damage from an intrusion. Port 25 should be blocked for local users, to prevent anonymous mail injection.

[Emmanuel Blot]

Replying to anonymous:

Since you may want to limit any action that could be performed by an intruder who has obtained the webserver UID (through some arbitrary vulnerability in some arbitrary webapp), you definitely want to identify, log, and eventually filter emails sent by the webserver user.

You can do this with an SMTP authenticated session, can't you?

[anonymous]

Replying to eblot:

Replying to anonymous:

Since you may want to limit any action that could be performed by an intruder who has obtained the webserver UID (through some arbitrary vulnerability in some arbitrary webapp), you definitely want to identify, log, and eventually filter emails sent by the webserver user.

You can do this with an SMTP authenticated session, can't you?

Yes, except that it needlessly blows up the complexity of the mail system - need to ensure that webserver user can auth from inside only, while other users can auth from outside; need to take care that webserver user cannot auth to services that share same authentication scheme (IMAP); maybe more loopholes to close. All for the sake of a single broken application that doesn't use the interface intended for local users.

[Emmanuel Blot]

Replying to anonymous:

Yes, except that it needlessly blows up the complexity of the mail system

Is it better to blow up the complexity of Trac, with a new bunch of options and settings, add a non-portable implementation - as it requires a Unix tool - for the sake of avoiding an SMTP server configuration? I'm not sure.

All for the sake of a single broken application that doesn't use the interface intended for local users.

Not supporting a platform-specific externality (sendmail), not mentionning the performance issues (spawning new processes from Python) does not appear as a broken feature to me. At worst it's a lack of a feature.

[nick+trac@…]

(I'm not the same as the anonymous poster, above)

Replying to eblot:

Replying to anonymous:

Yes, except that it needlessly blows up the complexity of the mail system

Is it better to blow up the complexity of Trac

IMHO, yes, yes, absolutely yes. It is more important for the deployment to be simple and able to be easily customizable to common deployment practice than for Trac to delegate a few lines of code to a plugin.

But does it really "blow up" the complexity to have a separate configuration that does everything the same as the other mail code except for the delivery?

, with a new bunch of options and settings, add a non-portable implementation - as it requires a Unix tool - for the sake of avoiding an SMTP server configuration? I'm not sure.

There are sendmail compatible programs for every platform out there, and a google search for "sendmail windows" turns up many results. True, it isn't commonly done that way, but I doubt that it is really as non-portable as you say.

All for the sake of a single broken application that doesn't use the interface intended for local users.

Not supporting a platform-specific externality (sendmail), not mentionning the performance issues (spawning new processes from Python) does not appear as a broken feature to me.

I have a hard time believing that shelling out to sendmail and writing to STDOUT, which is promptly saved to a queue on disk is considerably worse performance than opening a network connection and waiting for the SMTP server to respond, etc. In my experience (primarily with other tools written in other languages), going the sendmail route is consistently the more performant and scalable solution.

At worst it's a lack of a feature.

You have no argument from me there. It's just a puzzling lack of a feature. ;-)

I'm really sorry I can't follow my argument up with a patch… I looked into it a while back, but my Python skills are very sub-par, and I simply haven't had the time. But it seems like it would be an easy thing to do.

[Emmanuel Blot]

Replying to nick+trac@ekenosen.net:

But does it really "blow up" the complexity to have a separate configuration that does everything the same as the other mail code except for the delivery?

Two common issues with Trac:

• external dependencies (sendmail would be another one, and third party sendmail for Windows is even worse)
• configuration options

True, it isn't commonly done that way, but I doubt that it is really as non-portable as you say.

It brings another source of trouble, another optional piece of code to validate and support, another bunch of configuration issues. I'm not saying it is not worth to support it, I just say that is far from being as simple as it may look like.

I have a hard time believing that shelling out to sendmail and writing to STDOUT,

This can bring another bunch of troubles, as several Trac threads may run within the same process, among other things.

But it seems like it would be an easy thing to do.

Trac todo list is full of "easy things to do" ;-)

I think this thread is going round in circles.
Hopefully some other Trac developers are willing to support sendmail in a future Trac release. From my perspective this is still a plugin candidate or a long-term feature.

[nick+trac@…]

Replying to eblot:

Two common issues with Trac:

• external dependencies (sendmail would be another one, and third party sendmail for Windows is even worse)
• configuration options

Understood. But it isn't really a "dependency". It's an optional "if it makes sense for you, then you can use it". :-)

It brings another source of trouble, another optional piece of code to validate and support, another bunch of configuration issues. I'm not saying it is not worth to support it, I just say that is far from being as simple as it may look like.

I very much understand and sympathize with that.

But it seems like it would be an easy thing to do.

Trac todo list is full of "easy things to do" ;-)

This too. :-)

I think this thread is going round in circles.

Understood. I'll not make my case again. ;-)

Hopefully some other Trac developers are willing to support sendmail in a future Trac release. From my perspective this is still a plugin candidate or a long-term feature.

It isn't a high priority for your deployment scenarios, and there is some outlay of time and risk. That makes sense.

But, it would be nice if someone in the know would take a look at the patch that's already been posted on the thread and give it their blessing, or list what could be done to include it into core (if someone were to submit a patch with appropriate unit tests, would that help you out?). And I'll cross my fingers that, in the near future, some core team member needs to deploy in an environment where delegating to sendmail is the standard, so they can champion this. ;-)

[anonymous]

Replying to nick+trac@ekenosen.net:

I have a hard time believing that shelling out to sendmail and writing to STDOUT, which is promptly saved to a queue on disk is considerably worse performance than opening a network connection and waiting for the SMTP server to respond, etc. In my experience (primarily with other tools written in other languages), going the sendmail route is consistently the more performant and scalable solution.

In my experience, trac is quite slow as compared to other applications that do a similar amount of work, and I think this thread is quite revealing. Running an full-blown SMTP session in a web app that eventually gets timed out by the web server, instead of just "fire-and-forget" to sendmail, sounds like an outrageous design error. If this is done for simplicity (less code), and if this attitude pervades through the entire trac application, the performance (or lack thereof) is in fact understandable :)

In related news, trac has an 'enscript_path' ini variable, but if this is empty (hey, why? perhaps because there is no enscript executable?) it nevertheless shells out trying to run enscript (enlisting the shell's help to eventually find enscript). Shows that (a) shelling out is not a problem, since it's done quite gratitiously, (b) platform specific utilities are not seen as a problem, even if there are indications that they may be lacking. Duh.

[wu-lee]

I'd like to add, this would be useful to me too. I was in fact quite surprised it wasn't already available as an option, Trac is usually quite flexible.

Speaking personally, I really don't want to create an account to log into the local SMTP server just for Trac, and write the password into the config - it requires far more work checking the account is secure than just allowing local execution of some sendmail equivalent.

If it complicates Trac's internals significantly, I'd also be very surprised, since writing a formatted email to STDOUT is trivial.

[mgl@…]

Very surprised to find no sendmail support. I vote that this be fixed. SMTP is unnecessary overhead, requires running an SMTP server (or at least making an account on an existing server), and puts a password into the conf file.

[Emmanuel Blot]

Replying to mgl@…:

SMTP is unnecessary overhead, requires running an SMTP server (or at least making an account on an existing server), and puts a password into the conf file.

? This depends on your SMTP server/environment. YMMV.

[mr.troll]

i think that sendmail is the primary mail programm on linux systems. it's strange that you don't support it

[Almad]

Just hit that too. Surprised it's not implemented yet.

[anonymous]

Yeah, I wonder too. Sendmail or switch sendmail/smtp is really de-facto standart for web apps. Just look around.

So, it would be good if…

[anonymous]

I'd like to add my support for this feature too.

My employer uses a central SMTP server which is administered by a separate IT group. Today that SMTP server stopped accepting connection, and thus Trac became unusable (3-5 minutes to create/change a ticket).

Running a local SMTP server may be an acceptable workaround for some, but in my case it defeats the purpose of running a central mail server. It's one more thing I have to setup and administer. That's very frustrating considering lots of other software uses sendmail (which was designed for this sort of thing) without a problem.

Additionally, perhaps the way to do this is allow for an arbitrary number of notification plug-ins. Besides email (via direct SMTP, or sendmail, etc), what if I wanted SMS text messages through a third plug-in? This also leaves room for an IRC notifications module…

[anonymous]

Another vote for sendmail support — for all the reasons listed above. Compared to the de facto standard of sendmail, using SMTP for this kind of deployment scenario is simply slow and problem prone.

It seems to me that this ticket certainly has enough screaming users to merit some attention from the core team, no?

[Remy Blank]

Patch against trunk adding an IEmailSender interface and two implementations

[Remy Blank]

Ok, I'll bite. The patch above introduces an IEmailSender interface that abstracts sending of an email, with two alternative implementations: the usual SMTP and (you guessed it) sendmail.

The patch introduces two new configuration options in TracIni:

• email_sender: specifies the name of the component that should be used for sending e-mails. Two components are provided: SmtpEmailSender (the default) and SendmailEmailSender.

• sendmail_path: the path to the Sendmail-compatible executable.

So here's the deal: if I get some positive feedback from people testing this patch on different platforms (especially Windows, which I don't have around), and if there are no strong objections from other developers, I'll commit this to trunk.

Note that I'm calling Sendmail with the following command-line:

sendmail -i {address1} {address2} ...

It would be great if a Sendmail expert here could confirm that this is correct.

[Christian Boos]

Looks good, if only to make people believe that Trac developers are not an extinct species ;-)

Now if someone can send me a link to a free and real sendmail.exe (not the so-called "fake" one from glob.com.au, which seems rather limited), I could probably test this on Windows.

On a semi-related note, seeing the usage of NaivePopen, I wonder if we couldn't switch to using subprocess on trunk?

[Remy Blank]

Replying to cboos:

On a semi-related note, seeing the usage of NaivePopen, I wonder if we couldn't switch to using subprocess on trunk?

That's what I had in mind initially, but then I ended up copying the code from the Enscript renderer (yes, I felt lazy). I'll try subprocess anyway, looks interesting enough.

[Remy Blank]

The patch above uses the subprocess module instead of NaivePopen, and also sets the envelope sender with the -f option when using Sendmail.

[Remy Blank]

Updated patch that uses the subprocess module

[Remy Blank]

I'm a bit disappointed at the lack of feedback, especially from the more vocal people on this ticket.

Anyway, the patch was applied in [7935], and I'll update the documentation shortly.

[Remy Blank]

Documentation updated in TracNotification@41.

[nicholas a. evans <nick+trac@…>]

Replying to rblank:

I'm a bit disappointed at the lack of feedback, especially from the more vocal people on this ticket.

Sorry, I've been too busy at work to verify this (plus I don't have easy access to any Windows computers for verifying that platform). But I'm quite pleased about it. :) Thanks.

[ian]

Replying to rblank:

I'm a bit disappointed at the lack of feedback, especially from the more vocal people on this ticket.

Anyway, the patch was applied in [7935], and I'll update the documentation shortly.

Thanks! I'm really looking forward to 0.12

[Mitar]

I am also glad sendmail support has been introduced. Thanks.

[anonymous]

Hello,

I had similar problems [smtp transaction were very long] and managed to update the trac/notification.py with a simple patch:

[this applies to funcion notify within NotifyEmail

— Notify.notify(self, resid) ++ #Notify.notify(self, resid) ++ thread.start_new_thread(Notify.notify, (self, resid))

And now mails are being send in the backgournd (separate thread) and the webpage is response if fast. [there is additional line: import thread required on top].

[disench@…]

[Tim H]

Replying to anonymous:

And now mails are being send in the backgournd (separate thread) and the webpage is response if fast. [there is additional line: import thread required on top].

[disench@…]

Wow! Thanks disench, I had exactly the same problem as others above with a slow SMTP server, and have to run Trac 0.11 for now, so had almost given up on ever getting notifications to work without blocking the ticket submit page for up to a minute while my SMTP server responded. But your patch to call Notify.notify in a thread was so simple and works brilliantly, now the page returns quickly and the email gets sent in the background like you said. Thanks for posting this useful hack, maybe for a future version to help others this should be a configurable option in trac.ini?