Notification system silently drops failures to unicode email addresses

[John Hampton]

See #9082 for some background.

Currently, email addresses with unicode in them are silently discarded when sending ticket updates. While not trying to deliver the email is correct, it's a little misleading. If the user enters a unicode email address in the CC list, then it appears to get added (indeed it does) but they will never receive any emails. Unfortunately, there is no indication as to why the email will never be delivered. An admin can look in the log and see the email addresses which have been discarded, but they first have to know what they are looking for. Additionally, the same thing occurs for addresses in the smtp_always_cc and smtp_always_bcc fields.

We could show a warning indicating that the email wasn't delivered to the problem addresses. The problem with the CC list, is that the invalid address is put on the CC list, and so anytime an update occurred, the warning would be shown. An administrator would then need to clean out the CClist. The problem is even more complicated if a user has the invalid email address in their preferences and simply puts their username on the CC list (or as the owner/reporter/updater of a ticket). An admin would then need to clear out the email address in the user's session.

I think the solution is to warn when an email can't be delivered, as well as stricter input validation. Input validation would need to be in:

Preferences

when entering email address

CC list

would need to differentiate between entering an email address and a username. This might be really hard when taking into account use_short_addr and smtp_default_domain

smtp_always_cc and smtp_always_bcc

Don't know how to properly validate these, or whether it's worth it. Populating these with invalid values should be readily apparent to the admin.

[Christian Boos]

Milestone 0.13 only when there's a patch or a strong commitment to implement it for that release.

[Thijs Triemstra]

Added this to the TracDev/Proposals/EmailValidation proposal.

[Carsten Klein <carsten.klein@…>]

Thanks for the input.

Feel free to comment/review the notice I have provided under discussion/Additional Requirements and Non Requirements.

[Carsten Klein <carsten.klein@…>]

Yet, however, I fear that the discard of the email addresses is an MTA related one.

Do you use the smtp or the sendmail based mailer?

Does your MTA support unicode addresses with umlauts etc?

[Carsten Klein <carsten.klein@…>]

Reading through ​http://en.wikipedia.org/wiki/Email_address it seems that UTF-8 support in smtp headers is still in the works.

Thus, malformed e-mail addresses should definitely be prevented from entering into the system.

For smtp_always_cc and smtp_always_bcc I would definitely log a warning in the log files, for CC the ticket system must be adjusted to support this.

In the preferences module this must also be prevented for now.

+1 from me for this for the next major release or earlier if this can be done in a reasonable amount of time

[Carsten Klein <carsten.klein@…>]

Alas, the regular expression for validating email addresses on input is rather complex, see ​http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html.

[Carsten Klein <carsten.klein@…>]

I just had a good look at the rfc822 module.

The rfc822.parseaddr() function will parse addresses according to RFC822. Based on some arbitrary unicode input including umlauts I found that it will encode these using the \x.. notation.

However, and due to standard set forth in RFC822 nearly all input goes through, which basically also means that we cannot use this for pre-validation of a given address but we might use this for encoding the addresses to a format that is more compliant with existing mailers, for as long as they do not support the unicode character set natively in the headers.

But see for yourselves…

# -*- coding: utf-8 -*-

from rfc822 import parseaddr as p

e = ['öäü@üöö.cöm', 'Foo Bar <foo (grandmaster of all evil)@bar.com>', '"does this make any sense"@@not.at.all']

for a in e:
  print p(a)