empty rcpt TO with 1.4.4 on python2.7 (send: 'rcpt TO:<>\r\n')

[tractickets@…]

a trac instance of mine was running in version 1.2.2 for quite some time and worked flawlessly along email2trac.

It's hosted at uberspace and it used to run on python2.6 On 2024-07-25 they swapped /usr/bin/python to point to python2.7 and my mail issues began.

During my debug journey I upgraded to trac 1.4.4 but when trac wants to send mails (on ticket updates) I get Warning: The change has been saved, but an error occurred while sending notifications: {u'mail@mymail.com': (501, 'Syntax error in parameters or arguments'), u'mail@mymail.com': (501, 'Syntax error in parameters or arguments'), u'mail@mymail.com': (501, 'Syntax error in parameters or arguments')}

So I added

622             server.set_debuglevel(1)

to "python2.7/site-packages/trac/notification/mail.py"

and can now see the smtp dialog which is:

reply: '235 Authentication succeeded\r\n'
reply: retcode (235); Msg: Authentication succeeded
send: 'mail FROM:<sender@mymail.com> size=1697\r\n'
reply: '250 Requested mail action okay, completed\r\n'
reply: retcode (250); Msg: Requested mail action okay, completed
send: 'rcpt TO:<>\r\n'
reply: '501 Syntax error in parameters or arguments\r\n'
reply: retcode (501); Msg: Syntax error in parameters or arguments
send: 'rcpt TO:<>\r\n'
reply: '501 Syntax error in parameters or arguments\r\n'
reply: retcode (501); Msg: Syntax error in parameters or arguments
send: 'rcpt TO:<>\r\n'
reply: '501 Syntax error in parameters or arguments\r\n'
reply: retcode (501); Msg: Syntax error in parameters or arguments
send: 'rset\r\n'
reply: '250 OK\r\n'

I have proof that the mailing of this instance worked fine with the previus version of uberspace's python and my trac 1.2.2 mid-july. We recognized the lack of mails Mid-August and because of

lrwxrwxrwx. 1 root root 7 25. Jul 19:34 /usr/bin/python -> python2

my guess is, that it's connected to the swap of the python link. Unfortunately there is no more python2.6 left on the box which seems to run on CentOS Linux release 7.9.2009 (Core)

If I remember correctly, 1.2.2 and python27 had the issue with an empty 'from' field in the smtp dialog

send: 'mail FROM:<> size=1697\r\n'

causing 550, 'Requested action not taken: mailbox unavailable\nSender address is not allowed.

[anonymous]

my trac.ini has

[notification]
email_sender = SmtpEmailSender
smtp_enabled = enabled
smtp_from = sender@mymail.com
smtp_from_name = IT-Support Ticket-System
smtp_password = xxx
smtp_port = 587
smtp_replyto = sender@mymail.com
smtp_server = smtp.ionos.de
smtp_user = sender@mymail.com
use_public_cc = enabled
use_tls = enabled

[tractickets@…]

anything I could/should check on the python installation?

I also tried to upgrade to 1.6 with python3 (tried 3.6 and 3.11) and then smtp error is gone and mailing out of trac works like a charm again. But I need to have the email2trac tool running and that's not working with 1.6 yet

[tractickets@…]

trac.log has

2024-09-04 21:40:51,024 Trac[mail] INFO: Sending notification through SMTP at smtp.ionos.de:587 to [u'user@mymail.com', u'user1@mymail.com', u'user2@mymail.com']
2024-09-04 21:40:51,196 Trac[api] ERROR: Failure distributing event <TicketChangeEvent realm='ticket', category='changed', target=<Ticket 1916>, time=datetime.datetime(2024, 9, 4, 19, 40, 50, 903230, tzinfo=<FixedOffset "UTC" 0:00:00>), author=u'My.Name'>
Traceback (most recent call last):
  File "/home/user/.local/lib/python2.7/site-packages/trac/notification/api.py", line 380, in notify
    self.distribute_event(event, self.subscriptions(event))
  File "/home/user/.local/lib/python2.7/site-packages/trac/notification/api.py", line 408, in distribute_event
    distributor.distribute(transport, recipients, event)
  File "/home/user/.local/lib/python2.7/site-packages/trac/notification/mail.py", line 510, in distribute
    self._do_send(transport, event, message, cc_addrs, bcc_addrs)
  File "/home/user/.local/lib/python2.7/site-packages/trac/notification/mail.py", line 589, in _do_send
    notify_sys.send_email(from_addr, list(to_addrs), message.as_string())
  File "/home/user/.local/lib/python2.7/site-packages/trac/notification/api.py", line 372, in send_email
    self.email_sender.send(from_addr, recipients, message)
  File "/home/user/.local/lib/python2.7/site-packages/trac/notification/mail.py", line 644, in send
    server.sendmail(from_addr, recipients, message)
  File "/usr/lib64/python2.7/smtplib.py", line 746, in sendmail
    raise SMTPRecipientsRefused(senderrs)
SMTPRecipientsRefused: {u'user@mymail.com': (501, 'Syntax error in parameters or arguments'), u'user1@mymail.com': (501, 'Syntax error in parameters or arguments'), u'user2@mymail.com': (501, 'Syntax error in parameters or arguments')}
2024-09-04 21:40:51,196 Trac[web_ui] ERROR: Failure sending notification on change to ticket #1916: SMTPRecipientsRefused: {u'user@mymail.com': (501, 'Syntax error in parameters or arguments'), u'user1@mymail.com': (501, 'Syntax error in parameters or arguments'), u'user2@mymail.com': (501, 'Syntax error in parameters or arguments')}

[Jun Omae]

It seems to be caused by your email addresses and smtplib.quoteaddr implementation between Python 2.6 and 2.7. Make sure your email addresses are valid.

E.g.:

$ /usr/bin/python2.7 -c 'import smtplib as s; print(s.quoteaddr("aaaa@bbbb@domain"))'
<>
$ /usr/bin/python2.7 -c 'import smtplib as s; print(s.quoteaddr("aaaa@bbbb"))'
<aaaa@bbbb>
$ /usr/bin/python2.6 -c 'import smtplib as s; print(s.quoteaddr("aaaa@bbbb@domain"))'
<aaaa@bbbb>
$ /usr/bin/python2.6 -c 'import smtplib as s; print(s.quoteaddr("aaaa@bbbb"))'
<aaaa@bbbb>

From smtplib.py:

476     def mail(self, sender, options=[]):
477         """SMTP 'mail' command -- begins mail xfer session."""
478         optionlist = ''
479         if options and self.does_esmtp:
480             optionlist = ' ' + ' '.join(options)
481         self.putcmd("mail", "FROM:%s%s" % (quoteaddr(sender), optionlist))
482         return self.getreply()
483
484     def rcpt(self, recip, options=[]):
485         """SMTP 'rcpt' command -- indicates 1 recipient for this mail."""
486         optionlist = ''
487         if options and self.does_esmtp:
488             optionlist = ' ' + ' '.join(options)
489         self.putcmd("rcpt", "TO:%s%s" % (quoteaddr(recip), optionlist))
490         return self.getreply()

[tractickets@…]

Thanks for getting back so quick. Can you give me a hint how/where my email addresses are invalid?

They all are in the format xx@domain.tld and I just replaced them in the logs for this ticket.

The very same trac environment with the same trac.ini works in trac 1.6

Anything in detail I can check? Thanks!

[Jun Omae]

You can check using the following code:

$ python -c 'import sys, smtplib as s; print("\n".join("%r => %r" % (a, s.quoteaddr(a)) for a in sys.argv[1:]))' 'your-address-1st@domain.tld' 'your-address-2nd@domain.tld'
'your-address-1st@domain.tld' => '<your-address-1st@domain.tld>'
'your-address-2nd@domain.tld' => '<your-address-2nd@domain.tld>'

Otherwise, please let me know the email addresses in the following log line.

2024-09-04 21:40:51,024 Trac[mail] INFO: Sending notification through SMTP at smtp.ionos.de:587 to [...]

If you don't want to make them public, directly mail to jun66j5@gmail.com.

[tractickets@…]

thank for your reply, I've sent you a mail. The format of the mail addresses looks inconspicuous to me.

[Jun Omae]

Thanks for the mailing directly.

I just checked email addresses using smtplib.quoteaddr() on CentOS 7.9 but are correct. Your issue is unable to reproduce.

Could you please let me know installed libraries (pip list) and installed Trac plugins in your environment? I'm suspecting some kind of plugins do monkey patching to smtplib.quoteaddr() and/or email.utils.parseaddr().

[anonymous]

I removed all files in the plugins-dir and installed the missing via pip.

pip list shows

Package                          Version
-------------------------------- -------
Babel                            2.9.1
backports.ssl-match-hostname     3.5.0.1
Beaker                           1.5.4
boto                             2.45.0
cffi                             1.6.0
chardet                          2.2.1
cryptography                     1.7.2
decorator                        3.4.0
docutils                         0.18.1
duplicity                        0.7.19
enum34                           1.0.4
fasteners                        0.16.3
fstab                            1.4
Genshi                           0.7
getmail                          5.13
Glances                          2.5.1
GnuPGInterface                   0.3.2
google-api-python-client         1.6.3
httplib2                         0.18.1
idna                             2.4
iniparse                         0.4
iotop                            0.6
ipaddress                        1.0.16
IPy                              0.75
javapackages                     1.0.0
Jinja2                           2.11.3
keyring                          5.0
kitchen                          1.1.1
lockfile                         0.11.0
lxml                             3.2.1
Magic-file-extensions            0.2
Mako                             0.8.1
Markdown                         3.1.1
MarkupSafe                       1.1.1
monotonic                        0.1
MySQL-python                     1.2.5
oauth2client                     4.0.0
paramiko                         2.1.1
passlib                          1.7.4
Paste                            1.7.5.1
pexpect                          2.3
Pillow                           2.0.0
pip                              20.0.2
ply                              3.4
policycoreutils-default-encoding 0.1
psutil                           5.6.7
pyasn1                           0.1.9
pyasn1-modules                   0.0.8
pycparser                        2.14
pycurl                           7.19.0
PyDrive                          1.3.1
Pygments                         2.1.3
pygobject                        3.22.0
pygpgme                          0.3
pyliblzma                        0.5.3
PyMySQL                          0.9.3
pyOpenSSL                        0.13.1
pyparsing                        1.5.6
python-dateutil                  1.5
pytz                             2024.1
pyxattr                          0.5.1
PyYAML                           3.10
requests                         2.6.0
rsa                              3.4.2
s3cmd                            2.3.0
seobject                         0.1
sepolicy                         1.1
setuptools                       23.1.0
six                              1.10.0
slip                             0.4.0
slip.dbus                        0.4.0
Tempita                          0.5.1
textile                          2.3.2
Trac                             1.4.4
TracAccountManager               0.6.0
TracIniAdminPanel                1.4.1
TracMarkdownMacro                0.11.9
TracPermRedirect                 3.0
uritemplate                      3.0.1
urlgrabber                       3.10
urllib3                          1.10.2
virtualenv                       15.1.0
yum-metadata-parser              1.1.4

[tractickets@…]

so I setup a minimal test.py

import smtplib

...smtp_user, smtp_password, smtp_from copied from trac.ini and quoted the values

server = smtplib.SMTP(smtp_server)
server.starttls()
server.set_debuglevel(True)
server.login(smtp_user, smtp_password)

if I use (like mentioned in the trac.log)

server.sendmail(smtp_from, u'user@mydomain.com', message)

I get the very same

smtplib.SMTPRecipientsRefused: {u'user@mydomain.com': (501, 'Syntax error in parameters or arguments')}

but when I don't use a unicode string but an ascii one:

server.sendmail(smtp_from, 'user@mydomain.com', message)

it works.

So I added

from_addr.encode('ascii', 'ignore')
recipients = [s.encode('ascii', 'ignore') for s in recipients]

to ~/.local/lib/python2.7/site-packages/trac/notification/mail.py and my trac instance is able to send mails again. yay :-)

I have no clue if this python2.7 installation is broken or what else is going wrong

[tractickets@…]

forgot to add the conversion of from_addr - that also needs to be switched from unicode to ascii

         if isinstance(from_addr, (str, unicode)):
             from_addr = from_addr.encode('ascii', 'ignore')
         if isinstance(from_addr, list):
             from_addr = [s.encode('ascii', 'ignore') for s in from_addr]

still puzzled why it seems to be only me having this issue

[Jun Omae]

I just re-check it using CentOS 7 Docker image, however not reproduced.

$ dokcer run -it quay.io/centos/centos:centos7 /bin/bash
[root@8894f7417bc7 /]# python
Python 2.7.5 (default, Oct 14 2020, 14:45:30)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smtplib
>>> s = smtplib.SMTP('192.168.11.122')
>>> s.ehlo()
(250, 'localhost\nPIPELINING\nSIZE 10240000\nVRFY\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN\nCHUNKING')
>>> s.sendmail(u'root@localhost.localdomain', [u'user1@localhost.localdomain', u'user2@localhost.localdomain'], '')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/smtplib.py", line 746, in sendmail
    raise SMTPRecipientsRefused(senderrs)
smtplib.SMTPRecipientsRefused: {u'user1@localhost.localdomain': (554, '5.7.1 <user1@localhost.localdomain>: Recipient address rejected: Access denied'), u'user2@localhost.localdomain': (554, '5.7.1 <user2@localhost.localdomain>: Recipient address rejected: Access denied')}
>>>

I have no clue if this python2.7 installation is broken or what else is going wrong

At least, it is able to verify the installation being correct using rpm -V like the following:

[root@8894f7417bc7 /]# rpm -V python-libs; echo $?
0
[root@8894f7417bc7 /]# cp /dev/null /usr/lib64/python2.7/smtplib.py
cp: overwrite '/usr/lib64/python2.7/smtplib.py'? y
[root@8894f7417bc7 /]# rpm -V python-libs; echo $?
S.5....T.    /usr/lib64/python2.7/smtplib.py
1

[tractickets@…]

unfortunately I'm not root on the machine but rpm -V python-libs; echo $? returns with 0

[tractickets@…]

I wanted to compare the content of the installed version of /usr/lib64/python2.7/smtplib.py on the server running trac

so on the way I asked myself how in the world can those guys run such an old stuff without having tons of security issues and then I found out they use:
CentOS 7 Extended Lifecycle Support by TuxCare

and they have

rpm -qa | grep python-2.7.5
python-2.7.5-94.el7_9.tuxcare.els2.x86_64

and there is an info about this specific package-version:
​https://cve.tuxcare.com/els/releases/CLSA-2024:1711491407

Changelog

• CVE-2023-27043: reject malformed addresses in email.parseaddr()

and the guys at redhat write: ​https://access.redhat.com/articles/7051467

With the fix applied, the getaddresses and parseaddr functions from the email.utils module now include a new strict keyword argument to control the stricter behavior introduced by the fix.

The strict keyword argument can take the following values:

True (default) - the parsing is stricter to enhance security. False - the functions revert to the previous less strict, less secure behavior.

here is the diff between std-python 2.7.5 from 2013 and the version on the server

email/utils.py

@@ -100 +100 @@
     return address
 
 
-
 
-def getaddresses(fieldvalues):
-    """Return a list of (REALNAME, EMAIL) for each fieldvalue."""
-    all = COMMASPACE.join(fieldvalues)
-    a = _AddressList(all)
-    return a.addresslist
+def _iter_escaped_chars(addr):
+    pos = 0
+    escape = False
+    for pos, ch in enumerate(addr):
+        if escape:
+            yield (pos, '\\' + ch)
+            escape = False
+        elif ch == '\\':
+            escape = True
+        else:
+            yield (pos, ch)
+    if escape:
+        yield (pos, '\\')
 
 
-
 
+def _strip_quoted_realnames(addr):
+    """Strip real names between quotes."""
+    if '"' not in addr:
+        # Fast path
+        return addr
+
+    start = 0
+    open_pos = None
+    result = []
+    for pos, ch in _iter_escaped_chars(addr):
+        if ch == '"':
+            if open_pos is None:
+                open_pos = pos
+            else:
+                if start != open_pos:
+                    result.append(addr[start:open_pos])
+                start = pos + 1
+                open_pos = None
+
+    if start < len(addr):
+        result.append(addr[start:])
+
+    return ''.join(result)
+
+
+supports_strict_parsing = True
+
+def getaddresses(fieldvalues, strict=True):
+    """Return a list of (REALNAME, EMAIL) or ('','') for each fieldvalue.
+
+    When parsing fails for a fieldvalue, a 2-tuple of ('', '') is returned in
+    its place.
+
+    If strict is true, use a strict parser which rejects malformed inputs.
+    """
+
+    # If strict is true, if the resulting list of parsed addresses is greater
+    # than the number of fieldvalues in the input list, a parsing error has
+    # occurred and consequently a list containing a single empty 2-tuple [('',
+    # '')] is returned in its place. This is done to avoid invalid output.
+    #
+    # Malformed input: getaddresses(['alice@example.com <bob@example.com>'])
+    # Invalid output: [('', 'alice@example.com'), ('', 'bob@example.com')]
+    # Safe output: [('', '')]
+
+    if not strict:
+        all = COMMASPACE.join(unicode(v) for v in fieldvalues)
+        a = _AddressList(all)
+        return a.addresslist
+
+    fieldvalues = [unicode(v) for v in fieldvalues]
+    fieldvalues = _pre_parse_validation(fieldvalues)
+    addr = COMMASPACE.join(fieldvalues)
+    a = _AddressList(addr)
+    result = _post_parse_validation(a.addresslist)
+
+    # Treat output as invalid if the number of addresses is not equal to the
+    # expected number of addresses.
+    n = 0
+    for v in fieldvalues:
+        # When a comma is used in the Real Name part it is not a deliminator.
+        # So strip those out before counting the commas.
+        v = _strip_quoted_realnames(v)
+        # Expected number of addresses: 1 + number of commas
+        n += 1 + v.count(',')
+    if len(result) != n:
+        return [('', '')]
+
+    return result
+
+
+
 ecre = re.compile(r'''