Subject of notification email too long - add limit

[trac-trac@…]

I received a Trac notification email with a subject line like this:

[Pidgin] Batch modify: #4933, #5329, #5583, #5696, #6521, #6751,
 #6881, #6988, #7110, #7145, #7304, #7626, #7999, #8023, #8040,
 #8075, #8135, #8201, #8220, #8238, #8321, #8323, #8341, #8369,
 #8371, #8382, #8388, #8414, #8467, #8507, #8547, #8564, #8599,
 #8630, #8647, #8656, #8680, #8689, #8699, #8701, #8714, #8753,
 #8778, #8788, #8829, #8854, #8919, #8967, #8984, #8993, #9009,
 #9020, #9035, #9043, #9077, #9099, #9110, #9136, #9144, #9163,
 #9167, #9185, #9189, #9194, #9195, #9201, #9225, #9230, #9237,
...

The total length of the subject line was 5546 characters. Such long subject lines are not well-handled by email clients. Eg. Thunderbird doesn't allow the message to be viewed without a workaround. Trac should impose some hard limit on the length of the subject line for all emails. Subjects longer than that should be truncated with "…" appended.

For this specific email (batch-modification) I'd also suggest not listing all ticket numbers in the subject line if there are more than X of them.

[Ryan J Ollos]

I noticed some strange behavior of trac.util.text:shorten_line.

>>> from trac.util.text import shorten_line
>>> sl = shorten_line('0123456789', 10)
>>> print sl
'0123456789 ...'
>>> len(sl)
14
>>> sl = shorten_line('0123456789', 9)
>>> print sl
012345678 ...
>>> len(sl)
13

• When the length of the input text is maxlen, the text is still shortened.
• The shortened text does not have a length less than or equal to maxlen.

I'm considering the following patch:

trac/util/text.py

@@ -418 +418 @@
     This tries to be (a bit) clever and attempts to find a proper word
     boundary for doing so.
     """
-    if len(text or '') < maxlen:
+    if len(text or '') <= maxlen:
         return text
-    cut = max(text.rfind(' ', 0, maxlen), text.rfind('\n', 0, maxlen))
+    suffix = ' ...'
+    maxtextlen = maxlen - len(suffix)
+    cut = max(text.rfind(' ', 0, maxtextlen), text.rfind('\n', 0, maxtextlen))
     if cut < 0:
-        cut = maxlen
-    return text[:cut] + ' ...'
+        cut = maxtextlen
+    return text[:cut] + suffix
 
 
 class UnicodeTextWrapper(textwrap.TextWrapper):

Which results in:

>>> from trac.util.text import shorten_line
>>> sl = shorten_line('0123456789', 10)
>>> print sl
0123456789
>>> len(sl)
10
>>> sl = shorten_line('0123456789', 9)
>>> print sl
01234 ...
>>> len(sl)
9

I'll look more closely at existing places where shorten_line is used before making the changes. Typically I wouldn't suggest making changes to the behavior of the public API like this, but the behavior of shorten_line seems rather poor and I think we should fix it.

Unit tests would be added before committing changes to shorten_line.

As for the primary fix needed for this ticket, after a quick search I found ​SO:1592291, which seems to suggest that we limit the length of the subject of a notification email to 78 characters.

[Ryan J Ollos]

Replying to rjollos:

[…]

• When the length of the input text is maxlen, the text is still shortened.
• The shortened text is not length maxlen

The characteristics of the function have existed since it was introduced in [249].

[Peter Suter]

• When the length of the input text is maxlen, the text is still shortened.
• The shortened text is not length maxlen

I agree, that's not what I would have expected.

I'll look more closely at existing places where shorten_line is used before making the changes. Typically I wouldn't suggest making changes to the behavior of the public API like this, but the behavior of shorten_line seems rather poor and I think we should fix it.

Right, we should be careful not to break public API's needlessly. But here I can't even find any caller that explicitly specifies the maxlen. Probably all that's wanted is just a reasonably short message, without any specific requirements that need to be preserved. So it seems fine to improve the behavior as long as it stays reasonably similar.

As for the primary fix needed for this ticket, after a quick search I found ​SO:1592291, which seems to suggest that we limit the length of the subject of a notification email to 78 characters.

I would maybe even just go with the default of 75. 78 seems more like an approximate recommendation than an exact standard. For example someone in that discussion mentions Outlook will cut subjects >77.

Any idea if / how encodings affect this limit in practice?

Would a simple change in NotifyEmail​ to headers['Subject'] = shorten_line(self.subject) make sense?

[Jun Omae]

Would a simple change in NotifyEmail​ to headers['Subject'] = shorten_line(self.subject) make sense?

No. The limitation is very very short for CJK users. I don't think NotifyEmail should have any limitations.

I think we should cut only subject of batch-modify notification.

The following is real example. The subject has 68 characters (104 bytes) and 5 lines in mime encoding.

[tracpath] #1873: ExcelDownloadPlugin を Excel 2007 形式でエクスポートできるようにする

Subject: =?utf-8?q?=5Btracpath=5D_=231873=3A_ExcelDownloadPlugin_?=
	=?utf-8?q?=E3=82=92_Excel_2007_=E5=BD=A2=E5=BC=8F=E3=81=A7?=
	=?utf-8?q?=E3=82=A8=E3=82=AF=E3=82=B9=E3=83=9D=E3=83=BC=E3=83=88?=
	=?utf-8?q?=E3=81=A7=E3=81=8D=E3=82=8B=E3=82=88=E3=81=86=E3=81=AB?=
	=?utf-8?q?=E3=81=99=E3=82=8B?=

[Peter Suter]

No. The limitation is very very short for CJK users. I don't think NotifyEmail should have any limitations.

Ah, I suspected that. Thank you for the confirmation.

The following is real example. The subject has 68 characters

So shorten_line with maxlen=75 would leave it unchanged? (As mime encoding happens later​.)

---

(By the way, I got Re: [The Trac Project] #11164: Git persistent cache not refreshed on changeset added notification from another process recently (118 characters) and Thunderbird seemed to be fine with it.)

[Ryan J Ollos]

Replying to psuter:

Would a simple change in NotifyEmail​ to headers['Subject'] = shorten_line(self.subject) make sense?

That's the change I was going to propose, however I hadn't considered potential problems with languages other than English. I guess that is still unclear after your comment:6. Shortening simply by searching for Ascii whitespace and linefeed characters seems problematic for non-Latin languages based on the very little I know about how some non-Latin languages separate words. At a minimum, it seems shorten_line should truncate on Unicode whitespace.

If we shorten the subject for all emails, we might want to put the full Summary in the body of the email. Another consideration is that we would probably want the shortened summary to be the same before and after the Re: is prefixed in trunk/trac/ticket/notification.py@10285:171#L161​, otherwise email clients might not group the first and subsequent messages (i.e. it wouldn't treat Word Word Word ... and Re: Word Word ... as the same message).

Just fixing this for Batch notifications is looking quite a bit easier than the alternative!

[Ryan J Ollos]

Proposed changes can be found in log:rjollos.git:t11350.

[trac-trac@…]

Does this fix affect emails other than batch notification? I'd still suggest that there should be some hard limit for all emails - probably over 75 characters, but under 5546.

[Ryan J Ollos]

The proposed change does not shorten the summary for emails other than batch notification.

Trac allows configuring a limit on the length of the ticket description and comments, but not on the ticket summary: TracIni#ticket-section. A related possibility would be to add a [ticket] max_summary_size parameter.

[Ryan J Ollos]

Committed to 1.0-stable in [12237:12239] and merged to trunk in [12240].

In the future I'll look into limiting the max size of the summary for all notifications, or adding an option to limit the max size of the ticket summary.

[Ryan J Ollos]

Replying to rjollos:

In the future I'll look into limiting the max size of the summary for all notifications, or adding an option to limit the max size of the ticket summary.

See #11472.