Add TracSpamFilter to tracopt

[Ryan J Ollos]

First mentioned in comment:8:ticket:12270, I'd like to consider moving SpamFilter to tracopt. The plugin is a critical piece of many Trac installation, particularly for open source projects such as trac-hacks.org and bugs.jqueryui.com. We can't afford to not maintain it, therefore I think it makes sense to just include it in the Trac distribution.

I think all the dependencies​ are compatible with the 3-Clause BSD license:

• ​dnspython: "BDS style"
• ​spambayes: PSF license
• ​Pillow: appears to be "BSD style" license
• ​oauth2: MIT license
• ​httplib2: MIT license

For a few months I've been working on refactoring the code to meet our coding guidelines. There's still a lot of work to do, but there would be some advantages to integrating the plugin before the Trac 1.2 release that might outweigh the code not being in a pristine state.

In terms of continuing development of SpamFilter, I don't see much that needs that needs to change, but all development should following the guidelines: TracDev/DevelopmentWorkflow and TracDev/CodingStyle.

Brief outline of changes to move it to tracopt (WIP):

• Remove i18n:domain from templates
• Use translation functions directly from trac.util.translation

[Christian Boos]

I'm not convinced about this one. Consider the counter arguments:

• it has already some special maintenance status, as together with TracMercurial it's the only plugin we maintain directly on t.e.o
• there are basically two categories of Trac users, those using it on the open, and those using it within intranets; for the second category, which I suspect is the largest, the SpamFilter is entirely unnecessary; at the very least, we would have to make sure to mark all the introduced dependencies as optional

I think the possibility to have a different pace of releases for the SpamFilter if kept separate is also appreciable, given the relative volatility of the external services it depends on.

Now one thing that might make sense is to have the core of the SpamFilter in tracopt and a few modules that don't depend on any external service (3rd party package dependencies would be OK, like Pillow or spambayes, but not external services). External services could be added on a one by one basis (trac-spamfilter-akismet, trac-spamfilter-mollom, etc.).

But I'm not sure the added complexity of splitting the SpamFilter that way would be worth it, and bundling it wholesale feels scary to me. So for one, I'm happy with the status quo. I won't object either if people feel strongly positive about bundling it, I just wanted to voice my reservations.

[Ryan J Ollos]

Thanks for the reply. I'm considering the points and will follow-up soon.

[Dirk Stöcker]

I don't think that's a good idea. Development of this plugin will always be driven by the current situation in SPAM business and than you usually need to react fast or you get extremely poor performance. That conflicts with the idea of releases used for Trac.

[Ryan J Ollos]

Replying to Dirk Stöcker:

Development of this plugin will always be driven by the current situation in SPAM business and than you usually need to react fast or you get extremely poor performance. That conflicts with the idea of releases used for Trac.

In the past year, which changesets would have necessitated an immediate release? I'd like to understand the frequency of such changesets and the impact of having a delay in creating a formal release.

[Ryan J Ollos]

Replying to Christian Boos:

I'm not convinced about this one. Consider the counter arguments:

Thanks for the comments. I considered similar points, but reached different conclusions in most cases.

• it has already some special maintenance status, as together with TracMercurial it's the only plugin we maintain directly on t.e.o

I'm aiming to get away from the special maintenance status since I can't see the benefit. It's more difficult for users to discover SpamFilter, and it's one more thing to install. Wouldn't TracMercurial be included in tracopt were it not for the license incompatibility?

One of the big negatives I hear from users about Trac is that the installation is difficult to maintain with so many plugins. We can tackle the issues of plugin discoverability and site maintenance in other ways as well. Take for instance Jenkins, which has a really nice built-in page for installing and upgrading plugins with the push of a button. I'm also imagining being able to push-button clone a bare installation and be up and running with Trac on Google App Engine, Heroku, PythonAnywhere; along the lines of what Bitnami provides, but hitting more of the popular application platforms. Now I'm getting off topic.

One of my goals in the long run, if I manage to find time, is to pull in some of the commonly-used plugins, like the core of AccountManager. SpamFilter fits the pattern I have in my mind about the set of plugins that are common to a large number of Trac installations.

• there are basically two categories of Trac users, those using it on the open, and those using it within intranets; for the second category, which I suspect is the largest, the SpamFilter is entirely unnecessary; at the very least, we would have to make sure to mark all the introduced dependencies as optional

That was my reasoning for putting it in tracopt. tracopt by definition contains optional components, so SpamFilter would no different in this regard than the Subversion and Git connectors. For users that don't need SpamFilter the only effect would be a slightly larger egg, and it would be trivially larger.

You are probably right that the majority of users are running a "private" Trac instance, but SpamFilter is extremely important to public installations, some of which have been users of Trac for a long time and represent important and very visible projects: MacPorts, Django, WordPress, jQuery UI. I don't know that all of those are running SpamFilter, but assuming it's important to some. Moving SpamFilter to tracopt might help make sure it receives first-class maintenance.

I think the possibility to have a different pace of releases for the SpamFilter if kept separate is also appreciable, given the relative volatility of the external services it depends on.

Currently SpamFilter doesn't get any formal releases, so that's something we need to improve on one way or another. The time to create a Trac release has been shortened considerably and I think we can maintain the pace of every 2 months or so. I'm interested to know how often a change in SpamFilter would necessitate an immediate release because that is an important data point. From what I've seen there is maybe once or twice per year that a major change needs to be pushed, and the last incident I can think of was due to a service ending a few months from the time the change was made, giving adequate lead time to push the release. I'd be okay with moving up a planned release of Trac to push out an important change to SpamFilter, as long it's no more than once or twice per year.

Now one thing that might make sense is to have the core of the SpamFilter in tracopt and a few modules that don't depend on any external service (3rd party package dependencies would be OK, like Pillow or spambayes, but not external services). External services could be added on a one by one basis (trac-spamfilter-akismet, trac-spamfilter-mollom, etc.).

Besides their volatility, what are the downsides to communicating with external services? I was wondering about that with regard to Gravatar as well (comment:2:ticket:12210).

But I'm not sure the added complexity of splitting the SpamFilter that way would be worth it, and bundling it wholesale feels scary to me. So for one, I'm happy with the status quo. I won't object either if people feel strongly positive about bundling it, I just wanted to voice my reservations.

Adding that complexity is the opposite of what I'm trying to accomplish, so I fully agree with your conclusion on that point.

I have some reservations as well:

• Code quality and test coverage of SpamFilter is not as good as Trac
• If moved to tracopt, the Trac development process needs to be followed
• Trac needs to keep to its aim of being a minimalistic (but elegant) project management tool.
• The Trac codebase needs to be kept to a size that can be managed by the Trac dev team.

With regard to the latter point and considering a hypothetical, even if the Trac dev team doubled in size this year, we'd need to make sure that we've achieved sustainable momentum before adding many major features to Trac. The risk is that the size of the team could shrink, and Trac would collapse if the number of bugs grew too large for the dev team to maintain.

At this time moving SpamFilter to tracopt would have to wait for Trac 1.4. We can always revisit the debate later in the 1.3.x dev cycle. I can find plenty of other "pieces" to integrate to Trac in the meantime.

[Christian Boos]

Replying to Ryan J Ollos:

Wouldn't TracMercurial be included in tracopt were it not for the license incompatibility?

TracMercurial is much smaller.

That was my reasoning for putting it in tracopt. tracopt by definition contains optional components, so SpamFilter would no different in this regard than the Subversion and Git connectors. For users that don't need SpamFilter the only effect would be a slightly larger egg, and it would be trivially larger.

SpamFilter is so much bigger though. If it would be a couple of .py files, I probably wouldn't mind, but it has grown into a feature rich subsystem, comparable in size to the wiki subsystem.

Moving SpamFilter to tracopt might help make sure it receives first-class maintenance.

Well, I'm not sure about that. For example, I was never that familiar with the notification subsystem (old or new), so I never really touched it. I'm sure it's the same for other people vs. other subsystems. Bringing in under the repository is not a guaranty that more people will maintain it. Possibly the contrary, as maybe dedicated maintainers like Dirk might feel less confident to take ownership of it if it's in the main repository. I don't know for sure, but that's a possibility.

• The Trac codebase needs to be kept to a size that can be managed by the Trac dev team.

There's also the notion of a "critical mass" of code beyond which undertaking some major change would seem to be too daunting. Consider the switch to Jinja2: there were already quite a lot of templates to convert. Had the SpamFilter been in tracopt at that time, that would have been 1500 more lines of templates to port… That migration will still have to happen eventually, but as it's in a plugin it's much easier to decouple this task from the main one. Likewise, I still intend to change more fundamental stuff in the future, and adding more subsystems in Trac itself will only make that more difficult for the same reasons.

That being said, I'm not adverse to integrating more plugins into tracopt, especially small popular features. For example, I think even the ​TH:WikiExtras would be too big for tracopt and is better maintained separately. If someone manages to create a Mercurial plugin working with ​Mercurial:PythonHglib (#10411), that could fit besides the svn and git support. Well, it's not easy to give strict guidelines, but we can discuss each case as it comes up.

At this time moving SpamFilter to tracopt would have to wait for Trac 1.4. We can always revisit the debate later in the 1.3.x dev cycle. I can find plenty of other "pieces" to integrate to Trac in the meantime.

Besides, given all the work Dirk has put into it, I'd also say we should really give him a major weight into this decision.

[Ryan J Ollos]

See also comment:12:ticket:12270.