Ticket query renderer extension point

[Peter Suter]

I propose adding a new extension point ITicketQueryRenderer.

class ITicketQueryRenderer(Interface):
    """Extension point interface for components that add HTML renderers of
    ticket query results."""

    def get_ticket_query_formats(self):
        """Return a list of `(format, format_label)` tuples."""

    def render_ticket_query(self, context, query, query_string, tickets, format):
        """Render the ticket query in the specified format."""

The query page could then allow selecting one of the available implementations from a dropdown and display the query results in various different ways.

The existing tables view could become the default implementation. The existing progress bars from the roadmap could provide a second implementation. New third-party plugins could provide further alternate views like pie charts, burndown charts, Kanban boards etc.

The roadmap could perhaps also reuse these renderers to display the milestones in the various ways.

The TicketQuery plugin could also use these renderers, selected by the existing format parameter (which already allows format=table and format=progress).

(Third-party visualization plugins already exist of course, but this extension point would allow them to be instantly deeply integrated with both the ticket query system, and the roadmap, make them usable in any wiki content via macro for free.)

---

Further considerations:

Possibly the batch editing could be separated out.

The existing ITicketGroupStatsProvider extension point could perhaps eventually be deprecated and removed. In its current form it seems to enable only very limited possibilities, potentially also provided by this new extension point.

[Peter Suter]

Sorry, these quite old patches don't apply to trunk anymore, but I wanted to share them anyway just in case someone likes the idea and wants to finish it. I still really like the basic idea. If I remember correctly I was mainly dissatisfied with:

• Having to pass both the query and the query_string parameter to the extension point. (Maybe this can be refactored first, but working with the Query class always seems quite difficult to me.)
• Uncertainty about how some existing features would fit into this, e.g. the batch editing and maybe the milestone statistics progress bars.

[lkraav <leho@…>]

This sounds good. I'm currently exploring JSON-based ​http://datatables.net (especially ​inline editing) for ticket management UI upgrades. Feels like the ticket is applicable here.

[Peter Suter]

Attached refreshed ITicketQueryRenderer.2.patch​, applies to trunk@r14227. ITicketQueryRenderer-SelectFormatOnQueryPage.patch​ still applies on top of that.

[Peter Suter]

Prepared a wiki:TracDev/Proposals/ITicketQueryRenderer page that could be copied to TracDev/PluginDevelopment/ExtensionPoints/trac.ticket.api.ITicketQueryRenderer as documentation.

[Ryan J Ollos]

comment:4:ticket:9098 proposes rendering a timeline view for ticket query results. We should investigate a timeline format for the query view in this ticket.

[Peter Suter]

The refreshed patches ITicketQueryRenderer.3.patch​ and ITicketQueryRenderer-SelectFormatOnQueryPage.3.patch​ apply to trunk@r16600.

ITicketQueryRenderer-Plotly.3.patch​ is an additional proof-of-concept example using ​plotly.js — omitting the plotly-basic.min.js script itself to avoid attachment size limits — to draw pie charts​, bar charts​ etc.

[Ryan J Ollos]

There's one unittest failure with the patches. On investigating, the issue is associated with Query.template_data being called twice while rendering a table:

• rjollos.git/trac/ticket/query.py@daa75c74:1130#L1101​.
• rjollos.git/trac/ticket/query.py@daa75c74:1279#L1276​.

On the second call, when mutating the milestones field dict, the field['options'] has already been transformed to field['optgroups'] and field['options'] is empty, so the milestone query filter ends up empty: rjollos.git/trac/ticket/query.py@daa75c74:772-773,777-780#L759​.

The following change fixes the failure:

trac/ticket/query.py

@@ -769 +769 @@
                 field['options'].insert(0, {'name': '$USER',
                                             'value': '$USER'})
             if name == 'milestone' and not field.get('custom'):
+                field = field.copy()
                 milestones = [Milestone(self.env, opt)
                               for opt in field['options']]
                 milestones = [m for m in milestones

However, it looks like we need to refactoring Query.template_data. Isn't the Query class more like a model? It seems like it shouldn't be involved in preparing data for the view, rather that should be the responsibility of QueryModule.

[Ryan J Ollos]

Eliminated Query.template_data in log:rjollos.git:t12156_query_renderer_extension_point.2 ([aa8393a2/rjollos.git]).

[Jun Omae]

I think get_ticket_query_formats() should have priority attribute, like IDatabaseConnector.get_supported_schemes()​. Otherwise, it should be able to configure a component to render for each format.

[Ryan J Ollos]

Replying to Jun Omae:

Otherwise, it should be able to configure a component to render for each format.

Do you mean, move the ITicketQueryRenderer implementation from QueryModule to it's own component so that it could be replaced by disabling and activating another component providing the table format with different behavior? I think that makes sense, and we should discuss the trade-off of using a priority attribute vs component implementing only ITicketQueryRenderer.

There are a few plugins on trac-hacks.org using Query.template_data, so we should probably retain it for backwards compatibility, and review the use-cases in the plugins.

$ grep -R "query.template_data" . --exclude-dir=".svn"
./trachoursplugin/branches/refactoring2017/trachours/hours.py:        ticket_data = query.template_data(context, tickets, orig_list,
./trachoursplugin/branches/0.11/trachours/hours.py:        ticket_data = query.template_data(context, tickets, orig_list, orig_time, req)
./trachoursplugin/trunk/trachours/hours.py:        ticket_data = query.template_data(context, tickets, orig_list,
./tracstorypointsplugin/0.11/tags/0.1.1/tracstorypoints/macros.py:        data = query.template_data(formatter.context, tickets)
./tracstorypointsplugin/0.11/trunk/tracstorypoints/macros.py:        data = query.template_data(formatter.context, tickets)
./exceldownloadplugin/0.12/tracexceldownload/ticket.py:        data = query.template_data(context, tickets)
./ticketcalendarplugin/0.12/ticketcalendar/web_ui.py:        return query.template_data(context, tickets, None,

I squashes some changes in a new branch: log:rjollos.git:t12156_query_renderer_extension_point.3, based on assumption that [a01c6837/rjollos.git] is acceptable, but let me know.

[Peter Suter]

We should investigate a timeline format for the query view in this ticket.

I'm not 100% sure what you have in mind. We could try this:

• Extract this main part of timeline.html​ into a new template timeline_events.html. Presumably the rest wouldn't be needed for a query timeline format, right?
• Combine some code from TimelineModule​ and some from from TicketModule​. I'm not sure if the code can be easily refactored to be reused, or if it would just have to be partially copied.
• Use the query result to filter / select the tickets events. I'm not sure if we could get all the event data from the query result or if we still would have to use additional SQL queries to get ticket change event data.

ITicketQueryRenderer-TimelineFormatNewTicketEvents.patch​ implements a proof-of-concept of this, but only for the simple case of "new ticket" events. It seems simply adding the required columns to the query is enough for that?

Is this going in the intended direction or am I way off?

[Peter Suter]

Replying to Ryan J Ollos:

Replying to Jun Omae:

Otherwise, it should be able to configure a component to render for each format.

Do you mean, move the ITicketQueryRenderer implementation from QueryModule to it's own component so that it could be replaced by disabling and activating another component providing the table format with different behavior? I think that makes sense, and we should discuss the trade-off of using a priority attribute vs component implementing only ITicketQueryRenderer.

Either way sounds OK for me.

log:rjollos.git:t12156_query_renderer_extension_point.3, based on assumption that [a01c6837/rjollos.git] is acceptable, but let me know.

Looks good to me, thanks.

[Ryan J Ollos]

Replying to Peter Suter:

ITicketQueryRenderer-TimelineFormatNewTicketEvents.patch​ implements a proof-of-concept of this, but only for the simple case of "new ticket" events. It seems simply adding the required columns to the query is enough for that?

What is the lineage of that patch? I tried applying on my branch and on a branch containing your previous 3 patches, both failed to apply cleanly.

Is this going in the intended direction or am I way off?

Yeah, I think that's the intended direction. I was hoping it could be done with significant code reuse. Maybe it won't be clean or easy though.

[Ryan J Ollos]

Replying to Ryan J Ollos:

What is the lineage of that patch? I tried applying on my branch and on a branch containing your previous 3 patches, both failed to apply cleanly.

Oh, nm. It applies cleanly on top of:

• ITicketQueryRenderer.3.patch​
• ITicketQueryRenderer-SelectFormatOnQueryPage.3.patch​

[Ryan J Ollos]

I'm giving more thought to the timeline format and will post more on that soon.

Additional changes:

• [bf3deef88/rjollos.git]: Each ITicketQueryRenderer is a Component (addresses comment:9)
• [e086f4ed5/rjollos.git]: Adds a renderer for query options

I added method render_ticket_query_options after seeing that many of the display options (columns, group results by, max items per page, …) were not relevant to the progress bar format. I'm requesting feedback on this proof-of-concept change.

[Peter Suter]

I added method render_ticket_query_options after seeing that many of the display options (columns, group results by, max items per page, …) were not relevant to the progress bar format

Good idea. But "group results by" is actually relevant for the progress bar format? Maybe it can be added in a query_progressbar_format_options.html?

(For pie- and bar-charts group-by might even be required. For timeline it is irrelevant; but timeline prefs like date-range and event-kind-filters could be useful.)

How batch editing fits into this is also still unclear to me; it probably only works with table format.

[Ryan J Ollos]

Replying to Peter Suter:

Latest changes in log:rjollos.git:t12156_query_renderer_extension_point.5.

Good idea. But "group results by" is actually relevant for the progress bar format? Maybe it can be added in a query_progressbar_format_options.html?

In [676fd012e/rjollos.git], I tried to address this by allowing the controls in ticket_query_options.html to be individually enabled. This allows us to reuse the template for the progressbar.

Other ways I considered:

• Duplicating the code in query_progressbar_format_options.html
• Having a template of macros that could be reused across query_table_format_options.html, query_progressbar_format_options.html and others.
• Generating the formatting options in Python code.

We might need an interface method for handling the request arguments that are added by render_ticket_query_options. Then QueryModule.process_request would need go be refactored to pull out the arguments added by query_format_options.html. This might be related to #10983. I'm continuing to work on this part of the code.

(For pie- and bar-charts group-by might even be required. For timeline it is irrelevant; but timeline prefs like date-range and event-kind-filters could be useful.)

I'll try modifying some plugins to use the interface, like ​TracTicketStatsPlugin.

How batch editing fits into this is also still unclear to me; it probably only works with table format.

I see the page being roughly divided into section:

| Filters                                          |
| Result formatting options                        |
| Results                                          |
| Result modification controls (like batch modify) |

We could have an extension point for adding controls that act on the query results, one example being batch modify. The key question is whether there is likely to be another plugin that would use the interface (maybe something like ​GridModifyPlugin, but would need to investigate).

In the latest changes a template, data is returned by the interface methods ([ae7814d39/rjollos.git]). This is more consistent with other interfaces in Trac, but wouldn't allow generating HTML in Python code. Maybe we should support both a one-arg return value (rendered template) and two arg return value (template and data to be rendered).