Cumulative patch: New workflow system for tickets in Trac

[pkou@…]

A new workflow system is proposed (including implementation). See NewWorkflow for details. It includes the patch and comprehensive description of the changes.

Related tickets: #563, #301 (probably more).

IMHO it is one of most important features that need to be added to Trac.

[pkou <pkou at ua.fm>]

The patch in NewWorkflow has been updated and changes not related to workflow system have been removed (they have been registered as separate tickets #877, #878, #879, and #880).

I do not see anything that can be simplified or improved for now. Thus, I am waiting for comments from Trac development team.

I would not like to apply the patch to my real projects because it changes database structure. Until I get a feedback, there is a risk than next database change will break compatibility between Trac code and NewWorkflow code.

Thoughts?

[cmlenz]

First a couple of random thoughts on the proposal:

The one thing that I think is dangerous is that the workflow_version option is a pretty big switch that goes through the code-base, i.e. the "if workflow_version..." conditionals are too distributed.

Object-oriented programming theory says to use polymorphism instead of conditionals, so how about something like:

class SimpleWorkflow:

    def available_actions(self, ticket):
        if ticket['status'] in ['new', 'reopened']:
            actions = [ 'accept', 'reassign', 'resolve' ]
        elif ticket['status'] == 'assigned':
            actions = [ 'reassign', 'resolve' ]
        elif ticket['status'] == 'closed':
            actions = [ 'reopen' ]

    def ticket_changed(self, ticket, user, action, args):
        if action == 'accept':
            ticket['status'] =  'assigned'
            ticket['owner'] = user or ''
        if action == 'resolve':
            ticket['status'] = 'closed'
            ticket['resolution'] = args.get('resolve_resolution')
        elif action == 'reassign':
            ticket['owner'] = args.get('reassign_owner')
            ticket['status'] = 'new'
        elif action == 'reopen':
            ticket['status'] = 'reopened'
            ticket['resolution'] = ''


class ExtendedWorkflow:

   def get_actions(self, ticket, user):
        if ticket['status'] in ['new', 'reopened']:
            actions = [ 'accept', 'reassign', 'resolve' ]
        elif ticket['status'] == 'assigned':
            actions = [ 'reassign', 'resolve' ]
        elif ticket['status'] == 'resolved':
            actions = [ 'reopen', 'verify' ]
        elif ticket['status'] == 'verified':
            actions = [ 'reopen', 'close' ]
        elif ticket['status'] == 'closed':
            actions = [ 'reopen' ]

    def do_action(self, ticket, user, action, arg):
        if action == 'accept':
            ticket['status'] = 'assigned'
            ticket['owner'] = user or ''
        if action == 'resolve':
            ticket['status'] = 'resolved'
            ticket['resolution'] = arg
        elif action == 'reassign':
            ticket['status'] = 'new'
            ticket['owner'] = arg
        elif action == 'verify':
            ticket['status'] = 'new'
            ticket['owner'] = # XXX: get QA owner
        elif action == 'reopen':
            ticket['status'] = 'reopened'
            ticket['resolution'] = ''
        # XXX: etc

The ticket system/class would delegate to the workflow implementation where appropriate. The user would select between different workflows in trac.ini:

[ticket]
workflow = simple|extended

One could imagine users even plugging in custom workflow implementations by specifying the fully qualified name of the class:

[ticket]
workflow = mypackage.workflow.MyWorkflow

I think such an approach would also allow a more controlled introduction of the advanced workflow you propose: first refactor the ticket workflow so that it's basically pluggable, then add the new workflow implementation, together with the other changes such as adding qaowners to components, owners to milestones, etc.

What do you think?

Otherwise I think the patch is very high quality, and the feature has been requested often enough that we should seriously consider adding it to Trac. The best way forward would be (IMHO) to get you svn write access to a private branch in the repository (say pkou-dev/workflow), and try to merge the branch into trunk for 0.9 as soon as we all think it's ready.

BTW, I think attaching the patches to this ticket instead of to the Wiki page would be more appropriate.

[pkou <pkou at ua.fm>]

I like the idea with removing workflow_version and replacing it with a parameter that specifies a workflow class. This will give us flexible customization of a workflow for tickets. I will be working on this implementation.

One of interesting moments here is that it is possible to move action-specific HDF code from Ticket.cs into separate files. The Ticket.cs should include a file with actions using code like:

 <?cs include ticket.workflow.actions_template ?>

and a workflow class should have a method that returns the name of included file, e.g.

class SimpleWorkflow:
    def get_actions_template(self, ticket, user):
        return 'Ticket_simple.cs'

class ExtendedWorkflow:
    def get_actions_template(self, ticket, user):
        # possible changes: return different templates depending on ticket fields
        # and/or logged user
        return 'Ticket_extended.cs'

What do you think about this approach?

Also, it seems to be a good idea to have a commonly accessible place in SVN where we can change code. Let's discuss the details via email. I would also agree with proposed approach: Refactor current code first, and then apply changes that are specific to new workflow.

p.s.: The patches have been attached to Wiki page because it is necessary to maintain change history for the sources. I would like to suggest to keep the change history with NewWorkflow page even if we switch from patches to SVN repository. This way everybody can communicate with each other and describe additional changes that are made. And at the end we will have a good documentation for the change.

p.p.s: Thank you for the feedback.

[pkou <pkou at ua.fm>]

Support customized workflows in Trac

[pkou <pkou at ua.fm>]

Attached patch refactors existing code in trac/Ticket.py, templates/ticket.cs, and templates/newticket.cs in order to support customized workflows in Trac.

Note that the patch does not implement enhanced workflow, it just refactors existing code for existing workflow. However, the patch in NewWorkflow contains both this refactoring and support for enhanced workflow.

Section Refactoring of Trac code for easy support of customized workflows in NewWorkflow describes the changes with more details.

[cmlenz]

One thing I do not like about this patch is the coupling of workflow classes with templates. Templates are part of the web presentation layer, and backend/logic classes such as the workflow implementations should not need to know or even care about them.

(Granted, the separation between presentation and backend/logic isn't strong or even existant in the current code base, but that's something we need to move away from, not move closer to.)

So ideally, the workflow classes would somehow export a data structure that gives the normal ticket template enough information to display the available options. Going back to my initial example code, maybe something like:

class SimpleWorkflow:

    def available_actions(self, ticket):
        if ticket['status'] in ['new', 'reopened']:
            return [
                ('accept', None),
                ('reassign', 'owner'),
                ('resolve', 'resolution')
            ]
        elif ...

I don't think there's any other information than the new owner (reassign) or the new resolution (resolve etc) that we'd need on status changes, so such an approach may work. Probably, we'll also want to provide a hook for workflows to determine the available resolutions per status change, for example:

class SimpleWorkflow:

    def available_resolutions(self, ticket, newstatus):
        if newstatus == 'resolve':
            return [ 'fixed', 'invalid', 'duplicate', 'worksforme' ]
        return []

You get the idea.

And going into nitpicking mode, I think on_create would be a better method name than on_insert, as the latter is too SQL-layerish, and the Ticket class also uses create.

[pkou <pkou at ua.fm>]

Patch for the changes: Synchonized with trunk, changed according to comments on previous implementation

[pkou <pkou at ua.fm>]

New patch attached. Changes:

• Sources synchronized with trunk;
• Workflow operations on_insert and on_update renamed to on_create and on_save in order to match Trac conventions for tickets;
• Additional workflow contributed in order to prove current implementation (see below).

It seems that we have different vision on this issue and I would like to explain why current design is implemented as it is.

Trac has rather good separation between presentation layer and business logic. There are three types of entities:

• Python source code for business logic;
• ClearSilver templates for presentation layer. However, the templates are linked to business logic in terms of data that is passed between these layers;
• CSS templates for presentation layer.

An attempt to develop a customized workflow without appropriate changes in templates will not give a big value for developers of customized workflows.

Let's review what we cover by term customized workflow:

• It does not change statuses for tickets. Although this is an interesting idea, it is not discussed here. Many other places in Trac depend on statuses (timeline, milestone, reports, etc) and a separate research needs to be done for this task;
• A customized workflow does allow to customize actions on a ticket;
• A customized workflow allows to implement custom fields within an action. Examples of such fields include 1) ticket owner changes; 2) resolution field; 3) other fields like time that is spent on an action.

Thus, the idea behind current implementation is the following: Trac provides enough statuses natively but transitions between these statuses can be fully customized and extended. Currently, it is even possible to implement actions for new tickets, which can be a nice feature for some workflows.

A workflow developer should be responsible for the following areas:

• Which actions are allowed;
• Which additional parameters should be entered for an action:
  • A workflow should manage owner changes;
  • A workflow should manage resolution changes;
  • A workflow can manage other action-specific fields.
• Validation rules for a ticket;
• Automatic field calculation.

All these responsibilities depend on 1) ticket status; 2) other ticket fields; 3) currently logged user; and 4) current environment parameters for a project.

In order to prove this concept, I would like to ask you to review two custom workflows that have been developed:

• QaRmtWorkflow - See NewWorkflow for details;
• TimeTrackWorkflow - available in attached patch. It is implementation that subclasses any other workflow in order to allow users to specify time that they spend on an action.

It is important to underline that

1. These workflows do not require any changes in ticket-specific code in Trac;
2. These workflows do require changes in templates (at least I do not see a possibility to implement these workflows without appropriate changes in templates).

Thus, if fields owner and resolution are managed in main Trac code, we lose an important feature: It will not be possible to implement additional workflow-specific fields like time-for-action.

What I would like to suggest is to leave minimal support for owner and resolution fields in main Trac code. The rest should be done by workflow classes. With attached patch, owner and resolution have minimal meaning in main code already. As an option, it is possible to remove resolution constants from table enum and make a workflow responsible for generating this list.

If we follow your proposal, then we will need to add code specific for owner and resolution in main Trac code. And this will not solve the whole problem because templates will need to be changed in order to add new actions.

Ugh, I finished :-)

[anonymous]

Is there any update on this ticket?

I'm interested in using the patches, but would like to know that they are going to be committed. I'm in the process of writing conversion routines for our own custom tracking software for use with Trac, but I need the functionality offered by the patch. I've spent some time looking at just writing the custom fields using TracTicketsCustomFields. They do not seem to be a full solution to the problem as they do not address the issue of the two additional status codes (resolved and verified) or the workflow transitions to the respective entities.)

Thanks for the update.

[anonymous]

My engineering organization is also very interested in this patch. Any idea when this might be released?

Thanks!

[graham_foster@…]

I too am very interested in this patch being fully included ASAP. Will it make 0.9? Thx.

[pkou at ua.fm]

Doing a patch for 0.9 is rather hard due to significant source changes that have been made for this release in trunk.

Because the patch has not been integrated at beginning of 0.9, it is pending until all major refactorings will be done for 0.9 (in order to eliminate redundant work).

In order to simplify the patch, separate tickets have been created for step-by-step improvement of Trac code (notably, #1546, #1545, #1544, #1547, and optionally but highly required - #1549).

Having prepared source code in trunk, the patch will address workflow-related issues only.

Note about a major difference between existing patch for 0.8 and new patch for 0.9: The new patch will make Trac status-independent. This will allow customizing Trac for different workflows and statuses without additional changes in codebase.

[anonymous]

Is a patch available for 0.8.2? I tried using the patch-newworkflow-r1102.diff attachment, but had a couple of problems:

1. tried applying this with patch but it couldn't seem to get the file names--I had to type these in (or cut-n-paste them in);
2. also had lots of rejects once I entered the filenames.

It's the first time I've used patch, so perhaps I messed something up (I tried using -Np1 but that didn't help).

This new workflow is exactly what we need, so I'd love to get it set up prior to 0.9, unless that's imminent.

[anonymous]

New patch for Trac 0.8.2 attached in NewWorkflow.

[tom at vikus.com]

I've tried applying the 0.8.2 v2 patch onto versions 1752, 1768, and HEAD, but there are many Hunk failures, as well as missing files, specifically Ticket.py and WikiFormatter?.y.

I am anxious to apply this patch. Am I missing something?

[eblot]

I've tried applying the 0.8.2 v2 patch onto versions 1752, 1768, and HEAD

See [http:/trac/ticket/869#change_14 comment 14]. Trunk (i.e. pre-0.9) revisions are very different from the 0.8.2 branch, so do not expect to patch the trunk without going thru (heavy) modifications of the patch file.

[David M. Lee <dmlee@…>]

Add one more vote for an improved workflow.

[mgood]

#1989 has been marked as a duplicate of this ticket.

[cmlenz]

#2102 has been marked as duplicate of this ticket.

[anonymous]

My understanding is that this patch will not get into core Trac and needs to be rewritten using the new plug-in system. Is anyone working on this?

[anonymous]

yet another votes for this to be possible in .9

[anonymous]

I'd also like to put in my vote for this.

[mgood]

#2511 has been marked as a duplicate of this ticket

[tom@…]

I'm looking forward to this and would like to suggest an additional state which I've found very useful while working with Roundup, namely done, could be better a.k.a. donecbb.

In practice it happens often that something is fixed well enough to move on, but one still wants to have a list of all tickets that still could use some 'extra polish'. I found the donecbb state to be very helpful and would like to see it added to trac or that trac provide an easy way to customise states (as roundup does).

and just for the record: I think trac totally rocks ;-)

[Alec Thomas <alec@…>]

I've created a new patch, and a plugin which leverages this patch. They can be found here. This is against Trac's trunk, so if you are running 0.9.3 it may or may not work.

[cboos]

There's now a sandbox branch dedicated to this effort: log:sandbox/workflow

[totmacher@…]

Hello, My name is Christian Sprajc and I'm Project Manager of the Open Source Project "PowerFolder?" (http://www.powerfolder.com). We use Trac since three weeks and we like it very much. Thanks for this nice and useful Product. I'm looking forward to the Workflow component. I have read the "Whitepaper" of "New Trac Workflow" and I think something is still missing. I have the some Ideas to improve the "New" part of the workflow, the way a ticket comes into the system.

We have the problem, that someone who enters the ticket (a user or developer) is not responsible for sheduling a ticket on a milestone or version. Futhermore he might not have an idea to what component this ticket belongs to, nor what priority/Severity a ticket has. He also should not assign the ticket to a user.

I am thinging about a "Simple ticket entry" where all these properties have defaults which cannot be changed by the user. This ticket should then be evaluated by the project management, which sets these properties.

Workflow would be then: "Simple Ticket entry" (User, Developer) -> "Evaluation and assignment" (Project management) -> "Ticket new"

Best regards, Christian Sprajc

[anonymous]

We would love to see this feature implemented here.

[anonymous]

ASAP please!

[cboos]

#1581 has been marked as duplicate. That ticket reminds about the necessity to avoid hardcoding the status values in the templates.

Also, I've seen that this ticket doesn't yet link to the WorkFlow wiki page, which documents the way new flexible workflows will be possible in future Trac versions.

That page also keeps track of the development status of the workflow branch in the sandbox mentioned above.

[cmlenz]

#3056 marked as duplicate of this ticket.

[eblot]

(fix up CC list)

[anonymous]

I'd really like to see this integrated into Trac soon!

[sstarr]

Put the missing CCs back. 'Anonymous' please be careful with editing CCs.

[eblot]

(fix up CC: list)

[anonymous]

It would be good to get an update on this.

I'm running 0.1 and I'd like to use this feature, the ability to create new ticket states and actions is fundamental to a change control system. Do I have to wait or is there a work around/plug-in available?

[mgood]

Replying to anonymous:

I'm running 0.1 and I'd like to use this feature, the ability to create new ticket states and actions is fundamental to a change control system. Do I have to wait or is there a work around/plug-in available?

As mentioned before this is being developed in sandbox/workflow and it's scheduled for 0.11. This cannot be implemented as a plugin, so if you want to try it ahead of time you'll need to get the sandbox branch, though it is not yet up to date with all the fixes in 0.10.

[anonymous]

I really, really hope this get into the 0.11 release. Having used Trac for a while I say you cannot do without this feature for serious work with an off-shore testing team. The situation rapidly becomes confusing withouth the additional states of a flexible workflow. Also PM needs to track the defects in the various stages of the defect fixing process.

[sid]

#4338 was marked as duplicate of this ticket.

[anonymous]

Is there a patch available for 0.10.x?

P.S. Thank you for trac.

[sdyson@…]

Since the workflow sandbox no longer exists does anyone know the status of this work?

[cboos]

Check the WorkFlow page, there are links pointing to the old branch (on top of 0.10dev).

There has been rumors that a new port for 0.11dev will be attempted at the TracSprint during PyCon 2007.