Custom Ticket Time Fields

This page presents information on adding a new type 'time' to the list of all currently available custom ticket fields.

Proposed Implementation

Trac core

• add 'time' to available types, inheriting most of the definition from 'text', a simple (one line) text field
  

At that point TracQuery already works as intended by providing from..to input fields on selection of custom time fields for query. Praise to the flexible, modularized software design of Trac developers!

• extend ticket.model, ticket.web_ui, util.datefmt and ticket/templates/ticket.html to
  • allow for reading unix time stamp values from db table ticket_custom
  • convert unix time stamp values to localized time string
  • parse user input and consolidate different flavors of time stamps
  • write back unix time stamp to db
  • send out properly formatted notification

Related Trac Plugins

• ​CustomFieldAdminPlugin, modified version to ease the setup for new field type, private patch done
• ​DateFieldPlugin working like a charm with new time field, since it's still simple text input form field
• ​AnnouncerPlugin, modified version produces reports with pretty date/time strings, private patch done

Status of Development

Code was published but removed later from Mercurial repository (see custom-time branch) at

​http://traccustom-time.hg.sourceforge.net:8000/hgroot/traccustom-time/traccustom-time

because I re-based on clone of rblank's Mercurial repository with SVN changes up to trac SVN changeset r9443 and moved changes to Mercurial patch queue, and there is no sane way to push to repository (just volatile changesets), will not dare to push again before substantial clearing with core developers is reached, however patch files will get published somehow (starting with #9209)

Feedback could happen through

• comments to #1942,
• edits to (the discussion section of) this wiki page,
• private e-mail, or
• commits to VCS after initial per-developer arrangement inside the Sourceforge project,
• chat while I'm arround in IRC channel #trac at freenode.net

Known Bugs

I recognized at problems on creation of new tickets that needs to be fixed ASAP.

Making mxDateTime.Parser an option

While there is working code for all intended functions and I'm especially satisfied with the performance of the needed date/time parser I'll try to change it's implementation. Making mxDateTime.Parser just an (still recommended) option but not a dependency, like cboos suggests, clearly justifies some effort for the sake of maintaining Trac code highly self-sufficient.

Time stamp format

There was no argument against storing time stamps as string to arrange with current type definition "string" for values in ticket_custom db table. But switching to microseconds time stamp format recently adopted for Trac internal time fields time and changetime seems to be better than sticking with the former Unix time stamp, even if I prepared support for fractions of seconds by converting custom timestamps to float before.

Unit tests

I got advice, that presence of this would be a pre-requisite for inclusion into Trac.

Discussion

Additional code dependencies

Amongst the few new dependencies use of eGenix.com time string parser mxDateTime.Parser for Python is the most notable one [2]. We need to add such a powerful tool to provide a mature parser right from the start. If handled with care, similar but incompatible classes datetime.datetime and mxDateTime.DateTime are not much of an issue. In fact it doesn't matter at all, as long as we stick to unix time stamp format for saving times in the db for other (good) reasons as well.

(cboos): adding a new external dependency is a strong handicap for being considered for inclusion in Trac core. Have you really weighted the benefits of using mxDateTime? Maybe that helps to kick start the development, but you should really consider building your own utility functions, if you need functionality besides the one provided by the datetime classes. See for example how we integrate pytz: if present, we take benefit from it, if not, we provide our own replacement with reduced functionality.

Actually I put it in here, just because I expected opposition against this dependency. I understand your point and made this an own topic under the development section above.

Handling ticket changed notification

I plan to let the notification system handle the conversion of unix time stamp values to date/time strings on it's own. I tested with a patched version of ​AnnouncerPlugin that worked. This is a good start for the proposed replacement of current Trac Notification System by Code from ​AnnouncerPlugin. This might even make per-user preference-based date/time formatting possible in the future.

As adoption of code from ​AnnouncerPlugin is far from ready to replace current Ticket Notification, I feel the need to patch it as well. Right?

Development trace

Happened to have access to SourceForge, so I choose to go and push code into a Mercurial repository there to enable collaboration based on a distributed VCS as per suggestion from IRC channel #trac at freenode.net.

rblank offered to rebase my code on top of code from his own repository, so a least he would be able to rewiev more easily. I'll try it and combine this move with a trial on Mercurial patch queue.

Done, re-based on clone of rblank's Mercurial repository, all changes converted to patch series,
cleared repository on Sourceforge, because previous work was to dirty and non-functional in early tests, I found that I forgot to —addremove after pull/merge and so missed new files as well as deletions

Current development is based on Trac-0.12dev_r9115r9443 (see current state), testet with Python2.5 on Debian GNU/Linux stable. However it would be great to have it in trunk soon (trac-0.12.x?).

(cboos): that's way too old to base new development upon, especially for timestamp related development, as r9210 introduced a big change here (time are now stored as microseconds elapsed since epoch, as bigints).

Ah, well, thanks for the hint. We're at r9420 right now, any recommendation where to start from other than HEAD?

(cboos): why not HEAD? Latest trunk should be fine. We're about to release a rc1 or a beta soon, so that might provide a stable basis.

The question was obsoleted by current development. And I'll continue to work hard to keep pace and match all reasonable demands ASAP.

I'll have to reinstall my test platform here and re-base changes then. However starting with standard unix time seconds was easier to debug for me (i.e. comparing to date +%s). And to_timestamp() messed up the date values converted from mxDatetime for some reason. I went for the included mxDatetime.Datetime.ticks() function to get good unix time seconds. So I may still need an own function to calculate unix time microseconds from unix time seconds.

Checked changes between r9115 and r9435, prepare for updates this weekend. Found 23 changesets that did modify files I touched by now. Sounds quite big. Try to figure out a way to do it save for now, postpone to do it clever with least effort later. Might want to use Mercurial Queues or the like, but I fell I don't understand all the consequences.

Halfway done, 11 merges are in the repository now, most importantly merged with the microseconds time stamp changeset r9210. In fact especially that changeset was not at all related to my modifications. I didn't touch my old function, that still uses to_datetime(ts) instead of the new from_utimestamp(ts). I don't use to_timestamp(dt), but the conversion from within class mxDateTime.DateTime.ticks(). However for the sake of clarity I think it will be good to shift to microseconds timestamps for custom timefields too. So this is on my ToDo after finishing the whole merge marathon.

Done, merged changes for custom time field support with SVN changes up to trac SVN changeset r9442.

Related tickets

#710 asking for basic time tracking in Trac, especially missing native due_date and custom field type 'numeric'
#1942 the main resource on requests and considerations related to the topic
#1943 asking for time based calculations in queries, essentially based on #1943
#1962 asking for due dates on tickets & email notification on overdue dates, several tricks and workarounds needed for using date strings the way you'd expect true time values to match, sort, etc.
#2288 asked for date/time based ticket query functions, that exist by now
#8354 seeking to get more date/time related information from queries, following #2288
#6466 asked for higher time stamp precision introduced with microseconds for time and changetime in r9210
#9209 suggests code cleanup for trac/ticket/model.py, as side effect of my code review for this project

External related resources

Ideas and Open Source code was shamelessly taken from the following places:

[1] ​Date and Time Representation in Python by Jochen Voss
[2] ​eGenix.com, especially see mxDateTime ​related documentation
[3] ​Basic date and time types from module datetime in Python≥2.3

Needed help about Python and development tools and found it i.e. at:

​The Python Tutorial - the 'official' one
​Python Tutorial - and another
​Mercurial: The Definitive Guide by Bryan O'Sullivan
​Mercurial Frequently Asked Questions