Custom Ticket Time Fields

This page presents information on adding a new type 'time' to the list of all currently available custom ticket fields.

Proposed Implementation

Trac core

• add 'time' to available types, inheriting most of the definition from 'text', a simple (one line) text field
  

At that point TracQuery already works as intended by providing from..to input fields on selection of custom time fields for query. Praise to the flexible, modularized software design of Trac developers!

But now at least basic knowledge on how ticket data is handled inside Trac is needed.
(I found not much hints so had to study the code and initially did most things wrong. A multiple dozen not working or buggy local code revisions later, after discussion in #python, etc. I decided on the implementation detailed below. Please discuss, if you know better, I'm still a Python beginner.)

• extend ticket.model, ticket.web_ui, util.datefmt and ticket/templates/ticket.html to
  • allow for reading POSIX microsecond time stamps from db table ticket_custom
    • stored there as string for full compatibility with current custom fields implementation
    • convert to datetime.datetime values and go on using that values internally like is done for core time values time and changetime
  • prepare time field display in ticket view (ticket.html)
    • convert datetime.datetime time stamps to localized time strings
    • add single quote arround strings read from db, i.e. left from former string based date field definitions
    • add time format hint for input form fields
  • parse user input and consolidate different flavors of time stamps
    • initially done with help of mxDateTime.Parser
    • prepared for internal fall back to datetime based parser logic (still to be done) like is done for pytz with some basic internal logic
  • write back POSIX microsecond time stamps to db
  • take care for data used to send out properly formatted notification
    • patched version of ​AnnouncerPlugin for now
    • still unsure on patching Trac Ticket Notifications itself, since it looks like almost dead code (sorry, but Announcer is years ahead) and Announcer is already officially attempting to replace Trac Ticket Notifications (see below)

Related Trac Plugins

• ​CustomFieldAdminPlugin, modified version to ease the setup for new field type, private patch done
• ​DateFieldPlugin working like a charm with new time field, since it's still simple text input form field
• ​AnnouncerPlugin, modified version produces reports with pretty date/time strings, private patch done

Status of Development

Patches against Trac trunk are published as Mercurial queue at bitbucket.org now:

​http://bitbucket.org/hasienda/custom-time/overview

base: patches apply cleanly to Trac trunk r9443, cloned from rblank's Mercurial repository at bitbucket.org.
quality: beta - code flaws should no longer affect db, only minor changes expected until stable release
focus: completion of basic functionality, bug-fixes
reviews: so far just me

You may pull right now, or wait for an official call for beta-test, that will be announced after code works more flawlessly. There might still lure some serious bug I'm currently not aware of. You've been warned. See Known Bugs for more details. A release candidate is not far away now.

No longer bug-fixing on my initial approach, but doing a fresh and hopefully clean new implementation as detailed above. Assume to come out with some patches soon. I've setup ​my place at bitbucket.org recently, that'll make collaboration via a Mercurial repository easier than via Sourceforge.net.

Code of former implementation was published but removed later from Mercurial repository (see custom-time branch) at

http://traccustom-time.hg.sourceforge.net:8000/hgroot/traccustom-time/traccustom-time

because I re-based on a clone of rblank's Mercurial repository with SVN changes up to trac SVN changeset r9443 and moved changes to Mercurial patch queue, and there is no sane way to push to repository (just volatile changesets), will not dare to push again before substantial clearing with core developers is reached, however patch files will get published somehow (starting with #9209)

Feedback could happen through

• comments to #1942,
• edits to (the discussion section of) this wiki page,
• private e-mail, or
• commits to VCS after initial per-developer arrangement inside the Sourceforge project,
• chat while I'm arround in IRC channel #trac at freenode.net

Known Bugs

Problems (with priority) occurring arround ticket changes

• (A) preview time stamp format is throwing an exception
• (B) Trac Notification system unpatched, need to test and investigate, if necessary at all

fixed:

• ticket diff view for change history exposes raw values from db, should convert microseconds POSIX to pretty time stamp string
• bad conversion of date/time value None to db string value 'None' instead of '', gets overwritten with current date on next change, no date/time value deletion possible so far
• notification throwing an error note related to unexpected datetime.datetime input to Trac Notification system: gone while correcting functions preceding in ticket change process, not even sure it was Trac Notification, since this was not active at time of test
• current date is injected whenever time field is left/made empty: after extending all other functions the parser logic was the one place, where 'default_now' logic still happened overwriting undefined date/time
• change to switchable conversion of empty values with new attribute default_now = True(default)|False was incomplete: not done for older function to_timestamp(), so exception raised for some wiki pages (fix will be committed into next rev)
• move to microsecond timestamp caused a mess with calculations (added '* 10E07' instead of '* 10E06' and the like)
• server internal exception turned out to be caused by different versions of trac installed in /usr and /usr/local on my test system interfering on build time

Making mxDateTime.Parser an option

While there was working code for parser functions before and I was especially satisfied with the performance of the needed date/time parser I've re-arranged the implementation. Now mxDateTime.Parser is not a dependency but an option. This was suggested by cboos and others for the sake of maintaining Trac code highly self-sufficient, and I understood this a feature on its own. Still mxDateTime.Parser has superior performance, if it comes to more unusual date/time user input, so it is still a recommended for maximum user satisfaction.

Time stamp format

There was no argument against storing time stamps as string to arrange with current type definition "string" for values in ticket_custom db table. But switching to microseconds time stamp format recently adopted for Trac internal time fields time and changetime seems to be better than sticking with the former Unix time stamp, even if I prepared support for fractions of seconds by converting custom timestamps to float before.

Unit tests

I got advice, that presence of this would be a pre-requisite for inclusion into Trac.

Discussion

Development questions

In db table 'ticket_custom' a new row is created for every (custom time) field. Would it be worth the effort to delete this row, if the date/time is effectively purged? For now we end up with just an empty string ''.

No additional code dependency, but an option

Amongst the few new dependencies use of eGenix.com time string parser mxDateTime.Parser for Python is the most notable one [2]. We need to add such a powerful tool to provide a mature parser right from the start. If handled with care, similar but incompatible classes datetime.datetime and mxDateTime.DateTime are not much of an issue. In fact it doesn't matter at all, as long as we stick to unix time stamp format for saving times in the db for other (good) reasons as well.

(cboos): adding a new external dependency is a strong handicap for being considered for inclusion in Trac core. Have you really weighted the benefits of using mxDateTime? Maybe that helps to kick start the development, but you should really consider building your own utility functions, if you need functionality besides the one provided by the datetime classes. See for example how we integrate pytz: if present, we take benefit from it, if not, we provide our own replacement with reduced functionality.

Actually I put it in here, just because I expected opposition against this dependency. I understand your point and made this an own topic under the development section above.

See above about improved implementation, so this should be no problem anymore.

Considering demand for custom time field defaults

I can imagine various needs for defaults: pre-seed with

• empty value: this is current default
• fixed date/time (much less demand, I guess, but this could help i.e. when facing an already fixed project termination)
• current date/time (just make one of all the 'default_now = False' attributes along the processing optional)
• date/time with predefined offset from current date/time (like "now + 14d")
• …

Each one looks like a perfectly reasonable way to preset a suitable default for one of different demands. But only a preset, of course, still subject to change on ticket creation as well during ticket life. It looks like a discussion would help to possibly strip nonsense from sense. We'll see then, how generic or special, simple or complex a default configuration satisfying most/all real world demand would have to become.

Handling ticket changed notification

I plan to let the notification system handle the conversion of unix time stamp values to date/time strings on it's own. I tested with a patched version of ​AnnouncerPlugin that worked. This is a good start for the proposed replacement of current Trac Notification System by Code from ​AnnouncerPlugin. This might even make per-user preference-based date/time formatting possible in the future.

As adoption of code from ​AnnouncerPlugin is far from ready to replace current Ticket Notification, I feel the need to patch it as well. Right?

Development trace

Happened to have access to SourceForge, so I choose to go and push code into a Mercurial repository there to enable collaboration based on a distributed VCS as per suggestion from IRC channel #trac at freenode.net.

rblank offered to rebase my code on top of code from his own repository, so a least he would be able to rewiev more easily. I'll try it and combine this move with a trial on Mercurial patch queue.

Done, re-based on clone of rblank's Mercurial repository, all changes converted to patch series,
cleared repository on Sourceforge, because previous work was to dirty and non-functional in early tests, I found that I forgot to —addremove after pull/merge and so missed new files as well as deletions

Current development is based on Trac-0.12dev_r9115r9443 (see current state), testet with Python2.5 on Debian GNU/Linux stable. However it would be great to have it in trunk soon (trac-0.12.x?).

(cboos): that's way too old to base new development upon, especially for timestamp related development, as r9210 introduced a big change here (time are now stored as microseconds elapsed since epoch, as bigints).

Ah, well, thanks for the hint. We're at r9420 right now, any recommendation where to start from other than HEAD?

(cboos): why not HEAD? Latest trunk should be fine. We're about to release a rc1 or a beta soon, so that might provide a stable basis.

The question was obsoleted by current development. And I'll continue to work hard to keep pace and match all reasonable demands ASAP.

I'll have to reinstall my test platform here and re-base changes then. However starting with standard unix time seconds was easier to debug for me (i.e. comparing to date +%s). And to_timestamp() messed up the date values converted from mxDatetime for some reason. I went for the included mxDatetime.Datetime.ticks() function to get good unix time seconds. So I may still need an own function to calculate unix time microseconds from unix time seconds.

Checked changes between r9115 and r9435, prepare for updates this weekend. Found 23 changesets that did modify files I touched by now. Sounds quite big. Try to figure out a way to do it save for now, postpone to do it clever with least effort later. Might want to use Mercurial Queues or the like, but I fell I don't understand all the consequences.

Halfway done, 11 merges are in the repository now, most importantly merged with the microseconds time stamp changeset r9210. In fact especially that changeset was not at all related to my modifications. I didn't touch my old function, that still uses to_datetime(ts) instead of the new from_utimestamp(ts). I don't use to_timestamp(dt), but the conversion from within class mxDateTime.DateTime.ticks(). However for the sake of clarity I think it will be good to shift to microseconds timestamps for custom timefields too. So this is on my ToDo after finishing the whole merge marathon.

Done, merged changes for custom time field support with SVN changes up to trac SVN changeset r9442.

Related tickets

#710 asking for basic time tracking in Trac, especially missing native due_date and custom field type 'numeric'
#1942 the main resource on requests and considerations related to the topic
#1943 asking for time based calculations in queries, essentially based on #1943
#1962 asking for due dates on tickets & email notification on overdue dates, several tricks and workarounds needed for using date strings the way you'd expect true time values to match, sort, etc.
#2288 asked for date/time based ticket query functions, that exist by now
#6466 asked for higher time stamp precision introduced with microseconds for time and changetime in r9210
#8354 seeking to get more date/time related information from queries, following #2288
#9209 suggests code cleanup for trac/ticket/model.py, as side effect of my code review for this project
​th#3565 asking for periodical email notification per ticket after certain inactivity time, comment points to ​TracReminderScript
​th#5620 asking for reminder function sending notification before/on current date passing a due_date set as ticket field

External related resources

Ideas and Open Source code was shamelessly taken from the following places:

[1] ​Date and Time Representation in Python by Jochen Voss
[2] ​eGenix.com, especially see mxDateTime ​related documentation
[3] ​Basic date and time types from module datetime in Python≥2.3

Needed help about Python and development tools and found it i.e. at:

​The Python Tutorial - the 'official' one
​Python Tutorial - and another
​Learning with Python 2nd Edition at openbookproject.net
​SQLite Documentation
​Mercurial: The Definitive Guide by Bryan O'Sullivan
​Mercurial Frequently Asked Questions
​Using Mercurial Queues and bitbucket.org - a semi-official guide
​Define your world - explaining internet slang at urbandictionary.com
​Internet Slang Dictionary & Translator at noslang.com