Custom Ticket Time Fields

Notice This proposal has been integrated for the 1.1.1 release. See #1942 for more details about the integration process.

This page describes adding a new custom field type 'time' to the list of the currently available custom ticket fields. You may want to read the proposal page, that has more general information on this project.

HowTo testing

As stated above, the code has been added to Trac development branch in trunk r11333, accompanied by a fresh jQuery date-picker implementation.

Note: Information below is outdated now. Please checkout current trunk and report your findings.

Code review

Proposed code changes for Trac trunk reside in a Mercurial repository at bitbucket.org:

​https://bitbucket.org/hasienda/trac-1942

This repository isn't a repository clone, but a patch queue to Trac's SVN repository mirror to Mercurial instead. Mercurial Queue (MQ) is a powerful tool to bring dynamic, mutable changesets to otherwise immutable changesets of this distributed version control system (DVCS). But anyone can use it as a patch stack with some meta-data. Just see the series file for the patch order and optionally more related information ^{note 1}.

The patches will not always apply cleanly to current Mercurial tip/SVN HEAD. But re-basing is easy with MQ, and this has potential to drive even rapid development towards custom time field support ^{note 2}.

You're invited to pull right now, but save a backup of your database before testing, please. I never broke the database of my development system. However, there might still lure some serious bugs, that I'm currently not aware of but could hit you, so you may need your backup to recover from a bad database. You've been warned. See Known Bugs for more details.

Feedback could happen through

• chat, while I'm around in IRC channel #trac at freenode.net (nickname: hasienda)
• edits to (the discussion section of) this wiki page
• comments to #1942
• work on a publicly reachable clone of the Mercurial Queue
• private e-mail
• push to the Mercurial Queue after initial per-arrangement inside bitbucket.org (current Trac core developers enabled in advance)

I urge you to report back on any failure from code review or your testing. For test results, as always, mention your local setup (Trac version/revision, if not taken from my repo, Mercurial revision id of the patches). Any feedback will help to make the patches ready for production and later bring the code into Trac itself. That was my goal right from the start, not to produce just another ​Trac hack.

Notes

1. The common setup for Mercurial Queue puts the patches aside the main repository - into a Mercurial repo on it's own. So looking at ​the changes during development is as easy as looking at development in any repository. . Since there is no other sane way to push changes to a public repository without making them irreversible, this look like the ideal compromise until substantial clearing with core Trac developers has been reached. However, some code already got forwarded for adoption into Trac trunk, like the one published in #9209).

For help on applying the patches to Trac trunk source the traditional way, ie with patch, look at the patch documentation in the ​series file.

But you'll see, Mercurial will save you and me some time and hassle, even more for true collaboration on bugs and feature requests. For the lazy or impatient ones not familiar with Mercurial I recommend to go along the lines of the following walk-through (Debian way).

Before you start

I tested this on my own, really, but can't guarantee and so can't be responsible on behalf of you for any action on your system. Please try to understand, what you type and think twice. Look at documentation I referenced as external related resources at the bottom of this page. Be careful, especially while you act as root.

Quickstart

I assume that you have all pre-requisites like Python, setup-tools, Genshi and optionally Babel for the i18n support in place. If you're unsure or have issues with following procedure, look at the current Trac Install Guide.

$> apt-get install mercurial
$> mkdir ~/trac0.12-dev_custom-time
$> cd ~/trac0.12-dev_custom-time
$> hg qclone https://hasienda@bitbucket.org/hasienda/trac-1942
$> cd ./custom-time
$> hg qseries
$> hg qapplied
$> hg -v qpush custom-time_parser.patch
$> hg qapplied
$> hg -v qpush
$> hg qapplied
$> cp -r ./ ../custom-time_build
$> cd ../custom-time_build
$> python ./setup.py egg_info
$> python ./setup.py compile_catalog -f
$> python ./setup.py build
$> sudo python ./setup.py install --prefix=/usr/local
$> /usr/local/bin/tracd -d -s --port 8000 /trac-environments/sandbox

That will:

• install Mercurial, if not installed before,
• get Trac trunk sources from my Mercurial repo,
• apply all patches from the series but not the latest unstable development code,
• copy patched sources to a build dir,
• build Trac with custom time fields support and
• install it to your system, provided you have root permission or equivalent with sudo and last but not least
• fire up the standalone webserver tracd hosting a Trac environment you've prepared before.

You may have /usr/bin/tracd and /usr/local/bin/tracd than at different versions. Install to /usr/local will normally overrule the other Trac installation without replacing it. So calling just tracd will kick /usr/local/bin/tracd. However, just make sure, you always know, what is going on. I mention all this because I messed up on my own and now refrain from having multiple versions at the same system at a time.

Second hg -v qpush command from above is optional, but recommended to get verbose information from Trac log file, when Trac is configured with log level DEBUG, ie with a section in your trac.ini like the following:

[logging]
log_file = trac.log
log_level = DEBUG
log_type = file

See Trac Logging for more details on the matter.

Notes

Mercurial install/setup on Ubuntu 10.04 LTS ("Lucid Lynx")

On Ubuntu, you will have to apt-get install mercurial mercurial-common in order to get mercurial working. In addition, you will have to add the following entries to /etc/mercurial/hgrc:

# system-wide mercurial configuration file
# See hgrc(5) for more information
[extensions]
mq =
bookmarks =

Without these two entries, the above sequence of commands will fail.

Known Bugs

Issues (with priority) on new ticket creation

• (A) date/time stamp will not be saved, experience vary between failure on ticket creation (preventing ticket creation at all) and just trashing input silently - good news: fixed now by significant code rework, will publish soon together with requested rebase on official Mercurial mirror of Trac SVN repo

Issues (with priority) occurring around ticket changes

• (B) Trac Notification system unpatched, need to test and investigate, if necessary at all - WiP here, code done, think I'll have it in soon, blocked only by inability to test it right now
• (B) (Announcer) notification erroneously report custom time field as changed on any other change, maybe confused by conversion in a place, that is bad for checking an changed fields
• © invalid date/time strings from database get replaced by current day with time reset, should be displayed as empty or (better) the string in single quotes, look at related issue discussed in section 'Migrating from date/time strings' too

fixed:

• ​AnnouncerPlugin non-functional, that already worked before introducing datetime.datetime format for all internal time variables: actually must have stopped on introduction of microsecond time stamps
• ValueError: empty string for float() on TracQuery: handled now by trying to parse strings not convertible to float
• TypeError: unsupported type for timedelta microseconds component: unicode on TracQuery: simply needed to take care for conversation of variable type 'string' to a number and handle custom time values in trac/ticket/templates/query_results.html separately from standard time fields
• preview custom time stamp changes is throwing an exception on internal formatting: resolved after realising, that we deal with different formats in _render_property_diff()
  1. string with POSIX microsecond time stamp from database, directed to ticket changes list
  2. datetime.datetime values from user input for change-in-progress, directed to preview current custom time field changes
  3. old date/time strings, i.e. from previous use of custom text fields for date/time, directed to ticket changes list
  4. any weird value, that is actually take care for as well, as long it has a string representation
• ticket diff view for change history exposes raw values from database, should convert microseconds POSIX to pretty time stamp string
• bad conversion of date/time value None to database string value 'None' instead of '', gets overwritten with current date on next change, no date/time value deletion possible so far
• notification throwing an error note related to unexpected datetime.datetime input to Trac Notification system: gone while correcting functions preceding in ticket change process, not even sure it was Trac Notification, since this was not active at time of test
• current date is injected whenever time field is left/made empty: after extending all other functions the parser logic was the one place, where 'default_now' logic still happened overwriting undefined date/time
• change to switchable conversion of empty values with new attribute default_now = True(default)|False was incomplete: not done for older function to_timestamp(), so exception raised for some wiki pages (fix will be committed into next rev)
• move to microsecond timestamp caused a mess with calculations (added '* 10E07' instead of '* 10E06' and the like)
• server internal exception turned out to be caused by different versions of trac installed in /usr and /usr/local on my test system interfering on build time

ToDo

Regarding #1942, as discussed with rblank:

• add_warning() and
• on parser error recall /ticket or /newticket respectively,
• adding unittests
• check [[TicketQuery()]] functionality with new fields
• check yet-to-be-reintroduced optional mxDateTime.Parser functionality

Future

• add jQuery datepicker function to custom time fields or even more general to input forms for time stamps like was requested in a ​comment to ticket #1942

Discussion

Development questions

In database table 'ticket_custom' a new row is created for every (custom time) field. Would it be worth the effort to delete this row, if the date/time is effectively purged? For now we end up with just an empty string ''.

Time stamp format

There was no argument against storing time stamps as string to arrange with current type definition "string" for values in ticket_custom database table. But switching to microseconds time stamp format recently adopted for Trac internal time fields time and changetime seems to be better than sticking with the former Unix time stamp, even if I prepared support for fractions of seconds by converting custom timestamps to float before.

Migrating from date/time strings

I keep thinking of the case, where one already has a pile of date and wishes to switch to new date/time type now. Hope he/she finds it convenient that old values will be converted (silently) by current code, whenever a ticket is updated. But someone might come up with something like a database upgrade script for batch conversion/migration all a once. I think I'm not enough of a SQL guru to do that. So suggestions and contributions on the subject are very welcome.

The following issue is a side-effect of current fall-backs to parser. I'm unsure, if this is too ambiguous to be acceptable.

TracQuery based on custom time field ignores non-numeric time stamps from database but still displays values, that could be parsed to datetime.datetime in ticket list

Unit tests

I got advice, that presence of this would be a pre-requisite for inclusion into Trac.

No additional code dependency, but an option

Amongst the few new dependencies use of eGenix.com time string parser mxDateTime.Parser for Python is the most notable one [2]. We need I needed to add such a powerful tool to provide a mature parser right from the start. If handled with care, similar but incompatible classes datetime.datetime and mxDateTime.DateTime are not much of an issue. In fact it doesn't matter at all, as long as we stick to unix time stamp POSIX time format for saving times in the time stamps to db for other (good) reasons as well.

(cboos): adding a new external dependency is a strong handicap for being considered for inclusion in Trac core. Have you really weighted the benefits of using mxDateTime? Maybe that helps to kick start the development, but you should really consider building your own utility functions, if you need functionality besides the one provided by the datetime classes. See for example how we integrate pytz: if present, we take benefit from it, if not, we provide our own replacement with reduced functionality.

Actually I put it in here, just because I expected opposition against this dependency. I understand your point and made this an own topic under the development section above.

… so making mxDateTime.Parser an option While there was working code for parser functions before and I was especially satisfied with the performance of the needed date/time parser I've re-arranged the implementation. Now mxDateTime.Parser is not a dependency but an option. This was suggested by cboos and others for the sake of maintaining Trac code highly self-sufficient, and I understood this a feature on its own. Still mxDateTime.Parser has superior performance, if it comes to more unusual date/time user input, so it is still a recommended for maximum user satisfaction.

Considering demand for custom time field defaults

I can imagine various needs for defaults: pre-seed with

• empty value: this is current default
• fixed date/time (much less demand, I guess, but this could help i.e. when facing an already fixed project termination)
• current date/time (just make one of all the 'default_now = False' attributes along the processing optional)
• date/time with predefined offset from current date/time (like "now + 14d")
• …

Each one looks like a perfectly reasonable way to preset a suitable default for one of different demands. But only a preset, of course, still subject to change on ticket creation as well during ticket life. It looks like a discussion would help to possibly strip nonsense from sense. We'll see then, how generic or special, simple or complex a default configuration satisfying most/all real world demand would have to become.

Considering demand for (custom) time field format settings

As I learned from the discussion within ticket #2182 support for implicitly setting date and time formats by locale selection is not enough for some users. I'd like to make this enhancement available for custom time fields first. Than we might be prepared to go on extending it to an implementation for all time fields. Maintaining changes for custom and standard time field in different patched should be desired to allow for an independed application as proposed by me.

Handling ticket changed notification

I plan to let the notification system handle the conversion of unix time stamp values to date/time strings on it's own. I tested with a patched version of ​AnnouncerPlugin that worked. This is a good start for the proposed replacement of current Trac Notification System by Code from ​AnnouncerPlugin. This might even make per-user preference-based date/time formatting possible in the future.

As adoption of code from ​AnnouncerPlugin is far from ready to replace current Ticket Notification, I feel the need to patch it as well. Right?

Development traces

Happened to have access to SourceForge, so I choose to go and push code into a Mercurial repository there to enable collaboration based on a distributed VCS as per suggestion from IRC channel #trac at freenode.net.

rblank offered to rebase my code on top of code from his own repository, so a least he would be able to rewiev more easily. I'll try it and combine this move with a trial on Mercurial patch queue.

Done, re-based on clone of rblank's Mercurial repository, all changes converted to patch series,
cleared repository on Sourceforge, because previous work was to dirty and non-functional in early tests, I found that I forgot to —addremove after pull/merge and so missed new files as well as deletions

Current development is based on Trac-0.12dev_r9115 r9443 r9478 (see current state), tested with Python2.5 on Debian GNU/Linux stable - 5.0, codename lenny. However it would be great to have it in trunk soon (trac-0.12.x?).

(cboos): that's way too old to base new development upon, especially for timestamp related development, as r9210 introduced a big change here (time are now stored as microseconds elapsed since epoch, as bigints).

Ah, well, thanks for the hint. We're at r9420 right now, any recommendation where to start from other than HEAD?

(cboos): why not HEAD? Latest trunk should be fine. We're about to release a rc1 or a beta soon, so that might provide a stable basis.

The question was obsoleted by current development. And I'll continue to work hard to keep pace and match all reasonable demands ASAP.

I'll have to reinstall my test platform here and re-base changes then. However starting with standard unix time seconds was easier to debug for me (i.e. comparing to date +%s). And to_timestamp() messed up the date values converted from mxDatetime for some reason. I went for the included mxDatetime.Datetime.ticks() function to get good unix time seconds. So I may still need an own function to calculate unix time microseconds from unix time seconds.

Checked changes between r9115 and r9435, prepare for updates this weekend. Found 23 changesets that did modify files I touched by now. Sounds quite big. Try to figure out a way to do it save for now, postpone to do it clever with least effort later. Might want to use Mercurial Queues or the like, but I fell I don't understand all the consequences.

Halfway done, 11 merges are in the repository now, most importantly merged with the microseconds time stamp changeset r9210. In fact especially that changeset was not at all related to my modifications. I didn't touch my old function, that still uses to_datetime(ts) instead of the new from_utimestamp(ts). I don't use to_timestamp(dt), but the conversion from within class mxDateTime.DateTime.ticks(). However for the sake of clarity I think it will be good to shift to microseconds timestamps for custom timefields too. So this is on my ToDo after finishing the whole merge marathon.

Done, merged changes for custom time field support with SVN changes up to trac SVN changeset r9442.

Re-based on r9478 meanwhile, nothing to update inside my patches. Nice. Time to move on.

Related Trac plugins

• ​CustomFieldAdminPlugin, modified version to ease the setup for new field type ^*
• ​CalendarPopUpPlugin is working flawlessly with custom time fields, since these are still simple text input form fields
• ​DateFieldPlugin does content validation to enforce it's idea of proper date/time string formatting, that will in most cases conflict with the input rewrite done by the parser of the time field implementation, however comparing date picker with that of ​CalendarPopUpPlugin this one looks superior in general (integration of an external, stable code base, easy customization support, i18n/l10n support, etc.), so it qualified for this project (see ToDo section)
• ​AnnouncerPlugin, modified version produces reports with pretty date/time strings according to locale setting of trac(d) proccess ^*

^* patch is available inside repo, see subdirectory ./plugin

External related resources

Ideas and Open Source code was shamelessly taken from the following places:

[1] ​Date and Time Representation in Python by Jochen Voss
[2] ​eGenix.com, especially see mxDateTime ​related documentation
[3] ​Basic date and time types from module datetime in Python≥2.3