= Custom Ticket Time Fields = This page presents information on adding a new type 'time' to the list of all currently available [wiki:TracTicketsCustomFields custom ticket fields]. == Proposed Implementation == === Trac core === * add 'time' to available types, inheriting most of the definition from 'text', a simple (one line) text field[[BR]] At that point TracQuery already works as intended by providing from..to input fields on selection of custom time fields for query. Praise to the flexible, modularized software design of Trac developers! But now at least basic knowledge on how ticket data is handled inside Trac is needed.[[BR]] (I found not much hints so had to study the code and initially did most things wrong. A multiple dozen not working or buggy local code revisions later, after discussion in #python, etc. I decided on the implementation detailed below. Please discuss, if you know better, I'm still a Python beginner.) * extend ''ticket.model'', ''ticket.web_ui'', ''util.datefmt'' and ''ticket/templates/ticket.html'' to * allow for reading POSIX microsecond time stamps from db table ''ticket_custom'' * stored there as string for full compatibility with current custom fields implementation * convert to datetime.datetime values and go on using that values internally like is done for core time values ''time'' and ''changetime'' * prepare time field display in ticket view (ticket.html) * convert datetime.datetime time stamps to localized time strings * add single quote arround strings read from db, i.e. left from former string based date field definitions * add time format hint for input form fields * parse user input and consolidate different flavors of time stamps * initially done with help of ''mxDateTime.Parser'' * prepared for internal fall back to datetime based parser logic (still to be done) like is done for pytz with some basic internal logic * write back POSIX microsecond time stamps to db * take care for data used to send out properly formatted notification * patched version of [th:wiki:AnnouncerPlugin AnnouncerPlugin] for now * still unsure on patching Trac Ticket Notifications itself, since it looks like almost dead code (sorry, but Announcer is years ahead) and Announcer is already officially attempting to replace Trac Ticket Notifications (see below) === Related Trac Plugins === * [th:wiki:CustomFieldAdminPlugin CustomFieldAdminPlugin], modified version to ease the setup for new field type, private patch done * [th:wiki:DateFieldPlugin DateFieldPlugin] working like a charm with new time field, since it's still simple text input form field * [th:wiki:AnnouncerPlugin AnnouncerPlugin], modified version produces reports with pretty date/time strings, private patch done === Status of Development === No longer bug-fixing here and there, but doing a fresh and hopefully clean new implementation as detailed above. Assume to come out with some patches soon. I've setup [http://bitbucket.org/hasienda/shelf/ my place at bitbucket.org] recently, that'll make collaboration via a Mercurial repository easier than via Sourceforge.net. Code of former implementation was published but removed later from Mercurial repository (see ''custom-time'' branch) at http://traccustom-time.hg.sourceforge.net:8000/hgroot/traccustom-time/traccustom-time because I re-based on a clone of rblank's Mercurial repository with SVN changes up to trac SVN changeset r9443 and moved changes to Mercurial patch queue, and there is no sane way to push to repository (just volatile changesets), will not dare to push again before substantial clearing with core developers is reached, however patch files will get published somehow (starting with #9209) Feedback could happen through * comments to #1942, * edits to (the discussion section of) this wiki page, * private e-mail, or * ~~commits to VCS after initial per-developer arrangement inside the Sourceforge project~~, * chat while I'm arround in IRC channel #trac at freenode.net ==== Known Bugs ==== Problems on creation of new tickets * fixed: Server internal exception turned out to be caused by different versions of trac installed in /usr and /usr/local on my test system interfering on build time Problems on ticket change/preview * fixed: move to microsecond timestamp caused a mess with calculations (added '* 10E07' instead of '* 10E06' and the like) ==== Making ''mxDateTime.Parser'' an option ==== While there is working code for all intended functions and I'm especially satisfied with the performance of the needed date/time parser I'll try to change it's implementation. Making ''mxDateTime.Parser'' just an (still recommended) option but not a dependency, like cboos suggests, clearly justifies some effort for the sake of maintaining Trac code highly self-sufficient. ==== Time stamp format ==== There was no argument against storing time stamps as string to arrange with current type definition "string" for values in ticket_custom db table. But switching to microseconds time stamp format recently adopted for Trac internal time fields ''time'' and ''changetime'' seems to be better than sticking with the former Unix time stamp, even if I prepared support for fractions of seconds by converting custom timestamps to float before. ==== Unit tests ==== I got [ticket:1942#comment:23 advice], that presence of this would be a pre-requisite for inclusion into Trac. === Discussion === ==== Additional code dependencies ==== Amongst the few new dependencies use of ''eGenix.com'' time string parser ''mxDateTime.Parser'' for Python is the most notable one ![2]. We need to add such a powerful tool to provide a mature parser right from the start. If handled with care, similar but incompatible classes ''datetime.datetime'' and ''mxDateTime.!DateTime'' are not much of an issue. In fact it doesn't matter at all, as long as we stick to unix time stamp format for saving times in the db for other (good) reasons as well. (cboos): adding a new external dependency is a strong handicap for being considered for inclusion in Trac core. Have you really weighted the benefits of using mxDateTime? Maybe that helps to kick start the development, but you should really consider building your own utility functions, if you need functionality besides the one provided by the `datetime` classes. See for example how we integrate `pytz`: if present, we take benefit from it, if not, we provide our own replacement with reduced functionality. Actually I put it in here, just because I expected opposition against this dependency. I understand your point and made this an own topic under the [#StatusofDevelopment development section] above. ==== Considering demand for custom time field defaults ==== I can imagine various needs for defaults: pre-seed with * empty value (well, thats quite easy :-) * fixed date/time (much less demand, I guess, but this could help i.e. when facing an already fixed project termination) * current date/time * date/time with predefined offset from current date/time (like "now + 14d") * ... Each one looks like a perfectly reasonable way to preset a suitable default for one of different demands. But only a preset, of course, still subject to change on ticket creation as well during ticket life. It looks like a discussion would help to possibly strip nonsense from sense. We'll see then, how generic or special, simple or complex a default configuration satisfying most/all real world demand would have to become. ==== Handling ticket changed notification ==== I plan to let the notification system handle the conversion of unix time stamp values to date/time strings on it's own. I tested with a patched version of [th:wiki:AnnouncerPlugin AnnouncerPlugin] that worked. This is a good start for the [wiki:TracDev/Proposals/Announcer proposed replacement] of current Trac Notification System by Code from [th:wiki:AnnouncerPlugin AnnouncerPlugin]. This might even make per-user preference-based date/time formatting possible in the future. As adoption of code from [th:wiki:AnnouncerPlugin AnnouncerPlugin] is far from ready to replace current Ticket Notification, I feel the need to patch it as well. Right? ==== Development trace ==== Happened to have access to !SourceForge, so I choose to go and push code into a Mercurial repository there to enable collaboration based on a distributed VCS as per suggestion from IRC channel #trac at freenode.net. rblank offered to rebase my code on top of code from his own repository, so a least he would be able to rewiev more easily. I'll try it and combine this move with a trial on Mercurial patch queue. '''Done''', re-based on clone of rblank's Mercurial repository, all changes converted to patch series,[[BR]]cleared repository on Sourceforge, because previous work was to dirty and non-functional in early tests, I found that I forgot to ''--addremove'' after pull/merge and so missed new files as well as deletions Current development is based on Trac-0.12dev_~~r9115~~r9443 (see current state), testet with Python2.5 on ''Debian GNU/Linux'' stable. However it would be great to have it in trunk soon (trac-0.12.x?). (cboos): that's way too old to base new development upon, especially for timestamp related development, as r9210 introduced a big change here (time are now stored as microseconds elapsed since epoch, as bigints). Ah, well, thanks for the hint. We're at r9420 right now, any recommendation where to start from other than HEAD? (cboos): why not HEAD? Latest trunk should be fine. We're about to release a rc1 or a beta soon, so that might provide a stable basis. The question was obsoleted by current development. And I'll continue to work hard to keep pace and match all reasonable demands ASAP. I'll have to reinstall my test platform here and re-base changes then. However starting with standard unix time seconds was easier to debug for me (i.e. comparing to date +%s). And to_timestamp() messed up the date values converted from mxDatetime for some reason. I went for the included mxDatetime.Datetime.ticks() function to get good unix time seconds. So I may still need an own function to calculate unix time microseconds from unix time seconds. Checked changes between r9115 and r9435, prepare for updates this weekend. Found 23 changesets that did modify files I touched by now. Sounds quite big. Try to figure out a way to do it save for now, postpone to do it clever with least effort later. Might want to use Mercurial Queues or the like, but I fell I don't understand all the consequences. Halfway done, 11 merges are in the repository now, most importantly merged with the microseconds time stamp changeset r9210. In fact especially that changeset was not at all related to my modifications. I didn't touch my old function, that still uses ''to_datetime(ts)'' instead of the new ''from_utimestamp(ts)''. I don't use ''to_timestamp(dt)'', but the conversion from within class ''mxDateTime.!DateTime.ticks()''. However for the sake of clarity I think it will be good to shift to microseconds timestamps for custom timefields too. So this is on my !ToDo after finishing the whole merge marathon. '''Done''', merged changes for custom time field support with SVN changes up to trac SVN changeset r9442. == Related tickets == #710 asking for basic time tracking in Trac, especially missing native ''due_date'' and custom field type 'numeric'[[BR]] '''#1942 the main resource''' on requests and considerations '''related to the topic'''[[BR]] #1943 asking for time based calculations in queries, essentially based on #1943[[BR]] #1962 asking for due dates on tickets & email notification on overdue dates, several tricks and workarounds needed for using date strings the way you'd expect true time values to match, sort, etc.[[BR]] #2288 asked for date/time based ticket query functions, that exist by now[[BR]] #6466 asked for higher time stamp precision introduced with microseconds for ''time'' and ''changetime'' in r9210[[BR]] #8354 seeking to get more date/time related information from queries, following #2288[[BR]] #9209 suggests code cleanup for trac/ticket/model.py, as side effect of my code review for this project[[BR]] [th:ticket:3565 th#3565] asking for periodical email notification per ticket after certain inactivity time, comment points to [th:wiki:TracReminderScript TracReminderScript][[BR]] [th:ticket:5620 th#5620] asking for reminder function sending notification before/on current date passing a due_date set as ticket field == External related resources == Ideas and Open Source code was shamelessly taken from the following places: ![1] [http://seehuhn.de/pages/pdate Date and Time Representation in Python] by Jochen Voss[[BR]] ![2] [http://www.egenix.com eGenix.com], especially see ''mxDateTime'' [http://www.egenix.com/products/python/mxBase/mxDateTime/doc/#DateTime related documentation][[BR]] ![3] [http://docs.python.org/library/datetime.html#module-datetime Basic date and time types] from module ''datetime'' in Python>=2.3 Needed help about Python and development tools and found it i.e. at: [http://docs.python.org/tutorial/index.html The Python Tutorial] - the 'official' one[[BR]] [http://www.tutorialspoint.com/python/index.htm Python Tutorial] - and another[[BR]] [http://hgbook.red-bean.com/ Mercurial: The Definitive Guide] by Bryan O'Sullivan[[BR]] [http://mercurial.selenic.com/wiki/FAQ Mercurial Frequently Asked Questions]