Context Navigation

Changes between Version 37 and Version 38 of TracTicketsCustomTimeFields

Timestamp:: May 18, 2010, 9:47:20 PM (14 years ago)
Author:: hasienda <hoff.st@…>
Comment:: moved content to proposal page, sorting information by relevance for intro or develompent/testing

Legend:

: Unmodified
: Added
: Removed
: Modified

TracTicketsCustomTimeFields

-              v37
+              v38
 = Custom Ticket Time Fields  =
+This page presents information on adding a new type 'time' to the list of all currently available [wiki:TracTicketsCustomFields custom ticket fields].
+== Alternative approaches ==
+Making a long story short: They are there for years to cure the lack of custom time fields for various project management purposes. I've seen them, but I don't like all the string manipulation, at least, if it comes to queries and report generation. I met others feeling the same.
+Believe me, it's time for a sane extension of current custom fields support. And custom time fields backed by true POSIX microsecond time stamps stored in ticket_custom db table is the best way I could think of. Read on and, if you like it, join in to make it happen, please.
+== Proposed Implementation ==
+=== Changes to Trac ===
+ * add 'time' to available types, inheriting most of the definition from 'text', a simple (one line) text field[[BR]]
+At that point TracQuery already shows from..to input fields on selection of custom time fields for query as desired.
+Praise to the flexible, modularized software design of Trac developers! Making it actually work and display results in pretty date/time format is a different thing.
+But now more than basic knowledge on how ticket data is handled inside Trac is needed.[[BR]]
+(I found not much hints so had to study the code and initially did most things wrong. A multiple dozen not working or buggy local code revisions later, after discussion in #python, etc. I decided on the implementation detailed below. Please discuss, if you know better, I'm still a Python beginner.)
+ * extend ''ticket.model'', ''ticket.web_ui'', ''util.datefmt'' and ''ticket/templates/ticket.html'' to
+  * allow for reading POSIX microsecond time stamps from db table ''ticket_custom''
+   * stored there as string for full compatibility with current custom fields implementation
+   * convert to datetime.datetime values and go on using that values internally like is done for core time values ''time'' and ''changetime''
+  * prepare time field display in ticket view (ticket.html)
+   * convert datetime.datetime time stamps to localized time strings
+   * add single quote arround strings read from db, i.e. left from former string based date field definitions
+   * add time format hint for input form fields
+  * parse user input and consolidate different flavors of time stamps
+   * initially done with help of ''mxDateTime.Parser''
+   * prepared for internal fall back to datetime based parser logic (still to be done) like is done for pytz with some basic internal logic
+  * write back POSIX microsecond time stamps to db
+  * take care for data used to send out properly formatted notification
+   * patched version of [th:wiki:AnnouncerPlugin AnnouncerPlugin] for now
+   * still unsure on patching Trac Ticket Notifications itself, since it looks like almost dead code (sorry, but Announcer is years ahead) and Announcer is already officially attempting to replace Trac Ticket Notifications (see below)
+ * extend ''ticket.query'' and ''ticket/templates/query_results.html'' to
+  * convert custom time strings with POSIX microsecond time stamps from ticket_custom db table to datetime.datetime values
+  * format custom time values for display differently than standard time values
+=== Status of Development ===
+==== Screenshot ====
+[[Image(TracQuery_on_CustomTimeField.2.png)]][[BR]]
+Trac 0.12dev screenshot with ticket list from query based on custom time field
+==== Code ====
+''base'': patches apply cleanly to Trac trunk ~~r9443~~ r9478[[BR]]
+''quality'': beta - read: code flaws should have not much effect, at least not to trac db, '''start of testing announced on 21-Apr-2010''', now expect on only minor changes to core of implementation until my stable release[[BR]]
+''focus'': bug-fixes, working ticket change notification for both [TracNotification Trac Notification] system and [th:wiki:AnnouncerPlugin AnnouncerPlugin][[BR]]
+''reviews'': so far just me, I need your help now[[BR]]
+''prospect'': since 0.12 is on the way (congrats!) and considering all the new features coming in here, it's more likely to go for 0.13 right away
+This is work in progress. Proposed code changes for Trac trunk reside in a Mercurial repository at bitbucket.org ^note 1^. The Mercurial Queue (MQ) aside that repo is the patch stack itself with some meta-data ^note 2^:
+This page presents information on adding a new type 'time' to the list of all currently available [wiki:TracTicketsCustomFields custom ticket fields].[[BR]]
+You may want to start by reading the [wiki:TracDev/Proposals/TracTicketsCustomTimeFields proposal page], that has more general information on this project.
+== Code review ==
+Proposed code changes for Trac trunk reside in a Mercurial repository at bitbucket.org ^note 1^. The Mercurial Queue (MQ) aside that repo is the patch stack itself with some meta-data ^note 2^:
  http://bitbucket.org/hasienda/custom-time/
 …
 I urge you to report back on any failure from code review or your testing. For test results, as always, mention your local setup (Trac version/revision, if not taken from my repo, Mercurial revision id of the patches). Any feedback will help to make the patches ready for production and later bring the code into Trac itself. That was my goal right from the start, not to produce just another [http://trac-hacks.org Trac hack].
 ===== Notes =====
+=== Notes ===
 . My repository is a clone of rblank's Mercurial repository, that in turn consists of changesets imported from Trac SVN repo.
 . The common setup for Mercurial Queue puts the patches into a Mercurial repo on their own. So looking at [http://bitbucket.org/hasienda/custom-time/changesets the changes] during development is easy.
 . Since there is no other sane way to push the changes to a public repository without making them irreversible, I'll not dare to push real revisions before substantial clearing with core Trac developers. Some of the patches will get forwarded independently for early, stepwise adoption into Trac, like that published in #9209).
 ==== !HowTo testing ====
+== !HowTo testing ==
 For help on applying the patches to Trac trunk source the traditional way, i.e. with {{{patch}}}, look at the patch documentation in the [http://bitbucket.org/hasienda/custom-time/src/tip/series series] file.
 But you'll see, Mercurial will save you and me some time and hassle, even more for true collaboration on bugs and feature requests. For the lazy or impatient ones not familiar with Mercurial I recommend to go along the lines of the following walk-through (Debian way).
 ===== Before you start =====
+=== Before you start ===
 I tested this on my own, really, but can't guarantee and so can't be responsible on behalf of you for any action on your system. Please try to understand, what you type and think twice. Look at documentation I referenced as [#Externalrelatedresources external related resources] at the bottom of this page. Be careful especially while you act as root, and at the very least '''don't blame me''' for any mess YOU did.
 ===== Quickstart =====
+=== Quickstart ===
 I assume that you'll have all pre-requisites like Python, setup-tools, Genshi and optionally Babel for the i18n support in place. If you're unsure or have issues with following procedure, look at the current [wiki:TracInstall Trac Install Guide] before crying at me.
 …
  * fire up the standalone webserver tracd hosting a trac environment you've prepared before.
-'''Notes for Ubuntu users'''
-On Ubuntu, you will have to ''apt-get install mercurial mercurial-common'' in order to get it working. In addition, you will have to add the following entries to ''/etc/mercurial/hgrc'':
-{{{
-# system-wide mercurial configuration file
-# See hgrc(5) for more information
-[extensions]
-mq =
-bookmarks =
-}}}
-Without these two entries, the above sequence of commands will fail.
 You may have {{{/usr/bin/tracd}}} and {{{/usr/local/bin/tracd}}} than at different versions. Install to /usr/local will normally overrule the other Trac installation without replacing it. So calling just {{{tracd}}} will kick {{{/usr/local/bin/tracd}}}. However, just make sure, you always know, what is going on. (I mention all this because I messed up on my own and now refrain from having multiple versions at the same system at a time.)
 …
 See [wiki:TracLogging Trac Logging] for more details on the matter.
+==== Known Bugs ====
+=== Notes ===
+==== Mercurial install/setup on Ubuntu 10.04 LTS ("Lucid Lynx") ====
+On Ubuntu, you will have to ''apt-get install mercurial mercurial-common'' in order to get mercurial working.
+In addition, you will have to add the following entries to ''/etc/mercurial/hgrc'':
+{{{
+# system-wide mercurial configuration file
+# See hgrc(5) for more information
+[extensions]
+mq =
+bookmarks =
+}}}
+Without these two entries, the above sequence of commands will fail.
+=== Known Bugs ===
+Issues (with priority) on new ticket creation
+ * (A) date/time stamp will not be saved, experience vary between failure on ticket creation (preventing ticket creation at all) and just trashing input silently
 Issues (with priority) occurring around ticket changes
  * (B) [TracNotification Trac Notification] system unpatched, need to test and investigate, if necessary at all - WiP here, code done, think I'll have it in soon, blocked only by inability to test it right now
 …
  * server internal exception turned out to be caused by different versions of trac installed in /usr and /usr/local on my test system interfering on build time
 ==== ToDo ====
+== ToDo ==
  * add jQuery datepicker function to custom time fields or even more general to input forms for time stamps like was requested in a [http://trac.edgewall.org/ticket/1942#comment:18 comment to ticket #1942]
 === Discussion ===
 ==== Development questions ====
+== Discussion ==
+=== Development questions ===
 In db table 'ticket_custom' a new row is created for every (custom time) field. Would it be worth the effort to delete this row, if the date/time is effectively purged? For now we end up with just an empty string {{{''}}}.
 ==== Time stamp format ====
+=== Time stamp format ===
 There was no argument against storing time stamps as string to arrange with current type definition "string" for values in ticket_custom db table. But switching to microseconds time stamp format recently adopted for Trac internal time fields ''time'' and ''changetime'' seems to be better than sticking with the former Unix time stamp, even if I prepared support for fractions of seconds by converting custom timestamps to float before.
 ==== Migrating from date/time strings ====
+=== Migrating from date/time strings ===
 I keep thinking of the case, where one already has a pile of date and wishes to switch to new date/time type now. Hope he/she finds it convenient that old values will be converted (silently) by current code, whenever a ticket is updated. But someone might come up with something like a db upgrade script for batch conversion/migration all a once. I think I'm not enough of a SQL guru to do that. So suggestions and contributions on the subject are very welcome.
 …
 TracQuery based on custom time field ignores non-numeric time stamps from db but still displays values, that could be parsed to datetime.datetime in ticket list
 ==== Unit tests ====
+=== Unit tests ===
 I got [ticket:1942#comment:23 advice], that presence of this would be a pre-requisite for inclusion into Trac.
 ==== ''No'' additional code dependency, but an option ====
+=== ''No'' additional code dependency, but an option ===
 Amongst the few new dependencies use of ''eGenix.com'' time string parser ''mxDateTime.Parser'' for Python is the most notable one ![2]. ~~We need~~ I needed to add such a powerful tool to provide a mature parser right from the start. If handled with care, similar but incompatible classes ''datetime.datetime'' and ''mxDateTime.!DateTime'' are not much of an issue. In fact it doesn't matter at all, as long as we stick to ~~unix time stamp~~ POSIX time format for saving ~~times in the~~ time stamps to db for other (good) reasons as well.
  (cboos): adding a new external dependency is a strong handicap for being considered for inclusion in Trac core. Have you really weighted the benefits of using mxDateTime? Maybe that helps to kick start the development, but you should really consider building your own utility functions, if you need functionality besides the one provided by the `datetime` classes. See for example how we integrate `pytz`: if present, we take benefit from it, if not, we provide our own replacement with reduced functionality.
 …
   While there was working code for parser functions before and I was especially satisfied with the performance of the needed date/time parser I've re-arranged the implementation. Now ''mxDateTime.Parser'' is not a dependency but an option. This was suggested by cboos and others for the sake of maintaining Trac code highly self-sufficient, and I understood this a feature on its own. Still ''mxDateTime.Parser'' has superior performance, if it comes to more unusual date/time user input, so it is still a recommended for maximum user satisfaction.
 ==== Considering demand for custom time field defaults ====
+=== Considering demand for custom time field defaults ===
 I can imagine various needs for defaults:
 pre-seed with
 …
 Each one looks like a perfectly reasonable way to preset a suitable default for one of different demands. But only a preset, of course, still subject to change on ticket creation as well during ticket life. It looks like a discussion would help to possibly strip nonsense from sense. We'll see then, how generic or special, simple or complex a default configuration satisfying most/all real world demand would have to become.
 ==== Considering demand for (custom) time field format settings ====
+=== Considering demand for (custom) time field format settings ===
 As I learned from the discussion within ticket #2182 support for implicitly setting date and time formats by locale selection is not enough for some users. I'd like to make this enhancement available for custom time fields first. Than we might be prepared to go on extending it to an implementation for all time fields. Maintaining changes for custom and standard time field in different patched should be desired to allow for an independed application as [ticket:2182#comment:73 proposed by me].
 ==== Handling ticket changed notification ====
+=== Handling ticket changed notification ===
 I plan to let the notification system handle the conversion of unix time stamp values to date/time strings on it's own. I tested with a patched version of [th:wiki:AnnouncerPlugin AnnouncerPlugin] that worked. This is a good start for the [wiki:TracDev/Proposals/Announcer proposed replacement] of current Trac Notification System by Code from [th:wiki:AnnouncerPlugin AnnouncerPlugin]. This might even make per-user preference-based date/time formatting possible in the future.
  As adoption of code from [th:wiki:AnnouncerPlugin AnnouncerPlugin] is far from ready to replace current Ticket Notification, I feel the need to patch it as well. Right?
 ==== Development traces ====
+=== Development traces ===
 Happened to have access to !SourceForge, so I choose to go and push code into a Mercurial repository there to enable collaboration based on a distributed VCS as per suggestion from IRC channel #trac at freenode.net.
  rblank offered to rebase my code on top of code from his own repository, so a least he would be able to rewiev more easily. I'll try it and combine this move with a trial on Mercurial patch queue.
 …
 == Related Trac plugins ==
  * [th:wiki:CustomFieldAdminPlugin CustomFieldAdminPlugin], modified version to ease the setup for new field type ^*^
+ * [th:wiki:DateFieldPlugin DateFieldPlugin] working flawlessly with custom time fields, since these are still simple text input form fields
+ * [th:wiki:CalendarPopUpPlugin CalendarPopUpPlugin] is working flawlessly with custom time fields, since these are still simple text input form fields
+ * [th:wiki:DateFieldPlugin DateFieldPlugin] does content validation to enforce it's idea of proper date/time string formatting, that will in most cases conflict with the input rewrite done by the parser of the time field implementation, however comparing date picker with that of [th:wiki:CalendarPopUpPlugin CalendarPopUpPlugin] this one looks superior in general (integration of an external, stable code base, easy customization support, i18n/l10n support, etc.), so it qualified for this project (see [#ToDo ToDo section])
  * [th:wiki:AnnouncerPlugin AnnouncerPlugin], modified version produces reports with pretty date/time strings according to locale setting of trac(d) proccess ^*^
 ^*^ patch is available inside repo, see subdirectory ./plugin
-== Related tickets ==
- #710 asking for basic time tracking in Trac, especially missing native ''due_date'' and custom field type 'numeric'[[BR]]
- '''#1942 the main resource''' on requests and considerations '''related to the topic'''[[BR]]
- #1943 asking for time based calculations in queries, essentially based on #1943[[BR]]
- #1962 asking for due dates on tickets & email notification on overdue dates, several tricks and workarounds needed for using date strings the way you'd expect true time values to match, sort, etc.[[BR]]
- #2182 asking for better, directly (user- and system-)configurable date and time formats in contrast to current selection via locale[[BR]]
- #2288 asked for date/time based ticket query functions, that exist by now[[BR]]
- #6466 asked for higher time stamp precision introduced with microseconds for ''time'' and ''changetime'' in r9210[[BR]]
- #8354 seeking to get more date/time related information from queries, following #2288[[BR]]
- #9209 suggests code cleanup for trac/ticket/model.py, as side effect of my code review for this project[[BR]]
- [th:ticket:3565 th#3565] asking for periodical email notification per ticket after certain inactivity time, comment points to [th:wiki:TracReminderScript TracReminderScript][[BR]]
- [th:ticket:5620 th#5620] asking for reminder function sending notification before/on current date passing a due_date set as ticket field
 == External related resources ==
 …
  [http://ches.nausicaamedia.com/articles/technogeekery/using-mercurial-queues-and-bitbucket-org Using Mercurial Queues and bitbucket.org] - a semi-official guide[[BR]]
  [http://www.urbandictionary.com/ Define your world] - explaining internet slang at urbandictionary.com[[BR]]
  [http://www.noslang.com/ Internet Slang Dictionary & Translator] at noslang.com
+ [http://www.noslang.com/ Internet Slang Dictionary & Translator] at noslang.com[[BR]]