TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'> - Clone button bug

[pcelba]

How to Reproduce

While doing a POST operation on /newticket, Trac issued an internal error.

When creating a new ticket by the Clone button click and the original ticket contains Time field defined as

test_eight = time
test_eight.format = date
test_eight.label = An absolute date
test_eight.order = 5
test_eight.value = 

the Type error appears and also the warning on top saying:

Warning: The ticket field An absolute date is invalid: "1528603200000000" is an invalid date, or the date format is not known. Try "MMM d, y" or "YYYY-MM-DD" instead.

Request parameters:

{u'__FORM_TOKEN': u'af4e62a78a566df7aed49e9e',
 u'clone': u'+ Clone',
 u'field_blockedby': u'',
 u'field_blocking': u'',
 u'field_cc': u'',
 u'field_changetime': u'1528658560945708',
 u'field_client': u'',
 u'field_component': u'Infrastructure',
 u'field_description': u'Cloned from #1:\r\n> Some description here. Images will come later...\r\n> \r\n',
 u'field_estimatedhours': u'0',
 u'field_hours': u'0',
 u'field_keywords': u'',
 u'field_milestone': u'',
 u'field_owner': u'pcelba',
 u'field_parents': u'',
 u'field_priority': u'critical',
 u'field_summary': u'The first ticket with a large number of comments (cloned)',
 u'field_test_eight': u'1528603200000000',
 u'field_time': u'1528374973950465',
 u'field_totalhours': u'1640.5',
 u'field_type': u'task (no coding)',
 u'field_version': u'',
 u'preview': u''}

User agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36

System Information

Trac	1.3.2
Babel	2.6.0
Genshi	0.7 (without speedups)
Jinja2	2.10
pysqlite	2.6.0
Python	2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)]
pytz	2018.4
setuptools	39.2.0
SQLite	3.14.2
jQuery	1.12.4
jQuery UI	1.12.1
jQuery Timepicker	1.6.3

Enabled Plugins

timingandestimationplugin	1.5.9
traccustomfieldadmin	0.2.13
tracmastertickets	4.0.2
tracsubticketsplugin	0.5.3

Interface Customization

shared-htdocs
shared-templates
site-htdocs	malotraccloud3.jpg
site-templates

Python Traceback

Traceback (most recent call last):
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\main.py", line 640, in _dispatch_request
    dispatcher.dispatch(req)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\main.py", line 263, in dispatch
    method=method)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1418, in render_template
    fragment, iterable, method)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1477, in _render_jinja_template
    page = self.render_template_string(template, data, text)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1647, in render_template_string
    string = template.render(data)
  File "e:\pythonvirtual\trac13\lib\site-packages\jinja2\environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "e:\pythonvirtual\trac13\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket.html", line 132, in top-level template code
    <a href="#comment:${cnum}" class="${cls}">${prefix}${cnum}</a>
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\theme.html", line 22, in top-level template code
    # block body
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\theme.html", line 128, in block "body"
    # block content
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket.html", line 621, in block "content"
    #     include 'ticket_box.html'
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket_box.html", line 137, in top-level template code
    ${pretty_dateinfo(value, field.format,
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\timeline\web_ui.py", line 280, in pretty_dateinfo
    absolute = user_time(req, format_date, date)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 915, in user_time
    return func(*args, **kwargs)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 324, in format_date
    return _format_datetime(t, format, tzinfo, locale, 'date')
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 274, in _format_datetime
    t = to_datetime(t, tzinfo or localtz)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 179, in to_datetime
    type(t))
TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

[Jun Omae]

Maybe a related issue #12714.

[Jun Omae]

I'll post unit tests for the regression.

trac/ticket/web_ui.py

@@ -767 +767 @@
                     break
 
         # Data need for Javascript-specific logic
-        old_values = {name: ticket[name]
+        old_values = {name: unicode(ticket[name] or '')
                       for name in [field['name'] for field in ticket.fields]}
         old_values['id'] = ticket.id
         add_script_data(req, {'comments_prefs': self._get_prefs(req),

[Ryan J Ollos]

Yeah, I think that makes sense. TracJSONEncoder​ currently converts the datetime value to a timestamp, but it looks like the old_values should be strings because they are used as form values in the clone POST request: tags/trac-1.3.2/tracopt/ticket/htdocs/ticketclone.js@:26#L20​.

Do the old_values have another use outside of the clone action? If not, I might expect them to be added by clone.py​. I suppose even if they don't have another use in Trac it is good for the values to always be available for use by plugins.

[Jun Omae]

Replying to Ryan J Ollos:

Yeah, I think that makes sense. TracJSONEncoder​ currently converts the datetime value to a timestamp, but it looks like the old_values should be strings because they are used as form values in the clone POST request: tags/trac-1.3.2/tracopt/ticket/htdocs/ticketclone.js@:26#L20​.

I consider we should convert to an iso8601 string rather than a timestamp integer. It is not easy to use the timestamp integer from javascript.

trac/util/presentation.py

@@ -28 +28 @@
 from jinja2._compat import iteritems
 
 from trac.core import TracError
-from .datefmt import to_utimestamp
+from .datefmt import format_datetime, to_utimestamp, utc
 from .html import Fragment, classes, html_attribute, styles, tag
 from .text import javascript_quote
 
@@ -469 +469 @@
         if isinstance(o, Undefined):
             return ''
         elif isinstance(o, datetime):
-            return to_utimestamp(o)
+            return format_datetime(o, 'iso8601', o.tzinfo or utc)
         elif isinstance(o, Fragment):
             return '"%s"' % javascript_quote(unicode(o))
         return JSONEncoder.default(self, o)

It seems missing unit tests for to_json with datetime instance. We should add the unit tests.

Do the old_values have another use outside of the clone action? If not, I might expect them to be added by clone.py​. I suppose even if they don't have another use in Trac it is good for the values to always be available for use by plugins.

The old_values is introduced in [15435,15436] (#12639) to integrate Jinja2. As far as I know, we have no other uses.

[Ryan J Ollos]

Added a test for comment:4 patch in log:rjollos.git:t13040_ticket_clone_time_field.

Also fixed issue in which Clone button was not added if ticket description was empty, because Reply button not present.

[Ryan J Ollos]

Committed in r16944, r16945.