TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'> - Clone button bug

[pcelba]

How to Reproduce

While doing a POST operation on /newticket, Trac issued an internal error.

When creating a new ticket by the Clone button click and the original ticket contains Time field defined as

test_eight = time
test_eight.format = date
test_eight.label = An absolute date
test_eight.order = 5
test_eight.value = 

the Type error appears and also the warning on top saying:

Warning: The ticket field An absolute date is invalid: "1528603200000000" is an invalid date, or the date format is not known. Try "MMM d, y" or "YYYY-MM-DD" instead.

Request parameters:

{u'__FORM_TOKEN': u'af4e62a78a566df7aed49e9e',
 u'clone': u'+ Clone',
 u'field_blockedby': u'',
 u'field_blocking': u'',
 u'field_cc': u'',
 u'field_changetime': u'1528658560945708',
 u'field_client': u'',
 u'field_component': u'Infrastructure',
 u'field_description': u'Cloned from #1:\r\n> Some description here. Images will come later...\r\n> \r\n',
 u'field_estimatedhours': u'0',
 u'field_hours': u'0',
 u'field_keywords': u'',
 u'field_milestone': u'',
 u'field_owner': u'pcelba',
 u'field_parents': u'',
 u'field_priority': u'critical',
 u'field_summary': u'The first ticket with a large number of comments (cloned)',
 u'field_test_eight': u'1528603200000000',
 u'field_time': u'1528374973950465',
 u'field_totalhours': u'1640.5',
 u'field_type': u'task (no coding)',
 u'field_version': u'',
 u'preview': u''}

User agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36

System Information

Trac	1.3.2
Babel	2.6.0
Genshi	0.7 (without speedups)
Jinja2	2.10
pysqlite	2.6.0
Python	2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)]
pytz	2018.4
setuptools	39.2.0
SQLite	3.14.2
jQuery	1.12.4
jQuery UI	1.12.1
jQuery Timepicker	1.6.3

Enabled Plugins

timingandestimationplugin	1.5.9
traccustomfieldadmin	0.2.13
tracmastertickets	4.0.2
tracsubticketsplugin	0.5.3

Interface Customization

shared-htdocs
shared-templates
site-htdocs	malotraccloud3.jpg
site-templates

Python Traceback

Traceback (most recent call last):
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\main.py", line 640, in _dispatch_request
    dispatcher.dispatch(req)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\main.py", line 263, in dispatch
    method=method)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1418, in render_template
    fragment, iterable, method)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1477, in _render_jinja_template
    page = self.render_template_string(template, data, text)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1647, in render_template_string
    string = template.render(data)
  File "e:\pythonvirtual\trac13\lib\site-packages\jinja2\environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "e:\pythonvirtual\trac13\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket.html", line 132, in top-level template code
    <a href="#comment:${cnum}" class="${cls}">${prefix}${cnum}</a>
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\theme.html", line 22, in top-level template code
    # block body
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\theme.html", line 128, in block "body"
    # block content
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket.html", line 621, in block "content"
    #     include 'ticket_box.html'
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket_box.html", line 137, in top-level template code
    ${pretty_dateinfo(value, field.format,
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\timeline\web_ui.py", line 280, in pretty_dateinfo
    absolute = user_time(req, format_date, date)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 915, in user_time
    return func(*args, **kwargs)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 324, in format_date
    return _format_datetime(t, format, tzinfo, locale, 'date')
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 274, in _format_datetime
    t = to_datetime(t, tzinfo or localtz)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 179, in to_datetime
    type(t))
TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

[Jun Omae]

Maybe a related issue #12714.

[Jun Omae]

I'll post unit tests for the regression.

trac/ticket/web_ui.py

@@ -767 +767 @@
                     break
 
         # Data need for Javascript-specific logic
-        old_values = {name: ticket[name]
+        old_values = {name: unicode(ticket[name] or '')
                       for name in [field['name'] for field in ticket.fields]}
         old_values['id'] = ticket.id
         add_script_data(req, {'comments_prefs': self._get_prefs(req),

[Ryan J Ollos]

Yeah, I think that makes sense. TracJSONEncoder​ currently converts the datetime value to a timestamp, but it looks like the old_values should be strings because they are used as form values in the clone POST request: tags/trac-1.3.2/tracopt/ticket/htdocs/ticketclone.js@:26#L20​.

Do the old_values have another use outside of the clone action? If not, I might expect them to be added by clone.py​. I suppose even if they don't have another use in Trac it is good for the values to always be available for use by plugins.

[Jun Omae]

Replying to Ryan J Ollos:

Yeah, I think that makes sense. TracJSONEncoder​ currently converts the datetime value to a timestamp, but it looks like the old_values should be strings because they are used as form values in the clone POST request: tags/trac-1.3.2/tracopt/ticket/htdocs/ticketclone.js@:26#L20​.

I consider we should convert to an iso8601 string rather than a timestamp integer. It is not easy to use the timestamp integer from javascript.

trac/util/presentation.py

@@ -28 +28 @@
 from jinja2._compat import iteritems
 
 from trac.core import TracError
-from .datefmt import to_utimestamp
+from .datefmt import format_datetime, to_utimestamp, utc
 from .html import Fragment, classes, html_attribute, styles, tag
 from .text import javascript_quote
 
@@ -469 +469 @@
         if isinstance(o, Undefined):
             return ''
         elif isinstance(o, datetime):
-            return to_utimestamp(o)
+            return format_datetime(o, 'iso8601', o.tzinfo or utc)
         elif isinstance(o, Fragment):
             return '"%s"' % javascript_quote(unicode(o))
         return JSONEncoder.default(self, o)

It seems missing unit tests for to_json with datetime instance. We should add the unit tests.

Do the old_values have another use outside of the clone action? If not, I might expect them to be added by clone.py​. I suppose even if they don't have another use in Trac it is good for the values to always be available for use by plugins.

The old_values is introduced in [15435,15436] (#12639) to integrate Jinja2. As far as I know, we have no other uses.

[Ryan J Ollos]

Added a test for comment:4 patch in log:rjollos.git:t13040_ticket_clone_time_field.

Also fixed issue in which Clone button was not added if ticket description was empty, because Reply button not present.

[Ryan J Ollos]

Committed in r16944, r16945.

[Jun Omae]

Ah, I noticed javascript errors are lead by [16944].

22:48:37.282 Error: Syntax error, unrecognized expression: unsupported pseudo: 26 jquery.js:2:12733
    jQuery 9
    addCloneFromComments http://192.168.11.27:3000/1.3-sqlite/chrome/ticketopt/ticketclone.js:38
    <anonymous> http://192.168.11.27:3000/1.3-sqlite/chrome/ticketopt/ticketclone.js:69
    jQuery 4

ticketclone.js expects c.date is an utimestamp integer. The patch in comment:2 only should be applied…

 38       btns = $("#trac-change-" + c.cnum + "-" + c.date + " .trac-ticket-buttons");

[Jun Omae]

Proposed changes in [d1843a1b1/jomae.git].

[Ryan J Ollos]

Thanks for spotting. I forgot to test the clone-from-comment feature.

[Ryan J Ollos]

Sorry I forgot to commit these changes. Will commit and create 1.3.5.

[Ryan J Ollos]

Committed in r16966. I'll create the 1.3.5 release later in the day.

[massimo.b@…]

As you mentioned release 1.3.5, this issue should be already fixed in Trac 1.4?

We have a custom date field:

[ticket-custom]
duedate = time
duedate.format = date
duedate.label = Due date
duedate.order = 19

Cloning such a ticket fails like:

TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

The field data of the original ticket was set by the date dialog widget.

How to Reproduce

While doing a POST operation on /newticket, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{u'__FORM_TOKEN': u'9dcf799431961f51bc5afba8',
 u'clone': u'+',
 u'field_affectedmod': u'',
 u'field_cc': u'',
 u'field_ccb_prio': u'500',
 u'field_changedesc': u'',
 u'field_changetime': u'1580220545077606',
 u'field_component': u'server:central',
 u'field_config': u'',
 u'field_customer': u'0',
 u'field_description': u"Cloned from #2078:\r\n> An empty property value is rejected, if the corresponding pattern is not prefixed with '?' even if it allows zero length.\r\n> \r\n> {{{\r\n> mts_central   |    1782|200124T093234.070|CEN|D8|     [ENTER] int MT::Central::import_properties(const MT::Xof::Product&): product.getId(): '34' \r\n> mts_central   |    1782|200124T093234.070|CEN|D6|       Checked property 'Debitorennummer' = '19001'\r\n> mts_central   |    1782|200124T093234.070|CEN|ER|       int MT::Central::check_property(QString, QString, int):\r\n> mts_central   |    1782|200124T093234.070|CEN|ER|       Property 'GTIN-NTIN' = '' doesn't match the pattern '[0-9]{0,18}': 'Value must not be empty'\r\n> mts_central   |    1782|200124T093234.070|CEN|D8|     [LEAVE] int MT::Central::import_properties(const MT::Xof::Product&): product.getId(): '34' \r\n> mts_central   |    1782|200124T093234.072|DAB|D6|     1750:4: Transaction rollback\r\n> mts_central   |    1782|200124T093234.072|CEN|D8|   [LEAVE] int MT::Central::create_product(MT::Xof::Product&): product.getName(): '20401', product.getVersion(): '1', product.getId(): '34'\r\n> \r\n> }}}\r\n",
 u'field_developer': '',
 u'field_development': u'',
 u'field_documentation': u'',
 u'field_duedate': u'1580425200000000',
 u'field_editor': '',
 u'field_estimatedhours': u'0',
 u'field_impact': u'',
 u'field_integration': u'',
 u'field_integrator': '',
 u'field_issuecause': u'',
 u'field_issuedesc': u'',
 u'field_issueref': u'',
 u'field_keywords': u'',
 u'field_milestone': u'',
 u'field_owner': u'AKuit',
 u'field_parents': u'',
 u'field_platform': u'unspecified',
 u'field_priority': u'B',
 u'field_project': u'MT-IDT',
 u'field_propagation': u'',
 u'field_review': u'',
 u'field_reviewer': '',
 u'field_risk': u'',
 u'field_service': u'',
 u'field_serviceer': '',
 u'field_severity': u'undefined',
 u'field_summary': u'Central Service: Empty value does not match pattern with zero length allowed (cloned)',
 u'field_technicalnotes': u'',
 u'field_test': u'',
 u'field_tester': '',
 u'field_testnotes': u'',
 u'field_ticketsboard': u'0',
 u'field_time': u'1579863629713059',
 u'field_totalhours': u'0',
 u'field_type': u'Bug',
 u'field_version': u'',
 u'preview': u''}

User agent: #USER_AGENT#

System Information

Trac	1.4
Docutils	0.15.2
Genshi	0.7.3 (with speedups)
GIT	2.19.2
Jinja2	2.10.3
PostgreSQL	server: 11.5, client: 11.5
psycopg2	2.8.3 (dt dec pq3 ext lo64)
Pygments	2.5.2
Python	2.7.17 (default, Jan 16 2020, 13:39:29) [GCC 9.2.0]
pytz	2018.9
setuptools	44.0.0
Textile	3.0.3
jQuery	#JQUERY#
jQuery UI	#JQUERYUI#
jQuery Timepicker	#JQUERYTP#

Enabled Plugins

AZ-NotificationSubscriber	N/A
IncludeSource	0.4
KeepInterfaceSimple2Plugin	2.6
ticketsboardplugin	1.2.3.dev0
TracAccountManager	0.6.dev0
TracAutocompleteUsersPlugin	0.4.6
TracCodeReviewer	1.0.0.dev0
TracDiscussion	1.2.1
TracEstimationTools	0.5.0
TracFieldTooltip	0.8.5
TracFullBlogPlugin	0.1.6.3
TracHTTPAuth	1.2
TracMenus	0.3.0
TracpathTheme	1.0.1
TracQuiet	1.2.1
TracSimpleMultiProject	0.7.0.dev0
TracSQL	0.3.1
TracSqlQueryMacro	0.2
TracSubTicketsPlugin	0.5.4.dev0
TracTags	0.11
TracThemeEngine	2.2.4
TracWikiAutoComplete	1.4
TracWikiExtras	1.3.1
TracWikiPrintPlugin	3.0.dev0
TracWorkflowAdmin	0.12.0.6
TracXMLRPC	1.1.9.dev0
WikiTableMacro	0.3.dev0

Interface Customization

shared-htdocs
shared-templates
site-htdocs	favicon.ico, favicon.png, logo, report.css, permasoft-christmas_h64.png, permasoft_h64.png, sorttable.js, style.css
site-templates	account_details.html, admin_accountsconfig.html, admin_accountsnotification.html, admin_users.html, db_cleanup.html, login.html, prefs_account.html, register.html, reset_password.html, site_head.html, user_table.html, verify_email.html

Python Traceback

Traceback (most recent call last):
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/main.py", line 639, in dispatch_request
    dispatcher.dispatch(req)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/main.py", line 271, in dispatch
    method=method)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/chrome.py", line 1441, in render_template
    fragment, iterable, method)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/chrome.py", line 1516, in _render_jinja_template
    page = self.render_template_string(template, data, text)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/chrome.py", line 1675, in render_template_string
    string = template.render(data)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/ticket/templates/ticket.html", line 12, in top-level template code
    # extends 'layout.html'
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/templates/layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/templates/theme.html", line 22, in top-level template code
    # block body
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/templates/theme.html", line 128, in block "body"
    # block content
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/ticket/templates/ticket.html", line 630, in block "content"
    #     include 'ticket_box.html'
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/ticket/templates/ticket_box.html", line 137, in top-level template code
    ${pretty_dateinfo(value, field.format,
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/timeline/web_ui.py", line 280, in pretty_dateinfo
    absolute = user_time(req, format_date, date)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 917, in user_time
    return func(*args, **kwargs)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 324, in format_date
    return _format_datetime(t, format, tzinfo, locale, 'date')
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 274, in _format_datetime
    t = to_datetime(t, tzinfo or localtz)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 179, in to_datetime
    type(t))
TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

[Jun Omae]

Thanks for the report. The issue is not fixed yet.

I think the old_values should have original ticket values. Currently, the old_values has ticket values with submitted values in preview.

trac/ticket/web_ui.py

@@ -770 +770 @@
                     break
 
         # Data need for Javascript-specific logic
-        old_values = {name: ticket[name]
-                      for name in [field['name'] for field in ticket.fields]}
+        old_values = {}
+        for field in ticket.fields:
+            name = field['name']
+            value = ticket._old.get(name, ticket[name])
+            if name in ticket.time_fields and isinstance(value, (int, long)):
+                value = format_datetime(value, 'iso8601', utc)
+            old_values[name] = value
         old_values['id'] = ticket.id
         add_script_data(req, {'comments_prefs': self._get_prefs(req),
                               'old_values': old_values,

I'll post unit tests for the issue, again….

[Jun Omae]

Proposed changes in [66eeaf80c/jomae.git].

[Ryan J Ollos]

The changes look good to me.

[Jun Omae]

Thanks for the review. Committed in [17242] and merged in [17243].