Edgewall Software
Modify

Opened 2 years ago

Closed 6 months ago

#13040 closed defect (fixed)

TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'> - Clone button bug

Reported by: pcelba Owned by: Jun Omae
Priority: normal Milestone: 1.4.1
Component: ticket system Version: 1.3.2
Severity: normal Keywords: ticketclone time field
Cc: Branch:
Release Notes:

Fixed TypeError cloning a ticket that has a time custom field. Fixed Clone button not present when Reply button not present.

API Changes:
Internal Changes:

Description (last modified by Jun Omae)

How to Reproduce

While doing a POST operation on /newticket, Trac issued an internal error.

When creating a new ticket by the Clone button click and the original ticket contains Time field defined as

test_eight = time
test_eight.format = date
test_eight.label = An absolute date
test_eight.order = 5
test_eight.value = 

the Type error appears and also the warning on top saying:

Warning: The ticket field An absolute date is invalid: "1528603200000000" is an invalid date, or the date format is not known. Try "MMM d, y" or "YYYY-MM-DD" instead.

Request parameters:

{u'__FORM_TOKEN': u'af4e62a78a566df7aed49e9e',
 u'clone': u'+ Clone',
 u'field_blockedby': u'',
 u'field_blocking': u'',
 u'field_cc': u'',
 u'field_changetime': u'1528658560945708',
 u'field_client': u'',
 u'field_component': u'Infrastructure',
 u'field_description': u'Cloned from #1:\r\n> Some description here. Images will come later...\r\n> \r\n',
 u'field_estimatedhours': u'0',
 u'field_hours': u'0',
 u'field_keywords': u'',
 u'field_milestone': u'',
 u'field_owner': u'pcelba',
 u'field_parents': u'',
 u'field_priority': u'critical',
 u'field_summary': u'The first ticket with a large number of comments (cloned)',
 u'field_test_eight': u'1528603200000000',
 u'field_time': u'1528374973950465',
 u'field_totalhours': u'1640.5',
 u'field_type': u'task (no coding)',
 u'field_version': u'',
 u'preview': u''}

User agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36

System Information

Trac 1.3.2
Babel 2.6.0
Genshi 0.7 (without speedups)
Jinja2 2.10
pysqlite 2.6.0
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)]
pytz 2018.4
setuptools 39.2.0
SQLite 3.14.2
jQuery 1.12.4
jQuery UI 1.12.1
jQuery Timepicker 1.6.3

Enabled Plugins

timingandestimationplugin 1.5.9
traccustomfieldadmin 0.2.13
tracmastertickets 4.0.2
tracsubticketsplugin 0.5.3

Interface Customization

shared-htdocs
shared-templates
site-htdocs malotraccloud3.jpg
site-templates

Python Traceback

Traceback (most recent call last):
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\main.py", line 640, in _dispatch_request
    dispatcher.dispatch(req)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\main.py", line 263, in dispatch
    method=method)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1418, in render_template
    fragment, iterable, method)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1477, in _render_jinja_template
    page = self.render_template_string(template, data, text)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\web\chrome.py", line 1647, in render_template_string
    string = template.render(data)
  File "e:\pythonvirtual\trac13\lib\site-packages\jinja2\environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "e:\pythonvirtual\trac13\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket.html", line 132, in top-level template code
    <a href="#comment:${cnum}" class="${cls}">${prefix}${cnum}</a>
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\theme.html", line 22, in top-level template code
    # block body
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\templates\theme.html", line 128, in block "body"
    # block content
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket.html", line 621, in block "content"
    #     include 'ticket_box.html'
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\ticket\templates\ticket_box.html", line 137, in top-level template code
    ${pretty_dateinfo(value, field.format,
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\timeline\web_ui.py", line 280, in pretty_dateinfo
    absolute = user_time(req, format_date, date)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 915, in user_time
    return func(*args, **kwargs)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 324, in format_date
    return _format_datetime(t, format, tzinfo, locale, 'date')
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 274, in _format_datetime
    t = to_datetime(t, tzinfo or localtz)
  File "e:\pythonvirtual\trac13\lib\site-packages\trac\util\datefmt.py", line 179, in to_datetime
    type(t))
TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

Attachments (0)

Change History (19)

comment:1 by Jun Omae, 2 years ago

Description: modified (diff)
Priority: highnormal

Maybe a related issue #12714.

comment:2 by Jun Omae, 2 years ago

Keywords: ticketclone added

I'll post unit tests for the regression.

  • trac/ticket/web_ui.py

    diff --git a/trac/ticket/web_ui.py b/trac/ticket/web_ui.py
    index 351855e3a..cf1f0a35b 100644
    a b class TicketModule(Component):  
    767767                    break
    768768
    769769        # Data need for Javascript-specific logic
    770         old_values = {name: ticket[name]
     770        old_values = {name: unicode(ticket[name] or '')
    771771                      for name in [field['name'] for field in ticket.fields]}
    772772        old_values['id'] = ticket.id
    773773        add_script_data(req, {'comments_prefs': self._get_prefs(req),

comment:3 by Ryan J Ollos, 2 years ago

Yeah, I think that makes sense. TracJSONEncoder currently converts the datetime value to a timestamp, but it looks like the old_values should be strings because they are used as form values in the clone POST request: tags/trac-1.3.2/tracopt/ticket/htdocs/ticketclone.js@:26#L20.

Do the old_values have another use outside of the clone action? If not, I might expect them to be added by clone.py. I suppose even if they don't have another use in Trac it is good for the values to always be available for use by plugins.

Last edited 2 years ago by Ryan J Ollos (previous) (diff)

in reply to:  3 comment:4 by Jun Omae, 2 years ago

Replying to Ryan J Ollos:

Yeah, I think that makes sense. TracJSONEncoder currently converts the datetime value to a timestamp, but it looks like the old_values should be strings because they are used as form values in the clone POST request: tags/trac-1.3.2/tracopt/ticket/htdocs/ticketclone.js@:26#L20.

I consider we should convert to an iso8601 string rather than a timestamp integer. It is not easy to use the timestamp integer from javascript.

  • trac/util/presentation.py

    diff --git a/trac/util/presentation.py b/trac/util/presentation.py
    index 357d519ff..46165e94f 100644
    a b from jinja2.utils import soft_unicode  
    2828from jinja2._compat import iteritems
    2929
    3030from trac.core import TracError
    31 from .datefmt import to_utimestamp
     31from .datefmt import format_datetime, to_utimestamp, utc
    3232from .html import Fragment, classes, html_attribute, styles, tag
    3333from .text import javascript_quote
    3434
    class TracJSONEncoder(JSONEncoder):  
    469469        if isinstance(o, Undefined):
    470470            return ''
    471471        elif isinstance(o, datetime):
    472             return to_utimestamp(o)
     472            return format_datetime(o, 'iso8601', o.tzinfo or utc)
    473473        elif isinstance(o, Fragment):
    474474            return '"%s"' % javascript_quote(unicode(o))
    475475        return JSONEncoder.default(self, o)

It seems missing unit tests for to_json with datetime instance. We should add the unit tests.

Do the old_values have another use outside of the clone action? If not, I might expect them to be added by clone.py. I suppose even if they don't have another use in Trac it is good for the values to always be available for use by plugins.

The old_values is introduced in [15435,15436] (#12639) to integrate Jinja2. As far as I know, we have no other uses.

comment:5 by Ryan J Ollos, 2 years ago

Milestone: next-dev-1.3.x1.3.4

comment:6 by Ryan J Ollos, 23 months ago

Milestone: 1.3.41.3.5

comment:7 by Ryan J Ollos, 13 months ago

Milestone: 1.3.51.3.4
Owner: set to Ryan J Ollos
Status: newassigned

Added a test for comment:4 patch in log:rjollos.git:t13040_ticket_clone_time_field.

Also fixed issue in which Clone button was not added if ticket description was empty, because Reply button not present.

comment:8 by Ryan J Ollos, 13 months ago

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Committed in r16944, r16945.

comment:9 by Ryan J Ollos, 13 months ago

Owner: changed from Ryan J Ollos to Jun Omae

comment:10 by Jun Omae, 13 months ago

Ah, I noticed javascript errors are lead by [16944].

22:48:37.282 Error: Syntax error, unrecognized expression: unsupported pseudo: 26 jquery.js:2:12733
    jQuery 9
    addCloneFromComments http://192.168.11.27:3000/1.3-sqlite/chrome/ticketopt/ticketclone.js:38
    <anonymous> http://192.168.11.27:3000/1.3-sqlite/chrome/ticketopt/ticketclone.js:69
    jQuery 4

ticketclone.js expects c.date is an utimestamp integer. The patch in comment:2 only should be applied…

 38       btns = $("#trac-change-" + c.cnum + "-" + c.date + " .trac-ticket-buttons");
Last edited 13 months ago by Ryan J Ollos (previous) (diff)

comment:11 by Jun Omae, 13 months ago

Proposed changes in [d1843a1b1/jomae.git].

Last edited 13 months ago by Jun Omae (previous) (diff)

comment:12 by Ryan J Ollos, 13 months ago

Thanks for spotting. I forgot to test the clone-from-comment feature.

comment:13 by Ryan J Ollos, 13 months ago

Sorry I forgot to commit these changes. Will commit and create 1.3.5.

Last edited 13 months ago by Ryan J Ollos (previous) (diff)

comment:14 by Ryan J Ollos, 13 months ago

Committed in r16966. I'll create the 1.3.5 release later in the day.

comment:15 by massimo.b@…, 6 months ago

As you mentioned release 1.3.5, this issue should be already fixed in Trac 1.4?

We have a custom date field:

[ticket-custom]
duedate = time
duedate.format = date
duedate.label = Due date
duedate.order = 19

Cloning such a ticket fails like:

TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

The field data of the original ticket was set by the date dialog widget.

How to Reproduce

While doing a POST operation on /newticket, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{u'__FORM_TOKEN': u'9dcf799431961f51bc5afba8',
 u'clone': u'+',
 u'field_affectedmod': u'',
 u'field_cc': u'',
 u'field_ccb_prio': u'500',
 u'field_changedesc': u'',
 u'field_changetime': u'1580220545077606',
 u'field_component': u'server:central',
 u'field_config': u'',
 u'field_customer': u'0',
 u'field_description': u"Cloned from #2078:\r\n> An empty property value is rejected, if the corresponding pattern is not prefixed with '?' even if it allows zero length.\r\n> \r\n> {{{\r\n> mts_central   |    1782|200124T093234.070|CEN|D8|     [ENTER] int MT::Central::import_properties(const MT::Xof::Product&): product.getId(): '34' \r\n> mts_central   |    1782|200124T093234.070|CEN|D6|       Checked property 'Debitorennummer' = '19001'\r\n> mts_central   |    1782|200124T093234.070|CEN|ER|       int MT::Central::check_property(QString, QString, int):\r\n> mts_central   |    1782|200124T093234.070|CEN|ER|       Property 'GTIN-NTIN' = '' doesn't match the pattern '[0-9]{0,18}': 'Value must not be empty'\r\n> mts_central   |    1782|200124T093234.070|CEN|D8|     [LEAVE] int MT::Central::import_properties(const MT::Xof::Product&): product.getId(): '34' \r\n> mts_central   |    1782|200124T093234.072|DAB|D6|     1750:4: Transaction rollback\r\n> mts_central   |    1782|200124T093234.072|CEN|D8|   [LEAVE] int MT::Central::create_product(MT::Xof::Product&): product.getName(): '20401', product.getVersion(): '1', product.getId(): '34'\r\n> \r\n> }}}\r\n",
 u'field_developer': '',
 u'field_development': u'',
 u'field_documentation': u'',
 u'field_duedate': u'1580425200000000',
 u'field_editor': '',
 u'field_estimatedhours': u'0',
 u'field_impact': u'',
 u'field_integration': u'',
 u'field_integrator': '',
 u'field_issuecause': u'',
 u'field_issuedesc': u'',
 u'field_issueref': u'',
 u'field_keywords': u'',
 u'field_milestone': u'',
 u'field_owner': u'AKuit',
 u'field_parents': u'',
 u'field_platform': u'unspecified',
 u'field_priority': u'B',
 u'field_project': u'MT-IDT',
 u'field_propagation': u'',
 u'field_review': u'',
 u'field_reviewer': '',
 u'field_risk': u'',
 u'field_service': u'',
 u'field_serviceer': '',
 u'field_severity': u'undefined',
 u'field_summary': u'Central Service: Empty value does not match pattern with zero length allowed (cloned)',
 u'field_technicalnotes': u'',
 u'field_test': u'',
 u'field_tester': '',
 u'field_testnotes': u'',
 u'field_ticketsboard': u'0',
 u'field_time': u'1579863629713059',
 u'field_totalhours': u'0',
 u'field_type': u'Bug',
 u'field_version': u'',
 u'preview': u''}

User agent: #USER_AGENT#

System Information

Trac 1.4
Docutils 0.15.2
Genshi 0.7.3 (with speedups)
GIT 2.19.2
Jinja2 2.10.3
PostgreSQL server: 11.5, client: 11.5
psycopg2 2.8.3 (dt dec pq3 ext lo64)
Pygments 2.5.2
Python 2.7.17 (default, Jan 16 2020, 13:39:29)
[GCC 9.2.0]
pytz 2018.9
setuptools 44.0.0
Textile 3.0.3
jQuery #JQUERY#
jQuery UI #JQUERYUI#
jQuery Timepicker #JQUERYTP#

Enabled Plugins

AZ-NotificationSubscriber N/A
IncludeSource 0.4
KeepInterfaceSimple2Plugin 2.6
ticketsboardplugin 1.2.3.dev0
TracAccountManager 0.6.dev0
TracAutocompleteUsersPlugin 0.4.6
TracCodeReviewer 1.0.0.dev0
TracDiscussion 1.2.1
TracEstimationTools 0.5.0
TracFieldTooltip 0.8.5
TracFullBlogPlugin 0.1.6.3
TracHTTPAuth 1.2
TracMenus 0.3.0
TracpathTheme 1.0.1
TracQuiet 1.2.1
TracSimpleMultiProject 0.7.0.dev0
TracSQL 0.3.1
TracSqlQueryMacro 0.2
TracSubTicketsPlugin 0.5.4.dev0
TracTags 0.11
TracThemeEngine 2.2.4
TracWikiAutoComplete 1.4
TracWikiExtras 1.3.1
TracWikiPrintPlugin 3.0.dev0
TracWorkflowAdmin 0.12.0.6
TracXMLRPC 1.1.9.dev0
WikiTableMacro 0.3.dev0

Interface Customization

shared-htdocs
shared-templates
site-htdocs favicon.ico, favicon.png, logo, report.css, permasoft-christmas_h64.png, permasoft_h64.png, sorttable.js, style.css
site-templates account_details.html, admin_accountsconfig.html, admin_accountsnotification.html, admin_users.html, db_cleanup.html, login.html, prefs_account.html, register.html, reset_password.html, site_head.html, user_table.html, verify_email.html

Python Traceback

Traceback (most recent call last):
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/main.py", line 639, in dispatch_request
    dispatcher.dispatch(req)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/main.py", line 271, in dispatch
    method=method)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/chrome.py", line 1441, in render_template
    fragment, iterable, method)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/chrome.py", line 1516, in _render_jinja_template
    page = self.render_template_string(template, data, text)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/web/chrome.py", line 1675, in render_template_string
    string = template.render(data)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/ticket/templates/ticket.html", line 12, in top-level template code
    # extends 'layout.html'
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/templates/layout.html", line 12, in top-level template code
    # import "macros.html" as jmacros with context
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/templates/theme.html", line 22, in top-level template code
    # block body
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/templates/theme.html", line 128, in block "body"
    # block content
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/ticket/templates/ticket.html", line 630, in block "content"
    #     include 'ticket_box.html'
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/ticket/templates/ticket_box.html", line 137, in top-level template code
    ${pretty_dateinfo(value, field.format,
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/timeline/web_ui.py", line 280, in pretty_dateinfo
    absolute = user_time(req, format_date, date)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 917, in user_time
    return func(*args, **kwargs)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 324, in format_date
    return _format_datetime(t, format, tzinfo, locale, 'date')
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 274, in _format_datetime
    t = to_datetime(t, tzinfo or localtz)
  File "/mnt/data/trac/.local/lib64/python2.7/site-packages/trac/util/datefmt.py", line 179, in to_datetime
    type(t))
TypeError: expecting datetime, int, long, float, or None; got <type 'unicode'>

comment:16 by Jun Omae, 6 months ago

Milestone: 1.3.41.4.1
Resolution: fixed
Status: closedreopened

Thanks for the report. The issue is not fixed yet.

I think the old_values should have original ticket values. Currently, the old_values has ticket values with submitted values in preview.

  • trac/ticket/web_ui.py

    diff --git a/trac/ticket/web_ui.py b/trac/ticket/web_ui.py
    index 762485a40..0466d661b 100644
    a b class TicketModule(Component):  
    770770                    break
    771771
    772772        # Data need for Javascript-specific logic
    773         old_values = {name: ticket[name]
    774                       for name in [field['name'] for field in ticket.fields]}
     773        old_values = {}
     774        for field in ticket.fields:
     775            name = field['name']
     776            value = ticket._old.get(name, ticket[name])
     777            if name in ticket.time_fields and isinstance(value, (int, long)):
     778                value = format_datetime(value, 'iso8601', utc)
     779            old_values[name] = value
    775780        old_values['id'] = ticket.id
    776781        add_script_data(req, {'comments_prefs': self._get_prefs(req),
    777782                              'old_values': old_values,

I'll post unit tests for the issue, again….

comment:17 by Jun Omae, 6 months ago

Proposed changes in [66eeaf80c/jomae.git].

comment:18 by Ryan J Ollos, 6 months ago

The changes look good to me.

comment:19 by Jun Omae, 6 months ago

Resolution: fixed
Status: reopenedclosed

Thanks for the review. Committed in [17242] and merged in [17243].

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jun Omae.
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from Jun Omae to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.