Improve functional testing framework

[Ryan J Ollos]

It was previously discussed in comment:3:ticket:11014 that it would be nice to improve the existing functional test infrastructure. ​Selenium may be a good to utilize.

The current functional test infrastructure based on Twill has some deficiencies:

• The tests execute slowly.
• JavaScript and CSS cannot be tested.
• It is not possible to observe the functional tests manipulating pages in the browser.
• The functional tests are hand-coded using assertions with regexs. The tests are difficult to write, difficult to maintain and errors are easily made.
• Twill does not seem to be well-maintained (#11642).

See also #8170.

[figaro]

Selenium has a few ​drawbacks.

Has ​Twister ever been considered, which is also Python based and Apache licensed?

Finally, if it is GUI testing which is the focus of the functional tests, then jQuery may be of use for its ability to traverse the DOM.

[Christian Boos]

Unless I'm mistaken, looks like Twister needs Selenium as well for testing Web apps (​http://twistertesting.luxoft.com/blog/howto-twister-selenium/), so I'm not sure if that's buying us anything.

I think as a first step we should try to using Selenium directly (​http://seleniumhq.github.io/selenium/docs/api/py/).

(tentatively grabbing the ticket)

[figaro]

As a more end-user friendly alternative, we could record functional tests in GhostInspector and then ​export the test scripts as Selenium scripts.

[Ryan J Ollos]

Milestone renamed

[Peter Suter]

I experimented with using Selenium instead of twill (without any previous experience):

• pip install selenium
• ​Download Firefox selenium "webdriver" (or change webdriver.Firefox() below)
• Checkout log:jomae.git@1.5-py3 from #12130
• Apply experimental hacks: attachment:selenium_py3_tests_experiment.diff​
• Run trac\wiki\tests\functional.py

The browser starts, and Trac loads, but then I get stuck:

selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?[...]The connection to the server was reset while the page was loading.

[Jun Omae]

Replying to Peter Suter:

The browser starts, and Trac loads, but then I get stuck:

selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?[...]The connection to the server was reset while the page was loading.

It seems to be caused by login(). Fixed in log:jomae.git@1.5-py3-t11988.

python  -m unittest  trac.tests.functional.test_suite
E.EE.FFEEEF.EFFEEEEFFFFEEFFE.EEEEEEEFEEEEEEEEEEEEEEFEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEFEFEFFFFFFFEEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEEEEEFFE.FEFEFFFFEEFFFFEEFFEEFFEEFEFEFFEEEEEEEE.EEEEEEEEEEEEEEEEEEEEFEFFEFFEFFFFFFEFEFFFF.
...
FAILED (failures=91, errors=123)
Makefile:479: recipe for target 'functional-test' failed
make: *** [functional-test] Error 1

Updated the branch:

.....FFE..F.F.E..E.EEEE....E.EFEEEEEEEFEEEEEEEEEEEEEEFEEEEEEEEEFEEEFEEEFEEEFEEEEEEEEEEEEEEEEEEEEEE....FEEEEEEEFFFEEEEF...EEEEEEEEEEEEEEEE.F..EEEEEEEE.E.FFEEEEEEFFEEE.E.EEEEE.E..E...E..EEE...s.....EEEEFEEFEEFEEEEE.F.....E.
FAILED (failures=25, errors=141, skipped=1)

[Peter Suter]

Very good progress!

[Jun Omae]

Updated the branch: log:jomae.git@1.5-py3-t11988 (diff:jomae.git@1.5-py3:1.5-py3-t11988)

Ran 222 tests in 768.362s

OK (skipped=1)

TODO

• fix webdriver not working in AppVeyor
  • SessionNotCreatedException: Message: Unable to find a matching set of capabilities
  • ​https://ci.appveyor.com/project/jun66j5/trac/builds/31088303/job/3diok3jrdd996kq1#L1231
• restore html validation (html validator library is needed)

[Jun Omae]

Updated jomae.git@1.5-py3-t11988 branch. Firefox webdriver works fine on AppVeyor, now.

[Ryan J Ollos]

Nice work! I've been long hoping for this change. Finally we have a modern functional test framework.

I didn't have time to get an understanding of the changes yet, but I did run the tests on OSX @ [9fe048c78/jomae.git] (Selenium 3.141.0) and had a few failures:

======================================================================
ERROR: runTest (trac.ticket.tests.functional.admin.TestAdminVersionRemoveMulti)
Admin remove multiple versions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 908, in runTest
    tc.notfind("%s%s" % (name, i))
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 172, in notfind
    raise AssertionError('Regex matched: {!r} matches {!r} in {}'
AssertionError: Regex matched: b'MultiRemoveVersion0' matches 'MultiRemoveVersion0' in file:///Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/testenv/trac/log/TestAdminVersionRemoveMulti.html

======================================================================
ERROR: runTest (trac.ticket.tests.functional.admin.TestAdminVersionNonRemoval)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 279, in wrapper
    return fn(*args, **kwargs)
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 918, in runTest
    tc.find('No version selected')
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 180, in find
    raise AssertionError("Regex didn't match: {!r} not found in {}"
AssertionError: Regex didn't match: 'No version selected' not found in file:///Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/testenv/trac/log/TestAdminVersionNonRemoval.html

----------------------------------------------------------------------
Ran 205 tests in 749.062s

FAILED (errors=2)
make: *** [functional-test] Error 1

[Jun Omae]

Hmm, unable to reproduce the failures on my Ubuntu and OSX environments.

After running functional tests, please check testenv/trac/log/trac.log and geckodriver.log.

TestAdminVersionRemoveMulti

2020-03-13 20:05:15,806 Trac[cache] INFO: repos rev [0] != cached rev [None] in '(default)'
2020-03-13 20:05:15,811 Trac[api] INFO: Synchronized '(default)' repository in 0.01 seconds
2020-03-13 20:05:15,812 Trac[model] INFO: Deleting version 'MultiRemoveVersion2'
2020-03-13 20:05:15,817 Trac[model] INFO: Deleting version 'MultiRemoveVersion1'
2020-03-13 20:05:15,817 Trac[model] INFO: Deleting version 'MultiRemoveVersion0'
2020-03-13 20:05:15,854 Trac[cache] INFO: repos rev [0] != cached rev [None] in '(default)'
2020-03-13 20:05:15,856 Trac[api] INFO: Synchronized '(default)' repository in 0.00 seconds

TestAdminVersionNonRemoval

2020-03-13 20:05:16,384 Trac[cache] INFO: repos rev [0] != cached rev [None] in '(default)'
2020-03-13 20:05:16,389 Trac[api] INFO: Synchronized '(default)' repository in 0.01 seconds
2020-03-13 20:05:16,390 Trac[main] WARNING: [127.0.0.1] HTTPInternalServerError: 500 Trac Error (No version selected), <RequestWithSession "POST '/admin/ticket/versions'">, referrer 'http://127.0.0.1:34048/admin/ticket/versions'
2020-03-13 20:05:16,638 Trac[cache] INFO: repos rev [0] != cached rev [None] in '(default)'
2020-03-13 20:05:16,639 Trac[api] INFO: Synchronized '(default)' repository in 0.00 seconds

[Ryan J Ollos]

I commented out all the tests except those two and they pass running:

$ make test=trac.ticket.tests.functional.admin

However, if I run all the tests in that module the 2 failures occur.

I've attached the two logs. I currently have a problem with my SVN 1.14dev installation, so those errors are shown in the log.

[Jun Omae]

I found two TimedPromise timed out after 500 ms warnings after submitting form in the geckodriver.log. The same issue has been filed at ​https://github.com/mozilla/geckodriver/issues/1608.

When executing tc.submit("remove") in TestAdminVersionRemoveMulti:

1584129752945	Marionette	DEBUG	0 -> [0,6330,"WebDriver:ElementClick",{"id":"3f284e29-0b74-7542-accc-f6e4198191ad"}]
1584129753451	Marionette	WARN	TimedPromise timed out after 500 ms: stacktrace:
TimedPromise/<@chrome://marionette/content/sync.js:244:13
TimedPromise@chrome://marionette/content/sync.js:229:10
interaction.flushEventLoop@chrome://marionette/content/interaction.js:416:10
webdriverClickElement@chrome://marionette/content/interaction.js:182:31
1584129753652	Marionette	DEBUG	[68] Canceled page load listener because no navigation has been detected
1584129753653	Marionette	DEBUG	0 <- [1,6330,null,{"value":null}]

When executing tc.submit('remove', formname='version_table') in TestAdminVersionNonRemoval:

1584129754191	Marionette	DEBUG	0 -> [0,6340,"WebDriver:ElementClick",{"id":"1763a071-d428-dd47-9b4f-d3062250ec22"}]
1584129754704	Marionette	WARN	TimedPromise timed out after 500 ms: stacktrace:
TimedPromise/<@chrome://marionette/content/sync.js:244:13
TimedPromise@chrome://marionette/content/sync.js:229:10
interaction.flushEventLoop@chrome://marionette/content/interaction.js:416:10
webdriverClickElement@chrome://marionette/content/interaction.js:182:31
1584129754908	Marionette	DEBUG	[73] Canceled page load listener because no navigation has been detected
1584129754909	Marionette	DEBUG	0 <- [1,6340,null,{"value":null}]

I'm using the following in my OSX environment. What versions are you using?

$ /Applications/Firefox.app/Contents/MacOS/firefox-bin --version
Mozilla Firefox 73.0.1

$ geckodriver --version
geckodriver 0.26.0

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G103

[Ryan J Ollos]

Replying to Jun Omae:

What versions are you using?

Thank you for investigating.

$ /Applications/Firefox.app/Contents/MacOS/firefox-bin --version
Mozilla Firefox 74.

$ geckodriver --version
geckodriver 0.26.0

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.3
BuildVersion:	19D76

[Jun Omae]

• restore html validation (html validator library is needed)

We could use ​pytidylib to validate html with Python 2.7 and 3.5 through 3.8. (jomae.git@1.5-py3-tidylib [e575f67c4/jomae.git]). However, latest of the module has been released at Nov 16, 2016 (4 years ago).

Thoughts?

[Ryan J Ollos]

With the exception of the two tests noted in comment:12, all tests passing for me on OSX with Python 3.5.9, 3.6.10, 3.7.6, 3.8.2. I'll study pytidylib changes next.

[anonymous]

Considering edits in TracDev/UnitTests@29, there have been no releases for ​pypi:figleaf since 2008. I've never used it. I think we should just remove the references to it in the codebase, perhaps even for 1.4-stable.

[Ryan J Ollos]

Replying to anonymous:

Considering edits in TracDev/UnitTests@29, there have been no releases for ​pypi:figleaf since 2008. I've never used it. I think we should just remove the references to it in the codebase, perhaps even for 1.4-stable.

That comment was by me.

Proposed changes for 1.0-stable: [21412da3b/rjollos.git].

[Ryan J Ollos]

Replying to Jun Omae:

We could use ​pytidylib to validate html with Python 2.7 and 3.5 through 3.8. (jomae.git@1.5-py3-tidylib [e575f67c4/jomae.git]). However, latest of the module has been released at Nov 16, 2016 (4 years ago).

Thoughts?

They are ​looking for a new maintainer.

The library is only 812 lines, so it seems like it wouldn't take much effort to maintain or fork, if the need arises.

$ find pytidylib -name \*\.py -type f -print0 | xargs -0 cat | wc -l
     812

If you think it works well I'd say just go with it, since we can change the HTML validation library later with little or no impact to users of Trac.

Have you installed ​HTML Tidy on Windows?

[Ryan J Ollos]

On OSX I had:

$ tidy -v
HTML Tidy for Mac OS X released on 31 October 2006 - Apple Inc. build 17.1
$ which tidy
/usr/local/bin/tidy
$ brew search tidy
==> Formulae
perltidy                      tidy-html5                    tidyp

After brew install tidy-html5 I put /usr/local/Cellar/tidy-html5/5.6.0/bin on PATH and:

$ tidy -v
HTML Tidy for Apple macOS version 5.6.0
$ which tidy
/usr/local/Cellar/tidy-html5/5.6.0/bin/tidy

Seems to work well and catching some issues:

======================================================================
ERROR: runTest (trac.tests.functional.testcases.RegressionTestTicket7209)
Test for regression of https://trac.edgewall.org/ticket/7209
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/testcases.py", line 193, in runTest
    self._tester.go_to_ticket(ticketid)
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/tester.py", line 133, in go_to_ticket
    self.go_to_url(ticket_url)
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/tester.py", line 117, in go_to_url
    tc.go(url)
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 135, in go
    self._validate_html(self.get_source())
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 471, in _validate_html
    raise AssertionError('tidylib found %d error(s) in %s\n\n%s' %
AssertionError: tidylib found 1 error(s) in file:///Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/testenv/trac/log/RegressionTestTicket7209.html

line 476 column 11 - Warning: <div> attribute "id" lacks value
...

Ran 222 tests in 680.280s

FAILED (errors=27)
make: *** [functional-test] Error 1

Here is the full output: pytidylib_osx.txt​

[Ryan J Ollos]

Replying to Jun Omae:

• restore html validation (html validator library is needed)

We could use ​pytidylib to validate html with Python 2.7 and 3.5 through 3.8. (jomae.git@1.5-py3-tidylib [e575f67c4/jomae.git]). However, latest of the module has been released at Nov 16, 2016 (4 years ago).

What's your feeling about the state of the Python 3 work? Is there anything major to be done?

It seems to work well enough and I think it would be good to get it out to users as soon as possible to start getting feedback. Also, pushing to trunk soon would avoid too much divergence as I'm pushing changes to 1.4-stable and trunk.

[Ryan J Ollos]

Replying to anonymous:

Considering edits in TracDev/UnitTests@29, there have been no releases for ​pypi:figleaf since 2008. I've never used it. I think we should just remove the references to it in the codebase, perhaps even for 1.4-stable.

Edited TracDev/UnitTests@30.

Removed in r17311, r17315, merged in [17312:17314], [17316:17318].

[Ryan J Ollos]

Replying to Jun Omae:

• restore html validation (html validator library is needed)

We could use ​pytidylib to validate html with Python 2.7 and 3.5 through 3.8. (jomae.git@1.5-py3-tidylib [e575f67c4/jomae.git]). However, latest of the module has been released at Nov 16, 2016 (4 years ago).

Thoughts?

Could pytidylib also be used for validation in jinjachecker.py?

Would you want to backport it to 1.4-stable?

[Ryan J Ollos]

Replying to Ryan J Ollos:

Replying to Jun Omae:

What versions are you using?

Thank you for investigating.

$ /Applications/Firefox.app/Contents/MacOS/firefox-bin --version
Mozilla Firefox 74.

$ geckodriver --version
geckodriver 0.26.0

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.3
BuildVersion:	19D76

No failures now, but I'm not sure what fixed it:
On closer look, failures still occurring.

$ /Applications/Firefox.app/Contents/MacOS/firefox-bin --version
Mozilla Firefox 76.0.1

$ geckodriver --version
geckodriver 0.26.0

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.4
BuildVersion:	19E287

[Ryan J Ollos]

Replying to Ryan J Ollos:

Replying to anonymous:

Considering edits in TracDev/UnitTests@29, there have been no releases for ​pypi:figleaf since 2008. I've never used it. I think we should just remove the references to it in the codebase, perhaps even for 1.4-stable.

Edited TracDev/UnitTests@30.

Removed in r17311, r17315, merged in [17312:17314], [17316:17318].

I missed a few instances because I failed to do a case-insensitive search for figleaf.

Additional changes on 1.0-stable in r17381, merged in [17382:17384].

[Ryan J Ollos]

After [0db670dfb/rjollos.git] (log:rjollos.git:1.5-py3-tidylib.4), I get two different failures on the Version Admin pages (comment:15:ticket:11988), with TimedPromise timed out after 500 ms in the logs:

======================================================================
ERROR: runTest (trac.ticket.tests.functional.admin.TestAdminVersionDefault)
Admin set default version
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 987, in runTest
    tc.notfind('type="radio" name="default" checked="checked" value=".+"')
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 181, in notfind
    s, url))
AssertionError: Regex matched: b'type="radio" name="default" checked="checked" value="DefaultVersion"' matches 'type="radio" name="default" checked="checked" value=".+"' in file:///Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/testenv/trac/log/TestAdminVersionDefault.html

======================================================================
ERROR: runTest (trac.ticket.tests.functional.admin.TestTicketDefaultValues)
Test for regression of https://trac.edgewall.org/ticket/10772
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 1044, in runTest
    find_prop('version')
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 1023, in find_prop
    % {'field': field})
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 188, in find
    .format(s, url))
AssertionError: Regex didn't match: '<td headers="h_version">\\s*</td>' not found in file:///Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/testenv/trac/log/TestTicketDefaultValues.html

----------------------------------------------------------------------
Ran 58 tests in 211.043s

FAILED (errors=2)
make: *** [all] Error 1

Both failures appear to be due to the Clear default submit not being executed. For example, in TestAdminVersionDefault:

 982         print(self._testenv.get_config('ticket', 'default_version'))
 983         tc.submit('clear', formname='version_table')
 984         print(self._testenv.get_config('ticket', 'default_version'))

⇒

........................................................DefaultVersion

DefaultVersion

EE

In the third line of output, an empty string rather than DefaultVersion should be printed.

[Jun Omae]

Could pytidylib also be used for validation in jinjachecker.py?

I missed jinjachecker.py uses lxml for html validation. So that, I didn't intend to use in jinjachecker.py. However, I think that's good idea if possible.

Would you want to backport it to 1.4-stable?

Yes. The html validation in functional tests doesn't work in 1.4-stable. It works only when xhtml1-strict.dtd is in the contents but <!DOCTYPE html> is just generated from layout.html.

[Ryan J Ollos]

Follow-on to comment:29. If browser is run in non-headless mode:

python  -m unittest  trac.ticket.tests.functional.admin.test_suite
...................................................F......
======================================================================
FAIL: runTest (trac.ticket.tests.functional.admin.TestAdminVersionDetailTime)
Admin version detail set time
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 879, in runTest
    tc.submit('save')
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 367, in submit
    element.click()
  File "/Users/rjollos/.pyenv/versions/trac-py3610/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/Users/rjollos/.pyenv/versions/trac-py3610/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "/Users/rjollos/.pyenv/versions/trac-py3610/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Users/rjollos/.pyenv/versions/trac-py3610/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: Element <input class="trac-disable-on-submit" name="save" type="submit"> is not clickable at point (292,728) because another element <div id="ui-datepicker-div" class="ui-datepicker ui-widget ui-widget-content ui-helper-clearfix ui-corner-all"> obscures it


----------------------------------------------------------------------
Ran 58 tests in 194.128s

FAILED (failures=1)
make: *** [all] Error 1

That issue can be fixed by [89b0af0ac/rjollos.git]. However, going back to headless mode and comment:29 errors occur again.

However, commenting out all other test cases and running just TestAdminVersionDefault and TestTicketDefaultValues the errors don't occur.

The failure seems to occur when a certain number of versions are created. I experimented with running just the Version test cases.

1100     suite.addTest(TestAdminVersion())¬
1101     suite.addTest(TestAdminVersionAuthorization())¬
1102     suite.addTest(TestAdminVersionDuplicates())¬
1103     suite.addTest(TestAdminVersionDetail())¬
1104     suite.addTest(TestAdminVersionDetailTime())¬
1105     suite.addTest(TestAdminVersionDetailCancel())¬
1106     suite.addTest(TestAdminVersionRemove())¬
1107     suite.addTest(TestAdminVersionRemoveMulti())¬
1108     suite.addTest(TestAdminVersionNonRemoval())¬
1109     suite.addTest(TestAdminVersionDefault())¬
1110     suite.addTest(TestTicketDefaultValues())¬

Comment out TestAdminVersion and the errors don't occur. Add another instance of self._tester.create_version() in that function and a different error occurs:

python  -m unittest  trac.ticket.tests.functional.admin.test_suite
........E..
======================================================================
ERROR: runTest (trac.ticket.tests.functional.admin.TestAdminVersionNonRemoval)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 291, in wrapper
    return fn(*args, **kwargs)
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/ticket/tests/functional/admin.py", line 944, in runTest
    tc.find('No version selected')
  File "/Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/trac/tests/functional/better_twill.py", line 192, in find
    .format(s, url))
AssertionError: Regex didn't match: 'No version selected' not found in file:///Users/rjollos/Documents/Workspace/trac-dev/jomae-1.5-py3/testenv/trac/log/TestAdminVersionNonRemoval.html

----------------------------------------------------------------------
Ran 11 tests in 52.495s

FAILED (errors=1)
make: *** [all] Error 1

Very strange!

[Jun Omae]

...
selenium.common.exceptions.ElementClickInterceptedException: Message: Element <input class="trac-disable-on-submit" name="save" type="submit"> is not clickable at point (292,728) because another element <div id="ui-datepicker-div" class="ui-datepicker ui-widget ui-widget-content ui-helper-clearfix ui-corner-all"> obscures it
...

That issue can be fixed by [89b0af0ac/rjollos.git]. However, going back to headless mode and comment:29 errors occur again.

Could you please try the following patch (sending ESCAPE to close datepicker):

diff --git a/trac/tests/functional/better_twill.py b/trac/tests/functional/better_twill.py
index fa1ddbe81..d4ee85cb7 100755
--- a/trac/tests/functional/better_twill.py
+++ b/trac/tests/functional/better_twill.py
@@ -375,6 +375,12 @@ if selenium:
             element = self._find_by(*args, **kwargs)
             ActionChains(self.driver).move_to_element(element).perform()

+        def send_keys(self, *args):
+            chains = ActionChains(self.driver)
+            for arg in args:
+                chains.send_keys(arg)
+            chains.perform()
+
         def wait_for(self, condition, *args, **kwargs):
             element = self._find_by(*args, **kwargs)
             method = getattr(expected_conditions, condition)
diff --git a/trac/ticket/tests/functional/admin.py b/trac/ticket/tests/functional/admin.py
index d78869996..edd4e0459 100644
--- a/trac/ticket/tests/functional/admin.py
+++ b/trac/ticket/tests/functional/admin.py
@@ -874,9 +874,8 @@ class TestAdminVersionDetailTime(FunctionalTwillTestCaseSetup):
         name = self._tester.create_version()
         tc.follow(name)

-        # Clear value and send ENTER to close the datepicker.
-        tc.formvalue('edit', 'time', tc.keys.ENTER)
-        tc.wait_for('invisibility_of_element', id='ui-datepicker-div')
+        tc.formvalue('edit', 'time', '')
+        tc.send_keys(tc.keys.ESCAPE)  # close datepicker
         tc.submit('save')
         version_admin = self._tester.url + "/admin/ticket/versions"
         tc.url(version_admin, regexp=False)

Another idea is to use @tc.javascript_disabled to disable datepicker:

diff --git a/trac/ticket/tests/functional/admin.py b/trac/ticket/tests/functional/admin.py
index d78869996..6dda7a771 100644
--- a/trac/ticket/tests/functional/admin.py
+++ b/trac/ticket/tests/functional/admin.py
@@ -869,14 +869,13 @@ class TestAdminVersionDetail(FunctionalTwillTestCaseSetup):


 class TestAdminVersionDetailTime(FunctionalTwillTestCaseSetup):
+    @tc.javascript_disabled
     def runTest(self):
         """Admin version detail set time"""
         name = self._tester.create_version()
         tc.follow(name)

-        # Clear value and send ENTER to close the datepicker.
-        tc.formvalue('edit', 'time', tc.keys.ENTER)
-        tc.wait_for('invisibility_of_element', id='ui-datepicker-div')
+        tc.formvalue('edit', 'time', '')
         tc.submit('save')
         version_admin = self._tester.url + "/admin/ticket/versions"
         tc.url(version_admin, regexp=False)

[Ryan J Ollos]

I still get the same failure with the first comment:31 patch. However, sending ESCAPE to close timepicker seems like an improvement and it works in combination with the wait_for.

-        tc.formvalue('edit', 'time', tc.keys.ENTER)
+        tc.formvalue('edit', 'time', '')
+        tc.send_keys(tc.keys.ESCAPE)  # close datepicker
         tc.wait_for('invisibility_of_element', id='ui-datepicker-div')

I also like the addition of send_keys. That will be useful.

No test failures using @tc.javascript_disabled.

The problem with wait_for as I've written it, some ​expected conditions take an element argument, others take a tuple locator. It's possible it could be generalized , but it might be better to just add wait_for_invisibility_of_element and then add others as needed.

Also, I'll try adding headless=false from command line.

[Ryan J Ollos]

Replying to Jun Omae:

• restore html validation (html validator library is needed)

We could use ​pytidylib to validate html with Python 2.7 and 3.5 through 3.8. (jomae.git@1.5-py3-tidylib [e575f67c4/jomae.git]). However, latest of the module has been released at Nov 16, 2016 (4 years ago).

Thoughts?

I've only skimmed the documentation, but another possible candidate is ​pypi:uTidyLib.

[Ryan J Ollos]

choco has html-tidy:

> choco info html-tidy
Chocolatey v0.10.15
html-tidy 5.6.0 [Approved]
Title: HTML Tidy with HTML5 support | Published: 1/21/2018
...
Software Site: http://www.html-tidy.org/
...
Software Source: https://github.com/htacg/tidy-html5
...

Which is the same version as brew:

 brew info tidy-html5
tidy-html5: stable 5.6.0 (bottled), HEAD
Granddaddy of HTML tools, with support for modern standards
https://www.html-tidy.org/
/usr/local/Cellar/tidy-html5/5.6.0 (14 files, 2.6MB) *
  Poured from bottle on 2020-04-26 at 12:16:31
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/tidy-html5.rb
==> Dependencies
Build: cmake ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 21,247 (30 days), 67,798 (90 days), 270,761 (365 days)
install-on-request: 721 (30 days), 1,900 (90 days), 5,973 (365 days)
build-error: 0 (30 days)

Or we can install for Conda: ​tidy-html5.

I'll experiment.

[Jun Omae]

Replying to Ryan J Ollos:

Have you installed ​HTML Tidy on Windows?

I just tried with Python 3.8 and 2.7 on Windows.

pytidylib 0.3.2	tidy-5.6.0-vc14-64b.zip	Python 3.8	fine
pytidylib 0.3.2	tidy-5.6.0-vc10-64b.zip	Python 2.7	fine
uTidyLib 0.6	tidy-5.6.0-vc14-64b.zip	Python 3.8	OSError
uTidyLib 0.6	tidy-5.6.0-vc10-64b.zip	Python 2.7	fine

pytidylib 0.3.2 on Python 3.8

C:\>C:\usr\tmp\tidy-py38\Scripts\python.exe -c "from tidylib import tidy_document as t; print(t('<HTML')[1])"
line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 1 column 6 - Warning: inserting missing 'title' element

pytidylib 0.3.2 on Python 2.7

C:\>C:\usr\tmp\tidy-py27\Scripts\python.exe -c "from tidylib import tidy_document as t; print(t('<HTML')[1])"
line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 1 column 6 - Warning: inserting missing 'title' element

uTidyLib 0.6 on Python 3.8

C:\>C:\usr\tmp\tidy-py38\Scripts\python.exe -c "from tidy import parseString as p; print('\n'.join(map(str, p('<HTML').errors)))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\usr\tmp\tidy-py38\lib\site-packages\tidy\__init__.py", line 42, in <module>
    from tidy.lib import Document, ReportItem, parse, parseString
  File "C:\usr\tmp\tidy-py38\lib\site-packages\tidy\lib.py", line 67, in <module>
    _tidy = Loader()
  File "C:\usr\tmp\tidy-py38\lib\site-packages\tidy\lib.py", line 57, in __init__
    raise OSError("Couldn't find libtidy, please make sure it is installed.")
OSError: Couldn't find libtidy, please make sure it is installed.

According to ​https://docs.python.org/3/whatsnew/3.8.html#ctypes, the behavior has been changed since Python 3.8 on Windows. Work around is to call ​os.add_dll_directory() before import tidy or copy tidy.dll file to scripts directory of using virtualenv.

uTidyLib 0.6 on Python 2.7

C:\>C:\usr\tmp\tidy-py27\Scripts\python.exe -c "from tidy import parseString as p; print('\n'.join(map(str, p('<HTML').errors)))"
line 1 col 1 - Warning: missing <!DOCTYPE> declaration
line 1 col 6 - Warning: inserting missing 'title' element

[Ryan J Ollos]

Thanks for investigating. Seems pytidylib may be the way to go. I'll post work I've done modifying jinjachecker to use pytidylib soon, after posting proposed change for #13307.

[anonymous]

Replying to Ryan J Ollos:

<snip>

Both failures appear to be due to the Clear default submit not being executed. For example, in TestAdminVersionDefault:

 982         print(self._testenv.get_config('ticket', 'default_version'))
 983         tc.submit('clear', formname='version_table')

Maybe this is a timing issue with the testing framework. Perhaps a sleep() would trigger success? Or a transaction commit? A round trip to the test database is possibly required.

>  984         print(self._testenv.get_config('ticket', 'default_version'))
> 

⇒

........................................................DefaultVersion

DefaultVersion

EE

In the third line of output, an empty string rather than DefaultVersion should be printed.

[Ryan J Ollos]

Changes committed in r17483.

[Ryan J Ollos]

Replying to Ryan J Ollos:

Thanks for investigating. Seems pytidylib may be the way to go. I'll post work I've done modifying jinjachecker to use pytidylib soon, after posting proposed change for #13307.

⇒#13332.