Edgewall Software
Modify

Opened 10 years ago

Closed 10 years ago

Last modified 9 years ago

#8169 closed enhancement (fixed)

Ability to run functional tests on non-sqlite

Reported by: Tim Hatch Owned by: Tim Hatch
Priority: normal Milestone: 0.11.5
Component: general Version: none
Severity: normal Keywords: pycon-sprint testing
Cc: shanec@… Branch:
Release Notes:
API Changes:

Description

We need to be able to somehow configure the functional tests to run against other db backends. This got brought up last night at the Trac BoF at PyCon again, so I'm going to track down the patch I have and we should look into getting it integrated.

Attachments (0)

Change History (23)

comment:1 by shanec@…, 10 years ago

Right now I use a shell script to setup the postgres database in my dev environment. I can modify that to do a setup/teardown for testing. It could also be moved into python or separated out and left as a shell script.

There's also ticket:2304 I submitted some time ago that provides a backup interface, we'll chat about that since it might be useful for testing as well. I have a couple different postgres backup implementations that I'd be happy to make into a plugin or otherwise provide.

comment:2 by Christian Boos, 10 years ago

I did some further work on the branch, so that the functional tests with alternate db backends would work on Windows.

One thing I'm definitely not happy with at this point (r8135) is the FunctionalTestEnvironment.destroy_mysql method and its use of grep

in reply to:  2 comment:3 by Christian Boos, 10 years ago

One thing I'm definitely not happy with at this point (r8135) is the FunctionalTestEnvironment.destroy_mysql method and its use of grep

This should be fixed in r8155, however I'd like to get more feedback on this before merging on trunk (does it also work on Linux, things I may have missed compared to the previous solution, etc.)

comment:4 by Remy Blank, 10 years ago

I'm currently installing PostgreSQL, and I'll try to test on Linux and OS X.

comment:5 by Remy Blank, 10 years ago

All tests pass on OS X and Linux (actually, I get a single error on the latter, in the backup test, but this is due to the PostgreSQL server running on another machine, so pg_dump cannot be found locally).

I'm trying to set up a build slave running the tests with PostgreSQL. Once that is done, I'll do the same for MySQL.

comment:6 by Remy Blank, 10 years ago

I have set up the build slave for PostgreSQL, and created build configurations for 0.11-stable, trunk and rework-testing. However, there's an issue when emptying the database before creating the new environment. In testenv.py, the database is emptied in destroy() before the new environment is created, but only if a previous environment exists. This is not true when running the build slave, as the environment is created in a new checkout for every run. So the tests pass the first time, but subsequent runs fail to create the environment.

Fixing this is a bit tricky, as it involves emptying the database regardless of the existence of an environment, but then there's no environment to get a connection from. The easiest is probably to use a dummy environment just for cleaning the database.

comment:7 by Remy Blank, 10 years ago

I have also created a build configuration for MySQL, currently only for the rework-testing branch. All tests pass, but the same issue with the database cleanup occurs.

comment:8 by Christian Boos, 10 years ago

Is it possible to configure the build slaves so that they run in the same location, perhaps by doing an update instead of a checkout?

Unless you're already working on it, I'm thinking of moving the database cleanup code to trac.test and have the EnvironmentStub use it as well.

comment:9 by Remy Blank, 10 years ago

I am working on it. The idea is not to use the previous environment at all, but to instantiate PostgreSQLConnection or MySQLConnection directly just to clean the database. Something like:

  • trac/tests/functional/testenv.py

    diff --git a/trac/tests/functional/testenv.py b/trac/tests/functional/testenv.py
    a b  
    1616from trac.tests.functional.better_twill import tc, ConnectError
    1717from trac.env import open_environment
    1818from trac.db.api import _parse_db_str, DatabaseManager
     19from trac.db.mysql_backend import MySQLConnection
     20from trac.db.postgres_backend import PostgreSQLConnection
    1921from trac.util.compat import close_fds
    2022
    2123# TODO: refactor to support testing multiple frontends, backends (and maybe
     
    98100
    99101    def destroy(self):
    100102        """Remove all of the test environment data."""
    101         if os.path.exists(self.dirname):
    102             env = self.get_trac_environment()
    103             dburi = DatabaseManager(env).connection_uri
    104             scheme, db_prop = _parse_db_str(self.dburi)
    105             db = env.get_db_cnx()
    106             if scheme == 'postgres':
    107                 self.destroy_postgresql(db)
    108             elif scheme == 'mysql':
    109                 self.destroy_mysqldb(db, db_prop)
    110             env.shutdown()
     103        scheme, db_prop = _parse_db_str(self.dburi)
     104        if scheme == 'postgres':
     105            db = PostgreSQLConnection(**db_prop)
     106            self.destroy_postgresql(db)
     107        elif scheme == 'mysql':
     108            db = MySQLConnection(**db_prop)
     109            self.destroy_mysqldb(db, db_prop)
    111110
    112111        self.destroy_repo()
    113112        if os.path.exists(self.dirname):

First tests seem to indicate that this works.

comment:10 by Tim Hatch, 10 years ago

remy: I'm +1 on that, but couldn't figure out how to get the list of tables used by Trac (in particular when plugins are being used).

comment:11 by Remy Blank, 10 years ago

Christian has already implemented a generic solution. With PostgreSQL, the whole schema is dropped. With MySQL, the table names are retrieved from the information_schema.tables table.

comment:12 by Remy Blank, 10 years ago

Ok, the approach works well with both PostgreSQL and MySQL. Patch committed in [8156].

comment:13 by Christian Boos, 10 years ago

Just a quick update for avoiding duplicate efforts: I've been working on having the unit-test use the TRAC_TEST_DB_URI switch as well for selecting which DB backend to use. I'm nearly there (6 failures left in both mysql and postgres).

comment:14 by Christian Boos, 10 years ago

With [8191], the unit-tests also use the alternate DB backend specified by TRAC_TEST_DB_URI.

So instead of [8156], the FunctionalTestEnvironment now simply uses EnvironmentStub.get_db_cnx().

comment:15 by Christian Boos, 10 years ago

Change above ported to trunk in r8194. I noticed 8 failures (build:1579 and build:1580), due to the cache table being missing. I don't exactly get why - also had the error locally once, at the next run everything was fine.

What I'd like to do next on this topic is to add the possibility to specify TRAC_TEST_DB_URI=sqlite:trac/db/trac.db, i.e. run the tests with a SQLite database on the filesystem, as opposed to the in-memory one. Thanks to reset_db(), this should be now easy.

comment:16 by Remy Blank, 10 years ago

That's weird. I invalidated build:1579 and it passed the second time. Is it possible that there's a transient error when creating the database? Doesn't seem likely…

comment:17 by Christian Boos, 10 years ago

The error message was:

Traceback (most recent call last):
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/wiki/tests/formatter.py", line 139, in setUp
    wiki.save('joe', 'Entry page', '::1', datetime.now(utc))
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/wiki/model.py", line 153, in save
    WikiSystem(self.env).pages.invalidate(db)
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/cache.py", line 109, in invalidate
    CacheManager(self.env).invalidate(self.id, db)
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/cache.py", line 203, in invalidate
    "WHERE id=%s", (id,))
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
ProgrammingError: relation "cache" does not exist

but that was after the insert succeeded in the wiki table. So yes, it's a bit strange, how come the database could have the wiki table and not the cache table?

The only explanation for this is that there was already a schema in that test database that didn't contain the cache table. I thought this wouldn't happen in our case, as we had tested the cache before, but situations like that are certainly possible in the general case, e.g. just after a database version upgrade.

So I guess I have to complement reset_db() with a check of the database version before erasing the tables and do a destroy_db()/init_db() if the version doesn't match the expected one.

in reply to:  17 ; comment:18 by Christian Boos, 10 years ago

Replying to cboos:

So I guess I have to complement reset_db() with a check of the database version before erasing the tables and do a destroy_db()/init_db() if the version doesn't match the expected one.

… and this is needed rather sooner than later as the tests currently fail on multirepos (with the ported changes from trunk, not yet committed):

ProgrammingError: relation "repository" does not exist

And we need to be able to test back on an older database version, when we use the same TRAC_TEST_DB_URI and switching back and forth between branches…

in reply to:  18 ; comment:19 by Christian Boos, 10 years ago

So I guess I have to complement reset_db() with a check of the database version before erasing the tables and do a destroy_db()/init_db() if the version doesn't match the expected one.

This was implemented in [8205].

Now something strange happened - I tested the change on multirepos mainly on my version before integrating changes from upstream. Moments later I redid the testing with latest version and then noticed some failures, which I fixed by adding one more call to reset_db (r8208). That last change triggered a build failure in the functional tests for one platform (and only one), see build:1652. Remy, any chance to have a closer look at what happened there, as I can't reproduce this?

in reply to:  19 comment:20 by Remy Blank, 10 years ago

Replying to cboos:

That last change triggered a build failure in the functional tests for one platform (and only one), see build:1652. Remy, any chance to have a closer look at what happened there, as I can't reproduce this?

Yes, it's not the first time that this happens. I have invalidated that build, and the second time it succeeded. It may be due to my various build slaves not being completely isolated from each other. Somehow I don't manage to set a different temporary directory for each slave, so they all create their repository in /tmp/trac-svnrepos. The symptom for this is usually a failure to call trac-admin initenv to initialize the test environment, but maybe there are more subtle issues.

I'll revisit the build slaves soon, as I want to create a few scripts to make it easy to create build slaves on all platforms. Hopefully I can find out what's happening here.

comment:21 by Christian Boos, 10 years ago

Milestone: 0.120.11.5
Resolution: fixed
Status: newclosed

There are still a few corner-case issues left - see #8518 for those.

But the bulk of the change was completed for 0.11.5 and we used that feature a lot during the testing of 0.11.5, so I'm closing this ticket against this milestone.

comment:22 by Christian Boos, 10 years ago

Keywords: testing added

I still have the sqlite file database support to finish, but that will be for later. If anyone's interested in the meantime, just ask for the patch.

in reply to:  22 comment:23 by Christian Boos, 9 years ago

Replying to cboos:

I still have the sqlite file database support to finish, but that will be for later.

That's now possible starting with r10180 (make test db=sqlite).

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Tim Hatch.
The resolution will be deleted. Next status will be 'reopened'.
to as closed The owner will be changed from Tim Hatch to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.