Ability to run functional tests on non-sqlite

[Tim Hatch]

We need to be able to somehow configure the functional tests to run against other db backends. This got brought up last night at the Trac BoF at PyCon again, so I'm going to track down the patch I have and we should look into getting it integrated.

[shanec@…]

Right now I use a shell script to setup the postgres database in my dev environment. I can modify that to do a setup/teardown for testing. It could also be moved into python or separated out and left as a shell script.

There's also ticket:2304 I submitted some time ago that provides a backup interface, we'll chat about that since it might be useful for testing as well. I have a couple different postgres backup implementations that I'd be happy to make into a plugin or otherwise provide.

[Christian Boos]

I did some further work on the branch, so that the functional tests with alternate db backends would work on Windows.

One thing I'm definitely not happy with at this point (r8135) is the FunctionalTestEnvironment.destroy_mysql method and its use of grep…

[Christian Boos]

One thing I'm definitely not happy with at this point (r8135) is the FunctionalTestEnvironment.destroy_mysql method and its use of grep…

This should be fixed in r8155, however I'd like to get more feedback on this before merging on trunk (does it also work on Linux, things I may have missed compared to the previous solution, etc.)

[Remy Blank]

I'm currently installing PostgreSQL, and I'll try to test on Linux and OS X.

[Remy Blank]

All tests pass on OS X and Linux (actually, I get a single error on the latter, in the backup test, but this is due to the PostgreSQL server running on another machine, so pg_dump cannot be found locally).

I'm trying to set up a build slave running the tests with PostgreSQL. Once that is done, I'll do the same for MySQL.

[Remy Blank]

I have set up the build slave for PostgreSQL, and created build configurations for 0.11-stable, trunk and rework-testing. However, there's an issue when emptying the database before creating the new environment. In testenv.py, the database is emptied in destroy() before the new environment is created, but only if a previous environment exists. This is not true when running the build slave, as the environment is created in a new checkout for every run. So the tests pass the first time, but subsequent runs fail to create the environment.

Fixing this is a bit tricky, as it involves emptying the database regardless of the existence of an environment, but then there's no environment to get a connection from. The easiest is probably to use a dummy environment just for cleaning the database.

[Remy Blank]

I have also created a build configuration for MySQL, currently only for the rework-testing branch. All tests pass, but the same issue with the database cleanup occurs.

[Christian Boos]

Is it possible to configure the build slaves so that they run in the same location, perhaps by doing an update instead of a checkout?

Unless you're already working on it, I'm thinking of moving the database cleanup code to trac.test and have the EnvironmentStub use it as well.

[Remy Blank]

I am working on it. The idea is not to use the previous environment at all, but to instantiate PostgreSQLConnection or MySQLConnection directly just to clean the database. Something like:

trac/tests/functional/testenv.py

@@ -16 +16 @@
 from trac.tests.functional.better_twill import tc, ConnectError
 from trac.env import open_environment
 from trac.db.api import _parse_db_str, DatabaseManager
+from trac.db.mysql_backend import MySQLConnection
+from trac.db.postgres_backend import PostgreSQLConnection
 from trac.util.compat import close_fds
 
 # TODO: refactor to support testing multiple frontends, backends (and maybe
@@ -98 +100 @@
 
     def destroy(self):
         """Remove all of the test environment data."""
-        if os.path.exists(self.dirname):
-            env = self.get_trac_environment()
-            dburi = DatabaseManager(env).connection_uri
-            scheme, db_prop = _parse_db_str(self.dburi)
-            db = env.get_db_cnx()
-            if scheme == 'postgres':
-                self.destroy_postgresql(db)
-            elif scheme == 'mysql':
-                self.destroy_mysqldb(db, db_prop)
-            env.shutdown()
+        scheme, db_prop = _parse_db_str(self.dburi)
+        if scheme == 'postgres':
+            db = PostgreSQLConnection(**db_prop)
+            self.destroy_postgresql(db)
+        elif scheme == 'mysql':
+            db = MySQLConnection(**db_prop)
+            self.destroy_mysqldb(db, db_prop)
 
         self.destroy_repo()
         if os.path.exists(self.dirname):

First tests seem to indicate that this works.

[Tim Hatch]

remy: I'm +1 on that, but couldn't figure out how to get the list of tables used by Trac (in particular when plugins are being used).

[Remy Blank]

Christian has already implemented a generic solution. With PostgreSQL, the whole schema is dropped. With MySQL, the table names are retrieved from the information_schema.tables table.

[Remy Blank]

Ok, the approach works well with both PostgreSQL and MySQL. Patch committed in [8156].

[Christian Boos]

Just a quick update for avoiding duplicate efforts: I've been working on having the unit-test use the TRAC_TEST_DB_URI switch as well for selecting which DB backend to use. I'm nearly there (6 failures left in both mysql and postgres).

[Christian Boos]

With [8191], the unit-tests also use the alternate DB backend specified by TRAC_TEST_DB_URI.

So instead of [8156], the FunctionalTestEnvironment now simply uses EnvironmentStub.get_db_cnx().

[Christian Boos]

Change above ported to trunk in r8194. I noticed 8 failures (build:1579 and build:1580), due to the cache table being missing. I don't exactly get why - also had the error locally once, at the next run everything was fine.

What I'd like to do next on this topic is to add the possibility to specify TRAC_TEST_DB_URI=​sqlite:trac/db/trac.db, i.e. run the tests with a SQLite database on the filesystem, as opposed to the in-memory one. Thanks to reset_db(), this should be now easy.

[Remy Blank]

That's weird. I invalidated build:1579 and it passed the second time. Is it possible that there's a transient error when creating the database? Doesn't seem likely…

[Christian Boos]

The error message was:

Traceback (most recent call last):
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/wiki/tests/formatter.py", line 139, in setUp
    wiki.save('joe', 'Entry page', '::1', datetime.now(utc))
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/wiki/model.py", line 153, in save
    WikiSystem(self.env).pages.invalidate(db)
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/cache.py", line 109, in invalidate
    CacheManager(self.env).invalidate(self.id, db)
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/cache.py", line 203, in invalidate
    "WHERE id=%s", (id,))
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/Users/tracbuilder/src/bitten-2.5-pg/work/build_trunk_1580/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
ProgrammingError: relation "cache" does not exist

but that was after the insert succeeded in the wiki table. So yes, it's a bit strange, how come the database could have the wiki table and not the cache table?

The only explanation for this is that there was already a schema in that test database that didn't contain the cache table. I thought this wouldn't happen in our case, as we had tested the cache before, but situations like that are certainly possible in the general case, e.g. just after a database version upgrade.

So I guess I have to complement reset_db() with a check of the database version before erasing the tables and do a destroy_db()/init_db() if the version doesn't match the expected one.

[Christian Boos]

Replying to cboos:

So I guess I have to complement reset_db() with a check of the database version before erasing the tables and do a destroy_db()/init_db() if the version doesn't match the expected one.

… and this is needed rather sooner than later as the tests currently fail on multirepos (with the ported changes from trunk, not yet committed):

ProgrammingError: relation "repository" does not exist

And we need to be able to test back on an older database version, when we use the same TRAC_TEST_DB_URI and switching back and forth between branches…

[Christian Boos]

So I guess I have to complement reset_db() with a check of the database version before erasing the tables and do a destroy_db()/init_db() if the version doesn't match the expected one.

This was implemented in [8205].

Now something strange happened - I tested the change on multirepos mainly on my version before integrating changes from upstream. Moments later I redid the testing with latest version and then noticed some failures, which I fixed by adding one more call to reset_db (r8208). That last change triggered a build failure in the functional tests for one platform (and only one), see build:1652. Remy, any chance to have a closer look at what happened there, as I can't reproduce this?

[Remy Blank]

Replying to cboos:

That last change triggered a build failure in the functional tests for one platform (and only one), see build:1652. Remy, any chance to have a closer look at what happened there, as I can't reproduce this?

Yes, it's not the first time that this happens. I have invalidated that build, and the second time it succeeded. It may be due to my various build slaves not being completely isolated from each other. Somehow I don't manage to set a different temporary directory for each slave, so they all create their repository in /tmp/trac-svnrepos. The symptom for this is usually a failure to call trac-admin initenv to initialize the test environment, but maybe there are more subtle issues.

I'll revisit the build slaves soon, as I want to create a few scripts to make it easy to create build slaves on all platforms. Hopefully I can find out what's happening here.

[Christian Boos]

There are still a few corner-case issues left - see #8518 for those.

But the bulk of the change was completed for 0.11.5 and we used that feature a lot during the testing of 0.11.5, so I'm closing this ticket against this milestone.

[Christian Boos]

I still have the sqlite file database support to finish, but that will be for later. If anyone's interested in the meantime, just ask for the patch.

[Christian Boos]

Replying to cboos:

I still have the sqlite file database support to finish, but that will be for later.

That's now possible starting with r10180 (make test db=sqlite).