Edgewall Software
Modify

Opened 3 years ago

Closed 18 months ago

Last modified 6 months ago

#12299 closed enhancement (fixed)

Integrate functionality of TracMigratePlugin

Reported by: Ryan J Ollos Owned by: Ryan J Ollos
Priority: normal Milestone: 1.3.2
Component: database backend Version:
Severity: normal Keywords:
Cc:
Release Notes:

Added TracAdmin convert_db command, which migrates a database to the same or new environment while changing the database backend.

API Changes:

Description

Database is locked is one of the most frequent duplicates. For migrating databases we suggest using TracMigratePlugin. The plugin is stable and I've used it for several migrations. We could move the code into tracopt. I'm seeking feedback on that idea. Jun, are you okay with the code being moved into the Trac core?

We might consider naming the module convert.py rather than migrate.py. The TracMigrate page documents a slightly different scenario.

Attachments (0)

Change History (23)

comment:1 Changed 3 years ago by Ryan J Ollos

I did some initial work in log:rjollos.git:t12299_trac_migrate. More work would be needed to move ahead.

comment:2 in reply to:  description ; Changed 3 years ago by Jun Omae

Jun, are you okay with the code being moved into the Trac core?

Yeah. No problem.

We could move the code into tracopt. …. We might consider naming the module convert.py rather than migrate.py.

It would be good to move the code into trac.admin.convert or trac.db.convert. I think it would be useful to be able to use without changes of [components] section.

comment:3 in reply to:  2 ; Changed 3 years ago by Ryan J Ollos

Replying to jomae:

It would be good to move the code into trac.admin.convert or trac.db.convert. I think it would be useful to be able to use without changes of [components] section.

Those changes sound good.

I haven't been able to get the TracMigratePlugin tests to pass with PostgreSQL. After the following change they pass:

  • tracmigrate/tests/admin.py

     
    6767        att.insert('filename.txt', StringIO('test'), 4)
    6868        env.shutdown()
    6969
    70     if 'destroying' in inspect.getargspec(EnvironmentStub.__init__)[0]:
    71         def _destroy_db(self):
    72             EnvironmentStub(destroying=True)
    73     else:
    74         def _destroy_db(self):
    75             EnvironmentStub().destroy_db()
     70    # if 'destroying' in inspect.getargspec(EnvironmentStub.__init__)[0]:
     71    #     def _destroy_db(self):
     72    #         EnvironmentStub(destroying=True)
     73    # else:
     74    def _destroy_db(self):
     75        EnvironmentStub(destroying=True).destroy_db()
    7676
    7777    def _get_all_records(self, env):
    7878        def primary(row, columns):

comment:4 in reply to:  3 Changed 3 years ago by Jun Omae

Replying to Ryan J Ollos:

I haven't been able to get the TracMigratePlugin tests to pass with PostgreSQL. After the following change they pass: […]

Thanks for the investigating! Fixed in th:r15147. I didn't probably test with Trac 1.0+ and PostgreSQL.

comment:5 Changed 3 years ago by Ryan J Ollos

Milestone: next-major-releases1.2
Owner: set to Ryan J Ollos
Status: newassigned
Summary: Add TracMigratePlugin to tracoptIntegrate functionality of TracMigratePlugin

comment:6 Changed 3 years ago by Ryan J Ollos

There are some changes we should incorporate in th:#12696, th:#12697 and th:#12691.

comment:7 Changed 3 years ago by Ryan J Ollos

DONE Document obsolescence of TracMigratePlugin at 1.3/TracUpgrade.

Last edited 18 months ago by Ryan J Ollos (previous) (diff)

comment:8 Changed 3 years ago by Ryan J Ollos

For inplace migration, TracMigrate creates a temporary environment and for the SQLite case it copies the database file back to the migrated environment after the migration is finished. Would it work just as well if the source and destination environments were the same for an inplace migration?

comment:9 Changed 3 years ago by Ryan J Ollos

I'm looking at moving the database conversion methods to trac.db.convert.DatabaseConverter, and putting the migrate command in trac.env.EnvironmentAdmin. That would result in two trac-admin commands:

  • convertdb <dburi>: convert database (inplace)
  • migrate <dstenv> [dburi]: migrate environment and optionally convert the database

comment:10 Changed 2 years ago by Emmanuel Blot

I recently (and repeatedly) bumped into an issue with TracMigratePlugin and inherited trac.ini files:

Missing configuration file appkit/conf/trac.ini
Missing configuration file appkit/conf/site.ini
Missing configuration file appkit/conf/site.ini

I'm not sure what's going on here. Moreover I'm not sure yet if it tied but as the plugins are enabled in one of the inherited files, the DB tables for those plugins are not exported.

comment:11 in reply to:  10 Changed 2 years ago by Jun Omae

Replying to Emmanuel Blot:

I recently (and repeatedly) bumped into an issue with TracMigratePlugin and inherited trac.ini files:

[…]

I'm not sure what's going on here. Moreover I'm not sure yet if it tied but as the plugins are enabled in one of the inherited files, the DB tables for those plugins are not exported.

That may be caused by non-absolute path in [inherit] file option. Could you please create new ticket with the details on trac-hacks.org?

comment:12 Changed 2 years ago by Ryan J Ollos

Milestone: 1.21.3.1

comment:13 Changed 2 years ago by Christian Boos

Milestone: 1.3.1next-dev-1.3.x

Moving to some later 1.3.x version.

comment:14 Changed 20 months ago by Ryan J Ollos

Milestone: next-dev-1.3.x1.3.2
Release Notes: modified (diff)

comment:15 Changed 20 months ago by Ryan J Ollos

After additional refactoring, proposed changes can be found in log:rjollos.git:t12299_trac_migrate.1.

There was a test failure, fixed by [b78163a2/rjollos.git]. That may also need to be fixed in TracMigratePlugin.

comment:16 Changed 20 months ago by Peter Suter

Sounds great!

  • Was tracopt/db/migrate.py supposed to be deleted as well in changeset:15d7c62a/rjollos.git?
  • I have never used the the plugin. From the command name migrate (and its command help text) it was not immediately clear to me what its purpose is.
    • (For some reason I associate the term "migration" more with "DB schema upgrade".)
    • Is it mainly intended for migrating from one database backend (e.g. SQLite) to another (e.g. PostgreSQL)? (As described on wiki:DatabaseBackend#DatabaseConversion.)
    • Or also for migrating from older to newer database backend versions (e.g. from Pg8.0 to Pg9.6)?
    • Or also for migrating from one machine to another (e.g. when buying new server hardware)?
    • Should it also be used just to move the environment from one directory to another? (Moving a SQLite database with it, but leaving a PostgreSQL or MySQL database unaffected.)
    • The migrate command help text might need a grammar and clarity check.
    • If applicable maybe mention the word "database backend"?
    • If applicable maybe point out that the desired backend is selected / deduced from the db connection uri?
    • But maybe first decide about the final command names (comment:9)? convertdb sounds like an improvement to me.

comment:17 in reply to:  16 Changed 20 months ago by Ryan J Ollos

Replying to Peter Suter:

Sounds great!

Yes, the code is contained in trac/env.py and trac/db/convert.py. I also removed the trac-migrate.py script because I didn't see any value over the TracAdmin command.

The command name convertdb sounds good to me as well, but I welcome additional suggestions.

comment:18 in reply to:  16 Changed 19 months ago by Ryan J Ollos

Replying to Peter Suter:

  • (For some reason I associate the term "migration" more with "DB schema upgrade".)

There's good basis for associating migration with schema upgrade, for example Django migrations. From grepping the codebase I see we even use the term migration in some upgrade scripts and our database API, where the term is associated with database schema changes.

comment:19 Changed 19 months ago by Ryan J Ollos

Changed to command name convert_db in [7912475e5/rjollos.git].

I'm considering changing signature from <tracenv|-i|--in-place> <dburi> to <dburi> [newenv]. The command forms would then be:

  • trac-admin $env convert_db <dburi>
  • trac-admin $env convert_db <dburi> </path/to/newenv>

On the one hand it's good to be explicit about wanting to do a an in-place conversion, which the -i/--in-place switch provides. However, since the focus of the command is now correctly on the "conversion", the latter forms feel more natural to me.

comment:20 in reply to:  19 Changed 18 months ago by Ryan J Ollos

Replying to Ryan J Ollos:

I'm considering changing signature from <tracenv|-i|--in-place> <dburi> to <dburi> [newenv].

I've made those changes in log:rjollos.git:t12299_trac_migrate.3. Tests are passing, but I haven't done manual testing since making those changes. In absence of additional feedback, I'll do additional testing and then commit later in the week.

comment:21 Changed 18 months ago by Ryan J Ollos

Release Notes: modified (diff)
Resolution: fixed
Status: assignedclosed

Committed to trunk in r16024.

comment:22 Changed 16 months ago by Ryan J Ollos

Documented in 1.3/TracUpgrade@22.

comment:23 Changed 6 months ago by Ryan J Ollos

Python3 compatible revision of r16024 in r16611.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Ryan J Ollos.
The resolution will be deleted.
to The owner will be changed from Ryan J Ollos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.