Edgewall Software
Modify

Opened 15 years ago

Closed 14 years ago

Last modified 12 years ago

#1123 closed defect (fixed)

Exception error during hotcopy

Reported by: anonymous Owned by: Christian Boos
Priority: high Milestone: 0.9
Component: admin/console Version: 0.9b2
Severity: critical Keywords: windows hotcopy
Cc: jmberghold@…; rhind@…; johan.appelgren@…; cam@… nunterberg@…, gabriel.blum@…, agarino@… Branch:
Release Notes:
API Changes:

Description

I received an exception error during a hotcopy using trac-admin. I have Trac installed on Windows 2000 Server, Service Pack 4.

The error was as follows: Hotcopying d:\source\folder to d:\target\folder … [('d:
source
folder
db
trac.db-journal', 'd:
target
folder
db
trac.db-journal', <exceptions.IOError instance at 0x008BF3C8>)]

The 'attachments', 'conf' and 'db' folders were copied but it didn't get to the 'log', 'templates' and 'wiki-macros' folders. The 'trac.db' file appears to have been successfully copied across (at least, the size of the source and destination 'trac.db' files are the same).

Attachments (2)

robust_hotcopy.patch (2.7 KB ) - added by Christian Boos 14 years ago.
Implemented mgood's suggestion: on Windows, the trac.db-journal file can't be copied, and the shutil.copytree can't cope with that, therefore we need our own version, slightly adapted to cope with anticipated failures.
robust_hotcopy.2.patch (2.0 KB ) - added by Christian Boos 14 years ago.
Second version. copytree is now directly in admin.py, and the ignore list is now a "skip" list (we don't even attempt to copy the file)

Download all attachments as: .zip

Change History (30)

comment:1 by anonymous, 15 years ago

Cc: jmberghold@… added

comment:2 by rhind@…, 15 years ago

I'm just seeing this also. I can't backup the database at the moment because of it.

comment:3 by anonymous, 15 years ago

Cc: rhind@… added

comment:4 by anonymous, 15 years ago

Cc: johan.appelgren@… added

comment:5 by anonymous, 15 years ago

Cc: cam@… added

comment:6 by rhind@…, 15 years ago

Milestone: 0.9

Changed milestone to 0.9 to try and get some visibility on this as no-one seems to have commented on it for a while, but it is still happening for me

WinXP SP2 trac-0.8.1 python-2.3.4

Thanks

Russell

comment:7 by johan.appelgren@…, 15 years ago

I believe the error is because sqlite has opened trac.db-journal exclusively so that the copy fails. Don't know if the sqlite db will survive being copied with a transaction active and the journal file missing. But if it can survive that, excluding the journal file(es) in the hotcopy should fix this.

comment:8 by rhind@…, 15 years ago

I've stopped apache running on my system while trying to hotcopy, so there shouldn't be any active transactions should there?

comment:9 by johan.appelgren@…, 15 years ago

The hotcopy code in trac-admin creates a transaction.

comment:10 by m.v.kalmthout_1@…, 14 years ago

I had the same problem. Due to small error in TracOnWindows I installed pysqlite-1.0.1.win32-py2.3.exe in stead of pysqlite-1.1.6.win32-py2.3.exe.

After upgrading to pysqlite-1.1.6 this problem was gone.

comment:11 by rhind@…, 14 years ago

Ok, I had installed pysqlite-1.0.1.win32-py2.3.exe I uninstalled it from control panel, then installed pysqlite-1.1.6.win32-py2.3.exe as suggested. I now can't access the trac database at all. I get the following apache errors in my server log

[Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: Traceback (most recent call last): [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: File "C:
Python23
Lib
site-packages
mod_python
apache.py", line 299, in HandlerDispatch\n result = object(req) [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: File "C:
Python23
Lib
site-packages
trac
ModPythonHandler.py", line 186, in handler\n env = get_environment(req, mpr) [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: File "C:
Python23
Lib
site-packages
trac
ModPythonHandler.py", line 176, in get_environment\n env_cache[env_path] = open_environment(env_path, mpr) [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: File "C:
Python23
Lib
site-packages
trac
ModPythonHandler.py", line 137, in open_environment\n version = env.get_version() [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: File "C:
Python23
Lib
site-packages
trac
Environment.py", line 162, in get_version\n cursor.execute("SELECT value FROM system WHERE name='database_version'") [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: File "C:
Python23
Lib
site-packages
sqlite
main.py", line 244, in execute\n self.rs = self.con.db.execute(SQL) [Wed Mar 23 20:28:10 2005] [error] [client 10.0.127.18] PythonHandler trac.ModPythonHandler: DatabaseError: file is encrypted or is not a database

I've uninstalled 1.1.6 and gone back to 1.0.1 and everything works fine. Do I need to upgrade the trac-db somehow? I've tried a trac-admin upgrade but get the same error about the db being encrypted or is not a database.

Any help would be greatly appreciated.

Cheers

Russell

comment:12 by Mark Rowe, 14 years ago

You need the correct version of PySQLite for the version of SQLite you are using. If you are using SQLite 2.x, you need PySQLite 1.0.x. If you are using SQLite 3, you need PySQLite 1.1.x. As I understand it, the SQLite 2.x and 3.x file formats are incompatible.

comment:13 by rhind@…, 14 years ago

I don't believe I manually downloaded sqlite at all when I started, just pysqlite-1.0.1 and trac-0.7.1.exe (then upgraded to trac-0.8). I have now downloaded the sqlite3.exe. Is there a way I can upgrade the trac database so I can use the later pysqlite?

Thanks

Russell

comment:14 by anonymous, 14 years ago

Ok, I've just moved the database on to another server and looked again at upgrading from sqlite2 to sqlite3 (pysqlite-1.1.6). You need to upgrade db/trac.db sqlite database file in you trac directory. The bottom of this page details how to do this. I can now hotcopy trac!

I guess this ticket can be closed as worksforme?

comment:15 by Jonas Borgström, 14 years ago

Resolution: worksforme
Status: newclosed

I can't reproduce this either. Please reopen if this problem still exists.

comment:16 by anonymous, 14 years ago

Cc: cam@… added; cam@… removed
Resolution: worksforme
Severity: majorcritical
Status: closedreopened
Version: 0.80.9b2

I am reopening this ticket because I am seeing this problem on my new installation:

  • Trac 0.9 trunk revision [2288]
  • pysqlite 1.1.6
  • running on a windows 2003 domain server

Is it possible that we need pysqlite 2.x now? The problem might have been introduced in [2263], because you changed something in the transaction logic there (see ticket #2016)

comment:17 by Matthew Good, 14 years ago

Owner: changed from daniel to Matthew Good
Status: reopenednew

No, [2263] just made sure that only the standard DB-API methods were used instead of pysqlite 1.x-specific methods. The hotcopy command works fine with pysqlite 1.1 on Linux, but this appears to be Windows file-locking issue (bah, Windows shouldn't tell me I can't copy a file).

I guess I can do some testing of hotcopy with different pysqlite versions on Windows, but I don't think we really need the journal file anyways. If that's the only thing that's interfering with the copying on Windows it would be possible to create a modified version of the shutil.copytree method that would accept a list of files to ignore.

comment:18 by Alesque, 14 years ago

I upgraded trac from 0.8x to 0.9b2 on win2k3 today. The hotcopy command doesn't work anymore.

Hotcopying C:\Projects\Trac\XXXX to help ...Command failed: [('C:\\Projects\\Tra
c\\XXXX\\db\\trac.db-journal', 'help\\db\\trac.db-journal', <exceptions.IOError
instance at 0x00B38B20>)]

comment:19 by gabriel.blum@…, 14 years ago

Cc: gabriel.blum@… added

Does't work here either. Win2k3 server domain controller + 0.9b2 + pysqlite 1.1.6; I agree this looks like a file locking issue

A simple copy of the env dir would work as a backup?

comment:20 by Christian Boos, 14 years ago

Please try once again with recent trunk (r2355 might have solved your locking issue).

comment:21 by Alesque, 14 years ago

Cc: agarino@… added

I try a hot copy with r2357 and the same error occurs:

  • Windows 2003 domain server
  • pysqlite 2.0.4

comment:22 by Jonas Borgström, 14 years ago

Keywords: win32 added; trac.db-journal error removed

comment:23 by Christian Boos, 14 years ago

Owner: changed from Matthew Good to Christian Boos
Priority: normalhigh

I can reproduce this. I'll look into it.

by Christian Boos, 14 years ago

Attachment: robust_hotcopy.patch added

Implemented mgood's suggestion: on Windows, the trac.db-journal file can't be copied, and the shutil.copytree can't cope with that, therefore we need our own version, slightly adapted to cope with anticipated failures.

comment:24 by Christian Boos, 14 years ago

Status: newassigned

There's now a patch to test: attachment:robust_hotcopy.patch

For me at least, it appeared that the only problem was that the trac-db.journal can't be copied due to the way SQLite uses it to perform to lock of the database. So the copy will always fail for this file, we just have to cope with it. Also, it will not be a problem to not copy the journal file, as it was the hotcopy method itself which triggered the (dummy) transaction.

Therefore, I followed mgood's suggestion of skipping the copy of that file and added a variant of shutil.copytree to trac.util.

by Christian Boos, 14 years ago

Attachment: robust_hotcopy.2.patch added

Second version. copytree is now directly in admin.py, and the ignore list is now a "skip" list (we don't even attempt to copy the file)

comment:25 by Christopher Lenz, 14 years ago

Second patch looks good.

comment:26 by Christian Boos, 14 years ago

Resolution: fixed
Status: assignedclosed

Second patch applied in r2410.

comment:27 by sid, 13 years ago

Keywords: windows added; win32 removed

comment:28 by goran.wallgren@…, 12 years ago

This problem has reappeared in 0.11b1 on Windows with pysqlite-2.4.0, see new ticket #6627.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Christian Boos.
The resolution will be deleted. Next status will be 'reopened'.
to as closed The owner will be changed from Christian Boos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.