trac-admin hotcopy should (be able to) copy contents of symbolic links

[Thijs Triemstra]

I have some symlinks in the my trac project folder and when I run trac-admin myproject hotcopy it copies the symlink. Is it possible to make a copy of the folder that includes the content of those symlinks? That way I don't need to worry about those when I make a backup.. The htdocs and templates folders in my environment are symlinks as well (to checkouts of svn repositories) that contain the templates and config files for my trac project, so they're essential to be part of the hotcopy. If you don't want to do it by default for some reason, is it possible to add an option that makes this possible?

The test below shows the output of hotcopy contains a symbolic link to folder -> ../folder.

thijs@home:~/Sites$ mkdir test
thijs@home:~/Sites$ cd test/
thijs@home:~/Sites/test$ ls -l
thijs@home:~/Sites/test$ trac-admin myproject initenv
Creating a new Trac environment at /Users/thijs/Sites/test/myproject

Trac will first ask a few questions about your environment 
in order to initialize and prepare the project database.

[snip]

thijs@home:~/Sites/test$ ls -l
total 0
drwxr-xr-x  11 thijs  staff  374 10 Jan 14:31 myproject
thijs@home:~/Sites/test$ mkdir folder
thijs@home:~/Sites/test$ cd myproject/
thijs@home:~/Sites/test/myproject$ ln -s ../folder
thijs@home:~/Sites/test/myproject$ cd ..
thijs@home:~/Sites/test$ trac-admin myproject hotcopy mycopy
Hotcopying /Users/thijs/Sites/test/myproject to mycopy ...
Hotcopy done.
thijs@home:~/Sites/test$ ls -l mycopy
total 24
-rw-r--r--  1 thijs  staff   98 10 Jan 14:31 README
-rw-r--r--  1 thijs  staff   27 10 Jan 14:31 VERSION
drwxr-xr-x  2 thijs  staff   68 10 Jan 14:31 attachments
drwxr-xr-x  4 thijs  staff  136 10 Jan 14:31 conf
drwxr-xr-x  3 thijs  staff  102 10 Jan 14:35 db
lrwxr-xr-x  1 thijs  staff    9 10 Jan 14:35 folder -> ../folder
drwxr-xr-x  2 thijs  staff   68 10 Jan 14:31 htdocs
drwxr-xr-x  2 thijs  staff   68 10 Jan 14:31 log
drwxr-xr-x  2 thijs  staff   68 10 Jan 14:31 plugins
drwxr-xr-x  3 thijs  staff  102 10 Jan 14:31 templates

[Remy Blank]

I'm -1 on making this the default, as there are certainly people who don't want their symlinks followed. And -0.5 for an option, as I don't think this is the right approach to backing up environments.

In your case, you already have your htdocs and templates in a repository, so those are essentially taken care of already. I assume you have several environments, otherwise you wouldn't use symlinks in the first place. Following the symlinks during hotcopy therefore duplicates the symlinked parts in every backup.

hotcopy is really a kind of "poor man's backup" (except for the fact that it locks the database during the copy, therefore ensuring the consistency of the backup when using SQLite), so if you have more advanced needs, you should probably use more advanced tools as well.

One thing could be interesting, though. We could enhance hotcopy to allow calling an external program instead of the current copy operation. That way, you would still have the database consistency, while being able to copy the filesystem in any way you like.

How does that sound?

[Thijs Triemstra <lists@…>]

Replying to rblank:

hotcopy is really a kind of "poor man's backup" (except for the fact that it locks the database during the copy, therefore ensuring the consistency of the backup when using SQLite), so if you have more advanced needs, you should probably use more advanced tools as well.

Fair enough.

One thing could be interesting, though. We could enhance hotcopy to allow calling an external program instead of the current copy operation. That way, you would still have the database consistency, while being able to copy the filesystem in any way you like.

How does that sound?

Could you show an example of how that would work on the command line?

I was also wondering if it's possible to do a mysql dump when using hotcopy, since the user/pass for the mysql db are already known by trac? I threw together a quick bash script that will run as a cronjob and if Trac would do the mysql/pg export for me, then it would eliminate the need for defining the type of trac db in my bash script:

#!/bin/bash
# Backup script for Trac projects, includes database and files.

# general settings
TITLE=$1
TMP_DIR=/tmp/trac-tmp
TRAC_ROOT=/var/trac
BACKUP_TARBALL_FOLDER=/var/trac/backup
DATE=`date +%Y%m%d`

# mysql settings
DBUSER=$2
DBPASS=$3
MYSQL_OPTS="--log-error=db-errors.log --opt"

function setup {
     echo "Running Trac projects backup for '" $TITLE "'"

     # prepare folders
     echo "Cleaning build folder: " $TMP_DIR
     rm -rf $TMP_DIR
     mkdir -v $TMP_DIR
     cd $TMP_DIR

     mkdir -v $1
     mkdir -v $2
     echo
}

function ready {
     echo
     echo "Finished Trac projects backup."
     exit
}

function backupMySQLProject {
    echo "Archiving MySQL project: " $1
    echo "----------------------------------------"
    echo
    echo "Exporting database: " $2
    cd $TMP_DIR
    mysqldump -u $DBUSER -p$DBPASS $2 > backup.sql
    echo
    echo "Compressing database..."
    gzip --verbose backup.sql
    mv -v backup.sql.gz mysql/$2-backup-$DATE.sql.gz
    echo 
    echo "Exporting Trac files..."
    trac-admin $TRAC_ROOT/$1 hotcopy files/$1
    echo
    echo "============================================================"
    echo
}
     
function backupSQLiteProject {
    echo "Archiving SQLite project: " $1
    echo "----------------------------------------"
    echo
    echo "Exporting Trac files..."
    trac-admin $TRAC_ROOT/$1 hotcopy files/$1
    echo
    echo "============================================================"
    echo
}
         
# start backup
setup mysql files

backupMySQLProject project1 project_mysql_db_name
backupSQLiteProject project2
        
# compress and move files to backup
echo "Compressing files: " trac-$TITLE-backup-${DATE}.tar.gz
cd $TMP_DIR
tar zcf ${BACKUP_TARBALL_FOLDER}/trac-$TITLE-backup-${DATE}.tar.gz *
ls -lha ${BACKUP_TARBALL_FOLDER}/trac-$TITLE-backup-${DATE}.tar.gz

ready

[Remy Blank]

Replying to Thijs Triemstra <lists@…>:

Could you show an example of how that would work on the command line?

I was thinking of a TracIni option like [db] hotcopy_command where you could specify a command to be called when doing a hotcopy. If it is unset, the current behavior is executed (the recursive copy). If it is set, the command is called with two arguments, the path to the environment and the destination.

So you call trac-admin $ENV hotcopy $DEST, the database is locked, and your command is called. You could, for example, set it to "cp -RL" for dereferencing symlinks.

I was also wondering if it's possible to do a mysql dump when using hotcopy, since the user/pass for the mysql db are already known by trac?

We already have all the necessary functionality, as we do a backup during upgrades. So yes, that could be a good idea. We could store the dump in the db directory before the copy operation.

[Thijs Triemstra <lists@…>]

Replying to rblank:

I was thinking of a TracIni option like [db] hotcopy_command where you could specify a command to be called when doing a hotcopy. If it is unset, the current behavior is executed (the recursive copy). If it is set, the command is called with two arguments, the path to the environment and the destination.

So you call trac-admin $ENV hotcopy $DEST, the database is locked, and your command is called. You could, for example, set it to "cp -RL" for dereferencing symlinks.

I guess it would also make it possible to do something like a hotcopy of a svn diff? :) What would the default value of hotcopy_command be?

[db]
hotcopy_command=svn diff

$ trac-admin $ENV hotcopy $DEST

If such a _command in trac.ini is the easiest/best place to specify such a command then this sounds like a great idea :)

I was also wondering if it's possible to do a mysql dump when using hotcopy, since the user/pass for the mysql db are already known by trac?

We already have all the necessary functionality, as we do a backup during upgrades. So yes, that could be a good idea. We could store the dump in the db directory before the copy operation.

I've opened #8954 for this feature.

Thanks.

[Remy Blank]

Replying to Thijs Triemstra <lists@…>:

What would the default value of hotcopy_command be?

An empty value, meaning the current behavior: a recursive copy.

Let's keep this on the radar.

[Remy Blank]

Mmh, my comment edit in comment:4 made the whole comment show up as coming from me. This has been fixed in [9036], and the comment author will appear correct again when t.e.o is updated. Sorry about that.

[Christian Boos]

(this is bordering unscheduled to me…)