| View previous topic :: View next topic |
| Author |
Message |
jsiegers
Could work for Apple
Joined: 21 Oct 2008
Posts: 41
|
 Posted: Fri Nov 14, 2008 8:39 am Post subject: cvcpSync.sh : When rsync is too slow! |
 |
|
Hi Everybody,
As some of you know sometimes rsync is too slow to backup your volume. Especially to another Xsan volume.
I've heard a couple of times of the solution to only use rsync to build a file list and to use cvcp to do the actual file copy. But I never found a script that does this. So I made my own and thought perhaps some of you want to have it as well.
I did a small test with 145 GB of data. Just a normal folder any editor would use. A normal rsync did the job in 1 hour 7 minutes and 9 seconds.
cvcpSync.sh did it in 39 minutes and 29 seconds. So almost 30 minutes faster! I'm curious about your results.
Feel free to use it! And let me know if it's usefull and works.
If you need it as a normal file I'm happy to email it.
Cheers!
Jasper
| Code: |
#!/bin/bash
#############################################################################################
# NAME: cvcpSync.sh - Version 1.0 (14-11-2008) #
# AUTHOR: Jasper Siegers - jasper.siegers@eo.nl #
# DESCRIPTION: We use rsync to build a file list. Then we use cvcp to copy the files. #
#############################################################################################
########################################
# Variables #
########################################
# Paths
# If your source or target is a directory than use a "/" at the end!
# Needless to say that these should be existing directories!
SOURCE="/Volumes/Your Volume Name/Your Folder Name/"
TARGET="/Volumes/Your Backup Volume Name/Your Backup Folder Name"
TEMPFILE="/tmp/cvcpSync.txt"
# Run Rsync
# The problem with cvcp is that it needs an existing destination directory before it can copy files in it.
# So because it won't automatically make that directory this script makes it for cvcp.
# The problem with this is that this folder does not have the same owner, rights, modification date and
# resource forks as the original. So to make a good mirror it's wise to run the default rsync command to fix all of this.
# This rsync run won't backup all the large files anymore, only the folders and resource forks.
# 1 = Yes, 0 = No.
RUNRSYNC=1
# If you want you can write the outcome to a log. 1 = Yes , 0 = NO.
WRITETOLOG=0
# If WRITETOLOG=1 than we need a location for the logfile.
LOGFILE="/Library/Logs/cvcpSync.log"
# Do not edit below this line. Unless you know what you're doing. #
########################################################################
########################################
# PROCEDURES #
########################################
cvcpSync () {
echo "### cvcpSync ###"
echo "`date`"
echo ""
#
# Build the list and export it to our temp file location.
#
echo -n "# Building file list ... "
rsync -av --delete --dry-run "${SOURCE}" "${TARGET}" | awk 'NR>n{print a[NR%n]} {a[NR%n]=$0}' n=3 |sed "1d" > "${TEMPFILE}"
echo "done."
echo ""
#
# Delete the files on the TARGET.
#
echo "# Deleting files:"
cat $TEMPFILE |grep "deleting " | sed 's/deleting\ //g' |while read THEPATH
do
rm -rf "${TARGET}""${THEPATH}"
echo " ""${TARGET}""${THEPATH}"
done
echo ""
#
# Copy the files from the SOURCE to TARGET
#
echo "# Copy files:"
cat "${TEMPFILE}" |grep -v "deleting " |while read THEPATH
do
# Check if THEPATH is ./ We don't want that one.
if [[ $THEPATH != ./ ]] ;
then
# Check if this is a folder. If so we make one at the Target.
# We make a folder because cvcp is not able to make one.
if [[ -d "${SOURCE}""${THEPATH}" ]] ;
then
mkdir -p "${TARGET}""${THEPATH}"
chmod 775 "${TARGET}""${THEPATH}"
echo " Created Folder: ""${TARGET}""${THEPATH}"
# Check if this is a file. If so we copy it.
else if [[ -f "${SOURCE}""${THEPATH}" ]] ;
then
echo " File: ""${TARGET}""${THEPATH}"
echo -n " Copying..."
cvcp -xyz "${SOURCE}""${THEPATH}" "${TARGET}""${THEPATH}"
if [[ $WRITETOLOG == 1 ]] ;
then
echo ""
fi
fi
fi
fi
done
echo ""
# Check if we need to run rsync again for all the folders and resource forks.
if [[ $RUNRSYNC == 1 ]] ;
then
echo "# Running rsync."
rsync -av --delete "${SOURCE}" "${TARGET}"
fi
echo ""
#
# Clean up the tempfile and end.
#
rm -f "${TEMPFILE}"
echo `date`
echo "### End of cvcpSync ###"
echo ""
}
# Let's start the procedure.
if [[ $WRITETOLOG == 1 ]] ;
then
cvcpSync > "${LOGFILE}"
else
cvcpSync
fi
|
|
|
| Back to top |
|
 |
lucasnap
Xsan Master
Joined: 05 Oct 2006
Posts: 107
|
| Posted: Mon Nov 17, 2008 7:49 am Post subject: |
|
|
Nice!!
Thanks you very much!! |
|
| Back to top |
|
|
aaron
Site Admin
Joined: 19 Mar 2005
Posts: 405
|
| Posted: Mon Nov 17, 2008 8:55 am Post subject: |
|
|
This does look like a good script. Can you please post it as a story to our main page?
_________________
Aaron Freimark
http://www.tekserve.com/vcard/af.vcf |
|
| Back to top |
|
|
Ernesto Sanchez
Xsan Master
Joined: 31 Jul 2006
Posts: 74
|
| Posted: Mon Nov 17, 2008 5:17 pm Post subject: Thanks for the Script |
|
|
Just in time for my first backup.
Cheers,
Ernesto |
|
| Back to top |
|
|
SimonNZ
partially protected
Joined: 29 Jan 2008
Posts: 9
|
| Posted: Mon Nov 17, 2008 9:19 pm Post subject: |
|
|
Thanks for this Jasper, it has saved my neck! I was trying to write this exact thing and beating my head against the wall. Cheers  |
|
| Back to top |
|
|
jsiegers
Could work for Apple
Joined: 21 Oct 2008
Posts: 41
|
| Posted: Tue Nov 18, 2008 3:43 am Post subject: |
|
|
| aaron wrote: | | This does look like a good script. Can you please post it as a story to our main page? |
Love to. But I'm currently working on a better version of the script. Where you're able to do multiple cvcp's at the same time. This would make it much, much faster. So when that's finished I'll post it here and write you a story  |
|
| Back to top |
|
|
jsiegers
Could work for Apple
Joined: 21 Oct 2008
Posts: 41
|
|
| Back to top |
|
|
|
Xsanity Forum
Forum Search
Forum Help
Log in to check messages
Xsanity Articles
