Clone Data Script

This script automates the process of backing up files from a remote server to a local directory. It uses rsync over SSH to ensure that the local backup directory is synchronized with the remote directory. The script includes logging capabilities and retry mechanisms in case of connection issues.

• BACKUPS_HOST: The hostname or IP address of the backup server (e.g., backup.elosys.net).
• BACKUP_DIR: The local directory where backups will be stored (e.g., /home/amine/backups).
• LOGFILE: The file where logs will be stored (default: $BACKUP_DIR/out.log).

touch $LOGFILE
exec 2>&1
exec >> $LOGFILE

• Creates the log file if it doesn't exist.
• Redirects both stderr and stdout to the log file for comprehensive logging.

DATE="date +\%Y-\%m-\%d\ \%H:\%M:\%S"

• Defines a variable DATE to format and print the current date and time.

echo "["$(eval $DATE)"] Clone backup start"

• Logs the start time of the backup process.

while ! ssh -p 2224 $BACKUPS_HOST stat /upload/*.* \> /dev/null ;
do
    echo "["$(eval $DATE)"] Cannot contact backup server. Retrying in 1mn"
    sleep 30
done
sleep 2

• Uses a while loop to repeatedly check if backups exist on the remote server.
• If the server is not reachable, logs a message and retries after 30 seconds.
• Introduces a short delay after the server becomes reachable.

rsync -ahvz -H --delete --exclude='out.log' --exclude=".*" --exclude='anonymization' -e 'ssh -p 2224' $BACKUPS_HOST:/upload/ $BACKUP_DIR

• Uses rsync to synchronize the remote backup directory (/upload/) with the local backup directory ($BACKUP_DIR).
• Options used:
  • -a : Archive mode, which preserves permissions and other attributes.
  • -h : Human-readable output.
  • -v : Verbose mode.
  • -z : Compression during transfer.
  • -H : Preserve hard links.
  • –delete: Delete files in the local directory that are not in the remote directory.
  • –exclude: Exclude certain files or directories from being copied.
  • -e 'ssh -p 2224': Use SSH with a specified port for the connection.

rm $BACKUP_DIR/output.log

• Removes the output.log file from the local backup directory.

echo "["$(eval $DATE)"] Done"

• Logs the completion time of the backup process.

— Nadir Habib 2024/06/05 14:53