Fork of evobackup using rsync --link-dest.

This commit is contained in:
Benoît S. 2014-04-02 18:34:58 +02:00
parent 8a75570ef5
commit 1318ae89c9
18 changed files with 77 additions and 489 deletions

View file

@ -4,81 +4,21 @@ Installing EvoBackup
Backup server side
------------------
1) Git clone the project (i.e in /root/evobackup).
A rsync daemon launched in server mode.
2) Install configuration files.
```
root@backupserver:~/evobackup# install.sh
```
This will create /etc/evobackup and /etc/init.d/evobackup (or
/etc/init/evobackup.conf for Ubuntu).
3) Set up the first chroot.
```
root@backupserver:~/evobackup# chroot-new.sh -n client1 -i 192.168.0.10 -p 2222 -k /path/to/rsakeyclient1.pub
```
This will create the OpenSSH chroot for the machine "client1", listening on
port 2222 and accepting only connections from 192.168.0.10 using public key
rsakeyclient1.pub.
Tip: If you have already a chroot, you can commit the port option (-p), it
will be incremented from the last chroot.
4) Handle incrementals by modifying /etc/evobackup/conf.d/incs/client1
Syntax of this file is simple:
* +%Y-%m-%d.-0day Keep actual day
* +%Y-%m-%d.-1day Keep yesterday
* +%Y-%m-01.-0month Keep the firt day of the actual month
* +%Y-%m-01.-1month Keep the first day of the last month
Tip: You can use rdiff-backup in place of rsync, and choose to not use
EvoBackup incrementals method. You need to modify the cronjob.
5) Set up the scripts which will handle incrementals.
```
root@backupserver:~/evobackup# mkdir -p /usr/share/scripts
root@backupserver:~/evobackup# cp crons/evobackup-{inc,rm}.sh /usr/share/scripts/
root@backupserver:~/evobackup# chmod u+x /usr/share/scripts/evobackup-{inc,rm}.sh
root@backupserver:~/evobackup# crontab -e
```
Set this in the root crontab
```
29 10 * * * pkill evobackup-rm.sh && echo "Kill evobackup-rm.sh done" | mail -s "[warn] EvoBackup - purge incs interrupted" root
30 10 * * * /usr/share/scripts/evobackup-inc.sh && /usr/share/scripts/evobackup-rm.sh
Edit the configuration in /etc/evobackup/conf.d/incrementals.cf at least for MAIL_TO.
````
Client side
-----------
1) Git clone the project (i.e in /root/evobackup).
2) Generates OpenSSH key for user root (if user root don't have one already).
```
root@client1:~/evobackup# ssh-keygen
```
Do not set a passphrase, otherwise you will need to enter the passphrase (or
store it using an agent) for each backups!
3) Install configuration files.
2) Install configuration files.
```
root@client1:~/evobackup# install.sh client
```
4) Add the zzz_evobackup crontab into the daily cronjobs (recommended):
3) Add the zzz_evobackup crontab into the daily cronjobs (recommended):
```
root@client1:~/evobackup# cp crons/zzz_evobackup /etc/cron.daily/
@ -87,7 +27,7 @@ root@client1:~/evobackup# chmod 700 /etc/cron.daily/zzz_evobackup
Why "zzz"? Because we want the backup cronjob to be the last one.
5) Configure the cronjob.
4) Configure the cronjob.
In /etc/evobackup:
@ -97,19 +37,10 @@ In /etc/evobackup:
This will be launched before the rsync, using run-parts.
* What to backup using rsync filter rules in conf.d/include.cf
* Incrementals to keep in conf.d/incs.cf
* General config in conf.d/cron.cf
6) Connect to the OpenSSH chroot for the first time and accept the fingerprint.
```
root@client1:~/evobackup# ssh backupserver.mydomain.tld -p 2222
The authenticity of host 'backupserver.domain.tld (192.168.0.10)' can't be established.
RSA key fingerprint is a6:da:40:ac:72:f2:41:ec:7f:ca:d3:86:f6:27:19:77.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'backupserver.domain.tld,192.168.0.10' (RSA) to the list of known hosts.
```
7) Optional, test with ```sh -x &```, and see if it seems to works.
5) Optional, test with ```sh -x &```, and see if it seems to works.
```
root@client1:~/evobackup# sh -x /etc/cron.daily/zzz_evobackup &
@ -122,16 +53,4 @@ If it works, you can wait for it to finish or cancel it.
root@client1:~/evobackup# ^C
root@client1:~/evobackup# fg
root@client1:~/evobackup# ^C
```
Updating OpenSSH chroot
-----------------------
When you upgrade you system you may need to upgrade the OpenSSH chroot. To do
that launch update-chroot.sh.
```
root@backupserver:~/evobackup# chroot-update.sh
```
Then restart evobackup using init script.
```

View file

@ -1,49 +1,28 @@
EvoBackup
EvoBackup-linkdest
=========
EvoBackup is a bunch of shell scripts to create a backup server which will
handle the backup of many servers (clients). Licence is GPLv2.
(Fork of EvoBackup using rsync --link-dest as incrementals).
The main principle uses SSH chroot (called "jails" in the FreeBSD
world) for each client to backup. Each client will upload his data every day
using rsync in his chroot (using root account).
Incrementals are stored outside of the chroot using hard links. (So incrementals
are not available for clients). Using this method we can keep tens of backup of
each client securely and not using too much space.
EvoBackup is a bunch of shell scripts to backup a machine to a rsync server.
Licence is GPLv2.
```
Backup server
************
Server 1 ------ SSH/rsync -------> * tcp/2222 *
* *
Server 2 ------ SSH/rsync -------> * tcp/2223 *
************
```
This method uses standard tools (ssh, rsync, cp -al). EvoBackup is used for
This method uses standard tools (ssh, rsync). EvoBackup is used for
many years by Evolix for back up each day hundreds of servers which uses many
terabytes of data.
Backup server
-------------
The backup server need to be based on Debian. Tested on Debian Wheezy and
Ubuntu 13.04.
The backup server need to use rsync. Evobackup-linkdest was tested only on
Online backup offer using rsync and RPN.
Needed packages:
* openssh-server
* rsync
* bsd-mailx (or other package providing /usr/bin/mailx)
Backups are stored in a big partition mounted on /backup (you can change this).
For security reasons it is recommended to encrypt the backup partition (i.e
using LUKS).
Main directories:
* /backup/jails: chroot used by clients
* /backup/incs: incrementals
* /etc/evobackup: Config dir
To install and configure EvoBackup read INSTALL.md

View file

@ -1,30 +0,0 @@
#!/bin/bash
# Copy essential binaries into the chroot.
chrootdir=$1
# TODO: better detection of amd64 arch
cp -f /lib/ld-linux.so.2 $chrootdir/lib/ 2>/dev/null \
|| cp -f /lib64/ld-linux-x86-64.so.2 $chrootdir/lib64/
release=$(lsb_release -s -c)
if [ "$release" = "squeeze" ]; then
cp /lib/libnss* $chrootdir/lib/
else
if [ "$release" = "wheezy" ]; then
cp /lib/x86_64-linux-gnu/libnss* $chrootdir/lib/x86_64-linux-gnu/
else
# Others? Not tested...
cp /lib/x86_64-linux-gnu/libnss* $chrootdir/lib/x86_64-linux-gnu/
fi
fi
for dbin in /bin/bash /bin/cat /bin/chown /bin/mknod /bin/rm \
/bin/sed /bin/sh /bin/uname /bin/mount /usr/bin/rsync /usr/sbin/sshd \
/usr/lib/openssh/sftp-server; do
cp -f $dbin $chrootdir/$dbin;
for lib in `ldd $dbin | cut -d">" -f2 | cut -d"(" -f1`; do
cp -p $lib $chrootdir/$lib
done
done

View file

@ -1,7 +0,0 @@
root:x:0:
daemon:x:1:
shadow:x:42:
staff:x:50:
users:x:100:
nogroup:x:65534:
ssh:x:102:

View file

@ -1,4 +0,0 @@
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
sshd:x:100:65534::/var/run/sshd:/usr/sbin/nologin

View file

@ -1,4 +0,0 @@
root:x:13536:0:99999:7:::
daemon:*:13536:0:99999:7:::
nobody:*:13536:0:99999:7:::
sshd:!:13536:0:99999:7:::

View file

@ -1,35 +0,0 @@
Port 2222
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PasswordAuthentication no
X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
UseLogin no
Subsystem sftp /usr/lib/openssh/sftp-server
AllowUsers root@IP root@::ffff:IP

View file

@ -1,132 +0,0 @@
#!/bin/bash
# Set-up and configure an OpenSSH chroot.
BACKUP_PATH='/backup/jails'
#Are we root?
id=$(id -u)
if [ $id != 0 ]; then
echo "Error, you need to be root to install EvoBackup!"
exit 1
fi
usage() {
cat <<EOT
Add an OpenSSH chroot.
Usage: $0 -n name -i ip -p port -k pub-key-path
Mandatory parameters:
-n: Name of the chroot.
-i: IP address of the client machine.
-k: Path to the SSH public key of the client machine.
Optional parameters:
-p: SSH port which chroot/jail will listen on.
port can be ommited if there is already one chroot, it will be guessed.
EOT
}
newchroot() {
# Path to the chroot.
chrootdir=$1
mkdir -p $chrootdir
chown root:root $chrootdir
umask 022
# create jail
echo -n "1 - Creating the chroot..."
mkdir -p $chrootdir/{bin,dev,etc/ssh,lib,lib64,proc}
mkdir -p $chrootdir/lib/{x86_64-linux-gnu,tls/i686/cmov,i686/cmov}
mkdir -p $chrootdir/usr/{bin,lib,sbin}
mkdir -p $chrootdir/usr/lib/{x86_64-linux-gnu,openssh,i686/cmov}
mkdir -p $chrootdir/root/.ssh
mkdir -p $chrootdir/var/{log,run/sshd}
touch $chrootdir/var/log/{authlog,lastlog,messages,syslog}
touch $chrootdir/etc/fstab
echo "...OK"
echo -n "2 - Copying essential files..."
cp /proc/devices $chrootdir/proc
cp /etc/ssh/{ssh_host_rsa_key,ssh_host_dsa_key} $chrootdir/etc/ssh/
cp chroot-etc/sshd_config $chrootdir/etc/ssh/
cp chroot-etc/passwd $chrootdir/etc/
cp chroot-etc/shadow $chrootdir/etc/
cp chroot-etc/group $chrootdir/etc/
echo "...OK"
echo -n "3 - Copying binaries..."
./chroot-bincopy.sh $chrootdir
echo "...OK"
}
while getopts ':n:i:p:k:' opt; do
case $opt in
n)
jail=$OPTARG
;;
i)
ip=$OPTARG
;;
p)
port=$OPTARG
;;
k)
pub_key_path=$OPTARG
;;
?)
usage
exit 1
;;
esac
done
# Verify parameters.
if [ -z $jail ] || [ -z $ip ] || [ -z $pub_key_path ];
then
usage
exit 1
fi
# Test if the chroot exists.
if [ -d ${BACKUP_PATH}/${jail} ]; then
echo "Error, directory to chroot already exists!"
exit 1
fi
# Verify the presence of the public key.
if [ ! -f "$pub_key_path" ]; then
echo "Public key $pub_key_path not found."
exit 1
fi
# If port ommited try to guess it.
if [ -z $port ]; then
port=$(grep -h Port /backup/jails/*/etc/ssh/sshd_config \
| grep -Eo [0-9]+ | sort -n | tail -1)
port=$((port+1))
if [ -z $port ]; then
echo "Port cannot be guessed. Add -p option!"
exit 1
fi
fi
# Create the chroot
newchroot ${BACKUP_PATH}/${jail}
# Configure the chroot
echo -n "4 - Configuring the chroot..."
sed -i "s/^Port 2222/Port ${port}/" ${BACKUP_PATH}/${jail}/etc/ssh/sshd_config
sed -i "s/IP/$ip/g" ${BACKUP_PATH}/${jail}/etc/ssh/sshd_config
cat $pub_key_path > ${BACKUP_PATH}/${jail}/root/.ssh/authorized_keys
chmod -R 600 ${BACKUP_PATH}/${jail}/root/.ssh/
chown -R root:root ${BACKUP_PATH}/${jail}/root/.ssh/
cat <<EOT >/etc/evobackup/conf.d/incs/${jail}
+%Y-%m-%d.-0day
+%Y-%m-%d.-1day
+%Y-%m-%d.-2day
+%Y-%m-%d.-3day
+%Y-%m-01.-0month
+%Y-%m-01.-1month
EOT
echo "Done. OpenSSH chroot added! Restart evobackup service."

View file

@ -1,11 +0,0 @@
#!/bin/bash
# Update all OpenSSH chroot.
BACKUP_PATH='/backup/jails'
for i in `ls -1 ${BACKUP_PATH}/*/lib/libnss_compat.so.2`; do
chrootdir=`echo $i | cut -d"/" -f1,2,3,4`
echo -n "Updating $chrootdir ..."
./chroot-bincopy.sh $chrootdir
echo "Done!"
done

View file

@ -1,27 +0,0 @@
#!/bin/bash
# Handles creating incrementals backup.
. /etc/evobackup/conf.d/incrementals.cf
tmplog=$(mktemp --tmpdir=/tmp evobackup.tmplog.XXX)
# Don't return *, if bash glob don't find files/dir.
shopt -s nullglob
# Search for incrementals to do.
for client in ${CONFDIR}/*; do
start=$(date --rfc-3339=seconds)
backupname=${client#/etc/evobackup/conf.d/incs/}
echo "Incrementals of $backupname started at ${start}." \
>> $tmplog
[[ ! -d ${INCDIR}/${backupname} ]] && mkdir -p ${INCDIR}/${backupname}
# Do the incrementals.
cp -alx ${JAILDIR}/${backupname} ${INCDIR}/${backupname}/${DATEDIR}
stop=$(date --rfc-3339=seconds)
echo "Incrementals of $backupname ended at ${stop}." >> $tmplog
done
# Save tmplog to global log.
cat $tmplog >> $LOGFILE
# Send mail report.
< $tmplog mailx -s "[info] EvoBackup report of creating incrementals" $MAIL_TO
# Cleaning.
rm $tmplog

View file

@ -2,45 +2,43 @@
# Handle removing of incrementals.
. /etc/evobackup/conf.d/incrementals.cf
. /etc/evobackup/conf.d/cron.cf
tmpdir=$(mktemp --tmpdir=/tmp -d evobackup.tmpdir.XXX)
emptydir=$(mktemp --tmpdir=/tmp -d evobackup.empty.XXX)
tmplog=$(mktemp --tmpdir=/tmp evobackup.tmplog.XXX)
# Don't return *, if bash glob don't find files/dir.
shopt -s nullglob
dst="rsync://${RSYNC_USERNAME}@${BACKUPSERVER}/${RSYNC_PATH}"
# For each client (machine to backup), delete old incrementals according to the
# config file.
for client in ${CONFDIR}/*; do
# Get only the name of the backup.
backupname=${client#${CONFDIR}/}
# List actual incrementals backup.
for inc in ${INCDIR}/${backupname}/*; do
echo $inc
done > ${tmpdir}/${backupname}.files
# List non-obsolete incrementals backup.
for incConf in $(cat ${CONFDIR}/${backupname}); do
mydate=$(echo $incConf | cut -d. -f1)
before=$(echo $incConf | cut -d. -f2)
date -d "$(date $mydate) $before" "+%Y-%m-%d"
done > ${tmpdir}/${backupname}.keep
# Delete obsolete incrementals backup
for inc in $(grep -v -f ${tmpdir}/${backupname}.keep ${tmpdir}/${backupname}.files); do
start=$(date --rfc-3339=seconds)
echo "Deletion of ${backupname}/${inc#${INCDIR}/${backupname}/} started at ${start}." >> $tmplog
# We use rsync to delete since it is faster than rm!
rsync -a --delete ${emptydir}/ $inc
rmdir $inc
stop=$(date --rfc-3339=seconds)
echo "Deletion of ${backupname}/${inc#${INCDIR}/${backupname}/} ended at ${stop}." >> $tmplog
done
# List actual incrementals backup.
# RSYNC_PASSWORD=ojvafnabOn5 rsync --list-only
# rsync://evolix-1vhqa@localhost:8873/evolix-1vhqa_backup/ | sed -E 's#^([^\s]+\s+){4}##'
listincs=$(rsync --list-only ${dst} |
sed -E 's#^([^\s]+\s+){4}##' | sed -e '/\./d' -e '/current/d' | tr -s '\n' ' '
)
for inc in $listincs; do
echo $inc >> ${tmpdir}/incs.files
done
# List non-obsolete incrementals backup.
for incConf in $(cat ${CONFDIR}/incs.cf); do
mydate=$(echo $incConf | cut -d. -f1)
before=$(echo $incConf | cut -d. -f2)
date -d "$(date $mydate) $before" "+%Y-%m-%d"
done > ${tmpdir}/incs.keep
# Delete obsolete incrementals backup
for inc in $(grep -v -f ${tmpdir}/incs.keep ${tmpdir}/incs.files); do
start=$(date --rfc-3339=seconds)
echo "Deletion of $inc started at ${start}." >> $tmplog
echo rsync -a --delete ${emptydir}/ ${dst}/${inc}
echo rsync -a --delete --include="${inc}" --exclude="*" ${emptydir}/ $dst
stop=$(date --rfc-3339=seconds)
echo "Deletion of $inc ended at ${stop}." >> $tmplog
done
# Send mail report only if incrementals where deleted.
if [ -s $tmplog ]; then
# Save tmplog to global log & send mail.
cat $tmplog >> $LOGFILE
cat $tmplog >> $LOG
< $tmplog mailx -s "[info] EvoBackup - deletion of obsolete incrementals" $MAIL_TO
fi
# Cleaning

View file

@ -21,13 +21,24 @@ run-parts /etc/evobackup/actions.d/
hostname=$(hostname -f)
start=$(date --rfc-3339=seconds)
date=$(date +%Y-%m-%d)
tmplog=$(mktemp --tmpdir=/tmp evobackup.XXX)
# rsync command line to backup all data.
rsync -avzh --stats --force --ignore-errors --partial \
--include-from=/etc/evobackup/conf.d/include.cf \
--delete --delete-excluded / -e "ssh -p $SSHPORT -4" \
root@${BACKUPSERVER}:/var/backup/ > $tmplog
args="-avzh --stats --force --ignore-errors --partial "\
"--include-from=/etc/evobackup/conf.d/include.cf "\
"--delete --delete-excluded"
dst="rsync://${RSYNC_USERNAME}@${BACKUPSERVER}/${RSYNC_PATH}"
# Is there a current version? If no create it.
rsync --list-only ${dst}/current >/dev/null 2>&1
status=$?
if [ "$status" != 0 ]; then
# First backup.
rsync $args / ${dst}/current > $tmplog
else
# Backups with incrementials.
rsync $args --link-dest=/current/ / ${dst}/${date} > $tmplog
fi
status=$?
# Keep the last 30 lines & clean temporary log.
@ -46,4 +57,4 @@ if [ "$status" != 0 ]; then
else
tail -30 $LOG \
| mailx -s "[info] EvoBackup report for $hostname" $MAIL_TO
fi
fi

View file

@ -1,12 +1,21 @@
# Pid file
PIDFILE="/var/run/evobackup.pid"
# Port of the OpenSSH chroot on the backup server.
SSHPORT=2222
# Port of the rsync backup server. Default is 837.
PORT=837
# Hostname OR adress IP of the backup server.
BACKUPSERVER="mybackupserver.domain.tld"
# Username, if any.
RSYNC_USERNAME=""
# Password, if any.
export RSYNC_PASSWORD=""
# Path used by rsync server.
RSYNC_PATH="backup"
# A mail to send the report or alert.
MAIL_TO="jdoe@example.com"
@ -15,3 +24,5 @@ LOG="/var/log/evobackup.log"
# Used when you have more than one backup server.
NODE=$(( $(date +%-d) % 2 ))
CONFDIR=/etc/evobackup/conf.d

View file

@ -1,6 +0,0 @@
CONFDIR=/etc/evobackup/conf.d/incs
DATEDIR=$(date +"%Y-%m-%d")
LOGFILE=/var/log/evobackup-incs.log
JAILDIR=/backup/jails
INCDIR=/backup/incs
MAIL_TO=jdoe@example.com

View file

@ -0,0 +1,6 @@
+%Y-%m-%d.-0day
+%Y-%m-%d.-1day
+%Y-%m-%d.-2day
+%Y-%m-%d.-3day
+%Y-%m-01.-0month
+%Y-%m-01.-1month

View file

@ -1,51 +0,0 @@
#!/bin/bash
### BEGIN INIT INFO
# Provides: evobackup
# Required-Start: $syslog
# Required-Stop: $syslog
# Default-Start: 2
# Default-Stop: 1
# Short-Description: Backup manager using rsync and OpenSSH chroot.
### END INIT INFO
set -e
BACKUP_PATH=/backup
case "$1" in
start)
for jail in ${BACKUP_PATH}/jails/*; do
mount -t proc proc-chroot ${jail}/proc/
mount -t devtmpfs udev ${jail}/dev/
mount -t devpts devpts ${jail}/dev/pts
chroot ${jail} /usr/sbin/sshd > /dev/null
done
;;
stop)
for jail in ${BACKUP_PATH}/jails/*; do
kill $(chroot $jail cat /var/run/sshd.pid)
umount ${jail}/proc/
umount ${jail}/dev/pts/
# Need to wait a little time before unmounting /dev
sleep 0.2
umount ${jail}/dev
done
;;
reload|force-reload)
for jail in ${BACKUP_PATH}/jails/*; do
kill -HUP \
$(chroot $jail cat /var/run/sshd.pid)
done
;;
restart)
for jail in ${BACKUP_PATH}/jails/*; do
kill $(chroot $jail cat /var/run/sshd.pid)
chroot $jail /usr/sbin/sshd > /dev/null
done
;;
esac
exit 0

View file

@ -1,29 +0,0 @@
# evobackup
description "Backup manager using rsync and OpenSSH chroot."
author "Evobackup team <equipe@evolix.fr>"
start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]
env BACKUP_PATH=/backup
pre-start script
for jail in ${BACKUP_PATH}/jails/*; do
mount -t proc proc-chroot ${jail}/proc
mount -t devtmpfs udev ${jail}/dev
mount -t devpts devpts ${jail}/dev/pts
chroot $jail /usr/sbin/sshd > /dev/null
done
end script
post-stop script
for jail in ${BACKUP_PATH}/jails/*; do
kill $(chroot $jail cat /var/run/sshd.pid)
umount ${jail}/proc
umount ${jail}/dev/pts
# Need to wait a little time before unmounting /dev
sleep 0.2
umount ${jail}/dev
done
end script