From eda30a013c6de871b08429a15278537a45685f97 Mon Sep 17 00:00:00 2001 From: Patrick Marchand Date: Wed, 2 Jan 2019 13:45:21 -0500 Subject: [PATCH 01/26] Switch documentation to mdoc(7) Markdown is not a suitable language for technical documentation. --- bkctld.8 | 179 ++++++++++++++++++++++++++++++ bkctld.conf.5 | 67 ++++++++++++ docs/configuration.md | 92 +++++++++++++--- docs/incrementals.md | 93 +++++++++------- docs/usage.md | 249 +++++++++++++++++++++++++----------------- evobackup-incl.5 | 79 ++++++++++++++ 6 files changed, 606 insertions(+), 153 deletions(-) create mode 100644 bkctld.8 create mode 100644 bkctld.conf.5 create mode 100644 evobackup-incl.5 diff --git a/bkctld.8 b/bkctld.8 new file mode 100644 index 0000000..ae8d2f6 --- /dev/null +++ b/bkctld.8 @@ -0,0 +1,179 @@ +.Dd December 27, 2018 +.Dt BKCTLD 8 +.Os +.Sh NAME +.Nm bkctld +.Nd tool to manage evobackup jails +.Sh SYNOPSIS +.Nm +.Op Ar operand... +.Sh DESCRIPTION +.Nm +is a shell script that creates and manages a backup server +which can handle the backups of many other servers (clients). +.Pp +It uses +.Xr ssh 1 +and +.Xr chroot 8 +to sandbox every client's backups. +Each client will upload it's data every day +using +.Xr rsync 1 +in it's +.Xr chroot 8 +(using the root account). +.Pp +Prior backups are stored incrementally outside of the +.Xr chroot 8 +using +.Xr ln 1 +hard links or BTRFS snapshots. +(So they can not be affected by the client), +which backups are kept over time can be configured in the jail's nominal +.Xr evobackup-incl 5 +configuration file. +.Pp +A large enough volume must be mounted on +.Pa /backup , +if the filesystem is formatted with BTRFS, +.Nm +will use sub-volumes and snapshots to save space. +.Pp +It's default settings can be overridden in +.Xr bkctld.conf 5 +file. +.Pp +The following operands are available: +.Bl -tag -width Ds +.It Cm init Ar jailname +Create an evobackup jail +.It Cm update Cm all | Ar jailname +Update an evobackup jail +.It Cm remove Cm all | Ar jailname +Remove an evobackup jail +.It Cm start Cm all | Ar jailname +Start an evobackup jail +.It Cm stop Cm all | Ar jailname +Stop an evobackup jail +.It Cm reload Cm all | Ar jailname +Reload an evobackup jail +.It Cm restart Cm all | Ar jailname +Restart an evobackup jail +.It Cm sync Cm all | Ar jailname +Sync an evobackup jail, the mirror server is defined by the +.Ev $NODE +variable in +.Pa /etc/default/bkctld +.It Cm status Op Ar jailname +Print the status of all jails or only +.Op Ar jailname . +.It Cm key Ar jailname Op Ar keyfile +Print or set the +.Xr ssh 1 +public key of an evobackup jail +.It Cm port Ar jailname Op Cm auto | Ar port +Print or set the +.Xr ssh 1 +.Op Ar port +of an evobackup jail. +Using +.Op Cm auto +will set it to the next available port. +.It Cm ip Ar jailname Op Cm all | Ar address +Print or set the whitelisted IP +.Op Ar address +for an evobackup jail. +.Op Cm all +allows unrestricted access and is the default. +.It Cm inc +Generate incremental backups +.It Cm rm +Remove old incremental backups +.El +.Sh FILES +.Bl -tag -width Ds +.It Pa /usr/share/bkctld/bkctld.conf +Template for +.Xr bkctld.conf 5 +.It Pa /usr/share/bkctld/incl.tpl +Default rules for the incremental backups are stored here. +.El +.Sh EXAMPLES +Before creating a jail and backing up a client, +the backup server administrator will need: +.Bl -bullet +.It +The host name of the client system. +.It +The public RSA +.Xr ssh 1 +key for the +.Dq root +user of the client system, +it is recommended the private key be password-less if automation is desired. +.It +The IPv4 address of the client system is needed +if the administrator wishes to maintain a whitelist, +see +.Va FIREWALL_RULES +in +.Xr bkctld.conf 5 +.El +.Pp +He can then create the jail: +.Bd -literal -offset indent +# bkctld init CLIENT_HOST_NAME +# bkctld key CLIENT_HOST_NAME /root/CLIENT_HOST_NAME.pub +# bkctld ip CLIENT_HOST_NAME CLIENT_IP_ADDRESS +# bkctld start CLIENT_HOST_NAME +# bkctld status CLIENT_HOST_NAME +.Ed +.Pp +And override the default +.Xr evobackup-incl 5 +rules +.Bd -literal -offset indent +# $EDITOR /etc/evobackup/CLIENT_HOST_NAME +.Ed +.Pp +To sync itself, +the client server will need to install +.Xr rsync 1 . +It can then be run manually: +.Bd -literal -offset indent +# rsync -av -e "ssh -p JAIL_PORT" /home/ root@BACKUP_SERVER:/var/backup/home/ +.Ed +.Pp +If a more automated setup is required, +a script can be written in any programming language. +In this case, +it may be useful to validate the backup server's identity before hand. +.Bd -literal -offset indent +# ssh -p JAIL_PORT BACKUP_SERVER +.Ed +.Pp +A +.Xr bash 1 +example to be run under the +.Dq root +user's +.Xr crontab 5 +can be found in the +.Lk https://gitea.evolix.org/evolix/evobackup/src/branch/master/zzz_evobackup "source repository" +.\" .Sh EXIT STATUS +.\" For sections 1, 6, and 8 only. +.\" .Sh DIAGNOSTICS +.\" For sections 1, 4, 6, 7, 8, and 9 printf/stderr messages only. +.Sh SEE ALSO +.Xr rsync 1 , +.Xr ssh-keygen 1 , +.Xr bkctld 5 , +.Xr evobackup-incl 5 , +.Xr chroot 8 , +.Xr cron 8 , +.Xr sshd 8 +.Sh AUTHORS +.An Victor Laborie +.\" .Sh CAVEATS +.\" .Sh BUGS \ No newline at end of file diff --git a/bkctld.conf.5 b/bkctld.conf.5 new file mode 100644 index 0000000..f281354 --- /dev/null +++ b/bkctld.conf.5 @@ -0,0 +1,67 @@ +.Dd December 28, 2018 +.Dt BKCTLD.CONF 5 +.Os +.Sh NAME +.Nm bkctld.conf +.Nd configuration file for +.Xr bkctld 8 +.Sh SYNOPSIS +.D1 name=[value] +.Sh DESCRIPTION +The +.Nm +file contains variables that override the behavior of the +.Xr bkctld 8 +script. +By default, it is located at +.Pa /usr/share/bkctld/bkctld.conf . +.Pp +Each line must be a valid +.Xr bash 1 +variable definition. +Lines beginning with the +.Sq # +character are comments and are ignored. +The order of the definitions does not matter. +.Pp +The following variables may be defined: +.Bl -tag -width Ds +.It Va CONFDIR +Directory where +.Xr evobackup-incl 5 +files are kept. +It's default value is +.Pa /etc/evobackup/ . +.It Va JAILDIR +Directory where the jails are stored, +it is recommended that this be inside a BTRFS file system. +It's default value is +.Pa /backup/jails/ . +.It Va INCDIR +Directory where incremental backups are stored, +it is recommended that this be inside a BTRFS file system. +It's default value is +.Pa /backup/incs/ . +.It Va TPLDIR +Directory where the default configuration files are stored. +It's default value is +.Pa /usr/share/bkctld/ . +.It Va LOCALTPLDIR +Directory where custom configuration files are stored. +It's default is +.Pa /usr/local/share/bkctld/ . +.It Va LOGLEVEL +Defines the amount of information to log, follows the same scale as in +.In syslog.h +and defaults to 6. +.It Va FIREWALL_RULES +Configuration file containing the firewall rules that govern jail access. +This file must be sourced by your firewall. +It does not have a default value and, if unset, +.Xr bkctld 8 +will not automatically update the firewall. +.El +.Sh SEE ALSO +.Xr bash 1 , +.Xr evobackup-incl 5 , +.Xr bkctld 8 diff --git a/docs/configuration.md b/docs/configuration.md index 8045a6c..310bdb9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,20 +1,86 @@ -# CONFIGURATION VARS +BKCTLD.CONF(5) - File Formats Manual -bkctld configuration has to be set in /etc/default/bkctld file. +# NAME -## REQUIREDS VARS +**bkctld.conf** - configuration file for +bkctld(8) -Default required vars are defined in bkctld script. Alter them to override default values. +# SYNOPSIS -* CONFDIR (default: /etc/evobackup) : Dir where incremental backup is configured. See INCS CONFIGURATION section for details. -* JAILDIR (default : /backup/jails) : Dir for jail's root dir. BTRFS recommended. -* INCDIR (default : /backups/incs) : Dir where incremental backup is stored. BTRFS recommended. -* TPLDIR (default : /usr/share/bkctld) : Dir where jail template file is stored. -* LOCALTPLDIR (default : /usr/local/share/bkctld) : Dir for surcharge jail templates. -* LOGLEVEL (default : 6) : Define loglevel, based on syslog severity level. +> name=\[value] -## OPTIONALS VARS +# DESCRIPTION -Optionnals vars are no default value. No set them desactivate correspondant fonctionnality. +The +**bkctld.conf** +file contains variables that override the behavior of the +bkctld(8) +script. +By default, it is located at +*/usr/share/bkctld/bkctld.conf*. -* FIREWALL_RULES (default: no firewall auto configuration) : Configuration file were firewall was configured to allow jail access. This file must be sourced by your firewall configuration tool. +Each line must be a valid +bash(1) +variable definition. +Lines beginning with the +'#' +character are comments and are ignored. +The order of the definitions does not matter. + +The following variables may be defined: + +*CONFDIR* + +> Directory where +> evobackup-incl(5) +> files are kept. +> It's default value is +> */etc/evobackup/*. + +*JAILDIR* + +> Directory where the jails are stored, +> it is recommended that this be inside a BTRFS file system. +> It's default value is +> */backup/jails/*. + +*INCDIR* + +> Directory where incremental backups are stored, +> it is recommended that this be inside a BTRFS file system. +> It's default value is +> */backup/incs/*. + +*TPLDIR* + +> Directory where the default configuration files are stored. +> It's default value is +> */usr/share/bkctld/*. + +*LOCALTPLDIR* + +> Directory where custom configuration files are stored. +> It's default is +> */usr/local/share/bkctld/*. + +*LOGLEVEL* + +> Defines the amount of information to log, follows the same scale as in +> <*syslog.h*> +> and defaults to 6. + +*FIREWALL\_RULES* + +> Configuration file containing the firewall rules that govern jail access. +> This file must be sourced by your firewall. +> It does not have a default value and, if unset, +> bkctld(8) +> will not automatically update the firewall. + +# SEE ALSO + +bash(1), +evobackup-incl(5), +bkctld(8) + +OpenBSD 6.4 - December 28, 2018 diff --git a/docs/incrementals.md b/docs/incrementals.md index 9cd3c9d..046fc7d 100644 --- a/docs/incrementals.md +++ b/docs/incrementals.md @@ -1,63 +1,80 @@ -# INCS CONFIGURATION +EVOBACKUP-INCL(5) - File Formats Manual -Located by default in /etc/evobackup/, each incl.tpl file is named -after the EvoBackup for which the rules it contains must apply. +# NAME -The rules it defines decide which incremental backups are kept when -running `bkctl rm` +**evobackup-incl** - incremental backup configuration -Each line defines a single rule. The first part of the rule describes -when the backup was taken, the second part decides how long to keep -it. Lines beginning with the # character are comments and are -ignored. The order of the rules does not matter. +# SYNOPSIS -Evobackups that do not have their nominal incl.tpl file use the -default rules defined in /usr/share/bkctld/inc.tpl +> \+%Y-%m-%d.-%day + +# DESCRIPTION + +Located by default in +*/etc/evobackup/*, +each +**evobackup-incl** +file is named after the +bkctld(8) +backup for which the rules it contains must apply. + +The rules it defines decide which incremental backups are kept when running + + # bkctld rm + +Each line defines a single rule. +The first part of the rule describes when the backup was taken, +the second part decides how long to keep it. +Lines beginning with the +'#' +character are comments and are ignored. +The order of the rules does not matter. + +Evobackups that do not have their nominal +**evobackup-incl** +file use the default rules defined in +*/usr/share/bkctld/inc.tpl* + +# EXAMPLES Keep today's backup: -``` -+%Y-%m-%d.-0day -``` + +%Y-%m-%d.-0day Keep yesterday's backup: -``` -+%Y-%m-%d.-1day -``` + +%Y-%m-%d.-1day Keep the first day of this month: -``` -+%Y-%m-01.-0month -``` + +%Y-%m-01.-0month Keep the first day of last month: -``` -+%Y-%m-01.-1month -``` + +%Y-%m-01.-1month Keep backups for every 15 days: -``` -+%Y-%m-01.-1month -+%Y-%m-15.-1month -``` + +%Y-%m-01.-1month + +%Y-%m-15.-1month Keep a backup of the first day of january: -``` -+%Y-01-01.-1month -``` + +%Y-01-01.-1month Keep backups of the last 4 days and the first day of the last 2 months: -``` -+%Y-%m-%d.-0day -+%Y-%m-%d.-1day -+%Y-%m-%d.-2day -+%Y-%m-%d.-3day -+%Y-%m-01.-0month -+%Y-%m-01.-1month -``` \ No newline at end of file + +%Y-%m-%d.-0day + +%Y-%m-%d.-1day + +%Y-%m-%d.-2day + +%Y-%m-%d.-3day + +%Y-%m-01.-0month + +%Y-%m-01.-1month + +# SEE ALSO + +bkctld(8), +cron(8), +*/etc/evobackup/tpl/inc.tpl* + +OpenBSD 6.4 - December 28, 2018 diff --git a/docs/usage.md b/docs/usage.md index f67ca1d..bb22e64 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,162 +1,207 @@ +BKCTLD(8) - System Manager's Manual + # NAME -bkctld - tool to manage evobackup jail +**bkctld** - tool to manage evobackup jails # SYNOPSIS -~~~ -bkctld [] -~~~ +**bkctld** +\[*operand...*] # DESCRIPTION -bkctld is a shell script that creates and manages a backup server +**bkctld** +is a shell script that creates and manages a backup server which can handle the backups of many other servers (clients). -It uses OPENSSH and chroot's to sandbox every client's backups. -Each client will upload it's data every day using rsync in it's chroot + +It uses +ssh(1) +and +chroot(8) +to sandbox every client's backups. +Each client will upload it's data every day +using +rsync(1) +in it's +chroot(8) (using the root account). -Prior backups are stored incrementally outside of the chroot -using hard links or BTRFS snapshots (So they can not be affected -by the client). Which backups are kept over time can be configured in the jail's nominal [incl.tpl](incrementals.md) configuration file. A large enough volume must be mounted on `/backup`, if the filesystem is formatted -with BTRFS, bkctld will use sub-volumes and snapshots to save space. +Prior backups are stored incrementally outside of the +chroot(8) +using +ln(1) +hard links or BTRFS snapshots. +(So they can not be affected by the client), +which backups are kept over time can be configured in the jail's nominal +evobackup-incl(5) +configuration file. -It's default settings can be overridden in the [configuration file](configuration.md). +A large enough volume must be mounted on +*/backup*, +if the filesystem is formatted with BTRFS, +**bkctld** +will use sub-volumes and snapshots to save space. -# BKCTLD COMMANDS +It's default settings can be overridden in +bkctld.conf(5) +file. -Create an evobackup jail : +The following operands are available: -~~~ -bkctld init -~~~ +**init** *jailname* -Update an evobackup jail or all : +> Create an evobackup jail -~~~ -bkctld update |all -~~~ +**update** **all** | *jailname* -Remove an evobackup jail or all : +> Update an evobackup jail -~~~ -bkctld remove |all -~~~ +**remove** **all** | *jailname* -Start an evobackup jail or all : +> Remove an evobackup jail -~~~ -bkctld start |all -~~~ +**start** **all** | *jailname* -Stop an evobackup jail or all : +> Start an evobackup jail -~~~ -bkctld stop |all -~~~ +**stop** **all** | *jailname* -Reload an evobackup jail or all : +> Stop an evobackup jail -~~~ -bkctld reload |all -~~~ - -Restart an evobackup jail or all : +**reload** **all** | *jailname* -~~~ -bkctld restart |all -~~~ +> Reload an evobackup jail -Sync an evobackup jail or all. -Second server is defined by $NODE var in /etc/default/bkctld : +**restart** **all** | *jailname* -~~~ -bkctld sync |all -~~~ +> Restart an evobackup jail -Print status of all evobackup jail or one jail : +**sync** **all** | *jailname* -~~~ -bkctld status [] -~~~ +> Sync an evobackup jail, the mirror server is defined by the +> `$NODE` +> variable in +> */etc/default/bkctld* -Print or set the SSH public key of an evobackup jail : +**status** \[*jailname*] -~~~ -bkctld key [] -~~~ +> Print the status of all jails or only +> \[*jailname*]. -Print or set the SSH port of an evobackup jail. -Auto to set next available port (last + 1) : +**key** *jailname* \[*keyfile*] -~~~ -bkctld port [|auto] -~~~ +> Print or set the +> ssh(1) +> public key of an evobackup jail -Print or set allowed IP of an evobackup jail. -All for unrestricted access (default) : +**port** *jailname* \[**auto** | *port*] -~~~ -bkctld ip [|all] -~~~ +> Print or set the +> ssh(1) +> \[*port*] +> of an evobackup jail. +> Using +> \[**auto**] +> will set it to the next available port. -Generate inc of an evobackup jail : +**ip** *jailname* \[**all** | *address*] -~~~ -bkctld inc -~~~ +> Print or set the whitelisted IP +> \[*address*] +> for an evobackup jail. +> \[**all**] +> allows unrestricted access and is the default. -Remove old inc of an evobackup jail : +**inc** -~~~ -bkctld rm -~~~ +> Generate incremental backups + +**rm** + +> Remove old incremental backups + +# FILES + +*/usr/share/bkctld/bkctld.conf* + +> Template for +> bkctld.conf(5) + +*/usr/share/bkctld/incl.tpl* + +> Default rules for the incremental backups are stored here. + +# EXAMPLES -# CLIENT CONFIGURATION Before creating a jail and backing up a client, the backup server administrator will need: -* The host name of the client system. -* The public RSA OpenSSH key for the root user of the client system, -it is recommended the private key be password-less if automation is desired. -* The IPv4 address of the client system is needed -if the administrator wishes to maintain a whitelist, -see the FIREWALL_RULES variable in [bkctld.conf](configuration.md) +* The host name of the client system. + +* The public RSA + ssh(1) + key for the + "root" + user of the client system, + it is recommended the private key be password-less if automation is desired. + +* The IPv4 address of the client system is needed + if the administrator wishes to maintain a whitelist, + see + *FIREWALL\_RULES* + in + bkctld.conf(5) He can then create the jail: -``` -# bkctld init CLIENT_HOST_NAME -# bkctld key CLIENT_HOST_NAME /root/CLIENT_HOST_NAME.pub -# bkctld ip CLIENT_HOST_NAME CLIENT_IP_ADDRESS -# bkctld start CLIENT_HOST_NAME -# bkctld status CLIENT_HOST_NAME -``` + # bkctld init CLIENT_HOST_NAME + # bkctld key CLIENT_HOST_NAME /root/CLIENT_HOST_NAME.pub + # bkctld ip CLIENT_HOST_NAME CLIENT_IP_ADDRESS + # bkctld start CLIENT_HOST_NAME + # bkctld status CLIENT_HOST_NAME -And override the default [incremental](incrementals.md) rules +And override the default +evobackup-incl(5) +rules -``` -# $EDITOR /etc/evobackup/CLIENT_HOST_NAME -``` + # $EDITOR /etc/evobackup/CLIENT_HOST_NAME -To sync itself, the client server will need to install rsync. +To sync itself, +the client server will need to install +rsync(1). It can then be run manually: -``` -# rsync -av -e "ssh -p JAIL_PORT" /home/ root@BACKUP_SERVER:/var/backup/home/ -``` + # rsync -av -e "ssh -p JAIL_PORT" /home/ root@BACKUP_SERVER:/var/backup/home/ If a more automated setup is required, -a script can be written in any programming language. In this case, +a script can be written in any programming language. +In this case, it may be useful to validate the backup server's identity before hand. -``` -# ssh -p JAIL_PORT BACKUP_SERVER -``` + # ssh -p JAIL_PORT BACKUP_SERVER -A bash example to be run under the root user's crontab -can be found in the [source repository](https://gitea.evolix.org/evolix/evobackup/src/branch/master/zzz_evobackup) +A +bash(1) +example to be run under the +"root" +user's +crontab(5) +can be found in the +[source repository](https://gitea.evolix.org/evolix/evobackup/src/branch/master/zzz_evobackup) # SEE ALSO -rsync(1), sshd(8), chroot(8). +rsync(1), +ssh-keygen(1), +bkctld(5), +evobackup-incl(5), +chroot(8), +cron(8), +sshd(8) + +# AUTHORS + +Victor Laborie + +OpenBSD 6.4 - December 27, 2018 diff --git a/evobackup-incl.5 b/evobackup-incl.5 new file mode 100644 index 0000000..48f5453 --- /dev/null +++ b/evobackup-incl.5 @@ -0,0 +1,79 @@ +.Dd December 28, 2018 +.Dt EVOBACKUP-INCL 5 +.Os +.Sh NAME +.Nm evobackup-incl +.Nd incremental backup configuration +.Sh SYNOPSIS +.D1 +%Y-%m-%d.-%day +.Sh DESCRIPTION +Located by default in +.Pa /etc/evobackup/ , +each +.Nm +file is named after the +.Xr bkctld 8 +backup for which the rules it contains must apply. +.Pp +The rules it defines decide which incremental backups are kept when running +.Bd -literal -offset indent +# bkctld rm +.Ed +.Pp +Each line defines a single rule. +The first part of the rule describes when the backup was taken, +the second part decides how long to keep it. +Lines beginning with the +.Sq # +character are comments and are ignored. +The order of the rules does not matter. +.Pp +Evobackups that do not have their nominal +.Nm +file use the default rules defined in +.Pa /usr/share/bkctld/inc.tpl +.Sh EXAMPLES +Keep today's backup: +.Bd -literal -offset indent ++%Y-%m-%d.-0day +.Ed +.Pp +Keep yesterday's backup: +.Bd -literal -offset indent ++%Y-%m-%d.-1day +.Ed +.Pp +Keep the first day of this month: +.Bd -literal -offset indent ++%Y-%m-01.-0month +.Ed +.Pp +Keep the first day of last month: +.Bd -literal -offset indent ++%Y-%m-01.-1month +.Ed +.Pp +Keep backups for every 15 days: +.Bd -literal -offset indent ++%Y-%m-01.-1month ++%Y-%m-15.-1month +.Ed +.Pp +Keep a backup of the first day of january: +.Bd -literal -offset indent ++%Y-01-01.-1month +.Ed +.Pp +Keep backups of the last 4 days and the first day of the last 2 months: +.Bd -literal -offset indent ++%Y-%m-%d.-0day ++%Y-%m-%d.-1day ++%Y-%m-%d.-2day ++%Y-%m-%d.-3day ++%Y-%m-01.-0month ++%Y-%m-01.-1month +.Ed +.Sh SEE ALSO +.Xr bkctld 8 , +.Xr cron 8 , +.Pa /etc/evobackup/tpl/inc.tpl \ No newline at end of file From 15fa2dab0f656804d2264c695d81396a64dd6111 Mon Sep 17 00:00:00 2001 From: Patrick Marchand Date: Wed, 2 Jan 2019 12:13:38 -0500 Subject: [PATCH 02/26] Improved documentation for example script Added the description of variables to change into the zzz_evobackup script comments and updated the path to the repository. --- zzz_evobackup | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/zzz_evobackup b/zzz_evobackup index 6ac5a75..5524448 100755 --- a/zzz_evobackup +++ b/zzz_evobackup @@ -1,12 +1,20 @@ #!/bin/sh # # Script Evobackup client -# See https://forge.evolix.org/projects/evobackup +# See https://gitea.evolix.org/evolix/evobackup # # Author: Gregory Colpart # Contributor: Romain Dessort , Benoît Série , Tristan Pilat , Victor Laborie , Jérémy Lecour # Licence: AGPLv3 # +# The following variables must be changed: +# SSH_PORT: The Port used for the ssh(1) jail on the backup server +# MAIL: The email address to send notifications to. +# SRV: The hostname or IP address of the backup server. +# +# You must then uncomment the various +# examples that best suit your case +# PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin From 3af8ac1510dfc467c150aa408e08e307d4eafc35 Mon Sep 17 00:00:00 2001 From: Patrick Marchand Date: Wed, 2 Jan 2019 12:20:41 -0500 Subject: [PATCH 03/26] Improved README.md and adjacent files. * Fixed the english in README.md and docs/debian.md * Moved installation instructions from README.md to docs/install.md * Reworked the rest of README.md * Added a few comments to bkctld.conf --- README.md | 98 ++++++++++++++++++++----------------------------- bkctld.conf | 5 ++- docs/debian.md | 5 ++- docs/install.md | 47 ++++++++++++++++++++++++ 4 files changed, 93 insertions(+), 62 deletions(-) create mode 100644 docs/install.md diff --git a/README.md b/README.md index 5f77184..8e7e9c2 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,18 @@ Bkctld (aka evobackup) ========= -Bkctld is a shell script to create and manage a backup server which will -handle the backup of many servers (clients). Licence is AGPLv3. +Bkctld is a shell script that creates and manages a backup server +which can handle the backups of many other servers (clients). It +is licensed under the AGPLv3. -The main principle uses SSH chroot (called "jails" in the FreeBSD -world) for each client to backup. Each client will upload his data every day -using rsync in his chroot (using root account). -Incrementals are stored outside of the chroot using hard links or btrfs snapshots. -(So incrementals are not available for clients). Using this method we can keep tens -of backup of each client securely and not using too much space. +It uses SSH chroots (called "jails" in the FreeBSD world) to sandbox +every clients backups. Each client will upload it's data every day +using rsync in it's chroot (using the root account). Prior backups +are stored incrementally outside of the chroot using hard links or +BTRFS snapshots. (So they can not be affected by the client). + +Using this method, we can keep a large quantity of backups of each +client securely and efficiently. ~~~ Backup server @@ -20,56 +23,23 @@ Server 2 ------ SSH/rsync -------> * tcp/2223 * ************ ~~~ -This method uses standard tools (ssh, rsync, cp -al, btrfs subvolume). EvoBackup -is used for many years by Evolix for back up each day hundreds of servers which - uses many terabytes of data. +This method uses standard tools (ssh, rsync, cp -al, btrfs subvolume) +and has been used for many years by Evolix to backup hundreds of +servers, totaling many terabytes of data, each day. bkctld has +been tested on Debian Jessie and should be compatible with other +Debian versions or derived distributions like Ubuntu. -bkctld was test on Debian Jessie. It can be compatible with other Debian version -or derivated distribution like Ubuntu or Debian Wheezy. - -A big size volume must be mount on /backup, we recommend usage of **btrfs** for -subvolume and snapshot fonctionnality. -This volume can be encrypted by **luks** for security reason. +A large enough volume must be mounted on `/backup`, we recommend +the usage of **BTRFS** so you can use sub-volumes and snapshots. +This volume can also be encrypted with **LUKS**. ## Install -A Debian package is available in Evolix repository +See the [installation guide](docs/install.md) for instructions. -~~~ -echo "http://pub.evolix.net/ jessie/" >> /etc/apt/sources.list -apt update -apt install bkctld -~~~ +## Testing -### Chroot dependency - -Chroot jail use part of this package - -~~~ -apt install bash coreutils sed dash mount rsync openssh-server openssh-sftp-server libc6-i386 libc6 -~~~ - -#### Install cron for incremental backup - -Edit root crontab - -~~~ -crontab -e -~~~ - -Add this ligne - -~~~ -30 10 * * * /usr/sbin/bkctld inc && /usr/sbin/bkctld rm -~~~ - -> **Notes :** -> If you want mutiples backups in a day (1 by hour maximum) you can run `bkctld inc` multiples times -> If you want keep incremental backup **for ever**, you just need don't run `bkctld rm` - -## Test - -You can deploy tests environmments with Vagrant : +You can deploy test environments with Vagrant : ~~~ vagrant up @@ -77,7 +47,8 @@ vagrant up ### Deployment -Launch rsync-auto in a terminal for automatic synchronisation of your local code with Vagrant VM : +Launch rsync-auto in a terminal for automatic synchronization of +your local code with Vagrant VM : ~~~ vagrant rsync-auto @@ -85,7 +56,8 @@ vagrant rsync-auto ### Bats -You can run [bats](https://github.com/sstephenson/bats) test with *test* provisionner : +You can run [bats](https://github.com/sstephenson/bats) tests with +the *test* provision : ~~~ vagrant provision --provision-with test @@ -95,21 +67,31 @@ vagrant provision --provision-with test See [docs/usage.md](docs/usage.md). -Man page, in roff language, can be generated with pandoc : +The man(1) page, in troff(7) language, can be generated with pandoc: ~~~ -pandoc -f markdown -t man usage.md --template default.man -V title=bkctld -V section=8 -V date="$(date '+%d %b %Y')" -V footer="$(git describe --tags)" -V header="bkctld man page" +pandoc -f markdown \ + -t man usage.md \ + --template default.man \ + -V title=bkctld \ + -V section=8 \ + -V date="$(date '+%d %b %Y')" \ + -V footer="$(git describe --tags)" \ + -V header="bkctld man page" ~~~ #### Client configuration -You can save various systems on evobackup jail : Linux, BSD, Windows, MacOSX. Only prequisites is rsync command. +You can save various systems in the evobackup jails : Linux, BSD, +Windows, MacOSX. The only prerequisite is the rsync command. ~~~ rsync -av -e "ssh -p SSH_PORT" /home/ root@SERVER_NAME:/var/backup/home/ ~~~ -An example script is present in zzz_evobackup, clone evobackup repo and read **CLIENT CONFIGURATION** section of the manual. +An example synchronization script is present in `zzz_evobackup`, +clone the evobackup repository and read the **CLIENT CONFIGURATION** +section of the manual. ~~~ git clone https://forge.evolix.org/evobackup.git diff --git a/bkctld.conf b/bkctld.conf index b876241..ddcb872 100644 --- a/bkctld.conf +++ b/bkctld.conf @@ -1,4 +1,5 @@ -# Defaults for bkctld command (evobackup) +# bkctld.conf(5) +# Defaults for bkctld(8) command (evobackup) # sourced by /usr/sbin/bkctld and /etc/init.d/bkctld #CONFDIR='/etc/evobackup' @@ -10,5 +11,5 @@ #SSHD_PID='/var/run/sshd.pid' #SSHD_CONFIG='/etc/ssh/sshd_config' #AUTHORIZED_KEYS='/root/.ssh/authorized_keys' -#FIREWALL_RULES='/etc/firewall.rc.jails' +#FIREWALL_RULES='' #LOGLEVEL=6 diff --git a/docs/debian.md b/docs/debian.md index ddcc1b3..e99b0cf 100644 --- a/docs/debian.md +++ b/docs/debian.md @@ -1,6 +1,7 @@ # Debian Package -**bkctld** package can be build from the **debian** branch of this Git repository with git-buildpackage and sbuild. +The **bkctld** package can be built from the **debian** branch of +this git repository with git-buildpackage and sbuild. ## Dependencies @@ -37,7 +38,7 @@ sbuild-createchroot --include=eatmydata,ccache,gnupg unstable /srv/chroot/sid ht ## Build -You must be in **debian** branch : +You must be in the **debian** branch : ~~~ git checkout debian diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..22d3aed --- /dev/null +++ b/docs/install.md @@ -0,0 +1,47 @@ +# Install + +A Debian package is available in the Evolix repository + +~~~ +echo "http://pub.evolix.net/jessie/" >> /etc/apt/sources.list +apt update +apt install bkctld +~~~ + +Then edit `/etc//bkctld` + +## Chroot dependencies + +The chroot jails depend on these packages + +~~~ +apt install \ + bash \ + coreutils \ + sed \ + dash \ + mount \ + rsync \ + openssh-server \ + openssh-sftp-server \ + libc6-i386 \ + libc6 +~~~ + +## Client dependencies + +The clients only require OpenSSH and rsync. + +### Cron job for incremental backups + +Edit the root crontab + +~~~ +# crontab -e ++ 30 10 * * * /usr/sbin/bkctld inc && /usr/sbin/bkctld rm +~~~ + +## Notes +If you want mutiples backups in a day (1 by hour maximum) you can +run `bkctld inc` multiples times, if you want to keep incremental +backups **for ever**, just don't run `bkctld rm`. \ No newline at end of file From e960946285465cb1e3ae8f06a11b18d1b1ab4779 Mon Sep 17 00:00:00 2001 From: Victor LABORIE Date: Fri, 4 Jan 2019 13:51:05 +0100 Subject: [PATCH 04/26] Split bkctld into multiples scripts --- Vagrantfile | 1 + bkctld | 685 +++------------------------------------------ lib/bkctld-check | 67 +++++ lib/bkctld-inc | 23 ++ lib/bkctld-init | 29 ++ lib/bkctld-ip | 1 + lib/bkctld-key | 1 + lib/bkctld-params | 17 ++ lib/bkctld-port | 1 + lib/bkctld-reload | 12 + lib/bkctld-remove | 30 ++ lib/bkctld-restart | 11 + lib/bkctld-rm | 44 +++ lib/bkctld-start | 36 +++ lib/bkctld-stats | 23 ++ lib/bkctld-status | 17 ++ lib/bkctld-stop | 16 ++ lib/bkctld-sync | 21 ++ lib/bkctld-update | 11 + lib/config | 23 ++ lib/functions | 133 +++++++++ lib/logging | 42 +++ lib/mkjail | 44 +++ 23 files changed, 648 insertions(+), 640 deletions(-) create mode 100755 lib/bkctld-check create mode 100755 lib/bkctld-inc create mode 100755 lib/bkctld-init create mode 120000 lib/bkctld-ip create mode 120000 lib/bkctld-key create mode 100755 lib/bkctld-params create mode 120000 lib/bkctld-port create mode 100755 lib/bkctld-reload create mode 100755 lib/bkctld-remove create mode 100755 lib/bkctld-restart create mode 100755 lib/bkctld-rm create mode 100755 lib/bkctld-start create mode 100755 lib/bkctld-stats create mode 100755 lib/bkctld-status create mode 100755 lib/bkctld-stop create mode 100755 lib/bkctld-sync create mode 100755 lib/bkctld-update create mode 100755 lib/config create mode 100755 lib/functions create mode 100755 lib/logging create mode 100755 lib/mkjail diff --git a/Vagrantfile b/Vagrantfile index 9fa28e9..66e0cd7 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -16,6 +16,7 @@ Vagrant.configure('2') do |config| $install = <