Compare commits

...

30 commits

Author SHA1 Message Date
Mathieu Trossevin 53ae37fcea
Use correct variable when detecting local sshrc template
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2024-06-03 11:27:39 +02:00
Jérémy Lecour 201b22c145
Client - Release 24.05.1
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
gitea/evobackup/pipeline/tag This commit looks good
2024-05-14 08:19:37 +02:00
Jérémy Lecour 0e98c6d4cd
client: fix shell syntax error 2024-05-14 08:18:02 +02:00
Jérémy Lecour f3cfda48f7
Client - Release 24.05
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
gitea/evobackup/pipeline/tag This commit looks good
2024-05-02 10:22:39 +02:00
Jérémy Lecour dc003ecbca
Add previous release dates in CHANGELOG 2024-05-02 10:20:53 +02:00
Jérémy Lecour 0d32cb2cde
evobackupctl: update LIBDIR when copying the template 2024-05-02 10:18:25 +02:00
Jérémy Lecour 96aa0cf7a2
mention the Ansible role in the client README 2024-05-02 10:17:46 +02:00
Jérémy Lecour be6ebdf415
evobackupctl: simplify the program path retrieval 2024-05-02 10:17:22 +02:00
Jérémy Lecour ebad3bb8d9
README formatting (syntax highlighting)
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
gitea/evobackup/pipeline/tag This commit looks good
2024-04-30 10:02:35 +02:00
Jérémy Lecour 3bcc39088b
README formatting
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2024-04-30 09:59:37 +02:00
Jérémy Lecour ef8c04fe4e
client: release 24.04.1
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2024-04-30 09:57:53 +02:00
Jérémy Lecour 981420ed23
Add a general README and a client README 2024-04-30 09:56:28 +02:00
Jérémy Lecour 8a6db11ca7
zzz_evobackup: improve the documentation 2024-04-30 09:56:27 +02:00
Jérémy Lecour 939ced46c2
evobackupctl: fix args check 2024-04-30 09:56:25 +02:00
Jérémy Lecour 288d61d6b9
client: release 24.04
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2024-04-29 17:59:57 +02:00
Jérémy Lecour 1c44943143
Merge branch 'client-functions' 2024-04-29 17:58:51 +02:00
Jérémy Lecour 859bb800d4 Merge pull request 'fix-tempfiles' (#72) from fix-tempfiles into master
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
Reviewed-on: #72
2024-03-22 11:27:34 +01:00
Brice Waegeneire 21533c6fb9 client: Write temporary files in /tmp
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
By default "mktemp" creates a temporary file in the current working directory,
which in the context of Debian's cron is /. To avoid littering /, we add the
option "--tmpdir" that use $TMPDIR or /tmp instead.
2024-03-22 10:05:10 +01:00
Brice Waegeneire 9510546d48 client: Correctly clean up temporary files
The function "build_rsync_main_cmd" is called in a subshell, so it can't
effectively modify it's parent variable "temp_files". To correctly cleanup
those temporary files, we do it when this specific function exits.
2024-03-22 09:56:03 +01:00
Mathieu Trossevin f1d4e6ed9d
fix(includes): Avoid breaking is_btrfs if path doesn't exists
Some checks failed
gitea/evobackup/pipeline/head There was a failure building this commit
This should return with a err code of 1, not result in the entire
command breaking
2024-02-06 15:03:21 +01:00
Jérémy Lecour 491c839014
Add note for includes/excludes brace expansion
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-12-28 15:29:39 +01:00
William Hirigoyen 4298da250b Add WARNING and CRITICAL values to default conf
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-12-19 10:26:30 +01:00
William Hirigoyen d359883700 server: fix install.md paths
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-12-19 09:45:25 +01:00
Bruno TATU 4cd1554780 Correction url dans README.md
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-10-27 11:30:53 +02:00
William Hirigoyen 981f5118ce Ajout exclusion /var/lib/amavis/virusmails
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-06-15 11:18:49 +02:00
Jérémy Lecour 5dac827bb5 invert dry-run logic
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-05-03 22:08:17 +02:00
William Hirigoyen 4807dfbc99 Fix missing dump.rdb in Redis dump compression, which caused compression fail.
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-04-19 17:46:34 +02:00
William Hirigoyen fd9bb57f8b #71538 : fix Redis dump list in case directory is a symlink
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-04-17 17:13:23 +02:00
Alexis Ben Miloud--Josselin d62455cd1a Compress Redis dump
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-04-04 15:05:30 +02:00
Bruno Tatu 72f5900cf3 On backup les données dans les bases
All checks were successful
gitea/evobackup/pipeline/head This commit looks good
2023-03-30 16:37:36 +02:00
12 changed files with 256 additions and 31 deletions

20
README.md Normal file
View file

@ -0,0 +1,20 @@
EvoBackup
=========
EvoBackup is a combination of tools to manage backups on Evolix servers.
## The client side
_What you install on the servers you want to backup._
There is a backup script (usually executed by cron or similar), a utility script and some libraries.
More information in the [client README](/evolix/evobackup/src/branch/master/client/README.md).
## The server side
_What you install on the servers that store the backups._
This is also known as `bkctld` : a program to manage SSH servers in chroots to isolate backup destinations, daily copies and data retention.
More information in the [server README](/evolix/evobackup/src/branch/master/server/README.md).

View file

@ -13,6 +13,42 @@ The **patch** part changes is incremented if multiple releases happen the same m
### Added
### Changed
### Deprecated
### Removed
### Fixed
### Security
## [24.05.1] - 2022-05-14
### Fixed
* client: fix shell syntax error
## [24.05] - 2022-05-02
### Added
* evobackupctl: update LIBDIR when copying the template
### Changed
* evobackupctl: simplify the program path retrieval
## [24.04.1] - 2022-04-30
### Fixed
* evobackupctl: quote ARGS variable for options parsing.
## [24.04] - 2022-04-29
### Added
* Vagrant definition for manual tests
### Changed
@ -23,8 +59,6 @@ The **patch** part changes is incremented if multiple releases happen the same m
* use env-based shebang for shell scripts
* use $TMPDIR if available
### Deprecated
### Removed
* update-evobackup-canary is managed by ansible-roles.git
@ -34,9 +68,7 @@ The **patch** part changes is incremented if multiple releases happen the same m
* don't exit the whole program if a sync task can't be done
### Security
## [22.12]
## [22.12] - 2022-12-27
### Changed
@ -61,6 +93,6 @@ The **patch** part changes is incremented if multiple releases happen the same m
* Make start_time and stop_time compatible with OpenBSD
## [22.03]
## [22.03] - 2022-04-03
Split client and server parts of the project

166
client/README.md Normal file
View file

@ -0,0 +1,166 @@
EvoBackup — the client side
===========================
_What you install on the servers you want to backup._
## Design
### backup script
The tip of the iceberg is a script (often called `zzz_evobackup` because it is executed by cron at the very end of the _daily_ tasks list).
This is where you setup **what**, **how** and **where** to backup on remote server(s).
There are 2 main phases in the backup :
1. **local tasks**: everything you want to do locally (save state information, dump databases…).
2. **sync tasks**: which data (including what has been prepared in the local phase) you want to send to remote servers.
### libraries
The vast majority of the logic is in libraries, to help maintaining them without having to modify the backup script.
They contain mostly _dump_ functions that are called from the backup script.
Those functions contain a lot of code for logging, options parsing, error management, and some code specific to the dump task.
### utility script
A scripts named `evobackupctl` helps initializing a backup jail on remote servers or install the backup script where you want.
## Install and update
To install, copy these files :
* `lib/*``/usr/local/lib/evobackup`
* `bin/evobackupctl``/usr/local/bin/evobackupctl`
To update, simply overwrite them, since their content should (must?) not be customized locally.
There is also an [evobackup-client](https://gitea.evolix.org/evolix/ansible-roles/src/branch/unstable/evobackup-client) Ansible role to do this (and some other stuff) automatically.
## Usage
### backup script
#### minimal configuration
##### mail notifications
The absolute minimum you must do is set the `MAIL` variable to the email address you want to be notified at.
##### sync tasks
If you want to sync files to a remote server, you have to set the `SERVERS` variable with at least one host and port.
Beware that the default `evolix-system` _sync_ doesn't sync `/home`, `/srv`
If you want to sync files to multiple groups of servers, you can add as many _sync_ sections as you want.
A _sync_ section must contain something like this :
~~~bash
# The name of the "sync" (visible in logs)
SYNC_NAME="evolix-system"
# List of servers
SERVERS=(
host1:port1
host2:port2
)
# List of paths to include in the sync
RSYNC_INCLUDES=(
"${rsync_default_includes[@]}"
/etc
/root
/var
)
# List of paths to exclude from the sync
RSYNC_EXCLUDES=(
"${rsync_default_excludes[@]}"
)
# Actual sync command
sync "${SYNC_NAME}" "SERVERS[@]" "RSYNC_INCLUDES[@]" "RSYNC_EXCLUDES[@]"
~~~
##### local tasks
By default, the `local_tasks()` function only:
* executes [dump-server-state](https://gitea.evolix.org/evolix/dump-server-state) to put have a saved copy of a lot of information about the server
* saves a traceroute to some key network endpoints (using the `dump_traceroute()` function)
You can enable (by uncommenting) as many _dump_ functions as you want.
### advanced customization
Since this is a shell script, you can add any bash-compatible code you want.
If you do so you should read the libraries code to make sure that you don't overwrite existing functions.
##### sync tasks
If you don't want to sync files to any remote servers, you can simply replace the content of the `sync_tasks()` function by a no-op command (`:`).
`RSYNC_INCLUDES` and `RSYNC_EXCLUDES` refer to `${rsync_default_includes[@]}` and `${rsync_default_excludes[@]}` (defined in the `main.sh` library) to simplify the configuration. If you want to precisely customize the lists you can remove them and add you own.
##### local tasks
Existing _dump_ functions (as defined in libraries) are usable as-is, but you can also create your own local custom functions.
You have to define them in the backup script (or in a file that you source from the backup script).
You should prefix their name with `dump_` base your customization on the `dump_custom()` (documented in the backup script) to keep the boilerplate code (for logging, error management…).
You can customize some values inside the `setup_custom()`, like the server's hostname, the notification mail subject…
If you want to source libraries from a different path, you can change the `LIBDIR` variable at the end of the backup script.
### utility tool
The command is `evobackupctl`.
~~~
# evobackupctl --help
evobackupctl helps managing evobackup scripts
Options
-h, --help print this message and exit
-V, --version print version and exit
--jail-init-commands print jail init commands
--copy-template=PATH copy the backup template to PATH
# evobackupctl --version
evobackupctl version 24.04
Copyright 2024 Evolix <info@evolix.fr>,
Jérémy Lecour <jlecour@evolix.fr>.
evobackupctl comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to redistribute it under certain conditions.
See the GNU General Public License v3.0 for details.
~~~
#### jail init commands
It prints a list of commands you can execute on remote backup servers to configure a backup "jail".
~~~
# evobackupctl --jail-init-commands
Copy-paste those lines on backup server(s) :
----------
SERVER_NAME=example-hostname
SERVER_IP=203.0.113.1
echo 'ssh-ed25519 xxxxxx root@example-hostname' > /root/${SERVER_NAME}.pub
bkctld init ${SERVER_NAME}
bkctld key ${SERVER_NAME} /root/${SERVER_NAME}.pub
bkctld ip ${SERVER_NAME} ${SERVER_IP}
bkctld start ${SERVER_NAME}
bkctld status ${SERVER_NAME}
grep --quiet --extended-regexp "^\s?NODE=" /etc/default/bkctld && bkctld sync ${SERVER_NAME}
----------
~~~
#### copy-template
It copies the backups script template to the path of your choice, nothing more.
~~~
# evobackupctl --copy-template /etc/cron.daily/zzz_evobackup
New evobackup script has been saved to '/etc/cron.daily/zzz_evobackup'.
Remember to customize it (mail notifications, backup servers…).
~~~

View file

@ -1,9 +1,8 @@
#!/usr/bin/env bash
# shellcheck disable=SC2155
readonly PROGNAME=$(basename "${0}")
# shellcheck disable=SC2155
readonly PROGDIR=$(readlink -m "$(dirname "${0}")")
readonly PROGPATH=$(readlink -m "${0}")
readonly PROGNAME=$(basename "${PROGPATH}")
# shellcheck disable=SC2124
readonly ARGS=$@
@ -78,10 +77,14 @@ copy_template() {
if cp "${LIBDIR}/zzz_evobackup.sh" "${dest_path}"; then
chmod 750 "${dest_path}"
sed -i "s|@COMMAND@|${PROGDIR}/${PROGNAME} ${ARGS}|" "${dest_path}"
# Insert metadata about the template
sed -i "s|@COMMAND@|${PROGPATH} ${ARGS}|" "${dest_path}"
sed -i "s|@DATE@|$(date --iso-8601=seconds)|" "${dest_path}"
sed -i "s|@VERSION@|${VERSION}|" "${dest_path}"
# Make sure that the library directory is correct
sed -i "s|^LIBDIR=.\+|LIBDIR=\"${LIBDIR}\"|" "${dest_path}"
printf "New evobackup script has been saved to '%s'.\n" "${dest_path}"
printf "Remember to customize it (mail notifications, backup servers…).\n"
exit 0
@ -92,7 +95,7 @@ copy_template() {
main() {
# If no argument is provided, print help and exit
# shellcheck disable=SC2086
if [ -z ${ARGS} ]; then
if [ -z "${ARGS}" ]; then
show_help
exit 0
fi

View file

@ -1,7 +1,7 @@
#!/usr/bin/env bash
# shellcheck disable=SC2034,SC2317
readonly VERSION="24.04-pre1"
readonly VERSION="24.05.1"
# set all programs to C language (english)
export LC_ALL=C

View file

@ -105,7 +105,7 @@ test_server() {
pick_server() {
local -i increment=${1:-0}
local -i list_length=${#SERVERS[@]}
local sync_name=${2:""}
local sync_name=${2:-""}
if (( increment >= list_length )); then
# We've reached the end of the list

View file

@ -274,13 +274,14 @@ local_tasks() {
### # shellcheck disable=SC2174
### mkdir -p -m 700 "${dump_dir}" "${errors_dir}"
###
### # Log the start of the command
### # Log the start of the function
### log "LOCAL_TASKS - ${FUNCNAME[0]}: start ${dump_file}"
###
### # Execute your dump command
### # Send errors to the error file and the data to the dump file
### # Prepare the dump command (errors go to the error file and the data to the dump file)
### dump_cmd="my-dump-command 2> ${error_file} > ${dump_file}"
### log "LOCAL_TASKS - ${FUNCNAME[0]}: ${dump_cmd}"
###
### # Execute the dump command
### ${dump_cmd}
###
### # Check result and deal with potential errors
@ -293,7 +294,7 @@ local_tasks() {
### rm -f "${error_file}"
### fi
###
### # Log the end of the command
### # Log the end of the function
### log "LOCAL_TASKS - ${FUNCNAME[0]}: stop ${dump_file}"
### }

View file

@ -22,6 +22,7 @@ The **patch** part changes is incremented if multiple releases happen the same m
### Fixed
* Test presence of old config file before trying to delete it
* Use correct variable when detecting local sshrc template
### Security

View file

@ -15,3 +15,5 @@
#LOGLEVEL=6
#NODE=''
#ARCHIVESDIR='/backup/archives'
#WARNING=48
#CRITICAL=72

View file

@ -11,17 +11,17 @@ Warning: `cp`-ing the files without `-n` or `-i` will replace existing files !
~~~
# git clone https://gitea.evolix.org/evolix/evobackup.git
# cd evobackup
# cp bkctld /usr/local/sbin/
# cp server/bkctld /usr/local/sbin/
# mkdir -p /usr/local/lib/bkctld
# cp lib/* /usr/local/lib/bkctld/
# cp server/lib/* /usr/local/lib/bkctld/
# mkdir -p /usr/local/share/bkctld
# cp tpl/* /usr/local/share/bkctld/
# cp bkctld.service /lib/systemd/system/
# cp server/tpl/* /usr/local/share/bkctld/
# cp server/bkctld.service /lib/systemd/system/
# mkdir -p /usr/local/share/doc/bkctld
# cp zzz_evobackup /usr/local/share/doc/bkctld/
# cp client/zzz_evobackup /usr/local/share/doc/bkctld/
# mkdir -p /usr/local/share/bash_completion/
# cp bash_completion /usr/local/share/bash_completion/bkctld
# cp bkctld.conf /etc/default/bkctld
# cp server/bash_completion /usr/local/share/bash_completion/bkctld
# cp server/bkctld.conf /etc/default/bkctld
~~~
## Chroot dependencies

View file

@ -51,15 +51,15 @@ if dry_run; then
else
mv "${jail_path}" "${new_jail_path}"
fi
if dry_run; then
if [ -d "${incs_path}" ]; then
if [ -d "${incs_path}" ]; then
if dry_run; then
echo "[dry-run] rename ${incs_path} to ${new_incs_path}"
fi
else
if [ -d "${incs_path}" ]; then
else
mv "${incs_path}" "${new_incs_path}"
fi
fi
if [ -d "${jail_config_dir}" ]; then
if dry_run; then
echo "[dry-run] rename ${jail_config_dir} to ${new_jail_config_dir}"

View file

@ -136,7 +136,7 @@ is_btrfs() {
inode=$(stat --format=%i "${path}")
test $inode -eq 256
test "$inode" -eq 256
}
# Returns the list of jails found in the "jails" directory (default)
@ -294,7 +294,7 @@ setup_jail_chroot() {
[ -f "${LOCALTPLDIR}/passwd" ] && passwd="${LOCALTPLDIR}/passwd"
[ -f "${LOCALTPLDIR}/shadow" ] && shadow="${LOCALTPLDIR}/shadow"
[ -f "${LOCALTPLDIR}/group" ] && group="${LOCALTPLDIR}/group"
[ -f "${LOCALTPLDIR}/sshrc" ] && group="${LOCALTPLDIR}/sshrc"
[ -f "${LOCALTPLDIR}/sshrc" ] && sshrc="${LOCALTPLDIR}/sshrc"
cd "${jail_path}" || error "${jail_name}: failed to change directory to ${jail_path}."
umask 077