Added evobackup-client role #83
|
@ -17,6 +17,7 @@ The **patch** part changes incrementally at each release.
|
|||
* packweb-apache: Deploy opcache.php to give some insights on PHP's opcache status
|
||||
* webapps/evoadmin-web Overload templates if needed
|
||||
* redis: max clients is configurable
|
||||
* evobackup-client role to configure a machine for backups with bkctld(8)
|
||||
|
||||
### Changed
|
||||
* elasticsearch: listen on local interface only by default
|
||||
|
|
23
evobackup-client/README.md
Normal file
|
@ -0,0 +1,23 @@
|
|||
# evobackup-client
|
||||
|
||||
Allows the configuration of backups to one or more remote filesystems.
|
||||
|
||||
|
||||
The backup hosts and the ports in use need to be defined in
|
||||
evobackup-client__hosts before running it.
|
||||
|
||||
jlecour
commented
see my other comment below, the ssh port can be different for each server. see my other comment below, the ssh port can be different for each server.
|
||||
The default zzz_evobackup.sh configures a system only backup, but the
|
||||
template can be overriden to configure a full backup instead. If
|
||||
jlecour
commented
s/system/"system only"/ s/system/"system only"/
Ghost
commented
This is fixed, not sure why it's not marked as outdated. This is fixed, not sure why it's not marked as outdated.
|
||||
you change the variables in defaults/main.yml you can easily run
|
||||
this again and configure backups to a second set of hosts.
|
||||
|
||||
jlecour
commented
see my comment above about the command name and man section. see my comment above about the command name and man section.
|
||||
Do not forget to set the evobackup-client__mail variable to an
|
||||
email adress you control.
|
||||
|
||||
You can add this example to an installation playbook to create the
|
||||
ssh key without running the rest of the role.
|
||||
|
||||
~~~
|
||||
post_tasks:
|
||||
- include_role:
|
||||
name: evobackup-client tasks_from: ssh_key.yml
|
||||
~~~
|
13
evobackup-client/defaults/main.yml
Normal file
|
@ -0,0 +1,13 @@
|
|||
---
|
||||
evobackup_client__root_key_path: "/root/.ssh/evobackup_id"
|
||||
lpoujol
commented
I like the idea, but I'm not sure if we need a separate key for backups only I like the idea, but I'm not sure if we need a separate key for backups only
Ghost
commented
This idea was brought up by @jlecour I can let him expand on it. This idea was brought up by @jlecour I can let him expand on it.
|
||||
evobackup_client__cron_path: "/etc/cron.daily/zzz_evobackup"
|
||||
evobackup_client__cron_template_name: "zzz_evobackup"
|
||||
evobackup_client__mail: null
|
||||
evobackup_client__pid_path: "/var/run/evobackup.pid"
|
||||
evobackup_client__log_path: "/var/log/evobackup.log"
|
||||
evobackup_client__backup_path: "/home/backup"
|
||||
evobackup_client__hosts: null
|
||||
jlecour
commented
each backup server can have a different ssh port. Itr should be in the each backup server can have a different ssh port. Itr should be in the `evobackup-client___hosts` dictionary.
Ghost
commented
I havent met that case in the wild. It would be annoying to have to define the full dict for every host though. Not sure there is a clean way around this. I havent met that case in the wild. It would be annoying to have to define the full dict for every host though. Not sure there is a clean way around this.
|
||||
# - name: "backups.example.org"
|
||||
# ip: "xxx.xxx.xxx.xxx"
|
||||
# fingerprint: "ecdsa-sha2-nistp256 ..."
|
||||
# port: xxxx
|
15
evobackup-client/handlers/main.yml
Normal file
|
@ -0,0 +1,15 @@
|
|||
---
|
||||
- name: restart minifirewall
|
||||
command: /etc/init.d/minifirewall restart
|
||||
register: minifirewall_init_restart
|
||||
failed_when: "'starting IPTables rules is now finish : OK' not in minifirewall_init_restart.stdout"
|
||||
changed_when: "'starting IPTables rules is now finish : OK' in minifirewall_init_restart.stdout"
|
||||
|
||||
- name: 'created new jail'
|
||||
command: "bkctld restart {{ evolinux_hostname }}"
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
|
||||
- name: 'jail updated'
|
||||
command: "bkctld restart {{ evolinux_hostname }}"
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
when: evobackup_client__hosts|length > 1
|
60
evobackup-client/tasks/jail.yml
Normal file
|
@ -0,0 +1,60 @@
|
|||
---
|
||||
|
||||
- name: 'create jail'
|
||||
command: "bkctld init {{ evolinux_hostname }}"
|
||||
args:
|
||||
creates: "/backup/jails/{{ evolinux_hostname }}/"
|
||||
become: true
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
notify:
|
||||
- 'created new jail'
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
||||
|
||||
- name: 'create jail'
|
||||
command: "bkctld init {{ evolinux_hostname }}"
|
||||
args:
|
||||
creates: "/backup/jails/{{ evolinux_hostname }}/"
|
||||
become: true
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
notify:
|
||||
- 'created new jail'
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
||||
|
||||
# temp fix for bkctld 2.x because the ip and key command return 1
|
||||
# if the jail is not started, see https://gitea.evolix.org/evolix/evobackup/issues/31
|
||||
- name: 'start jail'
|
||||
command: "bkctld restart {{ evolinux_hostname }}"
|
||||
become: true
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
||||
|
||||
- name: 'add ip to jail'
|
||||
command: "bkctld ip {{ evolinux_hostname }} {{ ansible_host }}"
|
||||
become: true
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
notify: 'jail updated'
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
||||
|
||||
- name: 'get jail port'
|
||||
command: "bkctld port {{ evolinux_hostname }}"
|
||||
become: true
|
||||
register: bkctld_port
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
||||
|
||||
- name: 'register jail port'
|
||||
set_fact:
|
||||
evobackup_ssh_port={{ bkctld_port.stdout }}
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
26
evobackup-client/tasks/main.yml
Normal file
|
@ -0,0 +1,26 @@
|
|||
---
|
||||
|
||||
- include: "ssh_key.yml"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_ssh_key
|
||||
|
||||
- include: "jail.yml"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_jail
|
||||
|
||||
- include: "upload_scripts.yml"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_scripts
|
||||
|
||||
- include: "open_ssh_ports.yml"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_firewall
|
||||
|
||||
- include: "verify_ssh.yml"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_hosts
|
22
evobackup-client/tasks/open_ssh_ports.yml
Normal file
|
@ -0,0 +1,22 @@
|
|||
---
|
||||
|
||||
- name: Is there a Minifirewall ?
|
||||
stat:
|
||||
path: /etc/default/minifirewall
|
||||
register: evobackup_client__minifirewall
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_firewall
|
||||
|
||||
- name: Add backup SSH port in /etc/default/minifirewall
|
||||
jlecour
commented
We can have multiple backup sections (with heterogenous ssh ports) in the minifirewall file. Let's be extra careful with what we insert and/ore replace. We can have multiple backup sections (with heterogenous ssh ports) in the minifirewall file. Let's be extra careful with what we insert and/ore replace.
Ghost
commented
Again, heterogeneous ports is kind of an outlier case. But the main question is how to define it without it being a chore. Again, heterogeneous ports is kind of an outlier case. But the main question is how to define it without it being a chore.
Ghost
commented
So I removed this task, instead setting the port directly in the rule. So I removed this task, instead setting the port directly in the rule.
|
||||
blockinfile:
|
||||
dest: /etc/default/minifirewall
|
||||
marker: "# {mark} {{ item.name }}"
|
||||
block: |
|
||||
/sbin/iptables -A INPUT -p tcp --sport {{ item.port }} --dport 1024:65535 -s {{ item.ip }} -m state --state ESTABLISHED,RELATED -j ACCEPT
|
||||
with_items: "{{ evobackup_client__hosts }}"
|
||||
notify: restart minifirewall
|
||||
when: evobackup_client__minifirewall.stat.exists
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_firewall
|
31
evobackup-client/tasks/ssh_key.yml
Normal file
|
@ -0,0 +1,31 @@
|
|||
---
|
||||
|
||||
- name: Create SSH key
|
||||
user:
|
||||
name: root
|
||||
generate_ssh_key: true
|
||||
ssh_key_file: "{{ evobackup_client__root_key_path }}"
|
||||
ssh_key_type: rsa
|
||||
lpoujol
commented
I do have a preference for ed25519 over rsa. But this isn't mandatory :) I do have a preference for ed25519 over rsa. But this isn't mandatory :)
Ghost
commented
It's what we used up till now, but I have no problem with changing it. It's what we used up till now, but I have no problem with changing it.
|
||||
register: evobackup_client__root_key
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_ssh_key
|
||||
|
||||
- name: Print SSH key
|
||||
debug:
|
||||
var: evobackup_client__root_key.ssh_public_key
|
||||
when: evobackup_client__root_key.ssh_public_key is defined
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_ssh_key
|
||||
|
||||
- name: 'copy ssh public key to backup server'
|
||||
copy:
|
||||
content: "{{ evobackup_client__root_key.ssh_public_key }}"
|
||||
dest: "/root/{{ evolinux_hostname }}.pub"
|
||||
become: true
|
||||
delegate_to: "{{ evobackup_client__hosts[0].ip }}"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_ssh_key
|
||||
- evobackup_client_jail
|
16
evobackup-client/tasks/upload_scripts.yml
Normal file
|
@ -0,0 +1,16 @@
|
|||
---
|
||||
|
||||
- name: Upload evobackup script
|
||||
template:
|
||||
src: "{{ item }}"
|
||||
dest: "{{ evobackup_client__cron_path }}"
|
||||
force: true
|
||||
mode: 0755
|
||||
with_first_found:
|
||||
- "templates/evobackup-client/{{ evobackup_client__cron_template_name }}.{{ inventory_hostname }}.sh.j2"
|
||||
- "templates/evobackup-client/{{ evobackup_client__cron_template_name }}.{{ host_group }}.sh.j2"
|
||||
- "templates/evobackup-client/{{ evobackup_client__cron_template_name }}.sh.j2"
|
||||
- "zzz_evobackup.default.sh.j2"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_scripts
|
11
evobackup-client/tasks/verify_ssh.yml
Normal file
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
|
||||
- name: Verify evolix backup servers
|
||||
known_hosts:
|
||||
path: /root/.ssh/known_hosts
|
||||
name: "[{{ item.name }}]:{{ item.port }}"
|
||||
key: "[{{ item.name }}]:{{ item.port }} {{ item.fingerprint }}"
|
||||
with_list: "{{ evobackup_client__hosts }}"
|
||||
tags:
|
||||
- evobackup_client
|
||||
- evobackup_client_backup_hosts
|
365
evobackup-client/templates/zzz_evobackup.default.sh.j2
Normal file
|
@ -0,0 +1,365 @@
|
|||
#!/bin/sh
|
||||
#
|
||||
# Script Evobackup client
|
||||
# See https://gitea.evolix.org/evolix/evobackup
|
||||
#
|
||||
# Author: Gregory Colpart <reg@evolix.fr>
|
||||
# Contributors:
|
||||
# Romain Dessort <rdessort@evolix.fr>
|
||||
# Benoît Série <bserie@evolix.fr>
|
||||
# Tristan Pilat <tpilat@evolix.fr>
|
||||
# Victor Laborie <vlaborie@evolix.fr>
|
||||
# Jérémy Lecour <jlecour@evolix.fr>
|
||||
#
|
||||
# Licence: AGPLv3
|
||||
#
|
||||
# /!\ DON'T FORGET TO SET "MAIL" and "SERVERS" VARIABLES
|
||||
|
||||
##### Configuration ###################################################
|
||||
|
||||
# email adress for notifications
|
||||
MAIL={{ evobackup_client__mail }}
|
||||
|
||||
jlecour
commented
This probably is a bad copy/paste. This probably is a bad copy/paste.
Ghost
commented
Damnit, thought I had erased that. Damnit, thought I had erased that.
|
||||
# list of hosts (hostname or IP) and SSH port for Rsync
|
||||
SERVERS="{% for host in evobackup_client__hosts %}{{ host.name }}:{{ host.port }}{% if loop.index != loop.length %} {% endif %}{% endfor %}"
|
||||
jlecour
commented
i'm not sure this loop produces the intended output. i'm not sure this loop produces the intended output.
Ghost
commented
It produces:
The extra space at the end is not the cleanest, but it does not break the script. It produces:
```
SERVERS="hostname:port hostname:port "
```
The extra space at the end is not the cleanest, but it does not break the script.
Ghost
commented
But this is kind of reason why it's a pull request and not a straight merge. But this is kind of reason why it's a pull request and not a straight merge.
Ghost
commented
This should not be marked as outdated, this conversation is still open. This should not be marked as outdated, this conversation is still open.
|
||||
|
||||
# timeout (in seconds) for SSH connections
|
||||
SSH_CONNECT_TIMEOUT=30
|
||||
|
||||
## We use /home/backup : feel free to use your own dir
|
||||
LOCAL_BACKUP_DIR="{{ evobackup_client__backup_path }}"
|
||||
|
||||
# You can set "linux" or "bsd" manually or let it choose automatically
|
||||
SYSTEM=$(uname | tr '[:upper:]' '[:lower:]')
|
||||
|
||||
# Change these 2 variables if you have more than one backup cron
|
||||
PIDFILE="{{ evobackup_client__pid_path }}"
|
||||
LOGFILE="{{ evobackup_client__log_path }}"
|
||||
|
||||
## Enable/Disable tasks
|
||||
LOCAL_TASKS=${LOCAL_TASKS:-1}
|
||||
SYNC_TASKS=${SYNC_TASKS:-1}
|
||||
|
||||
##### SETUP AND FUNCTIONS #############################################
|
||||
|
||||
BEGINNING=$(/bin/date +"%d-%m-%Y ; %H:%M")
|
||||
|
||||
# shellcheck disable=SC2174
|
||||
mkdir -p -m 700 ${LOCAL_BACKUP_DIR}
|
||||
|
||||
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin
|
||||
|
||||
## lang = C for english outputs
|
||||
export LANGUAGE=C
|
||||
export LANG=C
|
||||
|
||||
## Force umask
|
||||
umask 077
|
||||
|
||||
## Initialize variable to store SSH connection errors
|
||||
SERVERS_SSH_ERRORS=""
|
||||
|
||||
# Call test_server with "HOST:PORT" string
|
||||
# It will return with 0 if the server is reachable.
|
||||
# It will return with 1 and a message on stderr if not.
|
||||
test_server() {
|
||||
item=$1
|
||||
# split HOST and PORT from the input string
|
||||
host=$(echo "${item}" | cut -d':' -f1)
|
||||
port=$(echo "${item}" | cut -d':' -f2)
|
||||
|
||||
# Test if the server is accepting connections
|
||||
ssh -q -o "ConnectTimeout ${SSH_CONNECT_TIMEOUT}" -i /root/.ssh/evobackup_id "${host}" -p "${port}" -t "exit"
|
||||
lpoujol
commented
While you allow to chose a different name for the SSH Key with evobackup_client__root_key_path, it's static here. While you allow to chose a different name for the SSH Key with evobackup_client__root_key_path, it's static here.
Ghost
commented
In practice, I end up overwriting this template almost all the time. But you're right. In practice, I end up overwriting this template almost all the time. But you're right.
|
||||
# shellcheck disable=SC2181
|
||||
if [ $? = 0 ]; then
|
||||
# SSH connection is OK
|
||||
return 0
|
||||
else
|
||||
# SSH connection failed
|
||||
new_error=$(printf "Failed to connect to \`%s' within %s seconds" "${item}" "${SSH_CONNECT_TIMEOUT}")
|
||||
SERVERS_SSH_ERRORS=$(printf "%s\n%s" "${SERVERS_SSH_ERRORS}" "${new_error}" | sed -e '/^$/d')
|
||||
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
# Call pick_server with an optional positive integer to get the nth server in the list.
|
||||
pick_server() {
|
||||
increment=${1:-0}
|
||||
list_length=$(echo "${SERVERS}" | wc -w)
|
||||
|
||||
if [ "${increment}" -ge "${list_length}" ]; then
|
||||
# We've reached the end of the list
|
||||
new_error="No more server available"
|
||||
SERVERS_SSH_ERRORS=$(printf "%s\n%s" "${SERVERS_SSH_ERRORS}" "${new_error}" | sed -e '/^$/d')
|
||||
|
||||
printf "%s\n" "${SERVERS_SSH_ERRORS}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Extract the day of month, without leading 0 (which would give an octal based number)
|
||||
today=$(date +%e)
|
||||
# A salt is useful to randomize the starting point in the list
|
||||
# but stay identical each time it's called for a server (based on hostname).
|
||||
salt=$(hostname | cksum | cut -d' ' -f1)
|
||||
# Pick an integer between 0 and the length of the SERVERS list
|
||||
# It changes each day
|
||||
item=$(( (today + salt + increment) % list_length ))
|
||||
# cut starts counting fields at 1, not 0.
|
||||
field=$(( item + 1 ))
|
||||
|
||||
echo "${SERVERS}" | cut -d' ' -f${field}
|
||||
}
|
||||
|
||||
## Verify other evobackup process and kill if needed
|
||||
if [ -e "${PIDFILE}" ]; then
|
||||
pid=$(cat "${PIDFILE}")
|
||||
# Killing the childs of evobackup.
|
||||
for ppid in $(ps h --ppid "${pid}" -o pid | tr -s '\n' ' '); do
|
||||
kill -9 "${ppid}";
|
||||
done
|
||||
# Then kill the main PID.
|
||||
kill -9 "${pid}"
|
||||
printf "%s is still running (PID %s). Process has been killed" "$0" "${pid}\n" >&2
|
||||
fi
|
||||
echo "$$" > ${PIDFILE}
|
||||
# shellcheck disable=SC2064
|
||||
trap "rm -f ${PIDFILE}" EXIT
|
||||
|
||||
##### LOCAL BACKUP ####################################################
|
||||
|
||||
if [ "${LOCAL_TASKS}" = "1" ]; then
|
||||
# You can comment or uncomment sections below to customize the backup
|
||||
|
||||
## OpenLDAP : example with slapcat
|
||||
# slapcat -l ${LOCAL_BACKUP_DIR}/ldap.bak
|
||||
|
||||
### MySQL
|
||||
|
||||
## example with global and compressed mysqldump
|
||||
# mysqldump --defaults-extra-file=/etc/mysql/debian.cnf -P 3306 \
|
||||
# --opt --all-databases --force --events --hex-blob | gzip --best > ${LOCAL_BACKUP_DIR}/mysql.bak.gz
|
||||
|
||||
## example with two dumps for each table (.sql/.txt) for all databases
|
||||
# for i in $(echo SHOW DATABASES | mysql --defaults-extra-file=/etc/mysql/debian.cnf -P 3306 \
|
||||
# | egrep -v "^(Database|information_schema|performance_schema|sys)" ); \
|
||||
# do mkdir -p -m 700 /home/mysqldump/$i ; chown -RL mysql /home/mysqldump ; \
|
||||
# mysqldump --defaults-extra-file=/etc/mysql/debian.cnf --force -P 3306 -Q --opt --events --hex-blob --skip-comments \
|
||||
# --fields-enclosed-by='\"' --fields-terminated-by=',' -T /home/mysqldump/$i $i; done
|
||||
|
||||
## example with compressed SQL dump for each databases
|
||||
# mkdir -p -m 700 /home/mysqldump/
|
||||
# for i in $(mysql --defaults-extra-file=/etc/mysql/debian.cnf -P 3306 -e 'show databases' -s --skip-column-names \
|
||||
# | egrep -v "^(Database|information_schema|performance_schema|sys)"); do
|
||||
# mysqldump --defaults-extra-file=/etc/mysql/debian.cnf --force -P 3306 --events --hex-blob $i | gzip --best > /home/mysqldump/${i}.sql.gz
|
||||
# done
|
||||
|
||||
## example with *one* uncompressed SQL dump for *one* database (MYBASE)
|
||||
# mkdir -p -m 700 /home/mysqldump/MYBASE
|
||||
# chown -RL mysql /home/mysqldump/
|
||||
# mysqldump --defaults-extra-file=/etc/mysql/debian.cnf --force -Q \
|
||||
# --opt --events --hex-blob --skip-comments -T /home/mysqldump/MYBASE MYBASE
|
||||
|
||||
## example with mysqlhotcopy
|
||||
# mkdir -p -m 700 /home/mysqlhotcopy/
|
||||
# mysqlhotcopy BASE /home/mysqlhotcopy/
|
||||
|
||||
## example for multiples MySQL instances
|
||||
# mysqladminpasswd=$(grep -m1 'password = .*' /root/.my.cnf|cut -d" " -f3)
|
||||
# grep -E "^port\s*=\s*\d*" /etc/mysql/my.cnf |while read instance; do
|
||||
# instance=$(echo "$instance"|awk '{ print $3 }')
|
||||
# if [ "$instance" != "3306" ]
|
||||
# then
|
||||
# mysqldump -P $instance --opt --all-databases --hex-blob -u mysqladmin -p$mysqladminpasswd > ${LOCAL_BACKUP_DIR}/mysql.$instance.bak
|
||||
# fi
|
||||
# done
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
## example with pg_dumpall (warning: you need space in ~postgres)
|
||||
# su - postgres -c "pg_dumpall > ~/pg.dump.bak"
|
||||
# mv ~postgres/pg.dump.bak ${LOCAL_BACKUP_DIR}/
|
||||
## another method with gzip directly piped
|
||||
# cd /var/lib/postgresql
|
||||
# sudo -u postgres pg_dumpall | gzip > ${LOCAL_BACKUP_DIR}/pg.dump.bak.gz
|
||||
# cd - > /dev/null
|
||||
|
||||
## example with all tables from MYBASE excepts TABLE1 and TABLE2
|
||||
# pg_dump -p 5432 -h 127.0.0.1 -U USER --clean -F t --inserts -f ${LOCAL_BACKUP_DIR}/pg-backup.tar -t 'TABLE1' -t 'TABLE2' MYBASE
|
||||
|
||||
## example with only TABLE1 and TABLE2 from MYBASE
|
||||
# pg_dump -p 5432 -h 127.0.0.1 -U USER --clean -F t --inserts -f ${LOCAL_BACKUP_DIR}/pg-backup.tar -T 'TABLE1' -T 'TABLE2' MYBASE
|
||||
|
||||
## MongoDB : example with mongodump
|
||||
## don't forget to create use with read-only access
|
||||
## > use admin
|
||||
## > db.createUser( { user: "mongobackup", pwd: "PASS", roles: [ "backup", ] } )
|
||||
# test -d ${LOCAL_BACKUP_DIR}/mongodump/ && rm -rf ${LOCAL_BACKUP_DIR}/mongodump/
|
||||
# mkdir -p -m 700 ${LOCAL_BACKUP_DIR}/mongodump/
|
||||
# mongodump --quiet -u mongobackup -pPASS -o ${LOCAL_BACKUP_DIR}/mongodump/
|
||||
# if [ $? -ne 0 ]; then
|
||||
# echo "Error with mongodump!"
|
||||
# fi
|
||||
|
||||
## Redis : example with copy .rdb file
|
||||
# cp /var/lib/redis/dump.rdb ${LOCAL_BACKUP_DIR}/
|
||||
|
||||
## ElasticSearch, take a snapshot as a backup.
|
||||
## Warning: You need to have a path.repo configured.
|
||||
## See: https://wiki.evolix.org/HowtoElasticsearch#snapshots-et-sauvegardes
|
||||
# curl -s -XDELETE "localhost:9200/_snapshot/snaprepo/snapshot.daily" -o /tmp/es_delete_snapshot.daily.log
|
||||
# curl -s -XPUT "localhost:9200/_snapshot/snaprepo/snapshot.daily?wait_for_completion=true" -o /tmp/es_snapshot.daily.log
|
||||
## Clustered version here
|
||||
## It basically the same thing except that you need to check that NFS is mounted
|
||||
# if ss | grep ':nfs' | grep -q 'ip\.add\.res\.s1' && ss | grep ':nfs' | grep -q 'ip\.add\.res\.s2'
|
||||
# then
|
||||
# curl -s -XDELETE "localhost:9200/_snapshot/snaprepo/snapshot.daily" -o /tmp/es_delete_snapshot.daily.log
|
||||
# curl -s -XPUT "localhost:9200/_snapshot/snaprepo/snapshot.daily?wait_for_completion=true" -o /tmp/es_snapshot.daily.log
|
||||
# else
|
||||
# echo 'Cannot make a snapshot of elasticsearch, at least one node is not mounting the repository.'
|
||||
# fi
|
||||
## If you need to keep older snapshot, for example the last 10 daily snapshots, replace the XDELETE and XPUT lines by :
|
||||
# for snapshot in $(curl -s -XGET "localhost:9200/_snapshot/snaprepo/_all?pretty=true" | grep -Eo 'snapshot_[0-9]{4}-[0-9]{2}-[0-9]{2}' | head -n -10); do
|
||||
# curl -s -XDELETE "localhost:9200/_snapshot/snaprepo/${snapshot}" | grep -v -Fx '{"acknowledged":true}'
|
||||
# done
|
||||
# date=$(date +%F)
|
||||
# curl -s -XPUT "localhost:9200/_snapshot/snaprepo/snapshot_${date}?wait_for_completion=true" -o /tmp/es_snapshot_${date}.log
|
||||
|
||||
## RabbitMQ : export config
|
||||
#rabbitmqadmin export ${LOCAL_BACKUP_DIR}/rabbitmq.config >> $LOGFILE
|
||||
|
||||
# backup MegaCli config
|
||||
#megacli -CfgSave -f ${LOCAL_BACKUP_DIR}/megacli_conf.dump -a0 >/dev/null
|
||||
|
||||
## Dump system and kernel versions
|
||||
uname -a > ${LOCAL_BACKUP_DIR}/uname
|
||||
|
||||
## Dump network routes with mtr and traceroute (warning: could be long with aggressive firewalls)
|
||||
for addr in 8.8.8.8 www.evolix.fr travaux.evolix.net; do
|
||||
mtr -r ${addr} > ${LOCAL_BACKUP_DIR}/mtr-${addr}
|
||||
traceroute -n ${addr} > ${LOCAL_BACKUP_DIR}/traceroute-${addr} 2>&1
|
||||
done
|
||||
|
||||
## Dump process with ps
|
||||
ps auwwx >${LOCAL_BACKUP_DIR}/ps.out
|
||||
|
||||
if [ "${SYSTEM}" = "linux" ]; then
|
||||
## Dump network connections with ss
|
||||
ss -taupen > ${LOCAL_BACKUP_DIR}/netstat.out
|
||||
|
||||
## List Debian packages
|
||||
dpkg -l > ${LOCAL_BACKUP_DIR}/packages
|
||||
dpkg --get-selections > ${LOCAL_BACKUP_DIR}/packages.getselections
|
||||
apt-cache dumpavail > ${LOCAL_BACKUP_DIR}/packages.available
|
||||
|
||||
## Dump MBR / table partitions
|
||||
disks=$(find /dev/ -regex '/dev/\([sv]d[a-z]\|nvme[0-9]+n[0-9]+\)')
|
||||
for disk in ${disks}; do
|
||||
name=$(basename "${disk}")
|
||||
dd if="${disk}" of="${LOCAL_BACKUP_DIR}/MBR-${name}" bs=512 count=1 2>&1 | egrep -v "(records in|records out|512 bytes)"
|
||||
fdisk -l "${disk}" > "${LOCAL_BACKUP_DIR}/partitions-${name}"
|
||||
done
|
||||
cat ${LOCAL_BACKUP_DIR}/partitions-* > ${LOCAL_BACKUP_DIR}/partitions
|
||||
else
|
||||
## Dump network connections with netstat
|
||||
netstat -finet -atn > ${LOCAL_BACKUP_DIR}/netstat.out
|
||||
|
||||
## List OpenBSD packages
|
||||
pkg_info -m > ${LOCAL_BACKUP_DIR}/packages
|
||||
|
||||
## Dump MBR / table partitions
|
||||
##disklabel sd0 > ${LOCAL_BACKUP_DIR}/partitions
|
||||
fi
|
||||
|
||||
fi
|
||||
|
||||
##### REMOTE BACKUP ###################################################
|
||||
|
||||
n=0
|
||||
server=""
|
||||
while :; do
|
||||
server=$(pick_server "${n}")
|
||||
test $? = 0 || exit 2
|
||||
|
||||
if test_server "${server}"; then
|
||||
break
|
||||
else
|
||||
server=""
|
||||
n=$(( n + 1 ))
|
||||
fi
|
||||
done
|
||||
|
||||
SSH_SERVER=$(echo "${server}" | cut -d':' -f1)
|
||||
SSH_PORT=$(echo "${server}" | cut -d':' -f2)
|
||||
|
||||
HOSTNAME=$(hostname)
|
||||
|
||||
if [ "${SYSTEM}" = "linux" ]; then
|
||||
rep="/bin /boot /lib /opt /sbin /usr"
|
||||
else
|
||||
rep="/bsd /bin /sbin /usr"
|
||||
fi
|
||||
|
||||
|
||||
if [ "${SYNC_TASKS}" = "1" ]; then
|
||||
# /!\ DO NOT USE COMMENTS in the rsync command /!\
|
||||
# It breaks the command and destroys data, simply remove (or add) lines.
|
||||
|
||||
# Remote shell command
|
||||
RSH_COMMAND="ssh -i /root/.ssh/evobackup_id -p ${SSH_PORT} -o 'ConnectTimeout ${SSH_CONNECT_TIMEOUT}'"
|
||||
lpoujol
commented
Same issue here, you're not using evobackup_client__root_key_path Same issue here, you're not using evobackup_client__root_key_path
Ghost
commented
Will fix Will fix
|
||||
|
||||
rsync -avzh --stats --delete --delete-excluded --force --ignore-errors --partial \
|
||||
--exclude "lost+found" \
|
||||
--exclude ".nfs.*" \
|
||||
--exclude "/var/log" \
|
||||
--exclude "/var/log/evobackup*" \
|
||||
--exclude "/var/lib/mysql" \
|
||||
--exclude "/var/lib/postgres" \
|
||||
--exclude "/var/lib/postgresql" \
|
||||
--exclude "/var/lib/sympa" \
|
||||
--exclude "/var/lib/metche" \
|
||||
--exclude "/var/run" \
|
||||
--exclude "/var/lock" \
|
||||
--exclude "/var/state" \
|
||||
--exclude "/var/apt" \
|
||||
--exclude "/var/cache" \
|
||||
--exclude "/usr/src" \
|
||||
--exclude "/usr/doc" \
|
||||
--exclude "/usr/share/doc" \
|
||||
--exclude "/usr/obj" \
|
||||
--exclude "dev" \
|
||||
--exclude "/var/spool/postfix" \
|
||||
--exclude "/var/lib/amavis/amavisd.sock" \
|
||||
--exclude "/var/lib/munin/*tmp*" \
|
||||
--exclude "/var/lib/php5" \
|
||||
--exclude "/var/spool/squid" \
|
||||
--exclude "/var/lib/elasticsearch" \
|
||||
--exclude "/var/lib/amavis/tmp" \
|
||||
--exclude "/var/lib/clamav/*.tmp" \
|
||||
--exclude "/home/mysqltmp" \
|
||||
--exclude "/var/lib/php/sessions" \
|
||||
${rep} \
|
||||
/etc \
|
||||
/root \
|
||||
/var \
|
||||
/home \
|
||||
jlecour
commented
Why /home/backup and not /home like it is in the official script https://gitea.evolix.org/evolix/evobackup/src/branch/master/zzz_evobackup#L344 ? Why /home/backup and not /home like it is in the official script https://gitea.evolix.org/evolix/evobackup/src/branch/master/zzz_evobackup#L344 ?
Ghost
commented
The way I've been using it mostly is for system backups not full backups, hence the '/home/backup'. If you're doing a complete backup, it's almost always going to be more custom, so I feel like the more common use case is more useful. But I dont have strong opinions about this. The way I've been using it mostly is for system backups not full backups, hence the '/home/backup'. If you're doing a complete backup, it's almost always going to be more custom, so I feel like the more common use case is more useful. But I dont have strong opinions about this.
Ghost
commented
This would be the main outstanding question. I feel like /home/backup is a better default, but I can change it. What do you think ? This would be the main outstanding question. I feel like /home/backup is a better default, but I can change it. What do you think ?
jlecour
commented
This reveals a blindspot in our setup. When we do a "system only" backup, we definitely want to keep the files with the list of packages, mtr output… They currently go to But when we also backup the whole server to another backup server and there are databases dumps in Maybe we should put lightweight important files to backup to This reveals a blindspot in our setup.
When we do a "system only" backup, we definitely want to keep the files with the list of packages, mtr output… They currently go to `/home/backup` by default.
But when we also backup the whole server to another backup server and there are databases dumps in `/home/backup` then thos files are also added to the "system-only" backup, which is not good.
Maybe we should put lightweight important files to backup to `/root/backup` and large data dumps to `/home/backup`. Then we would always add `/root/backup` to the `rsync` command, and only add `/home` to full backups.
Ghost
commented
That makes sense. We could then add a boolean switch to enable "full" backups from ansible. Could this cause any problems with the typical free space on /root ? That makes sense. We could then add a boolean switch to enable "full" backups from ansible.
Could this cause any problems with the typical free space on /root ?
|
||||
/srv \
|
||||
-e "${RSH_COMMAND}" \
|
||||
"root@${SSH_SERVER}:/var/backup/" \
|
||||
| tail -30 >> $LOGFILE
|
||||
fi
|
||||
|
||||
##### REPORTING #######################################################
|
||||
|
||||
END=$(/bin/date +"%d-%m-%Y ; %H:%M")
|
||||
|
||||
printf "EvoBackup - %s - START %s ON %s (LOCAL_TASKS=%s SYNC_TASKS=%s)\n" \
|
||||
"${HOSTNAME}" "${BEGINNING}" "${SSH_SERVER}" "${LOCAL_TASKS}" "${SYNC_TASKS}" \
|
||||
>> $LOGFILE
|
||||
|
||||
printf "EvoBackup - %s - STOP %s ON %s (LOCAL_TASKS=%s SYNC_TASKS=%s)\n" \
|
||||
"${HOSTNAME}" "${END}" "${SSH_SERVER}" "${LOCAL_TASKS}" "${SYNC_TASKS}" \
|
||||
>> $LOGFILE
|
||||
|
||||
tail -10 $LOGFILE | \
|
||||
mail -s "[info] EvoBackup - Client ${HOSTNAME}" \
|
||||
${MAIL}
|
s/a pair of bkctld(8) hosts/one or more backup servers/
The server side has nothing to do with bkctld as a command, so no need to confuse people with the name of the command and a man section. We can backup to anything as long as it is accessible via SSH.