Make zzz_evobackup more automation friendly

Ghost commented

2020-04-22 22:29:14 +02:00

The current script requires too much boilerplate when updating. I'd prefer having a clean file level separation between variables.

zzz_evobackup.cfg

This first file would contain all the top level variables necessary for the next two scripts. It could also contain the include / exclude list for rsync.

zzz_evobackup_operations.sh

This would be empty by default, but would contain any operations necessary to dump application information. This would be used for full backups.

zzz_evobackup.sh

This would be the main script in cron, it would do all initialization and finalization tasks (such as creating directories, syncing and logging) and include the two former scripts. It would also take care of dumping the information necessary for a system only backup.

This separation would mean that you generally only need to change zzz_evobackup.sh when updating. In rarer cases, you would also have to change the variable names in zzz_evobackup.cfg.

I'm not sure I should be spending time on this or if I should spend it on the bkctl client command instead, so I'm laying down my plans here to receive feedback.

The current script requires too much boilerplate when updating. I'd prefer having a clean file level separation between variables. zzz_evobackup.cfg This first file would contain all the top level variables necessary for the next two scripts. It could also contain the include / exclude list for rsync. zzz_evobackup_operations.sh This would be empty by default, but would contain any operations necessary to dump application information. This would be used for full backups. zzz_evobackup.sh This would be the main script in cron, it would do all initialization and finalization tasks (such as creating directories, syncing and logging) and include the two former scripts. It would also take care of dumping the information necessary for a system only backup. This separation would mean that you generally only need to change zzz_evobackup.sh when updating. In rarer cases, you would also have to change the variable names in zzz_evobackup.cfg. I'm not sure I should be spending time on this or if I should spend it on the bkctl client command instead, so I'm laying down my plans here to receive feedback.

Ghost added the

enhancement

question

zzz_evobackup

labels 2020-04-22 22:29:14 +02:00

jlecour was assigned by Ghost

2020-04-22 22:29:14 +02:00

Ghost commented

2020-04-22 22:38:10 +02:00

Author

This is the tool I referenced at the end: https://gitea.evolix.org/evolix/bkctl

jlecour commented

2020-04-22 23:38:51 +02:00

Owner

I wholeheartedly agree that the current evobackup script is a nightmare to upgrade.

The fact that it's a single script has a major benefit : it's simple to read, understand, copy, customize…

… but I think a more modular approach has much more potential.

I think we should have tasks (scripts, config, whatever) :

configuration : servers/ports to rsync to, email to send notifications…
common "system" tasks (list of packages, processes, open ports, firewall rules…) that generate local backups
dumps of various data sources (mysql, postgresql, redis, elasticsearch, ldap…)
remote sync

We should be able to enable/disable tasks, add custom tasks, execute them in any order we like.
We should be able to dump data sources once even if we have many remote sync tasks
We should be able to sync to different groups of servers, each with its own strategy (roundrobin, fallback…)
We shold have core tasks that can easily/safely be updated with Ansible, and custom tasks that are separate.

That said, I still don't have a clear picture in mind about the concrete architecture of this.

I wholeheartedly agree that the current evobackup script is a nightmare to upgrade. The fact that it's a single script has a major benefit : it's simple to read, understand, copy, customize… … but I think a more modular approach has much more potential. I think we should have tasks (scripts, config, whatever) : * configuration : servers/ports to rsync to, email to send notifications… * common "system" tasks (list of packages, processes, open ports, firewall rules…) that generate local backups * dumps of various data sources (mysql, postgresql, redis, elasticsearch, ldap…) * remote sync We should be able to enable/disable tasks, add custom tasks, execute them in any order we like. We should be able to dump data sources once even if we have many remote sync tasks We should be able to sync to different groups of servers, each with its own strategy (roundrobin, fallback…) We shold have core tasks that can easily/safely be updated with Ansible, and custom tasks that are separate. That said, I still don't have a clear picture in mind about the concrete architecture of this.

jlecour commented

2020-04-22 23:59:11 +02:00

Owner

If we accept to go for a big change in how we backup servers, we could also have a look at other existing software.

For example, I've been using BackupNinja for years (until a couple of years ago) on a personal server. It has all we want and more. It even is available as a Debian package. It has the same basic philosophy as ours.

If we accept to go for a big change in how we backup servers, we could also have a look at other existing software. For example, I've been using [BackupNinja](https://0xacab.org/riseuplabs/backupninja) for years (until a couple of years ago) on a personal server. It has all we want and more. It even is available as a Debian package. It has the same basic philosophy as ours.

Ghost commented

2020-04-23 14:35:04 +02:00

Author

BackupNinja looks interesting. Except the last release (and commit) was over a year ago. Is that because it's simple and mostly done or because it's not maintained enough ?

Ghost commented

2020-04-23 14:43:41 +02:00

Author

As for zzz_evobackup, I think the most modular approach would be to a configuration file and a remote syncing script that would dynamically execute all the scripts in a given folder, we could then have one file for each dump generation task.

The disadvantage is thats a lot of files. The advantage is that it would be manageable with ansible templates and a little clearer what tasks are activated.

I havent looked into the advantages of bkctl yet.

As for zzz_evobackup, I think the most modular approach would be to a configuration file and a remote syncing script that would dynamically execute all the scripts in a given folder, we could then have one file for each dump generation task. The disadvantage is thats a lot of files. The advantage is that it would be manageable with ansible templates and a little clearer what tasks are activated. I havent looked into the advantages of bkctl yet.

jlecour commented

2020-04-23 15:19:49 +02:00

Owner

BackupNinja looks interesting. Except the last release (and commit) was over a year ago. Is that because it's simple and mostly done or because it's not maintained enough ?

This piece of software has been maintained for over 9-10 years and has an actively maintained package in Debian. I would worry too much abouth the commit frequency.

As you say, it's been pretty stable and doesn't need a lot of work on a day-to-day basis.

> BackupNinja looks interesting. Except the last release (and commit) was over a year ago. Is that because it's simple and mostly done or because it's not maintained enough ? This piece of software has been maintained for over 9-10 years and has an actively maintained package in Debian. I would worry too much abouth the commit frequency. As you say, it's been pretty stable and doesn't need a lot of work on a day-to-day basis.

Ghost commented

2020-04-23 15:29:17 +02:00

Author

Great ! The main standing issue for me is whether it supports OpenBSD. I'm not seeing a package in the ports tree. But it seems to be bash code, so it should be possible to port.

Would you be willing to setup an internal linux server with BackupNinja so we have a working example ?

Great ! The main standing issue for me is whether it supports OpenBSD. I'm not seeing a package in the ports tree. But it seems to be bash code, so it should be possible to port. Would you be willing to setup an internal linux server with BackupNinja so we have a working example ?

Ghost commented

2020-04-23 15:32:08 +02:00

Author

Ideally, it would be accompanied by a new entry in https://wiki.evolix.org/

jlecour commented

2020-04-23 15:50:14 +02:00

Owner

Great ! The main standing issue for me is whether it supports OpenBSD. I'm not seeing a package in the ports tree. But it seems to be bash code, so it should be possible to port.

I've thought about that and I think it's feasible.

Would you be willing to setup an internal linux server with BackupNinja so we have a working example ?

I'll start with my own personal server and if it's good enough I'll use an internal server.

> Great ! The main standing issue for me is whether it supports OpenBSD. I'm not seeing a package in the ports tree. But it seems to be bash code, so it should be possible to port. I've thought about that and I think it's feasible. > Would you be willing to setup an internal linux server with BackupNinja so we have a working example ? I'll start with my own personal server and if it's good enough I'll use an internal server.

Ghost commented

2020-04-24 15:29:27 +02:00

Author

In the meantime, I will take a closer look at bkctl.

Make zzz_evobackup more automation friendly #40