Automate complete restore of a Restic server backup

Discussions about the usage of KeyHelp.
Post Reply
User avatar
24unix
Posts: 1560
Joined: Sun 21. Jun 2020, 17:16
Location: Kollmar
Contact:

Automate complete restore of a Restic server backup

Post by 24unix »

Mod edit -The thread has been detached. Original thread: viewtopic.php?t=11833

-------


That might be easily automated.

I've got time till the end of november for such an automation.

What would be preferable?
Simple shell script or Ansible Playbook?
mfg Micha
--
If Bill Gates had a nickel for every time Windows crashed …
… oh wait, he does.
PuntonetSvb
Posts: 33
Joined: Thu 7. Apr 2022, 19:16

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by PuntonetSvb »

Ok I'm checking how to do it, in case of hard disk failure or fire in the data center, or any other incident that involves data loss, so far I have three servers with i-mscp that replicate with rsync copies from one to another and two with keyhelp that make complete backups on a local server, but I have seen that the copy is not a file itself, but thousands of files without extensions and with non-descriptive names, so I don't know how it could be done
I think that if it can be automated as I have, for example, in i-mscp, in the event of a server failure in an hour or so I have it working, but here I am totally lost
PuntonetSvb
Posts: 33
Joined: Thu 7. Apr 2022, 19:16

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by PuntonetSvb »

24unix wrote: Mon 21. Nov 2022, 13:59 That might be easily automated.

I've got time till the end of november for such an automation.

What would be preferable?
Simple shell script or Ansible Playbook?
Shell script
User avatar
24unix
Posts: 1560
Joined: Sun 21. Jun 2020, 17:16
Location: Kollmar
Contact:

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by 24unix »

PuntonetSvb wrote: Mon 21. Nov 2022, 19:35 but I have seen that the copy is not a file itself, but thousands of files without extensions and with non-descriptive names, so I don't know how it could be done
You are talking about the restic repository?

You cannot use the raw repository, you must either restore via panel (maybe to different location) or mount the repository, then you see the regular files.
mfg Micha
--
If Bill Gates had a nickel for every time Windows crashed …
… oh wait, he does.
User avatar
24unix
Posts: 1560
Joined: Sun 21. Jun 2020, 17:16
Location: Kollmar
Contact:

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by 24unix »

PuntonetSvb wrote: Mon 21. Nov 2022, 22:23 Shell script
Work in Progress. That's was it already does:
SCR-20221122-pyv.png
It needs to be run as root (Debian) or via sudo (Ubuntu).

Code: Select all

keyhelp-php81 backup.php

TODO:
- KeyHelp-Groups
- TLS Stuff

And:

"To do this, call up "keyhelp-toolbox" on the console -> select point 1 and follow the instructions
of the script."

Is there a way of programmatically triggering that action?

In i-MSCP was a script for that available.

In older threads I found something about "rewrite-user-configs.php", but that doesn't exist on my system.


The current state of the script (most critical calls are commented, as they were tested and will be reenabled after the script is finished:

Code: Select all

<?php

error_reporting(error_level: E_ALL);

const COLOR_RED = "\033[31m";
const COLOR_GREEN = "\033[32m";
const COLOR_YELLOW = "\033[33m";
const COLOR_BLUE = "\033[34m";
const COLOR_WHITE = "\033[37m";
const COLOR_DEFAULT = "\033[39m";

const ETC_PASSWD = '/etc/passwd';
const ETC_SHADOW = '/etc/shadow';
const ETC_GROUP = '/etc/group';
const ETC_GSHADOW = '/etc/gshadow';



echo COLOR_YELLOW . 'Backup/Restore ' . COLOR_BLUE . 'Key' . COLOR_RED . 'Help' . PHP_EOL;
echo COLOR_WHITE . '----------------------' . PHP_EOL . PHP_EOL;
echo COLOR_DEFAULT . 'Preparation:' . PHP_EOL;
echo '1) Install of new system with KeyHelp' . PHP_EOL;
echo '2) Have your backup repository created and scanned in the new KeyHelp-Backup-Management' . PHP_EOL;
echo '3) Backup of the current MySQL user configuration: ';
runCommand(command: 'mysqldump --system=users 2>&1 > users.sql');
echo 'Users saved as ' . COLOR_YELLOW . 'users.sql' . COLOR_DEFAULT . '.' . PHP_EOL;
echo '4) Stop mail services:' . PHP_EOL;
//runCommand('systemctl stop postfix');
echo '   - Postfix is stopped' . PHP_EOL;
//runCommand('systemctl stop dovecot');
echo '   - Dovecot is stopped' . PHP_EOL . PHP_EOL;

echo COLOR_DEFAULT . 'Restore:' . PHP_EOL;
echo '1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore' . PHP_EOL;
$path = readline(prompt: "   Path to restored dir, defaults to /restore: ");
if ($path == '') {
    $path = '/restore';
}
echo '   Using ' . COLOR_YELLOW . $path . COLOR_DEFAULT . ' for restoring.' . PHP_EOL;
echo '   Processing ' . COLOR_YELLOW . $path . ETC_PASSWD . COLOR_DEFAULT . ':' . PHP_EOL;

$etcPasswd = file(filename: $path . ETC_PASSWD);
$etcShadow = file(filename: $path . ETC_SHADOW);
$etcGroup = file(filename: $path . ETC_GROUP);
$etcGShadow = file(filename: $path . ETC_GSHADOW);

$users = preg_grep(pattern: '/home\/users/', array: $etcPasswd);
foreach ($users as $user) {
    [$username, $password, $uid, $gid, $gecos, $home, $shell] = explode(':', $user);
    echo '   Found: ' . COLOR_YELLOW . $username . COLOR_DEFAULT;
    //file_put_contents(filename: ETC_PASSWD, data: $user, flags: FILE_APPEND);
    echo ', modified ' . COLOR_YELLOW . ETC_PASSWD;
    $shadow = preg_grep(pattern: "/^$username:/", array: $etcShadow);
    //file_put_contents(filename: ETC_SHADOW, data: $shadow, flags: FILE_APPEND);
    echo COLOR_DEFAULT . ', ' . COLOR_YELLOW . ETC_SHADOW;
    $group = preg_grep(pattern: "/^$username:/", array: $etcGroup);
    //file_put_contents(filename: ETC_GROUP, data: $group, flags: FILE_APPEND);
    echo COLOR_DEFAULT . ', ' . COLOR_YELLOW . ETC_GROUP;
    $gshadow = preg_grep(pattern: "/^$username:/", array: $etcGShadow);
    //file_put_contents(filename: ETC_GSHADOW, data: $gshadow, flags: FILE_APPEND);
    echo COLOR_DEFAULT . ', ' . COLOR_YELLOW . ETC_GSHADOW . COLOR_DEFAULT . PHP_EOL;
}
// handle KeyHelp groups

/*

Proceed in the same way with the group and gshadow files, with the difference that the keyhelp
groups are also taken over.:

Before appending to the actual system files /etc/group and /etc/gshadow, it is important to remove
the keyhelp groups from these files, as they would otherwise be duplicated.
2) restore of the default databases
Click on „restore backup“ in the new backup management and restore the default databases
keyhelp, mysql, phpmyadmin, rainloop and roundcube
This is the time to re-enter the commands saved in point 3 of the preparation via the MySQL
command line to set the MySQL user passwords again.
!!!Not needed!!!
Complete the entries with „flush privileges;“ or restart the database server.
!!!
3) restore of user data
It may be necessary to create the backup repository again, as a different Keyhelp config is now
active
Furthermore, all necessary PHP versions should now be installed via Configuration → PHP
Interpreter.
In the restore area, select the following elements:
- all email accounts
- all databaeses, except for the ones already restored in point 2!!!!
- all user folders /home/users/ and /var/spool/cron under NO circumstancers !!! select /etc
Restore /root later in another directory if necessary, /home/keyhelp is not needed in normal cases.
Wait until the restore process is complete
4) Restore SSL certificates
Change into the /restore folder and copy the certificates
cd /restore
mv /etc/ssl/keyhelp /etc/ssl/keyhelp.bak
cp -av etc/ssl/keyhelp/ /etc/ssl/

5) final operations
rewrite the user configuration
To do this, call up "keyhelp-toolbox" on the console -> select point 1 and follow the instructions
of the script.
Reboot the server.
Done.

*/

function runCommand(string $command): void
{
    $message = system(command: $command, result_code: $result);

    if ($result !== 0) {
        echo("Error: $result" . PHP_EOL);
        die($message . PHP_EOL);
    }
}
mfg Micha
--
If Bill Gates had a nickel for every time Windows crashed …
… oh wait, he does.
PuntonetSvb
Posts: 33
Joined: Thu 7. Apr 2022, 19:16

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by PuntonetSvb »

I have tried to restore a copy to a server from another and it has not worked for me, as I run the php it gives me errors of files not found in the path, should it work as is? Forgive my ignorance, I'm new to keyhelp, in imscp I cloned bd, web and mail directories and updated ip and that's it.
User avatar
Jolinar
Community Moderator
Posts: 3559
Joined: Sat 30. Jan 2016, 07:11
Location: Weimar (Thüringen)
Contact:

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by Jolinar »

PuntonetSvb wrote: Wed 4. Jan 2023, 17:54 it gives me errors of files not found in the path
Logfile entries?
Error messages?
Give us more information!
Wenn jemand inkompetent ist, dann kann er nicht wissen, daß er inkompetent ist. (David Dunning)

Data Collector für Community Support
___
Ich verwende zwei verschiedene Schriftfarben in meinen Beiträgen /
I use two different font colors in my posts:
  • In dieser Farbe schreibe ich als Moderator und gebe moderative Hinweise oder begründe moderative Eingriffe /
    In this color, I write as a moderator and provide moderative guidance or justify moderative interventions
  • In dieser Farbe schreibe ich als Community Mitglied und teile meine private Meinung und persönlichen Ansichten mit /
    In this color, I write as a community member and share my personal opinions and views
PuntonetSvb
Posts: 33
Joined: Thu 7. Apr 2022, 19:16

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by PuntonetSvb »

root@server:~# php copia.php
Backup/Restore KeyHelp
----------------------

Preparation:
1) Install of new system with KeyHelp
2) Have your backup repository created and scanned in the new KeyHelp-Backup-Management
3) Backup of the current MySQL user configuration: Users saved as users.sql.
4) Stop mail services:
- Postfix is stopped
- Dovecot is stopped

Restore:
1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore
Path to restored dir, defaults to /restore: 1
Using 1 for restoring.
Processing 1/etc/passwd:
PHP Warning: file(1/etc/passwd): Failed to open stream: No such file or directory in /root/copia.php on line 42
PHP Warning: file(1/etc/shadow): Failed to open stream: No such file or directory in /root/copia.php on line 43
PHP Warning: file(1/etc/group): Failed to open stream: No such file or directory in /root/copia.php on line 44
PHP Warning: file(1/etc/gshadow): Failed to open stream: No such file or directory in /root/copia.php on line 45
PHP Fatal error: Uncaught TypeError: preg_grep(): Argument #2 ($array) must be of type array, bool given in /root/copia.php:47
Stack trace:
#0 /root/copia.php(47): preg_grep()
#1 {main}
thrown in /root/copia.php on line 47
root@server:~# ^C
User avatar
24unix
Posts: 1560
Joined: Sun 21. Jun 2020, 17:16
Location: Kollmar
Contact:

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Post by 24unix »

Oh, I never finished this script, as there were no responses.
PuntonetSvb wrote: Mon 30. Jan 2023, 15:47 1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore
Path to restored dir, defaults to /restore: 1
Using 1 for restoring.
Processing 1/etc/passwd:
PHP Warning: file(1/etc/passwd): Failed to open stream: No such file or directory in /root/copia.php on line 42
PHP Warning: file(1/etc/shadow): Failed to open stream: No such file or directory in /root/copia.php on line 43
PHP Warning: file(1/etc/group): Failed to open stream: No such file or directory in /root/copia.php on line 44
PHP Warning: file(1/etc/gshadow): Failed to open stream: No such file or directory in /root/copia.php on line 45
You must enter a path, I guess "1" is neither a path nor does it contain the restored data, right?
mfg Micha
--
If Bill Gates had a nickel for every time Windows crashed …
… oh wait, he does.
PuntonetSvb
Posts: 33
Joined: Thu 7. Apr 2022, 19:16

Re: Automate complete restore of a Restic server backup

Post by PuntonetSvb »

Preparation:
1) Install of new system with KeyHelp
2) Have your backup repository created and scanned in the new KeyHelp-Backup-Management
3) Backup of the current MySQL user configuration: Users saved as users.sql.
4) Stop mail services:
- Postfix is stopped
- Dovecot is stopped

Restore:
1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore
Path to restored dir, defaults to /restore:
Using /restore for restoring.
Processing /restore/etc/passwd:
I make restore dir and other files manually, and now i see it when run script
Post Reply