Page 1 of 1

Automate complete restore of a Restic server backup

Posted: Mon 21. Nov 2022, 13:59
by 24unix
Mod edit -The thread has been detached. Original thread: viewtopic.php?t=11833

-------


That might be easily automated.

I've got time till the end of november for such an automation.

What would be preferable?
Simple shell script or Ansible Playbook?

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Mon 21. Nov 2022, 19:35
by PuntonetSvb
Ok I'm checking how to do it, in case of hard disk failure or fire in the data center, or any other incident that involves data loss, so far I have three servers with i-mscp that replicate with rsync copies from one to another and two with keyhelp that make complete backups on a local server, but I have seen that the copy is not a file itself, but thousands of files without extensions and with non-descriptive names, so I don't know how it could be done
I think that if it can be automated as I have, for example, in i-mscp, in the event of a server failure in an hour or so I have it working, but here I am totally lost

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Mon 21. Nov 2022, 22:23
by PuntonetSvb
24unix wrote: Mon 21. Nov 2022, 13:59 That might be easily automated.

I've got time till the end of november for such an automation.

What would be preferable?
Simple shell script or Ansible Playbook?
Shell script

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Tue 22. Nov 2022, 14:34
by 24unix
PuntonetSvb wrote: Mon 21. Nov 2022, 19:35 but I have seen that the copy is not a file itself, but thousands of files without extensions and with non-descriptive names, so I don't know how it could be done
You are talking about the restic repository?

You cannot use the raw repository, you must either restore via panel (maybe to different location) or mount the repository, then you see the regular files.

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Tue 22. Nov 2022, 18:47
by 24unix
PuntonetSvb wrote: Mon 21. Nov 2022, 22:23 Shell script
Work in Progress. That's was it already does:
SCR-20221122-pyv.png
It needs to be run as root (Debian) or via sudo (Ubuntu).

Code: Select all

keyhelp-php81 backup.php

TODO:
- KeyHelp-Groups
- TLS Stuff

And:

"To do this, call up "keyhelp-toolbox" on the console -> select point 1 and follow the instructions
of the script."

Is there a way of programmatically triggering that action?

In i-MSCP was a script for that available.

In older threads I found something about "rewrite-user-configs.php", but that doesn't exist on my system.


The current state of the script (most critical calls are commented, as they were tested and will be reenabled after the script is finished:

Code: Select all

<?php

error_reporting(error_level: E_ALL);

const COLOR_RED = "\033[31m";
const COLOR_GREEN = "\033[32m";
const COLOR_YELLOW = "\033[33m";
const COLOR_BLUE = "\033[34m";
const COLOR_WHITE = "\033[37m";
const COLOR_DEFAULT = "\033[39m";

const ETC_PASSWD = '/etc/passwd';
const ETC_SHADOW = '/etc/shadow';
const ETC_GROUP = '/etc/group';
const ETC_GSHADOW = '/etc/gshadow';



echo COLOR_YELLOW . 'Backup/Restore ' . COLOR_BLUE . 'Key' . COLOR_RED . 'Help' . PHP_EOL;
echo COLOR_WHITE . '----------------------' . PHP_EOL . PHP_EOL;
echo COLOR_DEFAULT . 'Preparation:' . PHP_EOL;
echo '1) Install of new system with KeyHelp' . PHP_EOL;
echo '2) Have your backup repository created and scanned in the new KeyHelp-Backup-Management' . PHP_EOL;
echo '3) Backup of the current MySQL user configuration: ';
runCommand(command: 'mysqldump --system=users 2>&1 > users.sql');
echo 'Users saved as ' . COLOR_YELLOW . 'users.sql' . COLOR_DEFAULT . '.' . PHP_EOL;
echo '4) Stop mail services:' . PHP_EOL;
//runCommand('systemctl stop postfix');
echo '   - Postfix is stopped' . PHP_EOL;
//runCommand('systemctl stop dovecot');
echo '   - Dovecot is stopped' . PHP_EOL . PHP_EOL;

echo COLOR_DEFAULT . 'Restore:' . PHP_EOL;
echo '1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore' . PHP_EOL;
$path = readline(prompt: "   Path to restored dir, defaults to /restore: ");
if ($path == '') {
    $path = '/restore';
}
echo '   Using ' . COLOR_YELLOW . $path . COLOR_DEFAULT . ' for restoring.' . PHP_EOL;
echo '   Processing ' . COLOR_YELLOW . $path . ETC_PASSWD . COLOR_DEFAULT . ':' . PHP_EOL;

$etcPasswd = file(filename: $path . ETC_PASSWD);
$etcShadow = file(filename: $path . ETC_SHADOW);
$etcGroup = file(filename: $path . ETC_GROUP);
$etcGShadow = file(filename: $path . ETC_GSHADOW);

$users = preg_grep(pattern: '/home\/users/', array: $etcPasswd);
foreach ($users as $user) {
    [$username, $password, $uid, $gid, $gecos, $home, $shell] = explode(':', $user);
    echo '   Found: ' . COLOR_YELLOW . $username . COLOR_DEFAULT;
    //file_put_contents(filename: ETC_PASSWD, data: $user, flags: FILE_APPEND);
    echo ', modified ' . COLOR_YELLOW . ETC_PASSWD;
    $shadow = preg_grep(pattern: "/^$username:/", array: $etcShadow);
    //file_put_contents(filename: ETC_SHADOW, data: $shadow, flags: FILE_APPEND);
    echo COLOR_DEFAULT . ', ' . COLOR_YELLOW . ETC_SHADOW;
    $group = preg_grep(pattern: "/^$username:/", array: $etcGroup);
    //file_put_contents(filename: ETC_GROUP, data: $group, flags: FILE_APPEND);
    echo COLOR_DEFAULT . ', ' . COLOR_YELLOW . ETC_GROUP;
    $gshadow = preg_grep(pattern: "/^$username:/", array: $etcGShadow);
    //file_put_contents(filename: ETC_GSHADOW, data: $gshadow, flags: FILE_APPEND);
    echo COLOR_DEFAULT . ', ' . COLOR_YELLOW . ETC_GSHADOW . COLOR_DEFAULT . PHP_EOL;
}
// handle KeyHelp groups

/*

Proceed in the same way with the group and gshadow files, with the difference that the keyhelp
groups are also taken over.:

Before appending to the actual system files /etc/group and /etc/gshadow, it is important to remove
the keyhelp groups from these files, as they would otherwise be duplicated.
2) restore of the default databases
Click on „restore backup“ in the new backup management and restore the default databases
keyhelp, mysql, phpmyadmin, rainloop and roundcube
This is the time to re-enter the commands saved in point 3 of the preparation via the MySQL
command line to set the MySQL user passwords again.
!!!Not needed!!!
Complete the entries with „flush privileges;“ or restart the database server.
!!!
3) restore of user data
It may be necessary to create the backup repository again, as a different Keyhelp config is now
active
Furthermore, all necessary PHP versions should now be installed via Configuration → PHP
Interpreter.
In the restore area, select the following elements:
- all email accounts
- all databaeses, except for the ones already restored in point 2!!!!
- all user folders /home/users/ and /var/spool/cron under NO circumstancers !!! select /etc
Restore /root later in another directory if necessary, /home/keyhelp is not needed in normal cases.
Wait until the restore process is complete
4) Restore SSL certificates
Change into the /restore folder and copy the certificates
cd /restore
mv /etc/ssl/keyhelp /etc/ssl/keyhelp.bak
cp -av etc/ssl/keyhelp/ /etc/ssl/

5) final operations
rewrite the user configuration
To do this, call up "keyhelp-toolbox" on the console -> select point 1 and follow the instructions
of the script.
Reboot the server.
Done.

*/

function runCommand(string $command): void
{
    $message = system(command: $command, result_code: $result);

    if ($result !== 0) {
        echo("Error: $result" . PHP_EOL);
        die($message . PHP_EOL);
    }
}

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Wed 4. Jan 2023, 17:54
by PuntonetSvb
I have tried to restore a copy to a server from another and it has not worked for me, as I run the php it gives me errors of files not found in the path, should it work as is? Forgive my ignorance, I'm new to keyhelp, in imscp I cloned bd, web and mail directories and updated ip and that's it.

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Wed 4. Jan 2023, 17:59
by Jolinar
PuntonetSvb wrote: Wed 4. Jan 2023, 17:54 it gives me errors of files not found in the path
Logfile entries?
Error messages?
Give us more information!

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Mon 30. Jan 2023, 15:47
by PuntonetSvb
root@server:~# php copia.php
Backup/Restore KeyHelp
----------------------

Preparation:
1) Install of new system with KeyHelp
2) Have your backup repository created and scanned in the new KeyHelp-Backup-Management
3) Backup of the current MySQL user configuration: Users saved as users.sql.
4) Stop mail services:
- Postfix is stopped
- Dovecot is stopped

Restore:
1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore
Path to restored dir, defaults to /restore: 1
Using 1 for restoring.
Processing 1/etc/passwd:
PHP Warning: file(1/etc/passwd): Failed to open stream: No such file or directory in /root/copia.php on line 42
PHP Warning: file(1/etc/shadow): Failed to open stream: No such file or directory in /root/copia.php on line 43
PHP Warning: file(1/etc/group): Failed to open stream: No such file or directory in /root/copia.php on line 44
PHP Warning: file(1/etc/gshadow): Failed to open stream: No such file or directory in /root/copia.php on line 45
PHP Fatal error: Uncaught TypeError: preg_grep(): Argument #2 ($array) must be of type array, bool given in /root/copia.php:47
Stack trace:
#0 /root/copia.php(47): preg_grep()
#1 {main}
thrown in /root/copia.php on line 47
root@server:~# ^C

Re: HOWTO - Instructions for the complete restore of a Restic server backup

Posted: Mon 30. Jan 2023, 16:04
by 24unix
Oh, I never finished this script, as there were no responses.
PuntonetSvb wrote: Mon 30. Jan 2023, 15:47 1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore
Path to restored dir, defaults to /restore: 1
Using 1 for restoring.
Processing 1/etc/passwd:
PHP Warning: file(1/etc/passwd): Failed to open stream: No such file or directory in /root/copia.php on line 42
PHP Warning: file(1/etc/shadow): Failed to open stream: No such file or directory in /root/copia.php on line 43
PHP Warning: file(1/etc/group): Failed to open stream: No such file or directory in /root/copia.php on line 44
PHP Warning: file(1/etc/gshadow): Failed to open stream: No such file or directory in /root/copia.php on line 45
You must enter a path, I guess "1" is neither a path nor does it contain the restored data, right?

Re: Automate complete restore of a Restic server backup

Posted: Mon 30. Jan 2023, 19:34
by PuntonetSvb
Preparation:
1) Install of new system with KeyHelp
2) Have your backup repository created and scanned in the new KeyHelp-Backup-Management
3) Backup of the current MySQL user configuration: Users saved as users.sql.
4) Stop mail services:
- Postfix is stopped
- Dovecot is stopped

Restore:
1) Now restore the /etc folder from restic to a new destinaion, e.g. /restore
Path to restored dir, defaults to /restore:
Using /restore for restoring.
Processing /restore/etc/passwd:
I make restore dir and other files manually, and now i see it when run script