ReaR notes

Intro

These are my notes for setting up ReaR bare-metal disaster recovery for my workstation at home.

Hopefully, they can be of use to you too!

Assumptions

This was done on a working installation of Kubuntu 19.10. This is the ReaR and nfs-client.

The Rear software used was ReaR 2.5 git.

The backup location is cyndane2, running CentOS, and is the ReaR and nfs-server.

General NFS-layout

On lanfear

/etc/fstab
192.168.0.8:/home/rear /home/rear nfs defaults,user 0 0

Folder permissions

/home
drwxrwxrwx 2 nobody nogroup 4096 Feb 1 21:06 rear/

On cyndane2

/etc/exports
/home 192.168.0.0/24(rw,fsid=root,no_root_squash,no_subtree_check)

If the exported folders change, run these commands before accessing them from nfs-client.

root@cyndane2:~ # exportfs -a
root@cyndane2:~ # service nfs restart
root@cyndane2:~ # showmount -e
Export list for cyndane2:
/home 192.168.0.0/2

Folder permissions

/home
drwxrwxrwx 3 root root 4096 Feb 1 22:34 rear

Hardware used

Lanfear (rear and nfs-client)

1x 128 GB SSD with /boot, swap and /.
3x 1 TB raid0-array with /home on /dev/md0.
Bootable Rescue/Recovery USB-stick is a common 4 GB Kingston DataTraveler 102.

Cyndane2 (rear and nfs-server)

1x 128 GB SSD with /boot, swap and /.
2x cages with 4x 1 TB SATA each, total effective storage is 4 TB in a RAID10-array (mirrored RAID0) with /home.

Official documentation

http://relax-and-recover.org/documentation/

Notes

Installation and prerequisites for ReaR client

Optional good-to-have stauff:
$ sudo apt install dd ethtool file grep gzip ip mount ps sed ssh strings tar lsscsi /
sg3_utils mkisofs syslinux syslinux-extlinux ebiso syslinux parted nfs-client cifs-utils

Required:
$ sudo apt install rear

$ sudo apt install -f

Prerequisites for ReaR server

The rear server basically only needs to have a working nfs-share. I won't go into the specifics as to how to set that up.

However, below is a nice guide for setting up both the nfs client and server.

https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-18-04

The ReaR client config

# /etc/rear/local.conf
#
# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is. 
# Our packages will never ship with a site.conf.

# Everything under here is by Sorin, 2020-01-31.
# Bootloader options on current running system
# Add ReaR boot option to grub, as a backup if the bootable usb-stick fails, or vice versa.
GRUB_RESCUE=1
# Make a note of the password, and please don't save it only on the backed up machine...
# Found to be *deprecated*, so disabled!
#GRUB_RESCUE_PASSWORD="S3cRe7"

# Where to output the rescue media when running rear mkrescue, in this case a bootable USB-stick.
# Note that the /dev/sde may change without further warning!
OUTPUT=USB
USB_DEVICE=/dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1
OUTPUT_URL=usb:///dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1

# Where to store the actual backed up stuff, in this case an existing NFS-mount.
BACKUP=NETFS
BACKUP_URL="nfs://192.168.0.8/home/rear/"

# Exclude this, as it can be restored by other means, or is not needed when doing a bare-metal recovery.
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/home/sorin/Nextcloud/*' '/home/sorin/.cache/*' '/home/sorin/.mozilla/*' /
'/home/sorin/VMs/*' '/home/sorin/Dropbox.SMC/*' '/home/sorin/Downloads/*' '/home/sorin/git/*' /
'/home/sorin/git-KIB/*' '/home/sorin/Nextcloud.musik/*' '/home/sorin/Nextcloud.video/*' '/home/sorin/Video/*' /
'/home/sorin/TMP/*' '/home/sorin/Trash/*' $VAR_DIR/output/* )

Common ReaR client commands

Create the rescue media, in this case to a bootable USB-stick.

# /usr/sbin/rear -v mkrescue

Backing up lanfear to cyndane2.

# /usr/sbin/rear -v mkbackup

Monitor the backup (example). The log may be on the bootable usb-stick, the local backed up machine or on the backup server ("BACKUP_URL").

# tail -f /var/log/rear/rear-lanfear.log

Recover the backed up system from the NFS.

# /usr/bin/rear -v recover

From from time to time, do check that the disk layout on the machine to be backed up, is identical with the layout on the bootable USB-stick.

# /usr/sbin/rear -v checklayout

Relax-and-Recover 2.5 / Git
Running rear checklayout (PID 9865)
Using log file: /var/log/rear/rear-lanfear.log.lockless
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda with GPT BIOS boot partition)
Verifying that the entries in /tmp/rear.u5wPacDHZowAfSB/tmp/checklayout.conf are correct ...
Disk layout is identical
Saving /var/log/rear/rear-lanfear.log.lockless as /var/log/rear/rear-lanfear.log
Exiting rear checklayout (PID 9865) and its descendant processes ...
Running exit tasks
root@lanfear:/etc/rear#

Problems

Yeah, took me a while to straigthen them out...

If you get any of the below errors, check the paths in /etc/rear/local.config. Above local.conf are now correct.

# Forgot the "/dev/disk/" before the node, as well as the triple-slash.
Wrong: OUTPUT_URL=pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0
Correct: OUTPUT_URL=usb:///dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0
ERROR: BACKUP_URL or OUTPUT_URL 'usb://usb:///pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0' requires tripple slash (i.e. 'usb:///path')

# A typo. /dev/disk/by.label/rear instead of /dev/disk/by-label/rear
Wrong: USB_DEVICE=/dev/disk/by.label/rear
Correct: USB_DEVICE=/dev/disk/by-label/rear
ERROR: USB device '/dev/disk/by.label/rear' is not a block device

COPY_AS_IS-messages when running "rear -v mkbackup": To be researched.

Symlink '/lib/modules/5.3.0-29-generic/build' -> '/usr/src/linux-headers-5.3.0-29-generic' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/src/linux-headers-5.3.0-29-generic' via the 'COPY_AS_IS' configuration variable.

Symlink '/usr/share/misc/magic' -> '/usr/share/file/magic' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/share/file/magic' via the 'COPY_AS_IS' configuration variable.

A possible work-around to keep all the available modules in the backup, is to add the below line in /etc/rear/local.conf. This way the backup can be used as a migration tool, at the expense of a bigger backup-size.
A further discussion can be found at https://github.com/rear/rear/issues/1202.
For my part, further investigations are needed. Please be careful when you're testing this!
As per 2017-05-12, a fix may have been added; https://github.com/rear/rear/issues/1202#issuecomment-301070169
```
# Files/dirs to copy as-is (with tar)
COPY_AS_IS=( "${COPY_AS_IS[@]:-}" /lib/modules )

# Fix
Now there is support for some new special MODULES values
MODULES=( 'all_modules' )
MODULES=( 'loaded_modules' )
MODULES=( 'no_modules' )
This way users can better specify what kernel modules get included in the rescue/recovery system.
See the explanation in default.conf [/usr/share/rear/conf/default.conf //SS]
```

A tar-problem after having done a mkbackup.

...
Archived 41943 MiB [avg 12257 KiB/sec] OK
WARNING: tar ended with return code 1 and below output:
 ---snip---
 tar: program: Warning: Cannot flistxattr: Operation not supported
 tar: gemensam: Warning: Cannot flistxattr: Operation not supported
 tar: hemkat: Warning: Cannot flistxattr: Operation not supported
 ----------
This means that files have been modified during the archiving
process. As a result the backup may not be completely consistent
or may not be a perfect copy of the system. Relax-and-Recover
will continue, however it is highly advisable to verify the
backup in order to be sure to safely recover this system.
Archived 41943 MiB in 3505 seconds [avg 12253 KiB/sec]
Exiting rear mkbackup (PID 2945) and its descendant processes ...
Running exit tasks

Program, gemensam and hemkat are mountpoints, possibly with active mounts.
This; https://github.com/rear/rear/issues/1175, implies some tar-versions aren't up to snuff with saving attributes. A possible fix is setting the below in local.conf:

BACKUP_PROG_OPTIONS="$BACKUP_PROG_OPTIONS --xattrs-include='*.*' --xattrs"

...

Tips'n'Tricks

I strongly suggest you test-boot your bootable USB-stick after having run a "rear -v mkrescue".
When I did, the path to the USB-drive changed from /dev/sde1 to /dev/sdd1, which made the rescue process a no-go, and necessitated a critical edit of the /etc/rear/local.config.
After the fix, the procedure worked as expected again. Keep an eye on that.

This is how my /etc/default/grub looks. I've edited a few lines to have a better control at boot when I need to boot to ReaR.
Note the bold lines.
Also, don't forget to run update-grub, as the first line suggests to activate the changes made - with the ReaR-boot-stick plugged in!

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=countdown
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="splash"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
GRUB_INIT_TUNE="480 440 1"

The USB OUTPUT_URL=usb:///dev/sde1 keeps jumping around. Today it was sdc1. Will test by /dev/disk/by-id instead. The id shouldn't change is my hope, but some research is needed to find a path that doesn't change, like ever...
```
USB_DEVICE=/dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1
OUTPUT_URL=usb:///dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1
```
So far, the by-id path i paragraph 3 seems to have worked, ie it doesn't jump around between reboots.
Ran into some permission problems with the NFS-connection on the client-side. The below articles helped me solve them.
Accessing NFS share without root privileges
https://unix.stackexchange.com/questions/451742/acecssing-nfs-share-without-root-privileges

NFS defaults, what are those? Add "user" to be able to access the nfs-share as a regular user.
Where are NFS defaults specified?
https://unix.stackexchange.com/questions/310178/where-are-nfs-defaults-specified
...