1. Intro

These are my notes for setting up ReaR bare-metal disaster recovery for my workstation at home.

Hopefully, they can be of use to you too!

 

 

2. Assumptions

This was done on a working installation of Kubuntu 19.10. This is the ReaR and nfs-client.

The Rear software used was ReaR 2.5 git.

The backup location is cyndane2, running CentOS, and is the ReaR and nfs-server.

 

2.1. General NFS-layout

2.1.1. On lanfear

/etc/fstab
192.168.0.8:/home/rear /home/rear nfs defaults,user 0 0

 

2.1.1.1. Folder permissions

/home
drwxrwxrwx 2 nobody nogroup 4096 Feb 1 21:06 rear/

 

2.1.2. On cyndane2

/etc/exports
/home 192.168.0.0/24(rw,fsid=root,no_root_squash,no_subtree_check)

If the exported folders change, run these commands before accessing them from nfs-client.

root@cyndane2:~ # exportfs -a
root@cyndane2:~ # service nfs restart
root@cyndane2:~ # showmount -e
Export list for cyndane2:
/home 192.168.0.0/2

 

2.1.2.1. Folder permissions

/home
drwxrwxrwx 3 root root 4096 Feb 1 22:34 rear

 

 

3. Hardware used

3.1. Lanfear (rear and nfs-client)

  • 1x 128 GB SSD with /boot, swap and /.
  • 3x 1 TB raid0-array with /home on /dev/md0.
  • Bootable Rescue/Recovery USB-stick is a common 4 GB Kingston DataTraveler 102.

 

3.2. Cyndane2 (rear and nfs-server)

  • 1x 128 GB SSD with /boot, swap and /.
  • 2x cages with 4x 1 TB SATA each, total effective storage is 4 TB in a RAID10-array (mirrored RAID0) with /home.

 

 

4. Official documentation

http://relax-and-recover.org/documentation/

 

 

5. Notes

5.1. Installation and prerequisites for ReaR client

Optional good-to-have stauff:
$ sudo apt install dd ethtool file grep gzip ip mount ps sed ssh strings tar lsscsi /
sg3_utils mkisofs syslinux syslinux-extlinux ebiso syslinux parted nfs-client cifs-utils

Required:
$ sudo apt install rear

$ sudo apt install -f

 

 

5.2. Prerequisites for ReaR server

The rear server basically only needs to have a working nfs-share. I won't go into the specifics as to how to set that up.

However, below is a nice guide for setting up both the nfs client and server.

https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-18-04

 

 

5.3. The ReaR client config

# /etc/rear/local.conf
#
# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.

# Everything under here is by Sorin, 2020-01-31.
# Bootloader options on current running system
# Add ReaR boot option to grub, as a backup if the bootable usb-stick fails, or vice versa.
GRUB_RESCUE=1
# Make a note of the password, and please don't save it only on the backed up machine...
# Found to be *deprecated*, so disabled!
#GRUB_RESCUE_PASSWORD="S3cRe7"

# Where to output the rescue media when running rear mkrescue, in this case a bootable USB-stick.
# Note that the /dev/sde may change without further warning!
OUTPUT=USB
USB_DEVICE=/dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1
OUTPUT_URL=usb:///dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1

# Where to store the actual backed up stuff, in this case an existing NFS-mount.
BACKUP=NETFS
BACKUP_URL="nfs://192.168.0.8/home/rear/"

# Exclude this, as it can be restored by other means, or is not needed when doing a bare-metal recovery.
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/home/sorin/Nextcloud/*' '/home/sorin/.cache/*' '/home/sorin/.mozilla/*' /
'/home/sorin/VMs/*' '/home/sorin/Dropbox.SMC/*' '/home/sorin/Downloads/*' '/home/sorin/git/*' /
'/home/sorin/git-KIB/*' '/home/sorin/Nextcloud.musik/*' '/home/sorin/Nextcloud.video/*' '/home/sorin/Video/*' /
'/home/sorin/TMP/*' '/home/sorin/Trash/*' $VAR_DIR/output/* )

 

 

5.4. Common ReaR client commands

Create the rescue media, in this case to a bootable USB-stick.

# /usr/sbin/rear -v mkrescue

 

Backing up lanfear to cyndane2.

# /usr/sbin/rear -v mkbackup

 

Monitor the backup (example). The log may be on the bootable usb-stick, the local backed up machine or on the backup server ("BACKUP_URL").

# tail -f /var/log/rear/rear-lanfear.log

 

Recover the backed up system from the NFS.

# /usr/bin/rear -v recover

 

From from time to time, do check that the disk layout on the machine to be backed up, is identical with the layout on the bootable USB-stick.

# /usr/sbin/rear -v checklayout

Relax-and-Recover 2.5 / Git
Running rear checklayout (PID 9865)
Using log file: /var/log/rear/rear-lanfear.log.lockless
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda with GPT BIOS boot partition)
Verifying that the entries in /tmp/rear.u5wPacDHZowAfSB/tmp/checklayout.conf are correct ...
Disk layout is identical
Saving /var/log/rear/rear-lanfear.log.lockless as /var/log/rear/rear-lanfear.log
Exiting rear checklayout (PID 9865) and its descendant processes ...
Running exit tasks
root@lanfear:/etc/rear#

 

 

6. Problems

Yeah, took me a while to straigthen them out...

  1. If you get any of the below errors, check the paths in /etc/rear/local.config. Above local.conf are now correct.
    # Forgot the "/dev/disk/" before the node, as well as the triple-slash.
    Wrong: OUTPUT_URL=pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0
    Correct: OUTPUT_URL=usb:///dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0
    ERROR: BACKUP_URL or OUTPUT_URL 'usb://usb:///pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0' requires tripple slash (i.e. 'usb:///path')

    # A typo. /dev/disk/by.label/rear instead of /dev/disk/by-label/rear
    Wrong: USB_DEVICE=/dev/disk/by.label/rear
    Correct: USB_DEVICE=/dev/disk/by-label/rear
    ERROR: USB device '/dev/disk/by.label/rear' is not a block device
  2. COPY_AS_IS-messages when running "rear -v mkbackup": To be researched.
    Symlink '/lib/modules/5.3.0-29-generic/build' -> '/usr/src/linux-headers-5.3.0-29-generic' refers to a non-existing directory on the recovery system.
    It will not be copied by default. You can include '/usr/src/linux-headers-5.3.0-29-generic' via the 'COPY_AS_IS' configuration variable.

    Symlink '/usr/share/misc/magic' -> '/usr/share/file/magic' refers to a non-existing directory on the recovery system.
    It will not be copied by default. You can include '/usr/share/file/magic' via the 'COPY_AS_IS' configuration variable.
  3. A possible work-around to keep all the available modules in the backup, is to add the below line in /etc/rear/local.conf. This way the backup can be used as a migration tool, at the expense of a bigger backup-size.
    A further discussion can be found at https://github.com/rear/rear/issues/1202.
    For my part, further investigations are needed. Please be careful when you're testing this!
    As per 2017-05-12, a fix may have been added; https://github.com/rear/rear/issues/1202#issuecomment-301070169
    # Files/dirs to copy as-is (with tar)
    COPY_AS_IS=( "${COPY_AS_IS[@]:-}" /lib/modules )

    # Fix
    Now there is support for some new special MODULES values
    MODULES=( 'all_modules' )
    MODULES=( 'loaded_modules' )
    MODULES=( 'no_modules' )
    This way users can better specify what kernel modules get included in the rescue/recovery system.
    See the explanation in default.conf [/usr/share/rear/conf/default.conf //SS]
  4. A tar-problem after having done a mkbackup.
    ...
    Archived 41943 MiB [avg 12257 KiB/sec] OK
    WARNING: tar ended with return code 1 and below output:
    ---snip---
    tar: program: Warning: Cannot flistxattr: Operation not supported
    tar: gemensam: Warning: Cannot flistxattr: Operation not supported
    tar: hemkat: Warning: Cannot flistxattr: Operation not supported
    ----------
    This means that files have been modified during the archiving
    process. As a result the backup may not be completely consistent
    or may not be a perfect copy of the system. Relax-and-Recover
    will continue, however it is highly advisable to verify the
    backup in order to be sure to safely recover this system.
    Archived 41943 MiB in 3505 seconds [avg 12253 KiB/sec]
    Exiting rear mkbackup (PID 2945) and its descendant processes ...
    Running exit tasks
    Program, gemensam and hemkat are mountpoints, possibly with active mounts.
    This; https://github.com/rear/rear/issues/1175, implies some tar-versions aren't up to snuff with saving attributes. A possible fix is setting the below in local.conf:
    BACKUP_PROG_OPTIONS="$BACKUP_PROG_OPTIONS --xattrs-include='*.*' --xattrs"
  5. ...

 

 

 

7. Tips'n'Tricks

    1. I strongly suggest you test-boot your bootable USB-stick after having run a "rear -v mkrescue".
      When I did, the path to the USB-drive changed from /dev/sde1 to /dev/sdd1, which made the rescue process a no-go, and necessitated a critical edit of the /etc/rear/local.config.
      After the fix, the procedure worked as expected again. Keep an eye on that.

    2. This is how my /etc/default/grub looks. I've edited a few lines to have a better control at boot when I need to boot to ReaR.
      Note the bold lines.
      Also, don't forget to run update-grub, as the first line suggests to activate the changes made - with the ReaR-boot-stick plugged in!
      # If you change this file, run 'update-grub' afterwards to update
      # /boot/grub/grub.cfg.
      # For full documentation of the options in this file, see:
      # info -f grub -n 'Simple configuration'

      GRUB_DEFAULT=0
      GRUB_TIMEOUT_STYLE=countdown
      GRUB_TIMEOUT=10
      GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
      GRUB_CMDLINE_LINUX_DEFAULT="splash"
      GRUB_CMDLINE_LINUX=""

      # Uncomment to enable BadRAM filtering, modify to suit your needs
      # This works with Linux (no patch required) and with any kernel that obtains
      # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
      #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

      # Uncomment to disable graphical terminal (grub-pc only)
      #GRUB_TERMINAL=console

      # The resolution used on graphical terminal
      # note that you can use only modes which your graphic card supports via VBE
      # you can see them in real GRUB with the command `vbeinfo'
      #GRUB_GFXMODE=640x480

      # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
      #GRUB_DISABLE_LINUX_UUID=true

      # Uncomment to disable generation of recovery mode menu entries
      #GRUB_DISABLE_RECOVERY="true"

      # Uncomment to get a beep at grub start
      GRUB_INIT_TUNE="480 440 1"
    3. The USB OUTPUT_URL=usb:///dev/sde1 keeps jumping around. Today it was sdc1. Will test by /dev/disk/by-id instead. The id shouldn't change is my hope, but some research is needed to find a path that doesn't change, like ever...
      USB_DEVICE=/dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1
      OUTPUT_URL=usb:///dev/disk/by-id/usb-Kingston_DataTraveler_102_001CC0EC3466BB20771E00B8-0:0-part1
    4. So far, the by-id path i paragraph 3 seems to have worked, ie it doesn't jump around between reboots.

    5. Ran into some permission problems with the NFS-connection on the client-side. The below articles helped me solve them.
      Accessing NFS share without root privileges
      https://unix.stackexchange.com/questions/451742/acecssing-nfs-share-without-root-privileges

      NFS defaults, what are those? Add "user" to be able to access the nfs-share as a regular user.
      Where are NFS defaults specified?
      https://unix.stackexchange.com/questions/310178/where-are-nfs-defaults-specified


    6. ...

 

 

8. Sources

http://relax-and-recover.org

http://relax-and-recover.org/documentation/

http://relax-and-recover.org/documentation/getting-started

https://github.com/rear/rear/blob/master/doc/user-guide/03-configuration.adoc

https://github.com/rear/rear/blob/master/doc/user-guide/04-scenarios.adoc#booting-from-usb-storage-device

https://github.com/rear/rear/issues/1738

https://www.google.com/search?q=rear+USB_DEVICE&oq=rear+USB_DEVICE&aqs=chrome..69i57.1731j1j7&sourceid=chrome&ie=UTF-8

https://www.google.com/search?q=rear+iso+image&oq=rear+iso+image&aqs=chrome..69i57j0.7200j1j7&sourceid=chrome&ie=UTF-8

https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-18-04

https://unix.stackexchange.com/questions/451742/acecssing-nfs-share-without-root-privileges

https://unix.stackexchange.com/questions/310178/where-are-nfs-defaults-specified

https://github.com/rear/rear/issues/1202

https://github.com/rear/rear/issues/1175