Knowledge Base

Updated!

This is part of the story, which has been completely rewritten as part 2.

Backstory

I picked up a mid-sized tower that will become my first offsite backup server! As my local network has matured, I have finally reached the point to establish my offsite backup storage. I had toyed with the idea of setting up a synology device, but I really only need storage; nothing else: no Plex, or docker, or anything of that kind. The name of this system is server2.

I am experimenting with using Devuan Ceres as a server. I could have used CentOS 7 which is still my standard for server OSes, but I can afford some experimentiation here.

Overview

I know I have a few requirements, and I chose my solutions and started the buildout.

• vpn (wireguard)
• RAID 5 (mdadm)
• backup scripts (rsync+sh)
• permissions (ipa user and various rules)

VPN

Install wireguard.

sudo apt-get install wireguard resolvconf

Wireguard already works with resolvconf to control the dns when entering/leaving the vpn, so it is accepted here.

Establish /etc/wireguard/wg0.conf. References include [servername]:/etc/wireguard/wg0.conf and server1:/etc/wireguard/wg0.conf.

[Interface]
Address = 10.222.0.4/24
ListenPort = 51820
# from `wg genkey`
PrivateKey = SCRUBBED
# server2 public key
# SCRUBBED
DNS = 192.168.1.10,192.168.1.11, ipa.internal.com, vm.internal.com, internal.com
[Peer]
# server1
PublicKey = SCRUBBED
AllowedIPs = 192.168.1.10/32, 192.168.1.11/32, 192.168.1.14/32, 192.168.1.18/32, 10.222.0.0/24
PersistentKeepalive = 25
Endpoint = www.example.com:51820

Also I had to add this as a peer on server1!

Set up a custom init script, /etc/init.d/wireguard.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119

#! /bin/sh
# adapted to sh from bash https://gist.github.com/kbabioch/5dd8801e702e519ed18d9b17cacae716
# 2021-11-18

# Copyright (c) 2021 Karol Babioch <karol@babioch.de>

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# LSBInitScript for Wireguard: This is a leightweight init script for
# Wireguard. While Wireguard itself requires only minimal overhead to setup and
# start, it still requires some script invocations (e.g. during boot).
#
# Most distributions are using systemd by now, and as such can use
# wg-quick@.service. However some distributions / images / Linux appliances
# are not (yet) using systemd. In such cases, this init script could be used
# to (re)start and/or stop Wireguard.
#
# It can handle all configured Wireguard interfaces (within /etc/wireguard)
# globally and/or individual interfaces, e.g. (/etc/init.d/wireguard start wg0).
#
# It relies on wg(8) and wg-quick(8) in the background.

### BEGIN INIT INFO
# Provides:          wireguard
# Required-Start:    $network $syslog
# Required-Stop:     $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts Wireguard interfaces
# Description:       Sets up Wireguard interfaces (by means of wg-quick).
### END INIT INFO

CONFIG_DIR=/etc/wireguard

function get_active_wg_interfaces() {
  INTERFACES=$(wg | awk '/interface:/{print $NF}')
  echo "$INTERFACES"
}

# This is required for wg-quick(1) to work correctly, i.e. for process
# substitution (`<()`) to work in Bash. If missing, wg-quick will fail with a
# "fopen: No such file or directory" error.
#[ -e /dev/fd ] || ln -sf /proc/self/fd /dev/fd

case "$1" in

  start)
    if [ -z "$2" ]; then
      echo "Starting all configured Wireguard interfaces"
      for CONFIG in $(cd $CONFIG_DIR; ls *.conf); do
        wg-quick up ${CONFIG%%.conf}
      done
    else
      echo "Starting Wireguard interface: $2"
      wg-quick up "$2"
    fi
    ;;

  stop)
    if [ -z "$2" ]; then
      echo "Stopping all active Wireguard interfaces"
      INTERFACES=$(get_active_wg_interfaces)
      for INTERFACE in $INTERFACES; do
        wg-quick down "$INTERFACE"
      done
    else
      echo "Stopping Wireguard interface: $2"
      wg-quick down "$2"
    fi
    ;;

  reload|force-reload)
    if [ -z "$2" ]; then
      echo "Reloading configuration for all active Wireguard interfaces"
      INTERFACES=$(get_active_wg_interfaces)
      for INTERFACE in $INTERFACES; do
        wg-quick strip "$INTERFACE" | wg syncconf "$INTERFACE"
      done
    else
      echo "Reloading configuration for Wireguard interface: $2"
      wg-quick strip "$2" | wg syncconf "$2"
    fi
    ;;

  restart)
    $0 stop "$2"
    sleep 1
    $0 start "$2"
    ;;

  status)
    # TODO Check exit codes and align them with LSB requirements
    if [ -z "$2" ]; then
      INTERFACES=$(get_active_wg_interfaces)
      for INTERFACE in $INTERFACES; do
        wg show $INTERFACE
      done
    else
      wg show "$2"
    fi
    ;;

  *)
    echo "Usage: $0 { start | stop | restart | reload | force-reload | status } [INTERFACE]"
    exit 1
    ;;

esac

Set it to start with the defaults.

sudo update-rc.d wireguard defaults

Setting up the RAID 5 disk array

I chose to use RAID 5, so the array can handle 1 disk failure and still keep going. I prefer to be able to survive 2 disks failed but I only have 4 disks due to the size of the chassis so this is a compromise I make. See Reference 2

sudo apt-get install mdadm #The following NEW packages will be installed: # bsd-mailx exim4-base exim4-config exim4-daemon-light libgnutls-dane0 libidn12 liblockfile1 libunbound8 mdadm # psmisc

Begin to make the array. This runs the job in the background already!

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sde 2>&1 | tee -a /root/mdadm.create.$( date "+%F" ).log

Check the status with

cat /proc/mdstat

When it is done, make the filesystem. I don't care about btrfs or zfs.

time sudo mkfs.ext4 /dev/md0 sudo mkdir -p /var/server2/ sudo mount -t ext4 -o noatime,nodev -v /dev/md0 /var/server2 sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

ARRAY /dev/md0 metadata=1.2 name=server2:0 UUID=671846fe:2d384a18:fa0c45af:8754ba8c

That last command added the ARRAY line to /etc/mdadm/mdadm.conf.

Add this filesystem to /etc/fstab:

/dev/md0 /var/server2 ext4 auto,rw,noatime,discard,nodev 0 0

And with those lines in those files, now we need to run:

sudo update-initramfs -u

Backup script

The main script is still in progress so check back for future blog posts, but I know I would need a number of permissions.

Permissions

This is what I have modified in my freeipa domain so far. See Reference 3.

Make a service account:

ipa user-add --first=sync --last=user --cn='syncuser' --homedir /home/syncuser --gecos='Service account for syncing data' --email='[myemail]' --password --noprivate --displayname='syncuser' --shell=/bin/bash syncuser --gidnumber=888800013

Establish the hbac rule that allows the user to log in to the two systems.

ipa hbacrule-add --servicecat=all syncuser_rule
ipa hbacrule-add-user --users=syncuser syncuser_rule
ipa hbacrule-add-host --hosts=dns2 syncuser_rule
ipa hbacrule-add-host --hosts=server2 syncuser_rule
ipa hbacrule-mod --desc='Allow syncuser to access the relevant systems for backups' syncuser_rule

As syncuser@dns2:

ssh-keygen
ipa user-mod "${USER}" --sshpubkey="$( cat ~/.ssh/id_rsa.pub )"

Establish sudoers permission in FreeIPA for service account syncuser.

ipa sudocmd-add --desc='rsync full permissions' rsync
ipa sudocmd-add --desc='rsync full permissions' /usr/bin/rsync
ipa sudorule-add syncuser-rsync
ipa sudorule-add-host syncuser-rsync --hosts server2
ipa sudorule-add-host syncuser-rsync --hosts dns2
ipa sudorule-add-allow-command syncuser-rsync --sudocmds rsync
ipa sudorule-add-allow-command syncuser-rsync --sudocmds /usr/bin/rsync
ipa sudorule-add-option syncuser-rsync --sudooption '!authenticate'
ipa sudorule-add-user syncuser-rsync --users syncuser
ipa sudorule-mod syncuser-rsync --desc='syncuser can run rsync on dns2, server2'

References

1. server2a:/etc/installed/server2a.log
2. How to Create a RAID 5 Storage Array with 'mdadm' on Ubuntu 16.04
3. Rsync 'Permission denied' for root - Top 4 reasons and solutions