Preparing my offsite backup server, part 2

bgstack15

2021-11-30 08:49

This post is an almost-complete story, and supersedes the previous post.

Overview

System server2 is a Dell PowerEdge T30 workstation with 4 3.5" 6TB WD Red Pro hard disks. It serves one primary goal, offsite backup of internal data. The remote location is [SCRUBBED]. The remote network where server2 will reside uses dhcp, so the default Debian dhcp client configuration is used.

The key design criteria are listed in the table below.

Item	Specific technology chosen
VPN	wireguard
Redundant storage	RAID 5 on WD Red Pro 6TB disks (one disk failure tolerated)
Firewall	nftables, without firewalld
Front-door net tunnel	autossh
Backup process	shell script and cron job

Related systems that will be modified to work with server2 include:

server1, bastion host for front-door entry to internal network
dns2, system where main offsite-sync.sh script will run
FreeIPA domain overall will get new config items

Initial preparation

Install low-level requirements.

apt-get install sudo screen vim curl gpg

Run the various scripts from http://www.example.com/internal/Support/Platforms/devuan/scripts/

apt-get --no-install-recommends install bgconf bgscripts-core sudo apt-get install python3-ipalib=4.9.7-1+devuan1 freeipa-client=4.9.7-1+devuan1 freeipa-helper systemctl-service-shim nfs-common cifs-utils mlocate parted rsync sssd-tools

time sudo bgconf.py -d10 2>&1 | sudo tee -a ~root/bgconf.$( date "+%F" ).log

/mnt/public/Support/Platforms/devuan/scripts/ipa-client-install.sh

sudo apt-get -V autopurge adwaita-icon-theme dbus-x11 util-linux-locales

Additional steps are described in /mnt/public/Support/Systems/server2/server2-plan.md

Main installation and preparation

Preparing the domain

The FreeIPA domain that is already used for access control will be modified.

Add a new user.

ipa user-add --first=sync --last=user --cn='syncuser' --homedir /home/syncuser --gecos='Service account for syncing data' --email='bgstack15+sync@gmail.com' --password --noprivate --displayname='syncuser' --shell=/bin/bash syncuser --gidnumber=960600013

The password is stored in my main keepass file. The chosen gid is group service-accounts which does not have login access to production systems.

Make a new hbac rule allow this user to log into server2.

ipa hbacrule-add --servicecat=all syncuser_rule
ipa hbacrule-add-user --users=syncuser syncuser_rule
ipa hbacrule-add-host --hosts=dns2 syncuser_rule
ipa hbacrule-add-host --hosts=server2 syncuser_rule
ipa hbacrule-mod --desc='Allow syncuser to access the relevant systems for backups' syncuser_rule

As syncuser@dns2, make a new ssh public key and add it to the ipa user.

# as user syncuser@dns2
ssh-keygen
ipa user-mod "${USER}" --sshpubkey="$( cat ~/.ssh/id_rsa.pub )"

Establish rules for the service account to log in the relevant systems. Reference: Weblink 3

Establish sudoers permission in ipa for syncuser:

ipa sudocmd-add --desc='rsync full permissions' rsync
ipa sudocmd-add --desc='rsync full permissions' /usr/bin/rsync
ipa sudorule-add syncuser-rsync
ipa sudorule-add-host syncuser-rsync --hosts server2
ipa sudorule-add-host syncuser-rsync --hosts dns2
ipa sudorule-add-allow-command syncuser-rsync --sudocmds rsync
ipa sudorule-add-allow-command syncuser-rsync --sudocmds /usr/bin/rsync
ipa sudorule-add-option syncuser-rsync --sudooption '!authenticate'
ipa sudorule-add-user syncuser-rsync --users syncuser
ipa sudorule-mod syncuser-rsync --desc='syncuser can run rsync on dns2, server2'

After ensuring the sssd cache was updated:

$ sudo -l -U syncuser
Matching Defaults entries for syncuser on server2:
   env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin,
   env_keep+="ftp_proxy http_proxy https_proxy no_prxy", env_keep+="DEBUG DEBUG_LEVEL DRYRUN VERBOSE"

User syncuser may run the following commands on server2:
   (root) NOPASSWD: /usr/bin/rsync, rsync

Installing wireguard

I chose wireguard because I already use it and adding a new peer is very easy.

sudo apt-get install wireguard resolvconf

Wireguard already works with resolvconf to control the dns when entering/leaving the vpn, so it is accepted here.

Establish /etc/wireguard/wg0.conf. References include internal document 1 and internal document 2.

[Interface] Address = 10.222.0.4/24 ListenPort = 51820 # from wg genkey PrivateKey = SCRUBBED # server2 public key # xozGLE4M5ncwGp4SpanAQGn1J6wMYv9JfGW4nS0e8UA= DNS = 192.168.1.10,192.168.1.11, ipa.internal.com, vm.internal.com, internal.com [Peer] # server1 PublicKey = KOQVWNY3+TMzkMrCTsG7DJm29wQGovEv1LfLrptfAjw= AllowedIPs = 192.168.1.10/32, 192.168.1.11/32, 192.168.1.14/32, 192.168.1.18/32, 10.222.0.0/24 PersistentKeepalive = 25 Endpoint = www.example.com:51820

Also had to add this as a peer on server1! That configuration is straightforward and not described here.

Set up a custom init script, /etc/init.d/wireguard.

#! /bin/sh
# Adapted from https://gist.github.com/kbabioch/5dd8801e702e519ed18d9b17cacae716
# 2021-11-18

# Copyright (c) 2021 Karol Babioch <karol@babioch.de>

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# LSBInitScript for Wireguard: This is a leightweight init script for
# Wireguard. While Wireguard itself requires only minimal overhead to setup and
# start, it still requires some script invocations (e.g. during boot).
#
# Most distributions are using systemd by now, and as such can use
# wg-quick@.service. However some distributions / images / Linux appliances
# are not (yet) using systemd. In such cases, this init script could be used
# to (re)start and/or stop Wireguard.
#
# It can handle all configured Wireguard interfaces (within /etc/wireguard)
# globally and/or individual interfaces, e.g. (/etc/init.d/wireguard start wg0).
#
# It relies on wg(8) and wg-quick(8) in the background.

### BEGIN INIT INFO
# Provides:          wireguard
# Required-Start:    $network $syslog
# Required-Stop:     $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts Wireguard interfaces
# Description:       Sets up Wireguard interfaces (by means of wg-quick).
### END INIT INFO

CONFIG_DIR=/etc/wireguard

function get_active_wg_interfaces() {
  INTERFACES=$(wg | awk '/interface:/{print $NF}')
  echo "$INTERFACES"
}

# This is required for wg-quick(1) to work correctly, i.e. for process
# substitution (`<()`) to work in Bash. If missing, wg-quick will fail with a
# "fopen: No such file or directory" error.
#[ -e /dev/fd ] || ln -sf /proc/self/fd /dev/fd

case "$1" in

  start)
    if [ -z "$2" ]; then
      echo "Starting all configured Wireguard interfaces"
      for CONFIG in $(cd $CONFIG_DIR; ls *.conf); do
        wg-quick up ${CONFIG%%.conf}
      done
    else
      echo "Starting Wireguard interface: $2"
      wg-quick up "$2"
    fi
    ;;

  stop)
    if [ -z "$2" ]; then
      echo "Stopping all active Wireguard interfaces"
      INTERFACES=$(get_active_wg_interfaces)
      for INTERFACE in $INTERFACES; do
        wg-quick down "$INTERFACE"
      done
    else
      echo "Stopping Wireguard interface: $2"
      wg-quick down "$2"
    fi
    ;;

  reload|force-reload)
    if [ -z "$2" ]; then
      echo "Reloading configuration for all active Wireguard interfaces"
      INTERFACES=$(get_active_wg_interfaces)
      for INTERFACE in $INTERFACES; do
        wg-quick strip "$INTERFACE" | wg syncconf "$INTERFACE" 
      done
    else
      echo "Reloading configuration for Wireguard interface: $2"
      wg-quick strip "$2" | wg syncconf "$2" 
    fi
    ;;

  restart)
    $0 stop "$2"
    sleep 1
    $0 start "$2"
    ;;

  status)
    # TODO Check exit codes and align them with LSB requirements
    if [ -z "$2" ]; then
      INTERFACES=$(get_active_wg_interfaces)
      for INTERFACE in $INTERFACES; do
        wg show $INTERFACE
      done
    else
      wg show "$2"
    fi
    ;;

  *)
    echo "Usage: $0 { start | stop | restart | reload | force-reload | status } [INTERFACE]"
    exit 1
    ;;

esac

Set wireguard to start with the defaults.

sudo update-rc.d wireguard defaults

Setting up the disk array

I chose to use RAID 5, so the array can handle 1 disk failure and still keep going. I prefer to be able to survive 2 disks failed but I only have 4 disks due to the size of the chassis so this is a compromise I make.

Reference: Weblink 2

sudo apt-get install mdadm
#The following NEW packages will be installed:
#  bsd-mailx exim4-base exim4-config exim4-daemon-light libgnutls-dane0 libidn12 liblockfile1 libunbound8 mdadm
#  psmisc

Begin to make the array. This runs the job in the background already!

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sde 2>&1 | tee -a /root/mdadm.create.$( date "+%F" ).log

Check the status with

cat /proc/mdstat

It is done now, so:

time sudo mkfs.ext4 /dev/md0
sudo mkdir -p /var/server2/
sudo mount -t ext4 -o noatime,nodev -v /dev/md0 /var/server2
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf 
> ARRAY /dev/md0 metadata=1.2 name=server2:0 UUID=671746fe:2d387a18:fa0c46af:8744bb7c

That last command added the ARRAY line to /etc/mdadm/mdadm.conf.

Add this filesystem to /etc/fstab:

/dev/md0       /var/server2     ext4     auto,rw,noatime,discard,nodev   0 0

And with those lines in those files, now we need to run:

sudo update-initramfs -u

Establish firewall

Reference: weblink 4

sudo apt-get install nftables sudo cp -p /usr/share/doc/nftables/examples/sysvinit/nftables.init /etc/init.d/nftables sudo chmod +x /etc/init.d/nftables sudo update-rc.d nftables defaults

The config file is /etc/nftables.conf. I set it initially with:

#!/usr/sbin/nft -f
# Startdate: 2021-11-23
# File: server2:/etc/nftables.conf
# Reference:
#    /usr/share/doc/nftables/examples/workstation.nft
# Documentation: /etc/installed/server2a.md
flush ruleset
table inet filter {
   chain input {
      type filter hook input priority 0;
      # accept any localhost traffic
      iif lo accept
      # accept traffic that originated from this system
      # accept traffic originated from us
      ct state established,related accept
      # this {} array is comma-separated
      tcp dport { 22 } ct state new accept
      # count and drop any other traffic
      counter drop
   }
   chain forward {
      type filter hook forward priority 0;
   }
   chain output {
      type filter hook output priority 0;
   }
}

Establish autossh

The remote server will also be configured to connect to my network through the front door, so to speak. This is another way to connect to the system in case of VPN failure.

In FreeIPA, allow syncuser to ssh in to the bastion host.

ipa hbacrule-add syncuser_ssh
ipa hbacrule-add-user --users=syncuser syncuser_ssh
ipa hbacrule-add-host --hosts=server1 syncuser_ssh
ipa hbacrule-mod --desc='Allow syncuser to ssh to server1 for autossh tunnel' syncuser_ssh

Generate on server2 an ssh key for syncuser.

ssh-keygen

Add this new key to the ipa user, without deleting the old ssh key. See my recent post about that.

eval ipa user-mod ${USER} $( ipa user-show ${USER} --all | awk '/SSH public key:/{$1="";$2="";$3="";print}' | sed -r -e 's/ *, */\n/g;' -e 's/^\s*//g;' | while read line ; do printf '%s ' "--sshpubkey='${line}'" ; done ; ) --sshpubkey="'$( cat ~/.ssh/id_rsa.pub )'"

Modify server1, the bastion host, to allow local port bindings on non-loopback IP addresses, and also open the firewall.

# on server1
sudo firewall-cmd --add-port=2201/tcp --permanent
sudo firewall-cmd --reload
sudo /usr/libexec/bgscripts/py/modconf.py -a -c '#' -l ' ' /etc/ssh/sshd_config set GatewayPorts yes
sudo service sshd restart

And now, the ssh command to connect to the front dynamic dns hostname of my main site is:

sshk -R:2201:localhost:22 -p2022 syncuser@www.example.com

When this tunnel is running, a user on the Internal network can run this command to connect to server2:

ssh -p2201 server1

Establish /etc/init.d/autossh.

#!/bin/bash
# Adapted from https://github.com/obfusk/autossh-init/blob/master/autossh.init
### BEGIN INIT INFO
# Provides:          autossh
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: autossh initscript
# Description:       Starts autossh tunnels.
### END INIT INFO

# --                                                            # {{{1
#
# File        : autossh.init
# Maintainer  : Felix C. Stegerman <flx@obfusk.net>
# Date        : 2013-04-09
#
# Copyright   : Copyright (C) 2013  Felix C. Stegerman
# Licence     : GPLv2
#
# --                                                            # }}}1

# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh
# script

PATH=/sbin:/usr/sbin:/bin:/usr/bin

DAEMON=/usr/bin/autossh
RUNNING=/usr/lib/autossh/autossh

DESC='autossh tunnels'
NAME=autossh

SCRIPT=/etc/init.d/$NAME
RUN=/var/run/$NAME

# --

AUTOSSH_USER=syncuser
AUTOSSH_OPTS='-N -R:2201:localhost:22 -p2022 -f'
AUTOSSH_TUNNELS=(syncuser@www.example.com)

function autossh_opts () { AUTOSSH_OPTS="$@"; }
function tunnel () { local x="$@"; AUTOSSH_TUNNELS+=( "$x" ); }

[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# --

[ ! -e "$RUN" ] && { mkdir "$RUN"; chown "$AUTOSSH_USER": "$RUN"; }

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

[ -n "$AUTOSSH_INIT_VERBOSE" ] && VERBOSE="$AUTOSSH_INIT_VERBOSE"

# --

function do_start()                                             # {{{1
{
  # Return:
  #   0 if daemon has been started
  #   1 if daemon was already running
  #   2 if daemon could not be started

  local tunnel n=0 pidfile args already=0 failed=0
  for tunnel in "${AUTOSSH_TUNNELS[@]}"; do
    pidfile="$RUN/$NAME.$n.pid"; (( ++n ))
    args=( $AUTOSSH_OPTS $tunnel )
    echo -n "[autossh tunnel start] CMD=$DAEMON ${args[@]} ... "
    touch "$pidfile"; chown "$AUTOSSH_USER": "$pidfile"
    export AUTOSSH_PIDFILE="$pidfile"
    start-stop-daemon -c "$AUTOSSH_USER" --start --quiet \
      --pidfile "$pidfile" --exec "$RUNNING" --test > /dev/null \
      || { (( ++already )); echo 'already running'; continue; }
    start-stop-daemon -c "$AUTOSSH_USER" --start --quiet \
      --pidfile "$pidfile" --exec "$DAEMON" -- "${args[@]}" \
      || { (( ++failed )); echo failed; continue ; }
    echo OK
  done
  [ "$failed"  -gt 0 ] && return 2
  [ "$already" -gt 0 ] && return 1
  return 0
}                                                               # }}}1

function do_stop()                                              # {{{1
{
  # Return:
  #   0 if daemon has been stopped
  #   1 if daemon was already stopped
  #   2 if daemon could not be stopped
  #   other if a failure occurred

  local pidfile p c r already=0 failed=0 retval=0
  for pidfile in $( ls -d "$RUN"/*.pid 2>/dev/null ); do
    p="$( cat "$pidfile" )"; c="$( ps -p "$p" -o command= )"
    echo -n "[autossh tunnel stop] PID=$p CMD=$c ... "
    start-stop-daemon -c "$AUTOSSH_USER" --stop --quiet \
      --retry=TERM/30/KILL/5 --pidfile "$pidfile" --name "$NAME"
    r="$?"
    case "$r" in
      0) rm -f "$pidfile"; echo OK ;;
      1) (( ++already )); echo 'already running' ;;
      2) (( ++failed )); echo failed ;;
      *) retval="$r"; rm -f "$pidfile"; echo "failed ($r)" ;;
    esac
  done
  [ "$retval"  -gt 2 ] && return "$retval"
  [ "$failed"  -gt 0 ] && return 2
  [ "$already" -gt 0 ] && return 1
  return 0
}                                                               # }}}1

function do_status ()                                           # {{{1
{
  # Return: 0 if all alive; 1 if some dead.
  local pidfile p c alive=0 dead=0 n=0 m info
  for pidfile in $( ls -d "$RUN"/*.pid 2>/dev/null ); do
    p="$( cat "$pidfile" )"; c="$( ps -p "$p" -o command= )"
    if [ -n "$c" ]; then
      echo "[autossh tunnel alive] PID=$p CMD=$c"; (( ++alive ))
    else
      echo "[autossh tunnel dead] PID=$p"; (( ++dead ))
    fi
    (( ++n ))
  done
  m="${#AUTOSSH_TUNNELS[@]}"
  info="alive=$alive dead=$dead total=$n config=$m"
  echo "[autossh tunnel status] $info"
  if [ "$dead" -eq 0 ]; then return 0; else return 1; fi
}                                                               # }}}1

# --

case "$1" in                                                    # {{{1
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
      0|1)  [ "$VERBOSE" != no ] && log_end_msg 0 ;;
      *)    [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
  ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
      0|1)  [ "$VERBOSE" != no ] && log_end_msg 0 ;;
      *)    [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
  ;;
  status)
    do_status
  ;;
  restart|force-reload)                                         # {{{2
    log_daemon_msg "Restarting $DESC" "$NAME"
    do_stop
    case "$?" in
      0|1)
        do_start
        case "$?" in
          0) log_end_msg 0 ;;
          1) log_end_msg 1 ;; # Old process is still running
          *) log_end_msg 1 ;; # Failed to start
        esac
      ;;
      *) log_end_msg 1 ;; # Failed to stop
    esac
  ;;                                                            # }}}2
  *)
    echo "Usage: $SCRIPT {start|stop|status|restart|force-reload}" >&2
    exit 3
  ;;
esac                                                            # }}}1

:

# vim: set tw=70 sw=2 sts=2 et fdm=marker :

Set this file as executable, apply the defaults, and start it.

sudo chmod +x /etc/init.d/autossh
sudo update-rc.d autossh defaults
sudo service autossh start

Additional reference: man autossh

Additional minor steps

Disable apparmor for sssd

sudo ln -sf /etc/apparmor.d/usr.sbin.sssd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.sssd

Reference: Weblink 6

RAID testing and alerting/monitoring

configuring postfix to relay through gmail

Reference: Weblink 9

sudo apt-get install postfix mailutils

Choose internet site when apt prompts.

Set contents of /etc/postfix/sasl_passwd:

[smtp.gmail.com]:587    bgstack15@gmail.com:PASSWORDPLAINTEXTHERE

Change permissions of file.

chmod 0600 /etc/postfix/sasl_passwd

Configure postfix's /etc/postfix/main.cf with these settings. This is only the partial contents!

relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Run this command to prepare the berkely database of the password file.

postmap /etc/postfix/sasl_passwd
service postfix restart

Send a test email.

mail -s "Test 8" bgstack15@gmail.com <<EOF
> this is the message contents
> are you sure?
> goodbye
> EOF

This test message worked.

Now this command will send an alert to me if something goes wrong:

mdadm --monitor --scan --mail=bgstack15@gmail.com

But the above is unnecessary because on Debian-based systems, mdadm already provides a cron.daily entry that will email root. So adjust /etc/aliases to include this definition:

root: bgstack15@gmail.com

And then run newaliases.

Simulate a failure. References include Weblink 7 and Weblink 8.

mdadm --manage --set-faulty /dev/md0 /dev/sdb

That command should cause that --monitor process to send an email.

Remove and re-add the hard disk.

mdadm /dev/md0 -r /dev/sdb
mdadm /dev/md0 -a /dev/sdb

Building the synchronization script

The main synchronization script runs on host dns2 which is on-prem. All steps in this section are for dns2. Server dns2 already mounts up serverx:/volume1/sword on /mnt/serverx for SpiderOakONE backup purposes.

The script is named /etc/installed/sync-offsite.sh. It reads exclusions from /etc/installed/sync-offsite.excludes. Establish the log directory.

# on dns2
sudo mkdir -p /var/log/sync-offsite
sudo chown syncuser:admins /var/log/sync-offsite
sudo chmod 0775 /var/log/sync-offsite

Establish /etc/installed/sync-offsite.sh

# on dns2
sudo touch /etc/installed/sync-offsite.sh /etc/installed/sync-offsite.excludes
sudo chown syncuser:admins /etc/installed/sync-offsite.sh /etc/installed/sync-offsite.excludes
sudo chmod 0770 /etc/installed/sync-offsite.sh
sudo chmod 0660 /etc/installed/sync-offsite.excludes

Check out my sync-offsite shell script for the shell script and its discussion.

This script will be run by cron on a schedule. On dns2, establish file /etc/cron.d/60_offsite_backup_cron.

# File: dns2:/etc/cron.d/60_offsite_backup_cron
# Startdate: 2021-11-24
# Purpose: Run offsite backup script which syncs to server2
15 5 * * *  syncuser  /etc/installed/sync-offsite.sh 1>/dev/null 2>&1

The excluded paths, which follow the rsync FILTER RULES format, are stored in /etc/installed/sync-offsite.excludes. The root of the transfer is /mnt/serverx because the first path parameter to rsync is /mnt/serverx/shares without a trailing slash.

# File: sync-offsite.excludes
# Location: dns2:/etc/installed/sync-offsite.excludes
# Author: bgstack15
# Startdate: 2021-11-19
# Title: Exclude patterns for rsync for offsite backup script
# Purpose: Separate config from commands
# History:
# Usage:
#    Lines that begin with pound # symbol are comments.
#    A starting slash / in this rules file means in the top directory of dns2:/mnt/serverx/shares so /public in this file would mean dns2:/mnt/serverx/shares/public
# Document:
# This should already be empty on serverx; it is a high-volume rotation of a few GB of music intended for my mobile phone.
/shares/public/Music/syncthing/*
# these are just practice sync dirs for FreeFileSync testing and are not required:
/shares/public/sync-left
/shares/public/sync-right
/shares/public/Video/temp

Preparing dns entry for server2

Server2 uses dhcp, so the extant dhcpd and dns solution on the internal network will properly add ddns and reverse records. When server2 is placed at the target network, it will only be directly addressable via the vpn. A domain name can be used with some setup.

On dns1 and dns2, establish a new zone database and its reverse database file. These paths are different for each host!

# on dns1
sudo touch /var/named/data/db.remote.internal.com /var/named/data/db.10.222.0
sudo chown named.named /var/named/data/db.remote.internal.com /var/named/data/db.10.222.0

# on dns2
sudo touch /var/named/slaves/db.remote.internal.com /var/named/slaves/db.10.222.0
sudo chown named.named /var/named/slaves/db.remote.internal.com /var/named/slaves/db.10.222.0

On dns1, add to file /etc/named/named.conf.local this clause.

# on dns1
zone "remote.internal.com" {
        type master;
        file "/var/named/data/db.remote.internal.com";
};
zone "0.222.10.in-addr.arpa" {
        type master;
        file "/var/named/data/db.10.222.0"; # 10.222.0/24 wireguard subnet
        allow-transfer { key DHCP_UPDATER; };
};

On dns2, add to file /etc/named/named.conf.local this clause.

zone "remote.internal.com" {
        type slave;
        file "slaves/db.remote.internal.com";
        masters { 192.168.1.10 key DHCP_UPDATER; };
};
zone "0.222.10.in-addr.arpa" {
        type slave;
        file "slaves/db.10.222.0"; # 10.222.0/24 wireguard subnet
        masters { 192.168.1.10 key DHCP_UPDATER; };
};

On dns1, populate /var/named/data/db.10.222.0 with this initial contents.

$ORIGIN .                       
$TTL 604800     ; 1 week        
0.222.10.in-addr.arpa   IN SOA  dns1.ipa.internal.com. admin.ipa.internal.com. (
                                1          ; serial 
                                604800     ; refresh (1 week)
                                86400      ; retry (1 day)
                                2419200    ; expire (4 weeks)
                                604800     ; minimum (1 week)
                                )
                        NS      dns1.ipa.internal.com.
                        NS      dns2.ipa.internal.com.
$ORIGIN 0.222.10.in-addr.arpa.
3                       PTR     danube.remote.internal.com.
4                       PTR     server2.remote.internal.com.
14                      PTR     server1.remote.internal.com.

On dns1, populate /var/named/data/db.remote.internal.com with this initial contents.

$ORIGIN .
$TTL 604800     ; 1 week
remote.internal.com     IN SOA  dns1.ipa.internal.com. admin.ipa.internal.com. (
                                1          ; serial
                                604800     ; refresh (1 week)
                                86400      ; retry (1 day)
                                2419200    ; expire (4 weeks)
                                604800     ; minimum (1 week)
                                )
                        NS      dns1.ipa.internal.com.
                        NS      dns2.ipa.internal.com.
$ORIGIN remote.internal.com.
$TTL 86400      ; 1 day
server2                A       10.222.0.4
server1                A       10.222.0.14

These zones are designed to be hardcoded. The additional entries are not related to server2 but are useful in the dns zone.

Reload dns on each dns server.

sudo service named reload

Validate the results.

$ nslookup server2.remote.internal.com
Server:     192.168.1.10
Address:    192.168.1.10#53

Name:   server2.remote.internal.com
Address: 10.222.0.4

$ nslookup 10.222.0.4
4.0.222.10.in-addr.arpa name = server2.remote.internal.com.

Operations

Here are some expected tasks that might be required in the future on server2.

Checking RAID health

To check the health of the mdadm device, run any of these commands.

cat /proc/mdstat
sudo mdadm --detail /dev/md0

To send an email if the status is degraded at all, run this command.

mdadm --monitor --scan --mail=bgstack15@gmail.com

Testing RAID failure alerts

To conduct a failure test, you can tell mdadm to pretend a drive is faulty.

mdadm --manage --set-faulty /dev/md0 /dev/sdb

To clear that faulty status (visible in the commands from heading Checking RAID health), run these two commands.

sudo mdadm /dev/md0 -r /dev/sdb
sudo mdadm /dev/md0 -a /dev/sdb

These commands remove and then re-add the disk.

Checking autossh

The ssh tunnel should always be kept alive by the system service autossh. Check its status with ps or this command.

$ sudo service autossh status
[autossh tunnel alive] PID=1629 CMD=/usr/lib/autossh/autossh -N -R:2201:localhost:22 -p2022    syncuser@www.example.com
[autossh tunnel status] alive=1 dead=0 total=1 config=1

Checking vpn status

The wireguard vpn (IP address 10.222.0.4) should always be running as well. Check its status with the following command.

$ sudo service wireguard status
interface: wg0
  public key: xozGPE4L5ncWgp4SpapAWGn1J6wMYv9JfGX4as1e8UA=
  private key: (hidden)
  listening port: 51820

peer: KOQVWMlb+T2zkLrCSsG7DJm29wQGovEV1LfLrPafKjp=
  endpoint: 35.133.216.104:51820
  allowed ips: 192.168.1.10/32, 192.168.1.11/32, 192.168.1.14/32, 192.168.1.18/32, 10.222.0.0/24
  latest handshake: 1 minute, 21 seconds ago
  transfer: 2.65 TiB received, 38.11 GiB sent
  persistent keepalive: every 25 seconds

Any system on the Internal network should be able to reach server2 via its wireguard interface IP address 10.222.0.4 as long as server1 is operating correctly.

Sending test email

Postfix is configured in /etc/postfix/main.cf and /etc/postfix/master.cf but these should not need to be changed unless the gmail account gets a new password or disables the "less-secure app access" mode. Sending a test email is really easy.

$ mail -s "This is the subject" -r "Pretty Name <root@server2.remote.internal.com>" <<EOF
> Contents go here
> EOF

Updating the excluded paths from the rsync command

The main sync-offsite.sh script runs fron server dns2. It reads file /etc/installed/sync-offsite.excludes which is fully documented inside that file. But simply, add new entries as uncommented lines to that file.

Performing a dry-run reverse sync for manual remediation

Check back for a future post on this topic!

This task should be similar to the master backup reverse script, so that I can clean up files that are truly not-needed.

Connecting to server2

During normal operations, server2 should always have a vpn connection to server1. Server2 uses IP address 10.222.0.4, which should be reachable from the Internal network at all times. Additionally, server2 runs autossh with a port being forwarded to server2:22 (ssh service). Any system on the Internal network can run any of the following commands to get to the ssh service on server2.

ssh -p2201 server1
ssh server2.remote.internal.com

Knowledge Base