This post is an almost-complete story, and supersedes the previous post.
Overview
System server2 is a Dell PowerEdge T30 workstation with 4 3.5" 6TB WD Red Pro hard disks. It serves one primary goal, offsite backup of internal data. The remote location is [SCRUBBED]. The remote network where server2 will reside uses dhcp, so the default Debian dhcp client configuration is used.
The key design criteria are listed in the table below.
| Item |
Specific technology chosen |
| VPN |
wireguard |
| Redundant storage |
RAID 5 on WD Red Pro 6TB disks (one disk failure tolerated) |
| Firewall |
nftables, without firewalld |
| Front-door net tunnel |
autossh |
| Backup process |
shell script and cron job |
Related systems that will be modified to work with server2 include:
- server1, bastion host for front-door entry to internal network
- dns2, system where main
offsite-sync.sh script will run
- FreeIPA domain overall will get new config items
Initial preparation
Install low-level requirements.
apt-get install sudo screen vim curl gpg
Run the various scripts from http://www.example.com/internal/Support/Platforms/devuan/scripts/
apt-get --no-install-recommends install bgconf bgscripts-core
sudo apt-get install python3-ipalib=4.9.7-1+devuan1 freeipa-client=4.9.7-1+devuan1 freeipa-helper systemctl-service-shim nfs-common cifs-utils mlocate parted rsync sssd-tools
time sudo bgconf.py -d10 2>&1 | sudo tee -a ~root/bgconf.$( date "+%F" ).log
/mnt/public/Support/Platforms/devuan/scripts/ipa-client-install.sh
sudo apt-get -V autopurge adwaita-icon-theme dbus-x11 util-linux-locales
Additional steps are described in /mnt/public/Support/Systems/server2/server2-plan.md
Main installation and preparation
Preparing the domain
The FreeIPA domain that is already used for access control will be modified.
Add a new user.
ipa user-add --first=sync --last=user --cn='syncuser' --homedir /home/syncuser --gecos='Service account for syncing data' --email='bgstack15+sync@gmail.com' --password --noprivate --displayname='syncuser' --shell=/bin/bash syncuser --gidnumber=960600013
The password is stored in my main keepass file. The chosen gid is group service-accounts which does not have login access to production systems.
Make a new hbac rule allow this user to log into server2.
ipa hbacrule-add --servicecat=all syncuser_rule
ipa hbacrule-add-user --users=syncuser syncuser_rule
ipa hbacrule-add-host --hosts=dns2 syncuser_rule
ipa hbacrule-add-host --hosts=server2 syncuser_rule
ipa hbacrule-mod --desc='Allow syncuser to access the relevant systems for backups' syncuser_rule
As syncuser@dns2, make a new ssh public key and add it to the ipa user.
# as user syncuser@dns2
ssh-keygen
ipa user-mod "${USER}" --sshpubkey="$( cat ~/.ssh/id_rsa.pub )"
Establish rules for the service account to log in the relevant systems. Reference: Weblink 3
Establish sudoers permission in ipa for syncuser:
ipa sudocmd-add --desc='rsync full permissions' rsync
ipa sudocmd-add --desc='rsync full permissions' /usr/bin/rsync
ipa sudorule-add syncuser-rsync
ipa sudorule-add-host syncuser-rsync --hosts server2
ipa sudorule-add-host syncuser-rsync --hosts dns2
ipa sudorule-add-allow-command syncuser-rsync --sudocmds rsync
ipa sudorule-add-allow-command syncuser-rsync --sudocmds /usr/bin/rsync
ipa sudorule-add-option syncuser-rsync --sudooption '!authenticate'
ipa sudorule-add-user syncuser-rsync --users syncuser
ipa sudorule-mod syncuser-rsync --desc='syncuser can run rsync on dns2, server2'
After ensuring the sssd cache was updated:
$ sudo -l -U syncuser
Matching Defaults entries for syncuser on server2:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin,
env_keep+="ftp_proxy http_proxy https_proxy no_prxy", env_keep+="DEBUG DEBUG_LEVEL DRYRUN VERBOSE"
User syncuser may run the following commands on server2:
(root) NOPASSWD: /usr/bin/rsync, rsync
Installing wireguard
I chose wireguard because I already use it and adding a new peer is very easy.
sudo apt-get install wireguard resolvconf
Wireguard already works with resolvconf to control the dns when entering/leaving the vpn, so it is accepted here.
Establish /etc/wireguard/wg0.conf. References include internal document 1 and internal document 2.
[Interface]
Address = 10.222.0.4/24
ListenPort = 51820
# from wg genkey
PrivateKey = SCRUBBED
# server2 public key
# xozGLE4M5ncwGp4SpanAQGn1J6wMYv9JfGW4nS0e8UA=
DNS = 192.168.1.10,192.168.1.11, ipa.internal.com, vm.internal.com, internal.com
[Peer]
# server1
PublicKey = KOQVWNY3+TMzkMrCTsG7DJm29wQGovEv1LfLrptfAjw=
AllowedIPs = 192.168.1.10/32, 192.168.1.11/32, 192.168.1.14/32, 192.168.1.18/32, 10.222.0.0/24
PersistentKeepalive = 25
Endpoint = www.example.com:51820
Also had to add this as a peer on server1! That configuration is straightforward and not described here.
Set up a custom init script, /etc/init.d/wireguard.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119 |
#! /bin/sh
# Adapted from https://gist.github.com/kbabioch/5dd8801e702e519ed18d9b17cacae716
# 2021-11-18
# Copyright (c) 2021 Karol Babioch <karol@babioch.de>
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# LSBInitScript for Wireguard: This is a leightweight init script for
# Wireguard. While Wireguard itself requires only minimal overhead to setup and
# start, it still requires some script invocations (e.g. during boot).
#
# Most distributions are using systemd by now, and as such can use
# wg-quick@.service. However some distributions / images / Linux appliances
# are not (yet) using systemd. In such cases, this init script could be used
# to (re)start and/or stop Wireguard.
#
# It can handle all configured Wireguard interfaces (within /etc/wireguard)
# globally and/or individual interfaces, e.g. (/etc/init.d/wireguard start wg0).
#
# It relies on wg(8) and wg-quick(8) in the background.
### BEGIN INIT INFO
# Provides: wireguard
# Required-Start: $network $syslog
# Required-Stop: $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Starts Wireguard interfaces
# Description: Sets up Wireguard interfaces (by means of wg-quick).
### END INIT INFO
CONFIG_DIR=/etc/wireguard
function get_active_wg_interfaces() {
INTERFACES=$(wg | awk '/interface:/{print $NF}')
echo "$INTERFACES"
}
# This is required for wg-quick(1) to work correctly, i.e. for process
# substitution (`<()`) to work in Bash. If missing, wg-quick will fail with a
# "fopen: No such file or directory" error.
#[ -e /dev/fd ] || ln -sf /proc/self/fd /dev/fd
case "$1" in
start)
if [ -z "$2" ]; then
echo "Starting all configured Wireguard interfaces"
for CONFIG in $(cd $CONFIG_DIR; ls *.conf); do
wg-quick up ${CONFIG%%.conf}
done
else
echo "Starting Wireguard interface: $2"
wg-quick up "$2"
fi
;;
stop)
if [ -z "$2" ]; then
echo "Stopping all active Wireguard interfaces"
INTERFACES=$(get_active_wg_interfaces)
for INTERFACE in $INTERFACES; do
wg-quick down "$INTERFACE"
done
else
echo "Stopping Wireguard interface: $2"
wg-quick down "$2"
fi
;;
reload|force-reload)
if [ -z "$2" ]; then
echo "Reloading configuration for all active Wireguard interfaces"
INTERFACES=$(get_active_wg_interfaces)
for INTERFACE in $INTERFACES; do
wg-quick strip "$INTERFACE" | wg syncconf "$INTERFACE"
done
else
echo "Reloading configuration for Wireguard interface: $2"
wg-quick strip "$2" | wg syncconf "$2"
fi
;;
restart)
$0 stop "$2"
sleep 1
$0 start "$2"
;;
status)
# TODO Check exit codes and align them with LSB requirements
if [ -z "$2" ]; then
INTERFACES=$(get_active_wg_interfaces)
for INTERFACE in $INTERFACES; do
wg show $INTERFACE
done
else
wg show "$2"
fi
;;
*)
echo "Usage: $0 { start | stop | restart | reload | force-reload | status } [INTERFACE]"
exit 1
;;
esac
|
Set wireguard to start with the defaults.
sudo update-rc.d wireguard defaults
Setting up the disk array
I chose to use RAID 5, so the array can handle 1 disk failure and still keep going. I prefer to be able to survive 2 disks failed but I only have 4 disks due to the size of the chassis so this is a compromise I make.
Reference: Weblink 2
sudo apt-get install mdadm
#The following NEW packages will be installed:
# bsd-mailx exim4-base exim4-config exim4-daemon-light libgnutls-dane0 libidn12 liblockfile1 libunbound8 mdadm
# psmisc
Begin to make the array. This runs the job in the background already!
sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sde 2>&1 | tee -a /root/mdadm.create.$( date "+%F" ).log
Check the status with
cat /proc/mdstat
It is done now, so:
time sudo mkfs.ext4 /dev/md0
sudo mkdir -p /var/server2/
sudo mount -t ext4 -o noatime,nodev -v /dev/md0 /var/server2
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
> ARRAY /dev/md0 metadata=1.2 name=server2:0 UUID=671746fe:2d387a18:fa0c46af:8744bb7c
That last command added the ARRAY line to /etc/mdadm/mdadm.conf.
Add this filesystem to /etc/fstab:
/dev/md0 /var/server2 ext4 auto,rw,noatime,discard,nodev 0 0
And with those lines in those files, now we need to run:
sudo update-initramfs -u
Establish firewall
Reference: weblink 4
sudo apt-get install nftables
sudo cp -p /usr/share/doc/nftables/examples/sysvinit/nftables.init /etc/init.d/nftables
sudo chmod +x /etc/init.d/nftables
sudo update-rc.d nftables defaults
The config file is /etc/nftables.conf. I set it initially with:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27 |
#!/usr/sbin/nft -f
# Startdate: 2021-11-23
# File: server2:/etc/nftables.conf
# Reference:
# /usr/share/doc/nftables/examples/workstation.nft
# Documentation: /etc/installed/server2a.md
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0;
# accept any localhost traffic
iif lo accept
# accept traffic that originated from this system
# accept traffic originated from us
ct state established,related accept
# this {} array is comma-separated
tcp dport { 22 } ct state new accept
# count and drop any other traffic
counter drop
}
chain forward {
type filter hook forward priority 0;
}
chain output {
type filter hook output priority 0;
}
}
|
Establish autossh
The remote server will also be configured to connect to my network through the front door, so to speak. This is another way to connect to the system in case of VPN failure.
In FreeIPA, allow syncuser to ssh in to the bastion host.
ipa hbacrule-add syncuser_ssh
ipa hbacrule-add-user --users=syncuser syncuser_ssh
ipa hbacrule-add-host --hosts=server1 syncuser_ssh
ipa hbacrule-mod --desc='Allow syncuser to ssh to server1 for autossh tunnel' syncuser_ssh
Generate on server2 an ssh key for syncuser.
ssh-keygen
Add this new key to the ipa user, without deleting the old ssh key. See my recent post about that.
eval ipa user-mod ${USER} $( ipa user-show ${USER} --all | awk '/SSH public key:/{$1="";$2="";$3="";print}' | sed -r -e 's/ *, */\n/g;' -e 's/^\s*//g;' | while read line ; do printf '%s ' "--sshpubkey='${line}'" ; done ; ) --sshpubkey="'$( cat ~/.ssh/id_rsa.pub )'"
Modify server1, the bastion host, to allow local port bindings on non-loopback IP addresses, and also open the firewall.
# on server1
sudo firewall-cmd --add-port=2201/tcp --permanent
sudo firewall-cmd --reload
sudo /usr/libexec/bgscripts/py/modconf.py -a -c '#' -l ' ' /etc/ssh/sshd_config set GatewayPorts yes
sudo service sshd restart
And now, the ssh command to connect to the front dynamic dns hostname of my main site is:
sshk -R:2201:localhost:22 -p2022 syncuser@www.example.com
When this tunnel is running, a user on the Internal network can run this command to connect to server2:
ssh -p2201 server1
Establish /etc/init.d/autossh.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185 |
#!/bin/bash
# Adapted from https://github.com/obfusk/autossh-init/blob/master/autossh.init
### BEGIN INIT INFO
# Provides: autossh
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: autossh initscript
# Description: Starts autossh tunnels.
### END INIT INFO
# -- # {{{1
#
# File : autossh.init
# Maintainer : Felix C. Stegerman <flx@obfusk.net>
# Date : 2013-04-09
#
# Copyright : Copyright (C) 2013 Felix C. Stegerman
# Licence : GPLv2
#
# -- # }}}1
# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh
# script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DAEMON=/usr/bin/autossh
RUNNING=/usr/lib/autossh/autossh
DESC='autossh tunnels'
NAME=autossh
SCRIPT=/etc/init.d/$NAME
RUN=/var/run/$NAME
# --
AUTOSSH_USER=syncuser
AUTOSSH_OPTS='-N -R:2201:localhost:22 -p2022 -f'
AUTOSSH_TUNNELS=(syncuser@www.example.com)
function autossh_opts () { AUTOSSH_OPTS="$@"; }
function tunnel () { local x="$@"; AUTOSSH_TUNNELS+=( "$x" ); }
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# --
[ ! -e "$RUN" ] && { mkdir "$RUN"; chown "$AUTOSSH_USER": "$RUN"; }
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions
[ -n "$AUTOSSH_INIT_VERBOSE" ] && VERBOSE="$AUTOSSH_INIT_VERBOSE"
# --
function do_start() # {{{1
{
# Return:
# 0 if daemon has been started
# 1 if daemon was already running
# 2 if daemon could not be started
local tunnel n=0 pidfile args already=0 failed=0
for tunnel in "${AUTOSSH_TUNNELS[@]}"; do
pidfile="$RUN/$NAME.$n.pid"; (( ++n ))
args=( $AUTOSSH_OPTS $tunnel )
echo -n "[autossh tunnel start] CMD=$DAEMON ${args[@]} ... "
touch "$pidfile"; chown "$AUTOSSH_USER": "$pidfile"
export AUTOSSH_PIDFILE="$pidfile"
start-stop-daemon -c "$AUTOSSH_USER" --start --quiet \
--pidfile "$pidfile" --exec "$RUNNING" --test > /dev/null \
|| { (( ++already )); echo 'already running'; continue; }
start-stop-daemon -c "$AUTOSSH_USER" --start --quiet \
--pidfile "$pidfile" --exec "$DAEMON" -- "${args[@]}" \
|| { (( ++failed )); echo failed; continue ; }
echo OK
done
[ "$failed" -gt 0 ] && return 2
[ "$already" -gt 0 ] && return 1
return 0
} # }}}1
function do_stop() # {{{1
{
# Return:
# 0 if daemon has been stopped
# 1 if daemon was already stopped
# 2 if daemon could not be stopped
# other if a failure occurred
local pidfile p c r already=0 failed=0 retval=0
for pidfile in $( ls -d "$RUN"/*.pid 2>/dev/null ); do
p="$( cat "$pidfile" )"; c="$( ps -p "$p" -o command= )"
echo -n "[autossh tunnel stop] PID=$p CMD=$c ... "
start-stop-daemon -c "$AUTOSSH_USER" --stop --quiet \
--retry=TERM/30/KILL/5 --pidfile "$pidfile" --name "$NAME"
r="$?"
case "$r" in
0) rm -f "$pidfile"; echo OK ;;
1) (( ++already )); echo 'already running' ;;
2) (( ++failed )); echo failed ;;
*) retval="$r"; rm -f "$pidfile"; echo "failed ($r)" ;;
esac
done
[ "$retval" -gt 2 ] && return "$retval"
[ "$failed" -gt 0 ] && return 2
[ "$already" -gt 0 ] && return 1
return 0
} # }}}1
function do_status () # {{{1
{
# Return: 0 if all alive; 1 if some dead.
local pidfile p c alive=0 dead=0 n=0 m info
for pidfile in $( ls -d "$RUN"/*.pid 2>/dev/null ); do
p="$( cat "$pidfile" )"; c="$( ps -p "$p" -o command= )"
if [ -n "$c" ]; then
echo "[autossh tunnel alive] PID=$p CMD=$c"; (( ++alive ))
else
echo "[autossh tunnel dead] PID=$p"; (( ++dead ))
fi
(( ++n ))
done
m="${#AUTOSSH_TUNNELS[@]}"
info="alive=$alive dead=$dead total=$n config=$m"
echo "[autossh tunnel status] $info"
if [ "$dead" -eq 0 ]; then return 0; else return 1; fi
} # }}}1
# --
case "$1" in # {{{1
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
*) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
*) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
status)
do_status
;;
restart|force-reload) # {{{2
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*) log_end_msg 1 ;; # Failed to stop
esac
;; # }}}2
*)
echo "Usage: $SCRIPT {start|stop|status|restart|force-reload}" >&2
exit 3
;;
esac # }}}1
:
# vim: set tw=70 sw=2 sts=2 et fdm=marker :
|
Set this file as executable, apply the defaults, and start it.
sudo chmod +x /etc/init.d/autossh
sudo update-rc.d autossh defaults
sudo service autossh start
Additional reference: man autossh
Additional minor steps
Disable apparmor for sssd
sudo ln -sf /etc/apparmor.d/usr.sbin.sssd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.sssd
Reference: Weblink 6
RAID testing and alerting/monitoring
configuring postfix to relay through gmail
Reference: Weblink 9
sudo apt-get install postfix mailutils
Choose internet site when apt prompts.
Set contents of /etc/postfix/sasl_passwd:
[smtp.gmail.com]:587 bgstack15@gmail.com:PASSWORDPLAINTEXTHERE
Change permissions of file.
chmod 0600 /etc/postfix/sasl_passwd
Configure postfix's /etc/postfix/main.cf with these settings. This is only the partial contents!
relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
Run this command to prepare the berkely database of the password file.
postmap /etc/postfix/sasl_passwd
service postfix restart
Send a test email.
mail -s "Test 8" bgstack15@gmail.com <<EOF
> this is the message contents
> are you sure?
> goodbye
> EOF
This test message worked.
Now this command will send an alert to me if something goes wrong:
mdadm --monitor --scan --mail=bgstack15@gmail.com
But the above is unnecessary because on Debian-based systems, mdadm already provides a cron.daily entry that will email root. So adjust /etc/aliases to include this definition:
root: bgstack15@gmail.com
And then run newaliases.
Simulate a failure. References include Weblink 7 and Weblink 8.
mdadm --manage --set-faulty /dev/md0 /dev/sdb
That command should cause that --monitor process to send an email.
Remove and re-add the hard disk.
mdadm /dev/md0 -r /dev/sdb
mdadm /dev/md0 -a /dev/sdb
Building the synchronization script
The main synchronization script runs on host dns2 which is on-prem. All steps in this section are for dns2. Server dns2 already mounts up serverx:/volume1/sword on /mnt/serverx for SpiderOakONE backup purposes.
The script is named /etc/installed/sync-offsite.sh. It reads exclusions from /etc/installed/sync-offsite.excludes. Establish the log directory.
# on dns2
sudo mkdir -p /var/log/sync-offsite
sudo chown syncuser:admins /var/log/sync-offsite
sudo chmod 0775 /var/log/sync-offsite
Establish /etc/installed/sync-offsite.sh
# on dns2
sudo touch /etc/installed/sync-offsite.sh /etc/installed/sync-offsite.excludes
sudo chown syncuser:admins /etc/installed/sync-offsite.sh /etc/installed/sync-offsite.excludes
sudo chmod 0770 /etc/installed/sync-offsite.sh
sudo chmod 0660 /etc/installed/sync-offsite.excludes
Check out my sync-offsite shell script for the shell script and its discussion.
This script will be run by cron on a schedule. On dns2, establish file /etc/cron.d/60_offsite_backup_cron.
# File: dns2:/etc/cron.d/60_offsite_backup_cron
# Startdate: 2021-11-24
# Purpose: Run offsite backup script which syncs to server2
15 5 * * * syncuser /etc/installed/sync-offsite.sh 1>/dev/null 2>&1
The excluded paths, which follow the rsync FILTER RULES format, are stored in /etc/installed/sync-offsite.excludes. The root of the transfer is /mnt/serverx because the first path parameter to rsync is /mnt/serverx/shares without a trailing slash.
# File: sync-offsite.excludes
# Location: dns2:/etc/installed/sync-offsite.excludes
# Author: bgstack15
# Startdate: 2021-11-19
# Title: Exclude patterns for rsync for offsite backup script
# Purpose: Separate config from commands
# History:
# Usage:
# Lines that begin with pound # symbol are comments.
# A starting slash / in this rules file means in the top directory of dns2:/mnt/serverx/shares so /public in this file would mean dns2:/mnt/serverx/shares/public
# Document:
# This should already be empty on serverx; it is a high-volume rotation of a few GB of music intended for my mobile phone.
/shares/public/Music/syncthing/*
# these are just practice sync dirs for FreeFileSync testing and are not required:
/shares/public/sync-left
/shares/public/sync-right
/shares/public/Video/temp
Preparing dns entry for server2
Server2 uses dhcp, so the extant dhcpd and dns solution on the internal network will properly add ddns and reverse records. When server2 is placed at the target network, it will only be directly addressable via the vpn. A domain name can be used with some setup.
On dns1 and dns2, establish a new zone database and its reverse database file. These paths are different for each host!
# on dns1
sudo touch /var/named/data/db.remote.internal.com /var/named/data/db.10.222.0
sudo chown named.named /var/named/data/db.remote.internal.com /var/named/data/db.10.222.0
# on dns2
sudo touch /var/named/slaves/db.remote.internal.com /var/named/slaves/db.10.222.0
sudo chown named.named /var/named/slaves/db.remote.internal.com /var/named/slaves/db.10.222.0
On dns1, add to file /etc/named/named.conf.local this clause.
# on dns1
zone "remote.internal.com" {
type master;
file "/var/named/data/db.remote.internal.com";
};
zone "0.222.10.in-addr.arpa" {
type master;
file "/var/named/data/db.10.222.0"; # 10.222.0/24 wireguard subnet
allow-transfer { key DHCP_UPDATER; };
};
On dns2, add to file /etc/named/named.conf.local this clause.
zone "remote.internal.com" {
type slave;
file "slaves/db.remote.internal.com";
masters { 192.168.1.10 key DHCP_UPDATER; };
};
zone "0.222.10.in-addr.arpa" {
type slave;
file "slaves/db.10.222.0"; # 10.222.0/24 wireguard subnet
masters { 192.168.1.10 key DHCP_UPDATER; };
};
On dns1, populate /var/named/data/db.10.222.0 with this initial contents.
$ORIGIN .
$TTL 604800 ; 1 week
0.222.10.in-addr.arpa IN SOA dns1.ipa.internal.com. admin.ipa.internal.com. (
1 ; serial
604800 ; refresh (1 week)
86400 ; retry (1 day)
2419200 ; expire (4 weeks)
604800 ; minimum (1 week)
)
NS dns1.ipa.internal.com.
NS dns2.ipa.internal.com.
$ORIGIN 0.222.10.in-addr.arpa.
3 PTR danube.remote.internal.com.
4 PTR server2.remote.internal.com.
14 PTR server1.remote.internal.com.
On dns1, populate /var/named/data/db.remote.internal.com with this initial contents.
$ORIGIN .
$TTL 604800 ; 1 week
remote.internal.com IN SOA dns1.ipa.internal.com. admin.ipa.internal.com. (
1 ; serial
604800 ; refresh (1 week)
86400 ; retry (1 day)
2419200 ; expire (4 weeks)
604800 ; minimum (1 week)
)
NS dns1.ipa.internal.com.
NS dns2.ipa.internal.com.
$ORIGIN remote.internal.com.
$TTL 86400 ; 1 day
server2 A 10.222.0.4
server1 A 10.222.0.14
These zones are designed to be hardcoded. The additional entries are not related to server2 but are useful in the dns zone.
Reload dns on each dns server.
sudo service named reload
Validate the results.
$ nslookup server2.remote.internal.com
Server: 192.168.1.10
Address: 192.168.1.10#53
Name: server2.remote.internal.com
Address: 10.222.0.4
$ nslookup 10.222.0.4
4.0.222.10.in-addr.arpa name = server2.remote.internal.com.
Operations
Here are some expected tasks that might be required in the future on server2.
Checking RAID health
To check the health of the mdadm device, run any of these commands.
cat /proc/mdstat
sudo mdadm --detail /dev/md0
To send an email if the status is degraded at all, run this command.
mdadm --monitor --scan --mail=bgstack15@gmail.com
Testing RAID failure alerts
To conduct a failure test, you can tell mdadm to pretend a drive is faulty.
mdadm --manage --set-faulty /dev/md0 /dev/sdb
To clear that faulty status (visible in the commands from heading Checking RAID health), run these two commands.
sudo mdadm /dev/md0 -r /dev/sdb
sudo mdadm /dev/md0 -a /dev/sdb
These commands remove and then re-add the disk.
Checking autossh
The ssh tunnel should always be kept alive by the system service autossh. Check its status with ps or this command.
$ sudo service autossh status
[autossh tunnel alive] PID=1629 CMD=/usr/lib/autossh/autossh -N -R:2201:localhost:22 -p2022 syncuser@www.example.com
[autossh tunnel status] alive=1 dead=0 total=1 config=1
Checking vpn status
The wireguard vpn (IP address 10.222.0.4) should always be running as well. Check its status with the following command.
$ sudo service wireguard status
interface: wg0
public key: xozGPE4L5ncWgp4SpapAWGn1J6wMYv9JfGX4as1e8UA=
private key: (hidden)
listening port: 51820
peer: KOQVWMlb+T2zkLrCSsG7DJm29wQGovEV1LfLrPafKjp=
endpoint: 35.133.216.104:51820
allowed ips: 192.168.1.10/32, 192.168.1.11/32, 192.168.1.14/32, 192.168.1.18/32, 10.222.0.0/24
latest handshake: 1 minute, 21 seconds ago
transfer: 2.65 TiB received, 38.11 GiB sent
persistent keepalive: every 25 seconds
Any system on the Internal network should be able to reach server2 via its wireguard interface IP address 10.222.0.4 as long as server1 is operating correctly.
Sending test email
Postfix is configured in /etc/postfix/main.cf and /etc/postfix/master.cf but these should not need to be changed unless the gmail account gets a new password or disables the "less-secure app access" mode. Sending a test email is really easy.
$ mail -s "This is the subject" -r "Pretty Name <root@server2.remote.internal.com>" <<EOF
> Contents go here
> EOF
Updating the excluded paths from the rsync command
The main sync-offsite.sh script runs fron server dns2. It reads file /etc/installed/sync-offsite.excludes which is fully documented inside that file. But simply, add new entries as uncommented lines to that file.
Performing a dry-run reverse sync for manual remediation
Check back for a future post on this topic!
This task should be similar to the master backup reverse script, so that I can clean up files that are truly not-needed.
Connecting to server2
During normal operations, server2 should always have a vpn connection to server1. Server2 uses IP address 10.222.0.4, which should be reachable from the Internal network at all times. Additionally, server2 runs autossh with a port being forwarded to server2:22 (ssh service). Any system on the Internal network can run any of the following commands to get to the ssh service on server2.
ssh -p2201 server1
ssh server2.remote.internal.com
References
Internal documents
-
- server1:/etc/wireguard/wg0.conf
Weblinks
- LSBInitScript for Wireguard: This is a leightweight init script for Wireguard
- How to Create a RAID 5 Storage Array with 'mdadm' on Ubuntu 16.04
- Rsync 'Permission denied' for root - Top 4 reasons and solutions
- nftables - Debian Wiki
- https://github.com/obfusk/autossh-init/blob/master/autossh.init
- Disable apparmor for sssd | Knowledge Base
- Using mdadm to send e-mail alerts for RAID failures | Support | SUSE
- Linux SW RAID MDADM System Test.pdf
- Configure Postfix to use Gmail as a Mail Relay
Online documentation
- /usr/share/doc/nftables/examples/workstation.nft
- man
autossh(1)