Knowledge Base

This month's OS updates were brutal! One small good thing: CentOS 7 was very mild: no kernel updates.

Things got rocky (no pun intended; I don't use Rocky Linux; I went with AlmaLinux for my CentOS 8 replacement) when I was validating my main file server. It operates way too many services, including my main wireguard "server." (In Wireguard, everything is merely a peer. I just make this one a peer to everything else because I like star topology.)

Well, I first realized that my wireguard routing (firewalld masquerading) was not working. Come to find out, Firewalld was malfunctioning. I spent a lot of time troubleshooting a weird error:

firewalld[903]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: No such file or directory

I troubleshot nftables, and python. There doesn't appear to be a binary named python-nftables. Some Ubuntu chap found a similar problem which he solved by manually adding the nft tables and chains somehow missing (that I assume firewalld should be adding on its own). I then wasted way too much time trying to parse the supposedly bad json blob from the logs with jq, and looping through to get my easy nft commands for chain and table. Unfortunately I didn't figure that out and ended up just running those commands manually. They didn't help.

So now I was holding multiple broken pieces of firewalld, wireguard, oh, and radicale (my calendar solution). Radicale was throwing errors about a read-only filesystem.

So, after trying to revert firewalld to just use trusty old iptables, which went sideways even faster than nftables, and searching those errors, I discovered that dmesg was throwing a fascinating and scary message:

missing module BTF, cannot register kfuncs

Which reliably shows up when restarting firewalld in its broken state. It was at this point I guess that this AlmaLinux 8 system had received a kernel update, and sure enough, it had. It was running 6.3.1 and the previous boot was 6.2.9. So I decide to just reboot. I visit the console (because I've never bothered to set up a cool network kvm device) and ensure I interrupt grub because who can be bothered to learn the new, current way to configure grub to pick a specific menu entry? I booted into the previous kernel entry, and then firewalld worked, and its masquerade (ip routing) function worked. So wireguard was back to functioning.

So by the way, this flapping of my nfs server affected my jellyfin instance which then coughed up some old entries from its database for some reason and spat out .nfo files for long-gone files once nfs got reestablished. Whatever. That's the least of my worries.

So now, I turned my attention to radicale. It was spewing a silly error:

radicale[939]: [939/Thread-3] [ERROR] An exception occurred during PROPFIND request on '/bgstack15/': [Errno 30] Read-only file system: '/var/lib/radicale/collections/.Radicale.lock'

I'm an old hat, classically trained (as I like to say), so I dropped SELinux into permissive mode, which didn't help. I then ran the radicale server on the command line, which did remove that error. It still operated incredibly slowly, so I eventually gave up on that. I saw in the radicale.service file an entry, ReadWritePaths= which led me to read about that in the systemd man pages. My collections are stored in there, but under a symlink. So I ran:

systemctl edit radicale.service
[Service]
ReadWritePaths=/the/readlink-f/path/to/my/collections

And then systemctl daemon-reload and restarted the app. And now, it didn't throw that error but it still ran incredibly slowly on my calendar(s). Well, come to find out, it wanted to reindex all 9,000 entries in my calendar, which takes a while. But once it had finished that, it was able to then serve them fast enough for my calendar front-end webapp.

And to think I say I enjoy this stuff!