Knowledge Base

Preserving for the future: Shell scripts, AoC, and more

I built Dosbox-X for Devuan

tl;dr

Dosbox-X dpkg is now available for Devuan Unstable.

Longer

The Dosbox-X community doesn't have a solid plan for building a dpkg of Dosbox-X. It's not a huge deal; they offer all sorts of packages! And thankfully, I even found the link to the COPR (Fedora RPM) package. Alas, I have abandoned Fedora GNU/Linux despite my love of SELinux (no joke). Systemd just is unncessary to proper computer operations. Anyway, I translated the nice and simple rpm spec into a dpkg build recipe!

The full story

I got the itch to work in Windows 98 again (yes, I said 98). Wine doesn't cut it, sometimes. So, I pulled out my old Windows 98 VM from last year. I had forgotten I never got the sound working. Also, the display refresh rate was just terrible. Sure, applications install in it, but forget about displaying video.

So, I investigated other options than qemu+spice. I read and followed the thorough demo of installing Windows 98 SE in vanilla Dosbox, but had no success. I very much enjoyed repeating this process 20+ years later. It even crashes and reboots more often than I remember! The setup never finished in vanilla Dosbox, so I had to move on.

So then I tried Dosbox-X! First of all, I wanted to build this application in a dpkg because I am not interested in Flatpak or other methods. I suppose I would have tolerated trying an AppImage but they didn't have one that I found. Anything that can be built in an rpm can be built in a dpkg (assuming of course the dependencies are available). Thankfully, there were no exotic dependencies, and writing a debian recipe was straightforward! The hardest part was some rather obtuse syntax of how to run the auto_configure statement, which is just a dpkg thing.

Stay tuned for more on this narrative about Windows 98 SE.

My small agenda project

I wrote a small program for myself to send me an email of the daily agenda. It uses python library caldav to get the events. It was for this project I had to write the previous post's solution.

It writes a small html output, and then I have a few scripts I use to turn it into an email to myself.

#!/usr/bin/env python3
# File: server3.py
# Startdate: 2023-05-17 15:47:40
# Purpose: script on server3 that generates email message content
import sys, datetime, os
sys.path.append("/home/agenda/agenda")
import agenda
url = "https://server3.ipa.internal.com/radicale/"
username = "bgstack15"
password = "plaintextpw"
thisdate = datetime.date.today()
try:
   thisdate = os.environ.get("AGENDA_DATE",datetime.date.today())
except:
   pass
print(f"Using date {thisdate}",file=sys.stderr)
print(agenda.summarize(thisdate,url=url,username=username,password=password),end="")

The above python script gets called by a shell script:

#!/bin/sh
# startdate: 2023-05-17-4 15:58
# Purpose: job for emailing daily agenda, for adding to cron
export AGENDA_DATE
results="$( /home/agenda/server3.py )"
test -z "${AGENDA_DATE}" && AGENDA_DATE="$( date "+%F" )"
subject="Daily agenda for ${AGENDA_DATE}"
if test -z "${results}" ;
then
   results="nothing on the agenda today"
   subject="nothing on agenda"
fi
echo "${results}" | mailx -s "$( printf '%s\n%s' "${subject}" "Content-Type: text/html" )" "bgstack15@gmail.com"

And then I threw this into cron:

# File: /etc/cron.d/85_agenda_email_cron
58  6  *  *  *  agenda   sh /home/agenda/server3.sh 1>>/var/server3/shares/public/Support/Systems/server3/var/log/agenda_log 2>&1

Sample html output (let's see if this works in my markdown file used in my SSG):

Online

All day: Pay water bill

Work

8:00am: Work

Main

11:30am: Lunch
2:00pm: Call Nick
5:00pm: remind Joe about AoE2

fixing user calendar access in radicale

Main

I have written before about my calendar solution. This time, I have improved my radicale installation for myself. I spent a long time investigating why a small caldav client script I was writing (post coming in a few days) couldn't get to my account.

I had to run the server on debuglevel "DEBUG" and carefully examine all the rights setup. I use the rights file method. I have previously described how I added my domain auth to my radicale instance.

At first, after a ton of work, I thought that the rights evaluations are not properly evaluating string {0} which should be a python re method for referring to the first replaced named expression in a regular expression, such as block:

[calendars-domain]
user: (.+)@IPA.EXAMPLE.COM
collection: {0}/[^/]+
permissions: rw

I am not entirely convinced it's operating as expected. I wanted user bgstack15@IPA.EXAMPLE.COM to access collections under namespace bgstack15 but it was not working. I tried adding a named variable in the interpolation list in the radicale source code to handle a username_without_domain but that didn't seem to work.

So eventually I just ended up adding a single line right after the variable user gets populated from the http Authorization header, in radicale/app/__init__.py:

user = user.split("@")[0]

Which due to user-friendly language design, safely handles when no at symbol is present also. So this just chomps off the @IPA.EXAMPLE.COM, and then I keep going.

I didn't fork the repo, or build a new rpm (since I'm now on AlmaLinux 8 and can just use the distro radicale3 package instead of the one I had to build for CentOS 7). I just modified the deployed file on my production system like a neanderthal. So any future updates will cause problems. Oh, so this is "technical debt." I guess I'm technically poorer now.

Second thought, unexecuted

And after I'd written my internal documentation about this whole process, I realized I should have just symlinked the collections like so:

cd /var/lib/radicale/collections/collection-root/
ln -s bgstack15 bgstack15@IPA.SMITH122.COM

Or just moved it. Absolutely all auth goes through the frontend reverse proxy because radicale listens only on loopback, so the usernames would always have the domain name appended. Ah, well. Perhaps in an alternate universe(timeline? parallelly-developed planet [Warning: TV Tropes links!]?) I solved it that way.

Why I blog

I read a thread on Ycombinator News about Why I blog, and since I've been out of technical topcis lately, here's my reasons for blogging, in no particular order.

  • I want a creative outlet. I need to practice writing so I keep that ability. I write documentation at work (aren't we all supposed to?).
  • Remembering neat tricks I wrote, or found. I do use this sometimes to remember how I did something.
  • Use my technical expertise. Even just using a wordpress instance took skills. Apparently not everybody has those. This blog is powered with Nikola static site generator on nginx on CentOS 7. All that had to be set up, and configured (because I'm old-school; no cattle here).
  • Have a technical presence on the World Wide Web for when people want to validate my existence.
  • Have a "pet project" that doesn't take up space in the living room.
  • Engage with my fansfriends! Drop me a comment, email, or irc (darn it, what replaced freenode? That was such a catchy name. Libera.chat, what a weak name compared to freenode) to let me know you read this. I need topic ideas!

OS Updates May 2023, and the Aftermath

This month's OS updates were brutal! One small good thing: CentOS 7 was very mild: no kernel updates.

Things got rocky (no pun intended; I don't use Rocky Linux; I went with AlmaLinux for my CentOS 8 replacement) when I was validating my main file server. It operates way too many services, including my main wireguard "server." (In Wireguard, everything is merely a peer. I just make this one a peer to everything else because I like star topology.)

Well, I first realized that my wireguard routing (firewalld masquerading) was not working. Come to find out, Firewalld was malfunctioning. I spent a lot of time troubleshooting a weird error:

firewalld[903]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: No such file or directory

I troubleshot nftables, and python. There doesn't appear to be a binary named python-nftables. Some Ubuntu chap found a similar problem which he solved by manually adding the nft tables and chains somehow missing (that I assume firewalld should be adding on its own). I then wasted way too much time trying to parse the supposedly bad json blob from the logs with jq, and looping through to get my easy nft commands for chain and table. Unfortunately I didn't figure that out and ended up just running those commands manually. They didn't help.

So now I was holding multiple broken pieces of firewalld, wireguard, oh, and radicale (my calendar solution). Radicale was throwing errors about a read-only filesystem.

So, after trying to revert firewalld to just use trusty old iptables, which went sideways even faster than nftables, and searching those errors, I discovered that dmesg was throwing a fascinating and scary message:

missing module BTF, cannot register kfuncs

Which reliably shows up when restarting firewalld in its broken state. It was at this point I guess that this AlmaLinux 8 system had received a kernel update, and sure enough, it had. It was running 6.3.1 and the previous boot was 6.2.9. So I decide to just reboot. I visit the console (because I've never bothered to set up a cool network kvm device) and ensure I interrupt grub because who can be bothered to learn the new, current way to configure grub to pick a specific menu entry? I booted into the previous kernel entry, and then firewalld worked, and its masquerade (ip routing) function worked. So wireguard was back to functioning.

So by the way, this flapping of my nfs server affected my jellyfin instance which then coughed up some old entries from its database for some reason and spat out .nfo files for long-gone files once nfs got reestablished. Whatever. That's the least of my worries.

So now, I turned my attention to radicale. It was spewing a silly error:

radicale[939]: [939/Thread-3] [ERROR] An exception occurred during PROPFIND request on '/bgstack15/': [Errno 30] Read-only file system: '/var/lib/radicale/collections/.Radicale.lock'

I'm an old hat, classically trained (as I like to say), so I dropped SELinux into permissive mode, which didn't help. I then ran the radicale server on the command line, which did remove that error. It still operated incredibly slowly, so I eventually gave up on that. I saw in the radicale.service file an entry, ReadWritePaths= which led me to read about that in the systemd man pages. My collections are stored in there, but under a symlink. So I ran:

systemctl edit radicale.service
[Service]
ReadWritePaths=/the/readlink-f/path/to/my/collections

And then systemctl daemon-reload and restarted the app. And now, it didn't throw that error but it still ran incredibly slowly on my calendar(s). Well, come to find out, it wanted to reindex all 9,000 entries in my calendar, which takes a while. But once it had finished that, it was able to then serve them fast enough for my calendar front-end webapp.

And to think I say I enjoy this stuff!

I tried building an AppImage, Part 1

I recently read a discussion about AppImage, and of course Ubuntu had its recent news that Flatpak is not installed by default in the latest silly-named Ubuntu version (23.04, I assume). I also recently had to use my first AppImage, for OpenShot. There was a bug in the Devuan unstable package of this app, where the user cannot drag the right edge of a video clip to shorten its duration. So I tried the AppImage, and the bug was fixed there!

I decided to investigate building an AppImage. It tends to be focused on desktop applications. The only (relatively simple) one I package that I could think of quickly is FreeFileSync, so I tried adapting my Open Build Service package to include an appimage.

Open Build Service uses an OpenSUSE 42.3 distribution environment, for which I know very little about the available packages. Unfortunately, FreeFileSync aggressively follows its upstream dependencies' version releases, and of course anything not bleeding edge does not have the right package versions. OpenSuSE 42.3 didn't even have some of the older versions of the (wxGTK-related) packages necessary, so my AppImage build process left me frustrated.

I will try again, and instead of packaging from source, I'll try to adapt the existing dpkg I do maintain.

TheMovieDb: Generate my Custom CSV for a show

I've discussed before how I use a CSV to help apply metadata to media files. One of my problems with just manually munging the Wikipedia list of TV show episodes is that TheMovieDB organizes episodes a little differently. Sometimes a two-parter is listed as a single episode, and so on. All minor things, but my episode number can get off which affects my filenames and metadata.

So I decided to investigate using TheMovieDb to generate my CSV/spreadsheet. I found a great python library for tmdbv3api. I think there's multiple python libraries for this, but they all get to the same thing: the contents of the show details.

Here's my little wrapper script: files/2023/04/listings/eplib.py (Source)

#!/usr/bin/env python3
# File: eplib.py
# Location: /mnt/public/Support/Programs/themoviedb
# Author: bgstack15
# Startdate: 2023-04-19-4 13:51
# Title: Episode CSV Generator
# Purpose: facilitate my csv list of episodes of a TV show
# History:
# Usage:
#    source ~/venv1/bin/activate ; cd /mnt/public/Support/Programs/themoviedb/
#    python3
#    >>> import importlib, eplib
# References:
#    1. https://pypi.org/project/tmdbv3api/
#    2. https://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-normalize-in-a-python-unicode-string
# Improve:
# Dependencies:
#    One time:
#       python3 -m venv ~/venv1
#       source ~/venv1/bin/activate
#       pip3 install tmdbv3api
from tmdbv3api import TMDb, TV, Season
import os
tmdb = TMDb()
tmdb.language = "en"
tmdb.debug = True
tv = TV()
def set_api_key(filename = None):
   """ Given a filename (or use hardcoded default file if None), load api key from file. """
   if filename is None:
      filename = "/mnt/public/Support/Programs/themoviedb/apikey"
   try:
      with open(filename,"r") as o:
         tmdb.api_key = o.read().rstrip('\n')
   except:
      return -1
   return 0
### Ripped from ref 2
from unicodedata import combining, normalize
LATIN = "ä  æ  ǽ  đ ð ƒ ħ ı ł ø ǿ ö  œ  ß  ŧ ü "
ASCII = "ae ae ae d d f h i l o o oe oe ss t ue"
def remove_diacritics(s, outliers=str.maketrans(dict(zip(LATIN.split(), ASCII.split())))):
    return "".join(c for c in normalize("NFD", s.translate(outliers)) if not combining(c))
### end ref 2
# bgstack15 function
def filename_safifier(s):
   """ Cruddy detox for myself. Should consider using my customized detoxrc """
   return remove_diacritics(s.replace("'","").replace('"',"").replace("...","").replace(",","").replace(":","_").replace("!","").replace(".","").replace("?","").replace("&","and").replace("/","_"))
def get_tv_details(show_id):
   """ Uses web connection to get details of the item, either id number or show name """
   if show_id.isdigit():
      details = tv.details(show_id)
   else:
      show_temp = get_tv_search(show_id)
      try:
         details = tv.details(show_temp[0]["id"])
      except:
         print("Invalid TV show name. Found these names though:")
         for i in show_temp:
            print(f"id {i['id']}: {i['name']}")
         return -1
   return details
def get_tv_search(show_id):
   """ Uses web connection to search for item, probably the name of the show """
   return tv.search(show_id)
def get_episodes_for_show(show_id = "", include_specials = False):
   """
   Given a show id (can be show_id integer, show name string, or a TV().details object), return useful list of episodes. If given show_id is a TV().details object, use that to avoid all the web calls. Example:
      a = TV().details(1855)
      get_episodes_for_show(a)
      OR
      a = get_tv_details("Star Trek Deep Space Nine")
      get_episodes_for_show(a)
   If include_specials, then also list episode titles and airdates for these special features but they still are not counted as absolute episode numbers.
   """
   if type(show_id).__name__ == "AsObj":
   #if show_details is not None:
      show = show_id
   else:
      if show_id.isdigit():
      # IMPROVE: accept show name?
         show = get_tv_details(show_id)
      else:
         show_temp = get_tv_search(show_id)
         try:
            show = get_tv_details(show_temp[0]["id"])
         except:
            print("Invalid TV show name. Found these names though:")
            for i in show_temp:
               print(f"id {i['id']}: {i['name']}")
            return -1
   #print(show)
   name = f"{show['name']} ({show['first_air_date'][0:4]})"
   season = Season()
   seasons = show["seasons"]
   print(f"Name: {name}")
   show_id = show["id"]
   abs_epnum = 0
   abs_epnum_including_specials = 0
   for s in show["seasons"]:
      snum = s["season_number"]
      season_eps = season.details(show_id, snum).episodes
      for e in season_eps:
         abs_epnum_including_specials += 1
         if snum != 0:
            abs_epnum += 1
         if include_specials or snum != 0:
            sep = e["episode_number"]
            epname = e["name"].replace("(1)","Part 1").replace("(2)","Part 2").replace("(3)","Part 3")
            if '"' in epname:
               epname = "'" + epname + "'"
            if ',' in epname and not '"' in epname:
               epname = '"' + epname + '"'
            airdate = e["air_date"]
            filename = filename_safifier(f"s{snum:02d}e{sep:02d} - {epname}")
            #print(f"{snum},{abs_epnum},{sep},{epname},{airdate},")
            if abs_epnum_including_specials == 1:
               print("s,ep,sep,epname,airdate,filename,")
            print(f"{snum:02d},{abs_epnum:03d},{sep:02d},{epname},{airdate},{filename},")
            #print("%02d,%03d,%02d,%s,%s," % (snum, abs_epnum, sep, epname, airdate))
# Run this regardless of __main__
set_api_key(os.environ.get("TMDB_APIKEY"))

So now I can get the output I want!

Infcloud improvement: keyboard shortcuts dialog

I added a dialog to my fork of Infcloud! This application originally had zero keyboard navigation, and I've slowly been adding some to make it more usable to me.

Right now the contents of the help dialog are hard-coded, but perhaps someday I will attempt to have the section that adds the keyboard shortcuts add their own text description to a variable that is used to populate this field.

All the work this time is in one merge commit.