Knowledge Base

Preserving for the future: Shell scripts, AoC, and more

Automated Devuan CI kickoff process

The Devuan project CI uses gitlab and jenkins (documentation).

To kick off a build of a git repository stored on https://git.devuan.org, you just make an issue with the label of the correct suite (unstable/experimental), and assigned to Releasebot. Oh, you also need a branch of code named suites/unstable, and a tag of upstream/1.32.0 where 1.32.0 is the upstream release version. Sometimes Debian doesn't actually store those, so you have to make those yourself.

files/2024/listings/make-issue.sh (Source)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
#!/bin/sh
# File: make-issue.sh
# Location: /mnt/public/work/git.devuan.org
# Author: bgstack15
# Startdate: 2024-07-02-3 10:26
# Title: Begin Build of Package in Devuan CI
# Purpose: make a oneliner to submit a build task for a project I maintain in Devuan
# History:
# Usage:
#    ./make-issue.sh lightdm
# Reference:
#    https://docs.gitea.com/api/1.20/#tag/organization/operation/orgGetLabel
#    https://demo.gitea.com/api/swagger#/issue/issueCreateIssue
#    https://docs.gitea.com/developers/api-usage
# Improve:
# Dependencies:
#    devuan-req: curl, jq

TOKEN="$( cat /mnt/public/packages/git.devuan.org-token )"
SUITE="${SUITE:-unstable}"
GITEAURL="${GITEAURL:-https://git.devuan.org}"
REPO="${REPO:-${1}}"

main() {
   # input: SUITE, TOKEN, REPO
   # Purpose: print the build job url, e.g, https://jenkins.devuan.dev/job/devuan-package-builder/2015
   # get label id of suite we want to use.
   test -z "${REPO}" && { echo "Fatal! Must set REPO. Aborted." ; return 1 ; }
   response="$( curl --silent "${GITEAURL%%/}/api/v1/orgs/devuan/labels" )"
   label_id="$( echo "${response}" | jq ".[] | select(.name == \"${SUITE}\").id" )"
   response="$( curl --silent "${GITEAURL%%/}/api/v1/repos/devuan/${REPO}/issues" -X POST -H "Accept: application/json" -H "Authorization: token ${TOKEN}" -H "Content-Type: application/json" --data-raw \
      "{\"description\":\"\",\"labels\":[${label_id}],\"title\":\"build\",\"assignees\":[\"releasebot\"]}"
   )"
   issue="$( echo "${response}" | jq '.number' )"
   sleep 8 # to give releasebot time to add its comment
   response="$( curl --silent "${GITEAURL%%/}/api/v1/repos/devuan/${REPO}/issues/${issue}/comments" -X GET -H "Accept: application/json" -H "Authorization: token ${TOKEN}" )"
   buildjob="$( echo "${response}" | jq --raw-output '.[] | select(.user.login = "ReleaseBot") | select(.body | startswith("Triggered")).body' | sed -r -e 's/^.*(https?)/\1/;' )"
   echo "${buildjob}"
}

# check if dot-sourced:
# Ref: https://stackoverflow.com/questions/2683279/how-to-detect-if-a-script-is-being-sourced/28776166#28776166
sourced=0
if [ -n "$ZSH_VERSION" ]; then 
  case $ZSH_EVAL_CONTEXT in *:file) sourced=1;; esac
elif [ -n "$KSH_VERSION" ]; then
  [ "$(cd -- "$(dirname -- "$0")" && pwd -P)/$(basename -- "$0")" != "$(cd -- "$(dirname -- "${.sh.file}")" && pwd -P)/$(basename -- "${.sh.file}")" ] && sourced=1
elif [ -n "$BASH_VERSION" ]; then
  (return 0 2>/dev/null) && sourced=1 
else # All other shells: examine $0 for known shell binary filenames.
     # Detects `sh` and `dash`; add additional shell filenames as needed.
  case ${0##*/} in sh|-sh|dash|-dash) sourced=1;; esac
fi

# if not dot-sourced:
if test $sourced -eq 0 ;
then
   test -z "${REPO}" && { echo "Fatal! Must set REPO. Aborted." ; exit 1 ; }
   main
fi

So now I don't have to manually navigate to do this every time. I just fix the repo with branches and tags, and then run this script. I love this modern API world! It's a shame it comes with some opinionated tech and companies that own tech that use it in ways against people. Tech doesn't have to be horrible. It can be great, like this!

Android and my printer

Default Print Service

For the built-in Android Default Print Service, omit the protocol.

print.ipa.example.com:631/printers/ml1865w

CUPS Printing

To use CUPS Printing with my printer, use this url:

https://print.ipa.example.com:631/printers/ml1865w

It complained about the untrusted cert, which suggests that CUPS Printing does not rely on the user-added certificates.

Alternative research

I researched, and unfortunately this fix for this app is not eough.

Perhaps the application needs to update the network_security_config.xml to include <certificates src="user" /> like F-droid itself needed

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <base-config cleartextTrafficPermitted="true">
        <trust-anchors>
            \<!-- Trust preinstalled CAs --\>
            <certificates src="system" />
            \<!-- Additionally trust user added CAs --\>
            <certificates src="user" />
        </trust-anchors>
    </base-config>

Troubleshooting problem in Infcloud, my web calendar solution

I've written about my web calendar solution. Over time I've noticed that my web calendar malfunctions and doesn't always hide/display my calendars when I check and uncheck the box.

I spent a bunch of time debugging this, and learned that it might be a commit where I forcefully set a variable at login time in commit d9ddf89b32a1514248f0943e72f704a8bcb523e4 to file webdav_protocol.js.

@@ -747,6 +772,8 @@ function netFindResource(inputResource, inputResourceIndex, forceLoad, indexR, l
                    if(globalSettings.activecalendarcollections.value.length>0 && globalVisibleCalDAVCollections.length==0)
                        globalDefaultCalendarCollectionActiveAll=true;
                }
+               // stackrpms,2 This forces all calendars to turn on at initial load
+               globalDefaultCalendarCollectionActiveAll=true;

                if(globalDefaultCalendarCollectionActiveAll)
                    for(var i=0; i<globalResourceCalDAVList.collections.length; i++)

I think even if a calendar is off, but I set that value to true, it misleads the variable globalVisibleCalDAVCollections. I found that using the java console to reset that value makes this work better.

globalVisibleCalDAVCollections=[]

So I haven't made any changes to the code, at this time. By just resetting that array with 5,445 entries (of the 11 calendars I have in production), my web calendar started behaving better.

add chapter titles to mkv automatically

Here's an old one I haven't used in years apparently, because the shebang was /bin/sh but it requires bash, so from my Fedora desktop days.

files/2024/listings/chapter-titles.sh (Source)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/bin/bash
# utility to simplify entering chapter titles into MKVToolNix GUI
# startdate: 2019-04-07 20:26
# dependencies:
#    INFILE needs one line per chapter. Each line is the exact contents of the chapter title.
#    xdotool

test -z "${INFILE}" && INFILE=/mnt/public/Video/chapters.txt
test -n "${1}" && INFILE="${1}"

echo "Using INFILE=${INFILE} with chapter titles:"
head -n3 "${INFILE}"
printf "%s\n" "..."
tail -n3 "${INFILE}"
echo "Press enter to start the 5-second countdown, and then move the cursor to the MKVToolNix window and Name field."
read foo
sleep 5

x=0
while read -u10 line ;
do
   # in case you want to count them
   x=$(( x + 1 ))
   echo "line=${line}"
   xdotool type --delay 12ms "${line}"
   xdotool key --delay 12ms "Return" "Return"
done 10< "${INFILE}"

echo "done!"

So while it's kind of a oneliner, you have to prepare your input file with the chapter names. That is left as an exercise for the reader.

To use this, you have to have the input file prepared, and you have to be running mkvtoolnix-gui already. (I never learned how to do with with mkvtoolnix cli which probably renders this whole script moot.) Open your mkv file on the chapter tab. Then in a nearby terminal, run:

chapter-titles.sh ~/chapters.txt

It will tell you to press enter, and then you have 5 seconds to position the cursor at the chapter 1 Name field.

Ironic that I go to all this effort, but only one of my video players (vlc, and not mpv or Jellyfin) seems to show chapter titles so why do I bother? Maybe that's why I haven't touched this script in 5 years. I got the itch to add all the metadata things to a file and got carried away, and was pleasantly surprised to learn I needed to describe it here!

Oneliner for installing network printer

After reinstalling my CentOS 7 print server with Rocky Linux 9, I realized I had changed the name of my cups printer. SO I had to reconfigure it on all my print clients.

#!/bin/sh
sudo lpadmin -D 'Samsung ML-1865w' -L "computer room" -p 'ml1865w' -E -v ipp://print.ipa.example.com:631/printers/ml1865w -m everywhere
sudo lpadmin -d 'ml1865w'

F-droid repo web frontend

In my previous post, I showed how I set up my fdroid mirror suitable for adding as an f-droid repository. I wrote a little web page generator, that parses the metadata of the packages.

files/2024/listings/fdroid_generate_web.py (Source)

#!/usr/bin/env python3
# vim: set ts=3 sw=3 sts=3 et:
# File: /etc/installed/fdroid/fdroid_generate_web.py
# Location: server3
# Author: bgstack15
# SPDX-License-Identifier: GPL-3.0-only
# Startdate: 2024-06-03-2 13:31
# Title: Generate simple web view of F-droid repo
# Purpose: Generate a small front-end web page of packages here
# History:
# Usage:
#    called by fdroid-sync.sh, or also just usable by itself
# Reference:
#    https://stackoverflow.com/questions/15940280/how-to-get-utc-time-in-python
# Improve:
# Documentation: design idea is to parse index-v1.json and build a front page, and also a per-app page that resembles f-droid.org/en/packages/org.jellyfin.mobile/
from configobj import ConfigObj
import os, json, datetime
CONF_FILE = os.environ.get("CONF_FILE",os.path.join("/etc","sysconfig","fdroid-generate-web"))
if os.path.exists(CONF_FILE):
   cfg = ConfigObj(CONF_FILE)
else:
   cfg = ConfigObj()
   print(f"Info: could not find CONF_FILE {CONF_FILE}. Using all defaults!")
# defaults
if "TOP_DIR" not in cfg:
   cfg["TOP_DIR"] = "/mnt/mirror/fdroid"
if "MIRROR_DIR" not in cfg:
   cfg["MIRROR_DIR"] = "/mnt/mirror/fdroid/repo"
if "INDEX_JSON" not in cfg:
   cfg["INDEX_JSON"] = "/mnt/mirror/fdroid/repo/index-v1.json"
print(cfg)
def get_app_attribute(app, attribute, localizations = ["en-US","en-GB"]):
   """
   Given app dictionary, get attribute, using the order of localizations if necessary.
   Returns that output, and locale of used value if any.
   """
   output = ""
   try:
      output = app[attribute]
      return output, ""
   except:
      if "localized" in app:
         some_l = app["localized"]
         for lchoice in localizations:
            if lchoice in app["localized"]:
               try:
                  output = app["localized"][lchoice][attribute]
                  return output, lchoice
               except:
                  pass
               some_l.pop(lchoice)
         for l in some_l:
            try:
               output = app["localized"][l][attribute]
               return output, l
            except:
               pass
      else:
         pass
   return output, ""
def get_app_icon(app, packageName, mirror_dir, localizations = ["en-US","en-GB"]):
   icon, localized = get_app_attribute(app, "icon", localizations)
   if localized:
      icon = f"{packageName}/{localized}/{icon}"
   else:
      icon = f"icons/{icon}"
   if not os.path.exists(os.path.join(mirror_dir,icon)):
      if os.path.exists(os.path.join(mirror_dir,packageName,"icon.png")):
         return f"{packageName}/icon.png"
      else:
         for l in localizations:
            if os.path.exists(os.path.join(mirror_dir,packageName,l,"icon.png")):
               return os.path.join(packageName,l,"icon.png")
   return icon
def generate_index_html(json_file, top_dir, mirror_dir):
   """
   Given the index-v1.json file, generate a top-level page for the front page, stored in top_dir.
   """
   with open(json_file,"r") as o:
      j = json.load(o)
   #print(len(json_contents))
   with open(os.path.join(top_dir,"index.html"),"w") as w:
      w.write("<html>\n")
      w.write("<head>\n")
      w.write("<link rel='stylesheet' href='fdroid.css'>\n")
      w.write("""<meta name="viewport" content="width=device-width, initial-scale=1">\n""")
      w.write(f"<title>{j['repo']['name']}</title>\n")
      w.write("</head>\n")
      w.write("<body>\n")
      w.write(f"<h1><img src='repo/icons/{j['repo']['icon']}' class='headimg'>{j['repo']['name']}</h1>\n")
      w.write(f"<p><a href='repo/'>Instructions to install this repository</a></p>")
      #apps = sorted(j["apps"],key=lambda i: i["name"])
      apps = j["apps"]
      if len(apps):
         w.write(f"<h2>Packages ({len(apps)})</h2>")
      for app in apps:
         name, _ = get_app_attribute(app,"name")
         packageName, _ = get_app_attribute(app,"packageName")
         icon = get_app_icon(app, packageName, mirror_dir)
         desc, _ = get_app_attribute(app,"summary")
         #w.write(f"<a href='packages/{packageName}.html'><img src='repo/{icon}'>{name}</a> {desc}<br/>\n")
         w.write(f"<img src='repo/{icon}' class='img'><b>{name}</b> {desc}<br/>\n")
      w.write("</body>\n")
      w.write("<footer>\n")
      w.write(f"Last generated: " + str(datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")))
      w.write("</footer>\n")
      w.write("</html>")
def generate_app_pages(json_file, top_dir):
   with open(json_file,"r") as o:
      j = json.load(o)
   apps = j["apps"]
   for app in apps:
      name, _ = get_app_attribute(app,"name")
      packageName, _ = get_app_attribute(app,"packageName")
      icon, localized = get_app_attribute(app,"icon")
      if localized:
         icon = f"../repo/{packageName}/{localized}/{icon}"
      else:
         icon = f"../repo/icons/{icon}"
      with open(os.path.join(top_dir,"packages",packageName+".html"),"w") as w:
         w.write("<html>\n")
         w.write("<head>\n")
         w.write("<link rel='stylesheet' href='../fdroid-package.css'>\n")
         w.write("""<meta name="viewport" content="width=device-width, initial-scale=1">\n""")
         w.write(f"<title>{name}</title>\n")
         w.write("</head>\n")
         w.write("<body>\n")
         w.write(f"<img src='{icon}' class='icon'/><h1>{name}</h1>")
         w.write("</body>\n")
         w.write("</html>\n")
if "__main__" == __name__:
   generate_index_html(cfg["INDEX_JSON"],cfg["TOP_DIR"],cfg["MIRROR_DIR"])
   # Not used at this time:
   #generate_app_pages(cfg["INDEX_JSON"],cfg["TOP_DIR"])

I wanted to get an amazing web frontend like f-droid.org has but I think they have more metadata available to them because they actually compile the apps and so have the source and that complex metadata available inside the various possible places. I didn't want to go through all that effort, so I gave up on the idea of an individual page per app.

F-droid partial mirror

I want to ensure the longevity of the tools I use on my mobile devices. Of course I use the amazing F-droid app store. I wanted to make sure the packages I use from this repository are available even if F-droid ever disappears. (A secondary benefit is that if I ever reinstall, or update, apps from my local repo, the downloads are way faster!)

So I spent a bunch of time reading the docs and learning how to set up my own, partial binary mirror of F-droid. A future step might include building packages, but for now I'm sticking to just the binaries.

Preparing the main F-droid custom mirror

All work is done on server3.

sudo useradd fdroid # set pw in keepass
sudo mkdir -p /mnt/mirror/fdroid /opt/android-sdk
sudo dnf install java-1.8.0-openjdk-headless
sudo dnf install --setopt=install_weak_deps=False java-1.8.0-openjdk-devel

The bare command would be this:

sudo -u fdroid -E /usr/bin/rsync -n -aHS --delete --delete-delay --info=progress2 plug-mirror.rcac.purdue.edu::fdroid/repo/ /mnt/mirror/fdroid/

But I of course wrote a script for myself, which is shown farther down in this post.

Read more…

openssl read cert template

Here's a post that combines all my favorite technologies! This is so fun.

If you want to use openssl to read the template name of a Microsoft Certificate Services certificate, you have to look up the OID that is stored on the cert and find it in the directory.

files/2024/listings/read-cert-template.sh (Source)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/usr/bin/env sh
# File: read-cert-template.sh
# Location: blog exclusive
# Author: bgstack15
# SPDX-License-Identifier: GPL-3.0-only
# Startdate: 2024-05-16-5 10:23
# Title: Read cert template
# Purpose: read certificate and print cert tempalte name if discoverable
# History:
# Usage:
# Reference: see blog post
# Improve:
# Dependencies:
#    openssl, ldapsearch, ldap credential in read-cert-template.conf

# Load conf, RCT_LDAPSERVER RCT_LDAPBASE RCT_LDAPAUTH1 RCT_LDAPAUTH2
RCT_CONF="${RCT_CONF:-${HOME}/.config/read-cert-template.conf}"
test -f "${RCT_CONF}" && . "${RCT_CONF}"

# use RCT_IN env var or first parameter, or else standard input
RCT_IN="${RCT_IN:-${1}}"
RCT_IN="${RCT_IN:-/dev/stdin}"

if echo "${RCT_IN}" | grep -qE -e '^-$|^stdin$' ;
then
   _input="$( cat )"
else
   _input="$( cat "${RCT_IN}" )"
fi

oid="$( echo "${_input}" | openssl x509 -in /dev/stdin -noout -text -certopt no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,ext_parse | sed -n -r -e '/1.3.6.1.4.1.311.21.7/,+2p' | awk '/OBJECT/{print $NF}' | sed -r -e 's/^://;' )"
test -n "${VERBOSE}" && printf 'oid=%s\n' "${oid}" 1>&2
LDAPTLS_REQCERT=never ldapsearch -LLL -o ldif-wrap=9000 -H "${RCT_LDAPSERVER}" ${RCT_LDAPAUTHUNQUOTED} "${RCT_LDAPAUTHQUOTED}" -b "CN=Certificate Templates,CN=Public Key,CN=Services,CN=Configuration,${RCT_LDAPBASE}" "(msPKI-Cert-Template-OID=${oid})" CN | awk '$1~/cn:/{$1="";print;}' | sed -r -e 's/^ +| +$//g;'

files/2024/listings/read-cert-template.conf (Source)

1
2
3
4
5
6
# File: ~/.config/read-cert-template.conf
RCT_LDAPSERVER=ldaps://example.corp
# The "CN=Certificate Templates,CN=Public Key,CN=Services,CN=Configuration," will be prepended to this:
RCT_LDAPBASE="DC=example,DC=corp"
RCT_LDAPAUTHUNQUOTED="-x -w see#keepass"
RCT_LDAPAUTHQUOTED="-D CN=Service Account 319 (sa319),OU=Accounts,DC=example,DC=corp"

References

man openssl man x509