Preserving for the future: Shell scripts, AoC, and more
If you want to run a program with a special restriction, such as without network access, you can do that with the unshare(1) utility.
unshare -r -n makemkv
I had noticed that makemkv makes some unwanted network traffic, and I was too lazy to insert my network proxy from the past. So a quick Internet search later, and I learned about unshare.
Unshare -n usually needs -r so the application will run. Read the man page for details.
So for just over a month I have noticed some errors in my logs for my main mirror sync scripts.
[2022-07-23T12:00:02Z]root@server3.ipa.internal.com: BEGIN errors @ERROR: max connections (150) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1661) [Receiver=3.1.3] @ERROR: Unknown module 'fedora-enchilada' rsync error: error starting client-server protocol (code 5) at main.c(1661) [Receiver=3.1.3] @ERROR: max connections (150) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1661) [Receiver=3.1.3] [2022-07-23T12:00:40Z]root@server3.ipa.internal.com: END errors
I didn't think much of it, but I have now learned that each of these "max connection" errors means that my entire connection to the upstream mirror failed. Two of these were kernel.org's mirrors. Apparently now they're so busy/popular at the time I connect to them that the server refuses the connection.
So my on-prem Fedora mirrors were out of date. This problem frustrated me, so I had to find a new mirror. So here is the Fedora releases mirror list and here is EPEL mirror list.
So now that I've picked a new one, my connections work and my local mirror is up-to-date!
You see, nowadays I have only one Fedora workstation. Everything else is now running Devuan GNU+Linux to avoid systemd. If I ever reinstall my Fedora machine, it will get Devuan too. I love my SELinux, but I don't actually care about it for desktop systems. And I'd like to avoid the bloatware on my systems when possible.
In some CI (continuous integration; i.e., Gitlab runners, Jenkins jobs, et al) shell jobs that I wrote, I finally turned some useful things into parameters. With code being checked into source control, I would hate to have to update the scm just to re-enable pipefail. But updating the invocation of a shell script in the CI is easy! So now I have this:
echo " $@ " | grep -qE " -v | --verbose " && set -x || : echo " $@ " | grep -qE " --pipefail " && set -oe pipefail || :
It is important to wrap the $@ (all parameters excluding $0 which is the name of the invoked command) with spaces, so that the checks work correctly. Also, the logical or || with a shell no-op : is important for those pipefail-enabled situations!
Share any small snippets that you use to make your life easier in your CI jobs!
The whole topic of self-hosted youtube alternatives is rather scant on the Internet, which surprised me.
I found a nice web-based yt-dlp frontend! I have a single Docker server for other purposes, so it was nice to play around with this metube. Ultimately it wasn't better than running yt-dlp in a terminal and dragging links into that command to append the url to the command.
And for the viewing frontend, I tried https://git.mills.io/prologic/tube which sounded so promising! The video upload operations never completed for me. I think it was trying to encode the videos (transcode?) but it never actually populated the web page with contents even after ffmpeg stopped running.
So I then tried MediaCMS which I really, really wanted to like. It's a Django (python) app, which I have no experience with. I was unable to get this reverse-proxied behind Apache httpd under a virtual path (or prefix, i.e., https://example.com/media/). I gave up and then used a different port number, so all the traffic on port 1285 went to the application.
This one worked with uploaded videos. I had to change a few settings in the deploy/docker/local_settings.py:
MINIMUM_RESOLUTIONS_TO_ENCODE = [240, 360, 1080]
USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
The last two are the bog-standard Django reverse-proxy instructions that deal with https. This is needed for any proxying tasks. I removed my manipulated STATIC_URL, MEDIA_URL, and FORCE_SCRIPT_NAME components which never actually completely worked.
The MediaCMS web presentation looks slick. With the 1080 minimum resolution (which ended up producing a file twice as large as the 1080p files I uploaded), it was acceptable. There is even a little button over the video for "Cast video." From a mobile browser, the cast failed to send to my Chromecast. From a desktop's chromium, it did work to cast to a Chromecast. It started as screen mirroring, and then it asked me if I wanted to send just the video.
So the usability was missing a little bit.
And none of this was actually better than Jellyfin, and also just visiting the files over nfs, except the awesome youtube interface. The quadruple files per video was a little excessive but storage is cheap these days, right?
In my efforts to de-google my life, I have established a Photoprism instance. It has a great third-party tool that helps build the same albums on Photoprism that I had in Google Photos.
I used to use images from Google Photos album as screensaver on Linux but now I am using PhotoPrism so I had to develop a new solution.
My goals include showing only the most recent photos (60 days in my case), and thankfully the Photoprism API endpoint for searching for images allows you to filter with "after {date}"! In fact, the logic for pulling the time range from Photoprism is easier than from Google Photos!
I wrote a whole python library and simple front end to accomplish my shell script which pulls the named albums' recent images.
This is a basic python lib for interacting with the API of a PhotoPrism instance.
This project's upstream is at https://bgstack15.ddns.net/cgit/photoprismpull and https://gitlab.com/bgstack15/photoprismpull.
A library written in Go is advertised by Photoprism.
For a shell interaction with the API to trigger imports, see https://bgstack15.ddns.net/cgit/photoprism-autoimport/.
For a script to create albums from a Google Photos Takeout in PhotoPrism see https://github.com/inthreedee/photoprism-transfer-album.
I do not know Go, but I am comfortable in python. My two original goals for this library were:
There is a small front-end, pp.py. Run its --help option.
I tend to use this library in an interactive python shell. One task included fetching a Google Photos album as a zip, extracting, and generating a sha1sums file. I had used that related photoprism-transfer-album tool, but my Takeout failed to include a few albums for no reason. Since I already had all the photos in PhotoPrism, I just needed to find them and add them to the correct album.
sha1sum -- * > input.album5
And then in a python shell:
import importlib, pplib # as needed after changes: importlib.reload(pplib) a = pplib.get_session("https://pp.example.com","admin","password") c = add_hashes_to_album(a, "Pretty Album Name", "/mnt/public/Images/photoprism/Named albums/Pretty Album Name/input.album5",apply=True)
The simpler example is downloading an album to a subdirectory.
import pplib a = pplib.get_session("https://pp.example.com","admin","password") c = download_album_to_directory("Pretty Album Name",directory="/mnt/public/Images/photoprism",extraparams="&after=2020-02-08",session=a)
This will make a new directory, /mnt/public/Images/photoprism/Pretty Album Name/ and populate it with the images from this named album in PhotoPrism. Note that the get_album_by_title() function explicitly excludes "albums" autogenerated from existing directories in your originals directory. It only searches through actual "album" albums. This is trivial to change for yourself as needed.
The file is that simple! There are so many more parameters you can pass to SearchPhotos. I only needed "after."
I spent some time trying to configure my self-hosted Jitsi Meet based on the documentation that says to make a custom-interface_config.js and custom-config.js for the volume that gets mounted as /config in the container. If you follow the instructions, that would be path ~/.jitsi-meet-cfg/web.
Unfortunately, the container does not actually process these files, so I had to develop my own way to inject my settings and branding into the application.
Part of the instructions for deploying involve getting the release tarball and extracting it (as opposed to cloning the git repo). Inside this tarball is the docker-compose.yml and the web/ directory for that Dockerfile. I had to modify this web/Dockerfile, by adding a new layer:
RUN ln -sf /custom/watermark.svg /usr/share/jitsi-meet/images/watermark.svg && \ ln -sf /custom/interface_config.js /defaults/interface_config.js && \ ln -sf /custom/config.js /defaults/config.js
I put this at the bottom, below the RUN declaration. Then, I built this new image and recorded the name of the image, e.g., 3129533498de.
~/jitsi-7439-2/web$ docker build .
And then update the docker-compose.yml to use this image for target web: and add the volume.
services: # Frontend web: #image: jitsi/web:${JITSI_IMAGE_VERSION:-stable-7439-2} image: 3129533498de restart: ${RESTART_POLICY:-unless-stopped} ports: - '${HTTP_PORT}:80' - '${HTTPS_PORT}:443' volumes: - ${CONFIG}/web:/config:Z - ${CONFIG}/web/crontabs:/var/spool/cron/crontabs:Z - ${CONFIG}/transcripts:/usr/share/jitsi-meet/transcripts:Z - /home/jitsi/.jitsi-meet-cfg/custom:/custom:Z
And then of course make my ~/.jitsi-meet-cfg/custom directory.
~/.jitsi-meet-cfg/custom$ ls config.js interface_config.js watermark.svg
I copied these javascript files from web/rootfs/defaults/ and modified them to suit my needs, and also I generated my own watermark.svg.
Upon preparing these files, I was able to then bring docker-compose up.
docker-compose up -d
And now my Jitsi Meet has my settings! That was way harder than it should have been, but the application was not acting as documented.
I recently got the urge to connect my old USB gamepads and play retro games with them. (Specifically, Jazz Jackrabbit 2 if you care). I spent a week whipping up a small GUI interface to do that.
On Devuan-like systems use packages:
You may attach the USB gamepad before or after running this program. For best results, do not connect multiple gamepads at a time.
Add new configs to config.py, with a simple declarative syntax that is documented briefly with the two given examples. Feel free to share additional configs with this upstream. Gather aliases for attached gamepads with the helper files in directory extras/.
The upstream for the project is at https://bgstack15.ddns.net/cgit/pyjstest/ or https://gitlab.com/bgstack15/pyjstest/.
I wanted to build a simple input-reaction program, and test my python skills, and learn some sdl2. SDL2 has python3 bindings available, but most of the documentation on the Internet is for the C library.
For basic input display on cli and gtk displays, use these utilities.
I need to redesign this to handle multiple attached devices. Right now, it supports only one.
I added an answer to AskUbuntu, based on a different answer. Here is a small function that easily enables/disables using the features provided by package xserver-xorg-input-joystick.
jsinput() {
# Insert your controller name here, as seen from `xinput --list`.
_c="$( xinput --list --id-only 'Logic3 Controller' )"
# Default is off, unless you pass "yes" or similar as first parameter.
_mode="0" ; echo "${1}" | grep -qiE '\<(yes|1|y|on)\>' && _mode=1
# Find ids of these property names, and then tell each one to go to the mode chosen in the previous line.
xinput list-props "${_c}" | sed -n -r -e '/Generate (Mouse|Key)/{s/.*\(([0-9]+)\).*/\1/;p}' | xargs -I@ xinput set-prop "${_c}" @ "${_mode}"
}
If you stick this in your ~/.bashrc you could then run jsinput to disable this controller as mouse input. To turn on mouse input you could then simply run jsinput 1 or jsinput on.
With the recent Gmail change that requires oauth2 for sending authenticated gmail (covered in Postfix use oauth2 for gmail), my jellyfin Webhook plugin that includes an smtp option has finally stopped working.
First of all, I had to ensure that I had network connectivity to my smtp server which is available over my wireguard connection.
nc server2.ipa.internal.com 25 Ncat: Connection refused.
So I had to modify the nftables rules on server2. That took me a while, but I finally got it. For a real-time modification, I used this command.
sudo nft add rule 'inet filter' input position 4 iif wg0 accept
This rule means "for input interface wg0 [wireguard], accept all packets." And insert this rule in a certain position, and not just at the end (so after the infamous "DROP ALL" of a well-behaved firewall.
And my full ruleset is now in /etc/nftables.conf.
flush ruleset table inet filter { chain input { type filter hook input priority 0; # accept any localhost traffic iif lo accept iif wg0 accept comment "trust all wireguard traffic" # accept traffic that originated from this system # accept traffic originated from us ct state established,related accept # this {} array is comma-separated tcp dport { 22 } ct state new accept # count and drop any other traffic counter drop } chain forward { type filter hook forward priority 0; } chain output { type filter hook output priority 0; } }
So finally my netcat worked.
$ nc server2.ipa.internal.com 25 220 server2.ipa.internal.com ESMTP Postfix (Debian/GNU)
So when I trigger a notification in Jellyfin, I get this error.
Jun 26 18:27:50 server2 postfix/smtpd[14319]: connect from server1.remote.internal.com[10.198.0.14] Jun 26 18:27:50 server2 postfix/smtpd[14319]: warning: TLS library problem: error:0A000126:SSL routines::unexpected eof while reading:../ssl/record/rec_layer_s3.c:308: Jun 26 18:27:50 server2 postfix/smtpd[14319]: lost connection after STARTTLS from server1.remote.internal.com[10.198.0.14] Jun 26 18:27:50 server2 postfix/smtpd[14319]: disconnect from server1.remote.internal.com[10.198.0.14] ehlo=1 starttls=1 commands=2
Researching on the Internet for "jellyfin webhook smtp starttls" led to information mostly about disabling starttls. I didn't even realize I had it enabled. So I made some changes to my postfix to disable the silly snakeoil TLS certificate.
And then I logged in again, and got this message in my postfix logs! So this is progress.
Jun 26 18:34:49 server2 postfix/smtpd[15802]: NOQUEUE: reject: RCPT from server1.remote.internal.com[10.198.0.14]: 454 4.7.1 <example@gmail.com>: Relay access denied; from=<example@gmail.com> to=<example@gmail.com> proto=ESMTP helo=<[192.168.58.18]> Jun 26 18:34:49 server2 postfix/smtpd[15802]: lost connection after RSET from server1.remote.internal.com[10.198.0.14] Jun 26 18:34:49 server2 postfix/smtpd[15802]: disconnect from server1.remote.internal.com[10.198.0.14] ehlo=1 mail=1 rcpt=0/1 rset=1 commands=3/4
After all the changes, my postfix main.cf includes at least these lines:
# Important to comment these out! #smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem #smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key #smtpd_tls_security_level=may smtpd_use_tls=no # This already existed, but... smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination # I added my wireguard subnet here. mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 10.198.0.0/24
And now I can receive notifications when my users visit my Jellyfin server.
And just for completeness's sake, here is my smtp notification information.
Add smtp destination. Name: smtp2 Webhook Url: (not relevant) https://www.example.com/internal/jellyfin-webhook2.html Items: playback start, playback stop, session start User filter: (all users) Item time: (all) Send all properties: no Template: <pre>Username: {{Username}} Action: {{NotificationType}} Timestamp: {{UtcTimestamp}} Title:: {{Name}} {{#if_exist SeriesName}} Series: {{SeriesName}} Season: {{SeasonNumber00}} Episode: {{EpisodeNumber00}} {{/if_exist}} DeviceName: {{DeviceName}} ClientName: {{ClientName}} PlaybackPosition: {{PlaybackPosition}} </pre> END TEMPLATE CONTENTS Sender: example@gmail.com Receiver: example@gmail.com smtp server address: server2.ipa.internal.com smtp port: 25 Use credentials: no Use ssl: no Is html body: yes Subject template: Jellyfin activity for {{Username}}
I have since learned that the nftables.conf contents should NOT have double-quotes around the wg0 interface name. The output of nft list table 'inet filter' shows double-quotes, but these do not work when placed in the rules file.
I finally got fed up with Wine's malfunction quite a few months ago about it passing my gamepads to Jazz Jackrabbit 2. For some reason, after a software update, the controllers did not work correctly.
Today, I spent a bunch of time researching how to adjust the mapping of the gamepads. I found a great tool which helped write custom SDL gamepad mapping strings, SDL2 Gamepad Tool by General Arcade. It is an interactive mapping tool where it guides you to press each button on your controller and it generates the right string.
I never got that string to work correctly with Wine, but I ended up solving my problem by adjusting the Wine registry. I ended up merely setting the key
HKLM\System\CurrentControlSet\Services\WineBus\Map Controllers (REG_DWORD) = 0x0
The relevant snippet from that link.
+->Map Controllers [DWORD value (REG_DWORD): Enable (0x1, default) or disable (0x0) conversion from SDL controllers to XInput-compatible gamepads. Only applies to SDL backend.]
And then, all my buttons worked, and my two-axis dpad worked!
I spent quite a while trying to customize the mappings, which didn't seem to stick when I tried them in the Wine registry in a few places.
HKCU\Software\Wine\DirectInput\"Tomee USB SNES Controller" = "X,Y" HKLM\System\CurrentControlSet\Services\WineBus\Map\"auniquename" = "03000000bd12000015d0000010010000,Tomee SNES USB Controller,platform:Linux,a:b2,b:b1,x:b3,y:b0,back:b8,start:b9,leftshoulder:b4,rightshoulder:b5,dpup:-a1,dpdown:+a1,dpleft:-a0,dpright:+a0,"
I'm pretty sure Wine doesn't honor the environment variable SDL_GAMECONTROLLERCONFIG, because it wants to use the Registry key above, which actually I never got it to work.
I wrote a little C program by adapting a large number of examples, mostly from the SDL2 documentation.
// File: js1.c // Startdate: 2022-06-29 // Purpose: Test my C skills, while trying to test SDL2 game controller stuff // References: // http://www.libsdl.org/release/SDL-1.2.15/docs/html/guideinput.html // https://wiki.libsdl.org/SDL_GameControllerMapping // https://stackoverflow.com/q/29589982 // https://generalarcade.com/gamepadtool/ // https://wiki.winehq.org/Useful_Registry_Keys // https://wiki.archlinux.org/title/Gamepad // https://stackoverflow.com/questions/29589982/does-sdl-on-linux-support-more-than-one-gamepad-joystick // Dependencies: // sudo apt-get install libsdl2-dev // Documentation: // try SDL_GAMECONTROLLERCONFIG env var. #include "SDL2/SDL.h" SDL_Joystick *joy; SDL_GameController *ctrl; char *guid[33]; char *mapping; const char *newmapping = "03000000bd12000015d0000010010000,Tomee SNES USB Controller,platform:Linux,a:b2,b:b1,x:b3,y:b0,back:b8,start:b9,leftshoulder:b4,rightshoulder:b5,dpup:-a1,dpdown:+a1,dpleft:-a0,dpright:+a0,"; int main () { if (SDL_Init( SDL_INIT_VIDEO | SDL_INIT_JOYSTICK ) < 0) { fprintf(stderr, "Couldn't initialize SDL: %s\n", SDL_GetError()); exit(1); } SDL_InitSubSystem(SDL_INIT_JOYSTICK); SDL_InitSubSystem(SDL_INIT_GAMECONTROLLER); printf("%i joysticks were found.\n\n", SDL_NumJoysticks() ); printf("The names of the joysticks are:\n"); for( int i=0; i < SDL_NumJoysticks(); i++ ) { joy = SDL_JoystickOpen(i); if (joy) { printf("Opened joystick #%d \"%s\"\n", i, SDL_JoystickNameForIndex(i)); printf("Axes count: %d\n", SDL_JoystickNumAxes(joy)); printf("Button count: %d\n", SDL_JoystickNumButtons(joy)); printf("Ball count: %d\n", SDL_JoystickNumBalls(joy)); SDL_JoystickGetGUIDString(SDL_JoystickGetGUID(joy),*guid,sizeof(guid)); printf("GUID string: \"%s\"\n",guid); if (SDL_IsGameController(i)) { ctrl = SDL_GameControllerOpen(i); mapping = SDL_GameControllerMapping(ctrl); printf("Mapping: \"%s\"\n", mapping); SDL_free(mapping); printf("Trying new mapping.\n"); int r = SDL_GameControllerAddMapping(newmapping); printf("Result: %d.\n", r); } } } return 0; }
And a makefile to make it faster to build.
CC=gcc SRC = $(wildcard *.c) OBJS = $(patsubst %.c, %.o, $(SRC)) NAME = js1 all: $(NAME) .PHONY: clean $(NAME): $(OBJS) $(CC) $(shell pkg-config --cflags sdl2 ) $< $(shell pkg-config --libs sdl2 ) -o $@ clean: rm -f $(OBJS) $(NAME)