Knowledge Base (Posts about troubleshooting)

Upgrading contentdb narrative

bgstack15 — Sun, 10 May 2026 12:55:00 GMT

I run an Internal ContentDB for Luanti users. I know the upstream ContentDB receives updates to its code.

I was months and months behind the upstream codebase, so I decided to try to upgrade my code. The most visible feature that I could use to confirm I had a new version is the display of "AI disclosure" on a package.

The code uses python3, flask, and SqlAlchemy. I've used all these technologies before, but not as deeply as the ContentDB team! It does make me happy that I have approximately the same skillset as that team.

The actions

In a containerized application, you normally run docker pull <imagename>, but there aren't pre-built images for this project. But it's not hard to run docker build . to build it yourself. So I did that, and then tried restarting the container, and it failed.

I failed to keep the exact error messages, but it boiled down to a "circular dependency problem."

I then connected to the running container to manually run alembic commands to upgrade the database. You can see the changes to the database schema. I've never used alembic directly before; I've always used flask db upgrade, and I eventually figured how to do that in this containerized fashion. But I had learned a lot about what I needed to add to the alembic configs so it could find its components. They're scribbled somewhere, but I'm not going to look for them because the flask db upgrade is a better and simpler way.

So the database could get upgraded, but I still cannot run the flask app because of the circular dependency problem. utils kept trying to import app.utils or vice versa. This was the real problem of my upgrade, which was solved after I looked through the code history and found Commit fa3697a: Split up app/utils/__init__.py to reduce issues. The problem still happens though, because of how the container loads the app from a mounted volume, into the location where it will run.

That file, app/utils/__init__.py, was deleted, but when the container startup script copies all files in, it doesn't remove files. It's one of those classic computer science problems, where it won't clean up a deleted file. It doesn't know that file exists, and doesn't care. It's not rsync with a --delete-after parameter. Therefore, I had to go into the container and delete app/utils/__init__.py. Then I could run that flask db upgrade from the utility script, and then start up the application.

The reason that file was deleted, was because you cannot have a python module import another module of the same name. That means directory name, even if it's a different directory. So utils cannot import app.utils, and vice versa. So I'm glad the ContentDB team rewrote it so one of them no longer has a module (__init__.py) directly, to choke.

And now for some reason, every time I restart the container, I have to re-run the flask db upgrade step. That's weird, but I can live with it.

Troubleshooting Mineclonia server lag, and podman

bgstack15 — Sun, 15 Mar 2026 13:24:00 GMT

Random Troubleshooting narrative

My new multiplayer Mineclonia survival world was suffering from severe lag. I wondered if it was related to the old CentOS 7-era hardware I run it on. Investigating showed some log entries about redstone events.

2026-03-05 16:00:12: ERROR[Server]: [mcl_redstone]: Maximum number of queued redstone events (65535) exceeded, deleting all of them.

This was a new world, with under 10 hours of gameplay, and we didn't have any redstone contraptions at all. I tried adjusting the incorrectly-labeled mcl_redstone_max_events (it was listed as mcl_redstone_event_max), which then let it actually count out all its redstone events, but then the errors occurred about max lag. So the server felt like it performed in the same bad way.

2026-03-05 16:00:10: WARNING[Server]: Server: Maximum lag peaked at 2.109 (steplen=0.05)

I found my old desktop computer, which was already running Rocky Linux 9, and set up podman, to conduct my first experiment with rootless podman. I've heard it supports compose compatibility, and I don't want to rewrite my docker-compose file into systemd units yet. I like the single file to control it all, rather than rewriting it in a different paradigm across 3 different service (unit) files.

I experimented with adjusting userns_mode and /run mounted from empty volumes. The game container fails because of s6 errors: It cannot chown various things. So I gave up on using rootless podman, and gave user luanti access to run sudo podman-compose. Then, the application would operate just fine.

So, with the game running, I connected and observed the same lagginess as before, even on the beefier host system. I investigated the game settings, and found an experimental flag I had turned on. By setting mcl_liquids_enable = false again, the game server operated normally.

Follow-up

Deep under the main play area, I found some large waterfalls. I could see how some performance-inhibiting waterfall calculations operated by the Mineclonia lua code could cause problems with that. Screenshot of gameplay

Screenshot of some waterfalls

Fixing Lego Universe client after OS updates November

bgstack15 — Wed, 19 Nov 2025 14:22:00 GMT

After the November OS updates, I could not run the client for Lego Universe anymore! It would fail with this mysterious error:

0024:err:winediag:ntlm_check_version ntlm_auth was not found. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
0024:err:ntlm:ntlm_LsaApInitializePackage no NTLM support, expect problems
0024:err:module:loader_init "Awesomium.dll" failed to initialize, aborting
0024:err:module:loader_init Initializing dlls for L"C:\\lego-universe-client\\legouniverse.exe" failed, status c0000005

The ntlm stuff is bogus; that warning has shown up for a long time, and also this game doesn't use ntlm. The real problem is its inability to load Awesomium.dll, which is in the current directory with legouniverse.exe. Some brief Internet searching didn't show anything interesting with wine 10.0~repack-11 and loading dlls, so I decided to try downgrading. Because I couldn't remember the complicated commands related to fancy downgrading using aptitude, I eventually derived the command for apt, after preparing my snapshot list.

For file /etc/apt/sources.list.d/snapshot.list.

deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20251001T203719Z/ unstable main

The command that got the working wine version.

sudo apt --allow-downgrades install wine=10.0~repack-6 wine32=10.0~repack-6 wine64=10.0~repack-6 libwine=10.0~repack-6 libwine:i386=10.0~repack-6 fonts-wine=10.0~repack-6

And then for some reason file /usr/share/wine/nls/locale.nls was absent (probably the wine-common=10.0~repack-11 being removed causes a conflict with the installation of libwine=10.0~repack-6), so then I did the same command as above but with reinstall.

And now my game runs again. Hopefully this will be solved by a future wine rebuild in Debian.

Screenshot of Lego Universe client running again

Troubleshooting Darkflame Lego Universe logins

bgstack15 — Wed, 22 Oct 2025 12:24:00 GMT

I spent about 3 hours troubleshooting my Darkflame Lego Universe server. My users could not connect. I turned the server back on after a few months of being offline, and nobody could connect. I wanted to test the new Chat Web API, and yes, I know it was limited to localhost specifically.

So I had 2 problems to solve: use the chat web api to list active players, and also fix the server so people can use it.

I could not confirm inside the docker container that the port was listening. There was no ss or netstat installed, so I searched and found a guide that provided this useful snippet.

Librewolf 136.0 upgrade problems

bgstack15 — Sat, 05 Apr 2025 13:27:00 GMT

During OS updates for April, I ran into a problem with Librewolf. This was for Librewolf 136.0, but it also happened back with 135.0 and I was able to revert when I upgraded to that. I decided I needed to solve the problems, because now they were multiple.

The first problem is that the new librewolf settings/prefs in the package file somehow set the "delete my history" checkbox even if it was unselected previously. So the first time you close the browser after upgrading to 136.0 (or 135.0), it deletes all your history, including links, cookies, and saved logins. I wanted to keep mine, and I was aware of this problem so I was able to uncheck those and keep going.

The second problem was that the search suggestions were not working as intended/(configured?). I want to type in a fragment of a url, such as "mtui" and I want it to suggest any open tabs with that expression in the url, or from my history. It didn't show any. The fix was complicated, but understandable.

The fix

I set up a new profile (by copying in a known good/blank one I've established in the past). Then I ran the browser, made sure the "delete my history" settings were unchecked. Then close the browser, and copy in these files to the new profile:

cd ~/.librewolf-off-2025-03-30/3ig9z5jv.default-release-1.off-2025-03-30-v136.0
cp -pi places.sqlite cookies.sqlite storage.sqlite sessionstore.jsonlz4 favicons.sqlite credentialstate.sqlite logins.json logins-backup.json cert9.db key4.db ~/.librewolf/liwjg22ii/
cp -pr chrome/ ~/.librewolf/liwjg22ii/

That last directory is for my userChrome fixes.

And then starting up Librewolf, and it worked. I lost my localStorage for one site but that was easy to re-add.

Hm, I guess I didn't keep it simple on this one, sorry.

Additional notes

For prefs.js.

privacy.resistFingerprinting.exemptedDomains *.amazon.com,*.themoviedb.org,*.freefilesync.org,freefilesync.org,finance.yahoo.com

References

Weblinks

Virt-manager fails with packet bytes received from server too large

bgstack15 — Wed, 01 May 2024 12:55:31 GMT

Recently, I had some problems connecting to one of my libvirt qemu hosts. When I ran virt-manager from my desktop which uses a qemu+ssh://root@vm3 connection with ssh key, I got this error:

Unable to connect to libvirt qemu+ssh://root@vm3/system.

packet 167772156 bytes received from server too large, want 33554432

Verify that the 'libvirtd' daemon is running on the remote host.

Libvirt URI is: qemu+ssh://root@vm3/system

Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/connection.py", line 923, in _do_open self._backend.open(cb, data) File "/usr/share/virt-manager/virtinst/connection.py", line 171, in open conn = libvirt.openAuth(self._open_uri, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/libvirt.py", line 147, in openAuth raise libvirtError('virConnectOpenAuth() failed') libvirt.libvirtError: packet 167772156 bytes received from server too large, want 33554432

After a lot of searching the Interwebs, I have learned the problem. It was hilarious and pointless.

I confirmed my ssh key still lets me ssh to root@vm3. (So sue me)

I confirmed libvirt is indeed running on the host (systemctl status -l libvirt) and noticed I had a bad image path, to a BSD/ directory I must have deleted on my network share. So I fixed that (already forgot the command but was probably virt edit-pool or something), but that didn't fix my issue today.

I logged to my good host (vm1) and bad host (vm3) just to tty and saw there's an additional newline when I log in to vm3. So I traced that down. In my .bashrc there was a command to run my "bash profile" /usr/bin/bp which I install via OS package on every system of mine. There's no reason it should be printing a newline. I had to set verbosity very high.

I eventually found that because file /var/spool/mail/root exists on vm3, it was printing the undefined variable $MAILMSG, and echo always throws in a newline.

Now, my root mail spool hasn't been updated since 2018, so I've had the one error message about something lame sitting there for 6 years, but it was only just now that virt-manager on Devuan Ceres fails to connect.

I'm guessing it has something to do with a newer version of virt-manager and older (unsupported now, apparently) version of libvirt on CentOS 7.

OS Updates May 2023, and the Aftermath

bgstack15 — Thu, 04 May 2023 13:13:46 GMT

This month's OS updates were brutal! One small good thing: CentOS 7 was very mild: no kernel updates.

Things got rocky (no pun intended; I don't use Rocky Linux; I went with AlmaLinux for my CentOS 8 replacement) when I was validating my main file server. It operates way too many services, including my main wireguard "server." (In Wireguard, everything is merely a peer. I just make this one a peer to everything else because I like star topology.)

Well, I first realized that my wireguard routing (firewalld masquerading) was not working. Come to find out, Firewalld was malfunctioning. I spent a lot of time troubleshooting a weird error:

firewalld[903]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: No such file or directory

I troubleshot nftables, and python. There doesn't appear to be a binary named python-nftables. Some Ubuntu chap found a similar problem which he solved by manually adding the nft tables and chains somehow missing (that I assume firewalld should be adding on its own). I then wasted way too much time trying to parse the supposedly bad json blob from the logs with jq, and looping through to get my easy nft commands for chain and table. Unfortunately I didn't figure that out and ended up just running those commands manually. They didn't help.

So now I was holding multiple broken pieces of firewalld, wireguard, oh, and radicale (my calendar solution). Radicale was throwing errors about a read-only filesystem.

So, after trying to revert firewalld to just use trusty old iptables, which went sideways even faster than nftables, and searching those errors, I discovered that dmesg was throwing a fascinating and scary message:

missing module BTF, cannot register kfuncs

Which reliably shows up when restarting firewalld in its broken state. It was at this point I guess that this AlmaLinux 8 system had received a kernel update, and sure enough, it had. It was running 6.3.1 and the previous boot was 6.2.9. So I decide to just reboot. I visit the console (because I've never bothered to set up a cool network kvm device) and ensure I interrupt grub because who can be bothered to learn the new, current way to configure grub to pick a specific menu entry? I booted into the previous kernel entry, and then firewalld worked, and its masquerade (ip routing) function worked. So wireguard was back to functioning.

So by the way, this flapping of my nfs server affected my jellyfin instance which then coughed up some old entries from its database for some reason and spat out .nfo files for long-gone files once nfs got reestablished. Whatever. That's the least of my worries.

So now, I turned my attention to radicale. It was spewing a silly error:

radicale[939]: [939/Thread-3] [ERROR] An exception occurred during PROPFIND request on '/bgstack15/': [Errno 30] Read-only file system: '/var/lib/radicale/collections/.Radicale.lock'

I'm an old hat, classically trained (as I like to say), so I dropped SELinux into permissive mode, which didn't help. I then ran the radicale server on the command line, which did remove that error. It still operated incredibly slowly, so I eventually gave up on that. I saw in the radicale.service file an entry, ReadWritePaths= which led me to read about that in the systemd man pages. My collections are stored in there, but under a symlink. So I ran:

systemctl edit radicale.service
[Service]
ReadWritePaths=/the/readlink-f/path/to/my/collections

And then systemctl daemon-reload and restarted the app. And now, it didn't throw that error but it still ran incredibly slowly on my calendar(s). Well, come to find out, it wanted to reindex all 9,000 entries in my calendar, which takes a while. But once it had finished that, it was able to then serve them fast enough for my calendar front-end webapp.

And to think I say I enjoy this stuff!

Wine error: Couldn't get desktop folder

bgstack15 — Mon, 16 Jan 2023 14:11:57 GMT

I copied a wine prefix to a different user, and was then unable to start my application. I struggled to get the app to work, and started backtracking to see if I could get more info. I learned that even explorer.exe was not working in Wine. I don't control that at all! That's a built-in component of Wine. So if that doesn't work, I've really messed something up!

WINEPREFIX=/home/bgstack15/.wine WINEARCH=win64 wine explorer.exe

When explorer.exe doesn't work in wine, you KNOW you have a problem! Here is the output of the error message.

Unhandled exception: page fault on read access to 0x00000000 in 32-bit code (0x0040be65).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:006b GS:0063
 EIP:0040be65 ESP:0078f7a0 EBP:0078fe68 EFLAGS:00010206(  R- --  I   - -P- )
 EAX:80070057 EBX:008a38c0 ECX:00000000 EDX:7149b720
 ESI:0089c980 EDI:0078f854
Stack dump:
0x0078f7a0:  0089cc48 00000000 00000000 0089ca88
0x0078f7b0:  00000000 00000000 00000280 00000018
0x0078f7c0:  0001005c 00000000 00400000 00000000
0x0078f7d0:  00000064 00891cc2 000000c4 00000000
0x0078f7e0:  7bc27600 688df3d0 00000000 00000000
0x0078f7f0:  008a1ffc 008a2004 01322014 0078f820
Backtrace:
=>0 0x0040be65 IShellFolder_Release+0x9(This=<internal error>) [Y:\include\shobjidl.h:1707] in explorer (0x0078fe68)
  1 0x0040be65 make_explorer_window+0x82a(params=<internal error>) [Y:\programs\explorer\explorer.c:372] in explorer (0x0078fe68)
  2 0x0040be65 wWinMain+0x995(hinstance=<couldn't compute location>, previnstance=<couldn't compute location>, cmdline=<couldn't compute location>, cmdshow=<couldn't compute location>) [Y:\programs\explorer\explorer.c:927] in explorer (0x0078fe68)
  3 0x00410568 wmain+0xa8(argc=<couldn't compute location>, argv=<couldn't compute location>) [Y:\dlls\msvcrt\crt_wwinmain.c:55] in explorer (0x0078fef8)
  4 0x004104b8 wmainCRTStartup+0x68() [Y:\dlls\msvcrt\crt_wmain.c:60] in explorer (0x0078ff30)
  5 0x7b6293e0 in kernel32 (+0x293e0) (0x0078ff48)
  6 0x7bc5ca37 in ntdll (+0x5ca37) (0x0078ff5c)
  7 0x7bc5d258 in ntdll (+0x5d258) (0x0078ffec)
0x0040be65 wWinMain+0x995 [Y:\programs\explorer\explorer.c:927] in explorer: movl   0x0(%ecx),%eax
Unable to access file 'Y:\programs\explorer\explorer.c'
Modules:
Module  Address         Debug info  Name (60 modules)
PE  00400000-0048f000   Dwarf-4         explorer
PE  00b40000-00d5f000   Deferred        rpcrt4
PE  016a0000-01b2c000   Deferred        ole32
PE  01b30000-01bb6000   Deferred        winex11
PE  01cb0000-020ec000   Deferred        actxprxy
PE  62500000-628ff000   Deferred        oleaut32
PE  63080000-630aa000   Deferred        zlib1
PE  64a80000-64ad8000   Deferred        win32u
PE  65680000-65900000   Deferred        msvcrt
PE  66080000-66180000   Deferred        shlwapi
PE  66640000-6665c000   Deferred        version
PE  667c0000-66809000   Deferred        shcore
PE  67500000-67552000   Deferred        imm32
PE  67bc0000-6805e000   Deferred        comctl32
PE  684c0000-68610000   Deferred        combase
PE  68880000-68d46000   Deferred        user32
PE  69840000-69938000   Deferred        advapi32
PE  6aac0000-6ada1000   Deferred        ucrtbase
PE  6bbc0000-6bc59000   Deferred        sechost
PE  6bc80000-6bcec000   Deferred        explorerframe
PE  6da80000-6dc9c000   Deferred        gdi32
PE  70d80000-70e23000   Deferred        uxtheme
PE  71400000-720ab000   Deferred        shell32
PE  7b000000-7b51a000   Deferred        kernelbase
PE  7b600000-7b757000   Dwarf-4         kernel32
PE  7bc00000-7beba000   Dwarf-4         ntdll
ELF 7d000000-7d005000   Deferred        <wine-loader>
ELF 7f416000-7f41e000   Deferred        libxfixes.so.3
ELF 7f41e000-7f42b000   Deferred        libxcursor.so.1
ELF 7f42b000-7f440000   Deferred        libxi.so.6
ELF 7f440000-7f445000   Deferred        libxcomposite.so.1
ELF 7f445000-7f454000   Deferred        libxrandr.so.2
ELF 7f454000-7f462000   Deferred        libxrender.so.1
ELF 7f462000-7f469000   Deferred        libxxf86vm.so.1
ELF 7f469000-7f46e000   Deferred        libxinerama.so.1
ELF 7f46e000-7f47d000   Deferred        libmd.so.0
ELF 7f47d000-7f494000   Deferred        libbsd.so.0
ELF 7f494000-7f49b000   Deferred        libxdmcp.so.6
ELF 7f49b000-7f4a0000   Deferred        libxau.so.6
ELF 7f4a0000-7f4ce000   Deferred        libxcb.so.1
ELF 7f4ce000-7f61f000   Deferred        libx11.so.6
ELF 7f61f000-7f635000   Deferred        libxext.so.6
ELF 7f64d000-7f6d7000   Deferred        winex11.so
ELF 7f6f7000-7f723000   Deferred        libexpat.so.1
ELF 7f723000-7f772000   Deferred        libfontconfig.so.1
ELF 7f772000-7f795000   Deferred        libbrotlicommon.so.1
ELF 7f795000-7f7b2000   Deferred        libz.so.1
ELF 7f7b2000-7f881000   Deferred        libfreetype.so.6
ELF 7f881000-7f986000   Deferred        libm.so.6
ELF 7fe0e000-7fe18000   Deferred        libuuid.so.1
ELF 7fe18000-7fe55000   Deferred        libpng16.so.16
ELF 7fe6d000-7ffe0000   Deferred        win32u.so
ELF f7c00000-f7e28000   Deferred        libc.so.6
ELF f7e2f000-f7ede000   Deferred        ntdll.so
ELF f7ee0000-f7ee5000   Deferred        libpthread.so.0
ELF f7ee5000-f7eea000   Deferred        libdl.so.2
ELF f7eea000-f7ef4000   Deferred        libgtk3-nocsd.so.0
ELF f7ef7000-f7f05000   Deferred        libbrotlidec.so.1
ELF f7f05000-f7f0a000   Dwarf           libwine.so.1
ELF f7f14000-f7f49000   Deferred        ld-linux.so.2
Threads:
process  tid      prio    name (all IDs are in hex)
00000020 start.exe
    00000024    0     
00000038 services.exe
    0000003c    0     
    00000040    0     wine_rpcrt4_server
    0000004c    0     wine_rpcrt4_io
    00000050    0     wine_threadpool_worker
    00000064    0     wine_rpcrt4_io
    00000084    0     wine_rpcrt4_io
    00000090    0     wine_rpcrt4_io
    000000b0    0     wine_rpcrt4_io
    000000d8    0     wine_threadpool_worker
    000000dc    0     wine_threadpool_worker
    000000e0    0     wine_threadpool_waitqueue
    00000114    0     wine_rpcrt4_io
00000044 svchost.exe
    00000048    0     
    00000054    0     
    00000058    0     wine_sechost_service
0000005c winedevice.exe
    00000060    0     
    00000068    0     
    0000006c    0     wine_sechost_service
    00000070    0     
    00000074    0     
    00000078    0     
    000000e4    0     
0000007c winedevice.exe
    00000080    0     
    00000088    0     
    0000008c    0     wine_sechost_service
    00000094    0     
    00000098    0     
    0000009c    0     
    000000a0    0     
    000000a4    0     
000000a8 plugplay.exe
    000000ac    0     
    000000b4    0     
    000000b8    0     wine_sechost_service
    000000bc    0     wine_rpcrt4_server
000000e8 conhost.exe
    000000ec    0     
000000f0 (D) C:\windows\syswow64\explorer.exe
    000000f4    0 <== 
    00000138    0     wine_threadpool_worker
    00000154    0     
000000f8 explorer.exe
    000000fc    0     
    00000100    0     
    00000104    0     wine_rpcrt4_server
    00000134    0     wine_rpcrt4_io
    0000013c    0     wine_threadpool_worker
    00000140    0     wine_rpcrt4_io
    00000144    0     wine_rpcrt4_io
    00000148    0     wine_rpcrt4_io
0000010c rpcss.exe
    00000110    0     
    00000118    0     
    0000011c    0     wine_sechost_service
    00000120    0     wine_rpcrt4_server
    00000124    0     wine_rpcrt4_server
    00000128    0     wine_rpcrt4_io
    0000012c    0     wine_threadpool_worker
    00000130    0     wine_rpcrt4_io
System information:
    Wine build: wine-8.0-rc1 (Debian 8.0~rc1~repack-1)
    Platform: i386 (WOW64)
    Version: Windows 7
    Host system: Linux
    Host version: 6.1.0-1-amd64

I finally noticed this little message on the command line:

00fc:err:wineboot:ProcessStartupItems Couldn't get desktop folder.

Who would have thought that this would be significant? When I bothered to read it, I checked:

[bgstack15@pcb-006|/home/bgstack15/.wine/drive_c/users]$ ll bgstack15/
total 4
drwxr-xr-x 4 bgstack15 bgstack15 4096 Jan 10 21:00 AppData/

When most "C:\users\bgstack15\" directories resemble:

$ ls -l ~/.wine/drive_c/users/bgstack15
total 4
drwxr-xr-x 4 bgstack15 bgstack15 4096 Jan 10 21:00 AppData/
lrwxrwxrwx 1 bgstack15 bgstack15   21 Jan 10 21:10 Desktop -> /home/bgstack15/Desktop/
lrwxrwxrwx 1 bgstack15 bgstack15   23 Jan 10 21:10 Documents -> /home/bgstack15/Documents/
lrwxrwxrwx 1 bgstack15 bgstack15   23 Jan 10 21:10 Downloads -> /home/bgstack15/Downloads/
lrwxrwxrwx 1 bgstack15 bgstack15   19 Jan 10 21:10 Music -> /home/bgstack15/Music/
lrwxrwxrwx 1 bgstack15 bgstack15   22 Jan 10 21:10 Pictures -> /home/bgstack15/Pictures/
lrwxrwxrwx 1 bgstack15 bgstack15   20 Jan 10 21:10 Videos -> /home/bgstack15/Videos/

So I fixed this weird wine error with:

cd ~/.wine/drive_c/users/bgstack15 ; for word in Desktop Documents Downloads Music Pictures Videos ; do ln -s ~/"${word}" . ; done

Which means that I made the symlinks that go to those real directories, and then Wine would work! This happened because the wineprefix came from a different user in Linux, so even the drive C user directory was different, and somehow it just didn't bother making a fully fleshed-out user directory there.

Jellyfin Chromecast selection never appears

bgstack15 — Thu, 22 Sep 2022 13:22:38 GMT

I had a problem with my production Jellyfin server a few days ago.

User "jellyfin" was unable to pull up the menu of available Chromecast targets, so Chromecast was entirely unavailable. Also, new logins for the user would not always go through. The server logs showed:

[2022-09-19 09:09:10.415 +00:00] [INF] [43] Emby.Server.Implementations.HttpServer.WebSocketManager: WS "258.1.15.15" request
[2022-09-19 09:09:10.526 +00:00] [ERR] [13] Microsoft.EntityFrameworkCore.Query: An exception occurred while iterating over the results of a query for context type '"Jellyfin.Server.Implementations.JellyfinDb"'."
""System.InvalidOperationException: The data is NULL at ordinal 5. This method can't be called on NULL values. Check using IsDBNull before calling.

A search of the Internet and more specifically github jellyfin showed a promising suggestion:

sbstp commented on Aug 7, 2021 @sweisgerber I actually found a better fix, this issue came back for me. The fix was to go into jellyfin.db with sqlite3 (command line) or sqlitebrowser (gui) and delete the rows of CustomItemDisplayPreferences where value is null. Make sure to backup the db before doing that just in case.
DELETE FROM CustomItemDisplayPreferences WHERE Value IS NULL;

Somehow the settings in the database table got corrupted. This is probably a failure of the application logic and not something done by a user. But it's something I can live with! Thankfully there are smarter people on the Internet who experienced this before I did.

Random troubleshooting story

bgstack15 — Fri, 24 Dec 2021 17:10:15 GMT

While programming my new (and still as-of-yet unpublished) python program that uses ldap authentication, I ran into a weird issue. I logged in to the web app, and got an ldap error! The ldap server was refusing connections. I logged into my freeipa servers, and investigated. sudo ipactl status showed all clear:

Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

Both servers showed the same results. So I tried my ldap login again and it failed again. So this time I ran an ldapsearch command on my client, and discovered that indeed, I couldn't get an ldap connection on dns2. Host dns1 was still working. The cool interactive flask troubleshooter session in the web page told me the app had picked dns2 (from my kerberos TXT nslookups) of the available pool. So, that's why the app was failing.

So, back to dns2. I then checked the directory service directly:

sudo journalctl -n200 -u dirsrv@IPA-EXAMPLE-COM.service

The output indicated the system was out of disk space! And sure enough, my root partition / was completely filled. Apparently my SpiderOakONE config directory takes up 50GB of space. So I moved it to /home partition which had way more space, and restarted ipa.