Knowledge Base

Preserving for the future: Shell scripts, AoC, and more

Walk through my vimrc

bgstack15

2021-02-14 08:48

I recently added a few more settings to my .vimrc file. I remember thinking years ago that, "Oh, I'll just use the vim defaults, because of all the systems I'll touch as a sysadmin." Yeah, over time as I realize the defaults are different across distros or major releases, or as I see the defaults stink, I add a few lines to my preferred vimrc. Here is my entire .vimrc in my own custom dotfiles implementation (which is so convoluted, but I'll save that for another day).

" Reference https://stackoverflow.com/questions/1878974/redefine-tab-as-4-spaces/1878984#1878984
set indentkeys= scrolloff=0 nocindent mouse= modeline ic
" for X mouse support: set mouse=a
set tabstop=3 shiftwidth=3 softtabstop=3 expandtab
set noincsearch hlsearch " search options
set ruler laststatus=2 " status bar at bottom
syntax on
au BufRead,BufNewFile,BufCreate,BufEnter */debian/rules set noexpandtab
au BufRead,BufNewFile,BufCreate,BufEnter */Makefile set noexpandtab
au BufRead,BufNewFile,BufFilePre,BufCreate,BufEnter *.md set filetype=markdown
au FileType yaml set tabstop=2 softtabstop=0 expandtab shiftwidth=2 smarttab smartindent
au FileType markdown set sw=4 ts=4 sts=4

Explanation

set indentkeys= scrolloff=0 nocindent mouse= modeline ic

I'm shoving multiple things into one line, because why burden my storage down with a few extra bytes? I undefine indentkeys because I'll add indentations myself, thank you. Scrolloff numerical value determines how many lines above and below the current line will always be visible. That is, if scrolloff=5, then as you scroll down, your cursor will not get closer to the bottom of the visible window than 5 lines. I find this visually distracting. Nocindent. Just no indents, even in C mode. Mouse=. Just disable interpreting mouse input within vim. I'm in a terminal for a reason. modeline (true) turns on interpreting a line within the file whose contents include, e.g., # vim: set ts=3 sw=3 sts=3 et: ic (true) enable case-insensitive search. I've noticed I usually want that on these days.

set tabstop=3 shiftwidth=3 softtabstop=3 expandtab

Yes, I use 3 by default, mostly to differentiate my original works from others'. And I like spaces instead of tabs. So much for saving bytes on my storage.

set noincsearch hlsearch " search options

Noincsearch disables the interactive "take cursor to first match as you're typing" feature. I remember that this is the default in Ubuntu 16.04 and its behavior just irked me. I'll tell vim when to update the screen (with a return key), dang it! Highlight all search matches.

set ruler laststatus=2 " status bar at bottom

Always show the cursor line number and column. I like to see that. laststatus=2 always show the status line (where the ruler values are shown). I like to see the filename and cursor location at all times.

syntax on

Use syntax highlighting. Works fine unless you're in yaml with mild depth levels or long quoted strings.

au BufRead,BufNewFile,BufCreate,BufEnter */debian/rules set noexpandtab
au BufRead,BufNewFile,BufCreate,BufEnter */Makefile set noexpandtab

So I actually discovered once how to disable the expandtab when in these files where the tab character is actually important.

au BufRead,BufNewFile,BufFilePre,BufCreate,BufEnter *.md set filetype=markdown

Perhaps this one is here because I had some environment that didn't have a syntax identifier for *.md even though markdown was one of the available syntaxes. I guess it's here just in case!

au FileType yaml set tabstop=2 softtabstop=0 expandtab shiftwidth=2 smarttab smartindent

Set tab depth of 2, and all the "smart" tab logic when in yaml.

au FileType markdown set sw=4 ts=4 sts=4

Set tab depth to 4 when in markdown, so that pressing the tab key takes me to the indentation required for the >pre< equivalent for markdown. Tabs aren't used very much in markdown so this seems acceptable to me. Check in with me in a few years to see how much more my preferred vimrc contains.

View details of a certificate

bgstack15

2021-02-10 08:38

Comments

Sometimes you need to inspect your certificate or certificate chain presented by a server. Here are several ways to do that.

Inspect certificate with web browser

If the certificate or cert chain in question is being used to present a web site, you can visit the site in a browser, such as Firefox. Visit your site, and select the padlock icon in the address bar beside the URL. Select the arrow pointing to the right, and then select the "More information" link. On the new modal window that appears, select the "View certificate" button. Firefox will show you the certificate and its chain (if Firefox knows about it, or the web server presents the chain) for your inspection.

Inspect certificate chain with openssl command

The openssl reference implementation is available for both Windows and Linux through various means. Sufficiently high versions of openssl (>=1.0.1a) will be able to perform these tasks. Openssl can make network connections to https sites, and can also inspect files.

Inspect certificate chain presented by web server

The simplest way is to search for the descriptors provided by openssl natively with s_client. You can make sure the number and order of certificates is what you expect to make a complete intermediate-server cert chain.

$ echo "" | openssl s_client -connect xkcd.com:443 -showcerts 2>&1 | grep -iE '[si]:'
 0 s:C = US, ST = California, L = San Francisco, O = "Fastly, Inc.", CN = i.ssl.fastly.net
   i:C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2
 1 s:C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA

A well-behaved web server will present, at a minimum, the server certificate and all intermediate certificates. Serving the root certificate is optional, because well-behaved clients will already trust the root certificate. You can also dump the whole chain to a file, so you can split it up and read each certificate with the commands farther down on this page.

$ echo "" | openssl s_client -connect xkcd.com:443 -showcerts > certchain.pem

Inspect certificate in a file

Openssl will only read one certificate per file! If you have a certificate chain in a file, split it into multiple files before running these commands.

$ openssl x509 -in cert1.pem -noout -subject -issuer -dates -fingerprint -serial
subject=C = US, ST = California, L = San Francisco, O = "Fastly, Inc.", CN = i.ssl.fastly.net
issuer=C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2
notBefore=Jun 16 18:30:07 2020 GMT
notAfter=Jul 28 18:43:49 2022 GMT
SHA1 Fingerprint=7A:63:0B:5F:F6:72:E8:4D:70:B7:8B:45:1D:CF:27:94:AF:2C:F1:9A
serial=0F40947DD38354936AD7D7D0

Symlink forest generator

bgstack15

2021-02-06 09:22

Comments

As part of my Web gallery solution, I wrote some logic that creates a YYYY/MM directory creator that places symlinks to my original files within the new directory structure. The links are added to the relevant directory based on exif metadata if possible, and file timestamp as a last resort. Check out the source files at my gitlab space:

generate.py front end for cli
genlib.py library

Here is my main sample invocation. I used this a lot while testing.

./generate.py -i /mnt/bgstack15/Backups/Images/Photos/camera/2018/ -o /mnt/public/www/gallery/my2018 -n -d2 -x 'October' -s --nr

The help text is generated with ArgumentParser.

$ ./generate.py --helpusage: generate.py [-h] [-d [{0,1,2,3,4,5,6,7,8,9,10}]] [-v] [-n | -a] [-z | --nz] [-c | -s | -m] [-r | --nr] -i
                   INDIR -o OUTDIR [-x EXCLUDE]

Make symlink forest for images

optional arguments:
  -h, --help            show this help message and exit
  -d [{0,1,2,3,4,5,6,7,8,9,10}], --debug [{0,1,2,3,4,5,6,7,8,9,10}]
                        Set debug level
  -v, --version         show program's version number and exit
  -n, --dryrun          Make no changes (default)
  -a, --apply           Actually make changes
  -z, -0, --zeropad     Zero pad month directories (default)
  --nz, --nozeropad, --no-zeropad
                        Do not zero pad
  -c, --copy            Copy files instead of symlinks. Not recommended.
  -s, --symlink         Make symlinks (default)
  -m, --move            Move files. Not recommended.
  -r, --relative        Make relative symlinks if possible.
  --nr, --norelative, --no-relative
                        Make absolute symlinks (default)
  -i INDIR, --indir INDIR
  -o OUTDIR, --outdir OUTDIR
  -x EXCLUDE, --exclude EXCLUDE
                        Exclude pathname matches. Can be used multiple times.

For posterity and in case my gitlab ever dies, here are their contents at the time of this writing.

generate.py

#!/usr/bin/env python3
# Startdate: 2021-01-29 15:35
# Dependencies:
#    devuan-req: python3-exifread
# Reference:
#    https://stackoverflow.com/questions/1192978/python-get-relative-path-of-all-files-and-subfolders-in-a-directory
#    https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory
#    https://stackoverflow.com/questions/600268/mkdir-p-functionality-in-python
from sys import stderr
from exifread import process_file
from os import walk, path, stat, symlink
from pathlib import Path
from datetime import date
from shutil import copy2, move

try:
   if debuglevel is None:
      debuglevel = 0
except:
   debuglevel = 0

# Functions
def eprint(*args, **kwargs):
   print(*args, file=stderr, **kwargs)

def list_files(indir,relative=False,debuglevel=debuglevel, excludes=None):
   x = 0
   f = []
   if debuglevel >= 9:
      eprint(f"Listing all files underneath {indir}")
      if len(excludes) > 0:
         eprint("While excluding path matches:",excludes)
   for (dirpath, dirnames, filenames) in walk(indir):
      for file in filenames:
         x += 1
         fullpath=path.join(dirpath,file)
         relpath=path.join(path.relpath(dirpath,indir),file)
         use = True
         use_pattern = ""
         for e in excludes:
            # simple match, no regex
            if e in relpath:
               use = False
               use_pattern = e
               break # short-circuit
         if use:
            if debuglevel >= 9: eprint(x,fullpath,relpath)
            f.append(relpath if relative else fullpath)
         else:
            if debuglevel >= 9: eprint(x,fullpath,f"ignored per {use_pattern}")
   return f

def limit(listf=None,filters=["image","video"]):
   # available types are "image", "video"
   newlist = []
   for f in listf:
      _, ext = path.splitext(f)
      ext = ext[1:].lower()
      if ext in ["jpg","jpeg","png","svg","tif","tiff","gif"] and "image" in filters:
         newlist.append(f)
      elif ext in ["mp4","webm","avi"] and "video" in filters:
         newlist.append(f)
   return newlist

def get_file_YM(tf,debuglevel=debuglevel,zero_pad=False):
   Y = 0 ; M = 0
   #eprint(f"Debuglevel {debuglevel}")
   try:
      # read exif data
      with open(tf,'rb') as f:
         tags = process_file(f, details=False, stop_tag="Image DateTime")
      if "Image DateTime" in tags:
         #if debuglevel >= 5: eprint(tf,tags["Image DateTime"])
         Y = str(tags["Image DateTime"]).split(' ')[0].split(':')
         if Y == "['']" or Y == ['']:
            Y = ['0','0']
         M = Y[1] ; Y = Y[0];
      # Any other image timestamps could be used here
      #eprint(tags)
   except KeyboardInterrupt:
      return -1, -1
   except:
      if debuglevel >= 1: eprint(f"Unable to extract any exif data for {tf}")
   if int(Y) == 0 or int(M) == 0:
      # need to just use timestamp
      try:
         ts = date.fromtimestamp(stat(tf).st_mtime)
      except KeyboardInterrupt:
         return -1, -1
      except:
         # no timestamp available from filesystem?
         if debuglevel >= 1: eprint(f"No timestamp available for {tf}")
      if int(Y) == 0: Y = ts.year
      if int(M) == 0: M = ts.month
   if zero_pad:
      M = str(M).zfill(2)
   else:
      M = str(int(M))
   if debuglevel >= 3:
      eprint(f"{tf} {Y}/{M}")
   return Y, M

def make_YM_forest(outdir, flist, action = "symlink", dryrun=True, debuglevel=debuglevel, zero_pad=False, relative_symlinks = False):
   """
   For each file in flist, [action] it to outdir, sorting into YYYY/MM subdirs based on exif metadata or timestamp.
   Action may be one of ['symlink', 'copy', 'move'].
   If relative_symlink = True, then make relative symlinks if possible.
   """
   result = 0
   stop = False

   # validate input
   if action not in ['symlink', 'copy', 'move']:
      eprint("make_YM_forest action may be one of ['symlink', 'copy', 'move'].")
      return -1

   # Learn all directories to make
   # and also cache the files with Y/M value
   fdict = {}
   destdirs = []
   for f in flist:
      try:
         Y, M = get_file_YM(f,debuglevel=debuglevel,zero_pad=zero_pad)
         if Y == -1 and M == -1:
            #eprint("Stopping due to keyboard variant 2.")
            stop = True
            break
      except KeyboardInterrupt:
         #eprint("Stopping due to keyboard variant 1.")
         stop = True
         break
      if zero_pad:
         M = str(M).zfill(2)
      else:
         M = str(int(M))
      YM = f"{Y}/{M}"
      if YM not in destdirs: destdirs.append(YM)
      fdict[f] = YM
   # finish the for f in flist

   # short-circuit if keyboard action happened in one of the nested functions
   if stop:
      return -1

   # Make directories
   for d in destdirs:
      dd = path.join(outdir,d)
      if debuglevel >= 3:
         eprint(f"Make dir: {dd}")
      if not dryrun:
         Path(dd).mkdir(parents=True, exist_ok=True)

   # Make symlinks
   for f in fdict:
      ff = f
      basename = path.basename(ff)
      destfile = path.join(outdir,fdict[f],basename)
      if action == "symlink" and relative_symlinks:
         ff = path.relpath(ff,destfile)
      if debuglevel >= 2:
         if action == "copy":
            eprint(f"cp -p {ff} {destfile}")
         elif action == "symlink":
            eprint(f"ln -s {ff} {destfile}")
         elif action == "move":
            eprint(f"mv {ff} {destfile}")

      if not dryrun:
      #if False:
         if action == "copy":
            copy2(ff,destfile)
         elif action == "symlink":
            symlink(ff,destfile)
         elif action == "move":
            move(ff,destfile)

   return result

genlib.py

#!/usr/bin/env python3
# Startdate: 2021-01-29 15:35
# Dependencies:
#    devuan-req: python3-exifread
# Reference:
#    https://stackoverflow.com/questions/1192978/python-get-relative-path-of-all-files-and-subfolders-in-a-directory
#    https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory
#    https://stackoverflow.com/questions/600268/mkdir-p-functionality-in-python
from sys import stderr
from exifread import process_file
from os import walk, path, stat, symlink
from pathlib import Path
from datetime import date
from shutil import copy2, move

try:
   if debuglevel is None:
      debuglevel = 0
except:
   debuglevel = 0

# Functions
def eprint(*args, **kwargs):
   print(*args, file=stderr, **kwargs)

def list_files(indir,relative=False,debuglevel=debuglevel, excludes=None):
   x = 0
   f = []
   if debuglevel >= 9:
      eprint(f"Listing all files underneath {indir}")
      if len(excludes) > 0:
         eprint("While excluding path matches:",excludes)
   for (dirpath, dirnames, filenames) in walk(indir):
      for file in filenames:
         x += 1
         fullpath=path.join(dirpath,file)
         relpath=path.join(path.relpath(dirpath,indir),file)
         use = True
         use_pattern = ""
         for e in excludes:
            # simple match, no regex
            if e in relpath:
               use = False
               use_pattern = e
               break # short-circuit
         if use:
            if debuglevel >= 9: eprint(x,fullpath,relpath)
            f.append(relpath if relative else fullpath)
         else:
            if debuglevel >= 9: eprint(x,fullpath,f"ignored per {use_pattern}")
   return f

def limit(listf=None,filters=["image","video"]):
   # available types are "image", "video"
   newlist = []
   for f in listf:
      _, ext = path.splitext(f)
      ext = ext[1:].lower()
      if ext in ["jpg","jpeg","png","svg","tif","tiff","gif"] and "image" in filters:
         newlist.append(f)
      elif ext in ["mp4","webm","avi"] and "video" in filters:
         newlist.append(f)
   return newlist

def get_file_YM(tf,debuglevel=debuglevel,zero_pad=False):
   Y = 0 ; M = 0
   #eprint(f"Debuglevel {debuglevel}")
   try:
      # read exif data
      with open(tf,'rb') as f:
         tags = process_file(f, details=False, stop_tag="Image DateTime")
      if "Image DateTime" in tags:
         #if debuglevel >= 5: eprint(tf,tags["Image DateTime"])
         Y = str(tags["Image DateTime"]).split(' ')[0].split(':')
         if Y == "['']" or Y == ['']:
            Y = ['0','0']
         M = Y[1] ; Y = Y[0];
      # Any other image timestamps could be used here
      #eprint(tags)
   except KeyboardInterrupt:
      return -1, -1
   except:
      if debuglevel >= 1: eprint(f"Unable to extract any exif data for {tf}")
   if int(Y) == 0 or int(M) == 0:
      # need to just use timestamp
      try:
         ts = date.fromtimestamp(stat(tf).st_mtime)
      except KeyboardInterrupt:
         return -1, -1
      except:
         # no timestamp available from filesystem?
         if debuglevel >= 1: eprint(f"No timestamp available for {tf}")
      if int(Y) == 0: Y = ts.year
      if int(M) == 0: M = ts.month
   if zero_pad:
      M = str(M).zfill(2)
   else:
      M = str(int(M))
   if debuglevel >= 3:
      eprint(f"{tf} {Y}/{M}")
   return Y, M

def make_YM_forest(outdir, flist, action = "symlink", dryrun=True, debuglevel=debuglevel, zero_pad=False, relative_symlinks = False):
   """
   For each file in flist, [action] it to outdir, sorting into YYYY/MM subdirs based on exif metadata or timestamp.
   Action may be one of ['symlink', 'copy', 'move'].
   If relative_symlink = True, then make relative symlinks if possible.
   """
   result = 0
   stop = False

   # validate input
   if action not in ['symlink', 'copy', 'move']:
      eprint("make_YM_forest action may be one of ['symlink', 'copy', 'move'].")
      return -1

   # Learn all directories to make
   # and also cache the files with Y/M value
   fdict = {}
   destdirs = []
   for f in flist:
      try:
         Y, M = get_file_YM(f,debuglevel=debuglevel,zero_pad=zero_pad)
         if Y == -1 and M == -1:
            #eprint("Stopping due to keyboard variant 2.")
            stop = True
            break
      except KeyboardInterrupt:
         #eprint("Stopping due to keyboard variant 1.")
         stop = True
         break
      if zero_pad:
         M = str(M).zfill(2)
      else:
         M = str(int(M))
      YM = f"{Y}/{M}"
      if YM not in destdirs: destdirs.append(YM)
      fdict[f] = YM
   # finish the for f in flist

   # short-circuit if keyboard action happened in one of the nested functions
   if stop:
      return -1

   # Make directories
   for d in destdirs:
      dd = path.join(outdir,d)
      if debuglevel >= 3:
         eprint(f"Make dir: {dd}")
      if not dryrun:
         Path(dd).mkdir(parents=True, exist_ok=True)

   # Make symlinks
   for f in fdict:
      ff = f
      basename = path.basename(ff)
      destfile = path.join(outdir,fdict[f],basename)
      if action == "symlink" and relative_symlinks:
         ff = path.relpath(ff,destfile)
      if debuglevel >= 2:
         if action == "copy":
            eprint(f"cp -p {ff} {destfile}")
         elif action == "symlink":
            eprint(f"ln -s {ff} {destfile}")
         elif action == "move":
            eprint(f"mv {ff} {destfile}")

      if not dryrun:
      #if False:
         if action == "copy":
            copy2(ff,destfile)
         elif action == "symlink":
            symlink(ff,destfile)
         elif action == "move":
            move(ff,destfile)

   return result

Web gallery solution

bgstack15

2021-02-02 08:43

Comments

A while back, I was inspired by Thealaskalinuxuser's article about how he set up his photo sharing server at home. I experimented briefly with piwigo back when I read that, but ended up stopping that project. When I returned to this topic, I realized I just didn't want to deal with php. I applaud his stamina and commitment, but I want something with fewer dependencies. Here is my solution. So, here is my internal documentation that is my alternative to Google Photos.

Gallery solution for internal network

Overview

As part of my goals to run self-hosted services for myself and my family, I intend to maintain a web gallery of photos and videos. This solution consists of several parts.

Tools that create symlink forests to the original image files, in a specific location
Static site generator sigal
Customized theme for sigal
Customized sigal config for that theme
Gallery id table
SELinux rules
CGI scripts for Apache httpd

Architecture

The solution runs on server1, the main file server for the internal network.

Tools that create symlink forests for the gallery source directories

Generate.sh is what I used for the proof of concept. This needs to be rewritten in python, and to handle files without exif data. My python implementation will show up on this blog at a later date.

Static site generator

The sigal site generator is installed with pip3, under local service account sigal on server1. A special script, /usr/local/bin/sigal.bin will generate the static pages for a site. To use this script, pass a parameter of the source directory where the sigal.conf.py exists.

/usr/local/bin/sigal.bin /mnt/public/www/example/images/.gallery

Sigal depends on ffmpeg for video thumbnailing and conversions. On CentOS 8, ffmpeg is in repository powertools. File sigal.bin

#!/bin/sh
# File: /usr/local/bin/sigal.bin
# Author: bgstack15@gmail.com
# Startdate: 2021-01-24
# Title: Run sigal static site generator
# Purpose: run sigal static site generator for provided path
# History:
# Usage:
#    called by regen.cgi, or by hand
# Dependencies:
#    60_regen_gallery_sudo
#    gallery.te
#    service account "sigal" with `pip3 install --user sigal`
# Reverse dependencies:
#    httpd conf 
sigal_home=~sigal

test -z "${GALLERY_ID}" && export GALLERY_ID="${1}"
test -z "${GALLERY_ID}" && { echo "Pass to this script the gallery_id or directory path to a sigal.conf.py. Aborted." 1>&2 ; exit 1 ; }
if test -d "${GALLERY_ID}" ;
then
   cd "${GALLERY_ID}"
else
   if test -e /etc/gallery-cgi.conf ;
   then
      . /etc/gallery-cgi.conf
   else
      # no gallery_id listing exists.
      echo "No gallery_id table exists. Aborted." 1>&2 ; exit 1
   fi
fi
if test "${GALLERY_ID}" = "init" ;
then
   sudo su sigal -s /bin/bash -c "${sigal_home}/.local/bin/sigal init"
else
   eval cd \"\${"${GALLERY_ID}"}\"
   sudo su sigal -s /bin/bash -c "${sigal_home}/.local/bin/sigal build"
fi

Customized theme for sigal

Use theme bgstack15-gallery-theme , in any location. You just need to put its full path as the value of the theme in a sigal.conf.py for a gallery. This is a fork of the default, included colorbox theme. The full theme is available in this project directory, as well as a diff of it to the original as of sigal version 2.1.1. The main changes are adding extra metadata handlers and link logic for the custom links related to editing metadata.

Customized sigal config for that theme

When you run sigal init in a directory, it generates a default config you can modify. The example theme, and this whole solution, depends on adding a number of settings. A summary of the specific options is here, but the full example file is in this project directory.

source = '/var/www/gallery/.my2018'
destination = '/var/www/gallery/my2018'
use_orig = True
edit_cgi_script = "/cgi-bin/gallery/edit.cgi" # web path to edit.cgi
edit_enabled = False
edit_password = "makeupapassword"
edit_string = '[edit]' # text of link to edit metadata
toggle_enable_string = "Enable editing"
toggle_disable_string = "Disable editing"
toggle_link = "/cgi-bin/gallery/toggle-editing.cgi" # web path to toggle-editing.cgi
toggle_editing = [
        (False, 'Enable editing'),
        (True, 'Disable editing')
    ]
gallery_id = "example_images"
# A list of links (tuples (title, URL))
# links = [('Example link', 'http://example.org'),
#          ('Another link', 'http://example.org')]
links = [
    ('Regenerate pages', '/cgi-bin/gallery/regen.cgi?id=' + gallery_id)
        ]

The gallery_id is very important, because the cgi scripts and theme rely on it.

Gallery id table

File /etc/gallery-cgi.conf contains a list of gallery_id translations to directories with sigal.conf.py rules.

example_images=/mnt/public/www/example/images/.gallery

SELinux rules

The reference system, server1, runs SELinux. A custom selinux module is needed to allow all the operations that are a part of this gallery solution, which include the following. File gallery.te can be installed as an enabled selinux module.

sudo checkmodule -M -m -o gallery.mod gallery.te && sudo semodule_package -o gallery.pp -m gallery.mod && sudo semodule -i gallery.pp

File gallery.te:

# Last modified 2021-01-30
module gallery 1.0;

require {
    type faillog_t;
    type security_t;
    type httpd_config_t;
    type init_t;
    type sssd_t;
    type mnt_t;
    type lastlog_t;
    type systemd_logind_sessions_t;
    type initrc_var_run_t;
    type tmpfs_t;
    type gconf_home_t;
    type chkpwd_t;
    type systemd_logind_t;
    type unconfined_t;
    type shadow_t;
    type httpd_sys_script_t;
    type sssd_selinux_manager_t;
    type sssd_conf_t;
    type var_t;
    type httpd_t;
    class capability { audit_write dac_read_search net_admin setgid setuid sys_resource };
    class process { noatsecure rlimitinh setrlimit siginh };
    class netlink_audit_socket { create nlmsg_relay read write };
    class netlink_selinux_socket { bind create };
    class passwd rootok;
    class dir { add_name read remove_name search write };
    class file { create execute execute_no_trans setattr getattr link lock map open read unlink write ioctl };
    class dbus send_msg;
    class fifo_file write;
    class security compute_av;
    class lnk_file read;
    class filesystem getattr;
    class process setfscreate;
}

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t faillog_t:file { open read };
allow httpd_sys_script_t var_t:file { create ioctl setattr unlink write };
allow httpd_sys_script_t var_t:dir { read add_name remove_name write };

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow httpd_sys_script_t gconf_home_t:file map;
allow httpd_sys_script_t gconf_home_t:file { execute execute_no_trans };
allow httpd_sys_script_t httpd_config_t:dir search;
allow httpd_sys_script_t initrc_var_run_t:file { lock open read };
allow httpd_sys_script_t lastlog_t:file { open read write };
allow httpd_sys_script_t mnt_t:lnk_file read;
allow httpd_sys_script_t security_t:dir read;
allow httpd_sys_script_t security_t:file { getattr open read write };
allow httpd_sys_script_t security_t:security compute_av;
allow httpd_sys_script_t self:capability { audit_write dac_read_search net_admin setgid setuid sys_resource };
allow httpd_sys_script_t self:netlink_audit_socket { create nlmsg_relay read write };
allow httpd_sys_script_t self:netlink_selinux_socket { bind create };
allow httpd_sys_script_t self:passwd rootok;
allow httpd_sys_script_t self:process setrlimit;
allow httpd_sys_script_t shadow_t:file { getattr open read };
allow httpd_sys_script_t sssd_conf_t:dir search;
allow httpd_sys_script_t sssd_conf_t:file { getattr open read };
allow httpd_sys_script_t systemd_logind_sessions_t:fifo_file write;
allow httpd_sys_script_t systemd_logind_t:dbus send_msg;
allow httpd_sys_script_t tmpfs_t:dir { add_name remove_name write };

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow httpd_sys_script_t tmpfs_t:file map;
allow httpd_sys_script_t tmpfs_t:file { create getattr link open read unlink write };
allow httpd_sys_script_t tmpfs_t:filesystem getattr;
allow httpd_sys_script_t self:process setfscreate;

#============= init_t ==============
allow init_t chkpwd_t:process siginh;
allow init_t unconfined_t:process siginh;

#============= sssd_t ==============
allow sssd_t sssd_selinux_manager_t:process { noatsecure rlimitinh siginh };

#============= systemd_logind_t ==============
allow systemd_logind_t httpd_sys_script_t:dbus send_msg;
#============= httpd_t ==============
allow httpd_t var_t:file { getattr map open read };

Sudo rules

For apache httpd to be able to run the sigal.bin, set up sudoers rules. File 60_regen_gallery_sudo adds the permission necessary.

# file: /etc/sudoers.d/60_regen_gallery_sudo
# Reference: server4:/etc/sudoers.d/60_starbound_sudo
apache ALL=(root)   NOPASSWD: /usr/local/bin/sigal.bin *

CGI scripts for Apache httpd

The main focus of the gallery project is the ability to edit metadata from the web view. While sigal is great for developers, some users might only care about editing metadata from where they are actually viewing the media.

edit.cgi is called from the custom theme's "edit" links, and includes the form for making changes to media metadata.
apply.cgi actually makes the changes, and is called from the edit.cgi form.
regen.cgi invokes sigal.bin which re-runs sigal.
toggle-editing.cgi enables or disables editing. Enabling requires a password.

These can be placed anywhere you have enabled CGI for httpd, but the canonical location is /var/www/cgi-bin/gallery/.

Operations

I anticipate that more work is needed on an ongoing basis. Here are some processes that can be used.

Make a new gallery

To establish a new gallery, change directory to the source directory for the gallery and run command

sigal.bin init

Which generates the basic sigal.conf.py. Add the pertinent variables, described in section "Customized sigal config for that theme" above.

Run sigal from command line

While the web links for "regen.cgi" are great for when you are viewing the web, you can also run the sigal.bin from the cli. You need to include a path to the directory that holds a sigal.conf.py, or else a gallery_id from /etc/gallery-cgi.conf.

sigal.bin example_images

Make metadata changes directly on filesystem

You can of course, as designed by the author of sigal, go edit any ${IMAGENAME%%.jpg}.md file with the relevant fields. See references 1 and 2 for the available fields. File index.md will be the metadata for the directory itself.

History

In 2020, I installed piwigo on a dev system. I didn't want to deal with php, so I dropped it. In January 2021, I started listing various options for self-hosted galleries. Read heading [Related Files] for those. Criteria I assembled includes

Metadata: description, date, comments

Related Files

These files are important to this gallery project. Check them all out at my gitlab space.

/usr/local/bin/sigal.bin
/var/www/cgi-bin/apply.cgi
/var/www/cgi-bin/edit.cgi
/var/www/cgi-bin/regen.cgi
/var/www/cgi-bin/toggle-editing.cgi
gallery.te
/etc/sudoers.d/60_regen_gallery_sudo
/etc/gallery-cgi.conf
sigal.conf.py
bgstack15-gallery-theme/

Alternatives

Piwigo uses PHP which I don't want to maintain.
Awesome selfhosted#photo-and-video-galleries is a whole list.

Ones I considered without trying

Ones I listed as tolerable, but not focused on what I need.

lychee
photato
PhotoFloat written by the cgit author. He has a large body of good works.

References

Weblinks

http://sigal.saimon.org/en/latest/album_information.html
http://sigal.saimon.org/en/latest/image_information.html
Home photo server, part 2: Apache, phpAlbum, and Piwigo | thealaskalinuxuser Thealaskalinuxuser's guide to a home photo server with piwigo

Package for Devuan: dragon-drag-and-drop

bgstack15

2021-01-29 09:12

Comments

I recently came across a great little project, dragon, which is a simple drag-and-drop source/sink for X11. This project follows the Unix concept of do one thing and do it well. The program was designed for users who do not use a graphical file manager, but want to be able to drag and drop files (into a web browser, for example). It has minimal dependencies, and is a breeze to compile and install and use. My contributions to the project include the packaging recipes for dpkg and rpm. You can go get dragon-drag-and-drop from my OBS repository.

Python json to csv

bgstack15

2021-01-25 09:18

Comments

I'm sure there are many tutorials for how to convert json arrays to csv. I wrote one that finds all the possible keys and then generates a csv. Check it out at my gitlab. The logic is broken up into several parts.

   # Learn keys at this level
   keys=[]
   for i in json_obj:
      for j in list(i.keys()):
         if j not in keys:
            keys.append(j)
            debugprint("Found key {0}".format(j),debug=debug)

And then I found the concatenation to a large string, and then a single write operation to a file, to be much more performant than writing to the file for each entry of each row (duh).

   x=0
   fullstring=""
   for i in json_obj:
      x += 1
      # only if there are actually contents of "all" do we print the headers
      if x == 1:
         for k in keys:
            fullstring += str(k) + ","
         fullstring += "\n"
      for k in keys:
         p = ""
         try:
            p = str(i[k]).replace(",",":")
         except:
            # no value for this key for this entry
            pass
         pp = "{0},".format(p)
         fullstring += pp
      fullstring += "\n"

   if fullstring != "":
      if csvfile == "stdout":
         print(fullstring)
      else:
         with open(csvfile,"a") as of:
            of.write(fullstring)

So you'll see that I also replace some commas with colons. I should probably have parameterized those, but I can leave that as an exercise to the reader. I even threw in a cli front-end to my library.

$ ./json-to-csv.py --help
usage: json-to-csv.py [-h] [-d [{0,1,2,3,4,5,6,7,8,9,10}]] [-i INFILE] [-o OUTFILE]

optional arguments:
  -h, --help            show this help message and exit
  -d [{0,1,2,3,4,5,6,7,8,9,10}], --debug [{0,1,2,3,4,5,6,7,8,9,10}]
                        Set debug level.
  -i INFILE, --infile INFILE
                        Json file to convert
  -o OUTFILE, --outfile OUTFILE
                        Csv file to output

Backstory

Some hospitals present their machine-readable data for pricing transparency as json. And some devices don't know how to read json, but a csv can be opened like a spreadsheet.

File Upload and Storage Service

bgstack15

2021-01-21 08:35

Comments

I present to the Internet my take on a python+flask web app! File Upload and Storage Service is a small webapp for uploading and serving files. Please run an instance behind authentication of some sort (which could be at the web server level) so malicious actors do not upload bad things to your server. Check out the code at https://gitlab.com/bgstack15/fuss. FUSS includes some cool features, like drag-and-drop upload on the optional upload page. If you disable the separate upload page, then users can just curl to the application base URL. The application dynamically discovers the path that the user used to get to the app! But admins who prefer to actually enforce a specific URL for the application can configure that in the config file. A sysvinit service script is provided, as well as a systemd service file. A dpkg recipe and also rpm spec are included. To run this application on CentOS 7, you will need pip3 and a few modules from it. Main page of FUSS web
app This program is designed mostly as a demo of all the various bits: javascript, python and flask, apache reverse proxying, and html.

Grub with a theme, when you have an encrypted root filesystem

bgstack15

2021-01-17 08:36

Comments

I use luks encryption for my root filesystem (i.e., "/"), so this includes /usr/share/desktop-base/grub-themes/ where a Debian-like grub installation looks for its themes. Unfortunately the encrypted filesystem prevents grub from reading the pretty theme, so it just shows me a basic menu. To get access at boot time to my grub theme, I copied it to /boot.

sudo mkdir -p /boot/themes
sudo cp -pr /usr/share/desktop-base/grub-themes/cinnabar-grub /boot/themes
sudo ln -s cinnabar-grub /boot/themes/live

And then I adjusted /etc/default/grub:

GRUB_THEME=/boot/themes/live/theme.txt

And then ran update-grub (a Debian alias for grub-mkconfig -o /boot/grub/grub.cfg ).

sudo update-grub


Generating grub configuration file ...
Found theme: /boot/themes/live/theme.txt

Credits

fsmithred of Refracta

exiftool notes

bgstack15

2021-01-13 09:27

Comments

Remove all metadata from an image.

exiftool -all= /path/to/image.jpg

My experience with certbot on CentOS 8

bgstack15

2021-01-09 09:09

Comments

I finally bit the bullet and set up Let's Encrypt for myself. The certbot instructions say to use a snap, but that is a hard negative for my environment. Thankfully, CentOS 8 has the certbot package from epel! (And don't hate on me! I had installed CentOS 8 about 2 weeks before the fateful news.) So I installed certbot, which pulls in some python dependencies.

sudo yum install certbot


Dependencies resolved.
======================================================================================================================
 Package                                Architecture        Version                      Repository              Size
======================================================================================================================
Installing:
 certbot                                noarch              1.10.1-1.el8                 epel                    49 k
Installing dependencies:
 python3-acme                           noarch              1.10.1-1.el8                 epel                    88 k
 python3-certbot                        noarch              1.10.1-1.el8                 epel                   387 k
 python3-configargparse                 noarch              0.14.0-6.el8                 epel                    36 k
 python3-josepy                         noarch              1.2.0-5.el8                  epel                    95 k
 python3-ndg_httpsclient                noarch              0.5.1-4.el8                  epel                    53 k
 python3-parsedatetime                  noarch              2.5-1.el8                    epel                    79 k
 python3-pyOpenSSL                      noarch              18.0.0-1.el8                 appstream              103 k
 python3-pyrfc3339                      noarch              1.1-1.el8                    epel                    19 k
 python3-requests-toolbelt              noarch              0.9.1-4.el8                  epel                    91 k
 python3-zope-component                 noarch              4.3.0-8.el8                  epel                   313 k
 python3-zope-event                     noarch              4.2.0-12.el8                 epel                   210 k
 python3-zope-interface                 x86_64              4.6.0-1.el8                  epel                   158 k

Transaction Summary
======================================================================================================================
Install  13 Packages

I have experience with apache httpd configs, so I wasn't interested in letting certbot do anything to my configs. So I opted for the webroot challenge mechanism, which just adds the challenge files to underneath your webroot location. Which, I learned, takes a small amount of manual work. Not a biggie, but worth knowing to simplify the process.

cd /var/www
mkdir -p .well-known/acme-challenge
restorecon -Rvn .well-known

I suppose it might be a good that even with sudo, certbot does not make directories or restore SELinux contexts. But now I was ready to run for real:

sudo certbot certonly --webroot -w /var/www -d www.example.com

It was fun to watch my apache logs and see the various IP addresses check my acme-challenge responses. It only took 7 seconds before the process was complete and I was issued my certificate!

 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/www.example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/www.example.com/privkey.pem

And now I can configure my httpd confs the way I want to, instead of letting somebody else fiddle with them. And all this because my friends don't know how to trust my root CA cert, let alone actually want to do that.

Operations

I set up my renewal with a shell script and cron.

References

Syntax of automatic command: https://community.letsencrypt.org/t/certonly- enter-a-webroot/27442 https://certbot.eff.org/lets-encrypt/centosrhel7-apache

Explanation

Inspect certificate with web browser

Inspect certificate chain with openssl command

Inspect certificate chain presented by web server

Inspect certificate in a file

See also

generate.py

genlib.py

Gallery solution for internal network

Overview

Architecture

Tools that create symlink forests for the gallery source directories

Static site generator

Customized theme for sigal

Customized sigal config for that theme

Gallery id table

SELinux rules

Sudo rules

CGI scripts for Apache httpd

Operations

Make a new gallery

Run sigal from command line

Make metadata changes directly on filesystem

History

Related Files

Alternatives

References

Weblinks

Backstory

Credits

Operations

References