Knowledge Base (Posts about symlink)

jellyfin show manager

bgstack15 — Sun, 10 Mar 2024 13:24:39 GMT

I'm back with an overengineered solution for myself. I have a complete show, but I want to have Jellyfin show only up to 2 new episodes at a time. I'll set a cron job to check and add new episodes from the real location as necessary.

So, the overcomplicated script:

files/2024/listings/show-manager.sh (Source)

#!/bin/sh
# File: show-manager.sh
# Location: /mnt/public/Support/Programs/jellyfin/scripts/
# Author: bgstack15
# Startdate: 2024-02-18-1 17:24
# SPDX-License-Identifier: GPL-3.0-only
# Title: Slowly add new episodes of a completed show to jellyfin
# Purpose: Check jellyfin for if the last season of a given show is greater than (x-2)/x episodes, and if so, then add that many new episodes.
# History:
# Usage:
#    . ./jellystack-autocomplete.bash ; manage_show stsnw
# Improve:
#    hook in to jellyfin deeper, by checking file paths of the episodes where watched=True, to know which episodes to skip trying to add
#    learn how to completely clear the "watched" information about the show in jellyfin.
# Documentation:
#    Assumptions:
#       1. media files are sorted in "Season 01/" type directories.
#       2. Media files are named "*s01e07 *" style which will be sorted alphabetically in episode number order.
#       3. watched episodes are in exact order!
#       4. input files are all mkv (this is to simplify filtering out non-episode files like images; it could be made more complex as needed)
# Dependencies:
#    jellystack_lib with watched_episodes_for_show()

# flow:
# get episodes-watched for last season of show (value is, e.g., "8/10"


# depends on content being in "Season 01/" directories.
get_episodes_watched_count() {
   # input env vars: library, show, DEBUG
   # output: prints watched= and total= numbers
   # usage: $( library="TV" show="Star Trek: Strange New Worlds" get_episodes_watched_count )
   _pyverbose=False
   test -n "${DEBUG}" && _pyverbose=True
   cd /mnt/public/Support/Programs/jellyfin/scripts
   . ~/.config/jellystack.viewing.user
   {
      python3 <<-EOF
import jellystack_lib
a = jellystack_lib.get_authenticated_client(url="${server}",username="${username}",password="${password}")
b = jellystack_lib.watched_episodes_for_show(a,"${library}","${show}","sum",${_pyverbose})
c = b.split("/")
print(f"watched={c[0]}")
print(f"total={c[1]}")
EOF
   }
}

symlink_file() {
   # input vars: infile SOURCE_DIR LINK_DIR LINK_PREFIX VERBOSE APPLY
   # reference: printf '%s\n' *s01e0* | while read line ; do ln -sf "../../../off.movies/Star Trek - Strange New Worlds (2021/Season 01/${line}" "/mnt/public/Video/TV/Star Trek - Strange New Worlds (2021/Season 01/" ; done
   # We now calculate season number based on file name
   #season_num_str="$( \printf '%02d' "${season_num}" )"
   base="$( basename "${infile}" )"
   season_num_str="$( \printf '%02d' "$( echo "${base}" | grep -oE '\<s[0-9]+e[0-9]+' | awk -F'e' '{print $1}' | tr -dc '0-9' )" )"
   test -n "${VERBOSE}" && echo ln -s "${LINK_PREFIX}/Season ${season_num_str}/${base}" "${LINK_DIR}/Season ${season_num_str}/"
   test -n "${APPLY}" && ln -s "${LINK_PREFIX}/Season ${season_num_str}/${base}" "${LINK_DIR}/Season ${season_num_str}/"
}

list_sorted_episode_files() {
   # input env vars: SOURCE_DIR
   # if changing from %P, probably do a cd in a sub-shell.
   find "${SOURCE_DIR}" -name '*mkv' -iregex '.*\<s[0-9]+e[0-9]+\>.*' -printf '%P\n' | sort -n
}

# Logic flow:
# diff=total-watched
# if diff < 2,
#    find the smallest-number episode files and add them in.
manage_show() {
   # input vars: MAX_NEW_EPISODES SOURCE_DIR VERBOSE LIBRARY SHOW
   # You may also run: manage_show "TV/Star Trek: Strange New Worlds"
   conf="${1}"
   test -f "/mnt/public/Support/Programs/jellyfin/scripts/input/${conf}.conf" && . "/mnt/public/Support/Programs/jellyfin/scripts/input/${conf}.conf"
   # Not necessary with the conf file method
   #test -z "${SHOW}" && test -z "${LIBRARY}" && {
   #   LIBRARY="$( echo "${1}" | awk -F'/' '{print $1}' )"
   #   SHOW="$( echo "${1}" | awk -F'/' '{print $2}' )"
   #}
   test -z "${MAX_NEW_EPISODES}" && MAX_NEW_EPISODES=2
   # this next function sets vars: watched, total
   eval $( library="${LIBRARY}" show="${SHOW}" get_episodes_watched_count )
   # hardcode these to test max_new_episodes
   #watched=6 total=8
   diff=$((total-watched))
   test -n "${VERBOSE}" && echo "Got watched=${watched} total=${total} diff=${diff}" 1>&2
   if test ${diff:-0} -lt ${MAX_NEW_EPISODES} ;
   then
      need=$((MAX_NEW_EPISODES-diff))
      echo "Must prepare next ${need} episodes." 1>&2
      w1=$((watched+1))
      end=$((watched+need))
      list_sorted_episode_files | sed -n -r -e "${w1},${end}p" | while read infile ;
      do
         # VERBOSE and APPLY are used in this function:
         infile="${infile}" symlink_file
      done
   fi
}

# Before the config file redesign, use one of these:
# These will still get "/Season 01" prepended to the file basename:
# SOURCE_DIR="/mnt/public/Video/off.movies/Star Trek - Strange New Worlds (2021)" LINK_DIR="/mnt/public/Video/TV/Star Trek - Strange New Worlds (2021)" LINK_PREFIX="../../../off.movies/Star Trek - Strange New Worlds (2021)" LIBRARY="TV" SHOW="Star Trek - Strange New Worlds (2021)" DIR="${SHOW}" manage_show

So I'm pretty sure this uses yet a new password file, this time as a shell snippet with export statements of the interesting variables, ~/.config/jellystack.viewing.user. (I don't use the admin user for actually watching, what am I, ridiculous?!)

export password="KEEPASS" username="jellyfin" server="http://vm4:8096"

And then with this autocomplete:

files/2024/listings/jellystack-autocomplete.bash (Source)

#!/bin/bash
# trimmed for blog
_show_manager_confs() {
   # reference: vm4:/etc/bash_completion.d/docker-nfs-check
   local cur prev words cword;
   _init_completion || return
   _tmpfile1="$( mktemp )"
   # populate list
   find /mnt/public/Support/Programs/jellyfin/scripts/input -mindepth 1 -maxdepth 1 ! -type d -name '*.conf' -printf '%P\n' | sed -r -e 's/\.conf$//;' > "${_tmpfile1}"
   COMPREPLY=($( compgen -W "$( cat ${_tmpfile1} )" -- "$cur" | sed -r -e "/^${prev}/d;" ))
   command rm -rf "${_tmpfile1:-NOTHINGTODEL}" 1>/dev/null 2>&1
   return 0
} &&
complete -F _show_manager_confs manage-show
. /mnt/public/Support/Programs/jellyfin/scripts/show-manager.sh
alias manage-show="manage_show"

And any number of files in that input/ directory:

files/2024/listings/strangenewworlds.conf.example (Source)

# Where all the episode mkv files are
SOURCE_DIR="/mnt/public/Video/off.movies/Star Trek - Strange New Worlds (2021)"
# Where they should be placed for Jellyfin to pick them up
LINK_DIR="/mnt/public/Video/TV/Star Trek - Strange New Worlds (2021)"
# Symlink prefix, so probably the relative path from LINK_DIR to SOURCE_DIR
LINK_PREFIX="../../../off.movies/Star Trek - Strange New Worlds (2021)"
# Jellyfin library name
LIBRARY="TV"
# Jellyfin show name
SHOW="Star Trek: Strange New Worlds"
# Show directory name. Not necessarily the same as SHOW.
DIR="Star Trek - Strange New Worlds (2021)"

So if I've watched 6/7 episodes, and MAX_NEW_EPISODES=2, then it should symlink in one new episode.

This process counts incorrectly sometimes (because I originally had all the episode files available to it and the counting got messed up), so I just keep increasing MAX_NEW_EPISODES until it puts the correct number of new episodes in the path for Jellyfin.

So I just run:

. jellystack-autocomplete.bash
VERBOSE=1 MAX_NEW_EPISODES=3 manage-show <tab>

And list the show confs, and pick one, and then run it. Once it looks good, I re-run with APPLY=1.

Convert .lnk shortcut to symlink

bgstack15 — Thu, 18 Feb 2021 13:56:41 GMT

Related to my Web gallery solution and Symlink forest generator, I wanted a way to convert Windows-style shortcuts (.lnk files) to symlinks. Apparently I am the first person to publish this research on the Internet. Nobody else in the world has discussed performing this operation. It's probably foolhardy, but it was technically interesting. I wrote the shortcut2symlink python library and cli tool.

Overview

A Windows client is used to maintain a website/ subdirectory of files that will be processed using the gallery project. To facilitate the process of preparing files for this website/ subdirectory, particularly over a wireless network connection, users have been instructed to just make Windows-style shortcuts (using CTRL+SHIFT dragging) in the website subdirectory in whatever layout desired. This project includes components that will scan the designated directory for .lnk files anywhere underneath, and interpret their targets, and generate symlinks in place of the .lnk files. Unfortunately my scanner (inotify) is slightly flaky, and sometimes I still need to manually run the s2s.py command directly. The point of using symlinks in this case is to reduce the disk usage and number of duplicate files and directories. To SMB users on the Windows client, the symlink files will look like copies, but that is acceptable because the main focus is the gallery generation for the web server.

Reason for existence

Reduce disk space by removing duplicate files, while still having a whole directory structure readable by the gallery solution for generating static site pages, and while making it easy for wireless network users to "manage" links to files at least initially.

Example usage

For the inotify script:

/mnt/public/Support/Programs/shortcuts/s2s-inotify.sh /mnt/public/Images/username1-photos/website

And for manually running:

/mnt/public/Support/Programs/shortcuts/s2s.py --indir /mnt/public/Images/username1-photos/website --apply --debug 8 --delete

References

Weblinks

.lnk format reference

Symlink forest generator

bgstack15 — Sat, 06 Feb 2021 14:22:21 GMT

As part of my Web gallery solution, I wrote some logic that creates a YYYY/MM directory creator that places symlinks to my original files within the new directory structure. The links are added to the relevant directory based on exif metadata if possible, and file timestamp as a last resort. Check out the source files at my gitlab space:

generate.py front end for cli
genlib.py library

Here is my main sample invocation. I used this a lot while testing.

./generate.py -i /mnt/bgstack15/Backups/Images/Photos/camera/2018/ -o /mnt/public/www/gallery/my2018 -n -d2 -x 'October' -s --nr

The help text is generated with ArgumentParser.

$ ./generate.py --helpusage: generate.py [-h] [-d [{0,1,2,3,4,5,6,7,8,9,10}]] [-v] [-n | -a] [-z | --nz] [-c | -s | -m] [-r | --nr] -i
                   INDIR -o OUTDIR [-x EXCLUDE]

Make symlink forest for images

optional arguments:
  -h, --help            show this help message and exit
  -d [{0,1,2,3,4,5,6,7,8,9,10}], --debug [{0,1,2,3,4,5,6,7,8,9,10}]
                        Set debug level
  -v, --version         show program's version number and exit
  -n, --dryrun          Make no changes (default)
  -a, --apply           Actually make changes
  -z, -0, --zeropad     Zero pad month directories (default)
  --nz, --nozeropad, --no-zeropad
                        Do not zero pad
  -c, --copy            Copy files instead of symlinks. Not recommended.
  -s, --symlink         Make symlinks (default)
  -m, --move            Move files. Not recommended.
  -r, --relative        Make relative symlinks if possible.
  --nr, --norelative, --no-relative
                        Make absolute symlinks (default)
  -i INDIR, --indir INDIR
  -o OUTDIR, --outdir OUTDIR
  -x EXCLUDE, --exclude EXCLUDE
                        Exclude pathname matches. Can be used multiple times.

For posterity and in case my gitlab ever dies, here are their contents at the time of this writing.

generate.py

#!/usr/bin/env python3
# Startdate: 2021-01-29 15:35
# Dependencies:
#    devuan-req: python3-exifread
# Reference:
#    https://stackoverflow.com/questions/1192978/python-get-relative-path-of-all-files-and-subfolders-in-a-directory
#    https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory
#    https://stackoverflow.com/questions/600268/mkdir-p-functionality-in-python
from sys import stderr
from exifread import process_file
from os import walk, path, stat, symlink
from pathlib import Path
from datetime import date
from shutil import copy2, move

try:
   if debuglevel is None:
      debuglevel = 0
except:
   debuglevel = 0

# Functions
def eprint(*args, **kwargs):
   print(*args, file=stderr, **kwargs)

def list_files(indir,relative=False,debuglevel=debuglevel, excludes=None):
   x = 0
   f = []
   if debuglevel >= 9:
      eprint(f"Listing all files underneath {indir}")
      if len(excludes) > 0:
         eprint("While excluding path matches:",excludes)
   for (dirpath, dirnames, filenames) in walk(indir):
      for file in filenames:
         x += 1
         fullpath=path.join(dirpath,file)
         relpath=path.join(path.relpath(dirpath,indir),file)
         use = True
         use_pattern = ""
         for e in excludes:
            # simple match, no regex
            if e in relpath:
               use = False
               use_pattern = e
               break # short-circuit
         if use:
            if debuglevel >= 9: eprint(x,fullpath,relpath)
            f.append(relpath if relative else fullpath)
         else:
            if debuglevel >= 9: eprint(x,fullpath,f"ignored per {use_pattern}")
   return f

def limit(listf=None,filters=["image","video"]):
   # available types are "image", "video"
   newlist = []
   for f in listf:
      _, ext = path.splitext(f)
      ext = ext[1:].lower()
      if ext in ["jpg","jpeg","png","svg","tif","tiff","gif"] and "image" in filters:
         newlist.append(f)
      elif ext in ["mp4","webm","avi"] and "video" in filters:
         newlist.append(f)
   return newlist

def get_file_YM(tf,debuglevel=debuglevel,zero_pad=False):
   Y = 0 ; M = 0
   #eprint(f"Debuglevel {debuglevel}")
   try:
      # read exif data
      with open(tf,'rb') as f:
         tags = process_file(f, details=False, stop_tag="Image DateTime")
      if "Image DateTime" in tags:
         #if debuglevel >= 5: eprint(tf,tags["Image DateTime"])
         Y = str(tags["Image DateTime"]).split(' ')[0].split(':')
         if Y == "['']" or Y == ['']:
            Y = ['0','0']
         M = Y[1] ; Y = Y[0];
      # Any other image timestamps could be used here
      #eprint(tags)
   except KeyboardInterrupt:
      return -1, -1
   except:
      if debuglevel >= 1: eprint(f"Unable to extract any exif data for {tf}")
   if int(Y) == 0 or int(M) == 0:
      # need to just use timestamp
      try:
         ts = date.fromtimestamp(stat(tf).st_mtime)
      except KeyboardInterrupt:
         return -1, -1
      except:
         # no timestamp available from filesystem?
         if debuglevel >= 1: eprint(f"No timestamp available for {tf}")
      if int(Y) == 0: Y = ts.year
      if int(M) == 0: M = ts.month
   if zero_pad:
      M = str(M).zfill(2)
   else:
      M = str(int(M))
   if debuglevel >= 3:
      eprint(f"{tf} {Y}/{M}")
   return Y, M

def make_YM_forest(outdir, flist, action = "symlink", dryrun=True, debuglevel=debuglevel, zero_pad=False, relative_symlinks = False):
   """
   For each file in flist, [action] it to outdir, sorting into YYYY/MM subdirs based on exif metadata or timestamp.
   Action may be one of ['symlink', 'copy', 'move'].
   If relative_symlink = True, then make relative symlinks if possible.
   """
   result = 0
   stop = False

   # validate input
   if action not in ['symlink', 'copy', 'move']:
      eprint("make_YM_forest action may be one of ['symlink', 'copy', 'move'].")
      return -1

   # Learn all directories to make
   # and also cache the files with Y/M value
   fdict = {}
   destdirs = []
   for f in flist:
      try:
         Y, M = get_file_YM(f,debuglevel=debuglevel,zero_pad=zero_pad)
         if Y == -1 and M == -1:
            #eprint("Stopping due to keyboard variant 2.")
            stop = True
            break
      except KeyboardInterrupt:
         #eprint("Stopping due to keyboard variant 1.")
         stop = True
         break
      if zero_pad:
         M = str(M).zfill(2)
      else:
         M = str(int(M))
      YM = f"{Y}/{M}"
      if YM not in destdirs: destdirs.append(YM)
      fdict[f] = YM
   # finish the for f in flist

   # short-circuit if keyboard action happened in one of the nested functions
   if stop:
      return -1

   # Make directories
   for d in destdirs:
      dd = path.join(outdir,d)
      if debuglevel >= 3:
         eprint(f"Make dir: {dd}")
      if not dryrun:
         Path(dd).mkdir(parents=True, exist_ok=True)

   # Make symlinks
   for f in fdict:
      ff = f
      basename = path.basename(ff)
      destfile = path.join(outdir,fdict[f],basename)
      if action == "symlink" and relative_symlinks:
         ff = path.relpath(ff,destfile)
      if debuglevel >= 2:
         if action == "copy":
            eprint(f"cp -p {ff} {destfile}")
         elif action == "symlink":
            eprint(f"ln -s {ff} {destfile}")
         elif action == "move":
            eprint(f"mv {ff} {destfile}")

      if not dryrun:
      #if False:
         if action == "copy":
            copy2(ff,destfile)
         elif action == "symlink":
            symlink(ff,destfile)
         elif action == "move":
            move(ff,destfile)

   return result

genlib.py

#!/usr/bin/env python3
# Startdate: 2021-01-29 15:35
# Dependencies:
#    devuan-req: python3-exifread
# Reference:
#    https://stackoverflow.com/questions/1192978/python-get-relative-path-of-all-files-and-subfolders-in-a-directory
#    https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory
#    https://stackoverflow.com/questions/600268/mkdir-p-functionality-in-python
from sys import stderr
from exifread import process_file
from os import walk, path, stat, symlink
from pathlib import Path
from datetime import date
from shutil import copy2, move

try:
   if debuglevel is None:
      debuglevel = 0
except:
   debuglevel = 0

# Functions
def eprint(*args, **kwargs):
   print(*args, file=stderr, **kwargs)

def list_files(indir,relative=False,debuglevel=debuglevel, excludes=None):
   x = 0
   f = []
   if debuglevel >= 9:
      eprint(f"Listing all files underneath {indir}")
      if len(excludes) > 0:
         eprint("While excluding path matches:",excludes)
   for (dirpath, dirnames, filenames) in walk(indir):
      for file in filenames:
         x += 1
         fullpath=path.join(dirpath,file)
         relpath=path.join(path.relpath(dirpath,indir),file)
         use = True
         use_pattern = ""
         for e in excludes:
            # simple match, no regex
            if e in relpath:
               use = False
               use_pattern = e
               break # short-circuit
         if use:
            if debuglevel >= 9: eprint(x,fullpath,relpath)
            f.append(relpath if relative else fullpath)
         else:
            if debuglevel >= 9: eprint(x,fullpath,f"ignored per {use_pattern}")
   return f

def limit(listf=None,filters=["image","video"]):
   # available types are "image", "video"
   newlist = []
   for f in listf:
      _, ext = path.splitext(f)
      ext = ext[1:].lower()
      if ext in ["jpg","jpeg","png","svg","tif","tiff","gif"] and "image" in filters:
         newlist.append(f)
      elif ext in ["mp4","webm","avi"] and "video" in filters:
         newlist.append(f)
   return newlist

def get_file_YM(tf,debuglevel=debuglevel,zero_pad=False):
   Y = 0 ; M = 0
   #eprint(f"Debuglevel {debuglevel}")
   try:
      # read exif data
      with open(tf,'rb') as f:
         tags = process_file(f, details=False, stop_tag="Image DateTime")
      if "Image DateTime" in tags:
         #if debuglevel >= 5: eprint(tf,tags["Image DateTime"])
         Y = str(tags["Image DateTime"]).split(' ')[0].split(':')
         if Y == "['']" or Y == ['']:
            Y = ['0','0']
         M = Y[1] ; Y = Y[0];
      # Any other image timestamps could be used here
      #eprint(tags)
   except KeyboardInterrupt:
      return -1, -1
   except:
      if debuglevel >= 1: eprint(f"Unable to extract any exif data for {tf}")
   if int(Y) == 0 or int(M) == 0:
      # need to just use timestamp
      try:
         ts = date.fromtimestamp(stat(tf).st_mtime)
      except KeyboardInterrupt:
         return -1, -1
      except:
         # no timestamp available from filesystem?
         if debuglevel >= 1: eprint(f"No timestamp available for {tf}")
      if int(Y) == 0: Y = ts.year
      if int(M) == 0: M = ts.month
   if zero_pad:
      M = str(M).zfill(2)
   else:
      M = str(int(M))
   if debuglevel >= 3:
      eprint(f"{tf} {Y}/{M}")
   return Y, M

def make_YM_forest(outdir, flist, action = "symlink", dryrun=True, debuglevel=debuglevel, zero_pad=False, relative_symlinks = False):
   """
   For each file in flist, [action] it to outdir, sorting into YYYY/MM subdirs based on exif metadata or timestamp.
   Action may be one of ['symlink', 'copy', 'move'].
   If relative_symlink = True, then make relative symlinks if possible.
   """
   result = 0
   stop = False

   # validate input
   if action not in ['symlink', 'copy', 'move']:
      eprint("make_YM_forest action may be one of ['symlink', 'copy', 'move'].")
      return -1

   # Learn all directories to make
   # and also cache the files with Y/M value
   fdict = {}
   destdirs = []
   for f in flist:
      try:
         Y, M = get_file_YM(f,debuglevel=debuglevel,zero_pad=zero_pad)
         if Y == -1 and M == -1:
            #eprint("Stopping due to keyboard variant 2.")
            stop = True
            break
      except KeyboardInterrupt:
         #eprint("Stopping due to keyboard variant 1.")
         stop = True
         break
      if zero_pad:
         M = str(M).zfill(2)
      else:
         M = str(int(M))
      YM = f"{Y}/{M}"
      if YM not in destdirs: destdirs.append(YM)
      fdict[f] = YM
   # finish the for f in flist

   # short-circuit if keyboard action happened in one of the nested functions
   if stop:
      return -1

   # Make directories
   for d in destdirs:
      dd = path.join(outdir,d)
      if debuglevel >= 3:
         eprint(f"Make dir: {dd}")
      if not dryrun:
         Path(dd).mkdir(parents=True, exist_ok=True)

   # Make symlinks
   for f in fdict:
      ff = f
      basename = path.basename(ff)
      destfile = path.join(outdir,fdict[f],basename)
      if action == "symlink" and relative_symlinks:
         ff = path.relpath(ff,destfile)
      if debuglevel >= 2:
         if action == "copy":
            eprint(f"cp -p {ff} {destfile}")
         elif action == "symlink":
            eprint(f"ln -s {ff} {destfile}")
         elif action == "move":
            eprint(f"mv {ff} {destfile}")

      if not dryrun:
      #if False:
         if action == "copy":
            copy2(ff,destfile)
         elif action == "symlink":
            symlink(ff,destfile)
         elif action == "move":
            move(ff,destfile)

   return result

Shell: find duplicate files and replace most with symlinks

bgstack15 — Thu, 20 Feb 2020 14:18:07 GMT

This is probably not the best solution, but it's the one I wanted to work with. I intended to reduce disk space of 3 similar projects. Here is my shell script to find duplicate files (by md5sum) and replace any secondary file with a symlink to the relative path of the original file.

#!/usr/bin/env sh
# File: set-symlinks.sh
# License: CC-BY-SA 3.0
# Author: bgstack15
# Startdate: 2020-02-07 14:03
# Title: Script that Replaces Duplicate Files with Symlinks
# Purpose:
# History:
# Usage:
# Reference:
#    https://stackoverflow.com/questions/2564634/convert-absolute-path-into-relative-path-given-a-current-directory-using-bash
# Improve:
# Dependencies:
#    coreutils >= 8.23

INDIR=~/src/project

#results="$( find "${INDIR}" ! -type d ! -type l -exec md5sum {} + | sort )"
find "${INDIR}" ! -type d ! -type l -exec md5sum {} + | sort | \
   awk '{a[$1]=a[$1]":"$2} END {for (i in a){print a[i]}}' | sed -r -e 's/^://;' | \
   while IFS=':' read main child1 child2 child3 child4 ;
   do
      x=0
      while test $x -lt 4 ;
      do
         x=$(( x + 1 ))
         eval thischild="\${child$x}"
         if test -n "${thischild}" ;
         then
            linkname="$( realpath --relative-to "$( dirname "${thischild}" )" "${main}" 2>/dev/null )"
            test -n "${DEBUG}" && echo "ln -sf ${linkname} ${thischild}" 1>&2
            test -z "${DRYRUN}" && ln -sf "${linkname}" "${thischild}"
         fi
      done
   done

I was going to do this task for myself by hand, but then a quick investigation showed 73 files that were duplicates. Because of the small size of the project, I decided to just run it in shell and not revert to Python. I don't need efficiency; I just need to run it once, really. The tricky bits are in the very front of the logic. The awk associate array builds a list of all filenames that correspond with an md5sum. Then, stripping out the leading colon (separator), I pipe the output to a while for easy variable naming. And then loop a few times (hard-coded to 4) and if item number X exists, get the relative path to the main file, and force create the symlink.

python readlink function

bgstack15 — Sat, 02 Sep 2017 13:17:23 GMT

tl;dr

def readlinkf(_inpath):
   if os.path.islink(infile):
      return os.path.join(os.path.dirname(_inpath),os.readlink(_inpath))
   else:
      return _inpath

infile=readlinkf(infile)

Explanation

In GNU, there's a fantastic utility named readlink. The main way I use it is to get the full and true path of a file or symlink. For example:

$ ls foo .config/qux
.config/qux  foo
$ readlink -f foo
/home/bgstack15/.config/qux

In python, I wanted to achieve the same thing. it is possible with the os.path.islink and os.readlink functions. Here is my quick function for a readlink -f , since I didn't find a specific implementation that outputs the full path.

Reference

Weblinks

https://stackoverflow.com/questions/11068419/check-if-file-is-symlink-in-python

Online reference

Search expressions

Shell one-liners for manipulating symlinks

bgstack15 — Tue, 22 Nov 2016 21:15:23 GMT

Showing symlink targets

I was trying to clean up a directory filled with files and symlinks. Many of the symlinks' targets did not exist anymore (old package versions). I wrote a few one-liners to help me examine and clean up the directory. Here's the first one-liner. find . -type l -print0 | sed -e 's/\.\///g;' | xargs -0 -n1 | while read line; do canon=$( readlink -f "${line}" 2>/dev/null ); test -e "${canon}" && echo "${line} is $( /bin/ls -d --color=always "${canon}" )" || echo "INVALID ${line}"; done In expanded form: find . -type l -print0 | \ sed -e 's/\.\///g;' | \ xargs -0 -n1 | \ while read line; do \ canon=$( readlink -f "${line}" 2>/dev/null ); test -e "${canon}" && echo "${line} is $( /bin/ls -d --color=always "${canon}" )" || echo "INVALID ${line}"; done The find command lists symlinks in the current directory, with special null characters as the separators. This will allow correct operation with spaces in filenames. You don't normally encounter this in GNU/Linux, but you never know! The sed removes the dot slash ./ from the beginning of filenames. I just don't like seeing that. If I had selected find * earlier, I wouldn't need the sed command, but I would miss any files whose names begin with dot (e.g., .gitignore) and I would still find file . which I would have to exclude somehow anyway (well, not here because I'm searching for symlinks, but I personally never try to run find * no matter what). xargs -0 -n1 now removes the null character separators, and feeds each individual entry one at a time to the next command. While: So for each entry, read the symlink. If the target is a file, display it in whatever color ls would show it (without entering directories), because colors are pretty! Otherwise, print the line with "INVALID" in front of it.

Cleaning up the invalid symlinks

find . -type l -print0 | sed -e 's/\.\///g;' | xargs -0 -n1 | while read line; do canon=$( readlink -f "${line}" 2>/dev/null ); test -e "${canon}" && true || { echo /bin/rm "./${line}"; }; done Replace the echo commands with true for the affirmative and the echo /bin/rm command. To actually effect the change, remove the echo in front of /bin/rm.