Knowledge Base

Preserving for the future: Shell scripts, AoC, and more

Building an apt repo on CentOS that supports apt-file operations

bgstack15

Comments

This is a newer version of Building an apt repository on CentOS

Overview

My network infrastructure consists of CentOS 7 systems and Devuan client systems. I maintain mirrors of Devuan Ceres locally so I only need one outbound operation. In addition to the official repos (albeit stored in an unofficial manner), I also maintain my own collections of packages, as the first link in this post describes. One thing I noticed though, is that apt-file does not operate on the packages in my own repositories. So I used the wonderful free and open source nature of all the great tools that make up Devuan's apt software, and wrote my own set of tools that add the support for apt-file. Adding the apt-file support requires modifications to the server, as well as the clients. This is acceptable because I of course am the admin on my own systems and can add /etc contents at will.

Configuring the repo server

On the CentOS 7 server, you need to generate gpg keys (see Weblink 1). I have elected to store the passphrase in plaintext (bottom of Weblink 1) as file /root/.gnupg/passphrasefile

Repo update script

Write the wrapper script. Looking back I realize I should have set this up as a config file and then I just need to invoke the lib script as the main script, with parameters, but I'll save that for a future refactoring.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
#!/bin/sh
# Filename: update-exampledeb.sh
# Location: /mnt/public/www/example/repo/deb
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2017-07-23
# Title: Script that updates apt repo "exampledeb"
# Purpose: automate rebuilding the repo
# History:
#    2020-10-23 adding apt-file compatibility.
# Usage:
#    just call this after adding a new package to the repository
# Reference:
# Improve:
# Documentation:
# Dependencies:

# Set variables
export repodir=/mnt/public/www/example/repo/deb/
export ownership="apache:admins"
export filetypes="deb"
export gpgkey_passphrase_file="/root/.gnupg/passphrasefile"
test -z "${SKIP_SCAN}" && export SKIP_SCAN=0
test -z "${SKIP_CONTENTS}" && export SKIP_CONTENTS=0

# load library, which validates the above variables
. /mnt/public/www/example/repo/deb/scripts/apt-repo-lib.sh

# Prepare directory and files
fix_owner_and_mode

# Prepare repo for apt
make_apt_repo

# create the Release and InRelease files
make_release_files

Update repo library

All of the logic is in the dot-sourced library script.

#/bin/sh
# File: /mnt/public/www/example/repo/deb/scripts/apt-repo-lib.sh
# Location: server1
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2020-10-23
# Title: Library for apt repo update scripts
# Purpose: provide common functions for each apt repo to reduce code duplication
# History:
#    2020-10-22 the make_contents functions were started 
#    2020-10-23 this library started and took basically all logic out of the update script itself, because it is all boilerplate minus a few variables.
# Usage: only dot-source from an upate-*.sh repo update script.
# Reference:
#    /mnt/public/www/example/repo/deb/update-exampledeb.sh
# Improve:
#    actually bother to build content files per architecture.
# Documentation:
#    designed to run on CentOS 7 where not all apt tools exist.
# Dependencies:
#    gzip, gpg2

# Functions
_mc_inner() {
   word="${1}"
      name="$( dpkg-deb --show "${word}" | awk '{print $1}' )"
      # list dpkg contents and remove directories from listing
      dpkg-deb --contents "${word}" | awk '!/\/$/ {$1="";$2="";$3="";$4="";$5="";print}' | sed -r -e 's/^\s+//' | \
         # change listings to be relative paths, like reference Contents file, and append package name
         # this appears to be incomplete compared to reference Contents file because we do not include the "section" information, e.g., "contrib/admin/zfs-test"
         sed -r -e 's:^\./::' -e "s:$:\t${name}:;" ;
      # reset var just in case
      echo "DONE: ${word}" 1>&2
}

make_contents() {
   # call: make_contents "/path/to/dir" "reponame" "file"
   _path="${1:-.}"
   _reponame="${2}"
   test "." != "${_path}" && pushd "${_path}" 1>/dev/null 2>&1
   # use the . for path so total count of parameter characters is shorter
   for word in $( find . -name "*.deb" ! -name '*teamviewer*' ) ; do
      _mc_inner "${word}" &
   done | \
      # only show new entries, which hopefully will make sort faster. It is possible this is not useful.
      awk '!x[$0]++' | \
      # sort output and show only unique lines. This is required in my repos because I leave multiple versions of a single package around.
      sort | uniq
   popd 1>/dev/null 2>&1
}

fail() {
   _ec="${1}" ; shift 1 ;
   echo "${@}" ;
   return "${_ec}"
}

fix_owner_and_mode() {
   find "${repodir}" -exec chown "${ownership}" {} + 1>/dev/null 2>&1
   find "${repodir}" -type f ! -name '*.sh' -exec chmod "0664" {} + 1>/dev/null 2>&1
   find "${repodir}" -type f -name '*.sh'   -exec chmod "0754" {} + 1>/dev/null 2>&1
   find "${repodir}" -type d -exec chmod "0775" {} + 1>/dev/null 2>&1
   restorecon -RF "${repodir}"
}

make_apt_repo() {
   pushd "${repodir}" 1>/dev/null 2>&1
   if ! test "${SKIP_SCAN}" = "1" ;
   then
      dpkg-scanpackages -m . > Packages # this takes a long time to run
      gzip -9c Packages > Packages.gz
   fi
   if ! test "${SKIP_CONTENTS}" = "1" ;
   then
      make_contents . "" "file" > "${repodir}/Contents"
      wait
      cd "${repodir}"
      # apt-file needs -amd64.gz and -i386.gz files, based on default /etc/apt/apt.conf.d/50apt-file.conf
      for word in amd64 i386 all ; do cp -pf Contents "Contents-${word}" ; done
      for word in Contents* ; do ! echo "${word}" | grep -qE '\.gz$' && gzip -9c "${word}" > "${word}.gz" ; done
   fi
   popd 1>/dev/null 2>&1
}

make_release_files() {
   pushd "${repodir}" 1>/dev/null 2>&1
   # create the Release and InRelease files
   md5s="$( for word in Packages* Contents* ; do printf "%s %s\n" "$( md5sum "${word}" | cut -d" " -f1 )" "$( wc -c "${word}" )" ; done | column -t | sed -r -e 's/^/ /;' )"
   sha1s="$( for word in Packages* Contents* ; do printf "%s %s\n" "$( sha1sum "${word}" | cut -d" " -f1 )" "$( wc -c "${word}" )" ; done | column -t | sed -r -e 's/^/ /;' )"
   sha2s="$( for word in Packages* Contents* ; do printf "%s %s\n" "$( sha256sum "${word}" | cut -d" " -f1 )" "$( wc -c "${word}" )" ; done | column -t | sed -r -e 's/^/ /;' )"
   cat <<EOF > Release
Architectures: all
Date: $(date -u '+%a, %d %b %Y %T %Z')
MD5Sum:
${md5s}
SHA1:
${sha1s}
SHA256:
${sha2s}
EOF
   gpg --batch --yes --passphrase-file "${gpgkey_passphrase_file}" --pinentry-mode loopback -abs -o Release.gpg Release
   gpg --batch --yes --passphrase-file "${gpgkey_passphrase_file}" --pinentry-mode loopback --clear-sign -o InRelease Release
   popd 1>/dev/null 2>&1
}

# When dot-sourcing the library, validate all input parameters
test -z "${repodir}" && { fail 1 "Fatal! Need \"repodir\" defined." || exit 1 ; }
test -z "${ownership}" && { fail 1 "Fatal! Need \"ownership\" defined, nominally \"apache:admins\"." || exit 1 ; }
test -z "${filetypes}" && { fail 1 "Fatal! Need \"filetypes\" defined, nominally \"deb\"." || exit 1 ; }
test -z "${gpgkey_passphrase_file}" && { fail 1 "Fatal! Need \"gpgkey_passphrase_file\" defined." || exit 1 ; }

Putting it all together on the server

When I have a new package underneath /mnt/public/www/example/repo/deb/, I just need to run sudo /mnt/public/www/example/repo/deb/update-exampledeb.sh. Previously my separate repositories were using duplicates of a single script. But now, they can all reference this one library and the in-tree update script is basically an executable config file.

Configuring clients

My architecture of the server, with the flat repository style (i.e., without dists/ceres/main/contrib directories) made it difficult to configure clients to always fetch the Contents files which matter. The clients need the apt repo definitions adjusted, and a custom apt-file conf.d added. The apt repo, nominally file /etc/apt/sources.list.d/exampledeb.list, needs to look like:

deb [target-=Contents-deb target+=Contents-stackrpms] http://www.example.com/example/repo/deb/ /

The target commands tell apt to use the custom indexing definition from this next file. Configure apt with file /etc/apt/apt.preferences.d/52apt-file- stackrpms.conf

# File: /etc/apt/apt.preferences.d/52apt-file-stackrpms.conf
# Part of support devuan scripts
# This enables the flat apt repos in example to be supported by apt-file
Acquire::IndexTargets {
    deb::Contents-stackrpms {
        MetaKey "Contents-$(ARCHITECTURE)";
        ShortDescription "Contents-$(ARCHITECTURE)";
        Description "$(RELEASE) $(ARCHITECTURE) Contents (deb)";

        flatMetaKey "Contents-$(ARCHITECTURE)";
        flatDescription "$(RELEASE) Contents (deb)";
        PDiffs "true";
        KeepCompressed "true";
        DefaultEnabled "false";
        Identifier "Contents-deb";
    };
};

Looking back, I probably should have just learned how to use the built-in "Contents-deb-legacy" but why make things too simple? I accomplish the client configuration on my own network with a script that does a couple of extra things, but I will not cover everything that it does here.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
 
#!/bin/sh
# File: /mnt/public/Support/Platforms/devuan/set-my-repos.sh
# Location:
# Author: bgstack15
# Startdate: 2019-08-10 16:02
# Title: Script that Establishes the repos needed for Devuan
# Purpose: Set up the 3 repos I always need on devuan clients
# History:
#    2020-02-01 customize clients for devuan-archive
#    2020-10-23 add apt-file compatibility
# Usage:
#    sudo set-my-repos.sh
# Reference:
#    /mnt/public/Support/Platforms/devuan/devuan.txt
# Improve:
#    need to control the sources.list file itself to have the main, contrib, etc., for ceres.
# Documentation:

test -z "${ALLREPOSGLOB}" && ALLREPOSGLOB="/etc/apt/sources.list /etc/apt/sources.list.d/*"
test -z "${REPOSBASE}" && REPOSBASE="/etc/apt/sources.list.d"
test -z "${PREFSBASE}" && PREFSBASE="/etc/apt/preferences.d"
test -z "${ADDLCONFBASE}" && ADDLCONFBASE="/etc/apt/apt.conf.d"

# confirm key
confirm_key() {
   # call: confirm_key "${PRETTYNAME}" "${SEARCHPHRASE}" "${URL_OF_KEY}"
   ___ck_repo="${1}"
   ___ck_sp="${2}"
   ___ck_url="${3}"
   if apt-key list 2>/dev/null | grep -qe "${___ck_sp}" ;
   then
      :
   else
      # not found so please add it
      echo "Adding key for ${___ck_repo}" 1>&2
      wget -O- "${___ck_url}" | sudo apt-key add -
   fi
}

# confirm repo
confirm_repo() {
   # call: confirm_repo "${PRETTYNAME}" "${SEARCHPHRASE}" "${SEARCHGLOB}" "${FULLSTRING}" "${PREFERRED_FILENAME}" "${OVERWRITE}"
   ___cr_repo="${1}"
   ___cr_sp="${2}"
   ___cr_sf="${3}"
   ___cr_full="${4}"
   ___cr_pref="${5}"
   ___cr_overwrite="${6}"
   if ! grep -E -qe "${___cr_sp}" ${___cr_sf} ;
   then
      # not found so please add it to preferred file
      echo "Adding repo ${___cr_repo}" 1>&2
      if test "${___cr_overwrite}" = "true" ;
      then
         # overwrite, instead of append
         echo "${___cr_full}" > "${REPOSBASE}/${___cr_pref:-99_misc.list}"
      else
         echo "${___cr_full}" >> "${REPOSBASE}/${___cr_pref:-99_misc.list}"
      fi
   fi
}

confirm_preferences() {
   # call: confirm_preferences "${PRETTYNAME}" "${FILENAME}" "{PACKAGE}" "${PIN_EXPRESSION}" "{PRIORITY}"
   ___cp_prettyname="${1}"
   ___cp_pref="${2}"
   ___cp_package="${3}"
   ___cp_pin_expression="${4}"
   ___cp_priority="${5}"

   ___cp_tempfile="$( mktemp )"
   {
      echo "Package: ${___cp_package}"
      echo "Pin: ${___cp_pin_expression}"
      echo "Pin-Priority: ${___cp_priority}"
   } > "${___cp_tempfile}"

   diff "${PREFSBASE}/${___cp_pref}" "${___cp_tempfile}" 1>/dev/null 2>&1 || {
      echo "Setting preferences for ${___cp_prettyname}"
      touch "${PREFSBASE}/${___cp_pref}" ; chmod 0644 "${PREFSBASE}/${___cp_pref}"
      cat "${___cp_tempfile}" > "${PREFSBASE}/${___cp_pref}"
   }

   rm -f "${___cp_tempfile:-NOTHINGTODEL}" 1>/dev/null 2>&1
}

# REPO 1: local exampledeb
confirm_key "exampledeb" "bgstack15.*example\.example\.com" "http://www.example.com/example/repo/deb/exampledeb.gpg"
confirm_repo "exampledeb" "target.*example\/repo\/deb" "${ALLREPOSGLOB}" "deb [target-=Contents-deb target+=Contents-stackrpms] http://www.example.com/example/repo/deb/ /" "exampledeb.list" "true"

# REPO 2: local devuan-deb
confirm_key "devuan-deb" "bgstack15.*example\.example\.com" "http://www.example.com/example/repo/deb/exampledeb.gpg"
confirm_repo "devuan-deb" "target.*example\/repo\/devuan-deb" "${ALLREPOSGLOB}" "deb [target-=Contents-deb target+=Contents-stackrpms] http://www.example.com/example/repo/devuan-deb/ /" "devuan-deb.list"

# REPO 3: local obs
#confirm_key "OBS bgstack15" "bgstack15@build\.opensuse\.org" "https://download.opensuse.org/repositories/home:bgstack15/Debian_Unstable/Release.key"
#confirm_repo "OBS bgstack15" "repositories\/home:\/bgstack15\/Debian_Unstable" "${ALLREPOSGLOB}" "deb http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable/ /" "home:bgstack15.list"
confirm_key "OBS bgstack15" "bgstack15@build\.opensuse\.org" "http://www.example.com/mirror/obs/Release.key"
confirm_repo "OBS bgstack15" "mirror\/obs" "${ALLREPOSGLOB}" "deb http://www.example.com/mirror/obs/ /" "home:bgstack15.list"

# REPO 4: local devuan-archive
confirm_key "devuan-archive" "bgstack15.*example\.example\.com" "http://www.example.com/example/repo/deb/exampledeb.gpg"
confirm_repo "devuan-archive" "target.*server1((\.ipa)?\.example\.com)?(:180)?.*example\/repo\/devuan-archive" "${ALLREPOSGLOB}" "deb [target-=Contents-deb target+=Contents-stackrpms] http://server.ipa.example.com:180/example/repo/devuan-archive/ /" "devuan-archive.list"
confirm_preferences "devuan-archive" "puddletag" "*" "origin server1.ipa.example.com" "700"

# ADDITIONAL APT PREFS
# important for the [target] stuff to work on repos so apt-file can work
cp -p "$( dirname "$( readlink -f "${0}" )")/input/52apt-file-stackrpms.conf" "${ADDLCONFBASE}/"

Backstory

I want to be able to view the contents of my packages without having to install them first. Apt supports this, but it took half a day to discover how to generate the Contents file, get it listed in the Release/InRelease file, actually generate an InRelease file (the gpg-signed Release file), and get clients to pull down the Contents files. Coincidentally, apt clients store the Contents files compressed with lz4, and not gzip like apt repos tend to provide. Go figure. I found nothing on the Internet for using apt-file with flat apt repositories, so I am assuming this is an original concept. I'm guessing all the shops that bother with custom repos use one of those [DebianRepository Setup tools. I assume all those Debian-focused apt repo tools already take all these steps, but my limitation here was CentOS. It makes for a fun challenge, that is solvable within my skillset and thanks to the open source nature of these great tools.

References

Weblinks

Building an apt repository on CentOS gpg key instructions Package for CentOS 7: gnupg2-2.2.18-2.el7 newer version of gpg so the apt tools work in CentOS

Local files

/usr/share/doc/apt-file/README.md.gz

Manpages

apt-file(1)

Comments