Knowledge Base (Posts about obs)

OBS and build dpkgs note

bgstack15 — Thu, 26 Sep 2024 12:31:00 GMT

The Open Build Service is fantastic. I need to figure how to donate a little money to the SuSE team that offers this to free software admins like me.

Their documentation is really hard to find, so when I had a new configuration for a _service file (seen below), I ran into a problem.

<services>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://bgstack15.ddns.net/cgit/stackrpms</param>
      <param name="subdir">debian</param>
      <param name="filename">for-dsc-only</param>
      <param name="revision">master</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">xz</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://bgstack15.ddns.net/cgit/fprintd-tk</param>
      <param name="subdir">debian</param>
      <param name="filename">debian</param>
      <param name="revision">master</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">xz</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://bgstack15.ddns.net/cgit/fprintd-tk</param>
      <param name="revision">master</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">gz</param>
   </service>
   <service name="extract_file">
      <param name="archive">for-dsc-only.tar.xz</param>
      <param name="files">*/*.dsc</param>
   </service>
</services>

The above is almost my standard method for pulling down the *.dsc file that tells OBS we are building a dpkg, and the source tarball. This time, we are pulling down the .dsc file in an archive separately from even the rest of the debian/ build recipe. And this method fails with an error:

 Too many files looking like a usable source tarball (would not know which to pick): fprintd-tk.tar.xz for-dsc-only.tar.xz

The solution, which was available after searching the Internet, should have been to add a debtransform command in the .dsc file:

DEBTRANSFORM-TAR: fprintd-tk.tar.xz

And that accepts things like @VERSION@ but I haven't bothered with that. I took the quicker, dirtier method in a later comment, which is to modify the _service definition.

The snippet that matters:

   <service name="tar_scm">
      ...
      <param name="extension">tar_</param>
      <param name="extract">*.dsc</param>
   </service>

And man, I never knew about the extract parameter before. Remember how those docs are hard to find? (And actually, in the documentation I could find nothing described parameter "extract" for service "tar_scm".

The full service definition is below.

<services>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://bgstack15.ddns.net/cgit/stackrpms</param>
      <param name="subdir">fprintd_tk/debian</param>
      <param name="filename">for-dsc-only</param>
      <param name="extension">tar_</param>
      <param name="revision">master</param>
      <param name="version">_none_</param>
      <param name="extract">*.dsc</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://bgstack15.ddns.net/cgit/fprintd-tk</param>
      <param name="subdir">debian</param>
      <param name="filename">debian</param>
      <param name="revision">master</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">xz</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://bgstack15.ddns.net/cgit/fprintd-tk</param>
      <param name="revision">master</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">gz</param>
   </service>
</services>

Librewolf 121.0.1 now in OBS

bgstack15 — Fri, 26 Jan 2024 13:48:22 GMT

I don't normally post on my blog merely when an application is updated in my OBS slice, but it's a big deal for this one because multiple pipeline problems have plagued this application.

I provide debianized LibreWolf sources so anyone can build his own package. I also use the public LibreWolf CI infrastructure to build the assets that then also get used by OBS to build the package.

Problem 1

The public (and only, that I know of) CI for librewolf hasn't worked in a few versions, and I haven't bothered to find my alt browser to sign in to whatever chat room it was to ask the LibreWolf folks what's up. I think it was a free instance or something and they probably reached their quota.

Problem 2

Due to the usrmerge drama in Debian, the OpenSUSE-focused Open Build Service doesn't have a great way to work with dpkgs. But finally, after enough project config modulations that include preinstall: systemd amongst a few other things, I can build packages again.

preinstall: usrmerge
prefer: usrmerge
prefer: libjpeg-dev, libjpeg62-turbo-dev
prefer: libjack-dev
prefer: libavcodec60
prefer: libavformat60
prefer: libavfilter9
Preinstall: libsystemd0
Preinstall: systemd

So I manually uploaded the 121.0.1 assets I built locally and dropped the _service file for now which tells the OBS worker node to fetch the publicly-built assets.

<services>
  <service name="download_url">
    <param name="protocol">https</param>
    <param name="host">storage.ci.librewolf.net</param>
    <param name="path">/artifacts/0/prepared/librewolf_120.0.1-1.debian.tar.xz</param>
    <param name="filename">librewolf_120.0.1-1.debian.tar.xz</param>
  </service>
  <service name="download_url">
    <param name="protocol">https</param>
    <param name="host">storage.ci.librewolf.net</param>
    <param name="path">/artifacts/0/prepared/librewolf_120.0.1-1.dsc</param>
    <param name="filename">librewolf_120.0.1-1.dsc</param>
  </service>
  <service name="download_url">
    <param name="protocol">https</param>
    <param name="host">storage.ci.librewolf.net</param>
    <param name="path">/artifacts/0/prepared/librewolf_120.0.1.orig.tar.xz</param>
    <param name="filename">librewolf_120.0.1.orig.tar.xz</param>
  </service>
</services>

And with my 3 manual assets, the package has now built. You're welcome, Infinity128, and everybody else!

An aside

The reason I use an alternate browser to visit the chat room with the LibreWolf guys is because it used to be gitter.im which used gitlab and wanted full access to your account, so I made a bgstack150 account on gitlab and used that to talk. I didn't want gitter.im having full access to whatever I have left on gitlab, so I used that alt account, and since it tied in SSO-like (read: Borg-like) to gitlab, I needed a different browser session and I've been too lazy to learn how to use multiple profiles in the same browser. But the last time I was in there, it was powered by matrix and is probably not necessary, but I still haven't bothered. The one other Librewolf CI project is alpine and that hasn't built in 2 months either so clearly they just weren't using it.

Improvements to obsmirror

bgstack15 — Thu, 07 Apr 2022 12:58:25 GMT

Previous posts on the topic of mirroring the OBS apt repos locally:

I have improved the shell script that is the main obsmirror logic. I also split out the rebuild-apt-repo logic into its own script so I could call it separately.

	`#!/bin/sh`
	`# File: /etc/installed/obsmirror.sh`
	`# Location: https://gitlab.com/bgstack15/former-gists/tree/master/obsmirror.sh`
	`# Author: bgstack15`
	`# Startdate: 2020-03-03 08:43`
	`# SPDX-License-Identifier: CC-BY-SA-4.0`
	`# Title: Script that scrapes down OBS site to serve a copy to intranet`
	`# Purpose: save down my OBS site so I can serve it locally`
	`# History:`
	`# 2020-01-05 v1: begin which used httrack`
	`# 2020-02-28 v2: complete rewrite to exclude httrack`
	`# 2020-03-03 v3: complete rewrite to get explicit files and loop through their contents, and rebuild apt repo`
	`# 2020-03-13 add on-prompt notifications`
	`# 2022-04-01 22:33`
	`# Usage:`
	`# in a cron job: /etc/cron.d/mirror.cron`
	`# 50 12 * * * root /etc/installed/obsmirror.sh 1>/dev/null 2>&1`
	`# Reference:`
	`# https://software.opensuse.org//download.html?project=home%3Abgstack15&package=freefilesync`
	`# /mnt/public/www/internal/repo/devuan-deb/update-devuan-deb.sh`
	`# https://medium.com/sqooba/create-your-own-custom-and-authenticated-apt-repository-1e4a4cf0b864`
	`# https://unix.stackexchange.com/questions/113898/how-to-merge-two-files-based-on-the-matching-of-two-columns/113903#113903`
	`# sed\|sed to get line numbers printed https://stackoverflow.com/questions/52882594/insert-line-numbers-into-file-with-sed/52884598#52884598`
	`# Improve:`
	`# Documentation:`
	`# Download the release key and trust it.`
	`# curl -s http://repo.example.com/mirror/obs/Release.key \| apt-key add -`
	`# Use a sources.list.d/ file with contents:`
	`# deb https://repo.example.com/mirror/obs/ /`
	`# Dependencies:`
	`# binaries: wget sed awk`
	`# user: obsmirror`
	`umask 0002`
	`exec 8>&0`

	`test -n "${OBSMIRROR_CONF}" && . "${OBSMIRROR_CONF}"`
	`test -z "${logfile}" && logfile="/tmp/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"`
	`test -z "${inurl}" && inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"`
	`test -z "${workdir}" && workdir=/tmp/obs`
	`test -z "${thisuser}" && thisuser=obsmirror`
	`test -z "${tempdir}" && tempdir="$( TMPDIR="${TMPDIR:-/tmp}" mktemp -d )"`
	`# also use include_sources resign_repo gpg_passfile gpg_keyfile DEBUG`

	`exec 3>&1`
	`show() {`
	`printf "%s" "${*}" 1>&3`
	`}`

	`reset_show() {`
	`printf "\r%s" "${*}" 1>&3`
	`}`

	`get_file() {`
	`# call: get_file "${tu}" "${md5sum}"`
	`___tu="${1}"`
	`___sum1="${2}" # nominally from the locally-generated Packages from previous run`
	`___sum2="${3}" # from current set`
	`tn="${___tu##${inurl}}"`
	`tf="${workdir}/${tn}" ; tf="$( readlink -m "${tf}" )"`
	`td="$( dirname "${tf}" )"`
	`test -d "${td}" \|\| mkdir -p "${td}"`
	`gotten="skipped "`
	`if test -z "${DRYRUN}" ;`
	`then`
	`# determine if file is good enough`
	`___matches_either=0`
	`___md5sum_file="$( md5sum "${tf}" 2>/dev/null \| awk '{print $1}' )"`
	`# sum1 check is disabled. If upstream obs rebuilds a package, we would never download the new package of the exact same name+version! So all my fancy sum1 logic is useless.`
	`#test "${___md5sum_file}" = "${___sum1}" && ___matches_either=1`
	`test "${___md5sum_file}" = "${___sum2}" && ___matches_either=1`
	`#test -n "${VERBOSE}" && printf "%s matches:%s\n" "${tn}" "${___matches_either}"`
	`if test -z "${___sum2}" \|\| test "${___matches_either}" = "0" ;`
	`then`
	`# so the checksum is empty, or the given checksum does not match the existing downloaded file`
	`# use the Link headers because provo-mirror sucks and presents a lot of 404s.`
	`# if Links header does not exist then this list will be short.`
	`___links="$( printf '%s\n%s\n' "${___tu}" "$( curl --head "${___tu}" --silent \| awk '/Link:/ && !/type=/{print $2}' \| tr -d '<>;' )" \| sed -e '/^\s*$/d' )"`
	`test -n "${VERBOSE}" && show "retrieving ${___tu}" 2>/dev/null \|\| :`
	`___valid=0`
	`___x=0`
	`while test ${___valid} -eq 0 && test ${___x} -lt $( echo "${___links}" \| wc -l ) ;`
	`do`
	`___x=$((___x+1))`
	`___tl="$( echo "${___links}" \| sed -n "${___x}p" 2>/dev/null )"`
	`test -n "${___tl}" && wget --content-disposition --no-verbose --quiet --output-document "${tf}" "${___tl}" && ___valid=1`
	`grep -qiE '404 Not Found' "${tf}" 2>/dev/null && ___valid=0`
	`! test -s "${tf}" && ___valid=0`
	`___matches_either=0`
	`___md5sum_file="$( mdtsum "${tf}" 2>/dev/null \| awk '{print $1}' )"`
	`#test "${___md5sum_file}" = "${___sum1}" && ___matches_either=1`
	`test "${___md5sum_file}" = "${___sum2}" && ___matches_either=1`
	`if test "${___matches_either}" = "1" ;`
	`then`
	`___valid=1`
	`fi`
	`done`
	`test ${___valid:-0} -eq 1 && gotten="DOWNLOADED"`
	`test ${___valid:-0} -eq 0 && gotten='x FAILED '`
	`fi`
	`fi`
	`test -n "${VERBOSE}" && reset_show 2>/dev/null \|\| :`
	`test -n "${VERBOSE}" && echo "${gotten} ${___tu} -> ${tf}"`
	`#echo "PAUSED: " ; read -u8 foo`
	`}`

	`wget_verbose=--quiet`
	`test -n "${VERBOSE}" && unset wget_verbose`
	`{`
	`test "${DEBUG:-NONE}" = "FULL" && set -x`
	`echo "logfile=${logfile}"`

	`# These files define an apt repo`
	`# archive the Packages file, which might be generated locally by rebuild-apt-repo.sh the previous time, and might have more useful md5sums of the packages`
	`/bin/cp -pf "${workdir}/Packages" "${tempdir}/Packages.$$"`
	`for word in InRelease Packages Packages.gz Release Release.gpg Release.key Sources Sources.gz ;`
	`do`
	`get_file "${inurl}/${word}"`
	`done`

	`# loop through named packages and download them`
	`# extract these 2 fields, from both old and new Packages files. Combine them into one line. Sed\|sed adds the line numbers so we can put the lines back in original order just because, and then sort.`
	`awk '/Filename:\|MD5/{print $2}' "${tempdir}/Packages.$$" \| xargs -n2 \| sed '=' \| sed 'N; s/\n/ /' \| sort -k2 > "${tempdir}/old_list"`
	`awk '/Filename:\|MD5/{print $2}' "${workdir}/Packages" \| xargs -n2 \| sed '=' \| sed 'N; s/\n/ /' \| sort -k2 > "${tempdir}/new_list"`
	`#awk '/Filename:\|MD5/{print $2}' "${workdir}/Packages" \| xargs -n2 \| while read word sum`
	`# The sort above was so join will work. Print these columns, sort by original line number, and then remove the lineno column with awk. Then split into these var names and process.`
	`join -j 2 -a 2 -o 2.1,2.2,1.3,2.3 "${tempdir}/old_list" "${tempdir}/new_list" \| sort -k1 \| awk '{$1="";print}' \| while read word sum_old sum_new`
	`do`
	`get_file "$( echo "${word}" \| sed -r -e "s@^\.@${inurl}@;" )" "${sum_old}" "${sum_new}"`
	`#echo "a=${a} b=${b}"`
	`done`

	`# loop through dsc, orig.tar.gz, and debian.tar.xz files`
	`test -n "${include_sources}" && {`
	`for word in $( sed -n -r -e '/Files:/,/^\s*$/{/^ /p;}' ${workdir}/Sources \| awk '{print $NF}' ) ;`
	`do`
	`get_file "${inurl}/${word}"`
	`done`
	`}`

	`test -n "${resign_repo}" && . /etc/installed/rebuild-apt-repo.sh`

	`chown -R "${thisuser}:$( id -G "${thisuser}" \| awk '{print $1}' )" "${workdir}"`
	`} 2>&1 \| tee -a "${logfile}"`

	`test -z "${NO_CLEAN}" && rm -rf "${tempdir:-NOTHINGTODEL}"`

Observe the ___sum1 lines that are commented out. I wrote a lot of logic to support reading the old (previous run's locally-generated) Packages file and accepting the .deb checksums from it. I realized that due to how I rebuild the exact same version of a package (yes, I'm a terrible person and I know it), accepting the old Packages's checksum for a file means that it would never download the newer build of the exact same version-release number. So I disabled allowing it from the old Packages file, which entirely defeats the purpose of my byzantine logic to join the two Packages entries' together and comparing against both checksums for a given file.

Here is the rebuild-apt-repo.sh script.

	`#!/bin/sh`
	`# Startdate: 2022-04-02 19:51`
	`# Purpose: To rebuild apt repo, primarily for obsmirror operations`
	`# Usage: Can be called by itself, with appropriate env vars, or from obsmirror.sh`
	`# ( . /etc/installed/obsmirror-cdemu.conf ; . /etc/installed/rebuild-apt-repo.sh ; )`
	`# Dependencies:`
	`# environment vars: workdir, gpg_passfile, gpg_keyfile`
	`# On CentOS7, gnupg2 package that supports --pinentry-mode loopback, 2.2.18-2.el7 from @copr:bgstack15:el7-gnupg2-debmirror`
	`# /usr/bin/dpkg-scanpackages`
	`# References:`
	`# obsmirror.sh`
	`# https://medium.com/sqooba/create-your-own-custom-and-authenticated-apt-repository-1e4a4cf0b864`

	`# rebuild release files`
	`echo "Rebuild apt repo in ${workdir}"`

	`repodir="${workdir}"`
	`cd "${repodir}"`
	`dpkg-scanpackages -m . > Packages`
	`gzip -9c < Packages > Packages.gz`
	`# create the Release file`
	`PKGS="$(wc -c Packages)"`
	`PKGS_GZ="$(wc -c Packages.gz)"`
	`old_headers1="$( grep -E '^(Archive\|Codename\|Origin\|Label\|Architectures):' Release )"`
	`old_headers2="$( grep -E '^(Description):' Release )"`
	`cat <<EOF > Release`
	`${old_headers1}`
	`Date: $(date -u '+%a, %d %b %Y %T %Z')`
	`${old_headers2}`
	`MD5Sum:`
	`$(md5sum Packages \| cut -d" " -f1) $PKGS`
	`$(md5sum Packages.gz \| cut -d" " -f1) $PKGS_GZ`
	`SHA1:`
	`$(sha1sum Packages \| cut -d" " -f1) $PKGS`
	`$(sha1sum Packages.gz \| cut -d" " -f1) $PKGS_GZ`
	`SHA256:`
	`$(sha256sum Packages \| cut -d" " -f1) $PKGS`
	`$(sha256sum Packages.gz \| cut -d" " -f1) $PKGS_GZ`
	`EOF`
	`test -e "${gpg_passfile}" && gpg --batch --yes --passphrase-file "${gpg_passfile}" --pinentry-mode loopback -abs -o Release.gpg Release`
	`test -e "${gpg_passfile}" && gpg --batch --yes --passphrase-file "${gpg_passfile}" --pinentry-mode loopback --clearsign -o InRelease Release`
	`# and because we are resigning it, replace Release.key with the one we used`
	`test -e "${gpg_keyfile}" && /bin/cp -pf "${gpg_keyfile}" Release.key`

To run this by itself, I recommend you run it in a sub shell, because you need to dot-source the conf file.

( . /etc/installed/obsmirror-cdemu.conf ; . /etc/installed/rebuild-apt-repo.sh ; )

And a reminder, the conf file looks like this:

# vim: syntax=sh
logfile="/var/server1/shares/public/Support/Systems/server1/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"
inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"
workdir=/var/server1/shares/public/www/mirror/obs
include_sources=
resign_repo=yes
gpg_passfile=/root/.gnupg/linuxadmin
gpg_keyfile=/var/server1/shares/public/www/internal/repo/deb/internaldeb.gpg
thisuser=obsmirror
VERBOSE=1

Trigger remote run on Open Build Service without changing _service file

bgstack15 — Wed, 16 Jun 2021 12:56:51 GMT

I like to use _service files on the OpenSUSE Open Build Service. I tend to have a very simple service file that pulls the orig.tar.gz (for dpkg) and .dsc files from git scm and those files then those files' presence indicate to the build server what to do next which is build the dpkg. Buried in the man page for osc is documentation for how to trigger a service run from the command line. When my service file points to a dev branch (see below example), the service file doesn't need to be updated at all. So I would normally have to go to the webpage and select the "trigger services" button. But with the following commands, I don't need to bother logging in to the webpage:

cd ~/dev/osc/home\:bgstack15/freeipa-helper
osc service remoterun

Why this is so obscure, I don't know.

Example _service file

<services>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://gitlab.com/bgstack15/freeipa-helper.git</param>
      <param name="subdir">debian</param>
      <param name="filename">debian</param>
      <param name="revision"> **dev** </param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">xz</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://gitlab.com/bgstack15/freeipa-helper.git</param>
      <param name="revision">dev</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">gz</param>
   </service>
   <service name="extract_file">
      <param name="archive">*.tar.xz</param>
      <param name="files">*/*.dsc</param>
   </service>
</services>

Download file from Open Build Service (OBS) from command line

bgstack15 — Sat, 19 Dec 2020 13:59:58 GMT

I really like the Open Build Service by the OpenSUSE team. I wrote a tool that will download an OBS service run asset on the cli. This tool interacts with the authentication mechanism so you can get the relevant cookies and accomplish your task. Check it out: https://gitlab.com/bgstack15/obs-dl Yes, it's insecure. Set environment variables USERNAME and PASSWORD and then run command:

./obs-dl https://build.opensuse.org/source/home:bgstack15/freefilesync/_service:recompress:tar_scm:FreeFileSync.tar.gz?rev=9410c094a8ce7fb9df4042c405ea0c7b

Update osc password for Open Build Service

bgstack15 — Sun, 17 May 2020 12:50:53 GMT

After the Open Build Service's effort to migrate to the new authentication system, I went ahead and just rotated my password.

adrianSuSE wrote 2 days ago: OBS has switched to the new authentification system: https://idp-portal-info.suse.com

Of course my osc commands started failing on the command line. The man page for osc is a little obtuse. You have to omit the word "section" when changing the password.

osc config https://api.opensuse.org --change-password

And then osc will be back on track!

Mirror an OBS repository locally -- update 1

bgstack15 — Wed, 11 Mar 2020 13:27:37 GMT

Previously, I wrote about how to mirror an Open Build Service apt repository. The original script used httrack and some crazy logic. Due to the unsatisfactory performance of the first version, I rewrote the entire script, twice! I am much more satisfied with my v3 of the script, which is hosting in the same place as before. So this rewrite is in some ways way simpler than it was before. Now, the script has a dedicated function for downloading a file, and it only does so if there is no checksum passed, or if the checksum of the local file is not the one passed to the function. How an apt repo works is it has checksums of all of its files in various metadata files, which are the only explicitly named files we download first. We loop through the filenames in the Packages file, and pass those filenames and checksums to the getter function. If the local file has the same checksum, we skip that file. A flag exists for choosing to download the dpkg sources as well. And then, for some reason I cannot quite explain, a few files of mine never downloaded and matched the checksums in the Package file. The deb files install, so they're valid, and I trust my own repo. So I decided to just re-sign all the apt repo files. That is, I rebuild the apt repo entirely and then sign it with my own gpg key for my internal network. And I finally split out the config into a config file so I can provide an example, and not store my actual data on the Internet!

#!/bin/sh
# File: /etc/installed/obsmirror.sh
# Location: https://gitlab.com/bgstack15/former-gists/tree/master/obsmirror.sh
# Author: bgstack15
# Startdate: 2020-03-03 08:43
# SPDX-License-Identifier: CC-BY-SA-4.0
# Title: Script that scrapes down OBS site to serve a copy to intranet
# Purpose: save down my OBS site so I can serve it locally
# History:
#    2020-01-05 v1: begin which used httrack
#    2020-02-28 v2: complete rewrite to exclude httrack
#    2020-03-03 v3: complete rewrite to get explicit files and loop through their contents, and rebuild apt repo
# Usage:
#    in a cron job: /etc/cron.d/mirror.cron
#       50  12  *   *   *   root    OBSMIRROR_CONF=/etc/installed/obsmirror.conf /etc/installed/obsmirror.sh 1>/dev/null 2>&1
# Reference:
#    https://software.opensuse.org//download.html?project=home%3Abgstack15&package=freefilesync
#    /mnt/public/www/repo/devuan-deb/update-devuan-deb.sh
#    https://medium.com/sqooba/create-your-own-custom-and-authenticated-apt-repository-1e4a4cf0b864
# Improve:
# Documentation:
#    Download the release key and trust it.
#       curl -s http://repo.example.com/mirror/obs/Release.key | apt-key add -
#    Use a sources.list.d/ file with contents:
#       deb https://repo.example.com/mirror/obs/ /
# Dependencies:
#    binaries: wget sed awk
#    user: obsmirror
umask 0002

test -n "${OBSMIRROR_CONF}" && . "${OBSMIRROR_CONF}"
test -z "${logfile}" && logfile="/tmp/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"
test -z "${inurl}" && inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"
test -z "${workdir}" && workdir=/tmp/obs
test -z "${thisuser}" && thisuser=obsmirror
# also use include_sources resign_repo gpg_passfile gpg_keyfile DEBUG

get_file() {
   # call: get_file "${tu}" "${md5sum}"
   ___tu="${1}"
   ___sum="${2}"
   tn="${___tu##${inurl}}"
   tf="${workdir}/${tn}" ; tf="$( readlink -m "${tf}" )"
   td="$( dirname "${tf}" )"
   test -d "${td}" || mkdir -p "${td}"
   gotten="skipped   "
   if test -z "${DRYRUN}" ;
   then
      if test -z "${___sum}" || test "$( md5sum "${tf}" 2>/dev/null | awk '{print $1}' )" != "${___sum}" ;
      then
         wget --content-disposition --no-verbose --quiet -O "${tf}" "${___tu}" && gotten=DOWNLOADED
      fi
   fi
   test -n "${VERBOSE}" && echo "${gotten} ${___tu} -> ${tf}"
}

wget_verbose=--quiet
test -n "${VERBOSE}" && unset wget_verbose
{
   test "${DEBUG:-NONE}" = "FULL" && set -x
   echo "logfile=${logfile}"

   # These files define an apt repo
   for word in InRelease Packages Packages.gz Release Release.gpg Release.key Sources Sources.gz ;
   do
      get_file "${inurl}/${word}"
   done

   # loop through named packages and download them
   #for word in $( awk '/Filename:/{print $2}' "${workdir}/Packages" ) ;
   awk '/Filename:|MD5/{print $2}' "${workdir}/Packages" | xargs -n2 | while read word sum
   do
      get_file "$( echo "${word}" | sed -r -e "s@^\.@${inurl}@;" )" "${sum}"
      #echo "a=${a}   b=${b}"
   done

   # loop through dsc, orig.tar.gz, and debian.tar.xz files
   test -n "${include_sources}" && {
      for word in $( sed -n -r -e '/Files:/,/^\s*$/{/^ /p;}' ${workdir}/Sources | awk '{print $NF}' ) ;
      do
         get_file "${inurl}/${word}"
      done
   }

   test -n "${resign_repo}" && {
      # rebuild release files
      repodir="${workdir}"
      cd "${repodir}"
      dpkg-scanpackages -m . > Packages
      gzip -9c < Packages > Packages.gz
      # create the Release file
      PKGS="$(wc -c Packages)"
      PKGS_GZ="$(wc -c Packages.gz)"
      old_headers1="$( grep -E '^(Archive|Codename|Origin|Label|Architectures):' Release )"
      old_headers2="$( grep -E '^(Description):' Release )"
      cat < Release
${old_headers1}
Date: $(date -u '+%a, %d %b %Y %T %Z')
${old_headers2}
MD5Sum:
 $(md5sum Packages  | cut -d" " -f1) $PKGS
 $(md5sum Packages.gz  | cut -d" " -f1) $PKGS_GZ
SHA1:
 $(sha1sum Packages  | cut -d" " -f1) $PKGS
 $(sha1sum Packages.gz  | cut -d" " -f1) $PKGS_GZ
SHA256:
 $(sha256sum Packages | cut -d" " -f1) $PKGS
 $(sha256sum Packages.gz | cut -d" " -f1) $PKGS_GZ
EOF
      test -e "${gpg_passfile}" && gpg --batch --yes --passphrase-file "${gpg_passfile}" --pinentry-mode loopback -abs -o Release.gpg Release
      test -e "${gpg_passfile}" && gpg --batch --yes --passphrase-file "${gpg_passfile}" --pinentry-mode loopback --clearsign -o InRelease Release
      # and because we are resigning it, replace Release.key with the one we used
      test -e "${gpg_keyfile}" && cp -p "${gpg_keyfile}" Release.key
   }

   chown -R "${thisuser}:$( id -G "${thisuser}" | awk '{print $1}' )" "${workdir}"
} 2>&1 | tee -a "${logfile}"

Package for devuan: keyboard-leds-trayicons

bgstack15 — Fri, 28 Feb 2020 13:33:56 GMT

I recently purchased a ThinkPad P50s and I love it! However, it doesn't have indicators for capslock and numlock. So I wrote a tool to simulate them! Keyboard-leds-trayicons is my solution. You can go install it in Devuan Ceres from my OBS. Reading the status of capslock and numlock is trivial on the command line:

xset q | awk 'function d2b(d,b) {while(d) {b=d%2b;d=int(d/2)}return(b)} /LED/{print d2b($NF)}'

The awk is there to convert the provided decimal value into a binary. But how do you poll this, and display it in your X11 session in an unobtrusive way? With a POSIX shell script, of course! But what about the actual icons? Use my fork of mktrayicon. My fork adds a few patches, and you can also get it at the OBS link above. Obviously most people have no need for this package. And by now anyone without the indicators probably doesn't need them. This project was 99% for myself, but I like to share. The default icons are a bit boring, which fits my style. But the tool uses the xdg icon spec, so you can add your own icons for capslock-on and similar. Patches are welcome to the project, so if you really think this package should include more icons, or use yours by default, please open a merge request!

Found: OBS service documentation

bgstack15 — Mon, 24 Feb 2020 14:20:04 GMT

For some unknown reason, it is really hard to find good documentation on how to use the _service file for the Open Build Service. The Open Build Service (OBS) is a project from the openSUSE team, and they run a public instance at https://build.opensuse.org/. This server application provides a build environment for many distributions' package methods. I use it for my public Devuan packages (see my "OBS" link in the links section of this site, usually a sidebar on each page). The _service file has some generic documentation in the main obs online man pages. I find this documentation extremely lacking in explaining how to use a service file fully. And I've learned by examples extant on the public OBS instance already. So, just to be clear, a _service file is a file in an obs project, that defines a number of steps for the server to take. By default, each service runs for each commit, or whenever you select the "trigger services" button on the web page for a project. I finally found a list of available service names to use! And the links take you to the projects on the Internet where they live and are fully documented. There are quite a few I plan on using in the future... like regex_replace.

Example _service file

With my service file set up to pull down the source debian/ directory as debian.tar.xz, and the entire source as a tar.gz, and extract out my .dsc file from that source, I can manage my OBS project way easier.

<services>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://gitlab.com/bgstack15/fluxbox-themes-stackrpms.git</param>
      <param name="subdir">debian</param>
      <param name="filename">debian</param>
      <param name="revision">local-dsc-file</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">xz</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://gitlab.com/bgstack15/fluxbox-themes-stackrpms.git</param>
      <param name="revision">local-dsc-file</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">gz</param>
   </service>
   <service name="extract_file">
      <param name="archive">*.tar.gz</param>
      <param name="files">*/*.dsc</param>
   </service>
</services>

The .dsc file is modified from the output from a proper debuild -us -uc locally. I learned the trick of zeroing out the checksums from another OBS project, waterfox-classic- kpe.

Format: 3.0 (quilt)
Source: fluxbox-themes-stackrpms
Binary: fluxbox-themes-stackrpms
Architecture: all
Version: 0.0.1-1+devuan
Maintainer: Ben Stack 
Homepage: /posts/
Standards-Version: 4.1.4
Build-Depends: debhelper (>= 12~)
Package-List:
 fluxbox-themes-stackrpms deb x11 optional arch=all
Files:
 00000000000000000000000000000000 1 fluxbox-themes-stackrpms_0.0.1.orig.tar.gz
 00000000000000000000000000000000 1 fluxbox-themes-stackrpms_0.0.1-1+devuan.debian.tar.xz

And since the filenames do not match, I think that section is not even necessary. Thankfully OBS figures out what tarballs to use where for the debuild. To quote my original distro (Korora): "Standing on the shoulders of giants."

Conclusion

I can control my osc package with only one file in the obs source control: the _service file! This reduces the need to pass multiple assets, which can be large.

References

Internet searches

How I actually found the main weblink open build service services recompress

Weblinks

https://en.opensuse.org/openSUSE:Build_Service_Concept_SourceService#All_OBS_services_available

Mirror an OBS deb repository locally

bgstack15 — Mon, 27 Jan 2020 13:56:50 GMT

Story

I run an OBS repository for all my packages, and it is available at the main site: https://build.opensuse.org/project/show/home:bgstack15. But I wanted to mirror this for myself, so I don't have to configure all my systems to point outward to get updates. I already host a Devuan ceres mirror for myself, and so mirroring this Open Build System repository is the last step to be self-hosting entirely for all systems except the mirror server. I first dabbled with debmirror, but it kept wanting to try rsync despite my best configuration, plus it really insists on using the dists/ directory which isn't used in the OBS deb repo design. So, I researched scraping down a whole site, and I found httrack which exists to serve a local copy of an Internet site. Bingo! After a few hours of work, here is my solution for mirroring an OBS deb repo locally.

Solution

Create a user who will own the files and execute the httrack command, because httrack didn't want to be run as root. Also, this new user can't munge other data.

useradd obsmirror

Configure a script (available at gitlab)

#!/bin/sh
# File: /etc/installed/obsmirror.sh
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2020-01-05 18:01
# Title: Script that scrapes down OBS site to serve a copy to intranet
# Purpose: save down my OBS site so I can serve it locally
# History:
# Usage:
#    in a cron job: /etc/cron.d/mirror.cron
#       50  12  *   *   *   root    /etc/installed/obsmirror.sh 1>/dev/null 2>&1
# Reference:
#    https://unix.stackexchange.com/questions/114044/how-to-make-wget-download-recursive-combining-accept-with-exclude-directorie?rq=1
#    man 1 httrack
#    https://software.opensuse.org//download.html?project=home%3Abgstack15&package=freefilesync
# Improve:
#    use some text file as a list of recently-synced URLs, and if today's URL matches a recent one, then run the httrack with the --update flag. Probably keep a running list forever.
# Documentation:
#    Download the release key and trust it.
#       curl -s http://repo.example.com/mirror/obs/Release.key | apt-key add -
#    Use a sources.list.d/ file with contents:
#       deb https://repo.example.com/mirror/obs/ /
# Dependencies:
#    binaries: curl httrack grep head tr sed awk chmod chown find rm ln
#    user: obsmirror

logfile="/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"
{
   test "${DEBUG:-NONE}" = "FULL" && set -x
   inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"
   workdir=/tmp/obs-stage
   outdir=/var/www/mirror/obs
   thisuser=obsmirror
   echo "logfile=${logfile}"

   mkdir -p "${workdir}" ; chmod "0711" "${workdir}" ; chown "${thisuser}:$( id -Gn obsmirror )" "${workdir}" 
   cd "${workdir}"
   # get page contents
   step1="$( curl -s -L "${inurl}/all" )"
   # get first listed package
   step2="$( echo "${step1}" | grep --color=always -oE 'href="[a-zA-Z0-9_.+\-]+\.deb"' | head -n1 | grep -oE '".*"' | tr -d '"' )"
   # get full url to a package
   step3="$( curl -s -I "${inurl}/all/${step2}" | awk '/Location:/ {print $2}' )"
   # get directory of the mirror to save down
   step4="$( echo "${step3}" | sed -r -e "s/all\/${step2}//;" -e 's/\s*$//;' )"
   # get domain of full url
   domainname="$( echo "${step3}" | grep -oE '(ht|f)tps?:\/\/[^\/]+\/' | cut -d'/' -f3 )"
   echo "TARGET URL: ${step4}"
   test -z "${DRYRUN}" && {
      # clean workdir of specific domain name in use right now.
      echo su "${thisuser}" -c "rm -rf \"${workdir:-SOMETHING}/${domainname:-SOMETHING}\""
      su "${thisuser}" -c "rm -rf \"${workdir:-SOMETHING}/${domainname:-SOMETHING}\"*"
      # have to skip the orig.tar.gz files because they are large and slow down the sync process significantly.
      echo su "${thisuser}" -c "httrack \"${step4}\" -*.orig.t* -v --mirror --update -s0 -r3 -%e0 \"${workdir}\""
      time su "${thisuser}" -c "httrack ${step4} -*.orig.t* -v --mirror --update -s0 -r3 -%e0 ${workdir}"
   }
   # -s0 ignore robots.txt
   # -r3 only go down 3 links
   # -%e0 follow 0 links to external sites

   # find most recent directory of that level
   levelcount="$(( $( printf "%s" "${inurl}" | tr -dc '/' | wc -c ) - 1 ))"
   subdir="$( find "${workdir}" -mindepth "${levelcount}" -maxdepth "${levelcount}" -type d -name 'Debian_Unstable' -printf '%T@ %p\n' | sort -n -k1 | head -n1 | awk '{print $2}' )"

   # if the work directory actually synced
   if test -n "${subdir}" ;
   then

      printf "%s " "DIRECTORY SIZE:"
      du -sxBM "${subdir:-.}"
      mkdir -p "$( dirname "${outdir}" )"
      # get current target of symlink
      current_target="$( find "${outdir}" -maxdepth 0 -type l -printf '%l\n' )"

      # if the current link is pointing to a different directory than this subdir
      if test "${current_target}" != "${subdir}" ;
      then
         # then replace it with a link to this one
         test -L "${outdir}" && unlink "${outdir}"
         echo ln -sf "${subdir}" "${outdir}"
         ln -sf "${subdir}" "${outdir}"
      fi

   else
      echo "ERROR: No subdir found, so cannot update the symlink."
   fi

   # disable the index.html with all the httrack comments and original site links
   find "${workdir}" -iname '*index.html' -exec rm {} +
} 2>&1 | tee -a "${logfile}"

And place this in cron!

#       50  12  *   *   *   root    /etc/installed/obsmirror.sh 1>/dev/null 2>&1

Explanation of script

So the logic is a little convoluted, because the OBS front page actually redirects downloads to various mirrors where the files are kept. So I needed to learn what the actual site is, and then pull down that whole site. I couldn't just use httrack --getfiles because it makes just a flat directory, which breaks the Packages contents' accuracy to the paths of the package files. But I didn't want the whole complex directory structure, just the repository structure. So I make a symlink to it in my actual web contents location.