Knowledge Base (Posts about yum)

Mirror Fedora cisco-openh264 repository

bgstack15 — Sun, 17 Dec 2023 14:13:38 GMT

So for some reason Fedora 39 VLC depends on something from the fedora-cisco-openh264 codecs repository for things, for no good reason. Considering the signing key is the same as the rest of Fedora, it sounds awfully official to me. Anyway, so I had to add this repo to my kickstart (coming up soon) for Fedora 39.

And since I like to use all-local resources for installing my systems, I had to mirror this repo. It took 7 seconds to run, and 30 minutes to research and learn the URL, find my utility for doing this, and preparing the config file.

If you wish to use yummirror.sh (also on gitlab) from my coprmirror project, you just need this config file.

files/2023/12/listings/yummirror-fedora-cisco-openh264-39-x86_64.conf (Source)

	`# vim: syntax=sh`
	`# startdate: 2023-12-13-4 22:28 for fc39x-ks.cfg kickstart to be able to access this locally`
	`# usage:`
	`# time sudo YUMMIRROR_CONF=/etc/installed/coprmirror/yummirror-fedora-cisco-openh264-39-x86_64.conf /etc/installed/coprmirror/yummirror.sh`
	`# improve:`
	`# this can probably be replaced with a shell script that loops through maybe a manual list of fedora release numbers, and then generates the inurl,workdir and then runs yummirror with env vars instead of various .conf files.`
	`logfile=~/log/coprmirror.$( date "+%FT%H%M%S" ).log`
	`# if we leave yum vars in inurl, it will be interpreted by whatever system is running the yummirror task, and not by a pretend version of the desired OS.`
	`#inurl='https://codecs.fedoraproject.org/openh264/$releasever/$basearch/os/'`
	`inurl='https://codecs.fedoraproject.org/openh264/39/x86_64/os/'`
	`workdir=/mnt/mirror/fedora-cisco-openh264/39/x86_64/os/`
	`thisuser=copruser`
	`VERBOSE=1`
	`DEBUG=1`
	`# skip .src.rpms`
	`include_sources=no`
	`# resign is not implemented anyways`
	`resign_repo=no`
	`# because gpgkey is just a file: for a basic fedora signing key that is already here, just leave it undefined so yummirror will skip it.`
	`#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch`

And in case gitlab/cgit goes away, here's the listing for yummirror script itself.

files/2023/12/listings/yummirror.sh (Source)

	`#!/bin/sh`
	`# File: yummirror.sh`
	`# Location: https://gitlab.com/bgstack15/coprmirror`
	`# Author: bgstack15`
	`# Startdate: 2021-08-16 09:57`
	`# SPDX-License-Identifier: GPL-3.0`
	`# Title: Script that copies a single copr yum repo`
	`# Project: coprmirror`
	`# Purpose: mirror copr locally, given baseurl; part of a larger set`
	`# History:`
	`# Usage:`
	`# Called by a larger script that catches the entire copr. This file only does a single yum repo for a single architecture and OS release. See coprmirror.sh`
	`# References:`
	`# https://unix.stackexchange.com/questions/19701/yum-how-can-i-view-variables-like-releasever-basearch-yum0`
	`# https://gitlab.com/bgstack15/former-gists/-/blob/master/obsmirror.sh/obsmirror.sh`
	`# Improve:`
	`# resign_repo is not implemented yet.`
	`# Dependencies:`
	`# jq, yum-utils, awk, sed, grep`
	`umask 0002`

	`test -n "${YUMMIRROR_CONF}" && . "${YUMMIRROR_CONF}"`
	`test -z "${logfile}" && logfile=./yummirror.$( date "+%FT%H%M%S" ).log`
	`test -z "${inurl}" && inurl='https://copr-be.cloud.fedoraproject.org/results/bgstack15/stackrpms/epel-7-$basearch/'`
	`test -z "${workdir}" && workdir=/tmp/copr`
	`test -z "${thisuser}" && thisuser=${USER}`
	`# also use include_sources resign_repo DEBUG VERBOSE`

	`exec 3>&1`
	`show() {`
	`printf "%s" "${*}" 1>&3`
	`}`

	`reset_show() {`
	`printf "\r%s" "${*}" 1>&3`
	`}`

	`## Functions`
	`get_file() {`
	`# call: get_file "${tu}" "${md5sum}" "absolute"`
	`___tu="${1}"`
	`___sum="${2}"`
	`___abs="${3}"`
	`if test "${___abs}" = "absolute" ; then`
	`tn="$( basename "${___tu}" )"`
	`else`
	`tn="${___tu##${inurl}}"`
	`fi`
	`tf="${workdir}/${tn}" ; tf="$( readlink -m "${tf}" )"`
	`td="$( dirname "${tf}" )"`
	`test -d "${td}" \|\| mkdir -p "${td}"`
	`gotten="skipped "`
	`#printf '\n%s\n' "inside get_file ${@}, DRYRUN=${DRYRUN}"`
	`if test -z "${DRYRUN}" \|\| test "${DRYRUN}" = "metadata";`
	`then`
	`if test -z "${___sum}" \|\| test "$( sha256sum "${tf}" 2>/dev/null \| awk '{print $1}' )" != "${___sum}" ;`
	`then`
	`test -n "${VERBOSE}" && show "retrieving ${___tu}" 2>/dev/null \|\| :`
	`wget --content-disposition --no-verbose --quiet -O "${tf}" "${___tu}" && gotten=DOWNLOADED`
	`fi`
	`fi`
	`test -n "${VERBOSE}" && reset_show 2>/dev/null \|\| :`
	`echo "${gotten} ${___tu} -> ${tf}"`
	`}`

	`## MAIN`

	`# Interpret any yum vars in the inurl`
	`if echo "${inurl}" \| grep -qE '\$' ;`
	`then`
	`echo "" \| jq 1>/dev/null 2>&1 \|\| { echo "Need jq to interpret yum vars in baseurl ${inurl}. Aborted." ; exit 1 ; }`
	`raw="$( python -c 'import yum, json; yb = yum.YumBase(); print json.dumps(yb.conf.yumvar, indent=2)' )"`
	`# validated on centos 7`
	`basearch="$( echo "${raw}" \| sed -n '1!p' \| jq '.basearch' \| tr -d '"' )"`
	`releasever="$( echo "${raw}" \| sed -n '1!p' \| jq '.releasever' \| tr -d '"' )"`
	`#echo "${inurl}" \| awk -v "basearch=${basearch}" -v "releasever=${releasever}" '{gsub("\$basearch",basearch,$0);gsub("\$releasever",releasever,$0);print}'`
	`newurl="$( echo "${inurl}" \| sed -r -e "s/\\\$basearch/${basearch}/g;" -e "s/\\\$releasever/${releasever}/g;" )"`
	`test -n "${DEBUG}" && {`
	`echo "Interpreting ${inurl} -> ${newurl}"`
	`} 1>&2`
	`inurl="${newurl}"`
	`fi`

	`# clean up trailing slashes`
	`inurl="${inurl%%/}"`

	`# sync to workdir`
	`wget_verbose=--quiet`
	`test -n "${VERBOSE}" && unset wget_verbose`
	`{`
	`test "${DEBUG:-NONE}" = "FULL" && set -x`
	`echo "logfile=${logfile}"`
	`mkdir -p "${workdir}" ; cd "${workdir}"`

	`# This file lists the files that define a yum repo`
	`for word in repodata/repomd.xml ;`
	`do`
	`# coprmirror.sh will pass DRYRUN=metadata if it has anything set in dryrun, and we interpret "metadata" here, because we absolutely need the repomd.xml and associated files no matter what, even for a dry run.`
	`DRYRUN="${DRYRUN:+metadata}" get_file "${inurl%%/}/${word}"`
	`done`

	`# loop through the important files listed in that file.`
	`# read sha256sum, so we don't have to download these if we already have them.`
	`metadata_list="$( grep -iE '<checksum \|href' repodata/repomd.xml \| awk -F'"' '/checksum/{print $3} /href/{print $2}' \| awk "/^>/{gsub(\"^>\",\"\",\$0);gsub(\"<.*$\",\"\",\$0);a=\$0;} /repodata/{print a,\$0}" )"`
	`echo "${metadata_list}" \| while read sum word ;`
	`do`
	`DRYRUN="${DRYRUN:+metadata}" get_file "${inurl%%/}/${word}" "${sum}" 3>/dev/null`
	`done`

	`# go ahead and fetch the gpgkey if resign_repo=no`
	`if echo "${resign_repo}" \| grep -qiE 'yes\|\<y\>\|1\|true' ;`
	`then`
	`# do something`
	`echo "Resigning functionality not built yet. Skipping..."`
	`else`
	`if test -n "${gpgkey}" ;`
	`then`
	`# fetch the gpgkey`
	`# note: yum variable parsing not yet implemented here. I would need to turn the variable parsing into a function and call it here.`
	`# cannot use get_file because it uses a relative path evaluation`
	`#wget --output-file="${workdir}/pubkey.gpg" "${gpgkey}"`
	`get_file "${gpgkey}" "" "absolute"`
	`fi`
	`fi`

	`# COPR always provides a primary.xml.gz file, which lists the assets to download`
	`full_list="$( zgrep -iE 'checksum\|href' $( grep -oiE 'href=.primary.xml.gz.$' repodata/repomd.xml 2>/dev/null \| awk -F'"' '{print $2}' ; echo "none.XXXXXXX" ) 2>/dev/null \| awk "/checksum/{gsub(\".\\\">\",\"\",\$0);gsub(\"</.\",\"\",\$0);a=\$0;} /href/{gsub(\".=\\\"\",\"\",\$0);gsub(\"\\\"/.\",\"\",\$0);print a,\$0;}" )"`
	`#echo "FULL_LIST=${full_list}"`

	`# protect against the "metadata only" run`
	`test "${DRYRUN}" = "metadata" && DRYRUN=1`
	`echo "${full_list}" \| while read sum word ;`
	`do`
	`if echo "${word}" \| grep -qiE "\.src\.rpm" ;`
	`then`
	`# if a srpm, only get it if user has set include_sources.`
	`if echo "${include_sources}" \| grep -qiE 'yes\|\<y\>\|1\|true' ;`
	`then`
	`get_file "${inurl}/${word}" "${sum}"`
	`fi`
	`else`
	`# always get all regular rpms and other non-.src.rpm files if any`
	`get_file "${inurl}/${word}" "${sum}"`
	`fi`
	`done`

	`chown -R "${thisuser}:$( id -G "${thisuser}" \| awk '{print $1}' )" "${workdir}"`
	`} 2>&1 \| tee -a "${logfile}"`

And now you have the entire (3-package) yum repository for local use.

dnf summarize

bgstack15 — Tue, 18 Jul 2023 13:02:31 GMT

I've been working on my Rocky Linux 9 vm kickstart installation process, and due to what is already available in my local Linux mirror server, I used some AlmaLinux sources. So not only did I have to make the symlinks for ln -s epel-9-x86_64 epel-9.2-x86_64 and ln -s 9 9.2 for my copr and epel mirrors, but I also see how many packages come from each source repo.

yum list installed | awk '{a[$NF]++} END{for(i in a){j=i;gsub(/@/,"",j);if(j!="Packages"){print j": "a[i]}}}' | sort

I don't have a cutesy shell function in my profile yet. It took me a while to remember that gsub outputs the number of substitutions. It modifies the third input value directly instead of outputting the changed string.

anaconda: 398
appstream: 297
AppStream: 6
base: 13
copr-bgstack15-stackrpms: 5
epel: 5
examplerpm: 1

Mirror a copr, all architectures and releases and versions

bgstack15 — Fri, 20 Aug 2021 12:44:20 GMT

coprmirror is my yum mirror solution, with a wrapper to mirror specifically a named COPR repository.

Overview

COPR is a distro-run community offering that lets users build yum/dnf repositories with their own software (primarily if not exclusively in rpm format). COPR performs the builds, and then hosts the binary and source rpms for client machines. My project today downloads using native GNU/Linux tools the yum repositories that collectively make up a COPR, so each release-releasever-basearch triplet. The end goal is to have a local copy of the entire current yum repos. This does not copy the build assets or log files; just what a yum repository defines and also the gpg public key. Notably this utility needs the yum python packages present only for evaluating yum variables in the inurl (baseurl from a .repo file), so if you have a string literal as the inurl value, you can run this on a system that does not have yum installed.

Using

Configure coprmirror.conf from the provided .example file, and then run:

COPRMIRROR_CONF=coprmirror.conf VERBOSE=1 DEBUG=1 ./coprmirror.sh

Upstream

Original content The get_file function was improved after being imported from my Reference 1 script.

Alternatives

I felt like ansible and system are overkill, but if you like those, this is perfect for you: https://github.com/ganto/ansible-copr_reposync

Dependencies

wget, grep, awk, sed, jq

References

obsmirror Mirror an OBS repository locally — update 1 [this blog]

Thoughts about yum repo server and defined repositories

bgstack15 — Wed, 22 May 2019 13:24:11 GMT

The internal server used as a yum repository runs EL6, and it servers EL6 and EL7 (read: CentOS) yum repos. An admin tried installing mkisofs, which wanted to come by default from the c7-base repository. It also wanted to upgrade bash and glibc. Well, installing a post-usrmerge bash (CentOS 7+) on CentOS6 caused all sorts of havoc. I had to load a rescue iso, boot, and copy /usr/bin/bash and /usr/bin/sh to the correct locations. Then my system would actually boot again. I was getting an interesting error:

init: Failed to spawn rcS pre-start process: unable to execute: No such file or directory
init: Failed to spawn rcS post-stop process: unable to execute: No such file or directory

Also, kernel options rghb and quiet are really annoying and I always disable them. So, the moral of the story is: always be very careful running yum on a yum repo server. Double-check what repos your package will pull in.

List yum repos that have packages installed and are still defined

bgstack15 — Sat, 18 May 2019 13:12:08 GMT

yum repolist all $( yum list installed | awk '$NF ~ /@/ && $NF !~ /anaconda/ {split($NF,a,"@");print a[2];}' | sort | uniq ) | awk 'NR > 2 && $1 !~ /repolist/ {split($1,a,"/");print a[1]}'

The idea is to list each installed package which also lists the repo it was installed from, and then organize those repos. Then list them and show the ones that are still defined in yum's repository files.

reposync fails: "No more mirrors to try."

bgstack15 — Sun, 05 May 2019 13:00:28 GMT

Symptom

The following message occurs, when running a yum reposync.

# time /usr/bin/reposync --source --repoid=EL7 -m --download-metadata --download_path="/var/www/html/yum/EL/EL7_Mirror"
Repository hosting is listed more than once in the configuration
firefox-60.6.0-3.0.1.el7_6.src FAILED
firefox-60.6.0-3.0.1.el7_6.src: [Errno 256] No more mirrors to try.                            ]  0.0 B/s |    0 B  --:--:-- ETA<[pre>

This could occur during an ansible play.

TASK [sync preprod EL7 to prod] ******************************************************************************************************************
fatal: [yumserver01]: FAILED! => {"changed": true, "cmd": "/usr/bin/reposync --source --repoid=EL7 -m --download-metadata --download_path=\"/var/www/html/yum/EL/EL7_Mirror\"", "delta": "0:04:35.805969", "end": "2019-04-17 08:27:22.119744", "msg": "non-zero return code", "rc": 1, "start": "2019-04-17 08:22:46.313775", "stderr": "", "stderr_lines": [], "stdout": "Repository hosting is listed more than once in the configuration\n\rfirefox-60.6.1-1.0.1.el7_6.src FAILED                                          \n\r(1/34): firefox-60.6.0-3.0 0% [                 ]  0.0 B/s |    0 B   --:-- ETA \r\rfirefox-60.6.0-3.0.1.el7_6.src FAILED                                          \n\r(1/34): firefox-60.6.0-3.0 0% [                 ]  0.0 B/s |    0 B   --:-- ETA \r\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1.0.2-15.el7_6.1.src.r FAILED                                          \n\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1.0.2-15.el7_6.1.src.r FAILED                                          \n\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1.0.2-15.el7_6.1.src.r FAILED                                          \n\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1
.............................. TRUNCATED ..............................
 0.0 B/s | 2.5 MB   --:-- ETA ", "", "tzdata-java-2019a-1.el7.noarch FAILED                                          ", "", "(1/34): tzdata-java-2019a- 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA ", "libwsman1-2.6.3-6.git4391e5c.el7_6.i686: [Errno 256] No more mirrors to try.", "openwsman-client-2.6.3-6.git4391e5c.el7_6.i686: [Errno 256] No more mirrors to try.", "tuned-utils-2.10.0-6.0.1.el7_6.3.noarch: [Errno 256] No more mirrors to try.", "openwsman-server-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "sbd-1.3.1-8.2.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "libwsman1-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "sbd-1.3.1-8.2.el7_6.1.src: [Errno 256] No more mirrors to try.", "freerdp-1.0.2-15.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "freerdp-plugins-1.0.2-15.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "openwsman-client-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "ocfs2-tools-1.8.6-11.el7.x86_64: [Errno 256] No more mirrors to try.", "python-2.7.5-77.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "tuned-2.10.0-6.0.1.el7_6.3.noarch: [Errno 256] No more mirrors to try.", "firefox-60.6.0-3.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "freerdp-libs-1.0.2-15.el7_6.1.i686: [Errno 256] No more mirrors to try.", "freerdp-1.0.2-15.el7_6.1.src: [Errno 256] No more mirrors to try.", "freerdp-libs-1.0.2-15.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "ghostscript-9.07-31.el7_6.10.i686: [Errno 256] No more mirrors to try.", "ocfs2-tools-1.8.6-11.el7.src: [Errno 256] No more mirrors to try.", "tzdata-java-2019a-1.el7.noarch: [Errno 256] No more mirrors to try.", "ghostscript-9.07-31.el7_6.10.x86_64: [Errno 256] No more mirrors to try.", "openwsman-2.6.3-6.git4391e5c.el7_6.src: [Errno 256] No more mirrors to try.", "tuned-profiles-cpu-partitioning-2.10.0-6.0.1.el7_6.3.noarch: [Errno 256] No more mirrors to try.", "firefox-60.6.1-1.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "thunderbird-60.6.1-1.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "openwsman-server-2.6.3-6.git4391e5c.el7_6.i686: [Errno 256] No more mirrors to try.", "libssh2-1.4.3-12.el7_6.2.src: [Errno 256] No more mirrors to try.", "pcs-0.9.165-6.0.3.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "openwsman-python-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "pcs-0.9.165-6.0.3.el7_6.1.src: [Errno 256] No more mirrors to try.", "tuned-2.10.0-6.0.1.el7_6.3.src: [Errno 256] No more mirrors to try.", "tzdata-2019a-1.el7.src: [Errno 256] No more mirrors to try.", "ghostscript-9.07-31.el7_6.10.src: [Errno 256] No more mirrors to try."]}
        to retry, use: --limit @/etc/ansible/retries/reposync-prod.retry
PLAY RECAP **********************************************************************************************************************************************
yumserver01               : ok=1    changed=0    unreachable=0    failed=1

Possible causes

The original rpm files could be malformed, so even though the reposync client is successfully downloading whatever the server provided, the rpm does not match the checksum in the yum metadata files.

Fix #1

Run the command manually to learn the whole list. Save it down to a file.
```
# time /usr/bin/reposync --source --repoid=EL7 -m --download-metadata --download_path="/var/www/html/yum/EL/EL7_Mirror"
```
Repository hosting is listed more than once in the configuration firefox-60.6.0-3.0.1.el7_6.src FAILED firefox-60.6.0-3.0.1.el7_6.src: [Errno 256] No more mirrors to try. ] 0.0 B/s | 0 B --:--:-- ETA

The file used in the documented example consisted of about 30 packages. file: bad.in

    firefox-60.6.0-3.0.1.el7_6.src
firefox-60.6.1-1.0.1.el7_6.src
freerdp-1.0.2-15.el7_6.1.src
freerdp-1.0.2-15.el7_6.1.x86_64
freerdp-libs-1.0.2-15.el7_6.1.i686
freerdp-libs-1.0.2-15.el7_6.1.x86_64
freerdp-plugins-1.0.2-15.el7_6.1.x86_64
ghostscript-9.07-31.el7_6.10.i686
ghostscript-9.07-31.el7_6.10.src

On the reposync client delete or move to a temp directory the failed files. You could use a shell script similar to the following.
1
#!/bin/sh
INFILE=/root/bad.in INDIR=/var/www/html/yum/EL/EL7_Mirror/EL7 OUTDIR=/root/orig/ mkdir -p "${OUTDIR}" find $( for word in $( cat "${INFILE}" ) ; do find "${INDIR}" -name "${word}*rpm" done ) -print -exec mv {} "${OUTDIR}" \;
On the repo server, manually download the files. You could modify the input file to have the upstream URLs. bad.in.url
```
http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackage/tuned-utils-2.10.0-6.0.1.el7_6.3.noarch.rpm
```
http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackage/tzdata-java-2019a-1.el7.noarch.rpm http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackageSource/firefox-60.6.0-3.0.1.el7_6.src.rpm http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackageSource/firefox-60.6.1-1.0.1.el7_6.src.rpm http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackageSource/freerdp-1.0.2-15.el7_6.1.src.rpm .... TRUNCATED .....

You could use a shell script similar to the following.

    #!/bin/sh
INFILE=/root/bad.in.url
INDIR=/var/www/html/yum/EL/EL7_Mirror/EL7/getPackage/
INDIRSRC=/var/www/html/yum/EL/EL7_Mirror/EL7/getPackageSource/

for word in $( cat "${INFILE}" ) ;
do
   if echo "${word}" | grep -q getPackageSource ;
   then
      cd "${INDIRSRC}"
   else
      cd "${INDIR}"
   fi
   #echo "$( pwd ) --> ${word}"
   wget "${word}"
done

Rerun the reposync manually and make sure it works.

yum: curl error #63: callback aborted

bgstack15 — Wed, 01 May 2019 12:37:31 GMT

Symptom

The following message occurs.

http://yum01.ad.example.com/yum/mirror/kernel/getPackage/kernel-doc-4.0.5-124.16.4.el7.noarch.rpm: [Errno 14] curl#63 - "Callback aborted"
Trying other mirror.


Error downloading packages:
  kernel-doc-4.1.12-124.16.4.el7.noarch: [Errno 256] No more mirrors to try.

Possible causes

When yum fails to download a file due to "curl error #63: Callback aborted," that means that reposync downloaded the rpm file incorrectly. It doesn't match the expected checksum in the metadata file.

Fix #1

On the mirror server (probably yum01), delete the offending files and download them manually from upstream.
1
#!/bin/sh
urldir=http://yum.upstream.example.com/repo/EL7/Mirror/x86_64/getPackage/ outdir=/var/www/html/yum/repo/EL7_Mirror/Mirror/getPackage cd "${outdir}"

for word in kernel-4.0.5-124.16.4.el7.x86_64.rpm kernel-debug-4.0.5-124.16.4.el7.x86_64.rpm kernel-debug-devel-4.0.5-124.16.4.el7.x86_64.rpm kernel-devel-4.0.5-124.16.4.el7.x86_64.rpm kernel-doc-4.0.5-124.16.4.el7.noarch.rpm kernel-firmware-4.0.5-124.16.4.el7.noarch.rpm; do /bin/rm -f "${word}" wget "${urldir}${word}" done
On the squid proxy server (probably proxy01), purge the cached files.
1
#!/bin/sh
reference:

time squidclient -h localhost -r -p 3128 -m PURGE http://yum01.ad.example.com/yum/Mirror/EL7/getPackage/kernel-4.0.5-124.16.4.el7.x86_64.rpm

urldir=http://yum.upstream.example.com/repo/EL7/Mirror/x86_64/getPackage/ outdir=/var/www/html/yum/repo/EL7_Mirror/Mirror/getPackage

for word in kernel-4.0.5-124.16.4.el7.x86_64.rpm kernel-debug-4.0.5-124.16.4.el7.x86_64.rpm kernel-debug-devel-4.0.5-124.16.4.el7.x86_64.rpm kernel-devel-4.0.5-124.16.4.el7.x86_64.rpm kernel-doc-4.0.5-124.16.4.el7.noarch.rpm kernel-firmware-4.0.5-124.16.4.el7.noarch.rpm; do squidclient -h localhost -r -p 3128 -m PURGE "${urldir}${word}" done

Setup Yum Repository with Security Metadata

bgstack15 — Mon, 16 Jul 2018 13:11:46 GMT

Define repository

Prepare the repo file on the server, so clients can download it.

cd /var/www/html/yum
cat <<'EOF' > hosting.repo
[hosting]
name=Hosting Delivery
baseurl=http://yum5.ipa.example.com/yum/hosting/
enabled=0
gpgcheck=0
EOF

Make or update repository

Use createrepo tool to make the repository. A wrapper script for creating or updating the existing repository is shown here.

tf=/usr/local/bin/updaterepo.sh
cat <<'EOF' > "${tf}"
#!/bin/sh
# reference:
#    https://gitlab.com/bgstack15/mirror/blob/master/usr/share/mirror/examples/rpm/update-examplerpm.sh

# Prepare directory and files
test -z "${UR_REPODIR}" && UR_REPODIR=/var/www/html/yum/hosting
test -z "${UR_BASEURL}" && UR_BASEURL=http://yum5.ipa.example.com/yum/hosting
test -z "${UR_OWNERSHIP}" && UR_OWNERSHIP="root.root"
test -z "${UR_FILETYPES}" && UR_FILETYPES="rpm"

find "${UR_REPODIR}" -exec chown "${UR_OWNERSHIP}" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type f -exec chmod "0664" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type d -exec chmod "0775" {} + 1>/dev/null 2>&1
chmod 0754 "$0"
restorecon -RF "${UR_REPODIR}"

# Prepare repo for rpm
cd "${UR_REPODIR}"
createrepo -v -u "${UR_BASEURL}" --basedir "${UR_REPODIR}" --simple-md-filenames --no-database --update --pretty .
EOF

Run this script.

/usr/local/bin/updaterepo.sh

Manually make the security metadata

The security metadata that yum interprets is stored in updateinfo.xml.gz. To make this file and include it in repomd.xml, you need to prepare it and learn some information about it. This is a trim example of updateinfo.xml. Please see the epel metadata for a full example. I do not have an automatic process for generating this file yet.

tf=updateinfo.xml
cat <<'EOF' > "${tf}"
<?xml version="1.0" encoding="UTF-8"?>
<updates>
  <update status="final" type="security" version="1" from="bgstack15@gmail.com">
    <id>HELP-210217</id>
    <title>bgscripts-core update</title>
    <release>Enterprise Linux 7</release>
    <issued date="2018-04-02"/>
    <rights>CC-BY-SA 4.0</rights>
    <description>bgscripts-core
[1.3-8]
- latest version from upstream
</description>
    <solution>This update is internal to the company.</solution>
    <references>
      <reference href="https://gitlab.com/bgstack15/bgscripts" type="self" title="bgscripts-core" />
    </references>
    <pkglist>
      <collection short="bgscripts">
        <name>bgscripts suite</name>
        <package name="bgscripts-core" version="1.3-8" release="" epoch="0" arch="noarch">
          <filename>bgscripts-core-1.3-8.noarch.rpm</filename>
          <sum type="md5">eaa20075720bf12d6e837a4f546241ab</sum>
        </package>
     </collection>
    </pkglist>
  </update>
</updates>
EOF

Update the repo metadata to include updateinfo.xml

A yum repository includes metadata of the package metadata, and stores this meta-metadata in repomd.xml. Insert the metadata for this new file, updateinfo.xml in the repomd file. This script is an update version of updaterepo.sh, which was listed earlier in this document.

tf=/usr/local/bin/updaterepo.sh
        cat <<'EOF' > "${tf}"
#!/bin/sh
# reference:
#    https://gitlab.com/bgstack15/mirror/blob/master/usr/share/mirror/examples/rpm/update-examplerpm.sh

# Prepare directory and files
test -z "${UR_REPODIR}" && UR_REPODIR=/var/www/html/yum/hosting
test -z "${UR_BASEURL}" && UR_BASEURL=http://yum5.ipa.example.com/yum/hosting
test -z "${UR_OWNERSHIP}" && UR_OWNERSHIP="root.root"
test -z "${UR_FILETYPES}" && UR_FILETYPES="rpm"
test -z "${UR_UPDATEINFO_INPUT}" && UR_UPDATEINFO_INPUT=/var/www/html/yum/build-hosting-repo/updateinfo.xml

find "${UR_REPODIR}" -exec chown "${UR_OWNERSHIP}" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type f -exec chmod "0664" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type d -exec chmod "0775" {} + 1>/dev/null 2>&1
chmod 0754 "$0"
restorecon -RF "${UR_REPODIR}"

# Prepare basic repo
cd "${UR_REPODIR}"
createrepo -v -u "${UR_BASEURL}" --basedir "${UR_REPODIR}" --simple-md-filenames --no-database --update --pretty .

# Inject custom updateinfo
# this task assumes the repomd file does not include node <data type="updateinfo"> yet.
UR_repomd="${UR_REPODIR}/repodata/repomd.xml"
UR_updateinfo_gz_short="repodata/updateinfo.xml.gz"
UR_updateinfo_gz="${UR_REPODIR}/${UR_updateinfo_gz_short}"

if ! test -e "${UR_UPDATEINFO_INPUT}" ;
then
   # file is absent, so decide how to fail.
   :
else
   # file exists, so continue with custom injection

   # learn open-size and open-checksum
   UR_updateinfo_opensize="$( /usr/bin/stat -c "%s" "${UR_UPDATEINFO_INPUT}" )"
   UR_updateinfo_openchecksum="$( /usr/bin/sha256sum "${UR_UPDATEINFO_INPUT}" | awk '{print $1}' )"

   # compress file and learn size and checksum
   /usr/bin/gzip < "${UR_UPDATEINFO_INPUT}" > "${UR_updateinfo_gz}"
   UR_updateinfo_size="$( /usr/bin/stat -c "%s" "${UR_updateinfo_gz}" )"
   UR_updateinfo_checksum="$( /usr/bin/sha256sum "${UR_updateinfo_gz}" | awk '{print $1}' )"
   UR_updateinfo_timestamp="$( /usr/bin/stat -c "%Y" "${UR_updateinfo_gz}" )"

   # insert information into repomd
   this_string="<data type=\"updateinfo\">
  <checksum type=\"sha256\">${UR_updateinfo_checksum}</checksum>
  <open-checksum type=\"sha256\">${UR_updateinfo_openchecksum}</open-checksum>
  <location xml:base=\"${UR_BASEURL}\" href=\"${UR_updateinfo_gz_short}\"/>
  <timestamp>${UR_updateinfo_timestamp}</timestamp>
  <size>${UR_updateinfo_size}</size>
  <open-size>${UR_updateinfo_opensize}</open-size>
</data>"

   {
      sed -r -e '/<\/repomd>/d' "${UR_repomd}"
      printf "%s\n%s\n" "${this_string}" "</repomd>"
   } > "${UR_repomd}.$$"
   /bin/touch --reference "${UR_repomd}" "${UR_repomd}.$$"
   /bin/mv -f "${UR_repomd}.$$" "${UR_repomd}"
fi
EOF

Summary

Using bash to modify xml files is obviously not ideal. However, this xml file is simple enough so this ugly mechanism suffices. For teams that know how to manage custom yum repositories and also want to just use yum update --security, this process should be a good basis or even complete solution!

Appendices

Appendix A: http proxy

If you use an http proxy for your yum traffic, the proxy might cache old versions of the metadata or package files. A quick and dirty way to clean up a squid proxy of the metadata file follows.

time squidclient -h localhost -r -p 3128 -m PURGE http://yum5.ipa.example.com/yum/hosting/repodata/updateinfo.xml.gz

Squid unfortunately does not allow recursive purging, so you will have to loop over all the metadata files and any package files you want to ensure get cleared.

References

Local file /var/cache/yum/x86_64/7Server/epel/69b82df00108c0ac8ac82fafbd0f3b89cc98d8dfe4fa350af7a23331a878fea2-updateinfo.xml.bz2

Find which package provides a file - Yum and Apt

bgstack15 — Tue, 17 Jan 2017 13:38:45 GMT

Last updated 2019-06-26

Yum-based distros

On Fedora-like systems, the default package manager is yum (or dnf for Fedora 22+). The main way you use it is to install packages and auto-resolve the dependencies.

yum install irfan

Similarly, apt-get on the Debian family of GNU/Linux installs packages with this command.

apt-get install irfan

Now, let's say you are looking for which package installs a particular file. The command for yum is:

yum provides */filename

If you want to use rpm, that is available too.

rpm -qf /etc/sudoers

Apt-based distros

On apt-based distros (Debian, Devuan, Ubuntu, etc.), you need to install the package apt-file before you can do similar lookup commands.

apt-get install apt-file

Now you can search with:

apt-file search */filename

You can use the lower-level package manager:

dpkg-query -S /usr/share/filename

Update just one repository on Linux

bgstack15 — Mon, 12 Dec 2016 15:25:07 GMT

So if you build your own packages (rpm/dpkg/other), you might have your own repository of packages for your systems. And if you want to tell your hosts to update just the one repository, you might be looking for a solution. A user on Ask Ubuntu had a fantastic answer for Ubuntu/debian flavor: http://askubuntu.com/questions/65245/apt-get-update-only-for-a-specific- repository/197532#197532. For those of you who don't want to click through, here is a summary of the entries, collapsed down to just one file (which might not be the best solution, but it works). Add these lines to ~/.bashrc: update-repo() { for source in "$@"; do sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/${source}" \ -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0" done } _ppa_lists(){ local cur _init_completion || return COMPREPLY=( $( find /etc/apt/sources.list.d/ -name "*$cur*.list" \ -exec basename {} \; 2> /dev/null ) ) return 0 } && complete -F _ppa_lists update-repo With rpm-based systems like RHEL/CentOS/Fedora, you are using yum or dnf. Here's my dnf implementation: _command_dnf=yum which dnf 1>/dev/null 2>&1 && _command_dnf="$( which dnf 2>/dev/null )" # update-repo command for dnf update just one repository update-repo() { case "${_command_dnf}" in *dnf) for source in "$@"; do sudo "${_command_dnf}" check-update -q --refresh --disablerepo=* --enablerepo="${source}" done ;; *yum) for source in "$@"; do sudo "${_command_dnf}" clean metadata -q --disablerepo=* --enablerepo="${source}" -q; yum check-update -q --disablerepo=* --enablerepo="${source}" done ;; esac } # autocomplete for update-repo _repo_lists() { local cur _init_completion || return COMPREPLY=( $( grep -hoiE -- "^\[.*\]" /etc/yum.repos.d/* | tr -d '[]' | grep -E "^${2:-.*}" ) ) return 0 } && complete -F _repo_lists -o filenames update-repo The functions whose names start with an underscore are the auto-complete commands. Who doesn't like a good tab auto-completion? Now that I learned how to do my own bash- completion, I'll be doing it a lot more!

Knowledge Base (Posts about yum)

Mirror Fedora cisco-openh264 repository

dnf summarize

Mirror a copr, all architectures and releases and versions

Overview

Using

Upstream

Alternatives

Dependencies

References

Thoughts about yum repo server and defined repositories

List yum repos that have packages installed and are still defined

reposync fails: "No more mirrors to try."

Symptom

Possible causes

Fix #1

yum: curl error #63: callback aborted

Symptom

Possible causes

Fix #1

reference:

time squidclient -h localhost -r -p 3128 -m PURGE http://yum01.ad.example.com/yum/Mirror/EL7/getPackage/kernel-4.0.5-124.16.4.el7.x86_64.rpm

Setup Yum Repository with Security Metadata

Define repository

Make or update repository

Manually make the security metadata

Update the repo metadata to include updateinfo.xml

Summary

Appendices

Appendix A: http proxy

References

Find which package provides a file - Yum and Apt

Last updated 2019-06-26

Yum-based distros

Apt-based distros

Update just one repository on Linux