Knowledge Base (Posts about cron)

Update ipasam rpm

bgstack15 — Fri, 13 May 2022 13:02:37 GMT

Overview

Update-ipasam-rpm is a project that facilitates building a custom rpm with just the one file that samba needs to authenticate users to ipa.

Update-ipasam-rpm upstream

Gitlab is the upstream. This is original work.

Reason for existing

The proper ipa-server-trust-ad rpm has a large number of dependencies which are not necessary merely when using samba with ipa user authentication, so this project copies that file out and builds a small rpm just for the one file. This makes my samba file server need fewer packages installed.

Alternatives

Install ipa-server-trust-ad with all of its dependencies, which is way more than I want to use when I just need ipasam.so.

Dependencies

For any rpm-based system, but primarily AlmaLinux 8, the server that runs the cron job needs a few things:

Http/s access to a package mirror
Https access to copr to initiate builds
Https access to gitlab to pull ipasam.spec template
Packages: copr-cli, rpmbuild
Unprivileged user, shown in this documentation as username copruser

The copr api can be used with a copr user with a generated api key (Reference 1).

Files in the project

~copruser/.config/copr
~copruser/.config/ipasam
~copruser/bin/update-ipasam-rpm.sh
/etc/cron/70_ipasam_cron
~copruser/.cache/ipasam.spec (generated by modifying spec from this repo)
~copruser/.cache/ipa-server-trust-ad.ver (generated)
~copruser/rpmbuild/SRPMS/iapsam*.srpm (generated)

Usage

Create user copruser or other unprivileged user as desired.
Deploy the non-generated files from this repository to the above locations and inspect the config files.
Prepare a user on the copr and generate an api key and place in file ~copruser/.config/copr.
Run the command manually to see it operate.

~/bin/update-ipasam-rpm.sh
Visit your copr project to see the ipasam rpm that you built!

Differences from upstream

N/A

References

COPR API introduction

Scheduling my blog updates

bgstack15 — Sun, 19 Sep 2021 13:08:14 GMT

So because I don't actually write these posts at the date they appear on the site (or are listed as when they appeared on the site), I need to schedule running my build and deploy scripts at those times.

Nikola has a feature for THE_FUTURE_IS_NOW, but I do want to wait until the appointed time before content is visible.

So my plan is to use a custom script to check for files whose scheduled publication time is in the same quarter-hour as this invocation of the cron job.

So in cron:

14,29,44,59  *  *  *  *  blog sh /var/server1/shares/public/Support/Programs/nikola/scripts/follow-schedule.sh 1>>/var/server1/shares/public/Support/Systems/server1/var/log/nikola_schedule/log 2>&1

Notice how in this cron entry I actually rely on cron to send the output to log files. This is a departure from the norm.

And file follow-schedule.sh:

#!/bin/sh
# Startdate: 2021-09-01 21:38
# Purpose: find nikola posts that should be released now and then build and deploy
# Usage:
#    ./follow-schedule.sh "2021-09-03 09:16"
# Design assumption:
#    expecting only one post per quarter-hour!
#    due to ssh possibility, cannot rely on DEBUG, VERBOSE, APPLY, DRYRUN environment variables. Just always be verbose.
# Dependencies:
#    plecho from bgscripts-core
# Documentation:
#    /mnt/public/Support/Programs/nikola/blog-README.md
# flow: find anything scheduled within this 15-minute block
if ! test "$( hostname -s )" = "server1" ;
then
   echo "Connecting to server1..."
   ssh server1 /mnt/public/Support/Programs/nikola/scripts/follow-schedule.sh
else

if ! test "${USER}" = "blog" ;
then
   echo "Switching to user blog..."
   sudo su blog -c '/mnt/public/Support/Programs/nikola/scripts/follow-schedule.sh'
else

echo "START nikola scheduled job" | plecho
INDIR=/mnt/public/Support/Programs/nikola/kb2
test -n "${1}" && DATE_AND_HOUR="${1}"
test -z "${DATE_AND_HOUR}" && DATE_AND_HOUR="$( date "+%F %H" )"
test -n "${2}" && MINUTE="${2}"
test -z "${MINUTE}" && MINUTE="$( date "+%M" )"
export DATE_AND_HOUR
export THISDATE="$( echo "${DATE_AND_HOUR}" | awk '{print $1}' )"
results="$( grep --include '*.md' -riIE '^\.\. date: [0-9]{4}(-[0-9]{2}){2}' "${INDIR}" 2>/dev/null | grep "${DATE_AND_HOUR}" 2>/dev/null )"
NOW_QUARTER="$( printf '%s\n' "scale=0;${MINUTE}/15" | bc )"
echo "Evaluating date+hour ${DATE_AND_HOUR} with quarter-hour ${NOW_QUARTER}"
if test -n "${results}" ;
then
   POST_QUARTER="$( printf "%s\n" "scale=0;$( echo "${results}" | awk '{print $NF}' | awk -F':' '{print $2}' )/15" | bc )"
   if test "${POST_QUARTER}" = "${NOW_QUARTER}" ;
   then
      tf="$( echo "${results}" | sed -r -e 's/:\.\..*$//;' )"
      scheduled_time="$( echo "${results}" | awk '{print $NF}' )"
      echo "Found a post for this quarter-hour: ${tf} at ${scheduled_time}"
      # determine if needs to be moved to YYYY/MM
      YYYY_MM_DD="$( echo "${THISDATE}" | awk -F'-' 'BEGIN{OFS="/"} {print $1,$2,$3}' )"
      if ! echo "${tf}" | grep -qiE "${YYYY_MM_DD}" ;
      then
         echo "mv \"${tf}\" \"${INDIR}/posts/${YYYY_MM_DD}\""
         mkdir -p "${INDIR}/posts/${YYYY_MM_DD}"
         mv "${tf}" "${INDIR}/posts/${YYYY_MM_DD}/"
      else
         echo "File \"${tf}\" does not need to be moved, but it is scheduled for now."
      fi
      # and now run build and deploy
      echo "Now will build and deploy."
      /mnt/public/Support/Programs/nikola/scripts/build.sh && \
      /mnt/public/Support/Programs/nikola/scripts/deploy.sh
   else # this quarter-hour
      echo "No files found for this quarter-hour."
   fi # end-if this quarter-hour
fi # end-if any 
echo "STOP nikola scheduled job" | plecho

# end-if-not-user-blog
fi

# end-if-not-server1
fi

Logrotate, audit.log, selinux, cron, and ansible

bgstack15 — Tue, 13 Feb 2018 14:19:44 GMT

The story

The disk space for /var/log/audit/audit.log tends to get filled up. The audit daemon has an ability to rotate its own logs. See the man page for auditd.conf.

max_log_file             =  100
max_log_file_action      =  rotate

That's swell and all, until you realize that auditd cannot compress its rotated logs. On a small /var/log/audit mount point, you'll fill it up with uncompressed logs.

/dev/mapper/os-var_log_audit          2490M 2136M      355M      86% /var/log/audit

So on a RHEL7 system with logrotate, you can adjust logrotate to handle the audit.log file. Now, logrotate is a finicky application. It has caused me many hours of grief in the past. You would want to set auditd.conf a certain way:

max_log_file             =  0
max_log_file_action      =  ignore

And set /etc/logrotate.d/audit:

/var/log/audit/*.log {
        weekly
        missingok
        compress
        #copytruncate
        rotate 30
        minsize 100k
        maxsize 200M
        postrotate
                touch /var/log/audit/audit.log ||:
                chmod 0600 /var/log/audit/audit.log ||:
                service auditd restart
        endscript
}

And ensure you've got a /etc/cron.weekly/logrotate:

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

After a few days, I learned that my logs were getting filled up so fast, the weekly rotation wasn't good enough. So I had to place it in my cron.hourly. And then I learned that it wasn't running every hour. I spent a few days investigating, and eventually learned that some systems use a specific status file for logrotate. I remember in the past logrotate needs an execution with a -f flag to force the rotation the first time and add a new file to the status file. So if a new file was never force-rotated, it won't be added to the status file. My manual logrotate -f command was indeed adding my audit.log log file to the status file, but to the wrong one! Some of my systems use -s /var/lib/logrotate/logrotate.status but the default is /var/lib/logrotate.status. So I had to reflect that in my ansible playbook. Actually, I had to write some logic to find the one used by the cronjob and then use that status file. So I got the correct logrotate status file set up in the ansible playbook. I spent the next week figuring out that logrotate simply couldn't rotate the file when called from cron. I piped the utility to tee, and also included the -v flag on logrotate. I saw a permission denied. With the permission issue, I had no choices left by selinux. I had to use the audit.log file to determine that the audit.log file is not readable by logrotate when called by cron. I finally set captured all the actions performed by logrotate by setting the selinux process context to be permissive:

semanage permissive -a logrotate_t


I let it run, and then had to collect all the actions it performed, and saw what had happened.


{ grep logrotate /var/log/audit/audit.log ; zgrep logrotate /var/log/audit/audit.log.1.gz ; } | audit2why

So I used audit2allow to convert it to an selinux policy.

{ grep logrotate /var/log/audit/audit.log ; zgrep logrotate /var/log/audit/audit.log.1.gz ; } | audit2allow -M logrotate-audit

And then after some searching online, I learned how I can keep the text definition file, and compile the policy from it when I need to:

grep logrotate /var/log/audit/audit.log | audit2allow -m logrotate-audit # saves to logrotate-audit.te
checkmodule -M -m -o logrotate-audit.mod logrotate-audit.te # intermediate step
semodule_package -o logrotate-audit.pp -m logrotate-audit.mod # compiled policy
semodule -i logrotate-audit.pp

The text definition of logrotate-audit policy:

#semodule -i logrotate-audit.pp

module logrotate-audit 1.0;

require {
        type auditd_etc_t;
        type logrotate_t;
        type auditd_log_t;
        class file { create getattr ioctl open read rename setattr unlink write };
        class dir { add_name read remove_name write };
}

#============= logrotate_t ==============
allow logrotate_t auditd_etc_t:file getattr;
allow logrotate_t auditd_log_t:dir { read write add_name remove_name };
allow logrotate_t auditd_log_t:file { create ioctl open read rename getattr setattr unlink write };

Now, I wrote a master ansible playbook that performs this whole operation, from loading the .te file and compiling it and installing it, to setting logrotate to watch the audit file, and telling auditd to ignore rotating it. Note : It is outside the scope of this task to ensure that the selinux tools are in place on each server. My environment already ensures package libselinux-python is present on each system, which should bring in all the dependencies of this ansible playbook.

---
# File: /etc/ansible/books/fix_var-log-audit.yml
# Author: bgstack15
# Startdate: 2018-01-24
# Title: Playbook that Fixes the /var/log/audit Space Issue
# Purpose: Logical Disk Free Space is too low
# History:
# Usage:
#    ansible-playbook -i /etc/ansible/inv/hosts /etc/ansible/configuration/fix_var-log-audit.yml -l hostwithproblem201
#    Use the -l host1,host2 parameter.
# Reference:
#    roles/general_conf/tasks/04_selinux.yml
#    roles/general_conf/tasks/05_auditd.yml
# Improve:
# Documentation:
#    The intention with auditd is to minimize the disk usage of the logs

- hosts: all
  remote_user: ansible_user
  become: yes

  vars:
    auditd_conf: /etc/audit/auditd.conf
    auditd_log_cleanup_regex: '.*audit\.log\.[0-9]+'
    auditd_log_dir: /var/log/audit
    auditd_logrotate_conf: /etc/logrotate.d/audit

  tasks:

# To make it possible to just drop in files to the files directory and have this module read them automatically, use these two.
#  - name: learn full list of semodules available to install, modular list version
#    shell: warn=no find /etc/ansible/roles/general_conf/files/selinux/ -regex '.*.te' -printf '%f\n' | sed -r -e 's/\.te$//;'
#    register: semodules_list
#    changed_when: false
#    delegate_to: localhost
#    ignore_errors: yes

#  - name: learn semodule versions to install, modular list version
#   shell: warn=no grep -E '^\s*module\s+{{ item }}\s+[0-9\.]+;\s*$' /etc/ansible/roles/general_conf/files/selinux/{{ item }}.te | awk '{print $3*1000;}'
#    register: selinux_pol_versions_target
#    changed_when: false
#    delegate_to: localhost
#    with_items:
#    - "{{ semodules_list.stdout_lines }}"

  - name: learn semodule versions to install, static version
    shell: warn=no grep -E '^\s*module\s+{{ item }}\s+[0-9\.]+;\s*$' /etc/ansible/templates/{{ item }}.te | awk '{print $3*1000;}'
    register: selinux_pol_versions_target
    changed_when: false
    delegate_to: localhost
    with_items:
    - logrotate-audit

  #- debug:
  #    msg: "{{ item.item }} should be {{ item.stdout }}"
  #  with_items:
  #  - "{{ selinux_pol_versions_target.results }}"

  - name: learn current semodule versions
    shell: warn=no semodule --list | awk '$1=="{{ item.item }}" {print $2*1000} END {print "0";}' | head -n1
    register: selinux_pol_versions_current
    changed_when: false
    with_items:
    - "{{ selinux_pol_versions_target.results }}"

  - debug:
      msg: "{{ item.item.item }} is currently {{ item.stdout }} and should be {{ item.item.stdout }}"
    with_items:
    - "{{ selinux_pol_versions_current.results }}"

  #- pause:
  #    prompt: "Does the above look good?........................"

  - name: download selinux modules that need to be installed
    copy:
      src: "/etc/ansible/templates/{{ item.item.item }}.te"
      dest: "/tmp/{{ item.item.item }}.te"
      mode: 0644
      owner: root
      group: root
      backup: no
      force: yes
    changed_when: false
    when:
    - "item.item.stdout > item.stdout"
    with_items:
    - "{{ selinux_pol_versions_current.results }}"

  - name: install selinux modules
    shell: chdir=/tmp warn=no /usr/bin/checkmodule -M -m -o "/tmp/{{ item.item.item }}.mod" "/tmp/{{ item.item.item }}.te" && /usr/bin/semodule_package -m "/tmp/{{ item.item.item }}.mod" -o "/tmp/{{ item.item.item }}.pp" && /usr/sbin/semodule -v -i "/tmp/{{ item.item.item }}.pp"
    when:
    - "item.item.stdout > item.stdout"
    with_items:
    - "{{ selinux_pol_versions_current.results }}"

  - name: clean any temporary selinux modules files
    file:
      path: "/tmp/{{ item[0].item.item }}.{{ item[1] }}"
      state: absent
    changed_when: false
    when:
    - "item[0].item.stdout > item[0].stdout"
    with_nested:
    - "{{ selinux_pol_versions_current.results }}"
    - [ 'te', 'pp', 'mod' ]

##### END SELINUX PORTION

  # modify auditd.conf which notifies the handler
  - name: auditd does not keep logs
    lineinfile:
      path: "{{ auditd_conf }}"
      regexp: "{{ item.r }}"
      backrefs: yes
      line: "{{ item.l }}"
      create: no
      state: present
      backup: yes
    #notify: auditd handler
    with_items:
    - { r: '^max_log_file_action.*$', l: 'max_log_file_action      =  ignore' }
    - { r: '^max_log_file\s.*$', l: 'max_log_file             =  0' }

  # tarball and cleanup any existing audit.log.1 files
  - name: list all old auditd logs which need to be compressed and cleaned up
    shell: warn=no find /var/log/audit -regex {{ auditd_log_cleanup_regex }}
    register: cleanup_list
    ignore_errors: yes
    changed_when: cleanup_list.stdout_lines | length > 0

  - name: get archive filename
    shell: warn=no echo "audit.log.{{ ansible_date_time.epoch }}.tgz"
    register: audit_log_tgz
    changed_when: audit_log_tgz.stdout_lines | length != 1

  - name: touch archive file
    file:
      path: "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}"
      state: touch
      owner: root
      group: root
      mode: 0600
    when: cleanup_list.stdout_lines | length > 0

  - name: archive and cleanup existing audit.log.1 files
    archive:
      dest: "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}"
      path: "{{ cleanup_list.stdout_lines }}"
      format: gz
      owner: root
      group: root
      remove: yes
    ignore_errors: yes
    when: cleanup_list.stdout_lines | length > 0

  - name: check for existence of new tarball
    stat:
      path: "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}"
    ignore_errors: yes
    register: audit_log_tarball

  - name: place audit log tarball in auditd_log_dir
    shell: warn=no /bin/mv "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}" "{{ auditd_log_dir }}/"
    ignore_errors: yes
    when:
    - audit_log_tarball.stat.exists is defined
    - audit_log_tarball.stat.exists

  - name: get current size of audit log
    stat:
      path: "{{ auditd_log_dir }}/audit.log"
    ignore_errors: yes
    register: audit_log_stat

  - name: apply logrotate script for audit
    copy:
      src: /etc/ansible/templates/etc-logrotate.d-audit
      dest: "{{ auditd_logrotate_conf }}"
      owner: root
      group: root
      mode: 0644
      backup: yes

  - name: learn the logrotate.status file to use, if any
    shell: warn=no grep -rE -- 'bin\/logrotate\>.*(-s|--state)(\s|=)[\/[A-Za-z0-9\.]+\>' /etc/cron.* 2>/dev/null | grep -oE '(-s|--state)(\s|=)[\/[A-Za-z0-9\.]+\>' | sort | uniq | head -n1
    ignore_errors: yes
    changed_when: false
    register: this_logrotate_flag

  - name: show which logrotate.status file to use, if any
    debug:
      msg: "The status file that will be used is {{ this_logrotate_flag.stdout }}"

  - name: run logrotate
    shell: warn=no /usr/sbin/logrotate {{ this_logrotate_flag.stdout }} -f "{{ auditd_logrotate_conf }}"
    register: run_logrotate
    when: ( cleanup_list.stdout_lines | length > 0 ) or ( audit_log_stat.stat.exists and audit_log_stat.stat.size > 190000000 )

  handlers:
...

Summary

So, logrotate can be configured to rotate the audit log. It just takes a few minutes to configure correctly, after about 2 weeks of research and testing.

References

Weblinks

Personal effort

Hours and hours of my original research Years of administering RHEL servers with logrotate