How to get useful system logs on CentOS 8

bgstack15 — Fri, 04 Jun 2021 13:18:03 GMT

Beware the short lifespan of CentOS 8. I haven't migrated to one of the replacements yet, so go investigate for me and tell me what you think of these: Rocky Linux AlmaLinux In the mean time, my lone CentOS 8 system needs real logging. Journald doesn't seem to store logs, because it is poorly written. Or given poor defaults, and what's the difference? So in file /etc/systemd/journald.conf go set variable:

[Journal]
Storage=persistent

But to get real logging, you need to go configure rsyslog.

sudo dnf install rsyslog

And that has sane defaults that actually log messages to disk.

References

Weblinks

Disable apparmor for sssd

bgstack15 — Thu, 03 Dec 2020 13:39:28 GMT

tl;dr

Turn it off

sudo ln -sf /etc/apparmor.d/usr.sbin.sssd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.sssd

Turn it back on

sudo unlink /etc/apparmor.d/disable/usr.sbin.sssd
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.sssd
sudo aa-status # to verify visually

The story

I use FreeIPA on Devuan GNU+Linux. It's only marginally supported in Debian, and even less so in Devuan. The sssd component, which is used to get entries in the passwd and group databases, tends to fill up /var/log/messages with way too many apparmor notices.

Nov 29 15:56:27 ws005 kernel: [2158971.927938] audit: type=1400 audit(1606683387.308:34490156): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/etc/host.conf" pid=16466 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 29 15:56:27 ws005 kernel: [2158971.928030] audit: type=1400 audit(1606683387.308:34490157): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/etc/resolv.conf" pid=16466 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 29 15:56:27 ws005 kernel: [2158971.928226] audit: type=1400 audit(1606683387.308:34490158): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/etc/resolv.conf" pid=16465 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 29 15:56:27 ws005 kernel: [2158971.928230] audit: type=1400 audit(1606683387.308:34490159): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=16465 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 29 15:56:27 ws005 kernel: [2158971.928233] audit: type=1400 audit(1606683387.308:34490160): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_sudo" name="/etc/host.conf" pid=16467 comm="sssd_sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I miss SELinux.

References

Adapted directly from Ubuntu Linux: Disable Apparmor For Specific Profile / Service Such As Mysqld Server - nixCraft

Plex Activity Logging and Alerting Project

bgstack15 — Fri, 21 Aug 2020 12:51:02 GMT

Overview

Plex media server has the ability to send webhooks when used with a Plex Pass (premium) account. This project documents how to set up a webhook to log and alert on media activity. The design goals include logging all media playing activity (play, stop, pause, resume) on my Plex server, and sending email notifications. Technologies used: apache, shell, selinux, jq, mailx

Architecture

Plex can be configured to point to a webhook, e.g., http://plex.example.com/cgi-bin/plex-log.cgi. The webhook file is documented below. The webhook is a script that acts upon the data passed from Plex. The actions include copying important values to different log files, and invoking a separate script that sends an email to the admin.

Involved files

Multiple files are a part of this project.

/var/server1/shares/public/www/cgi-bin/plex-log.cgi
/var/server1/shares/public/Support/Systems/server1/var/log/plex-webhook/activity.log
/var/server1/shares/public/Support/Systems/server1/var/log/plex-webhook/short.log
/usr/local/bin/send-plex-alert.sh
/usr/src/selinux/plexlog.te

Webhook script

#!/bin/sh
# File: server1:/var/server1/shares/public/www/cgi-bin/plex-log.cgi
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2020-08-06 07:55
# Title: Logging Webhook for Plex
# Project: Plex Activity Logging and Alerting Project
# History:
# References:
#    send from bgscripts
#    /posts/2017/08/05/send-authenticated-gmail-from-cli-with-mailx/
#    Not sure which mail I have https://unix.stackexchange.com/questions/15405/how-do-i-send-html-email-using-linux-mail-command/15463#15463
# Related files:
#    /usr/local/bin/send-plex-alert.sh
# Improve:
# Dependencies:
#    /usr/local/bin/send-plex-alert.sh
#    Webhook functionality of Plex with a Plex Pass account
#    apache cgi capability
# Documentation:
#    /mnt/public/Support/Programs/Plex/webhooks/palap-readme.md
LOGFILE=/var/server1/shares/public/Support/Systems/server1/var/log/plex-webhook/activity.log
LOGFILESHORT=/var/server1/shares/public/Support/Systems/server1/var/log/plex-webhook/short.log
VERBOSE=0 # set to anything to display extra things
ALERT_ON_EVENTS="play:stop:pause:resume" # options include play stop resume pause, colon separated
NOW="$( date -u "+%FT%TZ" )"
TMPFILE1="$( mktemp )"
TMPFILE2="$( mktemp )"
TMPFILE3="$( mktemp )"
printf "%s\n\n" "Content-type: text/html"
echo "<html><head><title>Plex webhook</title>"
echo "</head><body>"
echo "<pre>"
cat > "${TMPFILE1}"
boundary="$( echo "${CONTENT_TYPE}" | awk '{print $2}' | awk -F'=' '{print $2}' )"
{
    test "${VERBOSE}" -gt 0 && {
       printf "%s\n" "" >> "${LOGFILE}"
      echo "REQUEST-START: ${NOW}"
    env
    set
    echo "BEGIN RAW"
   }
    sed -r -e "s/-*${boundary}-*/\n/g;" "${TMPFILE1}" | tee "${TMPFILE2}" | \
      {
         test "${VERBOSE}" -gt 0 && { cat ; } || { cat >/dev/null ; }
      }
    test "${VERBOSE}" -gt 0 && {
      echo "END RAW"
      echo "BEGIN JQ"
   }
   json="$( grep -aE "^.{0,4}\"event\"" "${TMPFILE2}" | jq -c ". += {timestamp: \"${NOW}\", useragent: \"${HTTP_USER_AGENT}\" }" 2>&1 )"
   # full json for log
   echo "${json}"
   # send truncated json to mail account for alerts
   echo "${json}" | jq '{timestamp: .timestamp, event: .event, user: .Account.title, player: .Player.title, ipAddress: .Player.publicAddress, file: {type: .Metadata.type, title: .Metadata.title, parentTitle: .Metadata.parentTitle, grandparentTitle: .Metadata.grandparentTitle } }' > "${TMPFILE3}" 
   # log to short log and send email only if it has contents
   printf "%s" "${json}" | grep -qE '.' && {
      cat "${TMPFILE3}" >> "${LOGFILESHORT}"
      # only alert if it matches. This syntax relies on the event syntax from Plex to be "media.play" or "media.stop" et al.
      alert_regex="(media\.)$( echo "${ALERT_ON_EVENTS}" | sed -r -e 's/^://g;' -e 's/:$//g;' -e 's/:+/|/g;' )"
      if grep -qE "${alert_regex}" "${TMPFILE3}" ;
      then
         # magic to get fixed-width font in gmail by using html pre tags
         { echo "<html><body><pre>" ; cat "${TMPFILE3}" ; echo "</pre></html></body>" ; } | mailsubject="Media activity
Content-Type: text/html" /usr/local/bin/send-plex-alert.sh STDIN
      else
         test "${VERBOSE}" -gt 0 && {
            echo "SKIPPING EMAIL because event does not match ${ALERT_ON_EVENTS}"
         }
      fi
   }
   test "${VERBOSE}" -gt 0 && {
      echo "END JQ"
      echo "REQUEST-STOP: ${NOW}"
   }
} | tee -a "${LOGFILE}"
echo "</pre>"
echo "</body></html>"

# clean up
rm -f "${TMPFILE1}" "${TMPFILE2}" "${TMPFILE3}"

The above webhook script contains its own comments. The VERBOSE attribute can be set to a non-zero amount (usually 1 ) to log much more information useful for debugging. Attribute ALERT_ON_EVENTS is a colon-separated list of event types passed from Plex that should trigger an email to the admin. This syntax depends on Plex's syntax not changing. The two log files are configured in the main webhook script. The LOGFILE contains the verbose logging (if enabled) and the full json object from Plex. The LOGFILESHORT contains the trimmed contents that will be used for the email.

Email script

A separate script holds the logic to send the email, file /usr/local/bin/send-plex-alert.sh.

#!/bin/sh
# File: send-plex-alert.sh
# Project: Plex Activity Logging and Alerting Project
# startdate: 2020-08-07 17:21
# Documentation:
#    this one worked, but took a while, probably due to ISP smtp relay delay. echo "echo juliet oscar 10" | mailx -S 'from="bgstack15@ipa.example.com"' -S "Subject line" bgstack15@gmail.com ; echo $?
#    worked immediately: echo "hotel whiskey zulu 14" | mailx -v -S "Another thread here" -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp=smtp://smtp.gmail.com:587 -S from="B Stack <bgstack15@gmail.com>" -S smtp-auth-user="bgstack15@gmail.com" -S smtp-auth-password='FIXME' -S nss-config-dir=/etc/pki/nssdb/ bgstack15@gmail.com
# Improve:
#    learn how to use a custom MAILRC env var which holds the "set smtp-user-password=" contents
# Dependencies:
#    mailx (mailx-12.5-19.el7.x86_64 @base)
#    Google account security set to allow "insecure apps" or similar https://support.google.com/accounts/answer/6010255?authuser=2&p=lsa_blocked&hl=en&authuser=2&visit_id=637324418441299012-2559162990&rd=1

test -z "${mailmessage}" && export mailmessage="${1}"
test "${mailmessage}" = "STDIN" && export mailmessage="$( cat 2>/dev/null )"
test -z "${mailuser}" && export mailuser="bgstack15@gmail.com"
test -z "${mailpassword}" && export mailpassword='SUPERSAFESTRING;'
test -z "${mailfrom}" && export mailfrom="Plex activity <bgstack15@gmail.com>"
test -z "${mailsubject}" && export mailsubject="Media activity"
test -z "${mailto}" && export mailto=bgstack15@gmail.com
printf "%s\n" "${mailmessage}" | mailx -s "${mailsubject}" \
   -S smtp-use-starttls \
   -S ssl-verify=ignore \
   -S smtp=smtp://smtp.gmail.com:587 \
   -S smtp-auth=login \
   -S from="${mailfrom}" \
   -S smtp-auth-user="${mailuser}" \
   -S smtp-auth-password="${mailpassword}" \
   -S nss-config-dir=/etc/pki/nssdb \
   "${mailto}"

SELinux policy

To use the webhook script and mail script with SELinux enforcing, you need to add a custom policy. File /usr/src/selinux/plexlog.te contains the uncompiled policy.

# File: /usr/src/selinux/plexlog.te
# Startdate: 2020-08-07 20:45
# Title: SELinux policy to allow webhook to send email
# Project: Plex Activity Logging and Alerting Project
module plexlog 1.0;

require {
    type httpd_sys_script_t;
    type init_t;
    type httpd_t;
    type smtp_port_t;
    type var_t;
    class process { noatsecure rlimitinh siginh };
    class unix_stream_socket { read write };
    class capability net_admin;
    class file { append execute getattr open read };
    class tcp_socket name_connect;
}

#============= httpd_sys_script_t ==============

#!!!! WARNING: 'var_t' is a base type.
allow httpd_sys_script_t var_t:file { append execute getattr open read };

#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_connect, nis_enabled
allow httpd_sys_script_t smtp_port_t:tcp_socket name_connect;

#============= httpd_t ==============
allow httpd_t httpd_sys_script_t:process { noatsecure rlimitinh siginh };
allow httpd_t self:capability net_admin;

To use this policy, you need to compile and install it.

sudo checkmodule -M -m -o plexlog.mod plexlog.te && sudo semodule_package -o plexlog.pp -m plexlog.mod && sudo semodule -i plexlog.pp

Knowledge Base (Posts about log)

How to get useful system logs on CentOS 8

References

Weblinks

Disable apparmor for sssd

tl;dr

Turn it off

Turn it back on

The story

References

Plex Activity Logging and Alerting Project

Overview

Architecture

Involved files

Webhook script

Email script

SELinux policy

References

Weblinks