Logrotate, audit.log, selinux, cron, and ansible
The story
The disk space for /var/log/audit/audit.log tends to get filled up. The audit daemon has an ability to rotate its own logs. See the man page for auditd.conf.
max_log_file = 100
max_log_file_action = rotate
That's swell and all, until you realize that auditd cannot compress its rotated logs. On a small /var/log/audit mount point, you'll fill it up with uncompressed logs.
/dev/mapper/os-var_log_audit 2490M 2136M 355M 86% /var/log/audit
So on a RHEL7 system with logrotate, you can adjust logrotate to handle the audit.log file. Now, logrotate is a finicky application. It has caused me many hours of grief in the past. You would want to set auditd.conf a certain way:
max_log_file = 0
max_log_file_action = ignore
And set /etc/logrotate.d/audit:
/var/log/audit/*.log {
weekly
missingok
compress
#copytruncate
rotate 30
minsize 100k
maxsize 200M
postrotate
touch /var/log/audit/audit.log ||:
chmod 0600 /var/log/audit/audit.log ||:
service auditd restart
endscript
}
And ensure you've got a /etc/cron.weekly/logrotate:
1 2 3 4 5 6 7 8 |
|
After a few days, I learned that my logs were getting filled up so fast, the weekly rotation wasn't good enough. So I had to place it in my cron.hourly. And then I learned that it wasn't running every hour. I spent a few days investigating, and eventually learned that some systems use a specific status file for logrotate. I remember in the past logrotate needs an execution with a -f flag to force the rotation the first time and add a new file to the status file. So if a new file was never force-rotated, it won't be added to the status file. My manual logrotate -f command was indeed adding my audit.log log file to the status file, but to the wrong one! Some of my systems use -s /var/lib/logrotate/logrotate.status but the default is /var/lib/logrotate.status. So I had to reflect that in my ansible playbook. Actually, I had to write some logic to find the one used by the cronjob and then use that status file. So I got the correct logrotate status file set up in the ansible playbook. I spent the next week figuring out that logrotate simply couldn't rotate the file when called from cron. I piped the utility to tee, and also included the -v flag on logrotate. I saw a permission denied. With the permission issue, I had no choices left by selinux. I had to use the audit.log file to determine that the audit.log file is not readable by logrotate when called by cron. I finally set captured all the actions performed by logrotate by setting the selinux process context to be permissive:
semanage permissive -a logrotate_t
I let it run, and then had to collect all the actions it performed, and saw what had happened.
{ grep logrotate /var/log/audit/audit.log ; zgrep logrotate /var/log/audit/audit.log.1.gz ; } | audit2why
So I used audit2allow to convert it to an selinux policy.
{ grep logrotate /var/log/audit/audit.log ; zgrep logrotate /var/log/audit/audit.log.1.gz ; } | audit2allow -M logrotate-audit
And then after some searching online, I learned how I can keep the text definition file, and compile the policy from it when I need to:
grep logrotate /var/log/audit/audit.log | audit2allow -m logrotate-audit # saves to logrotate-audit.te
checkmodule -M -m -o logrotate-audit.mod logrotate-audit.te # intermediate step
semodule_package -o logrotate-audit.pp -m logrotate-audit.mod # compiled policy
semodule -i logrotate-audit.pp
The text definition of logrotate-audit policy:
#semodule -i logrotate-audit.pp
module logrotate-audit 1.0;
require {
type auditd_etc_t;
type logrotate_t;
type auditd_log_t;
class file { create getattr ioctl open read rename setattr unlink write };
class dir { add_name read remove_name write };
}
#============= logrotate_t ==============
allow logrotate_t auditd_etc_t:file getattr;
allow logrotate_t auditd_log_t:dir { read write add_name remove_name };
allow logrotate_t auditd_log_t:file { create ioctl open read rename getattr setattr unlink write };
Now, I wrote a master ansible playbook that performs this whole operation, from loading the .te file and compiling it and installing it, to setting logrotate to watch the audit file, and telling auditd to ignore rotating it. Note : It is outside the scope of this task to ensure that the selinux tools are in place on each server. My environment already ensures package libselinux-python is present on each system, which should bring in all the dependencies of this ansible playbook.
---
# File: /etc/ansible/books/fix_var-log-audit.yml
# Author: bgstack15
# Startdate: 2018-01-24
# Title: Playbook that Fixes the /var/log/audit Space Issue
# Purpose: Logical Disk Free Space is too low
# History:
# Usage:
# ansible-playbook -i /etc/ansible/inv/hosts /etc/ansible/configuration/fix_var-log-audit.yml -l hostwithproblem201
# Use the -l host1,host2 parameter.
# Reference:
# roles/general_conf/tasks/04_selinux.yml
# roles/general_conf/tasks/05_auditd.yml
# Improve:
# Documentation:
# The intention with auditd is to minimize the disk usage of the logs
- hosts: all
remote_user: ansible_user
become: yes
vars:
auditd_conf: /etc/audit/auditd.conf
auditd_log_cleanup_regex: '.*audit\.log\.[0-9]+'
auditd_log_dir: /var/log/audit
auditd_logrotate_conf: /etc/logrotate.d/audit
tasks:
# To make it possible to just drop in files to the files directory and have this module read them automatically, use these two.
# - name: learn full list of semodules available to install, modular list version
# shell: warn=no find /etc/ansible/roles/general_conf/files/selinux/ -regex '.*.te' -printf '%f\n' | sed -r -e 's/\.te$//;'
# register: semodules_list
# changed_when: false
# delegate_to: localhost
# ignore_errors: yes
# - name: learn semodule versions to install, modular list version
# shell: warn=no grep -E '^\s*module\s+{{ item }}\s+[0-9\.]+;\s*$' /etc/ansible/roles/general_conf/files/selinux/{{ item }}.te | awk '{print $3*1000;}'
# register: selinux_pol_versions_target
# changed_when: false
# delegate_to: localhost
# with_items:
# - "{{ semodules_list.stdout_lines }}"
- name: learn semodule versions to install, static version
shell: warn=no grep -E '^\s*module\s+{{ item }}\s+[0-9\.]+;\s*$' /etc/ansible/templates/{{ item }}.te | awk '{print $3*1000;}'
register: selinux_pol_versions_target
changed_when: false
delegate_to: localhost
with_items:
- logrotate-audit
#- debug:
# msg: "{{ item.item }} should be {{ item.stdout }}"
# with_items:
# - "{{ selinux_pol_versions_target.results }}"
- name: learn current semodule versions
shell: warn=no semodule --list | awk '$1=="{{ item.item }}" {print $2*1000} END {print "0";}' | head -n1
register: selinux_pol_versions_current
changed_when: false
with_items:
- "{{ selinux_pol_versions_target.results }}"
- debug:
msg: "{{ item.item.item }} is currently {{ item.stdout }} and should be {{ item.item.stdout }}"
with_items:
- "{{ selinux_pol_versions_current.results }}"
#- pause:
# prompt: "Does the above look good?........................"
- name: download selinux modules that need to be installed
copy:
src: "/etc/ansible/templates/{{ item.item.item }}.te"
dest: "/tmp/{{ item.item.item }}.te"
mode: 0644
owner: root
group: root
backup: no
force: yes
changed_when: false
when:
- "item.item.stdout > item.stdout"
with_items:
- "{{ selinux_pol_versions_current.results }}"
- name: install selinux modules
shell: chdir=/tmp warn=no /usr/bin/checkmodule -M -m -o "/tmp/{{ item.item.item }}.mod" "/tmp/{{ item.item.item }}.te" && /usr/bin/semodule_package -m "/tmp/{{ item.item.item }}.mod" -o "/tmp/{{ item.item.item }}.pp" && /usr/sbin/semodule -v -i "/tmp/{{ item.item.item }}.pp"
when:
- "item.item.stdout > item.stdout"
with_items:
- "{{ selinux_pol_versions_current.results }}"
- name: clean any temporary selinux modules files
file:
path: "/tmp/{{ item[0].item.item }}.{{ item[1] }}"
state: absent
changed_when: false
when:
- "item[0].item.stdout > item[0].stdout"
with_nested:
- "{{ selinux_pol_versions_current.results }}"
- [ 'te', 'pp', 'mod' ]
##### END SELINUX PORTION
# modify auditd.conf which notifies the handler
- name: auditd does not keep logs
lineinfile:
path: "{{ auditd_conf }}"
regexp: "{{ item.r }}"
backrefs: yes
line: "{{ item.l }}"
create: no
state: present
backup: yes
#notify: auditd handler
with_items:
- { r: '^max_log_file_action.*$', l: 'max_log_file_action = ignore' }
- { r: '^max_log_file\s.*$', l: 'max_log_file = 0' }
# tarball and cleanup any existing audit.log.1 files
- name: list all old auditd logs which need to be compressed and cleaned up
shell: warn=no find /var/log/audit -regex {{ auditd_log_cleanup_regex }}
register: cleanup_list
ignore_errors: yes
changed_when: cleanup_list.stdout_lines | length > 0
- name: get archive filename
shell: warn=no echo "audit.log.{{ ansible_date_time.epoch }}.tgz"
register: audit_log_tgz
changed_when: audit_log_tgz.stdout_lines | length != 1
- name: touch archive file
file:
path: "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}"
state: touch
owner: root
group: root
mode: 0600
when: cleanup_list.stdout_lines | length > 0
- name: archive and cleanup existing audit.log.1 files
archive:
dest: "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}"
path: "{{ cleanup_list.stdout_lines }}"
format: gz
owner: root
group: root
remove: yes
ignore_errors: yes
when: cleanup_list.stdout_lines | length > 0
- name: check for existence of new tarball
stat:
path: "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}"
ignore_errors: yes
register: audit_log_tarball
- name: place audit log tarball in auditd_log_dir
shell: warn=no /bin/mv "{{ auditd_log_dir }}/../{{ audit_log_tgz.stdout }}" "{{ auditd_log_dir }}/"
ignore_errors: yes
when:
- audit_log_tarball.stat.exists is defined
- audit_log_tarball.stat.exists
- name: get current size of audit log
stat:
path: "{{ auditd_log_dir }}/audit.log"
ignore_errors: yes
register: audit_log_stat
- name: apply logrotate script for audit
copy:
src: /etc/ansible/templates/etc-logrotate.d-audit
dest: "{{ auditd_logrotate_conf }}"
owner: root
group: root
mode: 0644
backup: yes
- name: learn the logrotate.status file to use, if any
shell: warn=no grep -rE -- 'bin\/logrotate\>.*(-s|--state)(\s|=)[\/[A-Za-z0-9\.]+\>' /etc/cron.* 2>/dev/null | grep -oE '(-s|--state)(\s|=)[\/[A-Za-z0-9\.]+\>' | sort | uniq | head -n1
ignore_errors: yes
changed_when: false
register: this_logrotate_flag
- name: show which logrotate.status file to use, if any
debug:
msg: "The status file that will be used is {{ this_logrotate_flag.stdout }}"
- name: run logrotate
shell: warn=no /usr/sbin/logrotate {{ this_logrotate_flag.stdout }} -f "{{ auditd_logrotate_conf }}"
register: run_logrotate
when: ( cleanup_list.stdout_lines | length > 0 ) or ( audit_log_stat.stat.exists and audit_log_stat.stat.size > 190000000 )
handlers:
...
Summary
So, logrotate can be configured to rotate the audit log. It just takes a few minutes to configure correctly, after about 2 weeks of research and testing.
References
Weblinks
- http://melikedev.com/2013/08/19/linux-selinux-semodule-compile-pp-module-from-te-file/
- https://linux.die.net/man/8/auditd.conf
Personal effort
Hours and hours of my original research Years of administering RHEL servers with logrotate
Comments