Linux use tee with color

bgstack15 — Tue, 21 Mar 2017 13:26:37 GMT

Output in color on the console

The console normally can display color. The most obvious example is when you do a standard ls command. By default on most systems, ls includes a --color=auto flag that changes filenames of certain types to different colors. Some applications display colorized output like systemctl status nfsd.service.

Using tee

Tee is a nifty cli utility that duplicates the input to both the standard out as well as to a file. You can use it like so:

ls -l --color=always | tee ~/ls.out

Some tools will disable colorized output if it is going to a pipe, but ls can be forced to provide color with the --color=always flag. For applications that don't have such an option, you can usually prepend the command unbuffer.

unbuffer ansible-playbook playbook.yml | tee ansible.log

When you examine the file, however, you will observe that the console color commands were saved. That's perfectly fine if you cat the file, or less -R, but normally you want a text file to just have the text. Sysadmins are quite used to dealing with logs that are all the standard color of the console.

The solution: ctee!

Using some fantastic resources on the Internet which I list at the bottom of this post, I assembled a tool that will tee the input, and send color to the stdout (usually the console) and send regular text to the file. I call my creation ctee.

Code walkthrough

So this script was generated using newscript which copies the template from the same bgscripts package. The template sources framework, which is my big flagship library of functions for shell scripts. Lines 39-54 are the parseFlag function, which parses the command line passed parameters. I implemented this before I ever learned about Python's argparse library. The only flag worthy to note is --append. All it does is set the variable to 1. The meat of this script starts at line 85 and goes to the end, which is shown below.

# this whole operation calculates where the stdout of this whole pipe goes
ttyresolved=0
_count=0
_pid=$$
_fd1="$( readlink -f /proc/${_pid}/fd/1 )"
while test ${ttyresolved} -lt 10;
do
   ttyresolved=$(( ttyresolved + 1 ))
   echo "before ${ttyresolved}, _pid=${_pid}, _fd1=${_fd1}" > ${devtty}
   case "${_fd1}" in
      *pipe:* )
         newpid=$( find /proc -type l -name '0' 2>/dev/null | xargs ls -l 2>/dev/null | grep -F "$( basename ${_fd1} )" | grep -viE "\/${_pid}\/"; )
         newpid=$( echo "${newpid}" | head -n1 | grep -oiE "\/proc\/[0-9]*\/" | grep -o "[0-9]*" )
         _pid=${newpid}
         _fd1="$( readlink -f /proc/${_pid}/fd/1 )"
         ;;
       *dev*|*/* )
         thisttyout="${_fd1}"
         ttyresolved=10
         ;;
   esac
done

echo "thisttyout ${thisttyout}" > ${devtty}

# MAIN LOOP
case "${append}" in
   1)
      tee ${thisttyout} | sed -u -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" >> "${outfile1}"
      ;;
   *) tee ${thisttyout} | sed -u -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" > "${outfile1}"
      ;;
esac

The idea behind ctee is that it uses tee, but inverts the outputs. Ctee uses the console tty as the file to output to, and then standard out continues on to a sed command that strips out the console color escape sequences which is then redirected to the file specified to ctee as a parameter. So one of the big tricks is to derive which console you are executing this command from. The while loop from lines 90-106 use a /proc filesystem trick to find the process on the other end of a pipe. This loop travels from this process's standard out to the standard in of another process, and then takes that standard out, and tracks that down until it includes the phrase "dev" or "/" in its name, indicating either a tty or a file (technically the dev is redundant, but it makes the line easier to understand). The second main trick of ctee is that it has a sed command strip out the console color control characters. I found a nice way to remove that, and you can see those commands on lines 113 and 115. One line is for appending to a file, and the other is for overwriting.

Summary

You can use ctee.sh like so:

unbuffer ansible-playbook playbook.yml | ctee.sh /var/log/ansible/playbook.$( date "+%Y-%m-%d-%H%M%S" ).log