Check log activity

Log activity can be an important sign for your Linux system's health. As errors usually produce additional log entries a higher log rate accordingly shows that something in your system does not work like it should. Nevertheless it can still have other reasons like that you have for example activated an additional service or increased the log level of an application.

Anyway I think it's a good idea to monitor your log rate. Unfortunately I could not find something which already does the job. As follows created a solution on my own. It consists of two scripts. As one needs root permissions to read most logs I tend to realize the data collection in a separate script as root cron job. This script searches for all files in /var/log/ and counts their lines. Note that this can take quite some time and cause load accordingly depending on size and amount of your log files. It's just fine to replace all the line count section by a simple LOGFILES=$(du -s /var/log/ | cut -f1) which is by far faster offering similar information.
Furthermore it drops the file check_logactivity.log in /var/log/ to store the last amount of counted log lines. Finally it gives user, group and others read access. If you don't want that e.g. for security reasons you have to change the chmod parameter to 400 and set the file's ownership to your nrpe user.

check_logactivity.sh:
#!/bin/sh
# This script is published under a Creative Commons BY license
# Written by XXLRay http://www.xxlray.bplaced.net
#
# Counts the amount of lines in all logfiles

COUNTFILE="/var/log/check_logactivity.log"
OLDCOUNT=0
if [ -f $COUNTFILE ]
then
  OLDCOUNT=$(cat $COUNTFILE | cut -f2 -d ';')
fi
# fetch files and folders from /var/log
LOGFILES=$(find /var/log/* 2>/dev/null)
for LOGFILE in $LOGFILES
do
  # check if we have a file and not a directory
  if [ -f $LOGFILE ]
  then
    # count lines and format output string
    NUMBER=$(wc -l $LOGFILE | cut -f1 -d ' ' | sed 's/ //g')
    # check if we really fetched a number
    echo $NUMBER | egrep '[0-9]+$' >/dev/null 2>&1
    ISNUMBER=$?
    if [ $ISNUMBER -eq 0 ]
  then
      # add up line count
      LOGLINES=$(($LOGLINES + $NUMBER))
    else
      ERRORCODE=1
      ERRORMESSAGE="${ERRORMESSAGE}\n${NUMBER} is not a line count"
    fi
  fi
done
# store line count
echo "${OLDCOUNT};${LOGLINES}" > $COUNTFILE
chmod 444 $COUNTFILE
exit 0


The second script checks if the log file exists or is outdated e.g. because the cron job has been messed up. Furthermore it compares the amount of counted log lines to parameters which have to be given. It can be run out of the box as cron job or Nagios script. Very few modifications are needed to let it fit Cacti or Zabbix. You may use both scripts under a CC BY license.

check_logactivity.sh:
#!/bin/sh
# This script is published under a Creative Commons BY license
# Written by XXLRay http://www.xxlray.bplaced.net

USAGE="\n
Checks the amount of logs written.\n
\n
Written by XXLRay http://www.xxlray.bplaced.net
\n
Usage: $(basename $0) [-h] -w <warnlevel> -c <critlevel>\n
  warnlevel: warning shall be given when that many lines have been added since the last check\n
  critlevel: critical shall be reported when that many lines have been added since the last check\n
\n
e.g.: ./$(basename $0) -w 1000 -c 2000\n"

while getopts hc:w: OPT
do
  case "$OPT" in
    h)  echo -e "$USAGE"
        exit 1
        ;;
    w)  WARNLEVEL=$OPTARG
        ;;
    c)  CRITLEVEL=$OPTARG
        ;;
  [?])  echo -e "$USAGE" >&2
        exit 1
        ;;
  esac
done

# check parameters
if [ "$WARNLEVEL" == "" -o "$CRITLEVEL" == "" ]
then
  echo -e "At least one of the arguments was empty!\n"
  echo -e $USAGE
  exit 1
fi
# initialize values
ERRORCODE=0
ERRORMESSAGE=""
LOGLINES=0
LASTLOGLINES=0
# maximum log file age
MAXLOGAGE=5
# count log lines and store old value
COUNTFILE="/var/log/check_logactivity.log"
if [ -f $COUNTFILE ]
then
  LASTLOGLINES=$(cat $COUNTFILE | cut -f1 -d ';')
  LOGLINES=$(cat $COUNTFILE | cut -f2 -d ';')
else
  echo "!Critical: $COUNTFILE not found!"
  exit 2
fi
# calculate difference
DIFFERENCE=0
# check if we already counted log lines before
if [ $LASTLOGLINES -ne 0 ]
then
  DIFFERENCE=$(($LOGLINES - $LASTLOGLINES))
fi
# check if log file is still sufficiently up to date
OLDFILES=$(find /var/log/ -maxdepth 1 -name "check_logactivity.log" -mtime +${MAXLOGAGE})
if [ "$OLDFILES" != "" ]
then
  ERRORMESSAGE="Warning: Log file too old! Maybe the cronjob updating it is not running.\n${ERRORMESSAGE}"
  ERRORCODE=1
fi

# check if levels are reached
if [ $DIFFERENCE -gt $CRITLEVEL ]
then
  ERRORMESSAGE="!Critical: $DIFFERENCE are more than $CRITLEVEL new log messages since last check!\n${ERRORMESSAGE}"
  ERRORCODE=2
elif [ $DIFFERENCE -gt $WARNLEVEL ]
then
  ERRORMESSAGE="Warning: $DIFFERENCE are more than $WARNLEVEL new log messages since last check!\n${ERRORMESSAGE}"
  ERRORCODE=1
fi
#
if [ $ERRORCODE -eq 0 ]
then
  echo "The amount of $DIFFERENCE log messages since last check is OK."
else
  # error detected
  echo -e "$ERRORMESSAGE"
fi
exit $ERRORCODE