So far I found no easy solution to monitor for Linux kernel messages. So here is a simple Nagios plugin to scan dmesg output for interesting stuff:
#!/bin/bash
SEVERITIES="err,alert,emerg,crit"
WHITELIST="microcode: |\
Firmware Bug|\
i8042: No controller|\
Odd, counter constraints enabled but no core perfctrs detected|\
Failed to access perfctr msr|\
echo 0 > /proc/sys"
# Check for critical dmesg lines from this day
date=$(date "+%a %b %e")
output=$(dmesg -T -l "$SEVERITIES" | egrep -v "$WHITELIST" | grep "$date" | tail -5)
if [ "$output" == "" ]; then
echo "All is fine."
exit 0
fi
echo "$output" | xargs
exit 1
"Features" of the script above:
- It gives you the 5 most recent messages from today
- It allows to whitelist common but useless errors in $WHITELIST
- It uses "dmesg" to work when you already have disk I/O errors and to be faster than syslog parsing
This script helped a lot to early on detect I/O errors, recoverable as well as corruptions. It even worked when entire root partition wasn't readable anymore, because then the Nagios check failed with "NRPE: unable to read output" which indicated that dmesg didn't work anymore. By always showing all errors from the entire day one cannot miss recovered errors that happened in non-office hours. Another good thing about the check is detecting OOM kills or fast spawning of processes.