Setting up a Kiosk Watchdog for your Ubuntu Blackbox Kiosk

Previously I put together a post which describes how to build a Kiosk computer using Ubuntu, Blackbox and Firefox. I'm following that up with details on how to monitor the kiosk so that you can be notified when/if the computer or services fail. I'll break this post into two sections.

1) Monitoring the computer services and network
2) Monitoring FireFox performance

Monitoring the computer services and network

Because our Kiosk computer is an Ubuntu server and running apache/php/mysql locally there are several open source network and service monitoring programs. The one I found most suitable for my solution is called Monit and I found a very good post at Ubuntu Geek describing how to install monit on ubuntu and configure it. I basically followed the Ubuntu Geek tutorial but made some modifications to the config file. My revised process is below.

1. Install Monit
   

   sudo apt-get install monit

2. Configure Monit. Open /etc/monit/monitrc in your favorite text editor. Below is an example of how I set up my own configuration file. It should be pretty self explanatory.
   

   ## Start monit in background (run as daemon) and check the services at 2-minute
   ## intervals.
   set daemon  120
   
   ## Set syslog logging with the 'daemon' facility.
   set logfile syslog facility log_daemon
   
   ## Set list of mailservers for alert delivery.
   ## I use my ISP's SMTP server for better reliability and means
   ## I don't need an smtp server running on my Kiosk
   set mailserver mail.shawcable.com
   
   ## Use event queue if mailserver unavailable
   set eventqueue
   basedir /var/monit  # set the base directory where events will be stored
   slots 100           # optionaly limit the queue size
   
   ## You can set the alert recipient here
   set alert someone@domain.com
   
   # Monitor Apache
   check process apache2 with pidfile /var/run/apache2.pid
   
   # Action to be taken when apache fails
   start program = "/etc/init.d/apache2 start"
   stop program = "/etc/init.d/apache2 start"
   # Admin will notify by mail if below the condition satisfied below
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 60% for 5 cycles then restart
   if children > 10 then alert
   if children > 50 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout
   group servers
   
   # Monitor MySQL
   check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if failed host 127.0.0.1 port 3306 then alert
   if 5 restarts within 5 cycles then timeout
   
   # Monitor SSH Service
   check process sshd with pidfile /var/run/sshd.pid
   start program = "/etc/init.d/ssh start"
   stop program = "/etc/init.d/ssh stop"
   if failed port 22 protocol ssh then restart
   if failed port 22 protocol ssh then alert
   if 5 restarts within 5 cycles then timeout
   group programs
   
   # Check services
   check system localhost
   if loadavg (1min) > 4 then alert
   if loadavg (5min) > 2 then alert
   if memory usage > 75% then alert
   if cpu usage (user) > 70% then alert
   if cpu usage (system) > 80% then alert
   if cpu usage (wait) > 20% then alert

3. Set Monit to start automagically. Open the file /etc/default/monit and change the startup value to 1. You can now start monit with the following command.
   

   sudo /etc/init.d/monit start

If you want to be able to access Monit's web based interface remotely then check out the Ubuntu Geek tutorial for more information. I am not enabling this ability in my kiosk at present.

Monitoring FireFox performance

By using Monit we are able to get good alerts regarding the overall health of our Kiosk. But Monit doesn't tell us is how our Kiosk client (Firefox) is behaving. If Firefox starts to eat up a percentage of your kiosk's available memory or CPU power you should be notified early.

I myself am not that great at building BASH scripts so I opted to create a PHP script which tests a few conditions and sends an email if Firefox isn't running or is using too many resources. This script can be run as a cron job every few minutes. My PHP script requires the open source class PHPMailer which makes sending email from PHP a snap.

Here is my php script.

 /**
* This script gets the CPU and MEM usage of Firefox
*/
$cpu = 0;
$mem = 0;
$failed = false;

// Get the PID of firefox
$pid = exec('pidof firefox-bin');

// If firefox is running get memory
if ( $pid )
{
$status_str = exec('ps aux | grep "'. $pid .'" | grep -v grep');
$status = explode( ' ', $status_str );
// Strip blanks
foreach( $status as $k=>$v )
{
  if ( trim($v) == '' ) unset($status[$k]);
}
$status = array_values($status);

$cpu = $status[2];
$mem = $status[3];

if ( $cpu >= 60 || $mem >= 60 )
{
   $failed = true;
   $message = "Firefox is using $cpu% of CPU and $mem% of MEM";
}

} else {

// Firefox is not running
$failed = true;
$message = "Firefox is NOT running";

}

if ( $failed )
{
$body = date('l jS \of F Y h:i:s A',time() )."\n\n";
$body .= $message;

// Create and Send Email
require_once( "/class/phpmailer/class.phpmailer.php");

$mail = new PHPMailer();
$mail->From     = ' firefox@mykiosk ';
$mail->FromName = " Firefox Status ";
$mail->Host     = 'mail.shawcable.com';
$mail->Mailer   = "smtp";

$mail->Subject = "Firefox Issue on ArtTouch";
$mail->AddAddress(" someone@domain.com ");

$mail->IsHtml(0);
$mail->Body    = $body;

// LOG RESULTS
if(! $mail->Send() ) {
   error_log("There was an error ending Firefox Performance Alert " . $mail->ErrorInfo );
}
}

If anyone wants to turn this into a bash script instead of PHP and share it that would be great.