Ubuntu Upstart Gotcha When Monitoring MySQL With God


Ubuntu Upstart Gotcha When Monitoring MySQL With GodI recently upgraded our two Slicehost VPSs from Ubuntu Hardy Heron (8.04) to Lucid Lynx (10.04). That’s from one LTS to another, and the process, overall, went very smoothly. To pull it off I followed the 8.04 to 10.04 instructions on the Slicehost Wiki. There were, however, two gotchas that I came across during the upgrade process.

For one, I needed to have the highest version kernel available installed on my VPS before starting the upgrade. The docs say “Make Sure Kernel 2.6.32 or Newer is Installed”, but that’s not high enough. I checked that condition before the upgrade, and met it, but after the upgrade when my slice rebooted it failed to come up. In a panic I emailed Slicehost support and they quickly got back to me saying “upgrade to the highest kernel version available”, which happened to be 2.6.35.4-rscloud. I did that, then my slice booted properly.

The bigger gotcha was with my server monitoring. I use god to monitor my web serving stack. That means god watches over MySQL. Ubuntu used to run the MySQL server with the tried and true /sbin/init which, when spawning the MySQL daemon, created a file /var/run/mysqld/mysqld.pid that contained the process id of the daemon. God relies upon that .pid file to detect whether or not the server is running. If the .pid file is missing, or the PID in the file doesn’t match the id in the process list that god knows about, it thinks the server is down and will start yelling at you via whatever notifications you have configured. Recent versions of Ubuntu use upstart to control the MySQL server. Upstart doesn’t create the .pid files that god needs for monitoring. That put a glitch in my setup.

I began searching for answers in god’s documentation and discovered that you can watch non-daemon processes (i.e. those that don’t create a .pid file) with god. To do so just don’t specify a pid_file in your god watch. In that case god will create  pid for you in /var/run/god and use it for monitoring. Simple enough, but no dice. For whatever reason that didn’t work for me. God would create /var/run/god/mysql.pid but still complain that the server is down. More searching led me to a Flowcoder post that provides a way to create a .pid file from an upstart script for server monitoring. The post is for a couchdb-lucene server, but the code was easily adapted for MySQL. I made it work by editing /etc/init/mysql.conf. Here are the important parts:

post-start script
    # Make a .pid file for god monitoring of MySQL
    PID=`status mysql |egrep -oi '([0-9]+)$' |head -n1`
    echo $PID > /var/run/mysqld/mysqld.pid
    ....
end script
 
post-stop script
   # Remove the .pid file used for god monitoring of MySQL
   rm -f /var/run/mysqld/mysqld.pid
end script

Voila! Now I can control the MySQL server via /usr/sbin/service and it will create and clean up /var/run/mysqld/mysqld.pid for me. Most important, god gets its .pid file and monitors happily. That’s all good, but I don’t like the fact that I had to edit a /etc/init file to do this. Anyone got a better solution?

Mighty Nice proudly serves its web content using Linux and other open source technologies

Incoming search terms:

  • upstart pid file
  • god upstart
  • upstart pid
  • ubuntu mysql pid
  • ubuntu mysql pid file
  • upstart god
  • upstart pidfile
  • upstart pid files
  • god upstart script
  • ubuntu mysqld pid

2 Responses to “Ubuntu Upstart Gotcha When Monitoring MySQL With God”

    • Chris says:

      Controlling the creation of MySQL’s .pid file via my.cnf is a cleaner solution, and it works like a charm. Thanks for sharing!

      Note that with MySQL 5.1 there is no [mysqlid] section (or so the MySQL docs suggest to me). I added “pid-file = /var/run/mysqld/mysqld.pid” to the [mysqld] section of my.cnf and it worked fine.

      I’ll leave the article as-is because the upstart script solution is useful for other servers that need monit or god monitoring.

Leave a Reply