data scientist, biophysicist, mac-unix zealot, pythonista

Automated Notifications of Experiment Progress: Combining Shell, SSH, Growl, and Prowl

In addition to covering my use of python in research, one of my goals for this blog is to share ways I use various other computational tools to automate basic research tasks. Such an opportunity arose this week with the onset of hardware issues causing one of our laboratory's NMR spectrometers to periodically stop running during a series of multiple day experiments. The instrument is still useable while waiting for a replacement part to arrive, but periodic reboots of the hardware are necessary for the experiment to continue where it left off. Thus, it is critical to know when such a reboot is necessary. Rather than make constant trips to the building where the instrument is located, I rigged up a notification system using some basic unix and Mac tools.

The instrument periodically writes data to a file when an experiment is running. So, my solution is a shell script that periodically monitors this file for updates and then sends Growl notifications to my Mac, which is located in another building.1 Because Growl on my Mac is configured to work with Prowl, a complimentary iOS notification system,2 I also receive notifications on my phone whenever the remote Mac is idle. As a bonus, I added the option to send the experimental data to my Mac for processing.

This solution requires SSH keys to be setup so the spectrometer's computer can login to my Mac when necessary. It also requires the installation of growlnotify, a command line accessory to Growl.

The script is written in the Z-shell3 and can be downloaded here. It asks for several configuration options, ensures the file that is used to track experiment progress exists, and then performs periodic checks on the modification time of this file until the script is halted. The optional transfer of data takes place via rsync4. The interval time must be adjusted so that the data file is written to more frequently than it is polled for modifications, otherwise the script will incorrectly indicate the experiment has stalled.

There are a few nuances associated with the encoding of the growlnotify string. First, the entire command has to be enclosed in quotes because it is sent through ssh to the remote Mac. Additionally, each of the arguments passed to growlnotify, such as the contents of the $message_string variable, must themselves be enclosed in quotes when executed on the remote Mac. I prefer to use single quotes for the growlnotify command and double quotes for the individual arguments. However, variables, such as $message_string, are not expanded within single quotes, so they must appear outside of the single quotes, as can be seen above. The contents of $message_string must still be enclosed in double quotes, which appear before and after the respective single quotes flanking $message_string.

Once I start my experiment on the spectrometer, I setup this script to run indefinitely. An added bonus is that it will notify you when the experiment has finished. If the instrument writes progress information to a plain text log file, this file can be queried and more detailed status information that can be included in the $message_string.

Here is an example5 of the Prowl status updates I receive on my iPhone. The actual script I use derives additional information from the log file of the NMR spectrometer.

"Prowl Status Updates"

This is a script I have been meaning to write for quite some time and I plan to use it, perhaps with less frequent notification times, even after the spectrometer is repaired.

  1. VNC could also be used to to check on the experiment, but I prefer a method that also notifies me when I am away from my computer. 

  2. There are other notification systems that will also work with Growl. I've also used Boxcar, but I switched to Prowl after Boxcar became unreliable at times. 

  3. I believe it will run in Bash with minimal or no modifications; however, if you're not using the Z-shell, you should be

  4. When using rsync in a script, I prefer the long version of argument names for readability because I never remember what the short versions mean. 

  5. With super secret science stuff obscured.