Email Digests with RVM, Clockwork, God, and Upstart

Update: I've been told that bluepill is 'better' than god. Check it out for yourself here. I must admit though, it has a nicer DSL.

This post is a result of needing to setup an automated mailer for an app at work; one approach I considered was to use the whenever gem but that is simply working with cron jobs and leveraging rake tasks for such a matter didn't sit well with me — especially since this may be sent out quite frequently.

Like Clockwork

Instead, I decided to look at the clockwork gem. The first step was to add clockwork as a gem and do bundle install — the same as doing bundle.

I proceeded to create app/clock.rb

require 'clockwork'
require File.expand_path('../config/boot', File.dirname(__FILE__))
require File.expand_path('../config/environment', File.dirname(__FILE__))

module Clockwork
  every(1.day, 'digest.send', :at => ['07:00', '10:00', '14:00', '17:00']) do
    Session.published.each do |s|
      Rails.logger.info "Sending DigestMailer" if Rails.env.development?
      DigestMailer.email_digest(s).deliver if ((Time.now < s.start) && (s.applicant_signups.count.to_i <= s.capacity.to_i))
    end
  end
end

As you can see, all it does it deliver the mailer if two criteria are met: (i) the 'Course' has not started yet and (ii) the 'Course' capacity has not been met.

Caveat: I've unfortunately gone with the very misleading name of 'Session' for courses. session happens to be a reserved word in Rails controllers but I've been able to work around this by making certain changes on how my forms submit.

When the event is triggered, I get to see this

$ tail -f log/clockwork.log
Starting clock for 4 events: [ digest.send digest.send digest.send digest.send ]
Triggering digest.send

Process monitoring like a God

god is a gem written by Tom Preston-Werner of Github (@mojombo). You can find detailed information on god over here. This is the god.conf I'm using to manage the clockwork process

RAILS_ENV   = ENV['RAILS_ENV']  || "production"
RAILS_ROOT  = ENV['RAILS_ROOT'] || "/home/appuser/sites/mytld.com"
PID_DIR     = "#{RAILS_ROOT}/log"

God.pid_file_directory = "#{PID_DIR}"

God.watch do |w|
  w.name     = "clockwork"
  w.interval = 30.seconds
  w.env      = { "RAILS_ENV" => RAILS_ENV }
  w.dir      = "#{RAILS_ROOT}"

  w.start    = "bundle exec clockwork #{RAILS_ROOT}/app/clock.rb"
  w.stop     = "kill -QUIT `cat #{PID_DIR}/clockwork.pid`"
  w.restart  = "kill -QUIT `cat #{PID_DIR}/clockwork.pid` && bundle exec clockwork #{RAILS_ROOT}/app/clock.rb"
  w.log      = "#{RAILS_ROOT}/log/clockwork.log"

  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = "#{PID_DIR}/clockwork.pid"

  w.uid = 'appuser'
  w.gid = 'appuser'

  # clean pid files before start if necessary
  w.behavior(:clean_pid_file)

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_exits)
  end

  # # restart if memory or cpu is too high
  # w.transition(:up, :restart) do |on|
  #   on.condition(:memory_usage) do |c|
  #     c.interval = 20
  #     c.above = 50.megabytes
  #     c.times = [3, 5]
  #   end
  # 
  #   on.condition(:cpu_usage) do |c|
  #     c.interval = 10
  #     c.above = 10.percent
  #     c.times = [3, 5]
  #   end
  # end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end  

end

I'd like to thank Jeff Rafter for his input and advice with regards to the above as well. You can see his god.conf for managing Unicorn here.

Configuring god alone though is not enough, as we need to ensure god is started on system boot and continues to operate as well. Hello, Upstart.

RVM & Upstart

To ensure that god continues to run, I simply saved the following as /etc/init/god.conf. Starting the service was as simple as doing start god and I was then able to do god status to view the status of the clockwork task.

The trick to get this to work was to create a RVM wrapper rvm wrapper ruby-1.9.2-p290 r192p290 god. RVM wrappers are further detailed here.

It is important to note the logs are saved at /var/log/god.log.

# god upstart service - myservice job file

description "God process monitoring"
author "Michael de Silva <[email protected]>"

# When to start and stop  the service
start on runlevel [2345]
stop on runlevel [016]

# Automatically restart process if crashed
respawn

# Essentially lets upstart know the process will detach itself to the background
expect fork

# Run before process
# pre-start script
#     [ -d /var/run/myservice ] || mkdir -p /var/run/myservice
#     echo "Put bash code here"
# end script

# Start the process
# Created a RVM wrapper via
#   `rvm wrapper ruby-1.9.2-p290 r192p290 god`
exec /usr/local/rvm/bin/r192p290_god -l /var/log/god.log -c /home/appuser/sites/mytld.com/config/mytld.god

Testing God

I'm sure your family Pastor would strongly advise against this but I wanted to make sure my setup works by killing the clockwork process. One problem though — for some bizarre reason, the clockwork PID is not being written to the log folder. This certainly requires further investigation!

Whilst perusing top I found a neat column to sort everything by, PPID or 'Parent Process ID'. I could see that god had a PID of 695 and, hey, what's that... a ruby process with a PPID of 695. That must be the clockwork process!

10792 appuser   20   0  156m  62m 4568 S    0  0.8   0:13.92   695   0:13 ruby
21681 appuser   20   0  166m  61m 1572 S    0  0.8   0:14.19     1   0:14 ruby
21687 appuser   20   0  166m  61m 1572 S    0  0.8   0:14.02     1   0:14 ruby
21693 appuser   20   0  166m  61m 1572 S    0  0.8   0:14.22     1   0:14 ruby
21699 appuser   20   0  166m  61m 1572 S    0  0.8   0:14.06     1   0:14 ruby
527 mysql       20   0  244m  37m 7496 S    0  0.5   0:30.76     1   0:30 mysqld
695 root        20   0  165m  20m 3428 S    0  0.3   3:01.38     1   3:01 god

Whilst restraining from making an "It's elementary, my Dear Watson" comment I killed off the process and kept an eye on clockwork.log

Starting clock for 4 events: [ digest.send digest.send digest.send digest.send ]
Triggering digest.send
Triggering digest.send
Triggering digest.send
Triggering digest.send
Triggering digest.send
Starting clock for 4 events: [ digest.send digest.send digest.send digest.send ]

It didn't come back straight away, as it had to reload the Rails environment. After a few seconds though, that last line appeared signaling that all was OK.

Confusion...

Why is clockwork.pid not being written to #{PID_DIR}?
On system boot, god reports the status is in the clockwork: init state. After killing the clockwork process, it changed to clockwork: start. It never reaches the up state — why?