Using a crontab @reboot as process-management is extremely limited and I'd only recommend it as a quick-and-dirty approach one step above manually managing the process entirely. I think the only option while sticking with this approach would be to manually start the process the first time. This would be good enough for a service that's on the production-daemon server for example, but not for a service that would need to be installed on either dynamically-provisioned frontends or ad-hoc instances.

If we want this to be managed in a more automated way, these are our options:

• Write an OS-specific service-init script from scratch, and install it using a Chef service resource.
  • Our servers are currently running Ubuntu 14.04, which supports sysvinit or Upstart scripts. However, future Ubuntu releases including 16.04 have eliminated Upstart and have instead adopted systemd as their service system of choice, so Upstart scripts will become obsolete with the next OS upgrade. For this reason, I'd prefer sticking with LSB-style sysvinit scripts. See newrelic_init.sh.erb for a minimal example of this approach.
• Using poise-service as a wrapper for setting up an OS-dependent service-init script with less fuss. See cdo-freegeoip::default for a minimal example of this approach.
• Adding a dependency to one of the many non-init process supervision wrapper utilities such as supervisord, monit, Circus, God, bluepill, Eye, [etc etc etc..] edit: realized that the process_queues_daemon already integrates daemons for this.
  • In order to launch automatically you'll still need to hook this into an OS-level service using one of the above approaches.

another quick-and-dirty approach that wouldn't require the full OS service integration would be to add a Chef execute resource to execute process_queues_daemon start if process_queues_daemon status shows it's not already running, something like:

execute 'process_queues_daemon' do
  cwd "#{dashboard_root}/bin/sqs"
  command 'bundle exec process_queues_daemon start'
  not_if 'bundle exec process_queues_daemon status'
  environment node['cdo-apps']['bundle_env']
  user node[:user]
  group node[:user]
end

Thanks for all the detailed suggestions!

The only place I really care about it running is on production-daemon. Test and Staging also have queues set up, so it could run there but we're not necessarily expecting it to process anything. The unit tests run with fake_sqs and don't need a processor daemon. I can always set it up manually if I need for an adhoc server, as we'll have to set up the queue_url configurations anyway.

That said, I like the sound of "less fuss", so I'd lean toward the poise_service or execute resource approach. In either case, would I need to create a new entry under cookbooks? I haven't worked with chef recipes before so I'm a little unclear on the details. Since this is already using the daemon gem, do we get added benefit with poise_service over execute?

Good point about the warning on restart. I do want to restart (or otherwise coerce it to pick up config changes) on deploy, so maybe do something similar to your proposed execute command but wth restart if it's already running and start otherwise?

Second attempt, now with poise_service. Please take another look.

Followup item: we still need to find a way to deal with config changes in the future.

I think this looks good, though worth thinking about whether we want to process queues only on the daemon, or if we want to also process them in parallel across all frontends as well (tasks would be distributed randomly on a first-come-first serve basis). Since this is only going to be a low-volume queue for now, maybe easier debugging-wise to have the service run only on the daemon for starters?

If so, wrapping an if node['cdo-apps']['daemon'] conditional around the poise_service resource should do the trick.

I agree only daemon for now. This is going to be low volume. We can reevaluate if we revive the high volume activity queue for HOC (which ran on two servers last year) or something similar.

Updated with the daemon conditional.

Great! So the current code will use a separate SQS queue for each of our daemon-managed server environments (staging, test, levelbuilder, production), as far as I can tell. Is this intended, Has each of these queues been created in our aws account? If so it all looks good to me 👍

If these queues were manually created via the AWS console, a future task would be to describe and maintain them via our cloudformation stack template - not necessary for this initial rollout (more a note to myself to circle back around for this later).

I created queues in the AWS console for staging, test, and production. I didn't do levelbuilder, but we shouldn't be creating workshops there and without a queue the action will simply process synchronously and the queue processor will no-op. Do you think we should have a queue for levelbuilder too?

Ah, so adding all of the queues isn't necessary with the synchronous fallback in place. Sounds good! 👍