librelist archives

« back to archive

Monitoring queue latency with the sidekiq_queue_status gem

Monitoring queue latency with the sidekiq_queue_status gem

Matthijs Langenberg
2014-06-06 @ 13:11
In search for ways to monitor our sidekiq processes, Mike inspired me
to monitor
the queue latency by using an end-point in my Rails routes.

In practice I have seen two problematic type of situations.

1. Queue too large because sidekiq process halted or too few workers for
the current load.
2. Failure rate skyrocketed because an old process is taking jobs and
failing them.

I would like to share my attempt to improve monitoring for our own Rails
projects, which results in the sidekiq_queue_status gem:

The gem takes Mike's approach and makes monitoring sidekiq a no-brainer. It
is very easy to reuse across Rails apps.

Just add the gem to your Gemfile:

  gem 'sidekiq_queue_status'

Now your app responds to GET /queue-status. Normally it returns a 200 OK,
by measuring two metrics it returns a 503 Service Unavailable in the
following situations:

a) When the latency of any queue is above 30 seconds.
b) When the failure rate for today is above 10%

The maximum latency is configurable per queue, as is the maximum failure

It fits our needs for monitoring sidekiq. Please share ideas that could
help others.

- Matthijs