Girders Blog
Notes on building internet applications

How Queuing with Resque works

Oct 30, 2011

Resque is a fast, lightweight, and powerful message queuing system used to run Ruby jobs asynchronously (or in the background) from your on-line software for scalability and response. I needed to integrate software written in different languages and environments for processing, and this is my understanding of the implementation.

How Queuing with Redis works

Resque’s real power comes with the Redis “NoSQL” Key-Value store. While most other Key-Value stores use strings as keys and values, Redis can use hashes, lists, set, and sorted sets as values, and operate on them atomically. Resque leans on the Redis list datatype, with each queue name as a key, and a list as the value.

Jobs are en-queued (the Redis RPUSH command to push onto the right side of the list) on the list, and workers de-queue a job (LPOP to pop off the left side of the list) to process it. As these operations are atomic, queuers and workers do not have to worry about locking and synchronizing access. Data structures are not nested in Redis, and each element of the list (or set, hash, etc.) must be a string.

Redis is a very fast, in-memory dataset, and can persist to disk (configurable by time or number of operations), or save operations to a log file for recovery after a re-start, and supports master-slave replication.

Redis does not use SQL to inspect its data, instead having its own command set to read and process the keys. It provides a command-line interface, redis-cli, to interactively view and manipulate the dataset. Here is a simple way to operate on a list in the CLI:

$ redis-cli
redis> rpush mylist "hello, redis"  # <= Adds the value to the right side of the list/queue
(integer) 1

redis> keys mylist*                 # <= Returns the matching key names
1) "mylist"

redis> type mylist                  # <= Returns the datatype of the value of this key
list

redis> lrange mylist 0 10           # <= Returns a elements 0 through 10 from the list/queue
1) "hello, redis"

redis> llen mylist                  # <= Returns the number of elements in the list/queue
(integer) 1

redis> lpop mylist                  # <= Pops the leftmost element from the list/queue
"hello, redis"

redis> lrange mylist 0 10
(empty list or set)

How Queuing with Resque works

Resque stores a job queue in a redis list named “resque:queue:name”, and each element is the list is a hash serialized as a JSON string. Redis also has its own management structures, including a “failed” job list.

$ redis-cli
redis> keys * 
1) "resque:stat:processed"          # <= Number of jobss successfully processed
2) "resque:failed"                  # <= This is the failed job list (not a queue)
3) "resque:queue:myqueue"           # <= This is your work queue!
4) "resque:queues"                  # <= The "Set" of work queues
5) "resque:stat:failed"             # <= The number of failed jobs
6) "resque:workers"                 # <= Set of workers
7) "resque:worker:host.example.com:79163:myqueue:started" # <= Count of jobs processed by worker
8) "resque:processed:host.example.com:79163:myqueue:started" # <= Timestamp of worker start

redis> get resque:stat:processed    # <= Returns the count of processed jobs 
"9"

redis> smembers resque:queues       # <= Prints the members of the set of queues
1) "myqueue"

redis> smembers resque:workers      # <= Prints the set of workers
1) "host.example.com:79163:myqueue"

Resque namespaces its data within redis with the “resque:” prefix, so it can be shared with other users.

Designed to work with Ruby on Rails, Resque jobs are submitted and processed like the following boilerplate:

class MyModel
  @queue = :myqueue                 # <= jobs will be placed in this queue name

  # call to queue processing in Resque until later
  def defer(*args)
     Resque.enqueue(MyModel, self.id, *args)
  end

  # Resque calls this method with the additional arguments. Must be named #process
  def self.process(id,*args)
    model = MyModel.find(id)
    # Do something here, raise an exception to send job to failure list
    raise "Oh Noes!" if failed?
  end
end

This does not serialize an object to the queue, instead it saved the (ActiveRecord) model name and record id which is re-instantiated from the database later. The additional arguments are saved in an array to call later. To keep the operation light, do not pass a lot of data to the job. Instead pass references to other records, files, etc.

Each job in Resque is a hash serialized as a JSON string (remember data structures can not be nested in Redis) of the format:

{"class":"MyModel", "args":[123, "arg1", "arg2", ...]}

When the job is popped from the queue, Resque instantiates the ActiveRecord object and calls its process method, passing the additional parameters. Functionally, the worker code behaves something like this (simplified):

klass, args = Rescue.reserve(queue_name)
model = klass.process(*args)

If processing raises an exception, the job and relevant information is placed on the failed list of the JSON format (as a string):

{ "failed_at":"2011/08/22 15:55:16 EDT",
  "payload":{"class":"MyModel","args":[123,"arg1","arg2"]},
  "exception":"NameError",
  "error":"uninitialized constant SalsaJob",
  "backtrace":[...],
  "worker":"host.example.com:56870:myqueue",
  "queue":"myqueue",
  "retried_at":"2011/08/22 16:07:50" }

A failed job can be retried (only once though) through the web interface started with the resque-web command.

Using Resque without Rails

Resque runs out of the box on Ruby on Rails. If you have a ruby application not in Rails, you can still run the Resque workers with the rake command by adding

require 'resque/tasks'

to your Rakefile.

Calling external systems with Resque

There are ports of Resque to other languages such as python, C, Java, .NET, node, PHP and Clojure. If your external system is written in one of these languages, then you can start workers listening to their queues. Since you are not talking to a ruby class with arguments, you can set up a placeholder class with the proper queue name. This will allow Resque plugins to fire on enqueue. (I assume the other libraries work the same way as the original, though some of the languages are not object-oriented—I have not verified them.)

class ExternalClass
  @queue = :external_class
end

Rescue.enqueue(ExternalClass, *args)

That class does not have to implement process() since that will be called in the real class.

If you need to call an external system to perform the task, either that system can be written to accept Resque-style queuing requests (hash of “class” and “args”), or you can push the expeted format directly to the queue

Resque.redis.rpush("queue:#{queue_name}", args.to_json)

The format does not have to be json, but has to be a string of a format the external system expects. You can not use the Resque workers

Calling the Ruby Resque from an external system

Maybe your external system needs to trigger a job to run on your Ruby Resque system, but can does not have a Resque implementation. You can drop your work (as a JSON hash of “class” and “args”) on the raw Redis list/queue yourself from the Redis library or the command line

redis-cli rpush "resque:queue:myqueue" '{"class":"MyModel","args":["arg1"]}'

Epilogue

I am new to Redis and Resque and wanted to dig into the Redis data structures used by Resque, and learned it more in depth while writing this. After understanding how it all fits together, I can now write some integration code! I hope you found this useful, and not too incorrect.