How Sidekiq really works - Paweł Dąbrowski

Have you ever wondered how Sidekiq works and what happen from the moment you queue the worker and the moment it is executed? I did. I checked step by step the whole process. Meanwhile I explored many useful Redis features and saw how, one of my favourite tools that I work with for many years, is designed under the hood.

Let's just begin with something that every Rails developer that is using Sidekiq is familiarized with - a simple worker:

class MySimpleWorker
  include Sidekiq::Worker

  def perform(text)
    puts text
  end
end

So the question is, what happen between the moment of execution the below line and printing the "I was performed" text in the console:

MySimpleWorker.perform_async("I was performed!")

Adding job to the queue

When you call a worker with the perform_async method, the worker is queued and executed asynchronously instead of being executed immediately after calling it. If you would like to execute the worker immediately, then you should call MySimpleWorker.new.perform("I was performed!") - the code will be executed in the console without hitting the Sidekiq.

Forming the options hash

After you call the perform_async method on a class that is extended with Sidekiq::Worker module, a hash with options is formed. The hash consists of arguments passed to the worker and the class name:

{"args" => ["I was performed"], "class" => MySimpleWorker}

if you would chain the call with the set method like this:

MySimpleWorker.set(queue: 'normal').perform_async("I was performed!")

then the hash from the set method will be also merged to the queue options and our options hash will look like the following:

{ "args" => ["I was performed"], "class" => MySimpleWorker, "queue" => "normal" }

Options assignment

After the item is queued, a validation is performed to check if a valid queue was given. Besides it, default options are assigned also - it means that the default queue will be assigned unless you specified the queue name in the worker definition or used the set method to set the queue when calling the worker.

When validation is successful, the following attributes are assigned to the job options:

class - a string that represents the name of the worker class
queue - a string that represents the name of the queue in which the job should be executed
jid - a string that represents a unique id for a given job. It's generated by SecureRandom.hex(12) call
created_at - the time when the job was created. It's set to Time.now.to_f if wasn't specified in the options of the worker

Calling middleware

Before the job is pushed to the Redis, Sidekiq is executing the middleware that was configured. Middleware is the code configured to be executed before or after a message is processed. An example of middleware can be Sidekiq Uniq jobs that is verifying if a job with the same arguments is not already processed.

The middleware classes receive the same arguments that were assigned in the previous step including jid, args, class, and created_at.

Pushing data to the redis

When middleware didn't reject the job, it is the time to push the data into the Redis so it can be processed. Two actions are now possible depending on the given case. I will now describe only the case when you want to process the worker asynchronously as soon as possible.

The second case is when you want to perform the worker in at a given moment in the future. You can achieve this by calling the following code:

MySimpleWorker.perform_in(5.minutes, "I was performed!")

For now, let's assume that you want to perform the job as soon as possible. The job payload (all arguments required to perform the job successfully, assigned in previous stages) is dumped into the JSON format. Two Redis methods are called: sadd and lpush.

Performing sadd

Imagine that there is an array of strings, with a unique name that you can refer to, and this array is stored in Redis:

["default", "high"]

If you want to add the next element to this array, Redis verifies first if the given element is already there. If the element already exists then the addition is ignored, otherwise, the element is added at the end of the array.

In Ruby you could replicate this behavior by creating the following class:

class Redis
  def initialize
    @lists = {}
  end

  def sadd(name, item)
    @lists[name] ||= []
    return @lists[name] if @lists[name].include?(item)

    @lists[name] << item
	  @lists[name]
  end
end

conn = Redis.new
conn.sadd("queues", "default") # => ["default"]
conn.sadd("queues", "default") # => ["default"]
conn.sadd("queues", "high") # => ["default", "high"]

Sidekiq is using the sadd method on the queues list to add a queue name of the job that it is currently processing:

conn.sadd("queues", queue)

Performing lpush

Imagine that you have an array of strings and each time you add an element to it, it's pushed at the beginning. That's how the lpush command works for in Redis.

In Ruby you could replicate this behavior by creating the following class:

class Redis
  def initialize
    @lists = {}
  end

  def lpush(name, *items)
    @lists[name] ||= []
    
    items.each do |item|
      @lists[name].unshift(item)
    end
  end
end

Sidekiq is using lpush method to push job to the given queue:

conn.lpush("queue:#{queue}", to_push)

Summary

When the call is successful and the job was queued, you receive the jid value as a return value. This id is now unique identification for the job you have just pushed to the queue. The job is now queued and if you have Sidekiq running using the bundle exec sidekiq command, it will pick it up and process it. How the process of picking up a job looks like, I will describe in the paragraph below.

Picking job from the queue

I mentioned above that your job will be picked as soon as you will execute the bundle exec sidekiq command. But what exactly is triggered then? Let's see.

Creating one instance of CLI object

The very first thing that is invoked when the command is executed is Sidekiq::CLI.instance method. The CLI class is including the Singleton module which means that by calling .instance on a class, we ensure that only one instance will be created. You can read more about it on the official documentation: https://ruby-doc.org/stdlib-2.5.1/libdoc/singleton/rdoc/Singleton.html

You can test how the Singleton pattern is working by using this simple snippet:

class Test
  include Singleton
end

Test.instance == Test.instance # => true

it means that even if you call bundle exec sidekiq twice in separated shell windows, the CLI instance will be the same.

Parsing the CLI

The second thing that Sidekiq is doing after executing the start up command is parsing. There are two steps of parsing executed one by one:

Options assignment

Options passed to bundle exec sidekiq command are parsed and assigned. So if you are passing config file as -C config/sidekiq.yml it will be detected and the config file location will be assigned for the future usage. If the file passed as a config won't be available, an error will be raised.

Default options are set as well if you don't provide the values when starting the Sidekiq. The default options set include concurrency and queues list.

Logger assignment

The logger is set on a debug level if the verbose option is specified. You can achieve this by calling the start up command either with -v flag or --verbose.

Validation

The last step in the parse phase is to validate if the value of concurrency and timeout options is not smaller than 0 and if Sidekiq has access to a Rails application or Ruby file. You can point Sidekiq to the Rails application config by using --require option.

Running the machine

This phase consists of multiple smaller steps that are needed to ensure that Sidekiq can perform its tasks fine. The following steps are performed:

If Sidekiq is executed in the development environment, the banner with kicking person is printed in the console
Information about licensing is printed
Upgrading to PRO version information is printed unless you are already using the PRO version
Error is raised if the Redis version is lower than 4
Error is raised if the Redis pool size is too small for Sidekiq (it needs the number of connections you set in concurrency option + 2)
Server middleware is touched so it's not lazily loaded by multiple threads
Information about applied client and server middleware is printed

The last step is to start the launcher which is a topic for the next paragraph!

Launching

If you are running Sidekiq in the development environment, the Starting processing, hit Ctrl-C to stop information is printed. Then the launcher is running.

The run method from the launcher does two things: start poller and manager.

Poller

A new thread called scheduler is called to take care of jobs that are enqueued to be performed in the future. There are two sets that are taken into the account: retry and schedule. Basically retry queue works the same as scheduled queue with one difference: the time of execution is set automatically depending on the retry number.

For each set, the zrangebyscore method from Redis is executed. The name is weird, so what actually it does? To understand better what it does, we have to take a step back to see how are pushed to the Redis jobs that suppose to be executed later not immediately. When a job should be executed in the future, Sidekiq is calling zadd method from Redis. This method adds an element to the list along with the score. The score in our case is the time when a given job should be executed. So if you call the following code:

MySimpleWorker.perform_in(3.minutes, "I was performed!")

then the score will be following:

Time.now.to_f + 3.minutes.to_f

So we have now some jobs and scores on the list and then we call zrangebyscore :

now = Time.now.to_f.to_s
jobs = conn.zrangebyscore(sorted_set, "-inf", now, limit: [0, 100])

It means that we want to pull all items from the sorted_set, which can be either schedule or retry queue, starting from any item in the past to now - this way we will pull all jobs that should be performed until now.

Each job is then passed to zrem method from Redis which removes the item from the sorted list with scores and pushed to Sidekiq just like you would call perform_async on the worker - the process we described while ago start from the beginning.

After all jobs are processed, the process sleeps for a while (the time for the process waits is called random pool interval) and then it's executed again.

Manager

Besides poller, there is a manager which takes care of processing jobs. Depending on the concurrency setting, it runs the defined number of workers. Each worker then picks one job from the specified queues using brpop method from Redis. This method receives a list of lists names and picks one element from the tail if the list is not empty. If there are no elements to pick, it blocks the connection.

When the job is picked, the following steps are performed:

job details are decoded and if a job has a wrong format, it is immediately added to the dead queue (it means it won't be automatically retried)
middleware is invoked
the job is performed just like you would call MySimpleWorker.new.perform

If the error is raised when a job is performed, logs are printed in the console along with the error backtrace.

Sidekiq dashboard

We just went through the complete process of queueing jobs and processing them but can't omit the dashboard as it is also an important part of the Sidekiq. The dashboard is built on the top of the Rack and it is just a simple application with a few routes defined where the data is displayed.

Views of the dashboard are just templates in the .erb format where Ruby code is rendered among HTML tags.

Summary

After reading the article you may think that the Sidekiq architecture is quite simple but in fact, the massive amount of work was done to make it a stable and reliable software for processing tons of workers.

After many years of working with Sidekiq, I still appreciate the way it works, the fact that it solves many problems, and improves the performance of the application. I hope you found it useful and fun to go through this guide.