Sidekiq good practices

It seems that writing Sidekiq’s workers it’s quite an effortless task. Still, during my eight-year journey with this great library, I experienced too many issues caused by poorly designed background jobs to believe that it’s an easy thing to do.

For sure, a badly designed background job can lead to the following issues:

  • Exhausting the API calls limit due to performing the same call too many times
  • Sending the same e-mail to the same recipient multiple times
  • Creating classes that are hard to test and debug

and probably many more. This article is my list of good practices that help me write better classes and design more reliable systems that heavily depend on the background job processing.

Keep the parameters primitive

When I build the worker and decide what params I should pass to it, I always follow the same rule: if the value can change between the invocation of the worker and the execution, don’t pass it. That’s why I don’t pass any objects as params.

This rule is not that obvious when talking about simpler values but let’s consider the following case: given we have a job that sends an e-mail to the user and is delayed by few days:

class SendEmailToUserJob
  include Sidekiq::Worker
 
  def perform(email)
    SomeMailer.send_email(email).deliver_now
  end
end

What can go wrong here? During those few days, the user can change his e-mail, and the action won’t succeed. Instead of passing the email, we should pass the reference to the object that is holding the e-mail address:

class SendEmailToUserJob
  include Sidekiq::Worker
 
  def perform(user_id)
    user = User.find(user_id)
    SomeMailer.send_email(user.email).deliver_now
  end
end

With this simple change, we would ensure that system would send the email to the correct address. The same rule applies to the scenario when the application can delete a user before executing the job, and then we would send an e-mail to someone who probably doesn’t want to hear from us anymore.

Few smaller workers are better than one big

Background job is a perfect way to perform time-extensive tasks without worrying about blocking the user thread in the application. However, when updating many things in worker, you can run into some troubles. Given you want to pull some information from the collected websites:

class CollectDataFromWebsitesJob
  include Sidekiq::Worker
 
  def perform
    Website.find_each do |website|
      page_data = Scraper.call(website.url)
      website.update(title: page_data.title)
    end
  end
end

The code is straightforward, and it works, but such design can cause the following issues:

  • If you want to process many websites, you can break the job by the deployment, and you would have to start over again.
  • If scraping of one of the websites raised an error, the worker would start again processing all jobs, even those already processed.
  • You can’t queue scraping of only one website if you would have to do this.
  • You can process only one website at a time

Such design is far from being efficient and reliable. However, the fix is simple, and all you have to do is to split the job into smaller workers that would process only one website per job:

class CollectDataFromWebsitesJob
  include Sidekiq::Worker
 
  def perform
    Website.find_each do |website|
      CollectDataFromWebsiteJob.perform_async(website.id)
    end
  end
end

With the above design, you fixed all the issues I mentioned before because:

  • The deployment won’t break the process as the primary job is rapid, and in case of Sidekiq’s shutdown, the jobs will be delayed, not interrupted.
  • Failure of one job does not affect other jobs so that you can retry just a single scraping
  • You can quickly queue scraping of the given website by hand
  • You can increase the performance by processing many jobs at the same time

Suppose you would need more control over the process and tell when all jobs are finished. In that case, you should use batches - they are also available as free extensions from other developers, but from my experience, the original implementation worked the best.

Keep the logic simple

Background job worker is just a class like any other class in your application. The main difference is that it is executed in the background. You should not overpack it with logic and stick to the single responsibility rule if possible and reasonable.

To keep the worker logic simple, I try to stick to the following patterns:

Use worker to queue many other, smaller workers:

class CollectDataFromWebsitesJob
  include Sidekiq::Worker
 
  def perform
    Website.find_each do |website|
      CollectDataFromWebsiteJob.perform_async(website.id)
    end
  end
end

Pull the data and send an e-mail

class NotifyAdministratorAboutPaymentJob
  include Sidekiq::Worker
 
  def perform(user_id, payment_id)
    user = User.find(user_id)
    payment = user.payments.find(payment_id)
 
    return unless payment.paid?
 
    NotificationMailer.notify_about_payment(payment.title).deliver_later
  end
end

Execute single service or command with additional policy logic

class SuspendUserJob
  include Sidekiq::Worker
 
  def perform(user_id)
    user = User.find(user_id)
  
    return if user.inactive? || user.in_trial?
 
    Users::SuspensionService.call(user)
  end
end

Of course, it’s not always possible to keep it clean as much as I described here, but that way of thinking helps me write more straightforward workers easily testable and readable for other developers.

Know how to debug the worker effectively

This point would be evident for many of you, but developers new to the Sidekiq sometimes have problems debugging the background job. To avoid issues with the debugging, you can stick to the following rules:

  • Use pry and binding.pry to debug the code when triggering the worker
  • Remember to start only one worker when using the debugger as otherwise, your console would become messy with multiple debugging points for various workers
  • Debug the worker in the console by using WorkerClass.new.perform instead of WorkerClass.perform_async

The Sidekiq monitoring dashboard is also your friend when debugging the background processes.

Keep the jobs organized

It’s always good to have the code well-organized, even if the application you are working on is small. I would say that background job workers are kind of design patterns, and we can define some rules that would help us to stay organized:

  • Keep all worker classes in one main directory, for example, app/jobs, and then structurize them inside by creating directories as namespaces, for example, app/jobs/users/suspend_job.rb
  • Use the same pattern for all worker names - {action}_job.rb where action should be meaningful and reflect the process that would be performed in the background

Being consistent in naming classes would always help other developers to familiarize themselves with the application code quicker.

Share connections to Redis

Suppose you are using Redis not only for Sidekiq. In that case, it is good to reuse the connections unless you use a separate Redis instance for Sidekiq and a separate instance for other features.

You can achieve that with Sidekiq.redis. Just simply wrap the call to Redis to reuse the connection pool:

Sidekiq.redis do |client|
  client.set('key', 'value')
end

Use prepend instead of inheritance

If you are not familiarized with prepend, you can read this quick guide about extending Ruby code using this helpful mechanism.

Sometimes, we have to wrap the worker logic with another logic that would be executed “around” the main call. With inheritance, you can do it like that:

class BaseJob
  include Sidekiq::Worker
 
  def perform(...)
    begin
      perform_worker(...)
    rescue Timeout::Error
      # do something
    end
  end
end

we can now use the base job for any other job we would like to extend with the timeout wrapper:

class CollectDataFromWebsiteJob < BaseJob
  private

  def perform_worker(user_id)
    website = Website.find(website_id)
 
    # It can raise Timeout::Error
    page_data = Scraper.call(website.url)
    website.update(title: page_data.title)
  end
end

This code works but has some serious disadvantages:

  • You can’t simply wrap the worker with the next wrapper as you have to extend the base job, and the class will grow
  • The extended worker has to change its structure, and testing becomes more problematic
  • You can't extend another job that already inherits from another class

With prepend, you can make the wrapper more reusable without changing the logic of the worker you would like to extend:

module TimeoutWrapper
  def perform(...)
    begin
      super
    rescue Timeout::Error
      # do something
    end
  end
end

You can now use just one line to apply the wrapper:

class CollectDataFromWebsiteJob
  include Sidekiq::Worker
  prepend TimeoutWrapper

  def perform(user_id)
    website = Website.find(website_id)
 
    # It can raise Timeout::Error
    page_data = Scraper.call(website.url)
    website.update(title: page_data.title)
  end
end

With such an approach, you can use many wrappers without worrying about the growing classes.

Happy background job processing!

Writing job workers can sometimes be tricky as it seems easy, but there are many details to be aware of. I hope you find those good practices useful.

If you would explore Sidekiq more, check out the deep-dive article about Sidekiq, where you will learn about how it’s working under the hood.