Asynchronous processing with Sidekiq gem in Rails
April 11, 2014
Introduction to Sidekiq
Sidekiq gem is used to move long running jobs to background for asynchronous processing.
It is more efficient, with respect to memory usage, than delayed_job and Resque as it uses threads instead of forks.
Need of background process
For example, in a mailing application you need to send emails to a large list of recipients. It will take large time to process with the large list of recipients, which is too long for a User to wait for a response after clicking a link. So, it’s better to move the long running task as the background process by using Sidekiq gem.
Integration of Sidekiq in a Rails application
Steps to integrate Sidekiq gem in a Rails Application :
Step#1 – Add the gem to the Gemfile to use Sidekiq with Active Record
You can also view the web interface to manage the background jobs. It isn’t included by default with Sidekiq. To use the web interface we need to add couple of gems to the Gemfile like ‘sinatra‘ and ‘slim‘
gem 'sidekiq', '2.9.0' gem 'sinatra', '1.3.6', require: false gem 'slim', '1.3.6'
Run “bundle install” to install the sidekiq gem
Step#2 – Install Redis Server to manage its job queue
Redis is an open-source, advanced key-value pair store. It is often referred to as a data structure server.
Sidekiq uses Redis to manage its job queue.
If you’re running OS X follow the steps as below:
Logged in as a Admin user or add sudo prefix on terminal
$ brew install redis
After installation completed starts the Redis server up with the following command
$ redis-server /usr/local/etc/redis.conf
if you’re running CentOS 5.6 follow the steps as below:
yum install make gcc wget telnet wget http://redis.googlecode.com/files/redis-2.2.12.tar.gz tar -xf redis-2.2.12.tar.gz cd redis-2.2.12 make && make install
Change the default redis.conf file to daemonize it.
mkdir /etc/redis /var/lib/redis sed -e "s/^daemonize no$/daemonize yes/" -e "s/^dir \.\//dir \/var\/lib\/redis\//" -e "s/^loglevel debug$/loglevel notice/" -e "s/^logfile stdout$/logfile \/var\/log\/redis.log/" redis.conf > /etc/redis/redis.conf
To make management easy, use an init script by using sed to update it.
Wget https://raw.github.com/gist/257849/9f1e627e0b7dbe68882fa2b7bdb1b2b263522004/redis-server sed -i "s/usr\/local\/sbin\/redis/usr\/local\/bin\/redis/" redis-server chmod u+x redis-server mv redis-server /etc/init.d /sbin/chkconfig --add redis-server /sbin/chkconfig --level 345 redis-server on /sbin/service redis-server start
Step#3 – Start up the jobs process
There are couple of ways to do this,
- If application is in development mode, we should use the below rake task file instead.
bundle exec sidekiq
If application is in production mode, then it is preferred to daemonizes the job process and allows multiple background processes to be spawned. To use this, follow the following step:
1. Add gem “daemon-spawn”, “0.4.2” to your Gemfile
2. Run bundle install
3. Run the below command to run it in background
bundle exec sidekiq -d -L log/delayed_job.log
4. You have to pass path of log file while using above command.
Step#4 – Add task to run in background
We need to add the background work in a separate class and store it in a new directory, let’s say app/background_jobs and ensures that it’s auto-loaded by the application.
We need to include the Sidekiq::Worker module.
class NotifyUsers include Sidekiq::Worker def perform User.find_each(is_subscribed: true) do |user| NewsMailer.newsletter_mail(user).deliver end end end
This class must have a perform method. Add the code that need be to run in the background. Then call NotifyUsers.perform_async method inside the controller which will add the job to Redis and then it will call function perform asynchronously.
Before background job in UsersController
class UsersController < ApplicationController def send_email User.find_each(is_subscribed: true) do |user| NewsMailer.newsletter_mail(user).deliver flash[:notice] = "Mail delivered" redirect_to root_path end end end
After implementation of UsersController after adding to background job
class UsersController < ApplicationController def send_email NotifyUsers.perform_async flash[:notice] = "Mail is being delivered" redirect_to root_path end end end
- No need to wait for a response after clicking a link to do a big stuff
- Sidekiq has client-side middleware which runs before a job is inserted into Redis
- It has server-side middleware which runs before the job is being processed. This middleware retry jobs, log them and handle exceptions
Sidekiq handles the processing of the jobs after they’re pulled from Redis and multithreaded behavior.
It is also providing a web interface there are a couple of gems that we need to add to the gem file too. To visit the web interface just go to the /sidekiq path
It tells us how many jobs have been processed, the number of failures, the currently-active workers and the number of jobs in queues.
Like this blog? I’d love to hear about your thoughts on this. Thanks for sharing your comments.