Posted on Aug 18 Background Job processing is a technique for running tasks that can take a long time to complete in a separate process from the main application server. This allows the main application server to continue to handle requests from users while the background jobs are running.There are many reasons why you might want to use background job processing in your Node.js server application. For example, you might use it to:There are a number of different ways to implement background job processing in Node.js. One popular approach is to use a message queue. A message queue is a first-in-first-out (FIFO) system that allows you to send messages between different processes. When you want to run a background job, you can add a message to the queue. The message will then be processed by a separate worker process, which is a process that is dedicated to running background jobs.In this article, I will share how to use the BullMQ npm package together with a Redis server to implement background job processing in Node.js. I'll also cover how to deploy your background job application to Heroku.To get started, you will need to have the following installed:Once you have these installed, you can follow the instructions in this article to create your background job application.You're welcome to skip if you already know or use Heroku.Heroku is a cloud platform that makes it easy to deploy and scale web applications. It provides a number of features that make it ideal for deploying background job applications, including:Automatic scaling: Heroku will automatically scale your application up or down based on demand, so you don't have to worry about managing your own infrastructure.Managed services: Heroku provides a number of managed services, such as Redis and Postgres, so you don't have to worry about provisioning and managing these services yourself.To get started with Heroku, you will need to create an account and install the Heroku CLI. Once you have done that, you can create a new Heroku app and deploy your application to it.The benefit of using Heroku for background job processing is that it is easily scalable, naturally fits long-running processes, and makes it easy to colocate your background job application with your main application server.BullMQ is a message queue library for Node.js. It allows you to send and receive messages between different processes. Redis is an in-memory data store that BullMQ uses as a message queue.BullMQ and Redis work together to provide a reliable and scalable way to process background jobs. BullMQ handles the queuing and routing of messages, while Redis stores the messages and provides a high-performance, consistent way to access them.Some key features of BullMQ include:To set up a Heroku application with Redis, you will need to:Here are the commands you can use to do this:The first command creates a new Heroku app named my-app. The second command provisions a Redis instance on the Hobby Dev plan. The third command creates the Heroku app infrastructure file Procfile, which tells Heroku to provision a background job processing resource and to run the node worker.js command for it.The worker.js file is where you will put your code for processing background jobs. We're getting to it next!The worker is the process that will consume work from the queue and run the background jobs. In Heroku, and generally speaking, it's a separate Node.js runtime process and as such it is entirely unrelated in terms of deployment from that of your main web API Node.js runtime, if you have one.It's also generally considered to be a long-running process, meaning it will run indefinitely until it is stopped. If there's no work, it simply idles. It will connect to the Redis instance and listen for jobs to be added to the queue. When a job is added to the queue, the worker will process the job and then mark it as complete.Setting up the BullMQ worker is as simple as creating a new file, worker.js, and adding the following code:We can hook into events in the worker lifecycle to perform actions when certain events occur. For example, we can hook into the completed event to perform an action when a job is completed. We can also hook into the failed event to perform an action when a job fails.Add the following to the worker.js file:Tip: Avoid try/catch for error handling. You don't need to customize error handling within the worker handler function. BullMQ automatically catches thrown errors from the function. Next, it retries failed jobs, and moves them to a dead letter queue after a certain number of retries.Node.js, being a single-threaded runtime, can only utilize a single CPU core. This means that if you have a multi-core CPU host assigned to the worker process then you'll constantly under-utilize your hardware resources. Instead, you can utilize all CPU cores by running multiple instances of the worker. This is often referred to as clustering and it's a common pattern in Node.js applications.We can use a small wrapper for Node.js's built-in clustering capabilities with the throng npm package in order to launch a separate Node.js runtime process for each CPU.We will change the worker.js file to accompany for throng:Your background job workers, while deployed separately from your main web APIs, will still need to access your application's services and dependencies. For example, you may want to access your database, or your file storage service, to which you'll need to pass the necessary configuration, credentials, and potentially other domain logic.In order to do this, we can use a dependency injection container to inject dependencies into the worker. We can use the Awilix npm package to do this.The pattern I suggest is as follows:The worker is co-located with the Node.js API service/microservice source code it is related to. This way, it can easily access domain logic via high-level abstraction APIs (the services, i.e: FileStorageService, DatabaseService, etc).The worker is instantiated with a dependency injection layer that is allows the worker to request access to any of the necessary dependencies, just as if it was another HTTP API route handler in the main service.This will end up looking something like this:In the above code example, we initialize configuration, and database connection details, then pass these to the dependency injection layer to make them available to DI consumers. As you notice, we wrap the worker code in a Factory Function to make the dependency injection container available to the worker.To make this guide complete, we'll also review shortly the BullMQ client which is responsible to add jobs to the queue for the worker to consume.It is in fact, as simple as:Tip: BullMQ will default to schedule jobs with a retry attempts value set to 0 which means it won't retry failed jobs. You can override this by setting the attempts value to a number greater than 1. More on this, jobs are retried immediately unless a delay is specified. You can override this by setting the backoff value be an object with type exponential and a delay specified in milliseconds.To accommodate for job retries, and other queue house-keeping configuration we can further customize the job queue configuration as follows for when we schedule jobs:Don't forget you have to actually deploy your worker to Heroku. You can do this by running the following command:If you are interested in learning more about BullMQ and Redis, I recommend checking out the following resources:You'd also want to look into alternative implementations for queue with RabbitMQ and the amqplib npm package.Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Nathaniel Arfin - May 25 Salah Eddine Lalami - Jun 8 Nilesh Prasad - Jun 3 Željko Šević - May 21 Once suspended, lirantal will not be able to comment or publish posts until their suspension is removed. Once unsuspended, lirantal will be able to comment and publish posts again. Once unpublished, all posts by lirantal will become hidden and only accessible to themselves. If lirantal is not suspended, they can still re-publish their posts from their dashboard. Note: Once unpublished, this post will become invisible to the public and only accessible to Liran Tal. They can still re-publish the post if they are not suspended. Thanks for keeping DEV Community safe. Here is what you can do to flag lirantal: lirantal consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy. Unflagging lirantal will restore default visibility to their posts. DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2023. We're a place where coders share, stay up-to-date and grow their careers.