Using Redis with Nodejs and MongoDB
In one of my previous articles, I talked about the basics of Redis. Today I am going to talk about how to use Redis as cache in a node js application with MongoDB as Database.
Prerequisites:-
Node js should be installed on your system
Redis cache should be installed on your the system
Use local or cloud atlas for mongoDB database.
Basic knowledge of express and CRUD operations with mongoose.
Here is a link to the Github repository of the whole project.
In this application, we are saving some data related to vehicles. Here is the schema of vehicle collection.
The app.js file has four routes
- ‘/vehicle’ (POST route): Using this route we save the data of a new vehicle in the database.
2. ‘/’ (GET route): Used to get all data in the database.
3. ‘/:vehicleType’ (GET route): Using this route we will send the vehicle type as a parameter and get a list of all vehicle of that vehicle type as response.
4. ‘/:vehicleType/:sno’ (GET route): Using this route we will send the ‘vehicletype’ and ‘serialno’ as parameters and get the details of that vehicle as response.
When we are adding new data or editing existing data in the database, the result of queries which were previously made might have changed. We should clear such data from the cache so that the updated data is sent to the user, not the old one. One way to do this may be to clear the whole cache. But a better way will be to clear only the data of the queries for the vehicle type in which data was edited or added. In the post route(‘/vehicle’) we are using clearCache( ) function for this purpose.
To enable us to group queries of a vehicle type together we have to use nested hash data structure. Here is a diagram illustrating how data will be structured in our cache.
In the above diagram, we can see query 1 and query 4 were made related to cars. Similarly, query 2 and 3 were related to trucks. Whenever one of these four queries are made again we can get the data directly from the cache instead of searching in MongoDB.
Now consider that a new vehicle of type ‘cars’ is added to the database. Then the result of query 1 and 4 might change. In most cases, it’s not possible to find the exact entries which will be effected in the cache when a new data is inserted in the database or existing data is edited in an efficient way. So we will clear all cached data related to the type ‘cars’.
We might not want to cache data of every query that is made to the database. Redis cache is pretty expensive so very large or less frequently queried data should not be cached. Like in the route ‘/’ we are getting all the data stored in our database.
To solve this issue we will create a property ‘useCache’ using which we will be able to specify if we want to cache those queries or not.
Now let's jump to how exactly we are going to implement the caching.
Only numbers and strings can be used as key and value in redis cache. So before storing key or values of other datatypes we need to stringify them.
Create a folder named ‘services’ in the root directory of your project and create cache.js inside it. In this, we will write all our caching logic.
Create a file vehicle.js inside models folder which will export the schema of vehicle to app.js.
Here is the directory structure of our application
Mongoose library is used to handle the interaction between node js and MongoDB. We will change some properties of mongoose in such a way that before sending the query to MongoDB we check our Redis cache to see if the query is already cached.
Don’t forget to start your Redis server by typing ‘redis-server’ in the terminal before you run your application.
Here are the npm libraries we will need to import and the code for connecting to Redis client.
Here is the useCache property I talked about earlier. When we will call the cache( ) function from app.js while making a query, ‘useCache’ will be set to true. Only then will the query will be cached. We also create the level 1 hash key in the cache( ) function which we will use later.
Before moving forward I would like to talk about the exec( ) function provided by Mongoose. Mongoose provides query functions like find( ), findOne( ) etc, which generate the query. exec( ) is used to execute that query and retrieve data from MongoDB. When we use async-await or promises(then( ) , catch( )) or callbacks, exec( ) is begin called in the background. So the simplest thing to do will be to manipulate the exec( ) function to check the Redis cache before MongoDB.
Here is how we are going to do that. The comments explain what each segment of code is doing.
In the end, we will create the clearCache( ) function about which we talked about earlier. It will take the level 1 hash key, i.e vehicleType as argument. We also need to import the cache.js in app.js as shown below.
The strategy for implementing cache will be different in different applications. In some applications, more levels of nesting might be required or some applications are better off without nesting at all. Strategy for how and when to clear cache will also vary accordingly.
Resources
- Github repo
- Redis basics
- Download Redis
