Search for a command to run...
Challenges, Benefits and Best Practices with NodeJs Example.
I am a seasoned Software Engineer with over 8 years of diverse experience, primarily focusing on Backend Software Engineering and Complex systems.
No comments yet. Be the first to comment.
The Invisible Architecture Behind All Systems
Keeping the Party Organised in complex systems
The importance of Fault tolerance and how to achieve it in distributed systems
Synchronization's Role in Upholding Consistency in Complex Systems
Considerations and Importance in database engineering
Modulus sharding is a method used in software engineering to spread data across various servers in a way that improves performance, scalability, and reliability. The method involves dividing data into smaller, easier-to-handle pieces and putting those pieces on different computers, or "shards." This article will give a detailed overview of modulus sharding, including its benefits, challenges, and best practices.
Modulus sharding involves dividing data into various shards based on several servers, or "nodes," which has already been decided. Each shard is given a unique identifier, which is usually an integer value, and the data is spread across the shards based on how much of the shard ID is left after dividing it by the number of nodes. For example, if we have three nodes and six shards, each node would be responsible for two shards, with the shards distributed as follows:
Node 1: Shards 1 and 4
Node 2: Shards 2 and 5
Node 3: Shards 3 and 6
This distribution makes sure that each shard is spread out evenly across the nodes, which helps to improve speed and scalability. Also, because each node is in charge of a subset of the shards, the system as a whole is more resistant to breakdowns. If any one node fails, it won't affect the availability of the whole system.
There are several benefits to using modulus sharding in software engineering:
Modulus sharding is a technique that can help to enhance the performance of a system by dividing data across numerous nodes. This allows each node to manage a smaller portion of the data, which in turn leads to improved performance. This can help to improve overall response times by reducing the burden that is placed on individual nodes.
Because more nodes can be added to the system as needed to accommodate expanding volumes of data, modulus sharding can also help to boost the scalability of a system. This is because of how the data is stored. In addition to this, the fact that each node is accountable for a part of the data makes it much simpler to add additional nodes without negatively affecting the performance of the nodes that are already present.
A system's fault tolerance can be improved by utilising modulus sharding because it allows for the failure of any individual node without having an effect on the overall system's availability. It is also possible to recover data that has been lost because it is distributed among numerous nodes, which makes it possible to reassemble the data using the nodes that are still available.
While modulus sharding can provide significant benefits, there are also several challenges to consider when implementing the technique:
When dealing with enormous amounts of data, the process of distributing the data among multiple shards can become very complicated. It is essential to give careful consideration to the distribution algorithm to guarantee that the data is dispersed uniformly across all of the shards.
If the system continues to expand and undergoes modifications, it is possible that rebalancing the shards may become essential to guarantee that the data will be dispersed uniformly throughout the nodes. This can be a difficult process because it involves moving data between nodes without affecting the system's availability.
It is possible that queries that span many shards would run more slowly than queries that just require data from a single shard because data is dispersed across multiple nodes. It is essential to give careful consideration to the design of queries to reduce the negative effects of shard distribution on the performance of queries.
As an example(PoC); We'll be writing a code which set up a sharded MongoDB cluster using four MongoDB shards and uses an Express server to expose a REST API for creating and retrieving user objects. The user objects are stored in the MongoDB shards and are distributed across the shards using a simple shard key based on the user ID.
Firstly, we'll import the required dependencies:
// Import the required packages
import express, { Request, Response } from 'express';
import mongoose from 'mongoose';
Creating an instance of the Express app:
// Create an instance of the Express application
const app = express();
Defining an interface/schema for the User object:
// Define the User interface that extends the mongoose.Document interface
interface User extends mongoose.Document {
id: number;
name: string;
}
// Define the schema
const userSchema = new mongoose.Schema({
id: Number,
name: String,
});
Initializing an empty array to store connections to each shard:
const shards: mongoose.Connection[] = [];
Defining a function to connect to a shard:
const connectToShard = async (shardNumber: number) => {
const shard = mongoose.createConnection(`mongodb://localhost/users${shardNumber}`);
await shard.once('open', () => {
console.log(`Connected to shard ${shardNumber}`);
});
shards.push(shard);
};
Connecting to each shard:
// Connect to all four MongoDB shards
for (let i = 0; i < 4; i++) {
connectToShard(i);
}
Defining a function to get the shard for a user:
// Define a function to get the MongoDB shard for a given user ID
const getUserShard = (userId: number): mongoose.Connection => {
const shardIndex = userId % shards.length;
return shards[shardIndex];
};
Defining a function to get a UserModel for a specific shard:
// Define a function to get the UserModel for a given shard
const UserModel = (shardIndex: number): mongoose.Model<User> => {
const shard = shards[shardIndex];
return shard.model<User>('User', userSchema);
};
Defining a route to get a user by ID:
// Define a route to get a user by ID
app.get('/users/:id', async (req: Request, res: Response) => {
const userId = parseInt(req.params.id);
// Get the MongoDB shard for the user ID
const shardIndex = userId % shards.length;
// Get the UserModel for the shard
const UserModelForShard = UserModel(shardIndex);
const user = await UserModelForShard.findOne({ id: userId });
if (!user) {
res.status(404).send('User not found');
return;
}
res.send(user);
});
The logic behind const shardIndex = userId % shards.length; is to determine which shard a given user's data should be stored on based on their user ID.
In this case, the userId is used to calculate a modulus value (%) based on the length of the shards array. The modulus operation returns the remainder of dividing the userId by the shards.length.
The resulting modulus value is then used as an index to access the corresponding shard in the shards array. This ensures that each user's data is stored on a specific shard based on their user ID, while also distributing the data evenly across all available shards for horizontal scaling.
Defining a route to create a new user:
app.post('/users', async (req: Request, res: Response) => {
const user: User = req.body;
// Get the MongoDB shard for the user ID
const shardIndex = user.id % shards.length;
// Get the UserModel for the shard
const UserModelForShard = UserModel(shardIndex);
const createdUser = await UserModelForShard.create(user);
res.send(createdUser);
});
Starting the Express server:
app.listen(3000, () => {
console.log('Server running on port 3000');
});
The code demonstrates how to connect to a MongoDB shard, how to create a user schema and model, and how to query and create user objects using the appropriate shard.
import express, { Request, Response } from 'express';
import mongoose from 'mongoose';
const app = express();
interface User extends mongoose.Document {
id: number;
name: string;
}
const userSchema = new mongoose.Schema({
id: Number,
name: String,
});
const shards: mongoose.Connection[] = [];
// Define a function to connect to a MongoDB shard
const connectToShard = async (shardNumber: number) => {
const shard = mongoose.createConnection(`mongodb://localhost/users${shardNumber}`);
await shard.once('open', () => {
console.log(`Connected to shard ${shardNumber}`);
});
shards.push(shard);
};
// Connect to all four MongoDB shards
for (let i = 0; i < 4; i++) {
connectToShard(i);
}
// Define a function to get the MongoDB shard for a given user ID
const getUserShard = (userId: number): mongoose.Connection => {
const shardIndex = userId % shards.length;
return shards[shardIndex];
};
// Define a function to get the UserModel for a given shard
const UserModel = (shardIndex: number): mongoose.Model<User> => {
const shard = shards[shardIndex];
return shard.model<User>('User', userSchema);
};
// Define a route to get a user by ID
app.get('/users/:id', async (req: Request, res: Response) => {
const userId = parseInt(req.params.id);
// Get the MongoDB shard for the user ID
const shardIndex = userId % shards.length;
// Get the UserModel for the shard
const UserModelForShard = UserModel(shardIndex);
const user = await UserModelForShard.findOne({ id: userId });
if (!user) {
res.status(404).send('User not found');
return;
}
res.send(user);
});
app.post('/users', async (req: Request, res: Response) => {
const user: User = req.body;
// Get the MongoDB shard for the user ID
const shardIndex = user.id % shards.length;
// Get the UserModel for the shard
const UserModelForShard = UserModel(shardIndex);
const createdUser = await UserModelForShard.create(user);
res.send(createdUser);
});
app.listen(3000, () => {
console.log('Server running on port 3000');
});
To ensure that modulus sharding is implemented effectively, there are several best practices that software engineers should follow:
Modulus sharding should be designed to handle future growth in terms of both the amount of data and the number of nodes. It's important to think carefully about the distribution algorithm and the way shards are rebalanced to make sure they can grow well.
It is important to keep an eye on important metrics like query response times, shard distribution, and node health to make sure the system is working well. This can help figure out if there are any problems or speed bottlenecks with certain nodes or shards.
When making the sharding plan, it's important to think about how the data is accessed. If the same data is frequently accessed at the same time, it may be best to keep it on the same shard to speed up queries.
Consistent hashing is a way to spread data across nodes so that shards don't have to be rebalanced as often. This can help make the system more scalable and lessen the effect of adding or taking away nodes.
It is important to set up data replication across various nodes to improve fault tolerance. This can help make sure that data is still accessible if a node fails.
Before putting a sharded system into production, it's important to test and validate it fully. This can help find problems or performance bottlenecks before they affect end users.
Modulus sharding is a powerful method that can help software systems run faster, grow, and handle errors better. But it's important to think carefully about the distribution algorithm, the way shards are rebalanced, and other things to make sure the implementation works. Software engineers can use modulus sharding to help their systems grow and work well by following best practises and keeping an eye on speed.