MongoDb becomes slow after sometime

I am working on a task in which I am running a cron job, work of cron job is to complete a task and send notifications(emails and sms).
Time interval for cron job is in every 8 minutes.

CronJob is running for 25000 merchants, and there was around 4-5 tasks for each merchant and around 60-70 db queries are running for each merchant, in which around 15-20 db queries are insertion and updation and around 40-50 queries are get queries.

I have run the cron job using goroutines, in which I have implemented worker pool, and I have set the worker to 200, which means 200 merchants are completing their tasks parallely.

I have setup a server for database, for all merchants exists and there was seperate db for each merchant. And I have used mongoDb as Database, and mongoDb is running on it’s default settings.

My servers system specifications are,

Database server:-
RAM:-192GB
Database size:- 570GB
OS:- Ubuntu 22.04

Cron are running on different server and specifications of that server is:-
RAM:- 16GB
OS:-Ubuntu 22.04

My problem is that, whenever I am starting the cron service, for first few merchants the db is working fine. all the Db queries including insert, update, delete, Get are running fast, but after a period of time, db becomes slow, all the queries run very slow.

The db becomes slow for every operations including cronJob or other operations. I have noticed that mongoDb goes into the locking condition for certain period of time.
And this locking time is increasing rapidly, i.e. Whenever it was stopped for first time it was again started in 1-2 seconds, but after some time the time is increased.
After 2-3 hours, It goes to a state in which db got locked for more than 5 minutes and run queries for only 1 minute after that again goes to the locking state.

I have noticed a log which was logged frequently whenever db is stopped

{“t”:{“$date”:“2023-03-31T06:38:04.021+00:00”},“s”:“W”, “c”:“COMMAND”, “id”:20525, “ctx”:“conn60701”,“msg”:“Failed to gather storage statistics for slow operation”,“attr”:{“opId”:2317177,“error”:“lock acquire timeout”}}

I have noticed the locking condition by examining the logs, whenever the db is started after the lock, I am seeing these type of slow query logs in which handleLock and schemaLock is high.

{“t”:{“$date”:“2023-03-31T06:40:34.908+00:00”},“s”:“I”, “c”:“COMMAND”, “id”:51803, “ctx”:“conn59118”,“msg”:“Slow query”,“attr”:{“type”:“command”,“ns”:“ausloc678_bk_db.providers”,“command”:{“find”:“providers”,“filter”:{“uid”:7},“limit”:1,“projection”:{“_id”:1,“show_payment_method_and_price”:1,“show_payment_method_and_price_for”:1,“is_team_member”:1,“who_see_payment_method_and_price”:1,“team_lead_id”:1,“hide_provider_payments”:1,“hidden_provider_payments”:1,“show_booking_price”:1,“show_booking_price_for”:1,“who_see_booking_price”:1},“singleBatch”:true,“lsid”:{“id”:{“$uuid”:“c6c4c42b-216c-48c4-92bf-8ca3b1db93f7”}},“$db”:“ausloc678_bk_db”},“planSummary”:“COLLSCAN”,“keysExamined”:0,“docsExamined”:52,“cursorExhausted”:true,“numYields”:1,“nreturned”:0,“queryHash”:“B89C5911”,“planCacheKey”:“B89C5911”,“reslen”:114,“locks”:{“FeatureCompatibilityVersion”:{“acquireCount”:{“r”:2}},“ReplicationStateTransition”:{“acquireCount”:{“w”:2}},“Global”:{“acquireCount”:{“r”:2}},“Database”:{“acquireCount”:{“r”:2}},“Collection”:{“acquireCount”:{“r”:2}},“Mutex”:{“acquireCount”:{“r”:1}}},“storage”:{“data”:{“bytesRead”:28496,“timeReadingMicros”:13},“timeWaitingMicros”:{“handleLock”:122143,“schemaLock”:15285487}},“protocol”:“op_msg”,“durationMillis”:15899}}

Can someone help me to find the solution to prevent these locking condition, I have optimized all the db queries, there was no lookup or joins are used in any query.

And I have some questions:-

  • What would be the reasons for this issue I am facing?
  • What changes should I do to resolve this problem?

You have

and

so your server haas at least 50000 open files. At least, because if you have indexes you even have more files. So you effectively implemented the Massive Number of Collections anti-pattern.

I am not too sure about that, since the logs you shared show a

context: I just have a total size of the design collection of 16gb and mongodb is taking forever (highly delaying) to reload the data and respond with my backend, even the users collection which is only about 500kb taking 5s approx, it just started to happen 1day ago. plz help me

design collection -

1. _id

:

ObjectId('65aa13f73e7b6c51ca7d004f')

userId

:

ObjectId('64a2fc75ca2300f3167e99f5')

userName

:

"Tester1"

title

:

"demoFour"

description

:

"yup demos"

tags

:

Array (2)

0

:

"Landing Page"

1

:

"Web Design"

files

:

Array (1)

0

:

Object

filename

:

"genre4.jpg"

data

:

"/9j/4AAQSkZJRgABAQIAOAA4AAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ…"

filesize

:

"0.22 MB"

type

:

"image/jpeg"

_id

:

ObjectId('65aa13f73e7b6c51ca7d0050')

no_files

:

1

uploadDate

:

2024-01-19T06:17:27.993+00:00

__v

:

60

likes

:

Array (1)

0

:

ObjectId('64a2fc75ca2300f3167e99f5')

views

:

Array (1)

0

:

ObjectId('64a2fc75ca2300f3167e99f5')

2. _id

:

ObjectId('65aa14233e7b6c51ca7d0054')

userId

:

ObjectId('64a2fc75ca2300f3167e99f5')

userName

:

"Tester1"

title

:

"japan"

description

:

"wonderful japan"

tags

:

Array (2)

files

:

Array (1)

no_files

:

1

uploadDate

:

2024-01-19T06:18:11.216+00:00

__v

:

20

likes

:

Array (1)

views

:

Array (1)

3. _id

:

ObjectId('65ad1ca97cf484eedd0bc406')

userId

:

ObjectId('64a2fc75ca2300f3167e99f5')

userName

:

"Tester1"

title

:

"Demo Video"

description

:

"Hello There"

tags

:

Array (2)

files

:

Array (1)

no_files

:

1

uploadDate

:

2024-01-21T13:31:21.642+00:00

__v

:

13

likes

:

Array (empty)

views

:

Array (1)

4. _id

:

ObjectId('65ae776c28af4b26b4470799')

userId

:

ObjectId('64a2fc75ca2300f3167e99f5')

userName

:

"Tester1"

title

:

"Last Demo"

description

:

"This are the most beautiful images, you cant even describe and yet you…"

tags

:

Array (3)

files

:

Array (2)

no_files

:

2

uploadDate

:

2024-01-22T14:10:52.124+00:00

__v

:

15

likes

:

Array (1)

views

:
Array (1)
``

Your sample documents are not formatted in a way that we can use them.

We need the list of indexes that you have.

We need the exact query that you perform.

We need the explain plan.

We need the specifications of your systems.

Ans1 → Plz excuse my layman language
{
“_id”: {
“$oid”: “65aa13f73e7b6c51ca7d004f”
},
“userId”: {
“$oid”: “64a2fc75ca2300f3167e99f5”
},
“userName”: “Tester1”,
“title”: “demoFour”,
“description”: “yup demos”,
“tags”: [
“Landing Page”,
“Web Design”
],
“files”: [
{
“filename”: “genre4.jpg”,
“data”:"/9j/4AAQSkZJRgABAQIAOAA4AAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBA…more large bunch of B64bits

similarly there are 3 more such sample data in a collection called ‘designs’

Ans2 →
Only id is the index so far in my collections

Ans3 →
For querying I am using the nodejs with express, mongoose

Some instances -
router.post(“/upload_design”, async (req, res) => {
try {
const sessionToken = req.header(‘Authorization’);
const user = await User.findOne({ sessionToken });
const { files } = req.body;
// console.log(files)
if (!user) {
return res.status(401).json({ error: ‘Unauthorized’ });
}
if (files != null) {
const designData = new Design({
userId: user._id,
userName: user.name,
title: files.title,
description: files.description,
tags: files.tags,
files: files.base64Files, // Array of file details
no_files: files.base64Files.length,
});

  await designData.save();
  // console.log('Design saved:', savedDesign);
  return res.status(200).json({ message: 'Successful' });
}

} catch (error) {
console.error(err);
}
})
router.get(‘/fetch_design_data’, async (req, res) => {
const sessionToken = req.header(‘Authorization’);
const user = await User.findOne({ sessionToken });
try {
if (!user) {
return res.status(401).json({ error: ‘Unauthorized’ });
}

const userId = user._id;
const designs = await Design.find({ userId });
if (!designs || designs.length === 0) {
  console.log("Not found")
  // return res.status(404).json(EMPTY_DESIGN);
  const EMPTY_DESIGN = { status: "empty", error: 'No file Uploaded' };
  return res.status(404).send(EMPTY_DESIGN);
  // return res.status(404).json({ error: 'Design not found' });
} else {
  const profilePic = user.profilePic;
  res.status(200).send({ designs, profilePic });
}

} catch (error) {
console.log(error)
}
// console.log(designs[0])
})
router.get(‘/fetch_Individual_design/:designId’, async (req, res) => {
try {
const sessionToken = req.header(‘Authorization’);
const designId = req.params.designId;
const user = await User.findOne({ sessionToken });

if (!user) {
  return res.status(401).json({ error: 'Unauthorized' });
}

const designDetail = await Design.findById(designId);
const DesignUserId = designDetail.userId;

// Query user details by DesignUserId
const userDetails = await User.findById(DesignUserId);

// Check if the user associated with the design exists
if (!userDetails) {
  return res.status(404).json({ error: 'User not found' });
}

if (DesignUserId.toString() !== user._id.toString()) {
  return res.status(403).json({ error: 'Permission denied' });
}
const ShortUserDetails = [
  (userDetails.name),
  (userDetails.profilePic)
]
if (!designDetail.views.includes(user._id)) {
  // Increment the view count
  designDetail.views.push(user._id);
  // Save the updated design
  await designDetail.save();
}

res.status(200).send({ designDetail, ShortUserDetails });

} catch (err) {
console.log(err);
res.status(500).json({ error: ‘Internal Server Error’ });
}
});
router.post(‘/like_design/:designId’, async (req, res) => {
try {
const sessionToken = req.header(‘Authorization’);
const { designId } = req.params;
if (!sessionToken) {
return res.status(401).json({ error: ‘Unauthorized’ });
}
const design = await Design.findById(designId);
if (!design) {
return res.status(404).json({ error: ‘Design not found’ });
}
const user = await User.findOne({ sessionToken });

const isLiked = design.likes.includes(user._id);

if (isLiked) {
  design.likes.pull(user._id);
} else {
  design.likes.push(user._id);
}
const updatedDesign = await design.save();
res.status(200).json({ likes: updatedDesign.likes, updatedDesign });

} catch (err) {
console.log(err);
}
})
Ans4 →
I guess the above things are adequate in terms of explanation

Ans5 →
Specifications = i3 11th gen, 8gb ssd ram, 256gb rom, win 10 (x64) bit OS

I guess, the issue has been solved, mongodb is working now as fast as usual, well thanks for overseeing the help.