Hi everybody,
Can anyone tell me how to upload a CSV file of about 1mil record to MongoDB? I’m working on it, but I recognized that the CSV file with about 200k records still uploads normally, but when I raise it to 300k, I get the error “Could not connect to any servers in your MongoDB Atlas cluster” right away, even though I set up the IP 0.0.0.0 and current IP.
Note: The CSV file is still uploaded to MongoDB normally via MongoDB Compass.
This is the CSV reader function, when i<= 200.000, it works well but i<= 300.000, the application crashes
function readCSVFile(file_path, delimiter = ',') {
return new Promise((resolve, reject) => {
fs.readFile(file_path, 'utf8', (err, data) => {
if (err) {
reject('Failed to read CSV file');
} else {
const lines = data.trim().split('\n');
const headers = lines[0].split(delimiter).map((header) => header.replace(/[\r]+/g, ''));
//console.log(headers)
const result = [];
for (let i = 1; i < lines.length; i++) {
const obj = {};
const currentLine = lines[i].split(delimiter);
if (currentLine.length !== headers.length) {
console.log(i)
continue;
//reject('Invalid CSV format');
}
else {
for (let j = 0; j < headers.length; j++) {
obj[headers[j]] = currentLine[j].trim()
}
}
result.push(obj);
}
resolve(result);
}
});
});
}
I have tried to split the file into smaller batches but it is not successful
const batchSize = 50000 // Set the batch size to 1000
readCSVFile('./datasets/2021-05.csv')
.then(data => {
const trips = data.map(row => new Trip(row))
// Split the trips array into three parts
const numTrips = trips.length
const numBatches = Math.ceil(numTrips / batchSize)
const batches = Array.from({ length: numBatches }, (_, i) =>
trips.slice(i * batchSize, (i + 1) * batchSize)
)
console.log(batches.length)
// Insert each batch separately using insertMany
Promise.all(
batches.map(batch => {
Trip.insertMany(batch).then(results => {
console.log(`Inserted ${results.length} trips to MongoDB`)
})
.catch(error => {
console.error(error)
})
})
)
})
.catch(error => {
console.error(error)
});
The error is :
err = new ServerSelectionError();
MongooseServerSelectionError: Could not connect to any servers in your MongoDB Atlas cluster. One common reason is that you're trying to access the database from an IP that isn't whitelisted. Make sure your current IP address is on your Atlas cluster's IP whitelist: https://www.mongodb.com/docs/atlas/security-whitelist/
Thank you for helping