Hi Wan, thanks for your comments.
I’m going to try to unfold the use case.
I’m building and app to manage company operations (purchases, sales, bill of material, inventories, accountancy, etc.). Also known as ERP (enterprise resource planner).
Although the app manages the forms to introduce this data, it also provides the option of uploading data from an excel or open xml file (a spreadsheet).
My intention is not only upload data, but also store the excel file in a s3 of aws (this is already done) so the user can keep track of the uploads it has performed and I’m able to write this excel, add a timestamp to the name and add a column with the id generated and a comment if the item could not be written in db for any reason. So the user only needs to download this excel and have this info
That’s way I’m trying to perform this operation from the backend. Also, maybe I’m mistaken, in this way I don’t use too much of the clients computer’s power and also make sure that low user memories or old computers don’t affect the upload operation. But I may be wrong about this believe.
Now, I’m sending the file from the front end to the server like this:
this.sendBtn.addEventListener('click', async () => {
const formData = new FormData();
// APPEDN EACH FILE TO THE FORMDATA
this.files.forEach(f => formData.append(f.name, f));
// PROVIDE JSON WITH PARAMS
formData.append('module', this.module);
const daoRes = await DaoUpdate.uploadElements(formData);
if(!daoRes.success) {
// REQUIRED MANAGE EVENTUALITIES
return;
};
The static method of the class Dao is quite simple:
// UPLOAD
static uploadElements = async obj => {
const res = await fetch(`${URL}/api/cloud/upload`, {
method: 'POST',
body: obj
});
return this._sendObj(res);
};
And it works. I tried with 140mb csv file and does not block the thread and the file get’s to the s3 bucket. And I think this is the proper way to tackle the issue. Because if I’m not mistaked, the formData object already manages the “descomposition-into-digestible-chunks-of-data” (don’t find the correct words) and that’s why there is not blocking thread. (Maybe I should test with bigger files, but 1M lines, what I’m uploading, I think is the max).
Now, the controller now uses the formidable module, because the Node.js project seems to recommand that. See: How to Handle Multipart Form Data (Official Node.js docs)
The whole function body looks like that:
try {
const user = req.user._id;
// UPLOAD FILES TO APP
const uploadsDir = path.join(path.resolve() + '/src/base/uploads');
const options = {
keepExtenstions: true,
maxFileSize: 200 * 1024 * 1024,
uploadDir: uploadsDir,
multiples: true
};
const data = formidable(options);
// INFO:
// data.parse rebuilds the file to the passed folder, and once is done
// calls teh callback function.
return data.parse(req, async (err, fields, files) => {
if(err) return console.info('err! ', err);
// UPLOAD OBJECT
const module = fields.module;
const uA = Object.keys(files).map(async fName => {
const fileRoute = files[fName].filepath;
const fileStream = fs.createReadStream(fileRoute);
const uploaded = await s3.send(new PutObjectCommand({
Bucket: `ek-knote-${module}`,
Key: `${user}/${fName}`,
Body: fileStream
}));
// REMOVE FILE FROM APP
if(uploaded.$metadata.httpStatusCode === 200) {
try {
await unlink(fileRoute);
} catch (err) {
console.info('Error by unliking uploaded fil --> ', err);
};
};
const obj = {
status: uploaded.$metadata.httpStatusCode,
fN: fName,
extension: fName.match(/[^.]+$/g).pop()
};
return obj;
});
return res.status(201).send(await Promise.all(uA));
});
} catch (err) {
console.info('uploadElement (upload.js) ERROR --> ', err);
return res.status(500).send();
};
Where I have problemes is in parsing the file before unlinking it and start sending json objects to the MongoDB. Here is where I’m stuck. The problem with csv extenssion files is that the user should feed several archives, because csv is plain text and knows nothing about workbooks.
Also, It would be a better approach to not hold files into the server disk, because it scales poorly.
Thanks!