I need an optimized workflow - to recalculate all records in a collection

Working with Data Connectors & Integrations
, ,
#1

Hello dear Lauren Schaefer,
your videos about mongoDB on Youtube are more than just brilliant - thank you!
I would still need a tip or a reference to a good tutorial please!
I just have to implement the following:
Every 4 hours I should go through all records of a certain collection and recalculate certain values - the calculation base I have to get from a different collection for each record!
I would like to realise this under node.js!
It is not clear to me what is the easiest way to proceed here!
Do you have a tip for me - or can you recommend some good literature?
Many thanks in advance for your help!
Kind regards
Markus

1 Like
#2

You can use node-cron task scheduler.

const cron = require('node-cron');
const Model = require('../models/custom-model)

cron.schedule('0 0 */4 * * *', async () => {
  try {
    const result = await Model.find();
    console.log('Result: ', result);
  } catch (error) {
    console.log('ERROR: ', error);
  }
});

Now, you can add your custom logic inside the handler.

#3

Here - i need also a tip:

#4

Can you add an example?

#5

The first collection has 50000 and more records including the following fields:

  • consignment number
  • weight
  • distance
  • price
  • (and 50 other fields - some with large amounts of data).
  • (… and 50 other fields)
    Based on the distance I have to get a multiplier from the second collection and then calculate the price in the first collection: Multiplier * Weight!
    I then have to repeat the whole thing every 4 hours - as the multipliers can change constantly!
    How do I implement this - to move as little data as possible between the backend and the database?
    Thanks in advance!
    Markus
#6

Hi @Markus_B,

Can you add an example document of both of your collections, and expected output?

Also, did you consider storing multiplier in the first collection? Is there a reason why it’s in the second collection?

#7

Unfortunately, I can’t describe it any further at this stage - as I’m in the process of developing it!
For me, the question is simply, what is the easiest way to go through the 50000 data records?

  • Can I first only determine the IDs of all existing data records via a query?
  • Can I then use the ID to load only the relevant fields “km”, “weight”, “price” and “date”?
  • then use “date” and “km” to search in the 2nd collection the “mutliplicator”.
  • and then calculate and update the “price” in the first collection!
    I can’t describe it any better at the moment!
    Thanks in advance!
#8

Is there really no one - who can show me - how I can most sensibly go through the collection with as little traffic as possible between backend and database?

Where can I find good tutorials on my topic?

#9

Most likely not. Because you wrote it yourself:

It is very hard to propose solutions with a vague description of a problem.

Take the course at university.mongodb.com.

#10

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.