Optimizing Real-Time Variable Calculations and Scaling with 10-Second Data Frequency

Working with Data
, ,
1

I’m seeking guidance on calculating a new variable from existing variables within my MongoDB collections, considering a data frequency of 10 seconds.

Specific Requirements:

  • Variable Calculation:

    • Combine “machine-1-power-1” and “machine-1-power-2” into a new variable named “machine-1-power”.

    • Incorporate seamless scaling for division by a user-specified value (e.g., 1000 or any other dynamic input).

    • Ensure efficient handling of 10-second data updates for real-time responsiveness

  • Data Frequency:

    • Account for data updates every 10 seconds.

Here is my db scherma

  • Devices :- _id , name , certificates : {pkey,cert}

  • Variables :- _id, name, deviceId ( reference to device (_id) )

  • Values (timeseries collection) :- _id, value, timestamp , metafield : {variableId}

Key Considerations:

  • Performance Optimization: How can I achieve optimal performance for these calculations amidst frequent data updates?
  • Indexing Strategies: Which indexing approaches would best support these calculations and real-time data retrieval?
  • Aggregation Framework: Can the Aggregation Framework effectively handle both calculations and scaling within query pipelines?
  • Alternative Approaches: Are there alternative methods, such as Map-Reduce or custom logic, that might prove more suitable?
  • Best Practices: What are the recommended best practices for handling real-time calculations and scaling in MongoDB?