MongoDB Time Series only insert when changed

Working with Data
1

Hi everyone,

I’m currently working on a project where I have to use the new Time Series collection of MongoDB, which has been working so far, but I’m curious about one aspect of it.

Currently I’m getting around a thousand new inserts every 10 minutes, but only around 5% of those inserts are actually different than their previous insert. Is it possible or practical within the Time Series collection to only insert the changed data, and not the unchanged data? I know that way of storing is possible, but how would one go over the querying and getting averages? Is there a way to fill in the “gaps”, which is essentially the document that comes before the requested timestamp, and all the timestamps in between.

Let me know if someone already has figured this out, thanks in advance!

2

Hi @Rolf_Oldenkotte ,

It is possible to ingest only data when something changed, wether its worth doing that and how it complex the code and effort is up to the use case, therefore not sure its worth it,

Newer versions of MongoDB like 6.0, offers aggregation operators that can fill timeseries gaps:

Let me know if that helps

Thanks
Pavel

3

To calculate averages or some similar quantity on a numerical time series with irregular timing, you can use standard numerical integration methods to calculate the sums, etc. that are needed - e.g. trapezoid rule or Simpson’s rule.

You can also use interpolators of various kinds to fit irregularly spaced data points. Which type works best depends on what you know about your data.