Using multiple measurements in a time series collection

[A_S_Gowri_Sankar]

Hello, Dear Community!

We’re exploring MongoDB time-series collections to export/stream, and aggregate resource metrics like CPU, Memory, etc., at various time intervals as emitted by the source

Going through the documentation to Create and Query a Time Series Collection - MongoDB Manual v7.0, I’m understanding that every document in a time-series collection should contain one measurement (or metric)

Each document you insert should contain a single measurement

So, I have a few questions now

1. should each document carry data of only one measurement? Or can we combine multiple measurements in one document?
2. The example mentioned in the documentation referenced refers to “Temperature” as the measurement stored inside a collection called “weather”. Does this mean we should use a dedicated collection for every measurement? In our case, should we have a separate collection for each of the metrics like CPU, Memory, Disk etc.?

Can we combine all the measurements into one document for a given timestamp? I guess this would avoid ending up having multiple collections and/or documents for every measurement we need to manage

E.g. we create a single collection named metrics supposed to contain all the different metric measurements at given time intervals

db.createCollection("metrics",{timeseries:{"timeField":"timestamp","metaField":"metadata"}});

Then, we insert a document which consists of the metaField metadata indicating what kind of measurement we’re inserting and then the actual measurement as another object like below

{
  "timestamp": "1704067200000",
  "metadata": {
    "metric_fields": [
      "prometheus.cpu_avg",
      "prometheus.mem_avg"
    ]
  },
  "metrics": {
    "prometheus.cpu_avg": 60,
    "prometheus.mem_avg": 70
  }
}

We’re using MongoDB v7. Any help in this modeling would be helpful. Please ask for any info you need. Thank you!

[emmanuel_bernard]

@A_S_Gowri_Sankar

Each document you insert should contain a single measurement

Hi,

I wanted to share a helpful MongoDB resource that explains Time Series Data : MongoDB Time Series Data.

Time series data consists of measurements recorded at specific time intervals from one or more sources.

You can create your collection

db.createCollection("metrics", 
{ timeseries: { 
               timeField: "date", 
               metaField: "source"  } })

Each measurement inserted should be a single measurement.

db.metrics.insertOne({
    date: ISODate("2024-08-14T11:00:00.000Z"),
    source: "prometheus",
    cpu: 60,
    memory: 70,
    networkIn: 327,
    networkOut: 663,
    ...
})

Then use aggregation pipeline operators ($avg, $sum, $count) and aggregation pipeline stages to analyse your data.

[A_S_Gowri_Sankar]

thank you very much, @Emmanuel_Bernard1! that is very helpful! so, this means we could put as many measurements or metrics as we need and not restricted to just one metric per document in a time-series collection

a few questions

1. can we embed these measurements under an object like "metrics":{"cpu":50,"mem":60,"disk":35} instead of having those metric fields directly in the root of the document?
2. similarly, can the metaField be an object instead of a single value?

Thank you again for sharing the document!

[steevej (Steeve Juneau)]

I am not very familiar with TS but I managed to quickly test both issues.

With the following TS collection

db.getCollectionInfos( { name : "ts-object"})[0].options
{
  timeseries: {
    timeField: 'date',
    metaField: 'meta',
    granularity: 'seconds',
    bucketMaxSpanSeconds: 3600
  }
}

I could create all sort of documents:

A metrics objects rather than field at the top level

{ date: 2024-08-14T17:29:28.492Z,
  meta: 'metric object',
  metrics: { cpu: 50, mem: 60, disk: 35 },
  _id: ObjectId("66bce978ec9f2d14eca4fc7f") }

String meta field and top level fields for measurements

{ date: 2024-08-14T17:29:56.983Z,
  meta: 'metric object',
  _id: ObjectId("66bce994ec9f2d14eca4fc80"),
  mem: 60,
  cpu: 50,
  disk: 35 }

An object as the meta field

{ date: 2024-08-14T17:30:51.484Z,
  meta: { type: 'object', x: 1, y: 2 },
  mem: 60,
  _id: ObjectId("66bce9cbec9f2d14eca4fc81"),
  disk: 35,
  cpu: 50 }

Note that the structured metrics version probably uses more space.

[emmanuel_bernard]

Hi @A_S_Gowri_Sankar,

Here is another MongoDB resource https://www.mongodb.com/docs/manual/core/timeseries-collections/

Metrics or values are better definitions and are not ambiguous like measurements.

Yes you can put as many metrics as needed.
Yes you can use Object or String.