I am needing to update my time series records after a schema review. We initially had static information outside the metaField and need to move it into the metaField.
For example, this record/document:
{
“timestamp”: {
“$date”: “2024-04-07T14:56:02.292Z”
},
“metadata”: {
“additional_identifier”: “”,
“apiKey”: “FF199”,
“drive”: “Apollo_HPS”,
“type”: “standard”
},
“__v”: 0,
“sensor”: “HeatSink”,
“value”: “16”,
“_id”: {
“$oid”: “6612b405615b477bcbc57af1”
},
“quality”: 192,
“createdAt”: {
“$date”: “2024-04-07T14:56:05.429Z”
}
}
needs to be modified to move the “sensor” field into the “metadata” subdoc like this:
{
“timestamp”: {
“$date”: “2024-07-01T00:00:00.003Z”
},
“metadata”: {
“additional_identifier”: “”,
“apiKey”: “FF323”,
“drive”: “IDEC_HPS”,
“sensor”: “Frequency”,
“type”: “Trends”,
“varName”: “FlexFlow.Trailer #323.IDEC_HPS.Trends.Frequency”
},
“value”: “58.81626”,
“_id”: {
“$oid”: “6681f185e322d4b702c43d41”
},
“quality”: 192,
“createdAt”: {
“$date”: “2024-07-01T00:00:05.717Z”
}
}
but looking through the documentation it looks like my only options is to do an $out to a new collection with the modifications, then drop the original collection and move it back. I have tried that, but we have millions of time series records and so I started to do them a month at a time, but $out seems to replace the collection if it already exists, so each month worth of records replaces the previous batch. And each batch takes almost an hour to process. Doing the whole collection at once doesn’t seem feasible, as it’s over 12 months worth… what other options do we have? Or am I missing something in the documentation? I’ll be honest, so far time series has been pretty challenging to work with.