Migrate Data into a Time Series Collection
On this page
To migrate data from an existing collection into a time series
collection, use an $out
stage in your aggregation pipeline. If your documents do not have a
suitable metadata field, use a $addFields stage before
the $out stage to add one.
Migrate Data to a Time Series Collection
Create a new time series collection
Create a new time series collection
by issuing the following command in the mongosh, changing the timeField, metaField, and granularity values as needed to match the data you're migrating:
db.createCollection( "weathernew", { timeseries: { timeField: "ts", metaField: "metaData", granularity: "hours" } } )
For detailed information on creating a new time series collection, see Create and Query a Time Series Collection.
(Optional) Transform your data and specify a metadata field
If your original collection doesn't have a metadata field, use the $addFields aggregation stage to add it. For the example in this procedure, documents require a metaData field.
Consider a collection with weather data that uses the format:
{ "_id" : ObjectId("5553a998e4b02cf7151190b8"), "st" : "x+47600-047900", "ts" : ISODate("1984-03-05T13:00:00Z"), "position" : { "type" : "Point", "coordinates" : [ -47.9, 47.6 ] }, "elevation" : 9999, "callLetters" : "VCSZ", "qualityControlProcess" : "V020", "dataSource" : "4", "type" : "FM-13", "airTemperature" : { "value" : -3.1, "quality" : "1" }, "dewPoint" : { "value" : 999.9, "quality" : "9" }, "pressure" : { "value" : 1015.3, "quality" : "1" }, "wind" : { "direction" : { "angle" : 999, "quality" : "9" }, "type" : "9", "speed" : { "rate" : 999.9, "quality" : "9" } }, "visibility" : { "distance" : { "value" : 999999, "quality" : "9" }, "variability" : { "value" : "N", "quality" : "9" } }, "skyCondition" : { "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" }, "cavok" : "N" }, "sections" : [ "AG1" ], "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 } }
The following pipeline stages add a metaData field as
specified in Step 1 and use $project to include or
exclude the remaining fields in the document:
{ $addFields: { metaData: { "st": "$st", "position": "$position", "elevation": "$elevation", "callLetters": "$callLetters", "qualityControlProcess": "$qualityControlProcess", "type": "$type" } }, }, { $project: { _id: 1, ts: 1, metaData: 1, dataSource: 1, airTemperature: 1, dewPoint: 1, pressure: 1, wind: 1, visibility: 1, skyCondition: 1, sections: 1, precipitationEstimatedObservation: 1 } }
Run the aggregation pipeline with $out as the
final stage
The example below uses the db.collection.aggregate() helper method:
db.weather_data.aggregate([ { $addFields: { metaData: { "st": "$st", "position": "$position", "elevation": "$elevation", "callLetters": "$callLetters", "qualityControlProcess": "$qualityControlProcess", "type": "$type" } }, }, { $project: { _id: 1, ts: 1, metaData: 1, dataSource: 1, airTemperature: 1, dewPoint: 1, pressure: 1, wind: 1, visibility: 1, skyCondition: 1, sections: 1, precipitationEstimatedObservation: 1 } }, { $out: "weathernew" } ])
After you run this command, you have the weathernew
collection below:
db.weathernew.findOne() { "_id" : ObjectId("5553a998e4b02cf7151190b8"), "ts" : ISODate("1984-03-05T13:00:00Z"), "dataSource" : "4", "airTemperature" : { "value" : -3.1, "quality" : "1" }, "dewPoint" : { "value" : 999.9, "quality" : "9" }, "pressure" : { "value" : 1015.3, "quality" : "1" }, "wind" : { "direction" : { "angle" : 999, "quality" : "9" }, "type" : "9", "speed" : { "rate" : 999.9, "quality" : "9" } }, "visibility" : { "distance" : { "value" : 999999, "quality" : "9" }, "variability" : { "value" : "N", "quality" : "9" } }, "skyCondition" : { "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" }, "cavok" : "N" }, "sections" : [ "AG1" ], "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 }, "metaData" : { "st" : "x+47600-047900", "position" : { "type" : "Point", "coordinates" : [ -47.9, 47.6 ] }, "elevation" : 9999, "callLetters" : "VCSZ", "qualityControlProcess" : "V020", "type" : "FM-13" } }
If your original collection had secondary indexes, manually recreate
them now. If your collection includes timeField values before
1970-01-01T00:00:00.000Z or after 2038-01-19T03:14:07.000Z,
MongoDB logs a warning and disables some query optimizations that make
use of the internal clustered index. Create a secondary index on the timeField to regain query
performance and resolve the log warning.