Copy data from standard collection to time-series collection

Okay, so I have some (TB sized) collections (observation data) that I’d like to copy over to equivalent time-series collections (currently I’m using 5.X, self-managed). The current collection has a proper UTC date-time as a “string” and time-series of course requires that to be a legit UTC “date”.

Option A, do I introduce a new UTC date field (based on the string that exists) into the existing collection that permeates the entire collection? First of all, is that bad? I’m not 100% sure what Mongo would do at the locking level if a command like that was issued (on a production system).

Option B: Pre-create the Time-Series collection, then attempt to populate it (from the original) while copy/converting that “string” UTC date to the actual UTC date required by time-series? Perhaps there is an option C that I haven’t thought of (other than "just retire already!)? I’d greatly appreciate a nudge in the right direction.

Hey @Allan_Chase,

Welcome to the MongoDB Community forums!

Could you please share the specific use case and the rationale behind opting for the time-series collections? Moreover, it’s essential to note that in time series collections, data insertion occurs in accordance with the timestamp, which ultimately results in the creation of meta-bucket collections.

I recommend you consider inserting data in a sorted manner based on the timestamp. By doing so, you can fully leverage the advantages of time series data, ensuring efficient querying and analysis.

Regards,
Kushagra

Hey @Allan_Chase,

You can use the MongoDB Kafka connector which will automatically convert the string to BSON Date:

Regards,
Kushagra

Great, thank you for responding. The types of use cases is 100% on par with the concept of weather sensors that repeatedly give new readings at well defined intervals. The rationale is that we were using a pre 5.X version of MongoDB that didn’t support Time Series Collections and thought that a handful of our collections fit the mold of Time Series. I will look at inserting/experimenting with sorted order with some of the smaller collections and see how that goes. Thanks again!

Thanks Kushagra, I’ll have a second look at Kafka. At first glance I thought it was too complicated for something I should be able to do with NoSQLBooster or export/import features. I’m totally old school when it comes to doing things like this, so I’ll take your advice and see how we go. Thank again.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.