I have a Mongo DB table that stores time-series data.
- _id is the only unique field in the database.
- I have an index of a time field with -1, so the index is created in descending order.
I am trying to Mongo DB schema transformation and I have written code to use AutoMapper library to do the transformation for each document.
I am reading document by document and transforming it and inserting them. Have configurable features for Bulk Read(Using Skip and Limit) and Bulk Write for speed.
I just wanted to know if suppose some error occurs and fails then how can I resume?
Can I use this _id value to resume the migration?
Wondering if I can use _id as a filter condition because there is an index on this column.
I can’t use the time to filter because this column is not unique and there we could miss few records to migrate if using greater than and will insert duplicate in using less than.
More importantly the _id column is unique. If I am able to apply $gt on this field then I will be able to jump to the last processed location without any overhead because of the index.
Basically what I am saying is I need to have one unique column that can be comparable for greater or lesser. _id column is one of them. Not sure if it is comparable. Is it?
Update: Maybe _id this is not unique https://docs.mongodb.com/manual/reference/bson-types/#objectid Not sure what this link meant. My head is biting because of reading too much information. Man this quarantine has given me a big problem and that’s time.