Ticket MIgrations

HI all stuck here when i run python movie_last_updated_migration.py
it says

45993 documents to migrate
0 documents updated

my predicate has been defined as

predicate = {'lastupdated':{'$exists':True}}
projection = None
movies_to_migrate = []
for doc in cursor:
    doc_id = doc.get('_id')
    #print(doc_id)
    lastupdated = doc.get('lastupdated', None)
    #print(lastupdated)
    movies_to_migrate.append(
        {
            "doc_id": ObjectId(doc_id),
            "lastupdated": parser.parse(str(lastupdated))
        }
    )

and my bulk upload code

try:
    # TODO: Complete the UpdateOne statement below
    # build the UpdateOne so it updates the "lastupdated" field to contain
    # the new ISODate() type
    bulk_updates = [UpdateOne(
        {"_id": movie.get("doc_id")},
        {"$set": {"lastupdated": movie.get("lastupdated")}}
    ) for movie in movies_to_migrate]

    # here's where the bulk operation is sent to MongoDB
    bulk_results = mflix.movies.bulk_write(bulk_updates)
    print(f"{bulk_results.modified_count} documents updated")

except InvalidOperation:
    print("no updates necessary")
except Exception as e:
    print(str(e))

when i go to frontend app click on migrate option it always show red irrespective of what i do,
i am not sure what i am doing wrong please help?

Where do you define cursor?

predicate = {‘lastupdated’:{’$exists’:True}}
projection = None

cursor = mflix.movies.find(predicate, projection)

I do not see that in the code you posted in the first message.

i have posted just now if u want i ca post full code

from pymongo import MongoClient, UpdateOne
from pymongo.errors import InvalidOperation
from bson import ObjectId
import isodate
import dateutil.parser as parser
from bson.json_util import dumps

“”"
Ticket: Migration

Update all the documents in the movies collection, such that the “lastupdated”
field is stored as an ISODate() rather than a string.

The parser.parse() method can transform date strings into ISODate objects for
us. We just need to make sure the correct operations are sent to MongoDB!
“”"

ensure you update your host information below!

#host = “mongodb://localhost:27017”
host = “mongodb+srv:###”
mflix = MongoClient(host)[“mflix”]

movies = mflix.movies
print(movies.count_documents({}))
#print(dumps(movies.find_one(),indent=2))

predicate = {‘lastupdated’:{’$exists’:True}}
projection = None

cursor = mflix.movies.find(predicate, projection)
#print(dumps(cursor,indent=2))

this will transform the “lastupdated” field to an ISODate() from a string

movies_to_migrate =
for doc in cursor:
doc_id = doc.get(’_id’)
#print(doc_id)
lastupdated = doc.get(‘lastupdated’, None)
#print(lastupdated)
movies_to_migrate.append(
{
“doc_id”: ObjectId(doc_id),
“lastupdated”: parser.parse(str(lastupdated))
}
)

print(movies_to_migrate[1])
print(f"{len(movies_to_migrate)} documents to migrate")

try:
# TODO: Complete the UpdateOne statement below
# build the UpdateOne so it updates the “lastupdated” field to contain
# the new ISODate() type
bulk_updates = [UpdateOne(
{"_id": movie.get(“doc_id”)},
{"$set": {“lastupdated”: movie.get(“lastupdated”)}}
) for movie in movies_to_migrate]

# here's where the bulk operation is sent to MongoDB
bulk_results = mflix.movies.bulk_write(bulk_updates)
print(f"{bulk_results.modified_count} documents updated")

except InvalidOperation:
print(“no updates necessary”)
except Exception as e:
print(str(e))

I would add a print statement of bulk_updates just before your call to bulk_results = mflix… to see oif it is the creation of the array that is wrong or if it is the criteria of UpdateOne.

I suspect that _id is wrong.

In your cursor loop, when you append to the array, you do “doc_id” : ObjectId(doc_id). I think that doc_id might already be an ObjectId since you doc_id = doc.get( _id ).

1 Like

I tried the suggestions above but my code still doesn’t work.
I’ve use the set method, even tried to pass the get(‘lastupdated’) through ISODate, to no avail.
Below is what is see when I place an print statement above the assertion.
assert isinstance(result.get(‘lastupdated’), datetime) 2015-08-15 00:15:54.630000000

So, it seems the format of lastupdated does not get changed. :frowning:

parser.parse(lastupdated) is not converting the date to ISODate format.
when I execute pytest -m migration, I get ‘2015-08-15 00:15:54.630000000’.

Tired to convert using $dateFromString, $toDate, still the same

{"$dateFromString": {“dateString”: lastupdated}}
{"$toDate": lastupdated}

Strange ticket…
dateFormat.parse will never work because Migrator uses single date pattern
Some movies have this format 2015-08-26 00:03:50.133000000 and few have this 2015-01-17 00:46:41
In order not to lose milliseconds I’ve used datePattern = “yyyy-MM-dd HH:mm:ss.SSS”
THen this silly trick
String lastUpdated = doc.getString(“lastupdated”);
String lastUpdatedTrunc = null;
int dotIdx = lastUpdated.indexOf(".");
if(dotIdx > -1) {
lastUpdatedTrunc = lastUpdated.substring(0, dotIdx + 4);
} else {
lastUpdatedTrunc = lastUpdated + “.000”;
}
and dateFormat.parse(lastUpdatedTrunc)

But I wonder if you would need to do this for real if this is right approach
Here we read movies in 2 different loops, accumulate list of WriteModels
Would it not be better to issue updateMany() with empty query and update document being a pipeline something like this
[ { $set: { lastupdated: { $toDate : “$lastupdated”} }]
We can cook up conditional doc for imdb,rating I’m sure…

Please provide the _id of the documents you found that were in the wrong format?

Thanks for looking into it Steeve
I don’t see how to do it since I’ve already changed lastupdated field type to date