Ticket: Migration

Hey

While trying to complete this ticket, I get this output

46014 documents to migrate
0 documents updated

-I have updated the host.
-My predicate works in Compass
-The existing code handles converting string to ISODate
-I have also updated the code for bulk update

bulk_updates = [UpdateOne(
{’_id’: movie.get(‘doc_id’)},
{’$set’: {‘lastupdated’: movie.get(‘lastupdated’)}}
) for movie in movies_to_migrate]

Can anyone tell me what I’m doing wrong

Can I see your predicate & projection answers? Need to be sure the lastupdated field has been converted and/or is not empty

Your code works for me.

predicate = {‘lastupdated’:{’$exists’:‘True’}, ‘lastupdated’:{’$type’:‘string’}}
projection = None

This is the ouput when I print one document in movies_to_migrate

{‘doc_id’: ObjectId(‘5c45dd45c068cd2c1ca5719f’), ‘lastupdated’: datetime.datetime(2015, 8, 26, 0, 3, 50, 133000)}

Please can you show the full python script. Seeing the full picture would help

from pymongo import MongoClient, UpdateOne
from pymongo.errors import InvalidOperation
from bson import ObjectId
import dateutil.parser as parser

“”"
Ticket: Migration

Update all the documents in the movies collection, such that the “lastupdated”
field is stored as an ISODate() rather than a string.

The parser.parse() method can transform date strings into ISODate objects for
us. We just need to make sure the correct operations are sent to MongoDB!
“”"

ensure you update your host information below!

host = “mongodb+srv://m220student:m220password@mflix-bav40.mongodb.net/test”
mflix = MongoClient(host)[“mflix”]

TODO: Create the proper predicate and projection

add a predicate that checks that the “lastupdated” field exists, and then

checks that its type is a string

a projection is not required, but may help reduce the amount of data sent

over the wire!

predicate = {‘lastupdated’:{’$exists’:‘True’}, ‘lastupdated’:{’$type’:‘string’}}
projection = None

cursor = mflix.movies.find(predicate, projection)

this will transform the “lastupdated” field to an ISODate() from a string

movies_to_migrate =
for doc in cursor:
doc_id = doc.get(‘doc_id’)
lastupdated = doc.get(‘lastupdated’, None)
movies_to_migrate.append(
{
“doc_id”: ObjectId(doc_id),
“lastupdated”: parser.parse(lastupdated)
}
)
print(movies_to_migrate[1])

print(f"{len(movies_to_migrate)} documents to migrate")

try:
# TODO: Complete the UpdateOne statement below
# build the UpdateOne so it updates the “lastupdated” field to contain
# the new ISODate() type
bulk_updates = [UpdateOne(
{’_id’: movie.get(‘doc_id’)},
{’$set’: {‘lastupdated’: movie.get(‘lastupdated’)}}
) for movie in movies_to_migrate]

# here's where the bulk operation is sent to MongoDB
bulk_results = mflix.movies.bulk_write(bulk_updates)
print(f"{bulk_results.modified_count} documents updated")

except InvalidOperation:
print(“no updates necessary”)
except Exception as e:
print(str(e))

Its the for loop.

for doc in cursor:
   doc_id = doc.get(‘doc_id’) # should be doc_id = doc.get('_id')

the movies collection has an _id not doc_id

Hey
Thanks a lot. Finally got it to work

Sure. Happy to help :smile:

Hi,
If anyone’s interested, I chose to use list comprehension vs the for loop.

[{“doc_id”: ObjectId(doc.get(’_id’)), “lastupdated”: parser.parse(doc.get(‘lastupdated’, None)) } for doc in cursor]
It worked!
:slight_smile:

1 Like

hi,
I think this is a fairly straight-forward solution and i don’t know what i am doing wrong. Would you take a look? I’m guessing its a typo somewhere because I’m getting Assertion Error.

predicate = {"lastupdated": {"$exists": 'True'}, "lastupdated":{"$type": "string"}}
projection = None

cursor = mflix.movies.find(predicate, projection)

# this will transform the "lastupdated" field to an ISODate() from a string
movies_to_migrate = []
for doc in cursor:
    doc_id = doc.get('_id')
    lastupdated = doc.get('lastupdated', None)
    movies_to_migrate.append(
        {
            "doc_id": ObjectId(doc_id),
            "lastupdated": parser.parse(lastupdated)
        }
    )

print(f"{len(movies_to_migrate)} documents to migrate")

try:
    # TODO: Complete the UpdateOne statement below
    # build the UpdateOne so it updates the "lastupdated" field to contain
    # the new ISODate() type
    bulk_updates = [UpdateOne(
        {"_id": movie.get("doc_id")},
        {"$set": {"lastupdated": movie.get("lastupdated")}}
    ) for movie in movies_to_migrate]

    # here's where the bulk operation is sent to MongoDB
    bulk_results = mflix.movies.bulk_write(bulk_updates)
    print(f"{bulk_results.modified_count} docusments updated")

except InvalidOperation:
    print("no updates necessary")
except Exception as e:
    print(str(e))

try removing quotes from the True boolean value.
{“lastupdated”: {"$exists": True}

Sorted. I had forgotten to run the migrations code actually. Hence the assertion error.
Thanks for responding though.

1 Like

Below are the migration code

ensure you update your host information below!

host = “mongodb+srv://username m220student:m220password@mflix-fqtnb.mongodb.net/test”
mflix = MongoClient(host)[“mflix”]

TODO: Create the proper predicate and projection

add a predicate that checks that the “lastupdated” field exists, and then

checks that its type is a string

a projection is not required, but may help reduce the amount of data sent

over the wire!

predicate = {“lastupdated”: {"$exists": True} , “lastupdated”: {"$type": “string”}}
projection = None

cursor = mflix.movies.find(predicate, projection)

this will transform the “lastupdated” field to an ISODate() from a string

movies_to_migrate =
for doc in cursor:
doc_id = doc.get(’_id’)
lastupdated = doc.get(‘lastupdated’, None)
movies_to_migrate.append(
{
“doc_id”: ObjectId(doc_id),
“lastupdated”: parser.parse(lastupdated)
}
)

print(f"{len(movies_to_migrate)} documents to migrate")

try:
# TODO: Complete the UpdateOne statement below
# build the UpdateOne so it updates the “lastupdated” field to contain
# the new ISODate() type
bulk_updates = [UpdateOne(
{"_id": movie.get(“doc_id”)},
{"$set": {“lastupdated”: movie.get(“lastupdated”)}}
) for movie in movies_to_migrate]

# here's where the bulk operation is sent to MongoDB
bulk_results = mflix.movies.bulk_write(bulk_updates)
print(f"{bulk_results.modified_count} documents updated")

except InvalidOperation:
print(“no updates necessary”)
except Exception as e:
print(str(e))

after the running the script getting the following messages with some errors:


File “movie_last_updated_migration.py”, line 32, in
for doc in cursor:
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\cursor.py”, line 1189, in next
if len(self.__data) or self._refresh():
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\cursor.py”, line 1104, in _refresh
self.__send_message(q)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\cursor.py”, line 931, in __send_message
operation, exhaust=self.__exhaust, address=self.__address)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\mongo_client.py”, line 1145, in _send_message_with_response
exhaust)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\mongo_client.py”, line 1156, in _reset_on_error
return func(*args, **kwargs)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\server.py”, line 85, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File “C:\ProgramData\Anaconda3\lib\contextlib.py”, line 112, in enter
return next(self.gen)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\pool.py”, line 1006, in get_socket
sock_info.check_auth(all_credentials)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\pool.py”, line 677, in check_auth
auth.authenticate(credentials, self)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\auth.py”, line 542, in authenticate
auth_func(credentials, sock_info)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\auth.py”, line 517, in _authenticate_default
return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\auth.py”, line 245, in _authenticate_scram
res = sock_info.command(source, cmd)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\pool.py”, line 579, in command
unacknowledged=unacknowledged)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\network.py”, line 150, in command
parse_write_concern_error=parse_write_concern_error)
File “C:\Users\LENOVO\AppData\Roaming\Python\Python37\site-packages\pymongo\helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Authentication failed.

Looking for guidance to sort out the issue.

Please follow forum guidelines and do not post code corresponding to potential answer.

This being mentioned. In the following the URI is wrong.

1 Like

Stuck with the below screenshot. No matter what I do. I have updated the host with the correct url:

  1. Connect to your Atlas cluster
  2. Find the doc whose _id corresponds to 573a13b8f29313caabd4c8c5 in the sample_mflix.movies collection
  3. What $type is the lastupdated field?

PS: It’s generally best to leave old threads alone and create your own

Hi @Anjani_Kumar_Agrawal_41728,

In addition to @007_jb,

Please make sure you update the field type of the lastupdated field to the expected type.

Thanks,
Shubham Ranjan
Curriculum Services Engineer

@007_jb… I have been filtering on { $type : “string”}. I checked Compass the type is mentioned as String. I tried String in Compass and got an error saying String alias not correct. Tried string in compass worked fine.
@Shubham_Ranjan The lastupdated field’s type is getting updated by existing code parser.parse and I did not change that.