Python reference causing error in queries

There is an issue with the queries which I was trying to do in pymongo that when I was inserting many dictionaries present in a list. Then due the reference behaviour of python it is modifying my original list due to which I was facing some issues but then I used deepcopy before inserting but I’m thinking can it be fixed internally?

Hi @Jai_Bhardwaj - welcome to the MongoDB community forum!

I can really feel your pain with this issue, and I’m glad that you’ve managed to fix it. Sometimes there’s a need to use deepcopy for this sort of problem, but I’ve found over the years that I very rarely need to use copy or deepcopy if I’m careful about how I build up the items in my lists and dicts.

With regards to your proposal of fixing it inside MongoDB, I don’t think that’s possible. PyMongo doesn’t modify any of the contents of the query, so it would be too late to do a deepcopy on the query. The copy operations need to be done while the query is being constructed, not when it is being traversed as part of executing the query.

If it was possible to solve the problem within the PyMongo driver, I’m not sure this would be desirable - deepcopy is an expensive operation and in most cases isn’t required. I think many of our users would be upset by the slower performance of the driver that isn’t in many cases offering any benefit to them.

Mark

Hi @Mark_Smith thanks for your reply!

I would like u to know more about the issue and please let me know what is wrong in my approach.

First I’m getting a list of dictionaries in a post request then i’m passing this list to a function which saves all the dicts in my collection inside the MongoDB, but then I’m sending this list to another function for the processing and the other function is doing some json.dumps on the list and what I’m getting is Object ID not serializable i.e, it is adding _id key in every dict of the list which I don’t
even need.

Is there another way or approach that I can use here?
like i can do the processing first then save it but what if I want to save it first?

For an explanation of the _id behavior, see Frequently Asked Questions - Why does PyMongo add an _id field to all of my documents?

then I’m sending this list to another function for the processing and the other function is doing some json.dumps on the list and what I’m getting is Object ID not serializable

You can use json_util.dumps() and json_util.loads() to JSON encode/decode documents containing bson types like ObjectId:
https://pymongo.readthedocs.io/en/latest/api/bson/json_util.html#module-bson.json_util

Alternatively you can remove the _id field that pymongo adds if you truly don’t need it.

2 Likes

I had no idea PyMongo did this! :flushed:

Sorry for the incorrect reply @Jai_Bhardwaj, and thanks for correcting my mistake @Shane

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.