ExceedTimeLimit error and not upgrading PyMongo version

Working with Data
,
1

I would like to let you know

  1. How many and which columns do we have
  2. What is the current error log (Based on the log please suggest if any index is required)
  3. MongoDB still showing PyMongo version 3.10.1 whereas the below command shows
    PyMongo Version 3.13 (Need help to properly upgrade PyMongo Version)
(env) ubuntu@playdb1:~$ sudo pip freeze |grep pymongo
pymongo==3.13.0
  1. Number of records in the document when the application is running fine and when it failed

Columns in the document (xxxxxx is the fabricated value)

db.user_messages.findOne()

{ "_id" : ObjectId("xxxxxxx"), "iid" : "xxxxxxx", "is_sender" : true, "_lm" : ISODate("2015-07-16T15:27:10.685Z"), "message_id" : ObjectId("xxxxxxxx"), "seq" : 1, "status" : "read", "thread_id" : ObjectId("xxxxxxxx"), "time_created" : ISODate("2015-07-16T15:27:10.685Z"), "time_read" : ISODate("2015-07-16T15:27:10.685Z"), "type" : "system", "user_id" : ObjectId("xxxxxxxxx"), "visibility" : "archived" }

Please see below the latest error log lines:

2023-03-31T06: 25: 12.932+0000 I COMMAND [conn39652] command local.$cmd command: **find { find:** "oplog.rs", filter: { ts: { $gte: Timestamp(1680243781,3) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 5, readConcern: { afterClusterTime: Timestamp(1680243781, 3) }, $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" }, $clusterTime: { clusterTime: Timestamp(1680243870, 1), signature: { hash: BinData(0, DDE58FBA0BEECDD6C6FF9677206EC95473E5BF23), keyId: 7213388223488196610 } }, $db: "local" } numYields: 0 ok: 0 errMsg: "**operation exceeded time limit**" errName:**MaxTimeMSExpired** errCode: 50 reslen: 665 locks: { ParallelBatchWriterMode: { acquireCount: { r: 3 } }, ReplicationStateTransition: { acquireCount: { w: 3 } }, Global: { acquireCount: { r: 3 } }, Database: { acquireCount: { r: 2 } }, Mutex: { acquireCount: { r: 2 } }, oplog: { acquireCount: { r: 2 } } } 
protocol:op_msg 36589ms 
2023-03-31T06: 25: 13.689+0000 I COMMAND [conn20369] command cureatr.$cmd command: **update { update**: "threads_garbage", ordered: true, lsid: { id: UUID("dfd49a79-f537-4ecd-952f-154d4d35364b") }, txnNumber: 4099331, autocommit: false, $clusterTime: { clusterTime: Timestamp(1680243863, 1), signature: { hash: BinData(0, 43057D869446C1D65E1CBCBEAE6F7227FFF446F6), keyId: 7213388223488196610 } }, $db: "cureatr" } numYields: 0 ok: 0 errMsg: "Transaction 4099331 has been aborted." errName:**NoSuchTransaction** errCode: 251 reslen: 306 locks: { ReplicationStateTransition: { acquireCount: { w: 1 } } } 
protocol:op_msg 42957ms 
2023-03-31T06: 30: 42.940+0000 I COMMAND [conn39720] command local.$cmd command: **find { find:** "oplog.rs", filter: { ts: { $gte: Timestamp(1680244080, 5) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 5, readConcern: { afterClusterTime: Timestamp(1680244080, 5) }, $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" }, $clusterTime: { clusterTime: Timestamp(1680244213, 1), signature: { hash: BinData(0, 86B28A16186A0F9FA16705757DCC5D617A664CD1), keyId: 7213388223488196610 } }, $db: "local" } numYields: 0 ok: 0 errMsg: "**operation exceeded time limit**" errName:**MaxTimeMSExpired** errCode: 50 reslen: 665 locks: { ParallelBatchWriterMode: { acquireCount: { r: 3 } }, ReplicationStateTransition: { acquireCount: { w: 3 } }, Global: { acquireCount: { r: 3 } }, Database: { acquireCount: { r: 2 } }, Mutex: { acquireCount: { r: 2 } }, oplog: { acquireCount: { r: 2 } } } 
protocol:op_msg 2011ms

PyMongo Version in the MongoDB log

2023-03-31T06:31:09.336+0000 I NETWORK [conn39802] received client metadata from 127.0.0.1:35286 conn39802: { driver: { name: "**PyMongo", version: "3.10.1**" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-1096-aws" }, platform: "CPython 3.7.5.final.0" }

There is an interesting part that the same code is working fine where records counts are less:

db.messages.count()
139774535

If failed when record count is double than above-mentioned count:

db.user_messages.count()
276214451
4

Hey :wave: @Shashank_Shekhar5,

Welcome to the MongoDB Community forums :sparkles:

So, to summarize the issue I have understood so far: when you run a findOne() command on a collection with a record count of 139774535, it works fine, but when you run the same command on a collection with a record count of 276214451, it throws an error.

errMsg: "**operation exceeded time limit**" errName:**MaxTimeMSExpired** errCode: 50

Let me know if I misunderstood the context here.

Furthermore, could you please share the following information to help us better understand the problem:

  • What version of MongoDB are you using? Could you please share the output of db.version() specifically?
  • Where is it deployed - on-premises or on MongoDB Atlas?
  • I noticed in the shared log that you are running an update() query. Could you provide more details about this query and also share the second find() query?
  • Have you tried running the same command from the mongo shell or MongoDB Compass? Is it also throwing the same error? If so, please share it with us.
  • Are you experiencing any detrimental effects in the app, such as lost queries, slow performance, or anything else?
  • Is the document structure consistent within the collection?
{ 
  "_id" : ObjectId("xxxxxxx"), 
  "iid" : "xxxxxxx", 
  "is_sender" : true, 
  "_lm" : ISODate("2015-07-16T15:27:10.685Z"), 
  "message_id" : ObjectId("xxxxxxxx"), 
  "seq" : 1, "status" : 
  "read", 
  "thread_id" : ObjectId("xxxxxxxx"), 
  "time_created" : ISODate("2015-07-16T15:27:10.685Z"), 
  "time_read" : ISODate("2015-07-16T15:27:10.685Z"), 
  "type" : "system", 
  "user_id" : ObjectId("xxxxxxxxx"), 
  "visibility" : "archived" 
}

Please follow the instructions here Installing / Upgrading - PyMongo 4.6.2 documentation to upgrade your pymongo version.

Best,
Kushagra