I have a job that triggers the lambda and get the count from mongodb everyday. When i try to retrieve the data from mongodb using the below query {’$and’: [{’_id.a’: 402}, {‘t.u’: {’$gte’: datetime.datetime(2022, 6, 29, 22, 0, tzinfo=datetime.timezone.utc), ‘$lte’: datetime.datetime(2022, 8, 9, 22, 0, tzinfo=datetime.timezone.utc)}}, {’_id.s’: {’$gt’: 99999999}}]} . I’m getting two different counts. when the job ran , i got the count as 661 whereas the actual count was 340. But when i ran it separately in the morning i got the count as 340. No changes were made in the query. Even when the job ran the same query fetched the different count.
Could you help me with some informations for the above issue mentioned:
By Lambda do you mean AWS Lambda?
Can you post a code snippet that involves the problematic count? Currently in the post you only have the query but not the actual code. Would also be helpful to know the driver language you’re using.
Is there a pattern for the count appearing to different in different executions or server responding with out of date responses?
Yes it is AWS Lambda.Please find the code snippet below,
query = {’$and’: [{’_id.a’: int(asset_id)}, {
‘t.u’: {’$gte’: datetime.strptime(utc_from_date, ‘%Y-%m-%dT%H:%M:%S%z’),
‘$lte’: datetime.strptime(utc_to_date, ‘%Y-%m-%dT%H:%M:%S%z’)}}, {’_id.s’: {’$gt’: 99999999}}]}
print(query)
data = spots_col.aggregate([
{’$match’: query},
{’$count’: ‘count’}
])
nielsen_mongo_count_list = [d for d in data]
print(nielsen_mongo_count_list[0][‘count’])
I dont get any server out of date response…The Mongodb count was different when aws lambda ran when it was triggered by the batch job. but when i trigger the lambda manually it i got the correct count. The query that ran in mongodb was exactly same with all the correct paramters.
4.i’m using pymongo version pymongo==3.12.1
Thank you for sharing the above information, but I think we do not have enough information to determine what’s going on.
Could you help with a few more details which may be helpful:
What is the MongoDB server version you are using?
What is the deployment that you are working on? Is it an on-prem deployment? Or, are you using Atlas? Can you please share the connection string URI for the database? Please redact any passwords.
Just a sanity check, can you please confirm if you are triggering the same function with and without the automatic trigger?
While automatically triggering the function, do you see a wrong output in every execution consistently? If not, do you see any pattern with different responses received?
I think a better analysis could be made when we have more logging from the Lambda function’s execution. I would suggest adding more logging statements to the function, that way we can see whether the function was executed differently when it was manually triggered.
Aside from logging, perhaps you can also check out:
Try to create a simple function to count the documents, and trigger it manually and automatically to ensure that the counting discrepancy was not due to an issue in the original function.