My find queries randomly return 0 result

Working with Data Node.js Frameworks
, , ,
1

I have these find queries in two different collections in the same database. They’re executed from a nodejs project (using mongoose) on a database that is hosted on Mongo atlas (Replica Set - 3 nodes).

The issue I have is that the queries return 0 result randomly. Would you know why? Note that the data requested are not edited between the query that works and the other one that doesn’t.

Would you have any idea?

2

No one can know with the information you supplied. And I would be very surprised that a query randomly works. So there is something else at stake that you are missing.

Please share the queries and documents from both collections that are supposed to match the queries. The culprit might also be the code that uses the queries so it would be helpful to also share it.

3

For sure I can provide more details. Actually here is my function

static async getHistoryData({ type, contract, areas, filters, timestamp, duration, isAwol, projection, lean = false, isUnavailable = false }) {
    return model.find(processFilter({ type, contract, areas, filters, timestamp, duration, isAwol, isUnavailable }), projection).lean(lean);
}

processFilter returns the following filter:

{
  "$and": [
    {
      "meta_data.data.area.id": {
        "$in": [
          "63f882df2d65a5368041937b",
          "6511a85fdbadd710cace1a28",
          "651d772aaa6d4dc89c14bfe2",
          "651d773aaa6d4dc89c14bff6",
          "651d774baa6d4dc89c14c00a",
          "651d7e4eaa6d4dc89c14c3ea",
          "651d7e5eaa6d4dc89c14c3fe",
          "651d7e75aa6d4dc89c14c412",
          "651d8455aa6d4dc89c14c613",
          "651d846baa6d4dc89c14c627"
        ]
      }
    },
    {
      "available_from": {
        "$lte": ISODate("2023-12-06T00:37:07.000Z")
      }
    },
    {
      "available_to": {
        "$gte": ISODate("2023-12-06T00:37:07.000Z")
      }
    },
    {
      "meta_data.data.area.type": {
        "$in": [
          "normal",
          "prm",
          "chargin_station"
        ]
      }
    },
    {
      "result": {
        "$ne": "unavailable"
      }
    }
  ]
}

it should return:

[
  {
    "_id" : ObjectId("656fc4f3be10b23d785f17f7"),
    "available_from" : ISODate("2023-12-06T00:36:50.000Z"),
    "available_to" : ISODate("2023-12-06T00:39:10.999Z"),
    "result" : "spot",
    "meta_data" : {
        "data" : {
            "area" : {
                "type" : "normal",
                "id" : "63f882df2d65a5368041937b"
            }
        }
    }
  },
...other results
]

but sometimes this kind of query returns 0 result and I can’t explain why.

4

The query you shared works all the time for me with the sample document you supplied.

You will need to share one that does not work.

Since your queries are built by a function, I suspect that the culprit is the function. Most likely there is an edge case that the function does not support and an invalid query is built. Or a valid query is built and there is no data matching the query.

Another possibility is that you read from a secondary and the data you expected to get with the query has not been replicated yet.

Storing an object id as its string representation is not optimal space wise and performance wise.

5

The thing is that this query (or another one) cans work and sometimes not. So I cannot provide a query that does not work.

It often has the following pattern (still have to determine if it is always the same one though):

1. query executed -> returns 0 documents 
2. query executed -> returns the expected document
3. query executed -> returns the expected document
...

I refactored my function like this:

try {
  const query = processToFilterHistoryData({
    type,
    contract,
    areas,
    filters,
    timestamp,
    duration,
    isAwol,
    isUnavailable,
  });
  if (Object.keys(query).length === 0) {
    throw new Error(`query is empty`);
  }
  const documents = await model.find(query, projection).lean(lean);
  return documents;
} catch (err) {
  throw new Error(
    `failed to process filter for history data: ${err} (${type}-${contract.ref_core_contract})`,
  );
}

But no error occur at all even when the query returns 0 document.

In my full algorithm, I perform 4 queries at the same time (in a Promise.all) using this function. 2 queries target the database where the live data are stored and the other 2 ones get data from the database that stores the “old” data.

But the queries that “fails” (i.e return 0 document) are always the ones that are performed on the database with the old data. Also, those data are coming from the last night so they’ve already been replicated (I suppose because they’ve been inserted for the first time 11-6 hours ago).

6

Assuming that

1 - you are certain that the query is the same on both DB
2 - you are certain that the data is present in the old DB

I can only conclude that the following code is misbehaving:

I am not familiar with Mongoose but I suspect that querying 2 DB with the same model might cause timing issue.

I would try to use native js driver find() rather than the Mongoose wrapper to investigate.

I think we are getting closer, because now we know that it does not fail randomly, as originally indicated, but that it works consistently with 1 DB and probably fails consistently on a 2nd DB when both are done almost simultaneously with the same global object.

7

With all the investigation we did here, I found out that when I delay my queries, it doesn’t work at all:

// before
Promise.all([query1(...args1), query2(...args2), query3(...args3), query4(...args4)]);

// after
await query1(...args1);
await query2(...args2);
await query3(...args3);
await query4(...args4);

With my queries delayed I found out that one of the argument I use is wrongly modified by another function. Wrongly is a way that it doesn’t follow the good practices and then, it breaks my algorithm.

By modifying the arguments handling in my other functions it now works perfectly so it was nothing related to mongo.

Thank you for you help @steevej !

1 Like