Optimizing Data Retrieval for the MongoDB Queries

db().collection('Webhookalerts').aggregate([
      {
        $addFields: {
          timestamp: {
            $toDate: "$timestamp"
          }
        }
      },
    {
      $match: {
        timestamp: {
          $gte:  new Date();,
          $lte: new Date()
        }
      }},
      {$sort: {timestamp: -1}},
      {
        $facet: {
         
         TotalCount: [
            { $count: "Totalcount" }
          ],
  event_log: [
            { $skip: 0 },
            { $limit: 500000 }
          ]
        }
      }
    ],{ allowDiskUse: true }).toArray() 

this query does not fetch 5lakh data so how can fetch and optimize

If you need to do

in most of your use-cases, you should be storing your timestamp as date rather than string. Anyway dates as Date are more efficient to store and compare and provide a richer date oriented API.

Since you $match and $sort on the computed timestamp your indexes are useless.

Finally, returning 500_000 document is not very efficient. You should leverage more all the power of the aggregation pipeline. I am pretty sure, it is not a human that looks at the 500_000 documents so what ever computation or filtering you do after could be done in the server.

ok thanks for your suggestion

 db.alerts.aggregate([ {
      $addFields: {
        timestamp: {
          $toDate: "$timestamp"
        }
      }
    }, 
  {
    $match: {
      timestamp: {
        $gte: new Date(Date.now() - 24* 60 * 60 * 1000),
        $lte: new Date() 
      },
    }
  },{
        $facet: {
          Total_log: [
            { $count: "Total_log" }
          ] } }] ,{ allowDiskUse: true }).toArray();

In this query I want to fetch data last 24 hours but it getting huge time to fetch in my collection data approx 300 million so i use an index also but it taking so much time

You are thanking me for the suggestion but you do not apply it. You still find things slow but you still do things with $addFields. I wrote

and you still think that

I will repeat your indexes are useless because you $match on a computed field. One thing you can do despite your unwillingness to store your timestamp in the appropriate format is to match using the string version of your 2 new Date(). It will still be much slower than using the appropriate format for timestamp. Because string comparison of date is much slower than comparing timestamp in the appropriate format.

See