$geoNear and $sort not working well together

Working with Data
1

So what i have is a geoNear aggregation followed by some $match then a sort at the end.

$geoNear: {
        near: {
          type: 'Point',
          coordinates: [
            longitude, latitude,
          ],
        },
        distanceField: 'dist.calculated',
        maxDistance: 20000,
        spherical: true,
        distanceMultiplier: 0.001,
      },

and a sort

{
        $sort: {
          score: -1,
          lastSeen: -1,
        },
      },

I have a 1 Million document, when i try to use geoNear alone, it gets executed in around 100 milliseconds.

However if i add the sort it takes more than 10 seconds.

If i try the sort alone without $geoNear it also gets executed in less than 100 milliseconds, but together its not working well. Any idea?