$geoNear and $sort not working well together

So what i have is a geoNear aggregation followed by some $match then a sort at the end.

$geoNear: {
        near: {
          type: 'Point',
          coordinates: [
            longitude, latitude,
        distanceField: 'dist.calculated',
        maxDistance: 20000,
        spherical: true,
        distanceMultiplier: 0.001,

and a sort

        $sort: {
          score: -1,
          lastSeen: -1,

I have a 1 Million document, when i try to use geoNear alone, it gets executed in around 100 milliseconds.

However if i add the sort it takes more than 10 seconds.

If i try the sort alone without $geoNear it also gets executed in less than 100 milliseconds, but together its not working well. Any idea?