The cluster consists of 5 data shards + 1 config shard. Each shard has 3 data members (except for one which is a PSA). We have one sharded collection by range, and we have 5 zones.
The query that triggers the error message is a simple find command, using the shard key and other fields. It only occurs on queries that target one particular shard, and the error is visible on the logs of all the replicas in the shard.
All instances are running 5.0.5, except for the problematic shard which is running 5.0.9 (just noticed this), and a newer shard running 6.0.5 (set on 5.0 compatibility mode)
As far as I know we don’t use transactions. Unless they are used internally by the clients we use (PHP, Java, Node.JS). But for the query in question I would say no.
Thank you for any assistance you might be able to provide!
Mismatching between the versions in a sharded cluster is allowed unless they are from adjacent major MongoDB server releases however, a sharded cluster should have consistent versions.
Hence, I would suggest to keep all shard at the same versions to avoid any inconsistencies.
Along with the shard servers, I would also suggest to upgrade mongos and the config servers also to the same versions and same feature compatibility version.
Once you have performed the necessary upgrades, try running the similar query again and let us know if the issue persists.