Mongos 7.0 slow on mongos,but fast on mongod

Other MongoDB Topics MongoDB Server Development
,
1

Hi. I’m using MongoDB 7.0 in my project.
I found the slow query msg in mongos log, it happened suddenly but periodically when I did the bench test （QPS is not high and the query is fast in mongsh shell). what should I do to fix it?
PS:3 mongo + 6 sharding
mongos log:

{"t":{"$date":"2023-08-24T16:27:20.680+08:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn39097","msg":"Slow query","attr":{"type":"command","ns":"bwms.blacklist","command":{"find":"blacklist","filter":{"mobile":{"$in":["88f25d36955e91914a24acd93a38c425ac675611"]},"labels":{"$elemMatch":{"ecid":0,"labels":{"$in":["TGHTC5SYGV4","TG63J19VLC0"]}}}},"hint":{"mobile":1},"limit":1,"maxTimeMS":10000,"projection":{"labels.$":1,"mobile":1},"$db":"bwms","$readPreference":{"mode":"primaryPreferred"},"lsid":{"id":{"$uuid":"94008ac1-0201-4304-939e-13d3ca01f458"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1692865639,"i":2}},"signature":{"hash":{"$binary":{"base64":"fnpFN9mxs2ag8epNnLlJNxTQleQ=","subType":"0"}},"keyId":7268229982285987858}}},"nShards":1,"nBatches":1,"cursorExhausted":true,"numYields":0,"nreturned":0,"queryHash":"6905B36F","reslen":228,"readConcern":{"level":"local","provenance":"implicitDefault"},"cpuNanos":660518,"remote":"172.16.215.25:43684","protocol":"op_msg","remoteOpWaitMillis":1,"durationMillis":806}}
{"t":{"$date":"2023-08-24T16:27:20.680+08:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn39090","msg":"Slow query","attr":{"type":"command","ns":"bwms.blacklist","command":{"find":"blacklist","filter":{"mobile":{"$in":["88f25d36955e91914a24acd93a38c425ac675611"]},"labels":{"$elemMatch":{"ecid":0,"labels":{"$in":["TGHTC5SYGV4","TG63J19VLC0"]}}}},"hint":{"mobile":1},"limit":1,"maxTimeMS":10000,"projection":{"labels.$":1,"mobile":1},"$db":"bwms","$readPreference":{"mode":"primaryPreferred"},"lsid":{"id":{"$uuid":"6414a0c3-51a5-423e-bc1b-1a4370a39338"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1692865639,"i":2}},"signature":{"hash":{"$binary":{"base64":"fnpFN9mxs2ag8epNnLlJNxTQleQ=","subType":"0"}},"keyId":7268229982285987858}}},"nShards":1,"nBatches":1,"cursorExhausted":true,"numYields":0,"nreturned":0,"queryHash":"6905B36F","reslen":228,"readConcern":{"level":"local","provenance":"implicitDefault"},"cpuNanos":660582,"remote":"172.16.215.25:43650","protocol":"op_msg","remoteOpWaitMillis":2,"durationMillis":711}}
{"t":{"$date":"2023-08-24T16:27:20.680+08:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn39103","msg":"Slow query","attr":{"type":"command","ns":"bwms.blacklist","command":{"find":"blacklist","filter":{"mobile":{"$in":["88f25d36955e91914a24acd93a38c425ac675611"]},"labels":{"$elemMatch":{"ecid":0,"labels":{"$in":["TGHTC5SYGV4","TG63J19VLC0"]}}}},"hint":{"mobile":1},"limit":1,"maxTimeMS":10000,"projection":{"labels.$":1,"mobile":1},"$db":"bwms","$readPreference":{"mode":"primaryPreferred"},"lsid":{"id":{"$uuid":"170a78cf-9a96-49c7-88e1-3c1a690fa409"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1692865639,"i":2}},"signature":{"hash":{"$binary":{"base64":"fnpFN9mxs2ag8epNnLlJNxTQleQ=","subType":"0"}},"keyId":7268229982285987858}}},"nShards":1,"nBatches":1,"cursorExhausted":true,"numYields":0,"nreturned":0,"queryHash":"6905B36F","reslen":228,"readConcern":{"level":"local","provenance":"implicitDefault"},"cpuNanos":731029,"remote":"172.16.215.25:43730","protocol":"op_msg","remoteOpWaitMillis":2,"durationMillis":542}}
2

help!!! I need some advice.

4

Hey @Hu_Peter,

As per the documentation , the client operations (such as queries) will appear in the log if their duration exceeds the slow operation threshold.

However, in order to better understand the issue here:

  • Could you share sample documents and the collection size?
  • The query you are executing on the same collection?
  • Do you have any indexes set up on this collection? If so, which ones?

Could you share more context on what type of bench test you are doing?

You mentioned query performance seems fast when run directly in the mongosh shell but is slow elsewhere. To clarify:

  • Where specifically are the queries running slow? From a driver application? From compass?
  • When running in mongosh, what are the typical response times you see?
  • Are the slow queries only happening under certain conditions like high QPS load?

More information regarding where the slowness occurs and example queries would be helpful for us to assist you better.

Regards,
Kushagra