Atlas search does not return document with must

Hello all!
I have an atlas search index with the following mappings:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "address": {
        "dynamic": false,
        "fields": {
          "city": {
            "analyzer": "custom_keyword_lowercase",
            "type": "string"
          },
          "region": {
            "analyzer": "custom_keyword_lowercase",
            "type": "string"
          }
        },
        "type": "document"
      },
      "friendlyId": {
        "type": "number"
      },
      "enabled": {
        "type": "boolean"
      },
      "name": {
        "analyzer": "custom_keyword_lowercase",
        "type": "string"
      },
      "phone": {
        "analyzer": "custom_keyword_lowercase",
        "type": "string"
      }
    }
  },
  "analyzers": [
    {
      "charFilters": [],
      "name": "custom_keyword_lowercase",
      "tokenFilters": [
        {
          "type": "lowercase"
        }
      ],
      "tokenizer": {
        "type": "keyword"
      }
    }
  ]
}

In general there was an issue, and one document that should be retrieved, it was not.
I began investigating the issue in Atlas Search test.

When i do the following query:

[
  {
    $search: {
      index: "search_shop",
      compound: {
        minimumShouldMatch: 1,
        must: [
          {
            equals: {
              path: 'enabled',
              value: true,
            },
          },
        ],
        should: [
          {
            equals: {
              path: 'friendlyId',
              value: 250272,
              score: { boost: { value: 2 } },
            },
          }
          ],
      }
    }
  }
]

It does not return the document(even though the boolean field enabled of the document is true). If i change the value of friendlyId to 250271 or 380 it does return the document(also 250273 is not working, most likely there are more). All these documents have enabled field to true, so i expect that all these documents are returned with the must property of the compound.

I do not know why this is happening. The strange thing is the following:
If i delete the must property of the compound object, the document 250272 is returned.

So does anybody know why this is happening? What am i doing wrong?

Hi @vasilis_m0 , can you share the document which are matching and which are not? Also, are there any other should clauses in your query? I noticed you have specified minimumShouldMatch: 1 but since there is only one should clause in your query, only documents where friendlyId: 250272 will be returned.

If you don’t have other should clauses that you want to specify, can you try removing the minimumShouldMatch line?

Hello, thanks a lot for the the answer!

I noticed you have specified minimumShouldMatch: 1 but since there is only one should clause in your query, only documents where friendlyId: 250272 will be returned.

Yes thats what i would want, but 250272 is not returned. If i change the query and give another friendlyId(say 250271) it does work and it returns the document with 250271. Removing the minimumShouldMatch: 1 does not see to have the behavior that i would want because it returns also documents with friendlyId not equal to 250272.

Query 1: With the query in the OP, i would expect to return the document:

{
  enabled: true,
  friendlyId: 250272,
  ...
}

But it does not.

Changing the query to the following and changing the friendlyId to 250271, it does return the expected result:

{
  enabled: true,
  friendlyId: 250271,
  ...
}

All these queries are tested in Atlas search tester, thus there i always provide the should clause.

The strange thing is when i remove the must clause(enabled true), Query 1. outputs the correct result. But that document has enabled: true.

By removing the must operator i meant:

[
  {
    $search: {
      index: "search_shop",
      compound: {
        minimumShouldMatch: 1,
        should: [
          {
            equals: {
              path: 'friendlyId',
              value: 250272,
              score: { boost: { value: 2 } },
            },
          }
          ],
      }
    }
  }
]

Which works but i really want to that must clause in order to get only documents that are enabled.

Any ideas?

Hi, apologies for the delay here, I’m having a hard time reproducing the issue. What happens if you add the below operator into the must clause instead of within should?

{
   equals: {
      path: 'friendlyId',
      value: 250272,
      score: { boost: { value: 2}}
   }
}

And just to double check, what is the data type of “friendlyID” stored within MongoDB? Are they all the same, or are some values different?

Hello,

Apologies for my super late response.

We were able to solve the issue with the Mongodb customer support team.

In a nutshell, the problem was that for some documents some fields were duplicate and I was told that this is an undefined behaviour in Mongodb and it is not supported in newer versions.

Specifically, there is also a phone field in the atlas search mappings. There were some documents in our collection that had that field twice. It was really weird and that problem originated from using Robo 3t. We were able to delete the duplicate fields with Robo 3T(we could see the fields twice there but we could not see them from other software, e.g. Studio 3T or Compass).

Thanks a lot!