Unexpected and occasional behavior: bitbucket pipelines + atlas search

MongoDB Atlas
,
#1

Hey guys,

So, I use mongo atlas search.
I have pytest cases all over the place to make sure that things are working as expected.

Here is a sample endpoint test case

def test_list_obj(self, test_profile) -> None:
        response = self.endpoint.get()
        assert response.status_code == 200
        assert len(response.json()) == 3
        for received in response.json():
            assert_list_obj_fields(
                received,
                ObjModel.objects.get(id=received["id"]),
            )

Here is the sample atlas search query we do in the api:

    search_query = {
        "compound": {
            "must": [
                {"text": {"query": OPEN_OBJ_CONSTANT, "path": PATH0}},
            ]
        },
    }

    if search_title is not None:
        search_query["compound"]["should"] = [
            {
                "autocomplete": {
                    "query": search_title,
                    "path": PATH1,
                    "fuzzy": {"maxEdits": 1}
                }
            },
            {
                "autocomplete": {
                    "query": search_title,
                    "path": PATH2,
                    "fuzzy": {"maxEdits": 1}
                }
            },
        ]
        search_query["compound"]["minimumShouldMatch"] = 1

 
    obj_search = obj_search.aggregate(
        [
            {"$search": search_query},
            {"$skip": start},
            {"$limit": size},
            {
                "$project": {
                    "_id": 0,
                }
            },
        ],
    )

I refactored the code and removed sensitive data, meaning I may have made punctuation errors in doing so.

Either way, when I run tests locally, everything and I mean every 120 endpoint test cases related to atlas search work well.

However, when these tests are run from bitbucket pipelines, things are uncertain.

  • There will be few weeks straight when every test case will pass well from bitbucket pipelines
  • Then there will be the other few weeks when there will always be something to fail. Sometimes test A will fail, other times test B or C or D. But something will surely fail.
    Meanwhile, those tests are all passing from every developers local environment.
  • Then again few weeks all will work well
  • Then again something failing for few weeks in straight

Bitbucket IPs are added to Mongo’s network connection access point.
Bitbucket devs have gotten back to me showing that there has been no network drops between test cases or during test cases that might have caused the issue.

This issue is becoming very annoying. Recently I saw that usually things begin to behave differently after there is a new mongo update.

Has anyone ever faced this issue?
What is the resolution? How can I achieve consistency here?

#3

Hello @Aleksandre_Bregadze ,

I notice you haven’t had a response to this topic yet - were you able to find a solution?

It seems that you are experiencing inconsistency in the results of your Atlas Search tests when running them from Bitbucket Pipelines compared to running them locally. While the tests consistently pass on your local environment, they intermittently fail on Bitbucket Pipelines.

To better understand the issue, it would be helpful to gather more information:

  1. Could you provide more details on the nature of the test failures? Are the tests failing due to network issues or something else? Any specific error messages or stack traces would be valuable in diagnosing the problem.
  2. Do the same tests consistently fail or do the failing tests change between each run? Identifying any patterns in the failures may help pinpoint potential causes.
  3. Are the tests from Bitbucket using the same database and collections as your local tests? It’s important to ensure that the test data is consistent between the two environments. Are the test collections populated with data, or is the data being populated as part of the test execution?
  4. How are the tests executed in Bitbucket Pipelines? Understanding the test setup, environment configuration, and any specific steps taken during the test execution can shed light on possible failure points.
  5. Have you tried running the same workload and tests on other providers, such as GitHub, to see if similar issues arise? This could help determine if the problem is specific to Bitbucket Pipelines or if it’s a more generalized issue.

Note: Since the tests consistently pass locally, the possibilities for failure lie within the Bitbucket Pipelines environment itself. It could be related to Bitbucket infrastructure, test setup, or other factors specific to that environment.

Best Regards,
Tarun