Atlas search indexes in automated tests on CI/CD no longer working

MongoDB Atlas Atlas Search
, , , ,
1

Hello,

I am having an issue with our automated testing pipeline for search implemented using an Atlas search index. We had initially set up the test using this article. The tests were operating as expected until July 21st.

The tests started failing Using docker image sha256:247341503d1d22610adf03bbf9652e32ef769acbadc2fb7a97c9951cf254676b for mongodb/mongodb-atlas-local:8.0.1 with digest mongodb/mongodb-atlas-local@sha256:817300f2924dca5e6e9ccc4d52ee3333a074a1e8b531b28233472c1ed739ab4a. Tests using the previous image (Using docker image sha256:dac7c0942089ba3195ffcf9890e76d28d0da3ade9a37b0bcb9f68a70c650d1b0 for mongodb/mongodb-atlas-local:8.0.1 with digest mongodb/mongodb-atlas-local@sha256:f4da4d31807ac09c7bb73315e74d88e97c4312d0401f97361d49109a5c4cbc9b) worked as expected.
I upgraded the image to the latest 8.0.4 but that did not resolve the issue.

Upon replicating the issue locally, I found that a 2 second timeout seemed no longer sufficient so I implemented polling to check if the index was ready and queryable. This resolved the issue locally, however, upon running the tests on CI/CD, I found that the index was always in a ‘DOES_NOT_EXIST’ status. Even dropping and recreating the index during the CI/CD would still result in DOES_NOT_EXIST.

For posterity here is the polling and recreation logic implemented so far:

Code 
export const waitForSearchIndex = async (params: {
  connection: Connection
  indexName: string
  maxWaitTime: number
}): Promise<void> => {
  const { connection, indexName = PARTICIPANT_SEARCH_INDEX, maxWaitTime = 10000 } = params
  // Initial wait for index propagation
  await setTimeout(500)
  if (!connection) {
    return
  }

  const startTime = Date.now()
  const pollInterval = 500

  while (Date.now() - startTime < maxWaitTime) {
    try {
      // Check if search indexes are ready
      const collection = connection.collection('participants')
      const indexes = (await collection
        .aggregate([{ $listSearchIndexes: { name: indexName } }])
        .toArray()) as SearchIndexStatus[]

      const targetIndex = indexes.find(idx => idx.name === indexName)

      if (targetIndex && targetIndex.queryable && targetIndex.status === 'READY') {
        console.log(`Search index '${indexName}' is ready and queryable`)
        return
      }

      if (!targetIndex || targetIndex.status === 'DOES_NOT_EXIST') {
        await createSearchIndexes({ connection, indexName })
      }

      console.log(
        `Waiting for search index '${indexName}' to be ready... Status: ${targetIndex?.status || 'NOT_FOUND'}`,
      )
    } catch (error) {
      console.warn('Error checking search index status:', error.message)
    }

    await setTimeout(pollInterval)
  }

  console.warn(`Search index '${indexName}' may not be ready after ${maxWaitTime}ms wait`)
}

/**
 * Creates MongoDB Atlas search indexes manually for testing
 * @param connection Mongoose connection to the database
 */
export const createSearchIndexes = async (params: {
  connection: Connection
  indexName: string
  maxWaitTime?: number
}): Promise<void> => {
  const { connection, indexName, maxWaitTime = 10000 } = params

  try {
    const collection = connection.collection('participants')

    // Check if index already exists
    const existingIndexes = await collection.aggregate([{ $listSearchIndexes: {} }]).toArray()
    const existingIndex = existingIndexes.find((idx: any) => idx.name === indexName)

    if (existingIndex) {
      if (existingIndex.status !== 'DOES_NOT_EXIST' && existingIndex.status !== 'NOT_FOUND') {
        console.log(`Search index '${indexName}' already exists`)
        return
      }
      await collection.dropSearchIndex(indexName)
    }

    // Wait for the index to be fully dropped
    let indexStillExists = true
    const startTime = Date.now()
    while (indexStillExists && Date.now() - startTime < maxWaitTime) {
      const indexes = await collection.aggregate([{ $listSearchIndexes: {} }]).toArray()
      indexStillExists = indexes.some((idx: any) => idx.name === indexName)
      if (indexStillExists) {
        await setTimeout(1000) 
      }
    }

    // Create the search index manually
    await collection.createSearchIndex(SEARCH_INDEX_DEFINITIONS[indexName])
    console.log(`Created search index '${indexName}' manually for testing`)
  } catch (error) {
    console.warn(`Failed to create search index: ${error.message}`)
    // Don't throw to prevent tests from failing if search functionality isn't available
  }
}

What changes have been made in the new images uploaded that may have resulted in this issue? Is there any specific configuration necessary to have atlas build indexes that may need to be enabled on the pipeline.

2

Hi @Mahad_Amir how are you starting up the local environment, are you using Docker or the Atlas CLI, in either case, can you share the command you’re using, please?

Ps. thanks for sharing the original article, this is now a little out of date and refers to an older implementation of Search via the Atlas CLI prior to Atlas Local Dev going into GA at the end of last year. I’ll work to get this updated.

Edit: here’s a more recent article on using Atlas Local to work with Search

3

Hey @Jonny_Roberts,

Thanks for providing the updated article! We managed to get things working. For the sake of anyone discovering this later down the line, the issue turned out to be caused by dropping and recreating the collection between each test. Changing that to just deleting all data from all the collections instead resulted in the indexes correctly updating.

1 Like