Troubleshoot mongot Deployment

Gather Diagnostic Information

Before you work through a specific scenario, collect the current state of the mongot pod, its logs, and recent Kubernetes events:

# Check the status and restart count of the mongot pod
kubectl get pods -n ${MDB_NS} --context ${K8S_CTX} | grep search
# Review the most recent mongot logs
kubectl logs statefulset/${MDB_RESOURCE_NAME}-search-0 \
  -n ${MDB_NS} --context ${K8S_CTX} --tail=100
# Inspect the MongoDBSearch resource status and conditions
kubectl describe mongodbsearch ${MDB_RESOURCE_NAME} \
  -n ${MDB_NS} --context ${K8S_CTX}
# List recent events in the namespace
kubectl get events -n ${MDB_NS} --context ${K8S_CTX} \
  --sort-by='.lastTimestamp'

If the pod restarts in a loop, view the logs from the previous container instance to capture the error that caused the restart:

kubectl logs statefulset/${MDB_RESOURCE_NAME}-search-0 \
  -n ${MDB_NS} --context ${K8S_CTX} --previous

Localize a Problem Using the Resource Status

A MongoDBSearch resource runs several independent components:

• One or more mongot StatefulSets

• An optional Kubernetes Operator-managed Envoy load balancer

• An optional Cloud Manager or Ops Manager metrics forwarder.

In a multi-cluster deployment, each of these runs per member cluster. When search does not behave as expected, use the resource status to localize the problem to a specific cluster and component before you inspect Pods or logs.

To read the status:

kubectl get mongodbsearch ${MDB_RESOURCE_NAME} \
  -n ${MDB_NS} --context ${K8S_CTX} -o yaml

Start with the top-level fields, then narrow down using status.clusters:

The following example shows a two-cluster deployment where the second cluster's Envoy load balancer is stuck mid-rollout. The resource is Pending even though mongot on both clusters is healthy because the managed load balancer's readiness affects the overall phase.

status:
  phase: Pending
  message: "Waiting for managed load balancer to be ready"
  clusters:
    - name: member-cluster-1
      index: 0
      search: Running
      loadBalancer: Running
    - name: member-cluster-2
      index: 1
      search: Running
      loadBalancer: Pending
      loadBalancerMessage: "Load balancer deployment mdb-search-search-lb-1
        rollout in progress: 1 unavailable replica(s)"

Use the name and index of the affected cluster to inspect the corresponding component:

• For a search sub-phase that is not Running, inspect that cluster's mongot StatefulSet and Pods. The searchMessage typically reports which StatefulSet is not ready. See The mongot Pod Fails to Start.

• For a loadBalancer sub-phase that is not Running, inspect that cluster's Envoy Deployment. Messages such as rollout in progress or unavailable replica(s) indicate the new Envoy Pods are not scheduling or becoming ready.

• For a metricsForwarder sub-phase that is not Running, inspect that cluster's metrics forwarder Deployment.

To collect the Pod state, logs, and events for the identified component, see Gather Diagnostic Information.

The mongot Pod Fails to Start

Symptoms: The mongot pod stays in Pending, ContainerCreating, or Error and never reaches Running, or the MongoDBSearch resource does not reach the Running state.

Diagnose: Run the commands in Gather Diagnostic Information and review the pod events and MongoDBSearch status. Common indicators include scheduling failures, volume-mount errors, and image-pull errors.

Resolve:

• If the pod cannot be scheduled, confirm that a node with sufficient CPU and memory is available and your StorageClass can bind the requested Persistent Volume Claim. To learn more about sizing mongot, see Search & Vector Search Resource Planning and Sizing.

• If the password Secret for the search-sync-source user is missing, create it. The MongoDBSearch resource expects a Secret named ${MDB_RESOURCE_NAME}-search-sync-source-password.

• If the mongot container image cannot be pulled, verify the image name and your registry credentials. Check for image-pull errors with kubectl get events -n ${MDB_NS} | grep -i pull.

• If the source mongod replica set is not in the Running state, resolve that issue first. The MongoDB Controllers for Kubernetes Operator waits for the source resource before it deploys mongot.

Search Queries Fail to Reach mongot

Symptoms: $search, $searchMeta, or $vectorSearch queries fail with an error that indicates mongod cannot reach the search service.

Diagnose: Confirm that the mongot pod is Running and that its service exists:

kubectl get pods,svc -n ${MDB_NS} --context ${K8S_CTX} \
  | grep search

Review the mongot logs for connection or authentication errors, and review the mongod logs for errors that reference the search host.

Resolve:

• If the mongot pod is not Running, work through The mongot Pod Fails to Start.

• If the logs show authentication errors, verify the search-sync-source user credentials and role. To learn more, see Secure the Connection from Search to MongoDB.

• If the logs show TLS errors, confirm that mongod and mongot trust the same CA. To learn more, see Secure the Connection from MongoDB to Search.

mongot Repeatedly Restarts

Symptoms: The mongot pod shows a high restart count or reports CrashLoopBackOff.

Diagnose: View the logs from the previous container instance to capture the error that triggered the restart:

kubectl logs statefulset/${MDB_RESOURCE_NAME}-search-0 \
  -n ${MDB_NS} --context ${K8S_CTX} --previous

Check the pod's last state and exit reason:

kubectl describe pod ${MDB_RESOURCE_NAME}-search-0-0 \
  -n ${MDB_NS} --context ${K8S_CTX}

Resolve:

• If the exit reason is OOMKilled, increase the memory resources for mongot. See mongot Runs Out of Memory.

• If the logs show a configuration or authentication error, correct the setting and let the MongoDB Controllers for Kubernetes Operator reconcile the change.

• If the logs show that mongot cannot read or write its index data, confirm that the Persistent Volume Claim is bound and writable. See The mongot Pod Fails to Start.

mongot Runs Out of Memory

Symptoms: The mongot pod is terminated with an OOMKilled reason and the restart count increases.

Diagnose: Confirm the pod's termination reason:

kubectl describe pod ${MDB_RESOURCE_NAME}-search-0-0 \
  -n ${MDB_NS} --context ${K8S_CTX}

mongot is a Lucene-based, memory-mapped workload. Memory pressure most often results from undersized memory limits relative to index size and query load.

Resolve:

• Increase the memory resources allocated to mongot in the MongoDBSearch resource. mongot restarts to apply the change.

• Review your index size and query load against the sizing guidance in Search & Vector Search Resource Planning and Sizing.

Initial Sync Is Slow or Replication Lag Grows

Symptoms: Newly created indexes stay in a building state for a long time, or query results lag behind recent writes to mongod.

Diagnose: Review the mongot logs for replication progress and errors. Use the metrics that you configured in Monitor Your Deployment to track index build progress and replication lag over time.

Resolve:

• Allow the initial sync to complete. The time to build an index scales with the size of the source data.

• If lag persists under steady load, the mongot instance might not have sufficient resources for the write volume. Review Search & Vector Search Resource Planning and Sizing.

• Confirm that storage performance meets the recommendations for a random-read-heavy Lucene workload. Slow storage is a common cause of sustained lag.

TLS Handshake Failures Between mongod and mongot

Symptoms: The mongot logs show TLS handshake or certificate-validation errors, and mongod cannot establish a connection to mongot.

Diagnose: Review the mongot logs for certificate errors and confirm that the TLS Secrets referenced by the MongoDBSearch resource exist:

kubectl get secrets -n ${MDB_NS} --context ${K8S_CTX} \
  | grep -i tls

Resolve:

• Confirm that mongod and mongot trust the same CA.

• After you replace or rotate a certificate, restart mongot so that it reads the new certificate. mongot reads certificates at startup and does not reload them while running.

• To learn more, see Secure the Connection from MongoDB to Search.

Automated Embedding Fails

Symptoms: Indexes that use automated embedding fail to build, or embedding requests return errors in the mongot logs.

Diagnose: Review the mongot logs for errors that reference the embedding provider, such as authentication failures or request timeouts.

Resolve:

• Verify that your Voyage AI API key is valid and that the Secret that holds it exists in the namespace.

• Confirm that mongot can reach the embedding provider endpoint from your Kubernetes cluster.

Capture Diagnostics for Support

If you contact MongoDB Support, include the mongot logs, the MongoDBSearch resource description, and recent namespace events:

kubectl logs statefulset/${MDB_RESOURCE_NAME}-search-0 \
  -n ${MDB_NS} --context ${K8S_CTX} > mongot.log
kubectl describe mongodbsearch ${MDB_RESOURCE_NAME} \
  -n ${MDB_NS} --context ${K8S_CTX} > mongodbsearch-describe.txt
kubectl get events -n ${MDB_NS} --context ${K8S_CTX} \
  --sort-by='.lastTimestamp' > events.txt

You can also include FTDC files from each affected search pod. mongot writes FTDC files to /mongot/data/diagnostic.data/ inside the pod.

The /mongot/data path is a PersistentVolumeClaim that the Kubernetes Operator mounts on the pod through a volumeClaimTemplate on the search StatefulSet, so the diagnostic data persists across pod restarts. When a pod is replaced, the StatefulSet reattaches the same PersistentVolumeClaim, so the diagnostic data is retained.

Copy the FTDC data from the search pod to your local machine. For sharded clusters, copy the FTDC data from each per-shard search pod to your local machine. Repeat for every shard:

kubectl cp -n ${MDB_NS} --context ${K8S_CTX} \
   ${MDB_SEARCH_RESOURCE_NAME}-search-0-${MDB_EXTERNAL_SHARD_0_NAME}-0:/mongot/data/diagnostic.data \
   ./mongot-diagnostic.data-${MDB_EXTERNAL_SHARD_0_NAME}

Learn More

• Monitor Your Deployment
• mongot Deployment Limitations