Restore Ops Manager from a Secondary Ops Manager

Considerations

Prerequisites

• Restoration Mode is enabled by default in Ops Manager 8.0.24 and later. If you disabled it, re-enable it on the primary Ops Manager before you restore. For the steps, see Configure a Secondary Ops Manager to Back Up Ops Manager.

• Confirm that the secondary Ops Manager has a completed snapshot and a continuous point-in-time recovery window for the backing databases.

• Confirm that you preserved the required per-host state for recovery, including the gen.key file from the original primary Ops Manager installation.

Preserve Per-Host State

A full primary Ops Manager recovery requires more than the application database. Preserve the following state on each host, in addition to the application database that the secondary Ops Manager backs up:

Restore the Primary Ops Manager

Restoration Mode and Reconciliation

After you restore the application database and start the primary Ops Manager, the primary Ops Manager compares each MongoDB Agent's configuration version with the restored configuration version. If an agent reports a later version, the primary Ops Manager enters Restoration Mode for that project and reconciles the configuration automatically:

• The primary Ops Manager isolates the project. It shows a Restoration Mode banner, returns an unchanged response to agent polls, and blocks deployment changes from the user interface and the API until reconciliation completes.

• The primary Ops Manager collects the configuration version from each agent, selects the agent with the latest configuration, and writes that configuration to the application database as the authoritative configuration.

• The primary Ops Manager exits Restoration Mode for the project and resumes normal operation. Backups resume automatically.

This reconciliation prevents a split-brain scenario. It converges all agents on a single authoritative configuration before any agent receives a new configuration.

When you roll the application database back to an earlier point in time, the restored configuration no longer includes deployment changes that you made after the restore point. Without reconciliation, agents receive the older configuration and stop the processes that the configuration no longer references. The impact depends on the deployment change:

Reconciliation prevents these outcomes. The primary Ops Manager converges all agents on the latest configuration and enqueues an on-demand snapshot before it exits Restoration Mode, which gives you a coherent backup point.

Validate the Restored Ops Manager

Confirm that the restored primary Ops Manager is healthy:

• Confirm that the MongoDB Agents reconnect and report as healthy.

• Confirm that automation, backup, and monitoring resume for your managed deployments.

• Confirm that the primary Ops Manager exits Restoration Mode. To check the Restoration Mode status, send a GET request to the following endpoint as a user with the Project Read Only role. The response shows the current state, the trigger reason, and timestamps for the project:

  GET /api/public/v1.0/groups/{PROJECT-ID}/restorationMode

Recover From a Stuck Reconciliation

If the primary Ops Manager restarts while it is in Restoration Mode, it re-triggers reconciliation on the next MongoDB Agent poll. You don't need to take action for the restart case.

Reconciliation might not complete, for example because agents are unreachable. In that case, use one of the following API endpoints as a user with the Project Owner role:

• To retry reconciliation, send a POST request to the following endpoint. The retry resets the reconciliation failure counter and re-runs reconciliation without exiting Restoration Mode:

  POST /api/public/v1.0/groups/{PROJECT-ID}/restorationMode/retry

• To force the primary Ops Manager to exit Restoration Mode and accept the restored configuration, send a DELETE request to the following endpoint:

  DELETE /api/public/v1.0/groups/{PROJECT-ID}/restorationMode

Cut Over to the Restored Ops Manager

This pattern restores the primary Ops Manager in place. It doesn't promote the secondary Ops Manager to replace the primary Ops Manager.

After you validate the restored primary Ops Manager, complete the cutover:

• Update the URL to Access Ops Manager setting and any DNS records to point to the restored primary Ops Manager.

• Confirm that the MongoDB Agents connect to the restored primary Ops Manager. The agents reconnect without re-registration when the mmsGroupId and mmsApiKey in their configuration match the restored project records.

Operational Guidance

Use the following practices to operate and validate this pattern over time.