Reset replica set version number using rs.reconfig()

Hi

Previously , rs.reconfig was ran with force: true and now I am having an arbitrary large number as a version number. I need to reset this number to a number within the allowed range.

I tried to run rs.reconfig(cfg) where cfg has the following:

cfg = rs.conf();
cfg.version=1;
rs.reconfig(cfg);

I confirmed cfg now has version = 1 . However, after rs.reconfig(cfg) version reverted back to old version number and I keep getting the out of range issue as before

rs.reconfig(cfg)
{
	"operationTime" : Timestamp(1659609419, 4),
	"ok" : 0,
	"errmsg" : "version field value of 2147483648 is out of range",
	"code" : 103,
	"codeName" : "NewReplicaSetConfigurationIncompatible",
	"$gleStats" : {
		"lastOpTime" : Timestamp(0, 0),
		"electionId" : ObjectId("7fffffff0000000000000045")
	},
	"lastCommittedOpTime" : Timestamp(1659609419, 4),
	"$configServerState" : {
		"opTime" : {
			"ts" : Timestamp(1659609434, 2),
			"t" : NumberLong(27)
		}
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1659609434, 2),
		"signature" : {
			"hash" : BinData(0,"hYlj4D5wuCoTuTjjLwNQRyqxbvQ="),
			"keyId" : NumberLong("7107200311055351831")
		}
	}
}

I don’t know why it’s ignoring the version number I supplied.

Hi @Reab_AB,

It sounds like you are experiencing an issue similar to replSetReconfig force generates too high version number.

It is expected that a force reconfiguration will increment the replica set config version but
that should be in the order of 10s or 100s of thousands, not millions. Replica set members compare the version number and replica set name to determine if their configuration is stale, so you cannot force an older version number.

Can you provide more details on your environment:

  • specific version of MongoDB server
  • O/S version
  • current version of the replica set config
  • steps to reproduce (for example, was this a single force reconfig or the result of several)

Thanks,
Stennie

1 Like

If we look at the documentation for rs.reconfig() we see the following text:

Replica set members propagate their replica configuration via heartbeats. Whenever a member learns of a configuration with a higher version and term , it installs the new configuration.

Since the server only looks for larger numbers for updating the new configuration, it will not update the version to a lower number as that could cause issues.

There are ways of rebuilding your replica set without losing your data. If not done properly this could result in loss of data, or a complete failure of your database system. You will want to test any potential process for resetting the replica set version thoroughly before running on a database that is storing data that you care about.

It will be interesting to see the answers to the questions that Stennie asked above.

1 Like

Hi @Stennie_X
Mongodb Server version is 4.4
Current version of replica set is 2147483648
O/S oracle linux 7.9
I am using Kubernetes operator, so this is the result of several rs.reconfig with force: true. Now the operator is fixed and it’s not using force flag, but I need to reset the version number in the replica set

Thanks

How to rebuild the replica set? I have a sharded replica set.