Getting replica set status via arbiter node

Joanne · May 7, 2020, 10:47pm

In MongoDB server version: 3.4.16 we are able to get the data via arbiter node but in same environment but MongoDB server version: v3.6.9, it is throwing error. Any reason, why? No config of replica set is changed

MongoDB shell version v3.6.9

# mongo  --host xxx.xxx.xxx.xx --port xxxxx --ipv6  --eval 'printjson(rs.status())'
MongoDB shell version v3.6.9
connecting to: mongodb://xxx.xxx.xxx.xx:xxxxx/
Implicit session: session { "id" : UUID("xxxxx-xxxxx-xxxxx-xxxxx-xxxxx") }
MongoDB server version: 3.6.9
{
        "ok" : 0,
        "errmsg" : "not authorized on admin to execute command { replSetGetStatus: 1.0, lsid: { id: UUID(\"xxxxx-xxxxx-xxxxx-xxxxx-xxxxx\") }, $db: \"admin\" }",
        "code" : 13,
        "codeName" : "Unauthorized"
}

MongoDB shell version v3.4.16

# mongo --host xxx.xxx.xxx.xx --port xxxxx  --ipv6  --eval 'printjson(rs.status())'
MongoDB shell version v3.4.16
connecting to: mongodb://xxx.xxx.xxx.xx:xxxxx/
MongoDB server version: 3.4.16
{
        "set" : "set01",
        "date" : ISODate("YYYY-MM-DDT17:25:54.460Z"),
        "myState" : 7,
        "term" : NumberLong(1),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                :                              
                :                              
                :                              
                :                              
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "hostname0:xxxxx",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                :                              
                :                              
                :                              
                :                              
                },
                {
                        "_id" : 1,
                        "name" : "hostname1:xxxxx",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                :                              
                :                              
                :                              
                :                              
                },
                {
                        "_id" : 2,
                        "name" : "hostname2:xxxxx",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 681823,
                :                              
                :                              
                :                              
                :                              
                }
        ],
        "ok" : 1
}

Stennie_X · May 7, 2020, 11:45pm

Hi,

I cannot reproduce this issue using MongoDB 3.6.9 with auth enabled.

Can you check that you are definitely connecting to an arbiter via:

mongo --host ... --port ... --eval "db.isMaster().arbiterOnly"

The arbiterOnly result should be true if you are connected to an arbiter.

Arbiters do not replicate any data (including user/auth data), but another possibility is that someone has manually created users to secure your arbiter.

Regards,
Stennie

Joanne · May 8, 2020, 12:51pm

This returned true. But how come this command db.isMaster() returned the result but rs.status() doesn’t? The authentication is enabled in the environment.

mongo --host xxx.xxx.xxx.xx --port xxxxx --ipv6 --eval 'printjson(rs.status())' —> This resulted in error
mongo --host xxx.xxx.xxx.xx --port xxxxx --ipv6 --eval "db.isMaster().arbiterOnly" → This returned true
How? Why? Sorry for being naive. But need to understand it better.
Is there any list of commands that work only with authentication enabled or regardless of authentication enabled they work too? Or am I understanding it wrong?

Doug_Duncan · May 8, 2020, 1:28pm

Does your user have access to run admin commands (commands in the admin database)?

The db.isMaster() shell helper runs db.runCommand( { isMaster: 1 } ) behind the scenes.

The rs.status() command runs db.adminCommand( { replSetGetStatus: 1, initialSync: 1 } ) behind the scenes.

Notice the the db.adminCommand(...) for rs.status() as opposed to db.runCommand(...) for db.isMaster().

If the user you are logging in with cannot run queries in the admin database then that would explain the errors you are receiving.

It would be helpful to see that user’s privileges and roles.

Joanne · May 8, 2020, 2:20pm

Hi Doug,

We are not specifying username in command. We are running the below commands by ssh’ing to arbiter ip and executing the commands

If I run db.listCommands() and in output I get whole list of db commands available:
Can you let me know if my understanding is correct

splitChunk: adminOnly → only returns result if Authentication is enabled, runs db.adminCommand and should be run from primary host?
splitVector: . → returns result regardless of authentication enabled or not and should be run from primary host?
startSession: slaveOk → returns result regardless of authentication enabled or not and can be run from either primary or secondary host?
top: adminOnly slaveOk . → only returns result if Authentication is enabled, and can be run from either primary or secondary host?

Really appreciate your help on this!
Thanks in advance.

Doug_Duncan · May 8, 2020, 4:24pm

Sorry about that. I remembered @Stennie_X mentioning auth earlier in the thread and didn’t look at the exact queries being run.

I have tested with version 3.6.9 on my Mac and I can run those commands without issue, whether I have auth enabled or not. The weird thing here is that I can run the following command (note that this is the same command that gets run by the shell helper rs.status()):

.../mongo/bin/mongodb-3.6.9/mongo --port 27019 --eval 'db.adminCommand({"replSetGetStatus": 1, "initialSync": 1})'

I get an error with the following:

/mongo/bin/mongodb-3.6.9/mongo --port 27019 --eval 'db.adminCommand({"top": 1})'
MongoDB shell version v3.6.9
connecting to: mongodb://127.0.0.1:27019/
Implicit session: session { "id" : UUID("17462515-7c74-44ed-9cef-152ac3463d7b") }
MongoDB server version: 3.6.9
{
        "ok" : 0,
        "errmsg" : "not authorized on admin to execute command { top: 1.0, lsid: { id: UUID(\"17462515-7c74-44ed-9cef-152ac3463d7b\") }, $db: \"admin\" }",
        "code" : 13,
        "codeName" : "Unauthorized"
}

Both are adminCommands() on the same Arbiter with auth enabled. Connecting to the secondary, but not logging in, I get the expected error when trying to run rs.status(). Really any adminCommand() call should require that you be authed and allowed to run commands in the admin database.

I am not sure why I can run the rs.status() commands against the arbiter and get results back where as you can’t. I am using a slightly older version than you, so not sure which is right in this case.

As for the splitChuck and splitVector commands, those are internal only methods and probably shouldn’t be called (if they even can be). The shell helpers sh.splitFind() or sh.splitAt() methods are what you would want to use. Also note that these methods are only needed if you’re using sharding.

As for top, you need to be logged in as someone with privileges to run commands in the admin database. This can be run from either a primary or secondary.

I’ve not used the startSession() method but does appear that it can be called whether or not you’ve authenticated and can be run from either a primary or secondary. Having said this, if you’re not authenticated, then you probably won’t be able to run many commands from this new session.

Hopefully some of this helps, and sorry that I can’t provide and explanation for why the rs.status() call fails for you from an arbiter.

Joanne · May 8, 2020, 9:39pm

Can someone please help us understand about Doug’s findings?

And I’d like to understand the output of db.listCommands() the meaning of combinations of adminOnly and slaveOk in case of authorisation enabled or disabled:
Can you let me know if my understanding is correct

command1: adminOnly → only returns result if Authentication is enabled, runs db.adminCommand and should be run from primary host?
command2: . → returns result regardless of authentication enabled or not and should be run from primary host?
command3: slaveOk → returns result regardless of authentication enabled or not and can be run from either primary or secondary host?
command4: adminOnly slaveOk . → only returns result if Authentication is enabled, and can be run from either primary or secondary host?

Really appreciate your help on this!
Thanks in advance.

chris · May 8, 2020, 10:33pm

I decided to have a play also.

PSA v3.6.9 auth enabled.
mongo locally on the arbiter could complete rs.status()

A fun scenario might be a node that was a replica but then was changed to an arbiter.
Now I get the unauthorized error!

So this arbiter could have been a ‘dirty’ node when it was added. Clean out the data dir on the arbiter and I can now rs.status()

Sample..

18:30 $ docker exec -it mongo-0-c mongo
MongoDB shell version v3.6.9
connecting to: mongodb://127.0.0.1:27017
Implicit session: session { "id" : UUID("a095286b-f632-4292-a14c-633861261df4") }
MongoDB server version: 3.6.9
s0:ARBITER> rs.status()
{
	"ok" : 0,
	"errmsg" : "not authorized on admin to execute command { replSetGetStatus: 1.0, lsid: { id: UUID(\"a095286b-f632-4292-a14c-633861261df4\") }, $db: \"admin\" }",
	"code" : 13,
	"codeName" : "Unauthorized"
}
s0:ARBITER> 
bye
18:30 $ docker rm -f mongo-0-c 
mongo-0-c
18:30 $ docker volume rm mongo-psa_mongo-0-c 
mongo-psa_mongo-0-c
18:30 $ docker-compose up -d mongo-0-c
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating volume "mongo-psa_mongo-0-c" with default driver
Creating mongo-0-c ... done
18:31 $ docker exec -it mongo-0-c mongo
MongoDB shell version v3.6.9
connecting to: mongodb://127.0.0.1:27017
Implicit session: session { "id" : UUID("63c73dab-8a0d-40d7-a97b-e40cf702d0e8") }
MongoDB server version: 3.6.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
s0:ARBITER> rs.status()
{
	"set" : "s0",
	"date" : ISODate("2020-05-08T22:31:28.865Z"),
	"myState" : 7,
	"term" : NumberLong(4),
	"syncingTo" : "",
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1588977081, 1),
			"t" : NumberLong(4)
		},
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1588977081, 1),
			"t" : NumberLong(4)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1588977081, 1),
			"t" : NumberLong(4)
		},
		"durableOpTime" : {
			"ts" : Timestamp(0, 0),
			"t" : NumberLong(-1)
		}
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "mongo-0-a:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 9,
			"optime" : {
				"ts" : Timestamp(1588977081, 1),
				"t" : NumberLong(4)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1588977081, 1),
				"t" : NumberLong(4)
			},
			"optimeDate" : ISODate("2020-05-08T22:31:21Z"),
			"optimeDurableDate" : ISODate("2020-05-08T22:31:21Z"),
			"lastHeartbeat" : ISODate("2020-05-08T22:31:27.615Z"),
			"lastHeartbeatRecv" : ISODate("2020-05-08T22:31:27.579Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "mongo-0-b:27017",
			"syncSourceHost" : "mongo-0-b:27017",
			"syncSourceId" : 1,
			"infoMessage" : "",
			"configVersion" : 7
		},
		{
			"_id" : 1,
			"name" : "mongo-0-b:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 9,
			"optime" : {
				"ts" : Timestamp(1588977081, 1),
				"t" : NumberLong(4)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1588977081, 1),
				"t" : NumberLong(4)
			},
			"optimeDate" : ISODate("2020-05-08T22:31:21Z"),
			"optimeDurableDate" : ISODate("2020-05-08T22:31:21Z"),
			"lastHeartbeat" : ISODate("2020-05-08T22:31:27.616Z"),
			"lastHeartbeatRecv" : ISODate("2020-05-08T22:31:27.743Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1588967519, 1),
			"electionDate" : ISODate("2020-05-08T19:51:59Z"),
			"configVersion" : 7
		},
		{
			"_id" : 2,
			"name" : "mongo-0-c:27017",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 11,
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"configVersion" : 7,
			"self" : true,
			"lastHeartbeatMessage" : ""
		}
	],
	"ok" : 1
}

Doug_Duncan · May 8, 2020, 11:05pm

Brilliant thinking @chris ! I’m glad you jumped in. I told you that I learn a lot from you.

@Joanne sorry that I misunderstood your question. You were wondering about the adminOnly and slaveOk designations and not the actual commands themselves. The best way to learn this is to run the command and see what happens.

It doesn’t matter if authentication is enabled or not for these commands to run. If authentication is enabled, then the user running the command would need to have the proper privileges to run them.

The adminOnly listed commands must be run in the admin database if using runCommand({...}) or with the adminCommand({...}) from any other database. These can be run from either the primary or the secondary.

I am not sure what the slaveOk designation means as I was able to run those commands from both a primary and a secondary that did not have rs.slaveOk ran on it. @Stennie_X, can you provide insight into the commands listed showing slaveOk, and what that label means, on them in the results of db.listCommands()?

Stennie_X · May 9, 2020, 1:45am

A few commands (such as isMaster, ping, connectionStatus, and authenticate) do not require any authentication even if auth is enabled. These are used to support connecting to a deployment.

The isMaster command is used by drivers as part of the connection handshake to discover server features and negotiate compatibility. The isMaster response includes information such as the wire protocol version, supported auth mechanisms, compression options, and the role of the current node.

The majority of commands (including serverStatus) will require authentication if auth is enabled.

Yes, this is a variation on the possibility I mentioned earlier:

Arbiters do not replicate any data (including user/auth data), but another possibility is that someone has manually created users to secure your arbiter.

This can be (ab)used as a feature, although any users created directly on an arbiter are (as of MongoDB 4.4) independent of the rest of the replica set. Dropping the data directory on the arbiter (as suggested by @chris) will ensure the arbiter is only used as designed.

The slaveOk flag (aka AllowedOnSecondary) indicates whether a command is permitted on a secondary, but does not mean this command does something useful from an end user point of view. Some commands are internal (although they should be generally noted as such) or require specific context to be useful.

The Database Commands section in the MongoDB documentation is the best reference for available commands relevant to your version of MongoDB.

The db.adminCommand() shell helper runs a command against the admin namespace and is a shorthand for db.getSiblingDB("admin").runCommand() or changing to the admin namespace and invoking db.runCommand(). You can run either admin or non-admin commands through the db.adminCommand() shell helper, as long as those are meaningful for the admin namespace.

Commands will fail unless the current user has appropriate privileges for the current configuration of access control and authentication.

Regards,
Stennie