Using clusterMonitor role Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: ....... $db: "local", $readPreference: { mode: "primary" } }

Ops and Admin Replication
, , ,
1

using MongoDB server version: 4.2.22

ii mongodb-org-server 4.2.22 amd64 MongoDB database server

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.5 LTS
Release:	20.04
Codename:	focal

I m trying to setup a monitoring on GCP - > MongoDB  |  Cloud Monitoring  |  Google Cloud

I can get the metric by
adding a new user for the monitoring :

db.getSiblingDB("admin").createUser(
  {
    user: "metricUser",
    pwd: "Password123@",
    roles: ["clusterMonitor"],
  })

metrics works !

but from the metric scraping process (google-cloud-ops-agent-opentelemetry-collector.service),
I get the following errors every 30seconds :

Jan 19 15:41:50 mongo-server-2.private.xxxx.com otelopscol[543784]: 2023-01-19T15:41:50.280Z        debug        scraperhelper/scrapercontroller.go:197        Error scraping metrics        {"error": "failed to fetch index stats metrics: (Unauthorized) not authorized on local to execute command { aggregate: \"system.replset\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"df5c11f9-e865-45d7-8c11-aeb6e1cdeddd\") }, $clusterTime: { clusterTime: Timestamp(1674142905, 1), signature: { hash: BinData(0, 9161B7FBCD2C952834EA0DAB5B87B6153B3BE5CA), keyId: 7138546094778089478 } }, $db: \"local\", $readPreference: { mode: \"primary\" } }", "scraper": "mongodb"}
Jan 19 15:41:50 mongo-server-2.private..xxxx.com otelopscol[543784]: go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
Jan 19 15:41:50 mongo-server-2.private.xxxx.com otelopscol[543784]:         /root/go/pkg/mod/go.opentelemetry.io/collector@v0.61.0/receiver/scraperhelper/scrapercontroller.go:197
Jan 19 15:41:50 mongo-server-2.private.xxxx.com otelopscol[543784]: go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
Jan 19 15:41:50 mongo-server-2.private.xxxx.com otelopscol[543784]:         /root/go/pkg/mod/go.opentelemetry.io/collector@v0.61.0/receiver/scraperhelper/scrapercontroller.go:172

from mongo logs every 30seconds:

2023-01-20T17:02:36.891+0000 I  ACCESS   [conn67] Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID("ec721915-45ce-4ac5-a8be-99c676afba63") }, $clusterTime: { clusterTime: Timestamp(1674234148, 1), signature: { hash: BinData(0, F171EA196DF7052781BD2A4B04117F0108CFC181), keyId: 7138546094778089478 } }, $db: "local", $readPreference: { mode: "primary" } }
2023-01-20T17:03:06.892+0000 I  ACCESS   [conn67] Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID("ec721915-45ce-4ac5-a8be-99c676afba63") }, $clusterTime: { clusterTime: Timestamp(1674234178, 1), signature: { hash: BinData(0, F8BF81DE3805E58A97310628BAB162337F481413), keyId: 7138546094778089478 } }, $db: "local", $readPreference: { mode: "primary" } }
2023-01-20T17:03:36.892+0000 I  ACCESS   [conn67] Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID("ec721915-45ce-4ac5-a8be-99c676afba63") }, $clusterTime: { clusterTime: Timestamp(1674234208, 1), signature: { hash: BinData(0, F02AE9EC78F26A6525DDE4363D2ABDD0A602501E), keyId: 7138546094778089478 } }, $db: "local", $readPreference: { mode: "primary" } }

I added also this role: readAnyDatabase to metricUser user but still getting the noisy Unauthorized errors log.

db.getRole("clusterMonitor",{showPrivileges:true})
{
	"role" : "clusterMonitor",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ],
	"privileges" : [
		{
			"resource" : {
				"cluster" : true
			},
			"actions" : [
				"checkFreeMonitoringStatus",
				"connPoolStats",
				"getCmdLineOpts",
				"getLog",
				"getParameter",
				"getShardMap",
				"hostInfo",
				"inprog",
				"listDatabases",
				"listSessions",
				"listShards",
				"netstat",
				"replSetGetConfig",
				"replSetGetStatus",
				"serverStatus",
				"shardingState",
				"top",
				"useUUID"
			]
		},
		{
			"resource" : {
				"db" : "",
				"collection" : ""
			},
			"actions" : [
				"collStats",
				"dbStats",
				"getDatabaseVersion",
				"getShardVersion",
				"indexStats"
			]
		},
		{
			"resource" : {
				"db" : "config",
				"collection" : ""
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"getDatabaseVersion",
				"getShardVersion",
				"indexStats",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "local",
				"collection" : ""
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"getDatabaseVersion",
				"getShardVersion",
				"indexStats",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "local",
				"collection" : "system.js"
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "config",
				"collection" : "system.js"
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "local",
				"collection" : "system.replset"
			},
			"actions" : [
				"find"
			]
		},
		{
			"resource" : {
				"db" : "",
				"collection" : "system.profile"
			},
			"actions" : [
				"find"
			]
		}
	],
	"inheritedPrivileges" : [
		{
			"resource" : {
				"cluster" : true
			},
			"actions" : [
				"checkFreeMonitoringStatus",
				"connPoolStats",
				"getCmdLineOpts",
				"getLog",
				"getParameter",
				"getShardMap",
				"hostInfo",
				"inprog",
				"listDatabases",
				"listSessions",
				"listShards",
				"netstat",
				"replSetGetConfig",
				"replSetGetStatus",
				"serverStatus",
				"shardingState",
				"top",
				"useUUID"
			]
		},
		{
			"resource" : {
				"db" : "",
				"collection" : ""
			},
			"actions" : [
				"collStats",
				"dbStats",
				"getDatabaseVersion",
				"getShardVersion",
				"indexStats"
			]
		},
		{
			"resource" : {
				"db" : "config",
				"collection" : ""
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"getDatabaseVersion",
				"getShardVersion",
				"indexStats",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "local",
				"collection" : ""
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"getDatabaseVersion",
				"getShardVersion",
				"indexStats",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "local",
				"collection" : "system.js"
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "config",
				"collection" : "system.js"
			},
			"actions" : [
				"changeStream",
				"collStats",
				"dbHash",
				"dbStats",
				"find",
				"killCursors",
				"listCollections",
				"listIndexes",
				"planCacheRead"
			]
		},
		{
			"resource" : {
				"db" : "local",
				"collection" : "system.replset"
			},
			"actions" : [
				"find"
			]
		},
		{
			"resource" : {
				"db" : "",
				"collection" : "system.profile"
			},
			"actions" : [
				"find"
			]
		}
	]
}
2

Please check mongo documentation buit-in-roles
Clustermonitor has only find action on system.replset
You have to give clustermanager or clusteradmin role
ReadwriteAnyDatabase does not give access to config & local DBs

3

@Ramachandra_Tummala I added clusterManager role to the metricUser user but I m still getting the same error logs.

image

2023-01-25T10:40:14.320+0000 I  ACCESS   [conn129] Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID("1ff3c60b-297f-4950-abea-ca7eb8279bcb") }, $clusterTime: { clusterTime: Timestamp(1674643214, 1), signature: { hash: BinData(0, A5FB6886D52D59CD67B9C752120EA6EB8DE3BD08), keyId: 7138546094778089478 } }, $db: "local", $readPreference: { mode: "primary" } }
2023-01-25T10:40:44.319+0000 I  ACCESS   [conn129] Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID("1ff3c60b-297f-4950-abea-ca7eb8279bcb") }, $clusterTime: { clusterTime: Timestamp(1674643244, 1), signature: { hash: BinData(0, 8F1E4EA53825279E8D13EF8FF3233F5E5846EF76), keyId: 7138546094778089478 } }, $db: "local", $readPreference: { mode: "primary" } }
2023-01-25T10:41:14.329+0000 I  ACCESS   [conn129] Unauthorized: not authorized on local to execute command { aggregate: "system.replset", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID("1ff3c60b-297f-4950-abea-ca7eb8279bcb") }, $clusterTime: { clusterTime: Timestamp(1674643274, 1), signature: { hash: BinData(0, A054F0CE3E53C5AB189A9CFCA61EC5D65E250F0B), keyId: 7138546094778089478 } }, $db: "local", $readPreference: { mode: "primary" } }

no difference if I just only grant clusterAdmin role

{
	"_id" : "admin.metricUser",
	"userId" : UUID("260cf642-1c10-49a5-bb4a-25875f2087ea"),
	"user" : "metricUser",
	"db" : "admin",
	"roles" : [
		{
			"role" : "clusterAdmin",
			"db" : "admin"
		}
	],
	"customData" : {

	},
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

Do you think google-cloud-ops-agent-opentelemetry-collector.service is trying to scrape some disallowed/non-existing metrics?

4

I resolved this by creating the below custom role and assigning it to user

Note that the operation and db may change for each case

use admin
db.createRole(
{
role: “customRoleConfig”,
privileges: [
{
actions: [ “collStats”, “indexStats” ],
resource: { db: “config”, collection: “system.indexBuilds” }
},
{
actions: [ “collStats”, “indexStats” ],
resource: { db: “local”, collection: “replset.election” }
},
{
actions: [ “collStats”, “indexStats” ],
resource: { db: “local”, collection: “replset.initialSyncId” }
},
{
actions: [ “collStats”, “indexStats” ],
resource: { db: “local”, collection: “replset.minvalid” }
},
{
actions: [ “collStats”, “indexStats” ],
resource: { db: “local”, collection: “replset.oplogTruncateAfterPoint” }
}],
roles: })

db.grantRolesToUser( “mongodb_exporter”, [ {role: “customRoleConfig”, db: “admin” }])