How can I troubleshoot mongos? Anyway to increase logging?

Ops and Admin
#1

I have a three node cluster with a mongos query router.
When connecting to the mongos instance it get’s stuck on connecting.
Connecting directly to the mongod instance works.

Is there anyway to increase the logging of mongos?
The logs just say it’s received a client connection.

#2

Hi @Martin_Nystrom and welcome in the MongoDB Community :muscle: !

I don’t understand the topology of your cluster. Usually a “3 nodes cluster” is a replica set (RS) and a RS doesn’t need a mongos. Only a sharded cluster would need a few mongos to work properly. If you are running a sharded cluster with 3 shards and they are not replicated, it’s really not safe. If one of the node fails, you will lose a third of your DB, given that you chose a shard key that distribute the data evenly across the 3 single node RS shards.

If this is really what you are doing:

  1. I hope it’s not a production environment.
  2. Did you add correctly the 3 single nodes RS as shards? Can you see them in the sh.status() command?
  3. You can set the verbosity level like so: https://docs.mongodb.com/manual/reference/log-messages/#log-messages-configure-verbosity

I hope this helps,
Maxime.

#3

Sorry I was a bit unclear. It’s really a 8 node cluster. 3 replicated config nodes, 3 replicated shard nodes and two query routers.
I followed this guide while setting up the cluster with a few modifications of my own:

716×556 9.67 KB

The only difference is that I have three shard nodes and it’s not open for Internet acess.
Thank you for the link to the verbosity settings. I will see if I can get some more info from the query router, seems like there is nothing wrong with the config replica set or the shard replica set. Since both replica sets have a working primary.

#4

Oh that looks great :slight_smile: ! Way more production ready :muscle: !

Can you please provide more information like:

  • The command line you use to start the mongos and mongod and their respective configurations?
  • The command line you use when you try to connect to the mongos? The one that is failing.

If I understand correctly, you were never able to connect to the mongos and thus finish the setup by adding the 3 replica sets in the configuration. Correct?

What’s the error that you get exactly?

#5

Well it’s very strange I was running this setup without issues for over 6 months and all of a sudden I was unable to connect through mongos.
These are the commands I use to connect to the mongos instance:

mongo 10.4.44.88 -u mongo-admin -p --authenticationDatabase admin

To rule out network issues I have tried SSHing directly to the host and using mongo client there:

mongo localhost -u mongo-admin -p --authenticationDatabase admin

I don’t really get any errors it sort of just times out… It just says:

connecting to: mongodb://10.4.44.88:27017/test

Even after increasing the verbosity of the mongos instance I was unable to determine what causes the client to get stuck.

Mongos configuration:

Start command:
/usr/bin/mongos --config /etc/mongos.conf

mongos.conf:

[Unit]
Description=Mongo Cluster Router
After=network.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongos --config /etc/mongos.conf
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

[Install]
WantedBy=multi-user.target
root@zenv-0689:~# cat /etc/mongos.conf 
# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongos.log
  logRotate: reopen
processManagement:
  pidFilePath: /var/run/mongodb/mongos.pid

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

security:
  keyFile: /var/lib/mongodb/keyfile

sharding:
  configDB: configReplSet/mongo-config01:27019,mongo-config02:27019,mongo-config03:27019

Mongod config servers:

Start command:

/usr/bin/mongod --config /etc/mongod.conf

mongod.conf:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
  logRotate: reopen
processManagement:
  pidFilePath: /var/run/mongodb/mongod.pid

# network interfaces
net:
  port: 27019
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

security:
  keyFile: /var/lib/mongodb/keyfile

replication:
  replSetName: configReplSet

sharding:
  clusterRole: "configsvr"

Mongod shard servers:

Start command:

/usr/bin/mongod --config /etc/mongod.conf

mongod.conf:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
  logRotate: reopen
processManagement:
  pidFilePath: /var/run/mongodb/mongod.pid


# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:
security:
  keyFile: /var/lib/mongodb/keyfile


#operationProfiling:
replication:
  replSetName: Shard00

#sharding:
sharding:
  clusterRole: shardsvr



#auditLog:

#snmp:
#6

Hi,

I resolved the issue by upgrading from 4.2 to 4.4.1

:+1:

1 Like
#7

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.