The replica set member host changed for itself from server IP address to hostname

Carlos_Manuel_55780 · November 29, 2022, 2:38pm

Hi

I set up Mongo 4.0.2 for linux server. The server hostname is dbaMongoServer.org.

/etc/hosts content:

  127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  192.168.10.10 dbaMongoServer

As you can see the hostname established in /etc/hosts file is not the same for server hostname

Then I changed the mongod.conf to establish it as replicaset and I executed the following commands in mongoshell:

  rs.initialize()
  rs.add("192.168.10.10:27017").

One test application that connects to MongoDB and it started to faill. This application connects by IP address, not by hostname database server. These are the following messages from application log:

  2022-11-28 08:54:53.848 INFO  14720 --- [XNIO-1 task-2] org.mongodb.driver.connection Closed connection [connectionId{localValue:3695}] to dbaMongoServer.org:27017 because there was a socket exception raised by this connection.
  2022-11-28 08:54:53.850 ERROR 14720 --- [XNIO-1 task-2] p.g.r.services.ValidateService     org.springframework.dao.DataAccessResourceFailureException: dbaMongoServer.org; nested exception is com.mongodb.MongoSocketException: dbaMongoServer.org
      at  org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:90)
      at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2774)
```
```
Caused by: com.mongodb.MongoSocketException: dbaMongoServer.org
      at com.mongodb.ServerAddress.getSocketAddress(ServerAddress.java:188)

I fixed the /etc/hosts changing its third line as follows:

192.168.10.10 dbaMongoServer.org

The application keeps failling.

After that, I checked the replicaset configuration by db.conf(), I realized the member host was changed for itself from IP address to hostname.

   "members" : [
            {
                    "_id" : 0,
                    "host" : "dbaMongoServer.org:27017",
                    "arbiterOnly" : false,
                    "buildIndexes" : true,
                    "hidden" : false,
                    "priority" : 1,
                    "tags" : {

                    },
                    "slaveDelay" : NumberLong(0),
                    "votes" : 1
            }
    ],

The question is why the replica set member host changed for itself from server IP address to hostname?

Ramachandra_Tummala · November 30, 2022, 1:17am

Did your rs.add succeed?
On which port your mongod was running?You are trying to add second node on 27017
Show us the output of rs.status()
The output you have shown must be that of primary
By default rs.initiate takes hostname with empty config or if the members were not defined at initiate time

Carlos_Manuel_55780 · November 30, 2022, 6:07pm

The rs.add succeded but not how I wanted. I executed rs.add(“192.168.10.10:27017”) after
rs.initialize(). Even though, the host member was stored as ‘dbaMongoServer:27017’. It never happenned to me before.

The port is 27017. It’s a replicaset for only one node (primary).

The output you have shown must be that of primary

Right
mongo_rsStatus_1.txt (2.5 KB)

chris · November 30, 2022, 11:14pm

rs.initiate() will have set the host up using its own fqdn. So it would not have changed it would have been dbMongoServer.org from the initialization. The subsequent rs.add() would likely have failed.

Click to expand: example: initiate, add

root@mdb:/# hostname -I
172.17.0.3 
root@mdb:/# mongo --quiet
> rs.initiate()
{
	"info2" : "no configuration specified. Using a default configuration for the set",
	"me" : "mdb:27017",
	"ok" : 1
}
s0:SECONDARY> 
s0:PRIMARY> rs.add('172.17.0.3')
{
	"operationTime" : Timestamp(1669848239, 1),
	"ok" : 0,
	"errmsg" : "The hosts mdb:27017 and 172.17.0.3:27017 all map to this node in new configuration version 2 for replica set s0",
	"code" : 103,
	"codeName" : "NewReplicaSetConfigurationIncompatible",
	"$clusterTime" : {
		"clusterTime" : Timestamp(1669848239, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}

Using fqdn’s is the recommended way to reference replicaset members.

Regardless, there are a few ways to change to change this to IP, I’ll add one here:

//matching my example from above
s0:PRIMARY> var c=rs.conf()
s0:PRIMARY> c.members[0].host='172.17.0.3:27017'
172.17.0.3:27017
s0:PRIMARY> rs.reconfig(c)
{
	"ok" : 1,
	"operationTime" : Timestamp(1669850026, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1669850026, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}

Carlos_Manuel_55780 · December 1, 2022, 2:14pm

I got a question.

If the /etc/hosts file was no configured well. I mean, it was containing:

  192.168.10.10 dbaMongoServer  // which it's a incorrect hostname. The correct is dbaMongoServer.org

After a period of time I corrected the hosts file. Would this have to do with a possible change of the replicaset’s member host to hostname?

chris · December 1, 2022, 3:16pm

On the client side understanding how the client connects to a replicaSet is important.

The client driver treats the hosts in the connection string as seed hosts, once a connection is successfully made the topology of the replicaset is retrieved by the equivalent of db.hello(). This is why the client would have tried to connect to dbaMongoServer.org if you specified 192.168.10.10 in your connection string.

Client: connect 192.168.10.10 → Server
Client: hello() → Server
Client ← Server: Replicaset looks like this, and primary is dbaMongoServer.org:27017
Client: connect dbaMongoServer.org → Cannot get an address for dbaMongoServer.org

You can specify directConnection=true to work around this behavior, but this is generally not want you want when connection to a replicaset as you want the application to connect to the next primary if the current one fails or is stepped down.

Yilmaz_Durmaz · December 1, 2022, 6:39pm

Hi @Carlos_Manuel_55780

I think this is related to your mongo.conf file. you possibly have set these following lines:

# network interfaces
net:
  port: 27017
  bindIp: dbaMongoServer.org, 127.0.0.1

doing so, your instances will listen to local connections and react to only requests made for “dbaMongoServer.org” (order matters). This also causes replica set use this name upon initialization. I am guessing you have set it there and forgot about it when you open this post.

although members mostly use same port but different names, you may have same name different ports from different machines but that would be hard to maintain in hosts file.

Carlos_Manuel_55780 · December 1, 2022, 8:58pm

This is the content of /etc/mongod.conf for net section

net:
  port: 27017
  bindIp: 0.0.0.0  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses ...

Yilmaz_Durmaz · December 1, 2022, 10:52pm

Honestly, the only time I got a similar result to yours was with that addition in the config file. I may still be wrong, but your server might be using another config file having this shape, other than “/etc/mongod.conf”.
Can you please check about this possibility? one way could be ps aux | grep mongod.

chris · December 2, 2022, 2:28pm

The initiate is using the servers own hostname as returned by the OS.

class CmdReplSetInitiate : public ReplSetCommand

github.com

mongodb/mongo/blob/e79fa3438f0fe33a4e688e29fad0a161a8681523/src/mongo/db/repl/repl_set_commands.cpp#L339


      
                      seeds->push_back(m);
                  }
                  if (*comma == 0) {
                      break;
                  }
                  p = comma + 1;
              }
          }
          }  // namespace
          
          class CmdReplSetInitiate : public ReplSetCommand {
          public:
              CmdReplSetInitiate() : ReplSetCommand("replSetInitiate") {}
              std::string help() const override {
                  return "Initiate/christen a replica set.\n"
                         "http://dochub.mongodb.org/core/replicasetcommands";
              }
              virtual bool run(OperationContext* opCtx,
                               const DatabaseName&,
                               const BSONObj& cmdObj,
                               BSONObjBuilder& result) {

HostAndPort me = someHostAndPortForMe();

github.com

mongodb/mongo/blob/e79fa3438f0fe33a4e688e29fad0a161a8681523/src/mongo/db/repl/repl_set_commands.cpp#L390


      
              StorageInterface::get(opCtx),
              ReplicationProcess::get(opCtx));
          std::string name;
          std::vector<HostAndPort> seeds;
          parseReplSetSeedList(&externalState, replSetString, &name, &seeds);  // may throw...
          
          BSONObjBuilder b;
          b.append("_id", name);
          b.append("version", 1);
          BSONObjBuilder members;
          HostAndPort me = someHostAndPortForMe();
          
          auto appendMember =
              [&members, serial = DecimalCounter<uint32_t>()](const HostAndPort& host) mutable {
                  members.append(
                      StringData{serial},
                      BSON("_id" << static_cast<int>(serial) << "host" << host.toString()));
                  ++serial;
              };
          appendMember(me);
          result.append("me", me.toString());

Which eventually leads to the call of getHostName() in someHostAndPortForMe():

github.com

mongodb/mongo/blob/e79fa3438f0fe33a4e688e29fad0a161a8681523/src/mongo/db/repl/repl_set_commands.cpp#L281


      
              if (localhost_only) {
                  // We're only binding localhost-type interfaces.
                  // Use one of those by name if available,
                  // otherwise fall back on "localhost".
                  return HostAndPort(addrs.size() ? addrs[0] : "localhost", bind_port);
              }
          
              // Based on the above logic, this is only reached for --bind_ip '0.0.0.0'.
              // We are listening externally, but we don't have a definite hostname.
              // Ask the OS.
              std::string h = getHostName();
              verify(!h.empty());
              verify(h != "localhost");
              return HostAndPort(h, serverGlobalParams.port);
          }
          
          void parseReplSetSeedList(ReplicationCoordinatorExternalState* externalState,
                                    const std::string& replSetString,
                                    std::string* setname,
                                    std::vector<HostAndPort>* seeds) {
              const char* p = replSetString.c_str();

Carlos_Manuel_55780 · December 2, 2022, 2:30pm

There is only one config file for mongod. On the other side, I can connect to the databse by MongoCompass from my PC. The hostname and IP are not added from Windows hosts file.

According to the previous messages. I keep in mind now that I have to check member host value for primary once added.

Thank you very much