Hi,
We have an existing replica set with 2 node and 1 arbiter. We want to add one more node to the existing replica set which I did and added the node to primary node using “rs.add” command however after running for over 20-30 hrs , the initial sync fails with below error:
2023-05-30T09:46:36.408-0400 E REPL [replication-404] Initial sync attempt failed -- attempts left: 0 cause: NetworkInterfaceExceededTimeLimit: error fetching oplog during initial sync: Operation timed out, request was RemoteCommand 16829255 -- target:mongodb02:27017 db:local expDate:2023-05-30T09:46:35.997-0400 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1685454318000|1 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000 }
2023-05-30T09:46:36.408-0400 F REPL [replication-404] The maximum number of retries have been exhausted for initial sync.
2023-05-30T09:46:36.458-0400 E REPL [replication-404] Initial sync failed, shutting down now. Restart the server to attempt a new initial sync.
2023-05-30T09:46:36.458-0400 I - [replication-404] Fatal assertion 40088 NetworkInterfaceExceededTimeLimit: error fetching oplog during initial sync: Operation timed out, request was RemoteCommand 16829255 -- target:mongodb02:27017 db:local expDate:2023-05-30T09:46:35.997-0400 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1685454318000|1 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000 } at src/mongo/db/repl/replication_coordinator_impl.cpp 635
2023-05-30T09:46:36.458-0400 I - [replication-404]
Target/Newly added node: MongoDB shell version v3.4.24
Source/existing node: MongoDB shell version v3.4.9
I tried to find rpm repo for 3.4.9 but could find only 3.4.24.
Size of the data on existing node : 30-35TB
No of databases : 32
No of collections : most of the DBs have less than 100 collections but 3-6 DBs have close to 5k-10k documents with more than few million documents in them.
Limitations:
- I can try to convince management to upgrade the mongo but that is long process to decide and implement
- we can not shutdown production server.
I am not expert in mongo and still trying to learn more and more from various sources so I would request the experts to help me achieve this target.
-Onkar