Q. How to change "replication buffer size"?

Ops and Admin Replication
1

I want to increase the size of the replication buffer.
When I looked at db.serverStatus().metrics.repl.buffer, the maximum size is 256MB.
Is there a command to change this???
I tried to find this in the manual but failed. :frowning:

3

Hi @Kim_Hakseon
Welcome to the MongoDB community forums!!.

As mentioned in the documentation, the maximum configured buffer size is 256 MB and is not configurable.
If you wish to change the size of the oplog, you can follow the documentation to Change the size of oplog for further details.

If the above documentation did not help, could you help me with your specific usecase to understand better?

Best regards
Aasawari

4

Hello, @Aasawari :smiley:

Currently, my situation is that a lot of write jobs are continuously coming in, and the secondary oplog buffer is also constantly full.

As a result, the replication lag is increasing.

If the write job is reduced, I expect it to be resolved, but I am looking for a way in a situation where it is not, so I left the above question.

5

Hi @Kim_Hakseon
Thank you for briefing further.

Can you also help me elaborate on the replica set configurations you have?
Also, can you confirm if all the members in the replica set belong to the same version?

As mentioned earlier, you can definitely try increasing the size of the oplog mentioned the documentation, but also starting from MongoDB 4.4 supports specifying a minimum oplog retention period in hours, where MongoDB only removes an oplog entry if:

  • The oplog has reached the maximum configured size, and
  • The oplog entry is older than the configured number of hours.

I would recommend you upgrading to the latest version and let us know if the issue still persists.

Regards
Aasawari

6

Hello,
First of all, I am using PSS’s 4.2 version of Replica Set.

The situation is that the Write Job is constantly coming in from the app, and the job is increasing.

Then one of the secondary lost several minutes of communication (after which communication was restored)

However, the amount of Write jobs that need to be replicated from the primary is so large that all of the secondary’s oplog buffers cannot handle it, so the replication lag is gradually increasing.

If there’s no way, I’ll have to consider scale-up as well, but I’m constantly looking to see if there’s a tuning point that I can do at the db level.

7

Hello, @Aasawari :smiley:
First, Happy New Year!

I have a question, so I ask you an additional question.
What resources should be scaled up to increase the speed of replication?

8

Has the secondary ‘fallen off the OpLog’? If the oldest entry in the other members oplog has moved beyond the most recent oplog in the member that went offline ‘for several minutes’ then it will not be able to continue replication an an initial sync will be required.

Indicators of this are an increase in replication lag and messages in the lagging members log such as:

We are too stale to use candidate as a sync source. Denylisting this sync source because our last fetched timestamp is before their earliest timestamp

REPL Info

{“t”:{“$date”:“2024-01-02T16:20:15.610+00:00”},“s”:“I”, “c”:“REPL”, “id”:21799, “ctx”:“ReplCoordExtern-0”,“msg”:“Sync source candidate chosen”,“attr”:{“syncSource”:“mongo-0-a:27017”}}
{“t”:{“$date”:“2024-01-02T16:20:15.611+00:00”},“s”:“I”, “c”:“REPL”, “id”:5579708, “ctx”:“ReplCoordExtern-0”,“msg”:“We are too stale to use candidate as a sync source. Denylisting this sync source because our last fetched timestamp is before their earliest timestamp”,“attr”:{“candidate”:“mongo-0-a:27017”,“lastOpTimeFetchedTimestamp”:{“$timestamp”:{“t”:1704211044,“i”:1}},“remoteEarliestOpTimeTimestamp”:{“$timestamp”:{“t”:1704211502,“i”:7492}},“denylistDurationMinutes”:1,“denylistUntil”:{“$date”:“2024-01-02T16:21:15.611Z”}}}
{“t”:{“$date”:“2024-01-02T16:20:15.611+00:00”},“s”:“I”, “c”:“REPL”, “id”:21798, “ctx”:“ReplCoordExtern-0”,“msg”:“Could not find member to sync from”}
{“t”:{“$date”:“2024-01-02T16:21:15.772+00:00”},“s”:“I”, “c”:“REPL”, “id”:21799, “ctx”:“BackgroundSync”,“msg”:“Sync source candidate chosen”,“attr”:{“syncSource”:“mongo-0-b:27017”}}
{“t”:{“$date”:“2024-01-02T16:21:15.772+00:00”},“s”:“I”, “c”:“REPL”, “id”:5579708, “ctx”:“ReplCoordExtern-0”,“msg”:“We are too stale to use candidate as a sync source. Denylisting this sync source because our last fetched timestamp is before their earliest timestamp”,“attr”:{“candidate”:“mongo-0-b:27017”,“lastOpTimeFetchedTimestamp”:{“$timestamp”:{“t”:1704211044,“i”:1}},“remoteEarliestOpTimeTimestamp”:{“$timestamp”:{“t”:1704211502,“i”:7492}},“denylistDurationMinutes”:1,“denylistUntil”:{“$date”:“2024-01-02T16:22:15.772Z”}}}

A comparison of rs.printReplicationInfo() can also determine if this is the case.

// from primary
oplog first event time
'Tue Jan 02 2024 16:05:02 GMT+0000 (Coordinated Universal Time)'

// from lagging secondary
oplog last event time
'Tue Jan 02 2024 15:57:24 GMT+0000 (Coordinated Universal Time)'

The solution to this is having enough OpLog headroom to allow a given amount of downtime, 48-72hrs is a general rule of thumb. It is long enough to allow for a member to be down over a weekend.

https://www.mongodb.com/docs/manual/tutorial/resync-replica-set-member/#resync-a-member-of-a-replica-set
https://www.mongodb.com/docs/manual/tutorial/change-oplog-size/#change-the-size-of-the-oplog