I have inherited a Ubuntu MongoDB cluster in turn servicing a Greylog cluster. The boxes are running NTPD, but they are synced to different external sources and are pretty badly out of sync. I need to reconfigure them to point to an internal network NTP server. The boxes will drift into alignment, so I’m not concerned about any abrupt shifts in time.
I would normally be inclined to correct the Primary server first, however this would increase the total discrepancy between the primary and the third node, so I’m not sure if I would be better off rectifying the 3rd node first.
Primary - approx 1m 23s slow
user@dca-glog-mgodb1:~$ ntpdate -q time.REDACTED.local
server 172.31.6.136, stratum 3, offset 83.265238, delay 0.04156 2 Aug 18:42:06
ntpdate[110694]: step time server 172.31.6.136 offset 83.265238 sec
Secondary1 - 11 seconds fast
user@dca-glog-mgodb2:~$ ntpdate -q time.REDACTED.local
server 172.31.6.136, stratum 3, offset -11.298167, delay 0.04149 2 Aug 18:43:49
ntpdate[62263]: step time server 172.31.6.136 offset -11.298167 sec
Secondary2 - 3m 20s seconds slow
user@dca-glog-mgodb3:~$ ntpdate -q time.REDACTED.local
server 172.31.6.136, stratum 3, offset 200.326086, delay 0.04155 2 Aug 18:40:21
ntpdate[110652]: step time server 172.31.6.136 offset 200.326086 sec
The boxes have been like this for at least a couple of years without any obvious problems, so I don’t think the application is heavily time dependent, but it is a Prod cluster so I want to tread carefully one-the-less. Can any offer any advice about the best order to correct these nodes in?
Thanks.