My server lost connection to M0 Cluster every few weeks(or days). The server implemented health check api is running on a docker container.
Because the API also checks DB connection, the container stops after a number of retries specified in docker-compose.yml but then I re-execute a last CI/CD job, the server runs well again as if nothing had happened.
I wanted to find even the smallest hint, but I couldn’t get any error messages.
These are What I find out:
- Time of disconnection
- There were no running DB operations.
(API server was just reading connection status during healthcheck)
index.js snippet
import Koa from 'koa'
const app = new Koa();
// ...
const MONGO_URI = `mongodb+srv://${ATLAS_USER}:${ATLAS_PASS}@${ATLAS_HOST}/${ATLAS_DB}?authSource=admin`
const MONGO_CONFIG = { useNewUrlParser: true, useUnifiedTopology: true, maxPoolSize: 30, useFindAndModify: false }
Mongoose.Promise = global.Promise;
Mongoose.set('useCreateIndex', true)
try {
Mongoose.connect(MONGO_URI, MONGO_CONFIG)
.then((db) => {
console.log('Mongoose connection Established')
db.connection.on('error', (err) => { console.error(err) }) // <- print nothing
db.connection.on('disconnected', () => { console.log('disconnected') }) // <- print once
db.connection.on('reconnected', () => { console.log('reconnected') }) // <- never printed
})
} catch (error) {
console.error(error.message)
console.log('Mongoose connection Failed')
process.exit(1)
}
// ...
healthcheck API
router.get('/_online', ctx => {
const mAvailable = mongoose.connection?.readyState === 1
if (!mAvailable) console.error('mongoose readyState : ' + mongoose.connection?.readyState)
ctx.body = mAvailable ? 'OK' : 'FAILED'
ctx.status = mAvailable ? 200 : 503
});
docker logs -n 100
<-- GET /_online
--> GET /_online 200 1ms 2b
<-- GET /_online
--> GET /_online 200 2ms 2b
<-- GET /_online
--> GET /_online 200 1ms 2b
<-- GET /_online
--> GET /_online 200 1ms 2b
<-- GET /_online
--> GET /_online 200 1ms 2b
<-- GET /_online
--> GET /_online 200 1ms 2b
<-- GET /_online
--> GET /_online 200 1ms 2b
disconnected
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 2ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 2ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 2ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 2ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 3ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
<-- GET /_online
--> GET /_online 503 1ms 6b
mongoose readyState : 0
2021-12-20T00:39:23: PM2 log: Stopping app:mm_api id:0
2021-12-20T00:39:23: PM2 log: App name:mm_api id:0 disconnected
2021-12-20T00:39:23: PM2 log: App [mm_api:0] exited with code [0] via signal [SIGINT]
2021-12-20T00:39:23: PM2 log: pid=18 msg=process killed
2021-12-20T00:39:23: PM2 log: PM2 successfully stopped