Change Stream Idle handling

Working with Data
,
#1

Hi All!

What is the best way for restarting a change stream service when the reading cursor is terminated/exhausted?
(could be on a primary switch and other cases)

I have tackled several times in situations when the service was doing nothing because of that, and I want to create a good way to inform k8s when it turns idle (health check?/readiness?).

does someone have an advice for me?

Thank you.

#3

Hi @Shay_I,

I’m not too familiar with k8s deployments but I have a few questions regarding the scenario you’ve detailed at the bottom of my reply.

What is the best way for restarting a change stream service when the reading cursor is terminated/exhausted?
(could be on a primary switch and other cases)

If the cursor is terminated, you can resume change streams by specifying a resume token to either resumeAfter or startAfter when opening the cursor.

Additionally, as per the change streams documentation:

While the connection to the MongoDB deployment remains open, the cursor remains open until one of the following occurs:

  • The cursor is explicitly closed.
  • An invalidate event occurs.
  • If the deployment is a sharded cluster, a shard removal may cause an open change stream cursor to close, and the closed change stream cursor may not be fully resumable.

With regards to the following:

I have tackled several times in situations when the service was doing nothing because of that, and I want to create a good way to inform k8s when it turns idle (health check?/readiness?).

Can you provide the following information:

  • Minimal reproducible code in which I can test this out. (Please redact any personal or sensitive information before posting it here)
  • What you have tackled / tried so far?
  • What you are expecting k8s to do when idling is encountered?
  • If change stream idling is also terminating the cursor? If so, please advise how you are seeing this
    Lastly, please advise what version of MongoDB you are using.

Regards,
Jason

1 Like