Change Streams: does config FullDocument = UpdateLookup have performance implications on the source DB?

Working with Data Connectors & Integrations
, , ,
1

I am trying to deep dive into Mongo Change Streams implementation to understand whether configuring full document update lookup will impact DB performance in a production environment .

I assume the full document lookup is just a simple query by ID. Therefore, my manly concern is how it will impact the source DB when there are a lot of writes happening to it. Maybe there is a configuration to query by batch of IDs, which might help.

If the change stream cursor will query my collection each time it sees an update in the oplog, it simply means that each applicative write to the collection is effectively a write and a read . I don’t want to impact an operational DB performance and thus impact the application performance.

From my understanding, change streams reading from the oplog will not impact the DB so much, but if my assumption above is right, this main advantage is gone.

Setup background:

Appreciate any piece of information on this matter.

Thank you all.

1 Like
2

First, I am not an expert,
but I am dealing with the same dilemma,
I think that your assumption is probably right that eventually every update will end up with another read after oplog cursor iteration.

but lets keep in our minds:

  1. That it’s an _id index read which is the most efficient for retrieving full document.
  2. What is the alternative? it has not being found yet (from what I know) the technology that knows how to
    mirror any database change without any performance implications.
  3. maybe other alternative is that you iterate the oplog cursor and aggregate a batch of _id that have being updated in small and defined period of time and make a bigger query from secondary node which could be preferable from operational point of view

Note: This Thread is very important in my opinion I hope others how could help will reply here