Approaches for implementing history tracking for debugging purposes

Working with Data
, ,
1

I have a web application split into multiple services which all use the same database. All services query and alter mostly the same collections. This arises the problem that bugs detected in production are very hard to debug as you don’t really know which services updated a given record and how. Given so, we are thinking to implement history tracking mechanism that would let us to track the last N versions of all records in a given collection alongside some metadata (like which service did the update and other relevant info).

For this, I was thinking that Change Streams are a good fit, but unfortunately there does not seem to be a way to append metadata to an update/insert operation without actually storing it in the record itself. I saw in the docs that there is a comment option (https://docs.mongodb.com/manual/reference/command/update/), but it won’t be available in the Change Stream.

What approaches would you suggest to implement this functionality without hurting performance? Those are the properties/constraints of the system:

  • the database is a MongoDB Replica Set
  • the history of changes will only be queried manually so it shouldn’t be highly performant
  • the number of stored versions should be limited so we don’t run out of storage space
  • the performance of existing queries should not be impacted
  • the performance of insert/update operations should be impacted as little as possible
  • preferably the history of changes won’t be stored on the same database as the records themselves, so there won’t be additional pressure on the production database

Thank you!

3

Hi @Mihail_Feraru , and welcome to the forums!

Depending on what kind of “bugs” you are trying to debug, as you mentioned, adding a comment on commands could be helpful in tracking down an operation. For example, adding a comment such below:

{ "name":"app A", 
  "method": "function B"
}

Would show up in the profiler output. The database profiler collects detailed information about Database Commands executed against a running mongod instance.

Also depending on the use case, an alternative is to log operations in an asynchronous method (fire and forget, fail with no impact) alongside the operations. Creating a metadata record into another collection/database.

Another approach is creating a common layer for all the services. For example, creating a REST API layer that talks to the database back-end and servicing all of the other services.

Regards,
Wan.

1 Like