Hi there,
I’m still new to MongoDB coming from the world of RDBMS and maybe my mindset is just wrong, but I’m struggling with how to design and implement ‘collection configuration management’ in Mongo. The way I view the problem is as follows, a specific version of our application has pre-requisites that must exist before it can run that are ‘application’/domain specific. These include things like:
- Configuration of the collection (e.g., is it capped, necessary validators, etc…)
- Configuration of indexes (e.g., unique indexes).
If the configuration isn’t applied prior to the application running, bad things can happen. We use mongo docker containers locally and then use Mongo Cloud solutions for our actual environment on production.
Ideally it seems that we want the configuration above to be tied to and part of the application version we are deploying. What I’m struggling with, is understanding potential ways to build it. In RDBMS land, we’ve used Liquibase in the application to manage the above, when the app is starting up it ensures that the configuration is correct before continuing. Some care is needed to ensure we aren’t disruptive to existing running instances, but this solution works everywhere (local and prod).
An important difference between the local and prod instances, is that prod may start instances concurrently. Liquibase manages this with an advisory locking system, that ensures that the migrations only happen by one instance at a time. I haven’t been able to find a tool that does this well (Liquibase recently added an extension for Mongo but it is not production ready in my view).
Another thought I had, is that we could just manage this in the application, in particular createIndex() calls are idempotent and so we could just reapply them in the background on start up. Testing revealed that while true, subsequent threads that call createIndex() do not block waiting for the background index, so the important happens-before relationship isn’t true and these threads can start inserting data that violates the index constraints, while building, and then cause the createIndex call to fail.
Another thought I had was to just use advisory locking, but unlike an RDBMS mongo doesn’t have those primitives available, and I’m not sure I want to build that over the primitives it does have in every application.
I haven’t found much discussion about this online so I’m wondering if I just am thinking about this the wrong way. One of the central tenets of CI/CD is keeping things in sync between environments, but I don’t see much on this topic w.r.t. Mongo. Another option is to use something like Ansible to manage it, but we were hoping for something with a lighter touch that applications could manage in a more choreographed way.