Coming in 4.2 - Compression, Key Rotation and Configuration

MongoDB 4.2 isn't just about new features for developers. Some of the changes are designed specifically with MongoDB operators and administrators in mind. In this Coming In 4.2 article, we take a look at some of those changes including new compression options, ways to reduce maintenance time and new ways to configure your MongoDB server.

Zstd compression arrives

Zstd compression is coming to MongoDB 4.2. To put this in context, this is the third compression option supported by MongoDB. MongoDB already has support for snappy and zlib compression of data. Snappy is the current default compression, used by the Wired Tiger storage engine for block and journal compression and by the MongoDB server for client communications.

The thing with compression is that it is a tradeoff between how much CPU is used to perform the compression and how much that CPU work results in more compressed data. Snappy is very much the baseline for modern compression. The older alternative zlib compressor uses more CPU but can more effectively compress data. It has been available for MongoDB users who were prepared to accept that trade-off.

Zstandard, referred to as zstd, is a more modern compressor which offers higher compression rates while using less CPU compared to other compression algorithms. MongoDB 4.2 will continue to use snappy compression by default. Where a driver already supports zstd compression, for example the MongoDB 3.11 Java driver, its use can be enabled by the URI or programmatically.

Rotatable Keys

If you have configured a cluster to use keys to authenticate the nodes of a cluster or a sharded cluster, you'll also know that changing those keys has been a chore. MongoDB nodes, such as mongod and mongos previously only knew about one key, so to change the key, all the nodes would need to go down so a new key could be put in place and everything restarted.

That's no longer the case in MongoDB 4.2. Database nodes can now understand config files with multiple keys in them and use those multiple keys. This means that you can now update the database nodes so they have both the old and the new keys in their configuration files. Restarting each of the nodes will make them use both keys. Once done, it's a matter of updating nodes to so they only use the new key and restarting them. The cluster will stay up throughout this entire process.

There's more details on how to do this in the documentation, for replica sets and for sharded clusters.

Configuration Expansion

Keeping secrets in configuration files is always a tricky balancing act between ensuring the files are secure and ensuring the secrets are up to date. It's a balancing act that is going to be easier to manage in MongoDB 4.2 with the addition of Externally Sourced Configuration File Values. When activated, this feature allows config files to contain directives which can retrieve configuration values or entire configuration files from REST endpoints or executable processes.

It's easiest to show by example. Let's start with this configuration file:

storage:
  dbPath: "/var/lib/mongod/"
systemLog:
  destination: file
  path: "/var/log/mongod/mongod.log"
net:
  bindIp: 192.168.123.50,127.0.0.1
  tls:
    mode: requireTLS
    certificateKeyFile: "/etc/tls/mongod.pem"
    certificateKeyFilePassword: "MySecretPassword"

That password for the certificate file is there in plain sight. Ideally, you'd like to move that data to somewhere more centralized and secure. Let's use the __rest directive to move that password somewhere else.

net:
  bindIp: 192.168.123.50,127.0.0.1
  tls:
    mode: requireTLS
    certificateKeyFile: "/etc/tls/mongod.pem"
    certificateKeyFilePassword:
        __rest: "https://configserver.example.net/api/passwords/currentKeyPassword"
        type: "string"

Now, if we started up the server as normal, it would error out given this file. These new options only work if we include --configExpand "rest" in the command line when launching mongod or mongos. When we add that, when the server encounters that __rest directive it will do a GET on the given URL (which has to be https). It will then decode the result as a string as specified by the type option below it.

This all, of course, assumes that you have a REST API server at that URL that will respond appropriately; that is left as an exercise for the system integrator.

The type option can also be yaml allowing a whole set of settings and values to be returned. This opens the way to retrieving the entire configuration file from a REST endpoint.

__rest: "https://configserver.example.net/api/config/fullConfig"
type: "yaml"

There's also the __exec directive which takes a command rather than a URL. The command will be executed locally to obtain the required value or values.

There's another new command line option, --outputConfig which when added to the command line for a mongod or mongos which already has a --configExpand in it, will run through all the directive expansions and then exit, after outputting the config file the server would start working. This allows you the chance to verify those expansions directives are working as expected.

The ability to expand configuration directives with REST retrieved and process generated values opens the way for a whole range of different integrations and we're looking forward to see how users put this to work.