Document Validation for Polymorphic Collections

In data modeling design reviews with customers, I often propose a schema where different documents in the same collection contain different types of data. This makes it efficient to fetch related documents in a single, indexed query. MongoDB's flexible data is great for optimizing workloads in this way, but people can be concerned about losing control of what applications write to these collections.

Customers are often concerned about ensuring that only correctly formatted documents make it into a collection, and so I explain MongoDB's schema validation feature. The question then comes: "How does that work with a polymorphic/single-collection schema?" This post is intended to answer that question — and it's simpler than you might think.

The banking application and its data

The application I'm working on manages customer and account details. There's a many-to-many relationship between customers and accounts. The app needs to be able to efficiently query customer data based on the customer id, and account data based on either the id of its customer or the account id.

Here's an example of customer and account documents where my wife and I share a checking account but each have our own savings account:

Code Snippet

{
    "_id": "kjfgjebgjfbkjb",
    "customerId": "CUST-123456789",
    "docType": "customer",
    "name": {
        "title": "Mr",
        "first": "Andrew",
        "middle": "James",
        "last": "Morgan"
    },
    "address": {
        "street1": "18 Hatfields",
        "city": "London",
        "postCode": "SE1 8DJ",
        "country": "UK"
    },
    "customerSince": ISODate("2005-05-20")
}

{
    "_id": "jnafjkkbEFejfleLJ",
    "customerId": "CUST-987654321",
    "docType": "customer",
    "name": {
        "title": "Mrs",
        "first": "Anne",
        "last": "Morgan"
    },
    "address": {
        "street1": "18 Hatfields",
        "city": "London",
        "postCode": "SE1 8DJ",
        "country": "UK"
    },
    "customerSince": ISODate("2003-12-01")
}

{
    "_id": "dksfmkpGJPowefjdfhs",
    "accountNumber": "ACC1000000654",
    "docType": "account",
    "accountType": "checking",
    "customerId": [
        "CUST-123456789",
        "CUST-987654321"
    ],
    "dateOpened": ISODate("2003-12-01"),
    "balance": NumberDecimal("5067.65")
}

{
    "_id": "kliwiiejeqydioepwj",
    "accountNumber": "ACC1000000432",
    "docType": "account",
    "accountType": "savings",
    "customerId": [
        "CUST-123456789"
    ],
    "dateOpened": ISODate("2005-10-28"),
    "balance": NumberDecimal("10341.21")
}

{
    "_id": "djahspihhfheiphfipewe",
    "accountNumber": "ACC1000000890",
    "docType": "account",
    "accountType": "savings",
    "customerId": [
        "CUST-987654321"
    ],
    "dateOpened": ISODate("2003-12-15"),
    "balance": NumberDecimal("10341.89")
}

As an aside, these are the indexes I added to make those frequent queries I referred to more efficient:

Code Snippet

Adding schema validation

To quote the docs…

Schema validation lets you create validation rules for your fields, such as allowed data types and value ranges.

MongoDB uses a flexible schema model, which means that documents in a collection do not need to have the same fields or data types by default. Once you've established an application schema, you can use schema validation to ensure there are no unintended schema changes or improper data types.

The validation rules are pretty simple to set up, and tools like Hackolade can make it simpler still — even reverse-engineering your existing documents.

It's simple to imagine setting up a JSON schema validation rule for a collection where all documents share the same attributes and types. But what about polymorphic collections? Even in polymorphic collections, there is structure to the documents. Fortunately, the syntax for setting up the validation rules allows for the required optionality.

I have two different types of documents that I want to store in my Accounts collection — customer and account. I included a docType attribute in each document to identify which type of entity it represents.

I start by creating a JSON schema definition for each type of document:

Code Snippet

const customerSchema = {
    required: ["docType", "customerId", "name", "customerSince"],
    properties: {
        docType: { enum: ["customer"] },
        customerId: { bsonType: "string"},
        name: {
            bsonType: "object",
            required: ["first", "last"],
            properties: {
                title: { enum: ["Mr", "Mrs", "Ms", "Dr"]},
                first: { bsonType: "string" },
                middle: { bsonType: "string" },
                last: { bsonType: "string" }
            }
        },
        address: {
            bsonType: "object",
            required: ["street1", "city", "postCode", "country"],
            properties: {
                street1: { bsonType: "string" },
                street2: { bsonType: "string" },
                postCode: { bsonType: "string" },
                country: { bsonType: "string" }  
            }
        },
        customerSince: {
            bsonType: "date"
        }
    }
}

const accountSchema = {
    required: ["docType", "accountNumber", "accountType", "customerId", "dateOpened", "balance"],
    properties: {
        docType: { enum: ["account"] },
        accountNumber: { bsonType: "string" },
        accountType: { enum: ["checking", "savings", "mortgage", "loan"] },
        customerId: { bsonType: "array" },
        dateOpened: { bsonType: "date" },
        balance: { bsonType: "decimal" }
    }
}

Those definitions define what attributes should be in the document and what types they should take. Note that fields can be optional — such as name.middle in the customer schema.

It's then a simple matter of using the oneOf JSON schema operator to allow documents that match either of the two schema:

Code Snippet

I wanted to go a stage further and add some extra, semantic validations:

For customer documents, the customerSince value can't be any earlier than the current time.
For account documents, the dateOpened value can't be any earlier than the current time.
For savings accounts, the balance can't fall below zero.

This document represents these checks:

Code Snippet

I updated the collection validation rules to include these new checks:

Code Snippet

If you want to recreate this in your own MongoDB database, then just paste this into your MongoDB playground in VS Code:

Code Snippet

const cust1 = {
    "_id": "kjfgjebgjfbkjb",
    "customerId": "CUST-123456789",
    "docType": "customer",
    "name": {
        "title": "Mr",
        "first": "Andrew",
        "middle": "James",
        "last": "Morgan"
    },
    "address": {
        "street1": "18 Hatfields",
        "city": "London",
        "postCode": "SE1 8DJ",
        "country": "UK"
    },
    "customerSince": ISODate("2005-05-20")
}

const cust2 = {
    "_id": "jnafjkkbEFejfleLJ",
    "customerId": "CUST-987654321",
    "docType": "customer",
    "name": {
        "title": "Mrs",
        "first": "Anne",
        "last": "Morgan"
    },
    "address": {
        "street1": "18 Hatfields",
        "city": "London",
        "postCode": "SE1 8DJ",
        "country": "UK"
    },
    "customerSince": ISODate("2003-12-01")
}

const acc1 = {
    "_id": "dksfmkpGJPowefjdfhs",
    "accountNumber": "ACC1000000654",
    "docType": "account",
    "accountType": "checking",
    "customerId": [
        "CUST-123456789",
        "CUST-987654321"
    ],
    "dateOpened": ISODate("2003-12-01"),
    "balance": NumberDecimal("5067.65")
}

const acc2 = {
    "_id": "kliwiiejeqydioepwj",
    "accountNumber": "ACC1000000432",
    "docType": "account",
    "accountType": "savings",
    "customerId": [
        "CUST-123456789"
    ],
    "dateOpened": ISODate("2005-10-28"),
    "balance": NumberDecimal("10341.21")
}

const acc3 = {
    "_id": "djahspihhfheiphfipewe",
    "accountNumber": "ACC1000000890",
    "docType": "account",
    "accountType": "savings",
    "customerId": [
        "CUST-987654321"
    ],
    "dateOpened": ISODate("2003-12-15"),
    "balance": NumberDecimal("10341.89")
}

const indexKeys1 = { accountNumber: 1 }
const indexKeys2 = {  accountNumber: 1, docType: 1 } 
const indexOptions1 = {partialFilterExpression: { docType: 'account' }}

const customerSchema = {
    required: ["docType", "customerId", "name", "customerSince"],
    properties: {
        docType: { enum: ["customer"] },
        customerId: { bsonType: "string"},
        name: {
            bsonType: "object",
            required: ["first", "last"],
            properties: {
                title: { enum: ["Mr", "Mrs", "Ms", "Dr"]},
                first: { bsonType: "string" },
                middle: { bsonType: "string" },
                last: { bsonType: "string" }
            }
        },
        address: {
            bsonType: "object",
            required: ["street1", "city", "postCode", "country"],
            properties: {
                street1: { bsonType: "string" },
                street2: { bsonType: "string" },
                postCode: { bsonType: "string" },
                country: { bsonType: "string" }  
            }
        },
        customerSince: {
            bsonType: "date"
        }
    }
}

const accountSchema = {
    required: ["docType", "accountNumber", "accountType", "customerId", "dateOpened", "balance"],
    properties: {
        docType: { enum: ["account"] },
        accountNumber: { bsonType: "string" },
        accountType: { enum: ["checking", "savings", "mortgage", "loan"] },
        customerId: { bsonType: "array" },
        dateOpened: { bsonType: "date" },
        balance: { bsonType: "decimal" }
    }
}

const semanticChecks = {
    "$expr": { 
        "$and": [
            {
                "$not": [
                    { "$and": [
                        { "$eq": ["$docType", "customer"] },
                        { "$gt": ["$customerSince", ISODate()] }
                    ]
                    }
                ]
            },
            {
                "$not": [
                    { "$and": [
                        { "$eq": ["$docType", "account"] },
                        { "$gt": ["$dateOpened", ISODate()] }
                    ]
                    }
                ]
            },
            {
                "$not": [
                    { "$and": [
                        { "$eq": ["$accountType", "savings"] },
                        { "$lt": ["$balance", 0] }
                    ]}
                ]
            }
        ]
    }
}

const schemaValidation = {
    "$and": [
        { $jsonSchema: { oneOf: [ customerSchema, accountSchema ] }},
        semanticChecks
    ]
}
const badDoc1 = {
    dummy: "some stuff"
}

const database = 'MongoBank';
const collection = 'Accounts';

use(database);
db.getCollection(collection).drop();
db.createCollection(collection, {validator: schemaValidation} )
db.getCollection(collection).replaceOne({"_id": cust1._id}, cust1, {upsert: true});
db.getCollection(collection).replaceOne({"_id": cust2._id}, cust2, {upsert: true});
db.getCollection(collection).replaceOne({"_id": acc1._id}, acc1, {upsert: true});
db.getCollection(collection).replaceOne({"_id": acc2._id}, acc2, {upsert: true});
db.getCollection(collection).replaceOne({"_id": acc3._id}, acc3, {upsert: true});
db.getCollection(collection).dropIndexes();
db.getCollection(collection).createIndex(indexKeys1, indexOptions1);
db.getCollection(collection).createIndex(indexKeys2);

Conclusion

I hope that this short article has shown how easy it is to use schema validations with MongoDB's polymorphic collections and single-collection design pattern.

I didn't go into much detail about why I chose the data model used in this example. If you want to know more (and you should!), then here are some great resources on data modeling with MongoDB:

Daniel Coupal and Ken Alger’s excellent series of blog posts on MongoDB schema patterns
Daniel Coupal and Lauren Schaefer’s equally excellent series of blog posts on MongoDB anti-patterns
MongoDB University Course, M320 - MongoDB Data Modeling

MongoDB

Document Validation for Polymorphic Collections

The banking application and its data

Adding schema validation

Conclusion

Related

Create a Data Pipeline for MongoDB Change Stream Using Pub/Sub BigQuery Subscription

Making Diabetes Data More Accessible and Meaningful with Tidepool and MongoDB

Build a Command Line Tool with Swift and MongoDBBuild a Command Line Tool with Swift and MongoDB

The Top 4 Reasons Why You Should Use MongoDB

Table of Contents