This is Part 2, continuing from “Part 1: Practical Considerations.” In Part 1, the foundational concerns were laid out. Here in Part 2, common use cases and patterns to address them with best practices are covered.
Schema design process for search
The search use cases that MongoDB Search is asked to support need to be evaluated within the identified physical constraints to produce a suitable schema design. MongoDB Search indexes content based directly on each field’s name and data type and returns the matching documents (or embedded documents) and facets from search requests. Modeling documents to use case needs is straightforward; what you index is what you get - you index “movies” and $search returns movies, or you index products, and $search returns products. The documents in the collection are what is returned and thus conceptually represent the types and granularity of “things” the user is being provided in response to a query.
Let’s use a movie search engine example: Movies themselves make sense to be a first-class search result item. What about cast members? Are they simply attributes and a means to find a movie? Or are they first-class entities that warrant being an independent item in search results? First-class search result items should be modeled as documents in a collection. How about genre? Would the users of a movie search system want to get to Documentaries as an entity unto itself, by typing only “doc” and it being the first suggestion?
Build with the end experience in mind
The search experience delivered by your application is a crucial part of the usefulness and satisfaction provided to your users; a lack of a quality search feature will deter users. Delightful search experiences provide facilities such as as-you-type suggestions, faceted navigation, and keyword highlighting, all while factoring in our explicit and implicit preferences by filtering out or boosting items. The quality of your application's search experience consists of much more than just a list of documents for a search query. It could be incorporating search into subtle areas such as “more like this” sections of existing pages, providing context-sensitive filtering and navigation of organized content, and other UX enhancements. The takeaway is this: Model and build to these experiences.
Some questions to ask yourself as you begin the modeling process:
What types of queries do you expect?
How is your content best organized for navigation and searchability?
When search results are presented, how can the app best help users narrow or expand their search?
Useful search design patterns
Don’t look at patterns to find a problem to solve, but rather find the patterns that best fit the challenges you’re facing. Several of the tried and true MongoDB design patterns work well for search use cases (and one in particular does not!!).
Attribute Pattern and search
The Attribute Pattern is a commonly used document database technique to represent key/value pairs in a structure that does not introduce a potentially unlimited number of field names (or keys) by representing a key/value pair as a JSON { k: “key”, v: “value”} structure. Keys are values to a k field in a sub-document. Because MongoDB Search uses field names for mapping, k is the field name for all attributes, and thus a specific attribute key cannot be mapped, and even more insidious is that v values could potentially be of different types of data, even if all values for a particular key are of the same type, because of the blending of all values together in a single index. The embeddedDocuments field type can help with conservative use of the Attribute Pattern for some capabilities, but that comes with other considerations (see the nested documents section above). The best course of action for attributes that need to be searched? Model or transform them as straightforward field_name: value fields on the documents, where they can be handled in a type-sensitive manner. All values of a given field are indexed together and separate from other fields.
Entity pattern for search
The Entity Pattern models all key domain entities in a way that supports the “find anything anywhere” and “relevant as you type suggestions” use cases. Entities are represented as first-class documents in a specialized collection and a MongoDB Search index configuration tuned for matching domain entities in many ways and allowing for customized ranking factors. All of your domain entities can be indexed and findable in a myriad of ways using this pattern. Of course, scale matters, so this perhaps isn’t a practical pattern for billions of things, though many hundreds of millions are generally a pretty tractable scale. How many unique entities do you have in your system? A quick ballpark estimate for a realistic product catalog system, representing products, customers, and orders, could have, say, 10k products across 10 categories, serving around 1M customers who each could have a billing and shipping address within any of the 50 United States. Across around 200k cities, that would be 10k products + 1M customers + 2M addresses + 50 states + 200k cities = ~3.2M entities, which is no big deal scale-wise.
If your situation consists of only a single collection, with only one or perhaps more entity types, and they are modeled as first-class documents already, this pattern is effectively what you’ve already got — all domain entities represented as first-class documents in a single collection. A collection of all domain entities blended together and typed exemplifies the Single-Collection Pattern.
It's often the case that your main collection represents one type of entity as documents, and other domain entities as metadata fields or nested documents. For example, let’s take the infamous sample movie data: As a user types, movie titles certainly should be suggested. But what about cast member names? Can I find movies starring Keanu Reeves by typing only "kea"? What about documentaries by only typing "doc"? Both cast (actor name) and genre fields (fields, not documents) are, for a movie search application, truly first-class domain entity objects, and in a findability sense, deserve to be first-class documents in a collection backed by MongoDB Search.
It’s a simple model with the following basic schema:
It’s important that entity documents have stable, unique identifiers, as the entities will be regularly refreshed from their source of truth collection. Assigning a type to each entity allows for filtering, grouping, boosting, or faceting by type.
Additionally, modeling entities directly as individual documents allows each to carry optional metadata fields to assist in ranking, displaying, filtering, or grouping them. In the movie example, cast member entity documents could carry a computed number of movies the actor has been in, and the average rating of all those movies. These types of computed values, aggregated during the entities collection sync process, exhibit the Computed Pattern.
At the heart of this solution, the straightforward document model feeds the name field through a sophisticated index configuration, which analyzes it in numerous ways, enabling incredible query and ranking power.
The Entity Pattern for Search, which debuted in Relevant As-You-Type Suggestions Search Solution (see the data modeling approach section), is specialized for the as-you-type/autocomplete/typeahead use case.
Entity collection uses
As-you-type suggest
Type-specific lookups (dropdown of Categories, States, Names)
General index of All (your) Things
This collection can be used to initiate lookups and limit the results to only a few (or even just one) result, making looking up the canonical, more detailed data straightforward and performant.
Basic search operations on an entity collection make these use cases straightforward and performant:
Filtering by type
Dropdown entry for cities (“type” equals “city”)
Saved searches: Log users’ “saved searches” as entity type “saved_search” with an additional “username” field on each of those types of documents. As a user types, their saved searches can be displayed. No additional collection would be needed — this could be the source of truth collection for “saved searches.” For this type of entity, it would also be wise to add a timestamp, so they could be sorted by recency.
Faceting by type
As a user types, entities are returned, and additionally, the types of entities found and their count within the matched result set can be returned. These counts are useful for users to see the bigger context of the types of things being matched and facilities filtered navigation by type from there.
With no filter criteria, a $seachMeta faceting by type will quickly give the full set of entity types available. These global counts can be useful, again, for navigation of your “domain entity space” and also for diagnostic purposes to ensure the number of entities of each type matches the counts from the source of truth data.
Boosting by type
Perhaps certain entity types make sense to boost over other entity types in the results. As a user types in a movie database search, genre entity types and movie entity types could be boosted over cast member name types, so more commonly searched-for entities surface ahead in a list of suggestions.
Similarly, a user's saved search entities could be prioritized over product or category types of entities to increase the value to your customers' experience.
Boosting by entity weights
In a movie database search, boost cast members by a factor related to the ratings of the movies they’ve been in. The average rating of all movies for an actor is not the best weight to use, as that prioritizes an actor who’s only been in one highly rated movie ahead of actors who have been in lots of movies, most of which have rated well. But there are ways to give seasoned actors a boost by using the sum of all their movie ratings rather than an average. The more good movies they’ve been in, the bigger their boost. Boost factors based on the entity data itself provide a flexible way to add “smart” weightings to results based on your business goals.
Populating an entity collection from existing data
You’ve already got data, and it’s well modeled for other use cases nested within sub-structures of related documents. However, the structure of that data doesn’t lend itself to being first-class findable. Leave the data where it fits the other use cases best, keeping the source of truth right where it already is. The unique entities nested within documents in other collections can generally be extracted and grouped uniquely, computing any useful values along the way. Maintaining a synchronization of data into the entities collection does require consideration and a process to implement.
A great example of an export/merge process comes from the movie collection data, where genre is an attribute of a movie, yet deserves to be a first-class domain entity. From the movies collection, we unwind the genres field from its nested per-movie value, then group genres into unique values, counting the number of movies for each genre along the way. Then, we format the unique genres into the entities collection schema (_id, type, name, etc.) where the _id is unique per genre and made to fit uniquely within the _id space of a blended collection of various entity types. And finally, these genre entity documents are merged into the catch-all entities collection. The $merge stage will overwrite documents with the same _id, making it important to uniquely and stably identify entities so they can easily be updated.
The following code snippet can be seen in full context on GitHub.
Each entity type could have its own synchronization process and type-specific additional fields.
Entity index configuration
The entities collection’s index configuration specifies the mappings on the type and name fields, otherwise leaving it dynamically mapped to accommodate additional custom fields as needed.
View a full example configuration for the as-you-type use case.
A few specific pieces of this configuration are worth highlighting. The entity type field is indexed in two different ways: as a token for filtering or boosting by exact value, and for faceting-by-type needs.
The heart of entity findability stems from the indexing strategies applied to the name field. MongoDB Search provides a powerful “multi” indexing technique for string fields, where that one string value is indexed numerous ways to facilitate any number of ways to match it. Strings are analyzed, a process creating a series of indexable units called terms. Terms represent words, character fragments, phonetics, or other lexical extraction tricks. In the example configuration mentioned above, the name field is indexed with just the left edge (“starts with”), as English text (“search”, “searches”, “searching”, “searched” all indexed the same), lowercase (case-insensitive matching), phonetic (“sounds like”), shingled (pairs of words concatenated and indexed together, good for relevancy boosting of words commonly used together), and using the default language-agnostic word-break delimited extraction analysis.
These various ways of analyzing the name field give the text and phrase operators potent ways of matching and ranking. Here’s an example as-you-type suggest query scenario, matching liberally on very little query text (“matr”) yet boosting best matches to the top:
From this MongoDB Search Playground, we weigh each analyzed variant of the name field differently, and then multiply the result by any specified weight on the matched entity:
Searching across multiple indexed fields, performantly, in combinations with one another, is a superpower of MongoDB Search — and it’s the key to nuanced and controllable result relevancy.
Type-specific metadata
To avoid field name collisions on any custom entity type-specific fields, the fields can be nested into type-specific sub-documents, like this book sub-structure:
The MongoDB Search index configuration of the entities collection could then add specific document-type mappings for overrides to the default dynamic mappings in place, perhaps making the number of pages of book entities facetable:
Summary: Key points for Data Modeling and Schema Design for Search
Model for the End Experience: Design your schema and search capabilities by starting with the desired UI/UX and identifying the exact objects or entities your users need searched and returned.
First-Class Documents: Represent searchable items and domain entities as first-class documents within their own collections rather than relying heavily on deeply nested structures or ambiguous attributes. This is a general best practice, though embeddedDocuments and returnScope are very powerful when the use case fits them.
Consistent Field Types: Ensure that fields with the same name maintain consistent data types across all collection documents to prevent query and faceting issues.
Optimize Indexing: Adhere to the principle that "data that is searched together, is indexed together," using tailored typeSets, views, and mappings to manage index size and work within physical constraints.
Next Steps
To learn more about MongoDB Search, visit our documentation.
Ready to start building? Get started with MongoDB Atlas for free today.