Isa Torres

Solutions Architect

Ethan Steininger

Search Specialist

Historically, the only way to improve your app’s search query relevance is through manual intervention. For example, you can introduce [score boosting](https://github.com/esteininger/atlas-search-guide/tree/master/patterns/3-advanced-scoring) to multiply a base relevance score in the presence of particular fields. This ensures that searches where a key present in some fields weigh higher than others. This is, however, fixed by nature. The results are dynamic but the logic itself doesn’t change.

The following project will showcase how to leverage synonyms to create a feedback loop that is self-tuning, in order to deliver incrementally more relevant search results to your users—*all without complex machine learning models!*

## Example

We have a food search application where a user searches for “Romanian Food.” Assuming that we’re logging every user's clickstream data (their step-by-step interaction with our application), we can take a look at this “sequence” and compare it to other results that have yielded a strong CTA (call-to-action): a successful checkout.

Another user searched for “German Cuisine” and that had a very similar clickstream sequence. Well, we can build a script that analyzes both these users’ (and other users’) clickstreams, identify similarities, we can tell the script to append it to a [synonyms document](https://github.com/esteininger/atlas-search-guide/tree/master/patterns/5-synonyms) that contains “German,” “Romanian,” and other more common cuisines, like “Hungarian.”

Here’s a workflow of what we’re looking to accomplish:

![diagram of what we're looking to accomplish](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/process_diagram_416a9bf4fc.png)

## Tutorial

### Step 1: Log user’s clickstream activity

In our app tier, as events are fired, we log them to a clickstreams collection, like:

```
[{
		"session_id": "1",
		"event_id": "search_query",
		"metadata": {
			"search_value": "romanian food"
		},
		"timestamp": "1"
	},
	{
		"session_id": "1",
		"event_id": "add_to_cart",
		"product_category":"eastern european cuisine",
		"timestamp": "2"
	},
	{
		"session_id": "1",
		"event_id": "checkout",
		"timestamp": "3"
	},
	{
		"session_id": "1",
		"event_id": "payment_success",
		"timestamp": "4"
	},
	{
		"session_id": "2",
		"event_id": "search_query",
		"metadata": {
			"search_value": "hungarian food"
		},
		"timestamp": "1"
	},
	{
		"session_id": "2",
		"event_id": "add_to_cart",
		"product_category":"eastern european cuisine",
		"timestamp": "2"
	}
]

```

In this simplified list of events, we can conclude that {"session_id":"1"} searched for “romanian food,” which led to a higher conversion rate, payment_success, compared to {"session_id":"2"}, who searched “hungarian food” and stalled after the add_to_cart event.
You can import this data yourself using [sample_data.json](https://github.com/esteininger/atlas-search-guide/blob/master/patterns/9-automated-search-tuning/sample_data.json).

Let’s prepare the data for our search_tuner script.

### Step 2: Create a view that groups by session_id, then filters on the presence of searches

By the way, it’s no problem that only some documents have a metadata field. Our $group operator can intelligently identify the ones that do vs don’t.

```
[
    # first we sort by timestamp to get everything in the correct sequence of events,
    # as that is what we'll be using to draw logical correlations
    {
        '$sort': {
            'timestamp': 1
        }
    },
    # next, we'll group by a unique session_id, include all the corresponding events, and begin
    # the filter for determining if a search_query exists
    {
        '$group': {
            '_id': '$session_id',
            'events': {
                '$push': '$$ROOT'
            },
            'isSearchQueryPresent': {
                '$sum': {
                    '$cond': [
                        {
                            '$eq': [
                                '$event_id', 'search_query'
                            ]
                        }, 1, 0
                    ]
                }
            }
        }
    },
    # we hide session_ids where there is no search query
    # then create a new field, an array called searchQuery, which we'll use to parse
    {
        '$match': {
            'isSearchQueryPresent': {
                '$gte': 1
            }
        }
    },
    {
        '$unset': 'isSearchQueryPresent'
    },
    {
        '$set': {
            'searchQuery': '$events.metadata.search_value'
        }
    }
]

```

Let’s create the view by building the query, then going into Compass and adding it as a new collection called group_by_session_id_and_search_query:

![screenshot of creating a view in compass](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/compass_view_474651ee64.png)

![screenshot of the view in compass](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/compass_view_output_e82d97d1bd.png)

Here’s what it will look like:

```
[
  {
    "session_id": "1",
    "events": [
      {
        "event_id": "search_query",
        "search_value": "romanian food"
      },
      {
        "event_id": "add_to_cart",
        "context": {
          "cuisine": "eastern european cuisine"
        }
      },
      {
        "event_id": "checkout"
      },
      {
        "event_id": "payment_success"
      }
    ],
    "searchQuery": "romanian food"
  }, {
    "session_id": "2",
    "events": [
      {
        "event_id": "search_query",
        "search_value": "hungarian food"
      },
      {
        "event_id": "add_to_cart",
        "context": {
          "cuisine": "eastern european cuisine"
        }
      },
      {
        "event_id": "checkout"
      }
    ],
    "searchQuery": "hungarian food"
  },
  {
    "session_id": "3",
    "events": [
      {
        "event_id": "search_query",
        "search_value": "italian food"
      },
      {
        "event_id": "add_to_cart",
        "context": {
          "cuisine": "western european cuisine"
        }
      }
    ],
    "searchQuery": "sad food"
  }
]

```

### Step 3: Build a scheduled job that compares similar clickstreams and pushes the resulting synonyms to the synonyms collection

```
// Provide a success indicator to determine which session we want to
// compare any incomplete sessions with
const successIndicator = "payment_success"

//  what percentage similarity between two sets of click/event streams
//  we'd accept to be determined as similar enough to produce a synonym
//  relationship
const acceptedConfidence = .9

//  boost the confidence score when the following values are present
//  in the eventstream
const eventBoosts = {
  successIndicator: .1
}

/**
 * Enrich sessions with a flattened event list to make comparison easier.
 * Determine if the session is to be considered successful based on the success indicator.
 * @param {*} eventList List of events in a session.
 * @returns {any} Calculated values used to determine if an incomplete session is considered to
 * be related to a successful session.
 */
const enrichEvents = (eventList) => {
  return {
    eventSequence: eventList.map(event => { return event.event_id }).join(';'),
    isSuccessful: eventList.some(event => { return event.event_id === successIndicator })
  }
}

/**
 * De-duplicate common tokens in two strings
 * @param {*} str1
 * @param {*} str2
 * @returns Returns an array with the provided strings with the common tokens removed
 */
const dedupTokens = (str1, str2) => {
  const splitToken = ' '
  const tokens1 = str1.split(splitToken)
  const tokens2 = str2.split(splitToken)
  const dupedTokens = tokens1.filter(token => { return tokens2.includes(token)});
  const dedupedStr1 = tokens1.filter(token => { return !dupedTokens.includes(token)});
  const dedupedStr2 = tokens2.filter(token => { return !dupedTokens.includes(token)});

  return [ dedupedStr1.join(splitToken), dedupedStr2.join(splitToken) ]
}

const findMatchingIndex = (synonyms, results) => {
  let matchIndex = -1
  for(let i = 0; i < results.length; i++) {
    for(const synonym of synonyms) {
      if(results[i].synonyms.includes(synonym)){
        matchIndex = i;
        break;
      }
    }
  }
  return matchIndex;
}
/**
 * Inspect the context of two matching sessions.
 * @param {*} successfulSession
 * @param {*} incompleteSession
 */
const processMatch = (successfulSession, incompleteSession, results) => {
  console.log(`=====\nINSPECTING POTENTIAL MATCH: ${ successfulSession.searchQuery} = ${incompleteSession.searchQuery}`);
  let contextMatch = true;

  // At this point we can assume that the sequence of events is the same, so we can
  // use the same index when comparing events
  for(let i = 0; i < incompleteSession.events.length; i++) {
    // if we have a context, let's compare the kv pairs in the context of
    // the incomplete session with the successful session
    if(incompleteSession.events[i].context){
      const eventWithContext = incompleteSession.events[i]
      const contextKeys = Object.keys(eventWithContext.context)

      try {
        for(const key of contextKeys) {
          if(successfulSession.events[i].context[key] !== eventWithContext.context[key]){
            // context is not the same, not a match, let's get out of here
            contextMatch = false
            break;
          }
         }
      } catch (error) {
        contextMatch = false;
        console.log(`Something happened, probably successful session didn't have a context for an event.`);
      }
    }
  }

  // Update results
  if(contextMatch){
    console.log(`VALIDATED`);
    const synonyms = dedupTokens(successfulSession.searchQuery, incompleteSession.searchQuery, true)
    const existingMatchingResultIndex = findMatchingIndex(synonyms, results)
    if(existingMatchingResultIndex >= 0){
      const synonymSet = new Set([...synonyms, ...results[existingMatchingResultIndex].synonyms])
      results[existingMatchingResultIndex].synonyms = Array.from(synonymSet)
    }
    else{
      const result = {
        "mappingType": "equivalent",
        "synonyms": synonyms
      }
      results.push(result)
    }

  }
  else{
    console.log(`NOT A MATCH`);
  }

  return results;
}

/**
 * Compare the event sequence of incomplete and successful sessions
 * @param {*} successfulSessions
 * @param {*} incompleteSessions
 * @returns
 */
const compareLists = (successfulSessions, incompleteSessions) => {
  let results = []
  for(const successfulSession of successfulSessions) {
    for(const incompleteSession of incompleteSessions) {
      // if the event sequence is the same, let's inspect these sessions
      // to validate that they are a match
      if(successfulSession.enrichments.eventSequence.includes(incompleteSession.enrichments.eventSequence)){
        processMatch(successfulSession, incompleteSession, results)
      }
    }
  }
  return results
}

const processSessions = (sessions) => {
  // console.log(`Processing the following list:`, JSON.stringify(sessions, null, 2));
  // enrich sessions for processing
  const enrichedSessions = sessions.map(session => {
    return { ...session, enrichments: enrichEvents(session.events)}
  })
  // separate successful and incomplete sessions
  const successfulEvents = enrichedSessions.filter(session => { return session.enrichments.isSuccessful})
  const incompleteEvents = enrichedSessions.filter(session => { return !session.enrichments.isSuccessful})

  return compareLists(successfulEvents, incompleteEvents);
}

/**
 * Main Entry Point
 */
const main = () => {
  const results = processSessions(eventsBySession);
  console.log(`Results:`, results);
}

main();

module.exports = processSessions;

```

Run [the script](https://github.com/esteininger/atlas-search-guide/blob/master/patterns/9-automated-search-tuning/realm/index.js) yourself.

### Step 4: Enhance our search query with the newly appended synonyms

```
[
    {
        '$search': {
            'index': 'synonym-search',
            'text': {
                'query': 'hungarian',
                'path': 'cuisine-type'
            },
            'synonyms': 'similarCuisines'
        }
    }
]

```

See [the synonyms tutorial](https://github.com/esteininger/atlas-search-guide/tree/master/patterns/5-synonyms).

## Next Steps

There you have it, folks. We’ve taken raw data recorded from our application server and put it to use by building a feedback that encourages positive user behavior.

By measuring this feedback loop against your KPIs, you can build a simple A/B test against certain synonyms and user patterns to optimize your application!

This blog will cover how to leverage synonyms to create a feedback loop that is self-tuning, in order to deliver incrementally more relevant search results to your users.

Improve Your App's Search Results with Auto-Tuning

Harshad Dhavale

Senior Technical Services Engineer

We live in an increasingly globalized economy. By extension, users have expectations that our applications will understand the context of their culture and by extension: language.

Luckily, most search engines—including, Atlas Search—support [multiple languages](https://www.mongodb.com/docs/atlas/atlas-search/analyzers/language/). This article will walk through three options of query patterns, data models, and index definitions to support your various multilingual application needs.

To illustrate the options, we will create a fictitious scenario. We manage a recipe search application that supports three cultures, and by extension, languages: English, Japanese (Kuromoji), and German. Our users are located around the globe and need to search for recipes in their native language.

## 1. Single field

We have one document for each language in the same collection, and thus each field is indexed separately as its own language. This simplifies the query patterns and UX at the expense of bloated index storage.

**Document:**

```
[
  {"name":"すし"},
  {"name":"Fish and Chips"},
  {"name":"Käsespätzle"}
]
```

**Index:**

```
{
  "name":"recipes",
  "mappings": {
    "dynamic": false,
    "fields": {
      "name": {
        "type": "string",
        "analyzer": "lucene.kuromoji"
      },
      "name": {
        "type": "string",
        "analyzer": "lucene.english"
      },
      "name": {
        "type": "string",
        "analyzer": "lucene.german"
      }
    }
  }
}
```

**Query:**

```
{
  "$search": {
    "index": "recipes",
    "text": {
      "query": "Fish and Chips",
      "path": "name"
    }
  }
}
```

**Pros:**

* One single index definition.
* Don’t need to specify index name or path based on user’s language.
* Can support multiple languages in a single query.

**Cons:**

* As more fields get added, the index definition needs to change.
* Index definition payload is potentially quite large (static field mapping per language).
* Indexing fields as irrelevant languages causing larger index size than necessary.

## 2. Multiple collections

We have one collection and index per language, which allows us to isolate the different recipe languages. This could be useful if we have more recipes in some languages than others at the expense of lots of collections and indexes.

**Documents:**

```
recipes_jp:
[{"name":"すし"}]

recipes_en:
[{"name":"Fish and Chips"}]

recipes_de:
[{"name":"Käsespätzle"}]
```

**Index:**

```
{
  "name":"recipes_jp",
  "mappings": {
    "dynamic": false,
    "fields": {
      "name": {
        "type": "string",
        "analyzer": "lucene.kuromoji"
      }
    }
  }
}

{
  "name":"recipes_en",
  "mappings": {
    "dynamic": false,
    "fields": {
      "name": {
        "type": "string",
        "analyzer": "lucene.english"
      }
    }
  }
}

{
  "name":"recipes_de",
  "mappings": {
    "dynamic": false,
    "fields": {
      "name": {
        "type": "string",
        "analyzer": "lucene.german"
      }
    }
  }
}
```

**Query:**

```
{
  "$search": {
    "index": "recipes_jp"
    "text": {
      "query": "すし",
      "path": "name"
    }
  }
}
```

**Pros:**

* Can copy the same index definition for each collection (replacing the language).
* Isolate different language documents.

**Cons:**

* Developers have to provide the language name in the index path in advance.
* Need to potentially copy documents between collections on update.
* Each index is a change stream cursor, so could be expensive to maintain.

## 3. Multiple fields

By embedding each language in a parent field, we can co-locate the translations of each recipe in each document.

**Document:**

```
{
  "name": {
    "en":"Fish and Chips",
    "jp":"すし",
    "de":"Käsespätzle"
  }
}
```

**Index:**

```
{
  "name":"multi_language_names",
  "mappings": {
    "dynamic": false,
    "fields": {
      "name": {
        "fields": {
          "de": {
            "analyzer": "lucene.german",
            "type": "string"
          },
          "en": {
            "analyzer": "lucene.english",
            "type": "string"
          },
          "jp": {
            "analyzer": "lucene.kuromoji",
            "type": "string"
          }
        },
        "type": "document"
      }
    }
  }
}
```

**Query:**

```
{
  "$search": {
    "index": "multi_language_names"
    "text": {
      "query": "Fish and Chips",
      "path": "name.en"
    }
  }
}
```

**Pros:**

* Easier to manage documents.
* Index definition is sparse.

**Cons:**

* Index definition payload is potentially quite large (static field mapping per language).
* More complex query and UX.

This article will walk through three options of query patterns, data models, and index definitions to support your various multilingual application needs. 

Atlas Search Multi-Language Data Modeling

## Introduction
In this walkthrough, I will show how the flexibility of Atlas Search's inverted indexes are a powerful option versus traditional b-tree indexes when it comes to supporting ad-hoc queries. 

## What is flexible querying?
Flexible query engines provide the ability to execute a performant query that spans multiple indexes in your data store. This means you can write ad-hoc, dynamically generated queries, where you don't need to know the query, fields, or ordering of fields in advance. 

Be sure to check out the [MongoDB documentation on this subject](https://www.mongodb.com/docs/manual/core/index-intersection/)!

It's very rare that MongoDB’s query planner selects a plan that involves multiple indexes. In this tutorial, we’ll walk through a scenario in which this becomes a requirement. 

### Your application is in a constant state of evolution

Let’s say you have a movie application with documents like:

```
{
  "title": "Fight Club",
  "year": 1999,
  "imdb": {
    "rating": 8.9,
    "votes": 1191784,
    "id": 137523
  },
  "cast": [
    "Edward Norton",
    "Brad Pitt"
  ]
}
```


### Initial product requirements

Now for the version 1.0 application, you need to query on title and year, so you first create a compound index via:

`db.movies.createIndex( { "title": 1, "year": 1 } )`

Then issue the query:

`db.movies.find({"title":"Fight Club", "year":1999})`

When you run an explain plan, you have a perfect query with a 1:1 documents-examined to documents-returned ratio:

```
{
  "executionStats": {
    "executionSuccess": true,
    "nReturned": 1,
    "executionTimeMillis": 0,
    "totalKeysExamined": 1,
    "totalDocsExamined": 1
  }
}
```

### Our query then needs to evolve

Now our application requirements have evolved and you need to query on cast and imdb. First you create the index:

`db.movies.createIndex( { "cast": 1, "imdb.rating": 1 } )`

Then issue the query:

`db.movies.find({"cast":"Edward Norton", "imdb.rating":{ $gte:9 } })`

Not the greatest documents-examined to documents-returned ratio, but still not terrible:

```
{
  "executionStats": {
    "executionSuccess": true,
    "nReturned": 7,
    "executionTimeMillis": 0,
    "totalKeysExamined": 17,
    "totalDocsExamined": 17
  }
}
```

### Now our query evolves again

Now, our application requires you issue a new query, which becomes a subset of the original:

`db.movies.find({"imdb.rating" : { $gte:9 } })`

The query above results in the dreaded **collection scan** despite the previous compound index (cast_imdb.rating) comprising the above query’s key. This is because the "imdb.rating" field is not the index-prefix, and the query contains no filter conditions on the "cast" field."

*Note: Collection scans should be avoided because not only do they instruct the cursor to look at every document in the collection which is slow, but it also forces documents out of memory resulting in increased I/O pressure.*

Our query plan results as follows:

```
{
  "executionStats": {
    "executionSuccess": true,
    "nReturned": 31,
    "executionTimeMillis": 26,
    "totalKeysExamined": 0,
    "totalDocsExamined": 23532
  }
}
```

Now you certainly could create a new index composed of just imdb.rating, which would return an index scan for the above query, but that’s three different indexes that the query planner would have to navigate in order to select the most performant response.
## Alternatively: Atlas Search
Because Lucene uses a different index data structure ([inverted indexes](https://en.wikipedia.org/wiki/Inverted_index) vs [B-tree indexes](https://en.wikipedia.org/wiki/B-tree)), it’s purpose-built to run queries that overlap into multiple indexes.

Unlike compound indexes, the order of fields in the Atlas Search index definition is not important. Fields can be defined in any order. Therefore, it's not subject to the limitation above where a query that is only on a non-prefix field of a compound index cannot use the index.

If you create a single index that maps all of our four fields above (title, year, cast, imdb):

```
{
  "mappings": {
    "dynamic": false,
    "fields": {
      "title": {
        "type": "string",
        "dynamic": false
      },
      "year": {
        "type": "number",
        "dynamic": false
      },
      "cast": {
        "type": "string",
        "dynamic": false
      },
      "imdb.rating": {
        "type": "number",
        "dynamic": false
      }                  
    }
  }
}
```


Then you issue a query that first spans title and year via a must (AND) clause, which is the equivalent of  `db.collection.find({"title":"Fight Club", "year":1999})`:

```
[{
  "$search": {
    "compound": {
      "must": [{
          "text": {
            "query": "Fight Club",
            "path": "title"
          }
        },
        {
          "range": {
            "path": "year",
            "gte": 1999,
            "lte": 1999
          }
        }
      ]
    }
  }
}]
```

The corresponding query planner results:

```
{
  '$_internalSearchIdLookup': {},
  'executionTimeMillisEstimate': 6,
  'nReturned': 0
}
```

Then when you add `imdb` and `cast` to the query, you can still get performant results:

```
[{
    "$search": {
      "compound": {
        "must": [{
            "text": {
              "query": "Fight",
              "path": "title"
            },
            {
              "range": {
                "path": "year",
                "gte": 1999,
                "lte": 1999
              },
              {
                "text": {
                  "query": "Edward Norton",
                  "path": "cast"
                }
              },
              {
                "range": {
                  "path": "year",
                  "gte": 1999,
                  "lte": 1999
                }
              }
            ]
          }
        }
      }]
```

The corresponding query planner results:

    {
      '$_internalSearchIdLookup': {},
      'executionTimeMillisEstimate': 6,
      'nReturned': 0
    }



## This isn’t a peculiar scenario

Applications evolve as our users’ expectations and requirements do. In order to support your applications' evolving requirements, Standard B-tree indexes simply cannot evolve at the rate that an inverted index can.

### Use cases

Here are several examples where Atlas Search's inverted index data structures can come in handy, with links to reference material:

- [GraphQL](https://www.mongodb.com/docs/atlas/app-services/graphql/): If your database's entry point is GraphQL, where the queries are defined by the client, then you're a perfect candidate for inverted indexes
- [Advanced Search](https://www.mongodb.com/docs/atlas/atlas-search/compound/): You need to expand the filtering criteria for your searchbar beyond several fields.
- [Wildcard Search](https://www.mongodb.com/docs/atlas/atlas-search/regex/#std-label-regex-ref): Searching across fields that match combinations of characters and wildcards.
- [Ad-Hoc Querying](https://www.mongodb.com/docs/atlas/atlas-search/queryString/): The need to dynamically generate queries on-demand by our clients.

# Resources

- Full code walkthrough via a [Jupyter Notebook](https://github.com/esteininger/atlas-search-guide/blob/master/patterns/12-flexible-querying/FLEXIBLE%20QUERYING.ipynb)


Atlas Search provides the ability to execute a performant query that spans multiple indexes in your data store. It's very rare, however, that MongoDB’s query planner selects a plan that involves multiple indexes. We’ll walk through a scenario in which this becomes a requirement. 

Ethan Steininger

Content by Ethan Steininger

Improve Your App's Search Results with Auto-Tuning

Atlas Search Multi-Language Data Modeling

Flexible Querying with Atlas Search