The MongoDB Pepsi Challenge

William (Will) Cross, Ph.D
June 19, 2015
#Company

Introduction

Back in February, there was a beer & wine tasting at MongoDB. I decided to implement a soda tasting (with some data collection) and piggyback on that. If you’re interested in the culture of MongoDB, this will give you a sense of some of the stuff that goes down at the end of the day from time to time.

I once read (and I think this is coming from Malcolm Gladwell in Blink, but I can’t find the quote) that if you give someone 3 glasses of soda, one of Coke, one of Pepsi, and one that is either Coke or Pepsi, and asked them, not just which are which, but which one is different from the other two, they can’t do it.

I’d been repeating this claim in conversation for several years, and the typical response I got was for the listener to accuse me of not knowing what the hell I was talking about. Apparently, I was repeating a claim that was, literally, unbelievable for a lot of people.

I wanted to know for sure, and this is the story of how I did some science to find out.

Summary of Results

You want to know the punchline, so here it is: MongoDB employees seem to be, on average, random number generators when it comes to determining what the odd soda out is. We were just as bad at figuring out whether a particular glass was Coke or Pepsi (which wasn’t the main scientific question I had, but I added it on since we were doing the science, anyway). Here are the results on individual questions:

Can MongoDB employees determine which of the 3 sodas is the odd one out?

Success Rate: 36%
Expected Rate from Randomized Guesses: 33%
Margin of Error: 17% (at two standard deviations)
Conclusion: MongoDB employees are not significantly better or worse than a randomized guessing machine in determining which soda is different from the other two.

Can MongoDB employees determine if the cup they chose was Coke or Pepsi?

Success Rate: 62%
Expected Rate from Randomized Guesses: 50%
Margin of Error: 17% (at two standard deviations)
Conclusion: MongoDB employees are not significantly better or worse than a randomized guessing machine at determining if a cup holds Coke or Pepsi.

Can MongoDB employees recognize regular soda?

Success Rate: 81%
Expected Rate from Randomized Guesses: 50%
Margin of Error: 13% (at two standard deviations)
Conclusion: MongoDB employees are significantly better than a randomized guessing machine at determining if a cup holds regular or diet soda (though the study didn’t anticipate asking this question and there may be methodological issues).

Were Confident People Better at Any of These Tests?

People were asked if they were confident in their answers. These are the results for the subset that answered yes. But yes, they did significantly better than chance when it comes to recognizing whether their Chosen Cup was Coke or Pepsi

76% +/-21% (two sigma), vs. 50% for random chance
In a quirk of statistics, this still wasn’t “significantly better” than the “not confident” group, and the results are at the edge of significance in differentiating itself relative to the “randomized guessing” hypothesis. More testing is recommended before we base any theories on this data.

At recognizing whether it’s diet, they did about as well as the rest of the group.

79% +/- 19% (two sigma) vs. 50% for a randomized guessing machine.

How about the not-confident people?

Nothing was significantly different.

Methodology

For those of you who’ve never tried to design an experiment, methodology can easily break an experiment. A well designed experiment will answer your questions about the universe, and a poorly designed one will seem to answer them while really just telling you nothing (if you’re smart and lucky enough to realize it), or else it will tell you something that has more to do with your biases than with the universe, and you won’t even realize that that’s the case. To ensure that I was measuring what I thought I was measuring, I designed the experiment as follows (deviations from plans and known design flaws are also described below):

The drinks were poured in a different room than the room in which they were served.

This prevents test subjects from seeing which beverage was poured into which cup.

The person serving the drinks didn’t know which beverage was in which cup.

This prevented the researchers from communicating information to the subjects
Towards the end of the experiment, this protocol wasn’t followed for the last 4-5 tests because a researcher left.

The person taking the drinks to the server wasn’t the researcher

This was a goal, but wasn’t followed due to low headcount.
If implemented, it would have been one additional layer preventing information about the beverages from being communicated to the subjects

The contents of each cup were determined by a pseudo-random algorithm.

This prevents the researchers from, say, always putting the odd one out in the 3rd cup; if we had done this, and early tasters had let it be known that the third cup is the “odd one out,” this might have biased laters subjects.

We had planned to let the subject choose regular or diet, but instead all subjects were given regular Coke/Pepsi, but were asked if they thought it was diet.

This was due to a miscommunication between the server/answer recorder (who asked test subjects and began marking it down), and the soda pourer, who was expecting to get requests for diet soda from time to time, but who never did, because he never told the server/data recorder to send it to him.

The soda consisted of 2-liter plastic bottles of standard Coke and Pepsi, purchased that day from the Times Square Duane Reade.

The cups were glass, and were washed between uses.

We had intended to use paper cups, and thought there would be lots available, but we were wrong, so we had to improvise at the last minute.

The test subjects were self-selected, and were recruited with an enticing email [1]. They should not be used as a representative sampling of either MongoDB employees nor of any larger demographic groups.

Subjects were anonymous, and were issued subject_id’s that they could later use to look up their results.

Code quality controls:

No unit tests were written
The code was written in haste over the course of an hour or two by yours truly. I initially felt moderately confident that I didn’t write in any bugs severe enough to invalidate the results.
A brief code review was given, with the results of (and I’m quoting from memory four months later) “Going forward, you might want to try to write code that you have a chance of reading a week from now. Still, it’s never going to be used by our team or by the company for anything that will in any way affect any of our jobs, and I have real work to do, so LGTM.”
When running the code, a series of major, crippling bugs became obvious. In each case, I then proceeded to change the script more or less at random until the outputs “looked right” to me.

Safety Procedures:

Because this test dealt with human experimentation, it is typical for any researcher to write up a research plan and submit their research plan to an IRB (Institutional Review Board) to ensure that the research is ethical, that it has at least a small chance of producing useful results, and that the test subjects will not be harmed. These tests are mandatory for all institutions (typically universities) that receive federal research grants.
This protocol wasn’t even attempted, here. Instead, we fed human research subjects a substance associated with weight gain in order to satisfy our own curiosity about whether or not they could tell which harmful substance they were ingesting.

Actual procedure:

In one room, the soda was poured (around 2 ounces per cup, but we did not measure it precisely), and placed on a plate with an experiment_id.
The cups were placed on one of 3 circles on the plate, labelled “A”, “B”, or “C”. At least one (and at most two) was Pepsi, and at least one (and at most two) was Coke, and they all contained High Fructose Corn Syrup.
The plate was then taken to the server/data recorder.
A subject who wished to take the test was issued a subject_id.
At the beginning of the trial, the server / data recorder would record the subject_id in the appropriate place for the experiment_id, and the subject was issued their plate.
The subject would taste the beverages in any order desired, at any speed desired.

When the subject was ready, the data recorder would ask the subject, and record their answers for the following:

Which cup was unique? - In other words, which of the 3 cups contained the odd beverage out? In other words, which cup contained the beverage that was only in that cup and not in any other cup?
What was in that unique cup? - Was it Coke? Or was it pepsi?
Are you confident in your answer? - Yes or No
Was it diet soda? - Yes or No; remember, all soda was regular, and no-one was given diet soda.

Experimenter’s Field Notes

In this section, I will discuss things that happened, and subjective experiences, and tips for anyone trying something similar.

First, if you plan to try this, make sure you have all materials. My understanding, going in, was that we had cups, but we didn’t, and this alone nearly doomed this experiment. Salvaging the experiment involved making human test subjects wash dishes after each trial.

Second, have at least 3 people making beverages, and at least 2 people taking data. This is a very labor intensive experiment.

Third, there was a human element I didn’t anticipate: most test subjects wanted to know, immediately after taking the test, if they were “right”. A single test run isn’t sufficient to determine if they know what they were doing (even a random answer would be correct one time in 3), but people still wanted to know. I didn’t release that data until the test was done (and they had to hold onto their test_id number), but I wish I’d given the test subjects the ability to learn the answer immediately.

Fourth, some test subjects complained that the test was “too easy,” because the carbonation levels of the sodas weren’t the same, making it obvious which soda was the “odd one out.” Therefore, as hard as this is to believe, our experiment may actually overestimate the ability of MongoDB employees to tell one cola from another.

Finally, the experimenters noticed that LOTS of people were extremely confident in their ability to perform well on this test before they took their first sip, but very unsure after their 3+ sips. It was immediately clear to the test subjects that this test was much harder than they thought it would be.

Data

I’ve put the results in CSV format; see 3 charts, attached. [2] I’ve also attached pictures of our raw data, on paper, [3].

In no case can we make claims about individuals’ abilities to determine the type of soda. We just didn’t have enough data from any individual to say with any statistical confidence that they did better than chance.

Those who were highly confident about their choices actually did slightly worse about choosing the “odd one out,” and slightly better at determining whether their choice was Coke or Pepsi, but not in a way that was statistically different from chance, partly because our error bars were pretty big.

Our data set was 32 people, with 36 test runs, so most people took only one test. Because of the small number of test subjects, even two people failing the test would have invalidated our ability to say, with p < 0.05, that we can tell coke from pepsi. 3 people failing to tell the odd drink out would have invalidated our ability to determine if we were better than chance at p < 0.05. In neither case did we get even close to that level of precision. Unfortunately, we didn’t have nearly enough data to make solid claims, except that this test is much harder than people thought it would be.

Conclusion

The one thing we can say with certainty is that the task (determining which of 3 cups of coke or pepsi was different) is difficult. We didn’t have enough data to say almost anything definitively. First, only one individual took the test the minimum number of times (3) to say whether or not they personally could determine the odd beverage out. Second, collectively, we did very close to chance. Third, confidence on the part of the test subject may have either helped or hurt, but our analysis doesn’t reveal which.

Everyone I spoke to about this had a lot of fun. There was a TON of interest, which is how I ended up writing this up in a blog post. When I ran into problems, people stepped up to help. When looking for research subjects, tons of people were jazzed to (1) show off their tasting abilities, and (2) do it for SCIENCE. When the results came out, that started a whole new conversation, and lots of people, from our interns to the board of directors weighed in.

If you’d like to be a test subject the next time we do something like this, you can apply below:

Work at MongoDB

Acknowledgments

First, I’d like to thank our own Richard Kreuter and David Percy, the former for serving drinks & recording data, the latter for assisting in preparing drinks when it became apparent that we were understaffed. I’d also like to thank Maria Hecheverry, who totally cleaned up my mess when I was doing some data analysis at the end of the evening, so that I came back to take care of things, and found that all of the cleaning had already been done.

About the Author - William (Will) Cross, Ph.D
Will is a curriculum developer on the Education team. Among other things, he:

Maintains and updates the online courses at MongoDB University
Builds certification exams
Leads trainings from time to time

His background is in physics; he spent his twenties measuring gravity, most recently publishing a value of G. He has also taught physics at Science Park High School in Newark, NJ.

Notes

_{[1] Email sent to MongoDB Employees to recruit test subjects}

_{Human Test Subjects, One Minion Needed for Human Experimentation}

_Summary: _{I want to perform taste tests to determine who can truly tell Coke from Pepsi (not necessarily which is which) by taste. I will be serving both, in small cups, to people who come by. I require at least one minion laboratory assistant (for reasons stated below), plus as many human test subjects soda drinkers as are sufficiently indifferent to their own health as to be interested in taking the test.}

_Time: _{tomorrow night (February 5) from 6 PM - 7PM}

_Venue: _{MongoDB's New York Headquarters at the Craft Beer & Artisanal Soda Tasting}

_{Experimental set-up:}

_{I will be in a conference room pouring soda into small cups. This will be out of sight of all test-takers and my most senior lab assistant, and I will be labeling the cups "A," "B," and "C," and pouring Coke and Pepsi into them using a completely random pseudo-random method. I will be recording an ObjectId() a sample id for each set of three, along with which beverage is in which container. When all of this is recorded, my lab assistant will then bring the beverage to the taster. The lab assistant will then record the taster's name, choice of which overpriced sugar water tasty beverage is different, and, if the taster is feeling particularly deserving of humiliation confident, whether it is Coke or Pepsi. You will also be asked if you are sure, so that we can determine if you are a poster child for the Dunning-Kruger effect the test subjects' aggregate degree of confidence is associated with greater success or not.}

_{Answers will be revealed at 7 PM. Tasters will have to wait to learn of their degree of humiliating ignorance success rate.}

_{Scientifically demonstrable knowledge for you, personally will be defined as:} _{You took 3 tests that were later all determined to be successfully carried out, and then stopped.} _{You took more than 3 tests, maybe got some wrong, but achieved an overall p-value of < 0.05.}

_{I will also look at some aggregate statistics and see what I can find about the quality of our co-workers.}

_-Will

_{[2] CSV results}
_{[3] View the raw data here and here.}

← Previous

On Selecting a Shard Key for MongoDB

One of the killer features of MongoDB is the built-in sharding capabilities. This feature lets you distribute your data, and your database workload, over multiple commodity-scale machines. While sharding is built-in to MongoDB, there are still a lot of things that you have to get right in order to have a successful installation. One of the trickiest ones is picking a good shard key. Why is picking a good shard key so tricky and so important? A number of reasons: If you pick the wrong shard key, you can totally trash the performance of your cluster Sharding a collection is a one-way parachute jump; if you get it wrong, you’ll need to migrate the data to a new collection with the right shard strategy. Picking the right shard key is more of an art than a science; there are 5 different considerations, and may not be possible to satisfy them all Nonetheless, there are some basic principles involved in picking a good shard key, and I'll go over them now. Recommended Background I'm going to assume that you know how sharding works in MongoDB , and have at least a basic understanding of what a shard key is. If not, you'll need to review the documentation , and ideally sit through a beginning and advanced presentation before reading on. The Perfect Shard Key If you think about it, the perfect shard key would have the following characteristics: All inserts, updates, and deletes would each be distributed uniformly across all of the shards in the cluster All queries would be uniformly distributed across all of the shards in the cluster All operations would only target the shards of interest: an update or delete would never be sent to a shard which didn't own the data being modified Similarly, a query would never be sent to a shard which holds none of the data being queried If your shard key fails to do one of these things, then the following Bad Things could happen: Poor Write Scalability If your write load (inserts, updates, and deletes) isn't uniformly distributed across your shards, then you could end up with a hot shard. Ideally, if you have a 4-shard cluster, you want each shard to take 25% of the write load. That way, your cluster can handle four times the write load that a single replica set could handle. If your shard key ends up directing all of your write load to a single shard, then you end up not having scaled up your write capacity at all. If your shard key ends up directing 75% of your write load to a single shard, and only 25% to the remaining 3 shards, then you've severely limited the benefits of sharding. Poor Read Scalability Similarly, if your read load from your find() operations isn't uniformly distributed across your shards, then you could end up with a hot shard for queries, and for the exact same reasons. Even though a hot shard for queries ends up having less of an impact than having a hot shard for writes, it's still less than optimal. There's another, more subtle, way that a poor shard key can limit read scalability. Ideally, the mongos process would be able to target the query to only the shards which had data. If the mongos cannot target the query, it will have to run a scatter/gather query, in which the query is broadcast to all the shards, and they all report back which data they have. While a scatter/gather query is low-impact on the shards which have no data, it still has some impact. The more shards you have, the more important it is to avoid scatter/gather queries: the impact of scatter/gather queries on a cluster with 50 shards is going to be much higher than on a cluster with 2 shards. Tradeoffs Alas, there is no such thing as the perfect shard key. There are criteria and considerations, but there may not be that you can pick a shard key that will be optimal for all of the operations that you'll perform on your collection. As with most things in MongoDB, you'll have to tune your shard key to the expected use case for your application. Is your application read-heavy? Write-heavy? What are your most common queries? What are your most common writes? You'll always need to make tradeoffs. The critical factor - and the one that you can't do without -- is to have a shard key that matches your workload. Shard Key Considerations With that said, there are five criteria for a good shard key. They are: Cardinality Write Distribution Read Distribution Read Targeting Read Locality These are discussed in the documentation, but here are my comments on each: Cardinality You need to pick a shard key which can be subdivided into small ranges. If you don't do this, MongoDB will be forced to put too many documents in a single chunk. When this happens, you will end up with "jumbo" chunks in your cluster: this will impact performance and manageability of your cluster. Consider an application that stores logs from multiple machines in a distributed system. I chose to shard the logs collection by the machine’s hostname. That choice means that all logs for a given machine go into the same chunk. Because my shard key is the machine’s hostname I have locked myself into having at most one chunk per machine. If a machine can be expected to generate more than 64 MB of logs, MongoDB will not be able to split the chunk. A much better shard key would be a compound shard key, using the machine's hostname along with a timestamp with one-second granularity: MongoDB will always be able to split the chunk, and will always be able to split it at a reasonable split point. Cardinality Reference . Write Distribution As discussed above, you want your write load to be uniformly distributed over the shards in the cluster. A monotonically increasing shard key (such as the date or an ObjectID) will guarantee that all inserts go into a single shard, thus creating a hot shard and limiting your ability to scale write load. There are other ways that you can create a hot shard, but using a monotonically increasing shard key is by far the most common mistake I've seen. If your write load is primarily updates, rather than inserts, you'll want to make sure that those are uniformly distributed across the shards as well. Write Distribution Reference . Read Distribution Similarly, you want your read load to be uniformly distributed over the shards in your cluster. How you need to do this depends on your specific application's anticipated read patterns. For example, consider a blogging application which was sharded by timestamp of article creation where your most common query is "Show me the last 20 articles created". This shard key will result in hot shard for inserts; as well as hot shards for reads. A better shard key would be a compound key where the first field is the 2 digit month number (i.e May is 05, June is 06), followed by a high-granularity field like author-id or hash. The coarse, month prefix is used to search for articles created in the current month while the high-granularity field provides the necessary cardinality to split and distribute chunks. Read Targeting As discussed above, the mongos query router can perform either a targeted query (query only one shard) or a scatter/gather query (query all of the shards). The only way for the 'mongos' to be able to target a single shard is to have the shard key present in the query. Therefore, you need to pick a shard key that will be available for use in the common queries while the application is running. If you pick a synthetic shard key, and your application can't use it during typical queries, all of your queries will become scatter/gather, thus limiting your ability to scale read load. Read Targeting Reference . Read Locality This criterion only applies if you're doing range queries; for example, "show me the last 10 articles posted by this user", or "show me the latest 10 comments on this posting", or even "show me all the articles posted last January". Note that any query with a sort and a limit is a range query. If you're doing range queries, you still want it to be targeted to a single shard, for all the reasons I explained above for "Read Targeting". In turn, this means you want the shard key to be such that all of the documents within the range are on the same shard. The way you typically do this is with a compound shard key. For example, your "articles” collection might be sharded by { userid:1, time_posted:1} If a particular user doesn't post that many articles, they'll all be on a single shard (based on the {userid:1} portion of the shard key) , so your range query (something like find({userid: 'Asya'}).sort({time_posted:-1}).limit(10) ) will only target the shard which has "Asya"'s posting. On the other hand, if "Asya" is a prolific poster, and there are hundreds of chunks with her postings in them, then the {time_posted:1} portion of the shard key will keep consecutive postings together on the same shard. Your query for the latest 10 postings will therefore only have to query one, or at most two, shards. Common Design Patterns There are two design patterns that I think work well for shard key selection. The first is using a hashed shard key, based on a field that is usually present in most queries. Hashed shard keys can often be a good option: out of the 5 criteria, the only one they don't provide is Read Locality. If your application doesn't use range queries, they may be ideal. Two important things to note about hashed shard keys: the underlying field that they're based on must provide enough cardinality, and the underlying field must be present in most queries in order to allow for Read Targeting. Consider the example of a massive multiplayer online game which uses MongoDB to persist user state between gaming sessions. My application describes a user’s state in an individual document per user, and I declare a hashed shard key on the _id field. The _id particularly well suited for a hashed shard key as it is the primary key MongoDB uses for identifying a single document. As such it is both a required field and unique within a collection. Hashing the _id field works great for this pattern since I predominately look up an individual’s game state by the user’s id. The other useful design pattern is a compound shard key, composed of of a low-cardinality ("chunky") first part, and a high-cardinality second part, often a monotonically increasing one. The {userid:1, time_posted:1} example from above is an example of this pattern. If there are enough distinct values in the first part (at least twice the number of shards) you'll get good write and read distribution; the high-cardinality second part gets you good cardinality and read locality. As with the hashed shard key, you need to have at least the first portion of the shard key present in queries in order to get some level of Read Targeting. Ideally, you'd have both portions of the key present in most queries, but it turns out that you can often get most of the benefit even if you only have the first portion . Tradeoffs, Tradeoffs, and More Tradeoffs The most important thing to remember is that it may be impossible to create the perfect shard key. For one thing, these five criteria I listed are typically mutually incompatible: it's very rare to be able to get good write distribution, read distribution, and read locality all with a single shard key. For another thing, your application may have multiple query patterns: a shard key that is perfectly tuned for one type of query may be sub-optimal for another type of query. For example, if you shard an "articles" collection by {userid:1, time_posted:1} , then queries for postings by a single user will be targeted queries, but queries for all recent postings made by all users will necessarily be scatter/gather. To further complicate things, different overall application workloads will call for you to select different shard keys. By arbitrarily specifying different types of read/write/update/sort loads, I can make up use cases where each one of the shard key criteria I listed does not affect performance. (The one exception is cardinality: cardinality is always important.) Here are some example workloads where you can ignore one or more of these criteria. For example: if your workload is 95% inserts and only 5% queries then you really really care about write distribution, care somewhat about cardinality, and the other factors barely matter at all. To take another example: if you have a cluster, and the workload is 90% read, 9.9% updates, and 0.1% inserts, it Really Doesn't Matter if you have a monotonically increasing shard key as long as the 'update' write load is uniformly distributed across the shard key range: your insert load won't be heavy enough to create a hot shard on its own. For a final example: if your application never does range queries, or does them only rarely, then there's no point in considering Read Locality. As such, the only reasonable way to approach MongoDB shard key selection is the way that you approach any other part of MongoDB schema design: you have to carefully consider the requirements arising from all of the different operations your application will perform. Once you have a good idea of the most important requirements, you structure your schema and your shard key to make sure that the important operations are optimized, and the other operations are possible, and reasonably efficient. Summary (aka -- TL;DR) Shard key selection requires thought . The key factors you have to consider are: Cardinality Write distribution Read distribution Read targeting Read locality You may not be able to come up with a shard key that works perfectly for all of your use cases: instead, you must consider all of your operations carefully, make sure that the important ones have been optimized, and that the other ones are reasonably efficient. Good luck -- and may you never have to re-shard a production system! If you’re interested in learning more about the performance best practices of MongoDB, read our guide: Read more about MongoDB performance best practices

June 18, 2015

Next →

Using MongoDB Skill Scanner to Build Better Training Programs

Technology leaders know that transformation is about more than just adopting modern technologies like MongoDB. The entire organization has to rally behind change — which is no easy task. The skills that modern development teams need are evolving faster than ever, and hiring to fill skills gaps can be too time-consuming and expensive of a process for many organizations. So it’s imperative that we plan for how we want to bring our people with us on our modernization journey, and proactively upskill them on the technologies we’re betting on. Because what happens if you choose MongoDB, but your developers don’t know how to use it? CIOs know that training programs are easier said than done. EY reported that 30% of CIOs acknowledge that their training programs are ineffective, and that they’re struggling to retain talent because of it. These leaders come to us to help them build and execute their MongoDB training programs , and seek advice on two extremely common yet critical challenges: How do we get away from the less effective one-size-fits-all approach? How do we measure the ROI of our training program and connect it to business impact? How we use MongoDB Skill Scanner to overcome training challenges Our Professional Services team uses a tool called MongoDB Skill Scanner to address both of these challenges. This tool helps us provide these three benefits to our customers looking to build a training program: Improve MongoDB proficiency: Teams can use Skill Scanner to quickly and easily assess the MongoDB skill gaps of their team members and gain a comprehensive understanding of their team’s MongoDB skills baseline. Increased productivity and accuracy: When team members have a comprehensive understanding of MongoDB, they are able to work more quickly and accurately on projects, leading to increased productivity and a higher quality of work. Save time and money with targeted Training: Using Skill Scanner, customers can avoid wasting time and money on trial-and-error learning. Instead, they can focus on improving their skills in a more targeted and efficient way with right-sized training plans. By leveraging this data, our customers’ engineers can engage in the right training at the right time, targeted for their job role and specific skill shortages. When a training program is built this way, engineers maximize their knowledge retention and minimize time away from their projects. Skill Scanner includes three role-based assessments, one for developers, database administrators, and DevOps respectively. Through a series of multiple choice questions, Skill Scanner provides customers with a clear understanding of their level of expertise across a set of technical skills that are critical for success in their role. After submitting the assessment, engineers will get results in each skill area outlining if they are beginner, intermediate, or advanced. Why data-driven training programs matter We’ve learned that it’s not enough to just tell teams to go watch training videos or webinars on their own, or to place everyone in the same one-size-fits-all program. Skills gaps vary from team to team, and individual to individual. The one-size-fits-all approach of some programs may not address individual learners' needs, wasting time and making it difficult for them to acquire new skills. By using Skill Scanner, we’re able to interpret this data to help determine which training courses your team should take. But we don’t only capture this data before doing training; we use Skill Scanner again after training programs are completed to see where immediate improvements have been made. This helps technology leaders prove the impact and ROI of their training, and gives them the confidence that their teams are ready to be successful with MongoDB. Developing a Precision Learning Program To go even further, our team can work with you to build a Precision Learning Program, where we use Skill Scanner data to build learning schedules that are unique to each individual. These schedules include a variety of short, blended, learning events such as classes, technical workshops, self-paced exercises, and project coaching. We’ve seen PLP lead to higher knowledge retention and of course, measurable project results. A customer who recently concluded their PLP saw a 43% increase in knowledge retention. Getting started building a personalized training program Skill gaps aren’t a novel problem IT leaders are facing. But with new digital courses, training, and technologies, the resources to close these gaps are at your fingertips. Skill Scanner and Precision Learning Program have been specifically designed to empower teams by offering targeted training that enhances their understanding of MongoDB. These short training events are carefully crafted to close skill gaps without compromising developer productivity. We’ve seen a variety of customers use this tool to help train their team’s individual needs, from needing to upskill new hires on their teams, projects with new MongoDB products, migrating to MongoDB Atlas, and more. It also saves your business the hours developers would've wasted searching for answers (and developers don’t want to spend their time that way, either). “We need help getting from point A to point B and feel MongoDB is uniquely positioned to help” — CTO at large insurance firm If you're interested in trying out MongoDB Skill Scanner or want to explore the MongoDB Precision Learning Program further, you can reach out to your account representative or contact us directly .

June 7, 2023