Here are some great articles about MongoDB to read this weekend:
ScaleGrid: Should You Enable MongoDB Journaling?, 10/18
Business Insider: MongoDB co-founder Dwight Merriman and CEO Max Schireson were chosen for the Silicon Alley Top 100 in New York Tech roundup, October
InfoWorld: Use MongoDB to Make Your App Location-Aware, 10/24
comSysto: Getting Started with MongoSoup, 10/25
Digital Misinformation: Collections and Embedded Documents in MongoDB, 10/22
Performance Tuning MongoDB on Solidfire
This is a guest post by by Chris Merz & Garrett Clark, SolidFire We recently had a large enterprise customer implement a MongoDB sharded cluster on SolidFire as the backend for a global e-commerce system. By leveraging solid-state drive technology with features like storage virtualization, Quality of Service (guaranteed IOPS per volume), and horizontal scaling, the customer was looking to combine the benefits of dedicated storage performance with the simplicity and scalability of a MongoDB environment. During the implementation the customer reached out to us with some performance and tuning questions, requesting our assistance with the configuration. After meeting with the team and reviewing the available statistics, we discovered response times that were deemed out of range for the application’s performance requirements. Response times were ~13-20ms (with an average of 15-17 ms). While this is considered acceptable latency in many implementations, the team was targeting < 5ms average query response times. When troubleshooting any storage latency performance issue it is important to focus on two critical aspects of the end-to-end chain: potential i/o queue depth bottlenecks and the main contributors to the overall latency in the chain. A typical end-to-end sequence with attached storage can be described by: MongoDB > OS > NIC > Network > Storage > Network > NIC > OS > MongoDB First off, we looked for any i/o queue depth bottlenecks and found the first one on the operating system layer. MongoDB was periodically sending an i/o queue depth of >100 to the operating system and, by default, iSCSI could only release a 32 queue depth per iSCSI session. This drop from an i/o queue depth of >100 to 32 caused frames to be stalled on the operating system layer while they were waiting to continue down the chain. We alleviated the issue by increasing the number of iSCSI sessions to the volume from 1 to 4, which proportionally increased the queue depth exiting the operating system to 128 (32*4). This enabled all frames coming off the application layer to immediately pass through the operating system and NIC, decreased the overall latency from ~15ms to ~4ms. Despite the latency average being 4ms, performance was still rather variable. We then turned our focus to pinpointing the sources of the remaining end-to-end latency. We were able to determine the latency factors in the stack through the study of three latency loops: First, the complete chain of: MongoDB > OS > NIC > Network > Storage > Network > NIC > OS > MongoDB . This loop took an average of 3.9ms to complete. Secondly, the subset loop of: OS > NIC > Network > Storage > Network > NIC > OS . This loop took ~1.1ms to complete. We determined the latency of this loop by the output of “iostat –xk 1” then greping for the corresponding volume. The last loop segment, latency on the storage subsystem, was 0.7ms and was obtained through a polling API command issued to the SolidFire unit. Our analysis pointed to the first layers of the stack contributing the most significant percent (>70%) of the end-to-end latency, so we decided to start there and continue downstream. We reviewed the OS configuration and tuning, with an eye towards both SolidFire/iSCSI best practices and MongoDB performance. Several OS-level tunables were found that could be tweaked to ensure optimal throughput for this type of deployment. Unfortunately, none of these resulted in any major reduction in the end-to-end latency for mongo. Having eliminated the obvious, we were left with what remained: MongoDB itself. A phrase oft-quoted by the famous fictional detective, Sherlock Holmes came to mind: “when you have eliminated the impossible, whatever remains, however improbable , must be the truth.” Upon going over the collected statistics runs with a fine-toothed comb, we noticed that the latency spikes had intervals of almost exactly 60 seconds. That’s when the light bulb went off… The MongoDB flush interval. The architecture of MongoDB was developed in the context of spinning disk, a vastly slower storage technology requiring batched file syncs to minimize query latency. The syncdelay setting defaults to 60 seconds for this very reason. In the documentation , it is clearly stated “In almost every situation you should not set this value and use the default setting”. ‘Almost’ was the key to our solution, in this particular case. It should be noted that changing syncdelay is an advanced tuning, and should be carefully evaluated and tested on a per-deployment basis. Little’s Law (IOPS = Queue Depth / Latency) indicated that lowering the flush interval would reduce the variance in queue depth thereby smoothing the overall latency. In lab testing, we had found that, under maximum load, decreasing the syncdelay to 1 second would force a ‘continuous flush’ behavior usually repeating every 6-7 seconds, reducing i/o spikes in the storage path. We had seen this as a useful technique for controlling IOPS throughput variability, but had not typically viewed it as a latency reduction technique. It worked! After implementing the change, the customer excitedly reported that they were seeing average end-to-end MongoDB response times of 1.2ms, with a throughput of ~4-5k IOPS per mongod (normal for this application), and NO obvious increase in extraneous i/o. By increasing the number of iSCSI sessions, normalizing the flush rate and removing the artificial 60s buffer, we reduced average latency more than an order of magnitude, proving out the architecture at scale in a global production environment. Increasing the iSCSI sessions increased parallelism, and decreased the latency by 3.5-4x. The reduction in syncdelay had the effect of smoothing the average queue depth being sent to the storage system, decreasing latency by slightly more than 3x. This customer’s experience is a good example of how engaging the MongoDB team early on can ensure a smooth product launch. As of today, we’re excited to announce that SolidFire is a MongoDB partner. Learn more about the SolidFire and MongoDB integration on our Database Solutions page . To learn more about performance tuning MongoDB on SolidFire, register for our upcoming webinar on November 6 with MongoDB. For more information on MongoDB performance, check out Performance Considerations for MongoDB , which covers other topics such as indexes and application patterns and offers insight into achieving MongoDB performance at scale.
How to Prepare for Your Engineering Interview at MongoDB
MongoDB’s Engineering team is full of creative individuals who play an impactful role in building our industry-leading technology. Our interview process is designed to ensure that you and MongoDB are a great match, and, no matter how many interviews you have done in the past, being prepared is the key to being successful. At MongoDB, we do our best to make sure you have a great interview experience and an opportunity to learn about our company, culture, and the people you will be working with. To help you prepare for your technical interviews, we want to share some tips. Research is key Candidates who do research and come prepared for interviews at MongoDB are able to make the most of their interview process. People sometimes think they do not need to do research because they are already familiar with our products, but that will set you up for unexpected surprises. Before beginning your interviews, you should have high-level knowledge of our company’s mission, values, and goals . The in-depth technical information you can learn about MongoDB and the role and team you are interviewing for may also help set you apart from other candidates. MongoDB has a variety of products and Engineering teams, and this information will give you a chance to learn more about what we are working on, technical stacks we use, and what you’d be contributing to if you joined. Take a look at some of the resources below, and use them to your advantage. MongoDB Blog : Our blog is updated regularly with new posts about life at MongoDB, news, products, and events. MongoDB University : This platform was created to empower developers through education. We offer completely free online courses led by Curriculum Engineers for any learner, whether you’re just getting started or already familiar with MongoDB. MongoDB Documentation : The documentation page has detailed information about our products and tools that will give you an idea of what you will be working on as an engineer. MongoDB Developer Hub : The developer hub provides articles on and resources for how to get started with MongoDB. Learn from our Developer Advocates and the MongoDB community! Types of interviews After doing some initial research, it is important to prepare for the actual interviews. Our interview process usually includes one or two virtual interviews and then an onsite interview, which we are currently conducting via Zoom. This may change in accordance with company and COVID-19 guidelines. These interviews and what they cover will vary by team, so it is important to speak with your recruiter and ask for any additional tips or insight into what to expect. Our recruiting process is primarily team-based, which means you’ll interview for a role on a specific team, and many of your interviewers will be team members, as well as your manager. In general, you can expect to receive questions about your background, interest in MongoDB, and why you are interviewing to work with that team. You’ll also have the opportunity to ask your interviewers questions about all things MongoDB. Technical Interviews Technical interviews have a variety of areas that may be covered, including concurrency, distributed systems, algorithms, system design, and language-specific coding. An important part of the technical interview that often goes under the radar is the need for effective communication when talking through your thought process or discussing the problems that are presented. Below are some of the things our engineers look for in a good technical performance. Writing code: strong understanding of the language being used, code is concurrency-safe, works in edge cases, good object-oriented design Software engineering: understanding of data structures and algorithms, considering trade-offs (e.g., run time vs. memory), testing your code Collaboration: clear and concise code that is readable and organized, responding well to suggestions or hints, effective communication about difficulties faced Systems design: design a solution to scale to high levels of concurrency, throughput, and reliability. Does it avoid common bottlenecks, how do we prove its correctness, and what are the trade-offs or alternative solutions? Behavioral Interviews Behavioral interviews focus on how you may add to the culture we continue to build at MongoDB. Reviewing our code of conduct and core values will show you how we operate as a company and what we expect from our employees. Other topics of discussion you should expect in these interviews are successes and failures, what you have learned from these experiences, and what you are looking for in your next role. We will also ask you about your experience with mentoring and learning from other engineers and leaders, your goals and aspirations for the future, and your experience with owning or leading projects. What we offer There are a few things we can promise if you decide to interview for an Engineering role at MongoDB. First, you’ll have a speedy and transparent process with a single, dedicated recruiter. We tailor each of our interview processes to fit the role’s responsibilities and seniority level, and you won’t be asked any riddle questions that aren’t related to the work you’d be doing. Our interview questions are typically sourced from real problems we have had to solve. You’ll also have the opportunity to interact with your future manager and some future teammates, and we hope you find that your interviewers are genuinely interested in you as a person and seeing you succeed at MongoDB. We believe different experiences, identities, and perspectives build a unique culture that helps us create and innovate the next generation of MongoDB. In short, following this guide will help prepare you for a successful interview at MongoDB. Ensure you have gained some knowledge about our company, mission, and goals; the role you’re interviewing for and the team you’d be working on; and the types of interview questions you may be asked. And be prepared with questions for us! We’re so glad you’re interested in joining our team, and we look forward to seeing you in the interview process. Interested in pursuing a career in engineering at MongoDB? We have several open roles on our teams across the globe and would love for you to transform your career with us!