MongoDB customers and community members are the people who realize GIANT ideas. We are excited to begin highlighting some of our community members, our MongoDB Giants, who are tackling challenging problems and bringing solutions to life with MongoDB.
March’s Giant of the Month is Mike Grayson, Senior MongoDB DBA at PayChex, a provider of payroll, human resource, and benefits outsourcing solutions for small to medium-sized businesses. Mike has been involved in many aspects of the MongoDB community since he started using the database in 2014. He received numerous internal awards from PayChex for his dedication to operationalizing their new system and educating his teams on the new database technology. In his own words, “with lots of help from Ops Manager, integrating MongoDB in to our ecosystem has been a great and painless process.”
Mike was also instrumental in the MongoDB 3.2 release and participated in Beta Testing for The Encrypted Storage Engine and MongoDB Compass. Paychex is a member of MongoDB’s customer advisory board and through their involvement Michael provides input into MongoDB’s product development.
In his spare time, if he’s not spending his time on the Advocacy Hub or reading about MongoDB and other databases, you can find Mike playing video games, rooting for Bayern Munich, or any of his favorite Philadelphia sports teams with his wife, three girls, and two dogs in the beautiful Finger Lakes Region of New York.
Have your voice heard in the MongoDB community. Join our Advocacy Hub and start getting involved today.
Using MongoDB, Kafka and Spark to Build Infrastructure for India’s First Affordable Smart-Homes Project
By Gautam Rege, Co-Founder of Josh Software and Co-Founder of SimplySmart Solutions In Sheltrex , a growing community about two hours outside of Mumbai, India, we’re part of a project that will put more than 100,000 people in affordable smart homes. To make those homes truly smart we’re building infrastructure that streams data from millions of sensors in near real-time. Citizens can then access the data through a mobile application that allows them to better manage their home. It’s a fantastic example of how technology can improve our lives, but building scalable and fast infrastructure is not simple. In this blog, I want to highlight how my team at Josh Software , one of India’s leading internet of things and web application specialists, is overcoming those challenges by using a stack of interesting data tools like Apache Kafka, Apache Spark and MongoDB . Of the planned 20,000 homes in Sheltrex, more than 1,500 have already been completed. Many people people are already living on site. The pilot is a proving ground for a whole host of smart township technologies. From mobile connected security to smart-meters monitoring power consumption. Along with the mobile application for individual citizens we’ve also built software that will aggregate this data for the entire community. This gives the township the ability to negotiate more competitive rates from India’s electricity providers. Sheltrex affordable home project in Karjat, India To provide homeowners and the community with accurate and timely utility data means processing information from millions of sensors quickly, then storing it in a robust and efficient way. The Smart City Application communicates with our stack APIs to make business sense for residents and the township management. The entire solution is split into two “universes.” Universe One is where we stream all the sensor data that is flooding in from the homes in real time. This could include data points like temperature or energy usage. The sensor and smart-meter data is first ingested into a messaging system powered by Kafka (an open source, high-throughput, distributed, publish-subscribe platform that can quickly process real-time data feeds at a large scale). Through Kafka the data is dropped into Spark , a large-scale data processing engine that is basically a much faster and simpler alternative to MapReduce. It’s in Spark, using Java and Python, that we do the processing and aggregation of the data - before it’s written on to our second “universe.” Universe Two is where the smart home data is stored and accessed by the mobile application. We need something fast, flexible and robust, so we turned to MongoDB. It is the primary database for all storage, analysis and archiving of the smart home data. This includes time-series data like regular temperature information, as well as enriched metadata such as accumulated electricity costs and usage rates. To connect the analytical and operational data sets we use the MongoDB Connector for Hadoop . We’ve found that the three technologies work well in harmony, creating a resilient, scalable and powerful big data pipeline, without the complexity inherent in other distributed streaming and database environments. Both in development, where it’s relatively simple to integrate them, and in production where the data flows smoothly between each stage. Smarter, faster I’ve been using MongoDB since the beginning, in fact, I’ve written a couple of books on the subject . It’s been great to see how the database itself has matured and kept adding the right features at the right time. Another big advantage for us is how much more productive MongoDB makes developers and operations staff. The devops team is continuously delivering code to support new requirements, so they need to make things happen fast. MongoDB’s ease of use means we can accelerate our development process and get new features integrated, tested and deployed quickly. Right now we’re operating across eight Amazon Web Services instances in the same zone. As the project expands and more citizens move into Sheltrex we expect to see huge growth. That’s why it’s been so important for us to leverage technologies that operate efficiently at scale. Sheltrex affordable home project in Karjat, India So far the pilot has been incredibly successful and we’re pleased with how our infrastructure is steadily increasing it’s capacity as thousands of new homes come online. But what we’re doing in Sheltrex is only the beginning. Housing is a volume game, as more people live in smart affordable homes the greater the effect will be for the community and the environment. I believe this type of affordable and intelligent housing should become standard across the world. Minor initial costs lead to massive efficiencies over the lifetime of the building. These are not simply monetary - consider the wasted water and electricity that we could save. To get there it will take political will and, of course, considerable funding, but from my point of view the technology is ready to go today. By building our giant idea on modern and mature technologies like MongoDB, we’re ready to change the world. About Josh Software & SimplySmart Driven by enthusiasm and passion, Josh is India’s leading company in building innovative web applications working exclusively in Ruby On Rails since 2007. The company thrives only on three basic needs - disruption, innovation, and learning. It builds products for customers who are able to fulfil at least two of these needs. Details are available at www.joshsoftware.com . Due to the diverse nature of building smart solutions for townships, Josh has incorporated another company called SimplySmart Solutions that builds and implements these solutions. As the name suggests SimplySmart Technologies relies on simple solutions for making things smarter. Details are available at www.simplysmart.tech . Who else runs on MongoDB? Find out: Who else uses MongoDB?
Splitit & MongoDB Atlas: Racing to Capture a Global Opportunity
Splitit is a global payment solution that allows businesses to offer installment plans for their customers. Unlike with other buy now, pay later (BNPL) solutions, Splitit shoppers can split their online purchases into monthly installments by using their existing credit, without the need for registration, application, or approval. “We have a very different proposition than others in this space,” says Splitit’s CTO, Ran Landau. “We’re not a financing company. We utilize the customer’s existing credit card arrangement, which allows us to accommodate smaller average deal values and a broader range of installment schedules.” Splitit works with online retailers across all market sectors and diverse price points, and recently raised $71.5 million in investment to fund global expansion. Following its IPO in January 2019, the business had seen strong growth as more consumers moved from brick and mortar to ecommerce. Then COVID-19 hit, and online shopping boomed. Landau recognized that the company needed to quickly scale its infrastructure in order to capture this large opportunity. The Need for Speed Landau joined Splitit in May 2019 and worked to modernize the company’s infrastructure. At the time, the team was using a traditional relational database. “As tech leaders, we need to make the right decision,” he says. “When I came to Splitit, I knew I needed a powerful NoSQL server so that my developers could develop faster and so that we could scale – both things that our relational databases were failing to deliver.” In the interest of getting up and running quickly, Ran’s team thought that they could move faster using a cloud-provider database that mimicked MongoDB functionality. He had used MongoDB before and saw that this solution offered the same drivers he was familiar with and claimed compatibility with MongoDB 3.6. Initially, the new solution seemed fine. But as the team started to migrate more data into the database, however, Landau noticed a few missing features. Scripts for moving documents from one collection to another were failing, and overall performance was deteriorating. The application became slow and unresponsive even though the load on the database was normal. “We were having issues with small things, like renaming collections. I couldn’t search or navigate through documents easily,” recalls Landau. Lost Data: A Breaking Point The application was unable to communicate with the database for 20 minutes, and when the database finally came back online, I could see it wasn't right. Landau contacted support, but the experience was not very helpful. “We were not pleased with the response from the database vendor,” he explains. “They insisted that the issue was our side. It wasn’t so collaborative.” Fortunately, he had taken a snapshot of the data so Splitit was able to revert to that version. But the incident was troubling. Other teams also had been complaining about how difficult it was to debug problems and connect to the database successfully. Landau knew he needed to find a better solution as soon as possible. MongoDB Atlas: A Reliable, Scalable Solution Landau believed that MongoDB was still the right choice for Splitit, and investigated whether the company offered a cloud solution. He discovered MongoDB Atlas and decided to give it a try. “The migration to MongoDB Atlas was so simple. I exported whatever data I had, then imported it into the new cluster. I changed the connection strings and set up VPC peering in all of my environments,” says Landau. “It was incredibly easy.” Not only was MongoDB Atlas built on actual MongoDB database software, but it was also secure, easy to use, and offered valuable features such as Performance Advisor . “It can tell you which indexes need to be built to increase speed. It’s such a powerful tool — you don’t need to think; it analyzes everything for you,” explains Landau. Another great feature was auto-scaling. “My biggest concern as I scale is that things keep working. I don’t have to stop, evaluate, and maintain the components in my system,” says Landau. “If we go back to doing database operations, we can’t build new features to grow the business.” Auto-archival Made Easy with Online Archive As a business in the financial services industry, Splitit needs to comply with various regulations, including PCI DSS . A key requirement is logging every transaction and storing it for auditing purposes. For Splitit, that adds up to millions of logs per day. Landau knew that storing this data in the operational database was not a cost-effective, long-term solution, so he initially used an AWS Lambda function to move batches of logs older than 30 days from one collection to another periodically. A few months ago, he discovered Online Archive , a new feature released at MongoDB.live in June 2020. With it, Landau was able to define a simple rule for archiving data from a cluster into a more cost-effective storage layer and let Atlas automatically handle the data movement. “The gem of our transition to Atlas was finding Online Archive,” says Landau. “There’s no scripting involved and I don’t have to worry about my aging data. I can store years of logs and know that it’s always available if I need it.” Online Archive gives me the flexibility to store all of my data without incurring high costs, and feel safe that I won't lose it. It's the perfect solution. Ran Landau, CTO, Splitit With federated queries, the team can also easily analyze the data stored in both the cluster and the Online Archive for a variety of use cases. Ready for Hypergrowth and Beyond Looking back, Landau admits that he learned his lesson. In trying to move quickly, he selected a solution that appeared to work like MongoDB, but ultimately paid the price in reliability, features, and scalability. You wouldn't buy a fake shirt. You wouldn't buy fake shoes. Why buy a fake database? MongoDB Atlas is the real thing. Ran Landau, CTO, Splitit Landau is confident that his investment in MongoDB puts in place a core building block for the business’ continued success. With a fully managed solution, his team can focus on building features that differentiate Splitit from competitors to capture more of the market. “We saw our growth triple in March due to COVID-19, but the sector as a whole is expanding,” he says. “Our technology is patent protected. Everything we build moving forward will be on MongoDB. As a company that’s scaling rapidly, the most important thing is not having to worry about my scaling. MongoDB Atlas takes care of everything.”