Slides and videos are now up from the largest ever MongoNYC, the one day conference in New York City dedicated to MongoDB. Based on feedback from attendees, here are the Top 5 Videos from MongoNYC, which range from a series of use cases, to best practices for using MongoDB in production.
- Growing Up MongoDB By Kiril Salvino, CTO and Founder, Gamechanger
- The right and wrong ways to implement MongoDB, Richard Kreuter, Consulting Manager, 10gen
- Managing a Maturing MongoDB Ecosystem, Charity Majors, Systems Engineer, Parse
- Real time integration between MongoDB and SQL Databases, Eugene Dvorkin, WebMD
- How to Keep Your Data Safe in MongoDB, Eliot Horowitz, CTO and Co-founder, 10gen
The Big Data Hoax That Wasn't
Welcome to the Age of Big Data. Or perhaps it’s the Age of Big Data Agnosticism. In a Newtonian twist, what started as a wave of hype for data’s transformational potential on organizations everywhere has turned into an equal and opposite backlash of big data naysaying. It is an understandable reaction to the great over-selling of big data as a kind of enterprise cure-all. Of course, in some companies, big data pilots have produced nothing but big piles of unfulfilled expectations. But the problem likely is not big data. Big data remains potentially the most powerful engine for business transformation to gain currency in the 21st century. The problem is that so much of what is sold as big data isn’t. It’s typically just lots of data. “Big data, that’s just data mining with a fancy new name.” How often have you heard that? It’s flatly false. The size or volume of the data does not matter in genuine big data analytics. Instead, savvy organizations already understand that big data is really about working with a mix of data types - structured and unstructured, from inside the organization and outside. It is CRM forms, but it also is Tweets, Facebook posts, TripAdvisor rants, Gmails, Outlook entries, even voicemail. In most organizations this does not add up to petabytes of data, as I’ve written before . Terabytes is the usual quantity even though that seems small by many measures. The complexity arises in the diversity of data. And that raises a problem. Not many databases have the flexibility to handle that many forms of data. And fewer databases have the agility to permit modifications on the fly - “Shouldn’t we add SMS data in here, too?” The right answer is, done. A database that cannot - with little fuss -- add a new row is too rigid for use in true big data analysis because the exciting - maybe maddening? - bit about big data today is that always there is new input that may enhance the overall result. Then there are the other questions: why are you collecting big data in the first place? What do you want from your analysis of it and this question is key because without targeted analytics, big data is just hoarding. As an insightful story in The Guardian recently posited, “Companies need to focus on big answers not big data. Instead of focusing upon the concept of big data, organizations should concentrate on the intelligence data can offer.” In other words, it’s not about the data: it’s about what intelligence can be drawn from it. The Guardian author calls himself a “big data sceptic” but, really, he isn’t. He just shares the frustration over the many mislabeled big data projects - that never were about big data - and also about the data hoarding that some companies do when they say they are committing to big data. Such projects rarely end well. Real big data - unstructured, from multiple sources - coupled with real analytics is a game changer that gives forward-thinking organizations insight that before was merely guesswork. One Texas city ran analyses to determine exactly what happened in parts of the city that experienced higher than anticipated growth and a resulting increase in value. This was true big data. In the mix were police reports, zoning violations, construction permits, parking tickets, you name it. If the data existed, it was fed into the analysis and the city began to see what it did - and didn’t do - to spur growth. Where could it get out of the way? Where could it proactively spur growth? It was real big data in action. And it’s why big data remains a big deal, despite the hype.
Splitit & MongoDB Atlas: Racing to Capture a Global Opportunity
Splitit is a global payment solution that allows businesses to offer installment plans for their customers. Unlike with other buy now, pay later (BNPL) solutions, Splitit shoppers can split their online purchases into monthly installments by using their existing credit, without the need for registration, application, or approval. “We have a very different proposition than others in this space,” says Splitit’s CTO, Ran Landau. “We’re not a financing company. We utilize the customer’s existing credit card arrangement, which allows us to accommodate smaller average deal values and a broader range of installment schedules.” Splitit works with online retailers across all market sectors and diverse price points, and recently raised $71.5 million in investment to fund global expansion. Following its IPO in January 2019, the business had seen strong growth as more consumers moved from brick and mortar to ecommerce. Then COVID-19 hit, and online shopping boomed. Landau recognized that the company needed to quickly scale its infrastructure in order to capture this large opportunity. The Need for Speed Landau joined Splitit in May 2019 and worked to modernize the company’s infrastructure. At the time, the team was using a traditional relational database. “As tech leaders, we need to make the right decision,” he says. “When I came to Splitit, I knew I needed a powerful NoSQL server so that my developers could develop faster and so that we could scale – both things that our relational databases were failing to deliver.” In the interest of getting up and running quickly, Ran’s team thought that they could move faster using a cloud-provider database that mimicked MongoDB functionality. He had used MongoDB before and saw that this solution offered the same drivers he was familiar with and claimed compatibility with MongoDB 3.6. Initially, the new solution seemed fine. But as the team started to migrate more data into the database, however, Landau noticed a few missing features. Scripts for moving documents from one collection to another were failing, and overall performance was deteriorating. The application became slow and unresponsive even though the load on the database was normal. “We were having issues with small things, like renaming collections. I couldn’t search or navigate through documents easily,” recalls Landau. “Then one day, my data was gone.” Lost Data: A Breaking Point The application was unable to communicate with the database for 20 minutes, and when the database finally came back online, a large collection had vanished. Landau contacted support, but the experience was not very helpful. “We were not pleased with the response from the database vendor,” he explains. “They insisted that we deleted the data ourselves. It wasn’t so collaborative.” Fortunately, he had taken a snapshot of the data so Splitit was able to recover the 60 million missing documents. But the incident was troubling. Other teams also had been complaining about how difficult it was to debug problems and connect to the database successfully. Landau knew he needed to find a better solution as soon as possible. MongoDB Atlas: A Reliable, Scalable Solution Landau believed that MongoDB was still the right choice for Splitit, and investigated whether the company offered a cloud solution. He discovered MongoDB Atlas and decided to give it a try. “The migration to MongoDB Atlas was so simple. I exported whatever data I had, then imported it into the new cluster. I changed the connection strings and set up VPC peering in all of my environments,” says Landau. “It was incredibly easy.” Not only was MongoDB Atlas built on actual MongoDB database software, but it was also secure, easy to use, and offered valuable features such as Performance Advisor . “It can tell you which indexes need to be built to increase speed. It’s such a powerful tool — you don’t need to think; it analyzes everything for you,” explains Landau. Another great feature was auto-scaling. “My biggest concern as I scale is that things keep working. I don’t have to stop, evaluate, and maintain the components in my system,” says Landau. “If we go back to doing database operations, we can’t build new features to grow the business.” Auto-archival Made Easy with Online Archive As a business in the financial services industry, Splitit needs to comply with various regulations, including PCI DSS . A key requirement is logging every transaction and storing it for auditing purposes. For Splitit, that adds up to millions of logs per day. Landau knew that storing this data in the operational database was not a cost-effective, long-term solution, so he initially used an AWS Lambda function to move batches of logs older than 30 days from one collection to another periodically. A few months ago, he discovered Online Archive , a new feature released at MongoDB.live in June 2020. With it, Landau was able to define a simple rule for archiving data from a cluster into a more cost-effective storage layer and let Atlas automatically handle the data movement. “The gem of our transition to Atlas was finding Online Archive,” says Landau. “There’s no scripting involved and I don’t have to worry about my aging data. I can store years of logs and know that it’s always available if I need it.” Online Archive gives me the flexibility to store all of my data without incurring high costs, and feel safe that I won't lose it. It's the perfect solution. Ran Landau, CTO, Splitit With federated queries, the team can also easily analyze the data stored in both the cluster and the Online Archive for a variety of use cases. Ready for Hypergrowth and Beyond Looking back, Landau admits that he learned his lesson. In trying to move quickly, he selected a solution that appeared to work like MongoDB, but ultimately paid the price in reliability, features, and scalability. You wouldn't buy a fake shirt. You wouldn't buy fake shoes. Why buy a fake database? MongoDB Atlas is the real thing. Ran Landau, CTO, Splitit Landau is confident that his investment in MongoDB puts in place a core building block for the business’ continued success. With a fully managed solution, his team can focus on building features that differentiate Splitit from competitors to capture more of the market. “We saw our growth triple in March due to COVID-19, but the sector as a whole is expanding,” he says. “Our technology is patent protected. Everything we build moving forward will be on MongoDB. As a company that’s scaling rapidly, the most important thing is not having to worry about my scaling. MongoDB Atlas takes care of everything.”