Narrative Science Quill™ and MongoDB: A Match Made In Heaven

MongoDB
November 10, 2014
#Releases

This is a guest post by Craig Booth and Katy De Leon of Narrative Science

Big Problems with Big Data

Billions of dollars have been invested in Big Data that have produced innumerable business intelligence and visualization tools. No amount of data is too large. No combination of algorithms is too many. No visualization is too complex. But no spreadsheet or dashboard can tell you what you need to know to improve your business. This you must figure out for yourself. Or rely on someone to figure it out for you.

At Narrative Science, we approach data in a different way. We find the insight hidden in data and communicate it in a form that makes natural sense to everyone: a written narrative. Whether it’s an earnings report, a system performance review or a product description, narratives leave you with nothing left to decipher. All you have to do is read.

Since Big Data is vital to what we do, we require a data storage solution that is reliable, high performing, scalable, flexible and expressive regarding the structure and hierarchy necessary for generating plain English language. MongoDB meets all of these criteria and has been the backend of our entire infrastructure since 2010. Over the years, our MongoDB replica sets have remained available even during region-wide outages in the Amazon cloud computing centers. We also leverage our MongoDB deploys as a job queuing system, and as a store for temporary data for our web services (using TTL indexes).

MongoDB also offers us schematic flexibility — a critical feature since we are constantly adding new data from a diverse range of industries and clients. Changes can be made iteratively and instantaneously, a huge advantage for us over a traditional SQL database. As our data requirements and client demands have grown ever more complex, MongoDB has grown with them; providing access to the data structures necessary to generate natural language directly from data.

Quill Automatically Explains MongoDB Usage

Quill™ is our automated narrative generation platform, powered by Artificial Intelligence. Quill integrates structured data from disparate sources, understands what is important to the end user and then generates perfectly written narratives that convey meaningful information to any intended audience. In essence, we do what data visualizations cannot. Although charts and dashboards look appealing, they still require people to explain them. Quill, on the other hand, adds value to data by identifying relevant data points and relaying them through professional, conversational language that people can immediately comprehend, act on and trust.

In order to diagnose potential performance and scaling issues before they affect us or our clients, we decided to put Quill on the task of monitoring our MongoDB usage. Here is a snapshot from a recent Quill-generated report explaining the weekly performance of one of our environments:

Data Use

This report, generated on July 25, 2014, compares today’s MongoDB usage to that from one week ago. Between last week and today:

The database “prod_ns4” has grown the most in the past week, increasing by 4.07 Gb. The collection inside of “prod_ns4” that drove most of this growth was “metadata”, which now contains 531,126 documents.
The total data size increased from 354.79 Gb to 365.31 Gb.
The total number of collections remained constant at 8407.
The number of documents has increased from 119.4 million to 121.1 million.
The following databases are the largest of those in your MongoDB instance: “prod_pub_service” (with 196 collections, containing a total of 18.4 million objects in 128 Gb) and “prod_harpoon” (with 25 collections, containing a total of 9.9 million objects in 44.54 Gb).
The following databases were dropped: “prod_assignment_service” and “stg_sla_monitor”.
No new databases were created in the past week.

Slow Queries

The three collections that accounted for the largest number of slow queries this week were ‘stg_finance.qm’, 'stg_cft.tweets’, and 'stg_statprovider.latest’ with 8675, 1405 and 601 slow queries respectively.

72% of the total number of slow queries were on the collection 'stg_finance.qmc. Consider adding indexes. The slowest queries were:

Wed Jul 16 05:01:48 [conn61548] query stg_finance.qqm query: { $query: { ticker: “LTG”, exchange: “NYSE” }, $orderby: { date: -1 } } ntoreturn:1 ntoskip:0 nscanned:643 scanAndOrder:1 keyUpdates:0 numYields: 20 locks(micros) r:9929394 nreturned:1 reslen:110 7102ms
Wed Jul 16 04:53:55 [conn61548] query stg_finance.qm query: { $query: { ticker: “ATR”, exchange: “NYSE” }, $orderby: { date: -1 } } ntoreturn:1 ntoskip:0 nscanned:643 scanAndOrder:1 keyUpdates:0 numYields: 18 locks(micros) r:13086372 nreturned:1 reslen:110 7050ms

Better Transparency and Decision-Making

This machine-generated explanation distills thousands of numbers, about over 5TB of data, into an easily digestible, relevant, and informative story that is sent to our engineering team every day and every week. It allows them to zero in on only those issues that are important and ignore the information that’s irrelevant to performance issues. And, unlike what you get from other monitoring tools, even a non-techy type can immediately understand the report. It gives our team what we need, when we need it - actionable insights specific to the performance of our MongoDB deployment.

These reports have already allowed us to make performance tweaks to our deployments. For example, when inefficient MongoDB queries make it into production code, Quill immediately flags them so we can place indexes in the relevant places. It also makes our lives easier by providing transparency into which projects are hogging data space so we can make the relevant collections smaller, either by making different choices about what data we store, or adding TTL indexes that allow irrelevant data to expire.

MongoDB is a powerful tool for dealing with the deeply nested documents that are needed for Natural Language Generation (NLG). It is a tool that is essential to Quill’s performance. At the same time, natural language reporting from Quill provides us with a powerful way to continually understand and improve our MongoDB usage.

Want to learn how Quill can immediately start adding value to your business? We’ve written a white paper that explains how Quill works and outlines examples of how companies are using it today. You can also visit our website, or contact us directly to see a demonstration.

← Previous

Four Ways to Optimize Your Cluster With Tag-Aware Sharding

By Francesca Krihely, Community Manager and Asya Kamsky, Principal Developer Advocate MongoDB supports partitioning data using either a user-defined shard key index (composed of one or more fields present in all documents) or based on a hashed index on a single field (which also much be present in all documents). A lesser known feature is tag-aware sharding , which allows you to create (and adjust) policies that affect where documents are stored based on user-defined tags and tag ranges. Tag-aware sharding is a very versatile feature: it can be used to maximize the performance of your cluster, optimize storage, and implement data center-awareness in your application. In this post, we’ll go through a few use cases for tag aware sharding. Optimizing Physical Resources In some cases, an application needs to store data that is not frequently accessed. Financial institutions are often required to save transaction details or claims for a number of years to stay in compliance with government regulation. It can be very expensive to keep several years worth of data on high-performance hardware, so a useful approach would be to use periodic batch jobs to move data to a more cost-effective archive storage–an operational headache for most applications. There are a number of failure points when moving data from one machine to the next and there is a lot of tedious application logic needed in order to execute these operations. Using MongoDB’s Tag-Aware sharding feature, you could assign tags to different storage tiers, associate a unique shard key range to a tag, and documents will move from one shard to the next during normal balancing operations. To ensure you’re moving the right number of document from one machine to the next over time, you could write a script to update shard tags once a month. This method is great because it keeps all the data in one database, so your application code doesn’t need to change as data moves between storage tiers. You can read an in-depth overview of this implementation in our blog post on tiered storage . Location-Based Separation of Data If you have a large, global user base, it is often preferable to store specific user information in localized data centers. In this case, you can use shard tags to keep documents to specified data center(s). There is a great tutorial on this in the (MongoDB Manual)[http://docs.mongodb.org/manual/data-center-awareness/]. While this is a classic use case for tag-aware sharding, there are a few things to look out for when implementing this new approach yourself. Remember that Shard Keys are immutable: you cannot change the value of your shard key, so if a user happens to change their location, they will in turn change their shard key, and you will need to delete and re-insert that document to change that information. See more about Shard Keys in our blog post here . You should include the shard key with every operation to avoid scattering the query to all shards. This may seem counter-intuitive if your shard key is “region, customerId” but it’s necessary to make sure that only shards holding a particular value of “region” field are sent the query. If you’re interested in learning more about this example, take a look at our new white paper on multi datacenter deployments which discusses sharding along with other configurations for distributing databases across regions. Balancing unsharded collections across a sharded cluster Sharding can help with many dimensions of scalability, including read and write scalability. With tag aware sharding, you can take this one step further, by balancing collections across shards to distribute read and write load. This would happen in a case where you have a large number of medium sized collections on a single shard – not typically big enough to warrant sharding– that make your workload unbalanced. Instead of splitting all of these medium-sized collections across all shards, you can distribute them “round-robin” style–one or several collections living on each shard. Some examples of how to implement that are here In the wild, we have seen this implemented for a “platform as a service” (i.e. an application development platform). If you have many tenants–or individual apps– and none of them are large enough to need sharding, there still may be too many to live on a single replica set. Having each tenant’s application on an individual shard would help distribute the load by splitting individual collections on to specific shards or a subset of shards. You would not have to partition any of the collections (which you would do if you sharded the collection) but they would live on individual shards, enabling you to balance your read and write load. Targeting data to individual servers Balancing collections can also be used to optimize physical resources. In some cases, your collections will have heavy indexes that are taking up a lot of memory. Rather than spreading them out on different machines, you can target that collection to a shard with more RAM, allowing you to separate it from other collections to reduce contention for limited physical resources. There are a number of ways to implement tag-aware sharding into your infrastructure to increase read and write efficiency, get better use of your hardware and improve overall performance in your application. If you want to learn more: See how Bugsnag implemented tag aware sharding to optimize their read and write load on specific collections Read more about the Tiered Storage model Learn more about multi-data center deployments in our free whitepaper Need help with your sharding configuration? MongoDB’s Technical Support Engineers can help you plan and architect your sharding setup and offer you guidance to ensure you’re on the right track. Learn more about MongoDB’s 24x365 Production Support , which will offer you proactive advice on your MongoDB deployment.

November 5, 2014

Next →

Congratulations to the 2023 APAC Innovation Award Winners

I’m thrilled to announce the nine winners of the 2023 MongoDB APAC Innovation Awards . The MongoDB Innovation Awards honor projects and people who dream big. They celebrate the groundbreaking use of data to build compelling applications and the creativity of professionals expanding the limits of technology with MongoDB. This year, we have broken the awards down regionally to celebrate organizations in APAC, from startups to industry-leading enterprises, across a wide variety of industries, who are delivering big results. We are delighted to announce the winners below: 2023 MongoDB APAC Innovation Award Winners: Positive Impact Open Government Products Open Government Products (OGP) is an in-house team of engineers, designers, and product managers, who is a part of the Singapore Government, and is responsible for building technologies for the public good. OGP used MongoDB’s developer data platform, MongoDB Atlas to create its digital form builder, FormSG. Used by the Singapore government and public healthcare institutions, FormSG securely collects data from residents and businesses and helps public officers to create digital government forms in minutes. It eliminates the use of paper forms and the manual process of transcribing physical documents, which had raised concerns around data privacy and protection. During the pandemic, FormSG enabled public officers to collect more than 100,000 daily temperature declarations nationwide. Today, FormSG has served more than 120,000 public officers from 155 agencies and it has created more than 500,000 digital forms to help the government collect data on travel and health declarations by visitors to the country, applications for COVID-19 swab tests, and applications for financial assistance. Organization Transformation Bendigo and Adelaide Bank Bendigo and Adelaide Bank is one of Australia’s largest banks, with around 7,000 employees helping more than 2.2 million customers achieve their financial goals. The bank has been on a multi-year journey of transformation using MongoDB's developer data platform to improve efficiency and deliver a better customer experience as they fulfill their vision to become Australia’s bank of choice. Recently, the cloud team launched Ready-Set-MongoDB (or RSM). This event-driven framework allows developers to streamline the consumption of internal or external APIs, and applies data transformations and storage automatically within a MongoDB collection of their choice. Using MongoDB Atlas Search, the bank also enabled developers to gain insights across its multi-cloud deployments, identifying cost savings, and providing inventory information to account owners and technical stakeholders. Within the first 18 months of launching these programmes, the automation had saved the organization more than 1,100 developers days. It also helped reduce human involvement, removed stale data, and allowed engineers to focus on the things that matter. The development of Ready-Set-MongoDB is ongoing and improving, as new Bendigo multi-cloud challenges arise and new MongoDB products are released. The application is a perfect representation of how Bendigo's Technology Department is using modern technology, rapid development, and innovation-led problem solving to drive organizational transformation. Heroes in Health Redcliffe Lifetech Private Limited Over the last few years, Redcliffe Labs has become India's fastest growing technology-driven diagnostics service provider. Redcliffe Labs is on a mission to serve 500 Million Indians by 2030 with fusion of technology and world- class laboratories. The company already serves thousands of people daily, with more than 73 labs and close to 1500 walk-in centers across 180 cities. Redcliffe Labs has relied on MongoDB Atlas’ flexible document model to power its innovative Smart Health Report, a patient resource that provides a number of indicators and trackers to gauge holistic health. The MongoDB developer data platform's best in class security, compliance, and privacy controls allows Redcliffe's team to confidently handle even the most sensitive patient data. MongoDB Atlas takes care of many of the traditional database management challenges, which means that developers can spend their time building diagnostics for patients, rather than managing databases. Redcliffe Labs is focusing on incorporating next-generation technologies in the diagnostics space with an AI platform that will make Interactive Diagnostics reports, Advanced Health Profiling and more detailed Diagnostics and Health Alerts. Industry Disruptor Cathay Pacific Cathay Pacific , Hong Kong’s home carrier operating in more than 60 destinations worldwide, has been on an impressive journey to become one of the very first airlines to create a truly paperless flight deck. Until recently, a flight from Hong Kong to New York would require a crew to review more than 150 pages of finely printed text and charts before their flight and make ongoing updates throughout the trip. In 2019, Cathay Pacific conducted the first zero paper flight, removing 50kg of manuals, charts, maps, and flight briefing paperwork. They achieved this enormous feat with the help of one seamless and highly customized iPad application: Flight Folder. Built on MongoDB Atlas, Flight Folder is designed to improve the pilot briefing experience. MongoDB helped consolidate dozens of different information sources into one place, and made it possible for flight crews to easily share their experiences with others. It also included a digital refueling feature that helps crews become much more efficient with fueling strategies – saving significant flight time and costs. The use of MongoDB Device Sync enables seamless syncing and no data loss even when the app goes on- and offline mid-flight. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. In addition to the greatly improved flight crew experience, flight times have been reduced, and digital refueling saves eight minutes of ground time on average. All these efficiencies have helped the company avoid the release of 15,000 tons of carbon. From Batch to Real-Time Adani Digital Labs Adani Digital Labs is the India-based digital innovation arm of the larger Adani group. The lab’s team's mission is to create one single platform – a SuperApp called AdaniOne – to empower a billion stories in India. To address several use cases and the huge scale that will be required by the superapp, the Adani Digital team selected MongoDB Atlas as its the main transactional database that will further enhance the application. A key component of the app is how it can bring together disparate data in order to provide a single view of activity across the application. In the first process, developers had taken out the data in batches and sent it to their database However, this was too slow and unpredictable as far as business requirements are concerned. Also, the consolidated view of customer history, orders, inventory, and supply chain network updates was likely to impact their customer's ability to generate revenue. Therefore, in order to find a better solution, Adani Digital Labs built a more modern architecture in line with MongoDB. Using MongoDB's Change Streams and the data platform's native Kafka connector, they created an event-based architecture that pushes the data out in real-time for analysis. Adani Digital Labs is still in the early phases of the SuperApp's rollout and collaborating with MongoDB as its developer data platform continues to help the firm to grow and deliver insights in real time. Industry 4.0 Dongwha Founded in 1948, the Dongwha Group has evolved from a singular focus on the wood and timber industry into a global leader across a number of sectors including building materials, chemicals and media. As part of its wider digital transformation strategy, Dongwha required smarter factories that would improve and optimize their production efficiency. Dongwha built an innovative Smart Factory Software platform that collects and analyzes data to enhance quality and production management capabilities. Originally, the platform was built with the community version of MongoDB. However, in order to scale and adapt, the team recently migrated to MongoDB Atlas in the cloud. This enabled them to store large volumes in the fastest and most secure way, optimize their solution for time series data, and make it easy to run machine learning across their data. Dongwha completed the migration seamlessly, without any disruption or downtime to their factories, and it has now been launched across five different sites. Over the last year, the application has significantly increased its availability and reliability while performance has improved by as much as 6x . As they look to the future, Dongwha plans to roll out the software to more of its international factories. Digital Native myBillBook India is home to more than 60 million small and medium-sized businesses (SMBs) but only a small portion of those SMBs are taking advantage of digitization and many still operate using pen and paper. In addition, many businesses in India still struggle with fluctuations in internet services, outages, and latency. FloBiz is on a mission to change that with myBillBook , a one-stop solution that helps SMBs create professional invoices, manage stock, collect payments, automate reminders through smart banking, engage with their customers, manage staff attendance and payroll and generate more than 25 business reports for accounting and decision making. The app is also mobile-first, so businesses can access them from their mobile devices and allows users to manage billing and inventory in both online and offline environments. The myBillbook app is powered by MongoDB Atlas, providing the flexible and scalable foundation for the business to do everything from building new features to performing complex analytical queries. In addition, MongoDB Realm, the mobile database within the data platform, supports offline usage and syncing to ensure there is never data loss or functionality for users due to poor internet connection. Because of its success in supporting customers with business critical operations, more than 6.5 million business owners in India are now using myBillbook for their billing, accounting, collection and business growth. Customer Focused KASIKORN Business-Technology Group Established in 1945, Kasikornbank (KBank) is one of the largest and oldest banks in Thailand. Their mission is to strive towards service excellence and empower every customer’s life and business. One of KBank’s subsidiaries, KASIKORN Business-Technology Group (KBTG) , developed a mobile banking application – MAKE by KBank. MongoDB Atlas’ flexibility and ease of development enabled MAKE’s development team to choose the best type of database for its tasks, to automate data tiering with Atlas Online Archive, and to reduce hours spent on operational maintenance. With more time to focus on delivering new innovations to customers, they created unique features like Cloud Pocket which can allocate funds into unlimited customizable pockets for separate usage. They also built Pop Pay, a feature that allows users to easily search for nearby friends and transfer money by clicking their profile picture as well as “Expense Summary" a spending analysis services that helps inform and manage users’ financial habits. As of January 2023, MAKE has acquired more than 1 million users, and increased the number of transactions in MAKE from 900,000 to more than 7.5 million in a span of one year. Massive Scale China Mobile China Mobile provides mobile voice and multimedia services via its nationwide mobile telecommunications network across mainland China and Hong Kong. It is the world's largest mobile network operator by total number of subscribers. The telecommunications leader is using MongoDB to support one of its largest and most critical push services, which sends out billing details to more than 1 billion users every month. Prior to MongoDB, the tech team relied on Oracle, but as the user numbers increased, performance degraded. Despite large investments, it was still taking too long to do basic requests like finalize and deliver bills to users. In 2019, after comprehensive testing, China Mobile migrated to MongoDB. By taking advantage of MongoDB's native sharding, they were able to improve performance by 80% and go from 50 Oracle machines, to just 12 machines for the same workload. The service now handles all current requirements and is set up to scale with future growth. With the support of MongoDB, China Mobile is growing steadily,with more than 168 million monthly users and has one of the highest customer satisfaction scores in the China Mobile group.

March 30, 2023