We recently introduced streamlined data sources in Atlas Charts, which eliminates the manual steps involved with adding data sources into Charts. With MongoDB Atlas project data automatically available in Charts, your visualization workflow can become quicker and simpler than ever.
For those unfamiliar with these data sources, here’s a quick summary:
A serverless instance is an Atlas deployment model that lets you seamlessly scale usage based on workload demand and ensures you are only charged for resources you need.
Online Archive enables automated data tiering of Atlas data, helping you scale your storage and optimize costs while keeping data accessible.
These data sources serve two distinct use cases, based on your needs. So, whether you are trying to eliminate upfront resource provisioning using a serverless instance or creating archives of your high-volume workloads, such as time-series or log data to reduce costs with Online Archive, Charts makes these sources natively available for visualization with zero ETL, just as it always has with your other Atlas clusters.
To learn how easy it is to visualize these new data sources, let’s create a serverless database called “ServerlessInstance0” and separately activate Online Archive on a database called “Cluster0” that will run daily in Atlas (Figure 1).
When setting up an Online Archive, Atlas creates two instances of your data (Figure 2). One instance includes only your archived data. The second instance contains your archive data and your live cluster data. This setup gives you additional flexibility to query data as your use case demands.
Moving on to the Data Sources page in Charts (Figure 3), all of the data sources are shown, including serverless instances and Atlas cluster data archived in Online Archive, neatly categorized based on the instance type and ready for use in charts and dashboards. (Note that project owners maintain full control of these data sources.) For more details about connecting and disconnecting data sources, review our documentation.
With these additions, Charts now supports all the cluster configurations you can create in Atlas, and we are excited to see how you achieve your visualization goals using these new data sources.
New to Atlas Charts? Get started today by logging into or signing up for MongoDB Atlas, deploying or selecting a cluster, and activating Charts for free.
MongoDB & IIoT: Turning Data into Business Intelligence
Manufacturing companies leverage business intelligence (BI) to sift through and analyze manufacturing and supply chain data in order to become more efficient and productive organizations. Often, the real hurdle with analytics is ensuring reliable access to relevant data sets. This article describes how to prepare data to yield strategic and operational insights through a combination of data tiering, federation, querying, and visualization. Consider the scenario of a car manufacturer looking to implement a predictive maintenance program to reduce maintenance costs for its car assembly machines. Establishing an optimal data storage infrastructure is critical to allow them to find correlations between live IoT sensor data and historical maintenance records, thereby gaining insights into maintenance trends and correlating sensor data. As shown in Figure 1, such a challenge falls under step 3 of our IIoT end-to-end data integration framework: Compute. Figure 1: Step 3 in end-to-end data integration framework for IIoT. Read the first , second , and third articles in this series on end-to-end data integration in the context of IIoT. Figure 2: Architecture overview of data visualization and analytics enabled by MongoDB. The proposed methodology leverages the different data tiering capabilities of MongoDB covering the full data lifecycle to create a single API access for BI/analytics. Figure 2 summarizes the different MongoDB features and third-party integrations available to take advantage of the volumes of data generated over time for data-driven business insights. The challenge of data tiering The car manufacturer in our example would most likely need to differentiate between the different types of data needed for its predictive maintenance model. Here we make a distinction between operational and analytical workloads. Operational workload: Refers to latency-sensitive data that affects functioning of equipment or powers critical applications/processes. Analytical workload: Refers to life and historical data that does not power mission-critical applications but is readily stored and queried for the purpose of reporting, analytics, or training of AI/ML models. Figure 3 provides a basic illustration of how MongoDB handles workload isolation leveraging MongoDB replica sets to support real-time BI and analytical workloads without additional ETL jobs. Figure 3: Illustration of workload isolation in MongoDB. More advanced architecture patterns for workload isolation or data tiering can be achieved through sharding. Although these approaches are suitable for many scenarios, they are still more like hot/warm data because storage and compute are still tightly coupled. For maximum cost efficiency at the expense of latency, we must consider newer cloud storage options, such as Amazon S3 or other Blob stores, which decouple storage and compute and are perfectly suited to store so-called cold data. The challenge, however, is how to extract the data from hot stores (such as MongoDB), bring it into the cold storage (such as S3) while maintaining the ability to query the data through a single API. MongoDB provides several options to facilitate fully automated data tiering, including: Online Archive Atlas Data Lake Online Archive: Rule-based data archiving Online Archive in MongoDB Atlas provides an automated rule-based mechanism for moving data out of live/hot clusters to more cost-effective/cold storage (for example, Amazon S3 buckets). This feature removes the burden of building and maintaining potentially complex ETL jobs and data purging functionality while allowing users to configure data offloading within a few simple configuration steps. Online Archive moves data based on criteria specified in archival rules (as shown in Figure 4). In our example of an auto manufacturing company, sensor data is an excellent use case for this type of data tiering. Sensor data is “hot” when it's created and cools down over time with less need for real-time queries. Our car manufacturer can easily configure an archival rule dependent on the timestamp and in combination with the number of days they want to keep the data in the MongoDB cluster . Figure 4: Animation showing how Online Archive works. A broad set of MongoDB Atlas customers across industries already uses Online Archive to save storage costs while maintaining query ability across hot and cold data. With Online Archive, we were able to save an astounding 60% in data storage costs and 70% in cloud backup costs — reducing our overall database spend by 35%. Martin Löper, Cloud Solutions Architect, Nesto Software Although offloading data already provides major cost savings, there is also potential for more efficient data processing on the consumer side by optimizing the data structures and file formats toward more column-oriented analytical queries. For this purpose, MongoDB has recently released a Data Lake feature set (currently in Preview) that allows users to take advantage of new features such as columnar indexing and an optimized analytical file format. Data Lake: Columnar indexing of database snapshots Data Lake is MongoDB’s offering of a fully managed analytical storage solution that provides the economics of cloud object storage and is optimized for high-performing analytical queries. It works by reformatting data from a backup snapshot of the Atlas cluster and creating partitioned indexes (illustrated in Figure 5). Figure 5: Diagram showing how Data Lake works. Fully integrated as part of MongoDB Atlas, Atlas Data Lake is provisioned alongside Atlas clusters with no infrastructure to set up or manage, and no storage capacity to predict, making the user experience, administration, and support easy. Returning to our example of predictive maintenance model development, performing columnar indexing on the collected data will result in high gains for analytical query performance. Data Federation: Data virtualization made simple Rarely do business analysts have all the required data in the same place. Often, it’s distributed among different domains and data stores as well as in different formats, like JSON, tabular, CSV, Parquet, Avro, and others. This leads to quite a complex landscape with different API languages, which makes it hard to get easy access to data across all these sources. That's where MongoDB's Atlas Data Federation comes in. Data Federation allows bridging of these data silos by consolidating all the discussed data sources behind a single API without the need for data duplication (Figure 6). Users can group different data sources to virtual databases and collections and query the data with MQL or SQL across the various sources just like talking to a single DBMS. This approach reduces the effort, time-sink, and complexity of pipelines and ETL tools when working with data in different formats. It also allows users to seamlessly query, transform, and aggregate data from one or more data stores (i.e., Atlas cluster, Atlas Data Lake, Amazon S3 buckets, Online Archive, and HTTP endpoints) to create a single virtual database using the full power of the aggregation pipeline (Figure 7). Figure 6: Diagram showing how Data Federation works in MongoDB Atlas. Figure 7: Creating a virtual database in the MongoDB Atlas GUI. Please refer to the documentation for a more detailed description of the process of creating a Federated Database Instance in MongoDB Atlas. Data Federation endpoints are not just read-only APIs. Results of querying a federated database instance can be stored back in MongoDB clusters or as files in S3 buckets to power other real-time enterprise or end-user applications, or for performing other analytical tasks and visualizations. In the case of our car manufacturer, real-time sensor data and maintenance history can be queried together and made available to an analytical engine training ML models for remaining useful life prediction. The fastest way to start building compelling visualizations and gaining insight into the data across MongoDB clusters and file-based data sources through federated instances is through the use of Charts , which comes fully integrated in the Atlas product suite. Data visualization with Charts Charts provides a quick, simple, and yet powerful way to visualize data with multiple widgets, dynamic filters, and automatic data refresh like you know it from traditional BI tools. Atlas users can connect dashboards created in Atlas Charts with federated databases and perform correlation analytics in a no-code environment. Charts is fully integrated with the MongoDB Atlas product suite, which means that data sources in Atlas are immediately accessible from the interface, allowing users to add federated databases as a source for a variety of dashboard visualizations. From displaying device sensor data to calculated values for more sophisticated insights, Charts provides widgets and custom fields calculations to achieve effective and insightful visualizations. Figures 8 and 9 show two examples of dashboards created in Charts showing time series sensor data from a smart factory and Overall Equipment Effectiveness (OEE) along with other manufacturing performance metrics information. Through the use of these powerful visualizations, the car manufacturer can understand the effect of optimal maintenance strategies on overall factory performance. Figure 8: Sample shop floor monitoring dashboard created in Atlas Charts. Figure 9: Sample OEE dashboard in Atlas Charts To harness existing knowledge and skills around familiar and popular BI tools such as Power BI and Tableau, MongoDB has developed Atlas SQL API , which gives users the option to connect SQL-based business intelligence and analytics tools to Atlas through a variety of drivers and connectors including: Tableau Connector Power BI Connector JDBC Driver ODBC Driver These Atlas SQL connectors and drivers leverage Data Federation functionality, thereby enabling users to query data across Atlas clusters and cloud storage (such as S3 buckets) and to maintain the comfort of existing SQL-based BI tools that they are familiar with. Getting started is easy using the Atlas SQL API at no cost with the detailed tutorial and the documentation . Register for a free Atlas user account to try it out. Thank you to Karolina Ruiz Rogelj for her contributions to this post. Watch our recorded webinar to see a live demonstration of how Atlas Federated Instances are created and used as a data source for MongoDB Charts and Tableau.
Building a Culture of Growth: SVP Simon Eid on MongoDB's Massive Opportunity in APAC
Simon Eid is Senior Vice President Asia-Pacific (APAC) at MongoDB and leads the sales teams across Australia and New Zealand, India, ASEAN, and Japan. Simon's go-to-market organisation in APAC is growing rapidly and has nearly tripled in size in the past three years. They are hiring in all regions . In this article, Simon discusses MongoDB’s opportunity in APAC and how he builds a culture of growth and accountability. Simon Eid, SVP APAC, MongoDB (left) and Anoop Dhankar, RVP ANZ, MongoDB (right) MongoDB's opportunity in Asia-Pacific Out of the top 13 economies by GDP in the world , five of them are located in APAC: China, Japan, Australia, India, and South Korea. And that's to say nothing of the ASEAN countries which alone have more than 650 million inhabitants. Combine this with the worldwide database market, one of the largest markets in the software industry. IDC estimates that it will grow to $137B in 2027, and MongoDB has just reached $1B in ARR. This gives you a sense of the massive market opportunity we have globally. Regardless of industry, product, or service, almost every company is becoming a technology company, which means that every company is becoming a data company. We believe MongoDB is the Developer Data Platform that is best placed to support and accelerate that trend. We’ve already captured thousands of customers around the globe, but it’s important to keep in mind that our world is still in the early stages of shifting to the cloud and changing how applications are built and run. Compared to other software, what's special about the market we play in is that the database is not a “nice-to-have”; it’s mission-critical for organisations. As our world continues to undergo this digital transformation, we have the opportunity to transform how our customers use software and data to innovate, create, and disrupt industries. For example, look at Cathay Pacific , Hong Kong's home airline carrier operating in more than 60 destinations worldwide. The company's digital team turned to MongoDB on their journey to become one of the first airlines to create a truly paperless flight deck. Flight Folder, their application built on MongoDB, consolidates dozens of different information sources into one place. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. Their innovation is enabled by MongoDB. Building a team across regions and cultures Our team in APAC is unique because of the different markets and cultures within the region. What this means is that we go to market differently in India than we do in Australia, in Singapore than we do in South Korea, and so on. Each market is completely different, but within all of them, there is a huge opportunity. Different from many of our peers, in APAC we've established business leaders who run regionalized teams in India, ASEAN, and ANZ with all functions reporting to them. These teams essentially operate as their own business and implement local best practices into their strategy. But, it doesn’t mean they’re operating in a silo. At the leadership level, there is an immense amount of collaboration and sharing of experiences to identify what’s working and what isn’t within each region. We also have a fantastic global sales organisation that rolls out extensive training and best practices to help enable our local teams to best help our customers and grow the business. Members of our APAC team at a recent offsite in Phuket Culture The most important thing is culture. We have a very high standard around everything we do and how we interact with each other. We don’t entertain politics. You can teach someone new skills and coach them on how to be successful in a new role, but if they’re not aligned with the culture, they will not be a fit. It’s a non-negotiable for me and why the most important aspect of the hiring process is the cultural aspect. If you get the culture right, everything else starts to fall into place. What I hear at MongoDB and from the teams I've built at other companies is that this is the kind of culture they can really thrive and grow. At MongoDB, our culture is defined and shaped by six core values . One of the values that’s most important to my team is “Embrace the Power of Differences”. Within APAC, there are a variety of cultural identities and nuances that can often be difficult to navigate, whether it is cultural values, beliefs, or go-to-market strategy. It’s important that everyone who joins my team is respectful of each other’s regional culture. What we’ve done within the APAC region, and with teams across the globe, is take everyone on a journey to understand and embrace these cultural differences. Our role as leaders is to develop our teams, from the bottom all the way up, which is part of MongoDB’s BDR to CRO career development initiative. We need to develop the next wave of leaders so that they’re prepared to step up when the time comes. For APAC, this means that regardless of where someone is from, each team member has been coached and developed on the cultural nuances so that they can lead people and go to market in each of the different regions. It’s also important that each team member contributes to a culture of psychological safety. Being part of a high-growth tech company requires taking risks and making mistakes. We have a high standard and we hold each other accountable, but it never comes at the cost of creating an environment where people are afraid to fail. When someone faces setbacks, I encourage them to share those experiences so that we can collectively learn. Through mutual support, we foster a stronger team capable of delivering exceptional results. The future of MongoDB in Asia-Pacific For any organisation to be successful, I believe it’s critically important for the entire ecosystem to act as one. As I mentioned earlier, at MongoDB the whole country ecosystem is aligned around one set of goals, so it's not a case of different teams running off in different directions. The teams are willing to lean in and do what's required to help each other build a great business. I can confidently say that in APAC, we are one team. This means sales, marketing, customer success, solutions consulting, and professional services all working together to focus on three things: making customers successful, building technical champions, and driving new workloads. As we continue to grow our team and MongoDB’s footprint in the region, these are the three things that will drive our success. As I mentioned earlier, there's a huge opportunity for MongoDB in APAC. Despite hiring slowing down or stopping completely at many other organisations, we're continuing to invest heavily in the region. To give you a sense of that - we've nearly tripled the size of our APAC go-to-market team in the past three years, and we've got more open roles across the different functions and regions. If you want to be part of this journey, there are three things I want to reiterate: First, we are extremely passionate about our culture, from the field level up to the leadership level. As a team, this is the brand we bring to the market. Second, the opportunity here is massive based on the total addressable market and our current share. And third, we place critical importance on development. By joining this team, I can promise that you’ll be provided with countless opportunities to develop your career and make an impact. I’m confident in my team and the leadership we have in place who are ready to take MongoDB APAC to the next level. Join us !