MongoDB Developer

Coding with MongoDB - news for developers, tips and deep dives

Take Advantage of Low-Latency Innovation with MongoDB Atlas, Realm, and AWS Wavelength

The emergence of 5G networking signals future growth for low-latency business opportunities. Whether it’s the ever-popular world of gaming, AR/VR, AI/ML, or the more critical areas of autonomous vehicles or remote surgery, there’s never been a better opportunity for companies to leverage low latency application services and connectivity. This kind of instantaneous communication through the power of 5G is still largely in its nascent development, but customers are adapting to its benefits quickly. New end-user expectations mean back-end service providers must meet growing demand. At the same time, business customers expect to have the ability to seamlessly deploy the same cloud-based back-end services that they’re familiar with, close to their data sources or end users. With MongoDB Realm and AWS Wavelength, you can now develop applications that take advantage of the low latency and higher throughput of 5G—and you can do it with the same tools you’re familiar with. The following blog post explores the benefits of AWS Wavelength, MongoDB Atlas, and Realm, as well as how to set up and use each service in order to build better web and mobile applications and evolve user experience. We’ll also walk through a real-world use case, featuring a smart factory as the example. Introduction to MongoDB Atlas & Realm on AWS MongoDB Atlas is a global cloud database service for modern applications. Atlas is the best way to run MongoDB on AWS because, as a fully managed database-as-a-service, it offloads the burden of operations, maintenance, and security to the world’s leading MongoDB experts while running on industry-leading and reliable AWS infrastructure. MongoDB Atlas enables you to build applications that are highly available, performant at a global scale, and compliant with the most demanding security and privacy standards. When you use MongoDB Atlas on AWS, you can focus on driving innovation and business value instead of managing infrastructure. Services like Atlas Search , Realm , Atlas Data Lake and more are also offered, making MongoDB Atlas the most comprehensive data platform in the market. MongoDB Atlas seamlessly integrates with many AWS products. Click here to learn more about common integration patterns. Why use AWS Wavelength? AWS Wavelength is an AWS Infrastructure offering that is optimized for mobile edge computing applications. Wavelength Zones are AWS infrastructure deployments that embed AWS compute and storage services within communications service providers’ (CSP) data centers. AWS Wavelength allows customers to use industry-leading and familiar AWS tools while moving user data closer to them in 13 cities in the US as well as London, UK, Tokyo and Osaka, Japan, and Daejeon, South Korea. Pairing Wavelength with MongoDB’s flexible data model and responsive Realm database for mobile and edge applications, customers get a familiar platform that can run anywhere and scale to meet changing demands. Why use Realm? Realm’s integrated application development services make it easy for developers to build industry-leading apps on mobile devices and the web. Realm comes with three key features: Cross-platform mobile and edge database Cross-platform mobile and edge sync solution Time-saving application development services 1. Mobile & edge database Realm’s mobile database is an open source, developer-friendly alternative to CoreData and SQLite. With Realm’s open source database, mobile developers can build offline-first apps in a fraction of the time. Supported languages include Swift, C#, Xamarin, JavaScript, Java, ReactNative, Kotlin, and Objective-C. Realm’s Database was built with a flexible, object-oriented data model, so it’s simple to learn and mirrors the way developers already code. Because it was built for mobile, applications built on Realm are reliable, highly performant, and work across platforms. 2. Mobile and edge sync solution Realm Sync is an out-of-the-box synchronization service that keeps data up-to-date between devices, end users, and your backend systems, all in real-time. It eliminates the need to work with REST, simplifying your offline-first app architecture. Use Sync to backup user data, build collaborative features, and keep data up to date whenever devices are online—without worrying about conflict resolution or networking code. Figure 2: High-level architecture of implementing Realm in a mobile application Powered by the Realm Mobile and Edge Database on the client-side and MongoDB Atlas on the backend, Realm is optimized for offline use and capable of scaling with you. Building a first-rate app has never been easier. 3. Application development services With Realm app development services, your team can spend less time integrating backend data for your web apps, and more time building the innovative features that push your business initiatives forward. Services include: GraphQL Functions Triggers Data access controls User authentication Reference Architecture High-level design Terminology wise, we will be discussing three main tiers for data persistence: Far Cloud, Edge, and Mobile/IOT. The Far Cloud is the traditional cloud infrastructure business customers are used to. Here, the main parent AWS regions (such as US-EAST-1 in Virginia, US-WEST-2 in Oregon, etc) are used for centralized retention of all data. While these regions are well known and trusted, the issue is that not many users or IOT devices are located in close proximity to these massive data centers and internet-routed traffic is not optimized for low latency. As a result, we use AWS Wavelength regions as our Edge Zones. An Edge Zone will synchronize the relevant subset of data from the centralized Far Cloud to the Edge. Partitioning principles are used such that users’ data will be stored closer to them in one or a handful of these Edge Wavelength Zones, typically located in major metropolitan areas. The last layer of data persistence is on the mobile or IOT devices themselves. If on modern 5G infrastructure, data can be synchronized to a nearby Edge zone with low latency. For less latency-critical applications or in areas where the Parent AWS Regions are closer than the nearest Wavelength Zone, data can also go directly to the Far Cloud. Figure 3: High Level Design of modern edge-aware apps using 5G, Wavelength, and MongoDB Smart factory use case: Using Wavelength, MQTT, & Realm Sync Transitioning from the theoretical, let’s dig one level deeper into a reference architecture. One common use case for 5G and low-latency applications is a smart factory. Here, IOT devices in a factory can connect to 5G networks for both telemetry and command/control. Typically signaling over MQTT, these sensors can send messages to a nearby Wavelength Edge Zone. Once there, machine learning and analysis can occur at the edge and data can be replicated back to the Far Cloud Parent AWS Regions. This is critical as compute capabilities at the edge, while low-latency, are not always full-featured. As a result, centralizing many factories together makes sense for many applications as it relates to long term storage, analytics, and multi-region sync. Once data is in the Edge or the Far Cloud, consumers of this data (such as AR/VR headsets, mobile phones, and more) can access this with low-latency for needs such as maintenance, alerting, and fault identification. Figure 4: High-level three-tiered architecture of what we will be building through this blog post Latency-sensitive applications cannot simply write to Atlas directly. Alternatively, Realm is powerful here as it can run on mobile devices as well as on servers (such as in the Wavelength Zone) and provide low-latency local reads and writes. It will seamlessly synchronize data in real-time from its local partition to the Far Cloud, and from the Far Cloud back or to other Edge Zones. Developers do not need to write complex sync logic; instead they can focus on driving business value through writing applications that provide high performance and low latency. For highly available applications, AWS services such as Auto Scaling Groups can be used to meet the availability and scalability requirements of the individual factory. Traditionally, this would be fronted by a load-balancing service from AWS or an open-source solution like HAProxy. Carrier gateways are deployed in each Wavelength zone and the carrier or client can handle nearest Edge Zone routing. Setting up Wavelength Deploying your application into Wavelength requires the following AWS resources: A Virtual Private Cloud (VPC) in your region Carrier Gateway — a service that allows inbound/outbound traffic to/from the carrier network. Carrier IP — address that you assign to a network interface that resides in a Wavelength Zone A public subnet An EC2 instance in the public subnet An EC2 instance in the Wavelength Zone with a Carrier IP address We will be following the “Get started with AWS Wavelength” tutorial located here . At least one EC2 compute instance in a Wavelength zone will be required for the subsequent Realm section below. The high level steps to achieve that are: Enable Wavelength Zones for your AWS account Configure networking between your AWS VPC and the Wavelength zone Launch an EC2 instance in your public subnet. This will serve as a bastion host for the subsequent steps. Launch the Wavelength application Test connectivity Setting up Realm The Realm components we listed above can be broken out into three independent steps: Set up a Far Cloud MongoDB Atlas Cluster on AWS Configure the Realm Serverless Infrastructure (including enabling sync) Write a reference application utilizing Realm 1. Deploying your Far Cloud with Atlas on AWS For this first section, we will be using a very basic Atlas deployment. For demonstration purposes, even the MongoDB Atlas Free Tier (called an M0) suffices. You can leverage the AWS MongoDB Atlas Quickstart to launch the cluster , so we will not enumerate the steps in specific detail. However, the high-level instructions are: Sign up for MongoDB Atlas account at cloud.mongodb.com and then sign in Click the Create button to display the Create New Database Deployment dialog Choose a “Shared” cluster, then choose the size of M0 (free) Be sure to choose AWS as the cloud and here we will be using US-EAST-1 Deploy and wait for the cluster to complete deployment 2. Configuring Realm and Realm Sync Once the Atlas cluster has completed deploying, the next step is to create a Realm Application and enable Realm Sync. Realm has a full user interface inside of the MongoDB Cloud Platform at cloud.mongodb.com however it also has a CLI and API which allows connectivity to CI/CD pipelines and processes, including integration with GitHub. The steps we are following will be a high-level overview of a reference application located here . Since Realm configurations can be exported, the configuration can be imported into your environment from that repository. The high level steps to create this configuration are as follows: While viewing your cluster at cloud.mongodb.com, click the Realm tab at the top Click “Create a New App” and give it a name such as RealmAndWavelength Choose the target cluster for sync to be the cluster you deployed in the previous step Now we have a Realm app deployed. Next, we need to configure the app to enable sync. Sync requires credentials for each sync application. You can learn more about authentication here . Our application will use API Key Authentication.To turn that on: Click Authentication on the left On the Authentication Providers tab, find API Keys, and click Edit Turn on the provider and Save If Realm has Drafts enabled, a blue bar will appear at the top where you need to confirm your changes. Confirm and deploy the change. You can now create an API key by pressing the “Create API Key” button and giving it a name. Be sure to copy this down for our application later as it cannot be retrieved again for security reasons Also, in the top left of the Realm UI there is a button to copy the Realm App ID. We will need this ID and API key when we write our application shortly. Lastly, we can enable Sync. The Sync configuration relies on a Schema of the data being written. This allows the objects (i.e. C# or Node.JS objects) from our application we are writing in the next step to be translated to MongoDB Documents. You can learn more about schemas here . We also need to identify a partition key. Partition keys are used to decide what subset of data should reside on each Edge node or each mobile device. For Wavelength deployments, this is typically a variation on the region name. A good partition key could be a unique one per API key or the name of the Wavelength Region (e.g. “BOS” or “DFW”). For this latter example, it would mean that your Far Cloud retains data for all zones, but the Wavelength zone in Boston will only have data tagged with “BOS” in the _pk field. The two ways to define a schema are either to write the JSON by hand or automatic generation. For the former, we would go to the Sync configuration, edit the Configuration tab, choose the cluster we deployed earlier, define a partition key (such as _pk as a string), then define the rules of what that user is allowed to read and write. Then you must write the schema on the Schema section of the Realm UI. However, it is often easier to let Realm auto-detect and write the schema for you. This can be done by putting the Sync into “Development Mode.” While you still choose the cluster and partition key, you only need to specify what database you want to sync all of your data to. After that, your application written below is where you can define classes, and upon connection to Realm Sync, the Sync Engine will translate the class you defined in your application into the underlying JSON representing that schema automatically. 3. Writing an application using Realm Sync: MQTT broker for a Smart Factory Now that the back-end data storage is configured, it is time to write the application. As a reminder, we will be writing an MQTT broker for a smart factory. IOT devices will write MQTT messages to this broker over 5G and our application will take that packet of information and insert it into the Realm database. After that, because we completed the sync configuration above, our Edge-to-Far-Cloud synchronization will be automatic. It also works bidirectionally. The reference application mentioned above is available in this GitHub repository . It is based on creating a C# Console application with the documentation here . The code is relatively straightforward: Create a new C# Console Application in Visual Studio Like any other C# Console Application, have it take in as CLI arguments the Realm App ID and API Key. These should be passed in via a Docker environment variable later and the values of these were the values you recorded in the previous Sync setup step Define the RealmObject which is the data model to write to Realm Process incoming MQTT messages and write them to Realm The data model for Realm Objects can be as complex as makes sense for your application. To prove this all works, we will keep a basic model: public class IOTDataPoint : RealmObject { [PrimaryKey] [MapTo("_id")] public ObjectId Id { get; set; } = ObjectId.GenerateNewId(); [MapTo("_pk")] public string Partition { get; set; } [MapTo("device")] public string DeviceName { get; set; } [MapTo("reading")] public int Reading { get; set; } } To sync an object, it must inherit from the RealmObject class. After that, just define getters and setters for each data point you want to sync. The C# implementation of this will vary depending on what MQTT Library you choose. Here we have used MQTTNet so we simply create a new broker with MqttFactory().CreateMqttServer() then start this with specific MqttServerOptionsBuilder where we need to define anything unique to your setup such as port, encryption, and other basic Broker information. However, we need to hook the incoming messages with .WithApplicationMessageInterceptor() so that way any time a new MQTT packet comes into the Broker, we send it to a method to write it to Realm. The actual Realm code is also simple: Create an App with App.Create() and it takes in the argument of the App ID which we are passing in as a CLI argument Log in with app.LogInAsync(Credentials.ApiKey()) and the API Key argument is again passed in as a CLI argument from what we generated before To insert into the database, all writes for Realm need to be done in a transaction. The syntax is straight forward: instantiate an object based on the RealmObject class we defined previously then do the write with a realm.Write(()=>realm.Add({message)}) Finally, we need to wrap this up in a docker container for easy distribution. Microsoft has a good tutorial on how to run this application inside of a Docker container with auto-generated Dockerfiles. On top of the auto-generated Dockerfile, be sure to pass in the arguments of the Realm App ID and API Key to the application as we defined earlier. Learning the inner workings of writing a Realm application is largely outside the scope of this blog post. However there is an excellent tutorial within MongoDB University if you would like to learn more about the Realm SDK. Now that the application is running, and in Docker, we can deploy it in a Wavelength Edge Zone as we created above. Bringing Realm and Wavelength together In order to access the application server in the Wavelength Zone, we must go through the bastion host we created earlier. Once we’ve gone through that jump box to get to the EC2 instance in the Wavelength Zone, we can install any prerequisites (such as Docker), and start the Docker container running the Realm Edge Database and MQTT application. Any new inbound messages received to this MQTT broker will be first written to the Edge and seamlessly synced to Atlas in the Far Cloud. There is a sample MQTT random number generator container suitable for testing this environment located in the GitHub repository mentioned earlier. Our smart factory reference application is complete! At this point: Smart devices can write to a 5G Edge with low latency courtesy of AWS Wavelength Zones MQTT Messages written to that Broker in the Wavelength Zone have low latency writes and are available immediately for reads since it is happening at the Edge through MongoDB Realm Those messages are automatically synchronized to the Far Cloud for permanent retention, analysis, or synchronization to other Zones via MongoDB Realm Sync and Atlas What's Next Get started with MongoDB Realm on AWS for free. Create a MongoDB Realm account Deploy a MongoDB backend in the cloud with a few clicks Start building with Realm Deploy AWS Wavelength in your AWS Account

October 14, 2021
Developer

Build a Single View of Your Customers with MongoDB Atlas and Cogniflare's Customer 360

The key to successful, long-lasting commerce is knowing your customers. If you truly know your customers, then you understand their needs and wants and can identify the right product to deliver to them—at the right time and in the right way. However, for most B2C enterprises, building a single view of the customer poses a major hurdle due to copious amounts of fragmented data. Businesses gather data from their customers in multiple locations, such as ecommerce platforms, CRM, ERP, loyalty programs, payment portals, web apps, mobile apps and more. Each data set can be structured, semi-structured or unstructured, delivered as stream or require batch processing, which makes compiling already fragmented customer data even more complex. This has led some organizations to bespoke solutions, which still only provide a partial view of the customer. Siloed data sets make running operations like customer service, targeted marketing and advanced analytics—such as churn prediction and recommendations—highly challenging. Only with a 360 degree view of the customer can an organization deeply understand their needs, wants and requirements, as well as how to satisfy them. A single view of that 360 data is therefore vital for a lasting relationship. In this blog, we’ll walk through how to build a single view of the customer using MongoDB’s database and Cogniflare’s Calledio Customer 360 tool. We’ll also explore a real-world use case focused on sentiment analysis. Building a single view with Calleido's Customer 360 With a Customer 360 database, organizations can access and analyze various individual interactions and touchpoints to build a holistic view of the customer. This is achieved by acquiring data from a number of disparate sources. However, routing and transforming this data is a complex and time-consuming process. Many of the existing Big Data tools often aren’t compatible with cloud environments. These challenges inspired Cogniflare to create Calleido . Figure 1: Calleido Customer 360 Use Case Architecture Calleido is a data processing platform built on top of battle-tested open source tools such as Apache NiFi. Calleido comes with over 300 processors to move structured and unstructured data from and to anywhere. It facilitates batch and real-time updates, and handles simple data transformations. Critically, Calleido seamlessly integrates with Google Cloud and offers one-click deployment. It uses Google Kubernetes Engine to scale up and down based on the demand, and provides an intuitive and slick low-code development environment. Figure 2: Calleido Data Pipeline to Copy Customers From PostgreSQL to MongoDB A real-world use case: Sentiment analysis of customer emails To demonstrate the power of Cogniflare’s Calleido , MongoDB Atlas , and the Customer 360 view, consider the use case of conducting a sentiment analysis on customer emails. To streamline the build of a Customer 360 database, the team at Cogniflare created flow templates for implementing data pipelines in seconds. In the upcoming sections, we’ll walk through some of the most common data movement patterns for this Customer 360 use case and showcase a sample dashboard. Figure 3: Sample Customer Dashboard The flow commences with a processor pulling IMAP messages from an email server (ConsumeIMAP). Each new email that arrives into the chosen inbox (e.g. customer service), triggers an event. Next, the process extracts email headers to determine topline details about the email content (ExtractEmailHeaders). Using the sender's email, Calleido identifies the customer (UpdateAttribute) and extracts the full email body by executing a script (ExecuteScript). Now, with all the data collected, a message payload is prepared and published through Google Cloud Platform (GCP) Pub/Sub (Kafka can also be used) for consumption by downstream flows and other services. Figure 4: Translating Emails to Cloud PubSub Messages The GCP Pub/Sub messages from the previous flow are then consumed (ConsumeGCPPubSub). This is where the power of MongoDB Atlas integration comes in as we verify each sender in the MongoDB database (GetMongo). If a customer exists in our system, we pass the email data to the next flow. Other emails are ignored. Figure 5: Validating Customer Email with MongoDB and Calleido Analysis of the email body copy is then conducted. For this flow, we use a processor to prepare a request body, which is then sent to Google Cloud Natural Language AI to assess the tone and sentiment of the message. The results from the Language Processing API then go straight to MongoDB Atlas so they can be pulled through into the dashboard. Figure 6: Making Cloud AutoML Call with Calleido End result in the dashboard: The Customer 360 database can be used in internal back-office systems to supplement and inform customer support. With a single view, it’s quicker and more effective to troubleshoot issues, handle returns and resolve complaints. Leveraging information from previous client conversations ensures each customer is given the most appropriate and effective response. These data sets can then be fed into analytics systems to generate learnings and optimizations, such as associating negative sentiment with churn rate. How MongoDB's document database helps In the example above, Calleido takes care of copying and routing data from the business source system into MongoDB Atlas, the operational data store (ODS). Thanks to MongoDB’s flexible data structure, we can transfer data in its original format, and subsequently implement necessary schema transformations in an iterative manner. There is no need to run complex schema migrations. This allows for the quick delivery of a single view database. Figures 7 & 8: Calleido Data Pipelines to Copy Products and Orders From PostgreSQL to MongoDB Atlas Calleido allows us to make this transition in just a few simple steps. The tool runs a custom SQL query (ExecuteSQL) that will join all the required data from outer tables and compile the results in order to parallelize the processing. The data arrives in Avro format, then Calleido converts it into JSON (ConvertAvroToJSON) and transforms it to the schema designed for MongoDB (JoltTransformJSON). End result in the Customer 360 dashboard: MongoDB Atlas is the market-leading choice for the Customer 360 database. Here are the core reasons for its world-class standard: MongoDB can efficiently handle non-standardized schema coming from legacy systems and efficiently store any custom attributes. Data models can include all the related data as nested documents. Unlike SQL databases, MongoDB avoids complicated join queries, which are difficult to write and not performant. MongoDB is rapid. The current view of a customer can be served in milliseconds without the need to introduce a caching layer. The MongoDB flexible schema model enables agility with an iterative approach. In the initial extraction, the data can be copied nearly exactly as its original shape. This drastically reduces latency. In subsequent phases, the schema can be standardized and the quality of the data can be improved without complex SQL migrations. MongoDB can store dozens of terabytes of data across multiple data centers and easily scale horizontally. Data can be shared across multiple regions to help navigate compliance requirements. Separate analytics nodes can be set up to avoid impacting performance of production systems. MongoDB has a proven record of acting as a single view database, with legacy and large organizations up and running with prototypes in two weeks and into production within a business quarter. MongoDB Atlas can autoscale out of the box, reducing costs and handling traffic peaks. The data can be encrypted both in transit and at rest, helping to accomplish compliance with security and privacy standards, including GDPR, HIPAA, PCI-DSS, and FERPA. Upselling the customer: Product recommendations Upselling customers is a key part of modern business, but the secret to doing it successfully is that it’s less about selling and more about educating. It’s about using data to identify where the customer is in the customer journey, what they may need, and which product or service can meet that need. Using a customer's purchase history, Calleido can help prepare product recommendations by routing data to the appropriate tools such as BigQuery ML. These recommendations can then be promoted through the call center and marketing teams for both online or mobile app recommendations. There are two flows to achieve this: preparing training data and generating recommendations: Preparing training data First, appropriate data from PostgreSQL to BigQuery is transferred using the ExecuteSQL processor. The data pipeline is scheduled to execute periodically. In the next step, appropriate data is fetched from PostgreSQL, dividing it to 1K row chunks with the ExecuteSQLRecord processor. These files are then passed to the next processor which uses load balancing enabled to utilize all available nodes. All that data then gets inserted to a BigQuery table using the PutBigQueryStreaming processor. Figure 9: Copying Data from PostgreSQL to BigQuery with Calleido Generating product recommendations Next, we move on to generating product recommendations. First, you must purchase Big Query capacity slots, which offer the most affordable way to take advantage of BigQuery ML features. Here, Calleido invokes an SQL procedure with the ExecuteSQL processor, then ensures that the requested BigQuery capacity is ready to use. The next processor (ExecuteSQL) executes an SQL query responsible for creating and training the Matrix Factorization ML model using the data copied by the first flow. Next in the queue, Calleido uses the ExecuteSQL processor to query our trained model to acquire all the predictions and store them in a dedicated BigQuery table. Finally, the Wait processor waits for both capacity slots to be removed, as they are no longer required. Figure 10 & 11: Generating Product Recommendations with Calleido Then, we remove old recommendations through the power of two processors. First, the ReplaceText processor updates the content of incoming flow files, setting the query body. This is then used by the DeleteMongo processor to perform the removal action. Figure 12: Remove Old Recommendations The whole flow ends with copying Recommendations to MongoDB. The ExecuteSQL processor fetches and aggregates the top 10 recommendations per user, all in chunks of 1k rows. Then, the following two processors (ConvertAvroToJSON and ExecuteScript) prepare data to be inserted into the MongoDB collection, by the PutMongoRecord processor. Figure 13: Copy Recommendations to MongoDB End result in the Customer 360 dashboard (the data used here in this example is autogenerated): Benefits of Calleido's 360 customer database on MongoDB Atlas Once the data is available in a centralized operational data store like MongoDB, Calleido can be used to sync it with an analytics data store such as Google BigQuery. Thanks to the Customer 360 database, internal stakeholders can then use the data to: Improve customer satisfaction through segmentation and targeted marketing Accurately and easily access compliance audits Build demand planning forecasts and analyses of market trends Reward customer loyalty and reduce churn Ultimately, a single view of the customer enables organizations to deliver the right message to prospective buyers, funneling those at the brand awareness stage into the conversion stage and ensures retention and post sales mechanics are working effectively. Historically, a 360 view of the customer was a complex and fragmented process, but with Cogniflare’s Calleido and MongoDB Atlas, a Customer 360 database has become the most powerful and cost efficient data management stack that an organization can harness.

October 14, 2021
Developer

Accelerate App Delivery with Cognizant's Next Gen Continuous Integrator

The phrase “digital transformation” is ubiquitous these days. But what does it actually mean? Often, the heart of a successful digital transformation lies in a company’s ability to migrate data from legacy systems to the cloud for easier access while also updating relevant applications quickly and seamlessly to deliver the most optimal experience to customers and end-users. To accomplish this journey in modernization, software teams have embraced agile practices founded on the pillars of collaboration, flexibility and continuous improvement and delivery. This is where continuous integration and continuous delivery (CI/CD) comes in. CI/CD is an agile practice that enables developers to implement small changes on a frequent basis. Conducted in an automated and secure environment that maintains version control, CI/CD not only accelerates app delivery, it also reduces infrastructure costs, minimizes security risks, and allows for zero-downtime deployments and easier rollbacks. The only problem? Many organizations lack the in-house expertise and resources needed to develop an agile CI/CD solution. How Cognizant's Next Generation Integrator can help The “Next Generation Continuous Integrator” is a UI-based, multi-component migration tool which utilizes a template-driven approach for migrating apps from existing CI/CD systems as well as on-boarding new apps to Concourse . The technology is built on two pillars: Templates and Containers. A template-driven approach is environment-agnostic (i.e. it can be run on private or public clouds, hybrid clouds, non-cloud, etc.) especially while handling numerous multi-tiered and multi-technology apps. A container is a standard unit of software that packages up code and all its dependencies and can be published into a Central Hub (public cloud) or a Customer Specific Hub (private). Containers can be used by various teams and applications to run applications reliably from one computing environment to another. How it works Based on the various needs of a project, including the services to be migrated or on-boarded, the Next Gen Continuous Integrator’s usage lifecycle is as follows: Reusable and shared infrastructure templates are built in association with the source CI/CD DevOps team. These form the core driving artefacts for the tool to work. These templates are pushed to GitHub repo and then to MongoDB. The templates are then exposed in the tool’s UI for the end user to choose the appropriate job configurations they need for their target pipelines. The services, according to the chosen configuration above, are then on-boarded or migrated from source CI/CD (Jenkins, GoCD etc) to Concourse. The migration/on-boarding reports are generated and pushed to MongoDB. Benefits Easier and quicker migration — CI/CD allows for system-agnostic migration across a broad spectrum of CI/CD tools through templates Quicker turn-around time — Easily accommodates Change Requests at any phase Improved quality — Standardizes processes faster. For instance, Next Gen Continuous Integrator designs a shared services layer for centralized management of the multiple environments and accounts Increased Savings — Projected cost savings of >70% as automation of the infrastructure leads to freeing up of internal resources who would otherwise be engaged in infrastructure management Why MongoDB for Next Gen Continuous Integrator tool? To understand why MongoDB is best positioned for the Next Gen Continuous Integrator tool, we first need to walk through the significant challenges of using relational databases (RDBMS) with the Next Gen Continuous Integrator. Challenges include: “Infrastructure-as-Code” templates are unstructured as they vary across teams and tools The reports that the tool generates are dynamic in nature, but reporting schema changes are difficult to make in an inflexible RDBMS Migrated source services are interdependent with multiple responsibilities and hence can become time-consuming to onboard them to Concourse The monolithic architecture of RDBMS affects management, scalability, and continuous deployment MongoDB, on the other hand, offers significant advantages: Flexible schema — Able to handle different data types and has the flexibility to store different templates of migration Ease of development — Design supports greater developer productivity and developers find it intuitive to use Rich JSON document and query language — Able to query templates at a deep hierarchical level Portability — For either the container or template driven approach, MongoDB can run across any platform There is no one-size-fits-all solution when it comes to workload migrations or onboarding applications for modern businesses. Next Generation Continuous Integrator can help clients develop a template-driven, containerized migration solution that fits the organization’s specific needs with ease and minimal operations support. Roadmap and contact details Based on the scale of adoption and performance analysis of Next Gen Continuous Integrator tool across multiple tenants, we may move from on-premise MongoDB to cloud-based MongoDB Atlas . Contact partner-presales@10gen.com for more information. Contact CognizantQEandA@cognizant.com for more information

September 29, 2021
Developer

Mainframe Data Modernization with MongoDB Powered by Wipro's "ModerniZ" Tool

This post will highlight a practical and incremental approach to modernizing mainframe data and workloads into cloud-hosted microservices using MongoDB’s modern, general purpose database platform. Enterprises modernize their mainframes for a number of reasons—to increase agility, lower the risk of aging legacy skills, and reduce total cost of ownership (TCO). But the greatest underlying benefit and reason for modernization lies in a company’s ability to access and make sense of their own data more quickly. Gleaning valuable business insights through the use of real-time data and AI/ML models is at the heart of today’s most successful and innovative companies. Consider the following business processes and their reliance on real-time insights: Real-time fraud detection, KYC, and score calculation Supporting new API requirements – PSD2, Open Banking, Fintech APIs etc Payment pattern analysis Moving away from hundreds of canned reports to template-based self-service configurable reporting Real-time management reporting With the continued emergence of mobile and web apps, enterprises are looking to render content even faster, as well as scale up and down on demand. However, mainframes often serve as the true system of record (SoR) and maintain the golden copy of core data. In a typical enterprise, inquiry transactions on the mainframe contribute to over 80% of the overall transaction volume—in some cases up to 95%. The goal for organizations is to increase the throughput of these inquiry transactions with improved response time. However, during real-time transaction processing, middleware must orchestrate multiple services, transform service response, and aggregate core mainframe and non-core applications. This architecture prevents the legacy services from seamlessly scaling on-demand with improved response time. At the same time, CIOs have significant concerns around the risks of big bang mainframe exit strategies, especially in the financial services, insurance and retail sectors where these complex applications serve the core business capabilities. Wipro and MongoDB’s joint offering, “ModerniZ,” can help alleviate these risks significantly by introducing a practical, incremental approach to mainframe modernization. An incremental solution: Offloading inquiry services data with ModerniZ To keep up with changing trends in an agile manner, and to bridge the gap between legacy monoliths and digital systems of engagement, a tactical modernization approach is required. While complete mainframe modernization is a strategic initiative, offloading inquiry services data and making it available off the mainframe is a popular approach adopted by several enterprises. Below are a few business requirements driving mainframe data offload: Seamlessly scale volume handling by up to 5X Improve response time by up to 2X Reduce mainframe TCO by up to 25% Direct API enablement for B2B and Partners (change the perception of being antiquated) Improve time to market on new enhancements by up to 3X Provision separate security mechanism for inquiry-only services Access single view data store for intra-day reporting, analytics and events handling These business requirements, as well as the challenges in current mainframe environments, warrant an offloaded data environment with aggregated and pre-enriched data from different sources in a format which can be directly consumed by channels and front-end systems. This is where our joint offering for mainframe modernization with MongoDB and Wipro’s ModerniZ tool comes into play. ModerniZ is Wipro’s IP platform specially focused on modernizing the UI, services and data layer of System Z (Mainframe). ModerniZ has multiple in-house tools and utilities to accelerate every phase of the legacy modernization journey. Let’s dig deeper into the solution elements. "CQRS with ModerniZ" for transactional and operational agility Command Query Responsibility Segregation (CQRS) is an architectural principle that can prescribe an operation as a ‘command,’ which performs an action, or a ‘query,’ which returns data to the requestor—but not both. CQRS achieves this by separating the read data model from the write data model. Separation of these two operations in a business process helps optimize performance, reduce the cost associated with inquiry transactions, and create a new model which can grow vertically and horizontally. MongoDB, with its document model and extensive set of capabilities, is best suited to house this offloaded ‘read data model.’ MongoDB’s JSON/BSON document model helps in pre-enriching the data and storing it in ‘inquiry ready’ format which simplifies the overhead for the front-end consumers. Enterprise use cases for CQRS: Customer demographic information – e.g. monetary and nonmonetary transaction inquiries in banking and financial services Payer view – Healthcare Single view across policy administration engines and pricing engines Consolidated participant view in benefit administration platforms Single view of manufacturing production control systems across plants or countries The below process indicates the step-by-step approach in enabling CQRS by offloading the data and transactional volumes into a modernized landscape, and continuously syncing the data (based on the business criticality) across platforms. Figure 1. Mainframe data modernization process The visual below indicates the conceptual target architecture where the service delivery platform (API/Middleware layer) identifies and routes the transaction to the respective systems. Any update which happens in the legacy system will be cascaded to the target MongoDB based on the business criticality of the fields. Figure 2. Post data modernization process view Mainframe services will continue to get exposed as domain APIs via zCEE for the Command Services (Update Transactions) and the newly built microservices will serve the inquiry transactions by fetching data from MongoDB. Any data updates in the mainframe will be pushed to MongoDB. The below table indicates how different fields can be synced between the mainframe and MongoDB, as well as their corresponding sync intervals. Java programs/Spark consumes the JSON document from Kafka Cluster, merges into MongoDB and creates new documents. table, th, td { border: 2px solid black; border-collapse: collapse; } Sync Type Field Classsification Sync Strategy Type-1 Near Real Time Sync for critical fields Using Kafka Queues / CICS Event Triggers / DB2 / 3rd Pardy CDC Replicators / Queue Replicators Type-2 Scheduled Batch Polling sync for less critical fields Using Mini Intra-day Batch / Replicators / ELT Type-3 EoD Batch sync for non-critical fields Batch CDC Sync Sync / Update / ELT How MongoDB and Wipro's ModerniZ helps Wipro’s ModerniZ platform provides multiple tools to accelerate the modernization journey across phases, from impact analysis to design to build to deployment. For data modernization, ModerniZ has 5 tool sets like PAN (Portfolio Analyzer), SQL Converter, automated data migration, and so on—all which can be leveraged to yield a minimum committed productivity gain of 20%. Figure 3. Mainframe to cloud transformation using ModerniZ Why MongoDB for mainframe modernization? MongoDB is built for modern application developers and for the cloud era. As a general purpose, document-based, distributed database, it facilitates high productivity and can handle huge volumes of data. The document database stores data in JSON-like documents and is built on a scale-out architecture that is optimal for any kind of developer who builds scalable applications through agile methodologies. Ultimately, MongoDB fosters business agility, scalability and innovation. Some key benefits include: Deploys across cloud in nearly all regions Provides a document model that is flexible and maps to how developers think and code Costs a fraction of the price compared to other offloading solutions built using relational or other NoSQL databases and even a larger, sharded MongoDB environment brings magnitudes of savings compared to the traditional mainframe MIPS-based licensing model Allows complex queries to be run against data via an extremely powerful aggregation framework. On top of that, MongoDB provides a BI connector for dedicated reporting/business intelligence tools as well as specific connectors for Hadoop and Spark to run sophisticated analytical workloads Offers enterprise-grade management through its EA product and includes security features which cover all areas of security – authentication, authorization, encryption and auditing. Competitors often only offer a subset of those capabilities Provides a unified interface to work with any data generated by modern applications Includes in-place, real-time analytics with workload isolation and native data visualization Maintains distributed multi-document transactions that are fully ACID compliant MongoDB has successfully implemented mainframe offloading solutions before and customers have even publicly talked about the success (e.g. a top financial service enterprise in the EU) Roadmap and contact details Work in progress on offloading mainframe read workloads to GCP native services and MongoDB. Contact partner-presales@mongodb.com for more information.

September 16, 2021
Developer

Serverless Instances Now Offer Extended Regional and Cloud Provider Support

Today’s applications are expected to just work, regardless of time of day, user traffic, or where in the world they are being accessed from. But in order to achieve this level of performance and scale, developers have to meticulously plan for infrastructure needs, sometimes before they even know what the success of their application may be. In many cases, this is not feasible and can lead to over provisioning and over paying. But what if you could forgo all of this planning and the database would seamlessly scale for you? Well, now you can - with serverless instances on MongoDB Atlas. Since we announced serverless instances in preview at MongoDB.live we have been actively working toward implementing new functionality to make them more robust and widely available. With our most recent release, serverless instances now offer expanded cloud providers and regions, and support MongoDB tools. Deploy a serverless instance on the cloud provider of your choice With our dedicated clusters on MongoDB Atlas, you have the flexibility to run anywhere with global reach on the cloud provider of your choice, so you can deliver responsive and reliable applications wherever your users are located. Our goal is to provide this same flexibility for serverless instances. We’re happy to announce that you can now deploy a serverless instance in ten regions on AWS, Google Cloud, and Azure. You’ll see when deploying a serverless instance there are now more regions supported on AWS, as well as two available regions on both Google Cloud and Azure - so you can get started with the cloud provider that best suits your needs or the region that’s closest to you. We will be continuing to add new regions over time to ensure coverage where you need it most. Easily import your data with MongoDB tools With this release, we have also made it easier to work with your data. You can now easily import data from an existing MongoDB deployment using the MongoDB Tools including mongodump, mongorestore, mongoexport , and mongoimport . In order to use MongoDB tools with serverless instances, you will need to be using the latest version . If you have additional feature requests that would make your developer experience better, share them with us in our feedback forums . Database deployment made simple With serverless instances, you can get started with almost no configuration needed - MongoDB Atlas will automatically scale to meet your workload needs, whether you have variable traffic patterns or you’re looking for a sandbox database for your weekend hobby project. If you haven’t yet given serverless instances a try, now is a great time to see what they can offer. If you have feedback or questions, we’d love to hear them! Join our community forums to meet other MongoDB developers and see what they’re building with serverless instances. Create your own serverless instance on MongoDB Atlas. Try the Preview .

September 16, 2021
Developer

A Guide to Freeing Yourself from Legacy RDBMS

Oracle introduced the first commercial relational database (RDBMS) to the market in 1979 — more than a decade before the World Wide Web. Now, digital transformation is reshaping every industry at an accelerating pace. In an increasingly digital economy, this means a company's competitive advantage is defined by how well they build software around their most critical asset — data. MongoDB and Palisade Compliance have helped some of the largest and most complex Oracle customers transform their architecture and shift to a cloud-first world. Although every client is unique, we have identified three important steps to moving away from Oracle software, reducing costs, and achieving their digital transformation goals: Understand your business and technical requirements for today and tomorrow, and identify the technical solution and company that will be by your side to help future-proof your organization. Decipher your Oracle contracts and compliance positions to maximize cost reduction initiatives and minimize any risks from Oracle audits and non-compliance that may derail your ultimate goals. Mobilize internal momentum and traction to make the move. MongoDB can help with #1, Palisade Compliance assists with #2, and you have to supply #3. This is a guide to getting started, as outlined by the main pillars of success above. 1. Understand your requirements and find the right partner — MongoDB The most common requirements we hear from organizations are that they need to move faster, increase developer productivity, and improve application performance and scale -- all while reducing cost and breaking free from vendor lock-in. For example , to keep pace with demands from the business, Travelers Insurance modernized its development processes with a microservices architecture supported by agile and DevOps methodologies. But the rigidity of its existing Oracle and SQL Server databases created blockers to move at the speed they needed. The solution was MongoDB and its flexible data model. They eliminated the three-day wait to make any database changes, creating a software development pipeline supporting continuous delivery of new business functionality. Similarly, Telefonica migrated its customer personalization service from Oracle to MongoDB. Using Oracle, it took 7 developers, multiple iterations and 14 months to build a system that just didn't perform. Using MongoDB, a team of 3 developers built its new personalization service in 3 months, which now powers both legacy and new products across the globe. MongoDB helps Telefonica be more agile, save money and drive new revenue streams. While some organizations try to innovate by allowing siloed, modern databases to coexist with their legacy relational systems, many organizations are moving to fully replace RDBMS. Otherwise, a level of complexity remains that creates significant additional work for developers because separate databases are required for search, additional technologies are needed for local data storage on mobile devices, and data often needs to be moved to dedicated analytics systems. As a result, development teams move slowly, create fewer new features, and cost the organization more capital. MongoDB provides the industry’s first application data platform that allows you to accelerate and simplify how you build with data for any application. Developers love working with MongoDB’s document model because it aligns with how they think and code. The summarized functional requirements that we typically hear from leading companies and development teams regarding what they require from a data platform include: A data structure that is both natural and flexible for developers to work with Auto-scaling and multi-node replication Distributed multi-document transactions that are fully ACID compliant Fully integrated full-text search that eliminates the need for separate search engines Flexible local datastore with seamless edge to cloud sync In-place, real-time analytics with workload isolation and native data visualization Ability to run federated queries across your operational/transactional databases and cloud object storage Turnkey global data distribution for data sovereignty and fast access to Data Lake Industry-leading data privacy controls with client-side, field level encryption Freedom to run anywhere, including the major clouds across many regions MongoDB delivers everything you need from a modern data platform. But it’s not just about being the right data platform; we’re also the right modernization partner. Through our Modernization Program we have built and perfected modernization guides that help you select and prioritize applications, review best practices, and design best-in-class, production-grade, migration frameworks. We’ve built an ecosystem around accelerating and simplifying your journey that includes: deployment on the leading cloud providers to enable the latest innovations technology companies that help with data modeling, migration, and machine learning, and expert System Integrators to provide you with tools, processes and support to accelerate your projects. We are proud to be empowering development teams to create faster and develop new features and capabilities, all with a lower total cost of ownership. 2. Manage Oracle as you move away — Palisade Compliance Oracle’s restrictive contracts, unclear licensing rules, and the threat of an audit can severely impact a company’s ability to transform and adopt new technologies that are required in a cloud-first world. To move away from Oracle and adopt new solutions, companies must be sure they can actually reduce their costs while staying in compliance and avoiding the risks associated with an audit. There will be a time when you are running your new solution and your legacy Oracle software at the same time. This is a critical phase in your digital transformation as you do not want to be tripped up by Oracle’s tactics and forced to stay with them. It may seem counterintuitive, but as you spend less with Oracle you must be even more careful with your licensing. As long as you keep spending money with Oracle and renewing those expensive contracts, the threat of an audit and non-compliance will remain low. Oracle is unlikely to audit a company that keeps giving it money. However, the moment you begin to move to newer technologies, your risk of an audit significantly increases. As a result, you must be especially vigilant to prevent Oracle from punishing you as you move away from them. Even if you’ve found a technical partner and managed your Oracle licenses and compliance to ensure no surprises, you still have to find a way to reduce your costs. It’s not as simple as terminating Oracle licenses and seeing your support costs go down. As stated above, Oracle contracts are designed to lock in customers and make it nearly impossible to actually reduce costs. Palisade Compliance has identified eleven ways to manage your Oracle licenses and reduce your Oracle support. It is critical that you understand and identify the options that work for your firm, and then build and execute on a plan that ensures your success. 3. Mobilize internal momentum and traction to make the move Legacy technology companies excel at seeding doubt into organization and preventing moves that threaten their antiquated solutions. Unfortunately, too many companies succumb to these tactics and are paralyzed into a competitive disadvantage in the market. In software, as in life, it’s easier to stay the course than to follow through with change. But when it comes to technical and business decisions that impact the overall success and direction of an organization, innovation and change aren’t just helpful, they’re necessary to survive--especially in a world with high customer demands and easy market entry. Ensuring you have the right technical partner and Oracle advisor is the best way to build the confidence and momentum needed to make your move. Creating that momentum is easier with MongoDB’s Database Platform, consisting of a fully managed service across 80+ regions, and Palisade’s expertise in Oracle licensing and contracts. Technical Alternative (MongoDB) + Independent Oracle Advisors (Palisade) ⇒ Momentum Parting thoughts To schedule a preliminary health check review and begin building the right strategy for your needs, fill out your information here . And to learn more about MongoDB’s Modernization Program, visit this page . About Palisade Compliance With over 400 clients in 30 countries around the world, Palisade is the leading provider of Oracle-independent licensing, contracting, and cost reduction services. Visit the website to learn more. To schedule a complementary one-hour Oracle consultation send an email to info@palisadecompliance.com.

September 2, 2021
Developer

Highlight What Matters with the MongoDB Charts SDK

We're proud to announce that with the latest release of the MongoDB Charts SDK you can now apply highlights to your charts. These allow you to emphasize and deemphasize your charts with our MongoDB query operators . Build a richer interactive experience for your customers by highlighting with the MongoDB Charts embedding SDK . By default, MongoDB Charts allows for emphasizing parts of your charts by series when you click within a legend. With the new highlight capability in the Charts Embedding SDK, we put you in control of when this highlighting should occur, and what it applies to. Why would you want to apply highlights? Highlighting opens up the opportunity for new experiences for your users. The two main reasons why you may want to highlight are: To show user interactions: We use this in the click handler sandbox to make it obvious what the user has clicked on. You could also use this to show documents affected by a query for a control panel. Attract the user’s attention: If there's a part of the chart you want your users to focus on, such as the profit for the current quarter or the table rows of unfilled orders. Getting started With the release of the Embedding SDK , we've added the setHighlight method to the chart object, which uses MQL queries to decide what gets highlighted. This lets you attract attention to marks in a bar chart, lines in a line chart, or rows in a table. Most of our chart types are already supported, and more will be supported as time goes on. If you want to dive into the deep end, we've added a new highlighting example and updated the click event examples to use the new highlighting API: Highlighting sandbox Click events sandbox Click events with filtering sandbox The anatomy of a click In MongoDB Charts, each click produces a wealth of information that you can then use in your applications , as seen below: In particular, we generate an MQL expression that you can use called selectionFilter , which represents the mark selected. Note that this filter uses the field names in your documents, not the channel names. Before, you could use this to filter your charts with setFilter , but now you can use the same filter to apply emphasis to your charts. All this requires is calling setHighlight on your chart with the selectionFilter query that you get from the click event, as seen in this sandbox . Applying more complex highlights Since we accept a subset of the MQL language for highlighting, it's possible to specify highlights which target multiple marks, as well as multiple conditions. We can use expressions like $lt and $gte to define ranges which we want to highlight. And since we support the logical operators as well, you can even use $and / $or . All the Comparison , Logical and Element query operators are supported, so give it a spin! Conclusion This ability to highlight data will make your charts more interactive and help you better emphasize the most important information in your charts. Check out the embedding SDK to start highlighting today! New to Charts? You can start now for free by signing up for MongoDB Atlas , deploying a free tier cluster and activating Charts. Have an idea on how we can make MongoDB Charts better? Feel free to leave an idea at the MongoDB Feedback Engine .

September 2, 2021
Developer

MongoDB Atlas as a Data Source for Amazon Managed Grafana

Amazon Managed Grafana is a fully managed service that is based on open source Grafana. Amazon Managed Grafana makes it easy to visualize and analyze operational data at scale. With Amazon Managed Grafana, organizations can analyze data stored in MongoDB Atlas without having to provision servers, configure or update software, or do the heavy lifting involved in securing and scaling Grafana in production. Connecting MongoDB Atlas to AMG The MongoDB Grafana plug-in makes it easy to query MongoDB with Amazon Managed Grafana. Simply select MongoDB as a data source, then connect to theMongoDB cluster using an Atlas connection string and proper authentication credentials (see Figure 1). Figure 1. Set up: MongoDB Grafana plug-in Now, MongoDB is configured as a data source. To visualize the data through Amazon Managed Grafana, select the Explore tab in the side panel and ensure that MongoDB is selected as the data source. Users can then write the first query in the query editor (see Figure 2). sample_mflix.movies.aggregate([ {"$match": { "year": {"$gt" : 2000} }}, {"$group": { "_id": "$year", "count": { "$sum": 1 }}}, {"$project": { "_id": 0, "count": 1, "time": { "$dateFromParts": {"year": "$_id", "month": 2}}}} ] ).sort({"time": 1}) Figure 2. AMG query editor Grafana will graph the query, illustrating how certain fields change over time. For more granular detail, users can review the data view below the visualization. (see Figure 3). Figure 3. AMG data view Using MongoDB as a data source in Amazon Managed Grafana allows users to easily analyze MongoDB data alongside other data sources, affording a singular point of reference for all of the most important data in an application. There’s no hassle; once connected to MongoDB from Amazon Managed Grafana, it simply works. Try out MongoDB Atlas with Amazon Managed Grafana today.

September 1, 2021
Developer

Simplifying Data Migrations From Legacy SQL to MongoDB Atlas with Studio 3T and Hackolade

Migrating data from SQL relational databases to MongoDB Atlas may initially seem to be a straightforward process. Export the data from the relational database, import the tables into MongoDB Atlas, and then start writing queries for it. But the deeper you look, the more it can start to feel like an overwhelming task. Decades of irrelevant indexes, rare relationships and forgotten fields that need to be migrated all make for a more complicated process. Not to mention, converting your old schemas to work well with the document-based world of MongoDB can take even longer. Making this process easier and more achievable was one of Studio 3T’s main goals when they created their SQL to MongoDB migration tools. In fact, Studio 3T's SQL migration doesn't just make this process easier; it also makes it reliably repeatable. This means you can tune your migration to produce the perfect documents in an ideal schema. There's no big bang cut over; you can proceed at your own pace. Delivering the ability to perfect your new schema is also why we've integrated with Hackolade's schema design and data modeling tools. In this article, we look at how the data modernization process works with a focus on the SQL migration powers of Studio 3T and the data modelling technology of Hackolade. So the problem was, how do I migrate the data that’s been collected over the last 10 years from MySQL over to MongoDB? And that’s when I found Studio 3T. I could not imagine trying to do it myself by hand... If it hadn’t been for Studio 3T, I probably would have just left it all in the SQL database. Rand Nix, IT Director, Wakefield Inspection Services Why MongoDB Atlas Building on MongoDB’s intuitive data model and query API, MongoDB Atlas gives engineering organizations the versatility they need to build sophisticated applications that can adapt to changing customer demands and market trends. Not only is it the only multi-cloud document database available, it also delivers the most advanced security and data distribution capabilities of any fully managed service. Developers can get started in minutes and leverage intelligent automation to maintain performance at scale as applications evolve over time. Mapping the move The crucial component to a SQL migration is mapping the SQL tables and columns to JSON documents and their name/value fields. While the rigid grid of relational tables has a familiar feeling, a document in MongoDB can be any shape and, ideally, it should be the shape that makes sense for your business logic. Doing this by hand can prove extremely time-consuming for developers. Which situations call for reshaping your data into documents and which should you construct new documents for? These are important questions which require time and expertise to answer. Or, you can simply use Studio 3T's SQL Migration tool which is a powerful, configurable import process. It allows you to create your MongoDB documents with data retrieved from multiple relational tables. Using foreign keys, it can capture the relationships between records in document-centric form. "When we started out, we built it on SQL Server. All was good to start, but by the time we got up to fifty customers and more, each with some thousands of their staff logging into the system concurrently, we ran into scaling issues. "Now we’re using a hosted, paid-for subscription on MongoDB Atlas. They do all the management and the sharding and all that stuff. And when we switch to regional provisioning, they can handle all that too, which is a huge relief. "We all use Studio 3T constantly, it’s used as extensively as Visual Studio. We use it for all sorts of things because Studio 3T makes it really easy to navigate around the data." Dan Cummings, Development Mgr. Terryberry Inc. For example, a relational database may have a table of customers and a table of orders by each customer. With the SQL Migration tool, you can automatically create a collection of customer documents, containing an array of all the orders that customer has placed. This exploits the power of the document model by keeping related data local. You can also preview the resulting JSON documents before running a full import so you are sure you're getting what you want. Figure 1. Studio 3T’s SQL to MongoDB Migration In Figure 1 (see above), we can see Studio 3T's SQL Migration pulling in multiple SQL tables, embedding rental records into the customer records, and previewing the JSON output. For more radical restructuring, SQL queries can also provide the source of the data for import. This allows for restructuring of the data by, for example, region or category. Hackolade integrated As you can see, Studio 3T is fully capable of some complex migrations through its tooling, SQL Migration, Import and Reschema. However, if you are of the mindset that you should design your migration first, then this is where Hackolade comes in. Hackolade is a "polyglot data" modelling application designed for a world where models are implemented in everything from graphs to APIs to storage formats. Figure 2. Entity Relationships Mapped in Hackolade Using Hackolade, you can import the schema with relationships from an SQL database and then visually reassemble the SQL fields to compose your MongoDB document schema. Watch this demo video for more details. Once you've built your new models, you can use Studio 3T to put them into practice by importing a Hackolade schema mapping into a SQL Migration configuration. Figure 3. Importing Hackolade Models into Studio 3T This helps eliminate the need to repeatedly query and import from the SQL database as you fine-tune your migration. Instead, you can construct and document the migration within Hackolade, review with your team, and then implement with confidence. Once the data is in MongoDB Atlas, support from Studio 3T continues. Studio 3T's Reschema tools allow you to restructure your schema without re-importing, working entirely on the MongoDB Atlas data. This can be particularly useful when blending data from existing document databases with freshly imported, formerly relational data. The idea that migrations have to be team-breaking chores no longer holds. It is eminently possible, with tools such as Studio 3T's SQL Migration and Hackolade, to not only perform more complex migrations easily, but to be able to design them up front and put them into practice as often as is needed. With both tools working in integrated harmony, the future of migration has arrived.

August 24, 2021
Developer

Ready to get Started with MongoDB Atlas?

Start Free