MongoDB Developer

Coding with MongoDB - news for developers, tips and deep dives

Client-Side Field Level Encryption is now on Azure and Google Cloud

We’re excited to announce expanded key management support for Client-Side Field Level Encryption (FLE). Initially released last year with Amazon’s Key Management Service (KMS), native support for Azure Key Vault and Google Cloud KMS is now available in beta with support for our C#/.Net, Java, and Python drivers. More drivers will be added in the coming months. Client-Side FLE provides amongst the strongest levels of data privacy available today. By expanding our native KMS support, it is even easier for organizations to further enhance the privacy and security of sensitive and regulated workloads with multi-cloud support across ~80 geographic regions. My databases are already encrypted. What can I do with Client-Side Field Level Encryption? What makes Client-Side FLE different from other database encryption approaches is that the process is totally separated from the database server. Encryption and decryption is instead handled exclusively within the MongoDB drivers in the client, before sensitive data leaves the application and hits the network. As a result, all encrypted fields sent to the MongoDB server – whether they are resident in memory, in system logs, at-rest in storage, and in backups – are rendered as ciphertext. Neither the server nor any administrators managing the database or cloud infrastructure staff have access to the encryption keys. Unless the attacker has a compromised DBA password, privileged network access, AND a stolen client encryption key, the data remains protected, securing it against sophisticated exploits. MongoDB’s Client-Side FLE complements existing network and storage encryption to protect the most highly classified, sensitive fields of your records without: Developers needing to write additional, highly complex encryption logic application-side Compromising your ability to query encrypted data Significantly impacting database performance By securing data with Client-Side FLE you can move to managed services in the cloud with greater confidence. This is because the database only works with encrypted fields, and you control the encryption keys, rather than having the database provider manage the keys for you. This additional layer of security enforces an even finer-grained separation of duties between those who use the database and those who administer and manage the database. You can also more easily comply with “right to erasure” mandates in modern privacy legislation such as the GDPR and the CCPA . When a user invokes their right to erasure, you simply destroy the associated field encryption key and the user’s Personally Identifiable Information (PII) is rendered unreadable and irrecoverable to anyone. Client-Side FLE Implementation Client-Side FLE is highly flexible. You can selectively encrypt individual fields within a document, multiple fields within the document, or the entire document. Each field can be optionally secured with its own key and decrypted seamlessly on the client. To check-out how Client-Side FLE works, take a look at this handy animation. Client-Side FLE uses standard NIST FIPS-certified encryption primitives including AES at the 256-bit security level, in authenticated CBC mode: AEAD AES-256-CBC encryption algorithm with HMAC-SHA-512 MAC. Data encryption keys are protected by strong symmetric encryption with standard wrapping Key Encryption Keys, which can be natively integrated with external key management services backed by FIPS 140-2 validated Hardware Security Modules (HSMs). Initially this was with Amazon’s KMS, and now with Azure Key Vault and Google Cloud KMS in beta. Alternatively, you can use remote secure web services to consume an external key or a secrets manager such as Hashicorp Vault. Getting Started To learn more, download our Guide to Client-Side FLE . The Guide will provide you an overview of how Client-Side FLE is implemented, use-cases for it, and how it complements existing encryption mechanisms to protect your most sensitive data. Review the Client-Side FLE key management documentation for more details on how to configure your chosen KMS. Safe Harbor The development, release, and timing of any features or functionality described for our products remains at our sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality.

November 9, 2020
Developer

Accelerating Mainframe Offload to MongoDB with TCS MasterCraft™

Tata Consultancy Services (TCS), a leading multinational information technology services and consulting company, leverages its IP-based solutions to accelerate and optimize service delivery. TCS MasterCraft™ TransformPlus uses intelligent automation to modernize and migrate enterprise-level mainframe applications to new, leading-edge architectures and databases like MongoDB. In this blog, we’ll review the reasons why organizations choose to modernize and how TCS has made the process easy and relatively risk-free. Background Legacy Modernization Legacy modernization is a strategic initiative that enables you to refresh your existing database and applications portfolio by applying the latest innovations in development methodologies, architectural patterns, and technologies. At the current churn rate, about half of today’s S&P 500 firms will be replaced over the next 10 years $100T of economic value is ready to be unlocked over the next decade via digital transformation Source Legacy System Challenges Legacy technology platforms of the past, particularly monolithic mainframe systems, have always been challenged by the pace of disruptive digitalization. Neither the storage nor the accessibility of these rigid systems is agile enough to meet the increasing demands of volume, speed, and data diversity generated by modern digital applications. The result is noise between the legacy system of record and digital systems of engagement. This noise puts companies at a competitive disadvantage. It often manifests as a gap between customer service and user experience, impeding the delivery of new features and offerings and constraining the business from responding nimbly to changing trends. Operational costs of mainframe and other legacy systems have also skyrocketed. With each million instructions per second (MIPS) costing up to $4,000 per year, these older systems can create the equivalent of nearly 40% of an organization’s IT budget in technical debt, significantly increasing the overall annual run cost. And as qualified staff age and retire over the years, it’s becoming harder to find and hire people with the required mainframe skills. To manage MIPS consumption, a large number of our customers are offloading commonly accessed mainframe data to an independent operational data layer (ODL), to which queries are redirected from consuming applications. IT experts understand both the risk and the critical need to explore modernization options like encapsulation, rehosting, replatforming, refactoring, re-architecting, or rebuilding to replace these legacy systems. The key considerations when choosing an approach are familiar: risk of business disruption, cost, timelines, productivity, and the availability of the necessary skills. MongoDB + TCS MasterCraft™ TransformPlus = Transformation Catalyst To stay competitive, businesses need their engineering and IT teams to do these three things, among others: Build innovative digital apps fast Use data as a competitive moat to protect and grow their business Lower cost and risk while improving customer experience Some customers use a “lift and shift” approach to move workloads off the mainframe to cloud for immediate savings, but that process can’t unlock the value that comes with microservice architectures and document databases. Others gain that value by re-architecting and rewriting their applications, but this approach can be time consuming, expensive, and risky. More and more, customers are using a tools-driven refactoring approach to intelligently automate code conversion. What TCS MasterCraft™ TransformPlus Brings to the Table TCS MasterCraft™TransformPlus automates the migration of legacy applications and databases to modern architectures like MongoDB. It extracts business logics from decades-old legacy mainframe systems as a convertible, NoSQL document data model for deployment. This makes extraction faster, easier, and more economical, and reduces the risk that comes with rewriting legacy applications. With more than 25 years of experience, TCS’s track record includes: 60+ modernization projects successfully delivered 500M+ lines of COBOL code analyzed 25M+ lines of COBOL code converted to Java 50M+ new lines of Java code auto-generated What MongoDB Brings to the Table MongoDB’s document data model platform can help make development cycles up to 5 times faster. Businesses can drive innovation faster, cut costs by 70% or more, and reduce their risk at the same time. As a developer, MongoDB gives you: The best way to work with data The ability to put data where you need it The freedom to run anywhere Why is TCS collaborating with MongoDB for Mainframe Offload? Cost. Redirecting queries away from the mainframe to the ODL significantly reduces costs. Even cutting just 20%-30% in MIPS consumption can save millions of dollars in mainframe operating costs. Agility. As an ODL built on a modern data platform, MongoDB helps developers build new apps and digital experiences 3—5 times faster than is possible on a mainframe. User Experience. MongoDB meets demands for exploding data volumes and user populations by scaling out on commodity hardware, with self-healing replicas that maintain 24x7 service. More details can be found here . How TCS MasterCraftTM Accelerates Mainframe Offload to MongoDB Data Migration Configures target document schema to corresponding relational schema Automatically transforms relational data from mainframe sources to MongoDB documents Loads data to MongoDB Atlas with the latest connector support Application Migration Facilitates a cognitive code analysis-based application knowledge repository Ensures complete, comprehensive application knowledge extraction Automates conversion of application logic from COBOL to Java, with data access layer accessing data from MongoDB Splits monolithic code into multiple microservices Automates migration of mainframe screens to AngularJS-based UI Together, TCS MasterCraft™ TransformPlus and MongoDB can simplify and accelerate your journey to the cloud, streamlining and protecting your data while laying the foundation for digital success. Download the Modernization Guide to learn more.

October 28, 2020
Developer

Designing a CLI: 11 Principles from the MongoDB Realm Design Team

Don Norman saw this coming. Back in 2007 , Norman, one of the giants of user-centered design, predicted that the next UI breakthrough in the design world would be the command-line interface (CLI). Noting the limitations of graphical user interfaces and the capabilities of command-line language in search functions, Norman staked this bold claim: “Command line interfaces are back again, hiding under the name of search. Now you see them, now you don't. Now you see them again. And they will get better and better with time: Mark my words, that is my prediction for the future of interfaces.” In 2020, the return of the CLI is, perhaps, debatable. In the world of software development, CLI's still dominate with software professionals , due to their wide availability, high capacity for automation, and cultural fit with the developer ethos. But in the design world, “user experience” is still largely associated with modern web and mobile interfaces — clean whites, curved borders, and gradient buttons. Design interventions of the CLI are often left to the wayside. Here at MongoDB, our designers understand the importance of the CLI to our users. Through internal discussion and collaboration with our product and engineering teams, we are working hard to match the user experience of the CLI to our user’s needs. As such, the MongoDB Realm team has been working on a revamp of our Realm CLI to be released by the end of Q4. In order to improve the experience of using the CLI, our UX research and Realm design teams have conducted primary and secondary research, attempting to figure out how our CLI is used, and how developers commonly interact with CLIs writ large. Based on our findings, we created a list of 11 CLI UX principles for the Realm CLI . We call this our CLI Design Cheat Sheet. Modeled off Nielson-Norman’s Usability Heuristics , this set of principles has allowed us to inform and streamline our CLI design process, and to foster a user-centered approach across our product’s operations. It’s my hope that presenting these principles will help others design better CLIs, too. We’ll provide some CLI illustrations that are not representative of the new Realm CLI, but can help show these principles visually. 1. Allow users to create and clone Realm applications and assets via the CLI Develop short and easy-to-understand commands for app creation and cloning. Consider dividing this command into an object and an action: For example, the command may read realm app clone , where “app” is the object, and “clone” is the action. 2. Use accessible language to bridge the CLI and the real world The system should speak the users' language, with words, phrases, and concepts familiar to the user, rather than system-oriented terms. When creating CLI commands and prompts, use questions or phrases that resemble sentences. Avoid positional arguments, where the order matters. These types of arguments can be confusing. Use flags instead of args. Although they require a little more typing, flags better prevent input errors (e.g., realm fork --from sourceapp --to destapp ) compared to args (e.g., realm fork -sourceapp ) When collecting user information, use questions to make the CLI more conversational. For example, when initializing, the CLI can ask the user questions such as, “What’s your project name?” when determining the user’s framework. 3. Simplify CLI outputs to increase user control Complicated and messy outputs reduce user control and diffuse action. CLI users expect high-level outputs following a command. Make outputs simple so that users don’t need to scroll through multiple lines of text to find what they need. The MongoDB CLI provides a good example of human-friendly CLI output: Human-friendly MongoDB CLI output. Additionally, the CLI should allow for both human-friendly (plaintext) output, and machine-friendly (JSON) output. A user should be able to define which output they’d like to see. For example, the MongoDB CLI has the following functionalities that allow users to define their output as plaintext or JSON: Illustration: Allowing users to choose between different output formats. 4. Make terminologies consistent Try to use the same terminology consistently across your product’s system. In the case of Realm, the product’s CLI should use the same terminology as its GUI, as well as other CLI’s across the MongoDB platform. Additionally, try to draw from the user’s context with terminologies from the tools and CLIs that are already familiar to your users. 5. Prevent errors As mentioned before, the use of flags -- as opposed to args -- is one way to accomplish this. Another consideration is to have clear warnings and retype commands. These can better prevent destructive mistakes. For instance, instead of using a “Y/N” prompt, consider having the user re-type their input, and prefacing with a warning that it may result in drastic changes to their program. Illustration: Providing a warning and a retype command to prevent destructive errors. 6. Maximize user recognition by combining commands or prompting input Make the CLI experience more efficient and easy-to-use by easing the user’s memory load. One way to do that is by giving users a single command to perform a task, which doesn’t require them to remember certain inputs. For MongoDB Realm, this principle can be mobilized to improve our authentication experience. A single command can be called to automatically generate CLI credentials from the Realm CLI for Atlas. If it isn’t possible or ideal to combine several commands, then consider a prompt showing users a choice of complicated options in the CLI. Rather than asking users to type and remember an input, the CLI can show them a series of options. Illustration: A prompt suggesting complicated inputs that are hard to remember 7. Make the CLI more flexible by allowing users to easily set and change configuration options Understood generally, this principle is meant to highlight user freedom and customization capability. The CLI should provide straightforward commands that can allow users to easily set and change things as they wish. In the case of Realm, we are looking to make MongoDB auth configuration file creation automatic upon Realm initialization, with the access token stored in the user’s home folder. This makes configuration more efficient. Additionally, we are considering allowing users to change their config file directly in the CLI to increase user flexibility. 8. Implementing aesthetics and interactions There are different visual considerations that can improve CLI interactions and experiences. Here are a few examples: Adding better color support (e.g., Yellow or green = good; red = wrong) Adding visual hierarchy for tables (e.g., making headers stand out through highlighting) Adding spinners, progress bars, and/or step count to show long-running tasks Illustrations: Different ways to show task runtime. 9. Help users recognize, diagnose, and recover from errors Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. They should be informative and should indicate next steps. Consider providing or linking to a troubleshooting guide or links to documentation as part of your error messages. 10. Help and documentation For users that decide not to go through the CLI docs, the help command is vital. We recommend writing a small description of the CLI’s purpose in the --help command. Also, consider providing links to documentation when necessary (i.e. as part of an error message). An example of an effective --help output is seen in the MongoDB CLI: The MongoDB CLI's --help output is both descriptive and concise. 11. Maintain user control and freedom by allowing users to opt-out of data collection Allow users to control data collection settings from their config in the CLI. This functionality can be provided with a command. In the case of realm, this may look like: --realm telemetry-disable . We hope these principles provide a solid foundation for designers and developers looking to create effective, user-centered CLIs. Be on the lookout for a new version of our Realm CLI by the end of Q4! Interested in trying MongoDB Realm ? Start building your app today! Start Free

October 21, 2020
Developer

Using MongoDB CLI to Get Atlas Performance Advice From Your Terminal

MongoDB CLI brings the power of your MongoDB cloud (MongoDB Atlas, MongoDB Cloud Manager, and MongoDB Ops Manager) to your terminal, helping you write advanced scripts and automate different workflows for your MongoDB cloud infrastructure. We released MongoDB CLI's first stable version back in June. So far we've seen more than 100 clusters deployed with the tool, as well as with many other operations such as getting metrics for these clusters or managing backups. Meanwhile, we've been busy adding new features and listening to your feedback as to what we should add next. With the latest release of mongocli , we have added the ability to get Atlas performance recommendations. Let's see how they work. $ mongocli atlas clusters create slowQueriesDemo \ --region EASTERN_US \ --members 3 \ --tier M30 \ --provider GCP \ --mdbVersion 4.4 \ --diskSizeGB 30 This command deploys a three-member replica named slowQueriesDemo, with GCP as the backing cloud provider. You can always use mongocli atlas clusters create --help to review all available options when creating a cluster. Getting your new cluster ready can take some time. Use mongocli to be notified when your changes are available with a watch : $ mongocli atlas clusters watch slowQueriesDemo Performance Advisor works by scanning your Atlas Cluster logs and finding any slow operations that could be affecting the performance of your queries. For this example, I loaded my demo Atlas cluster with some dummy data. I executed a series on MongoDB commands where I expect to see the Performance Advisor recommend new indexes to improve my queries. If you're not seeing any suggestions, please check our docs . The performance advisor command requires a process identifier. To check for the processes available to our project: $ mongocli atlas processes ls ID REPLICA SET NAME SHARD NAME VERSION atlas-00-00.gcp.mongodb.net:27017 atlas-fjszq1-shard-0 4.4.1 atlas-00-01.gcp.mongodb.net:27017 atlas-fjszq1-shard-0 4.4.1 atlas-00-02.gcp.mongodb.net:27017 atlas-fjszq1-shard-0 4.4.1 We want to pick one process ID related to our cluster, and then use it with the performance advisor: $ mongocli atlas performanceAdvisor suggestedIndexes ls --processName atlas-00-01.gcp.mongodb.net:27017 ID NAMESPACE SUGGESTED INDEX 5f75c6d38788f14f816f80ee loadTest.saildrone { timeUTC: 1 } 5f75c6d38788f14f816f80ef loadTest.saildrone { id: 1, arrayOperation: 1, decimal: 1, nested.field: 1, quotedReservedWordField: 1, date: 1, binary: 1, embeddedArrayDocument: 1, embeddedDocument: 1, embeddedArray: 1, embeddedDocumentArray: 1, nullValue: 1, coordinates.latitude: 1 } 5f75c6d38788f14f816f80f0 loadTest.saildrone { RH_MEAN: 1 } When running this command, you'll get all recommended indexes for the given process, along with the related namespace. You can use this information to make a decision on creating new indexes that could improve different operations over your database. To automate this process, we can write a small script to create all recommended indexes. This script will leverage the existing mongocli command to create rolling indexes. Then we’ll rerun our index recommendations and transform that into arguments to the create index command.: mongocli atlas performanceAdvisor suggestedIndexes ls \ --processName atlas-00-01.gcp.mongodb.net:27017 \ | awk 'BEGIN{ORS=" ";}{if (NR!=1){split($2,a,"."); print a[1]; print a[2]; for(i=4;i<=NF-1;++i)printf $i} printf"\n"}' \ | xargs -n3 sh -c 'echo "Creating index db: $1 collection: $2 index: $3"; mongocli atlas clusters indexes create --clusterName slowQueriesDemo --db $1 --collection $2 --key $3' sh This more advanced example shows how to combine or pipe different commands and automate different workflows, in this case saying goodbye to any slow queries. This is just one of the many things you can do with mongocli. We are constantly adding new features, so let us know what you would like to see next . Ready to see for yourself? Try MongoDB Atlas today! Sign up now Get MongoDB CLI! Download

October 19, 2020
Developer

1Data - PeerIslands Data Sync Accelerator

Today’s enterprises are in the midst of digital transformation, but they’re hampered by monolithic, on-prem legacy applications that don’t have the speed, agility, and responsiveness required for digital applications. To make the transition, enterprises are migrating to the cloud. MongoDB has partnered with PeerIslands to develop 1Data, a reference architecture and solution accelerator that helps users with their cloud modernization. This post details the challenges enterprises face with legacy systems and walks through how working with 1Data helps organizations expedite cloud adoption. Modernization Trends As legacy systems become unwieldy, enterprises are breaking them down into microservices and adopting cloud native application development. Monolith-to-microservices migration is complex, but provides value across multiple dimensions. These include: Development velocity Scalability Cost-of-change reduction Ability to build multiple microservice databases concurrently One common approach for teams adopting and building out microservices is to use domain driven design to break down the overall business domain into bounded contexts first. They also often use the Strangler Fig pattern to reduce the overall risk, migrate incrementally, and then decommission the monolith once all required functionality is migrated. While most teams find this approach works well for the application code, it’s particularly challenging to break down monolithic databases into databases that meet the specific needs of each microservice. There are several factors to consider during transition: Duration. How long will the transition to microservices take? Data synchronization. How much and what types of data need to be synchronized between monolith and microservice databases? Data translation in a heterogeneous schema environment. How are the same data elements processed and stored differently? Synchronization cadence. How much data needs syncing, and how often (real-time, nightly, etc.)? Data anti-corruption layer. How do you ensure the integrity of transaction data, and prevent the new data from corrupting the old? Simplifying Migration to the Cloud Created by PeerIslands and MongoDB, 1Data helps enterprises address the challenges detailed above. Migrate and synchronize your data with confidence with 1Data Schema migration tool. Convert legacy DB schema and related components automatically to your target MongoDB instance. Use the GUI-based data mapper to track errors. Real-time data sync pipeline. Sync data between monolith and microservice databases nearly in real time with enterprise grade components. Conditional data sync. Define how to slice the data you’re planning to sync. Data cleansing. Translate data as it’s moved. DSLs for data transformation. Apply domain-specific business rules for the MongoDB documents you want to create from your various aggregated source system tables. This layer also acts as an anti-corruption layer. Data auditing. Independently verify data sync between your source and target systems. Go beyond the database. Synchronize data from APIs, Webhooks & Events. Bidirectional data sync. Replicate key microservice database updates back to the monolithic database as needed. Get Started with Real-Time Data Synchronization With the initial version of 1Data, PeerIslands addresses the core functionality of real-time data sync between source and target systems. Here’s a view of the logical architecture: Source System. The source system can be a relational database like Oracle, where we’ll rely on CDC, or other sources like Events, API, or Webhooks. **Data Capture & Streaming.**Captures the required data from the source system and converts them into data streams using either off-the-shelf DB connectors or custom connectors, depending on the source type. 1Data implements data sharding and throttling, which enable data synchronization at scale, in this phase. Data Transformation. The core of the accelerator, when we convert the source data streams into target MongoDB document schemas. We use LISP-based Domain Specific Language to enable simple, rule-based data transformation, including user-defined rules. Data Sink & Streaming. Captures the data streams that need to be updated into the MongoDB database through stream consumers. The actual update into the target DB is done through sink connectors. Target system. The MDB database used by the microservices. Auditing. Most data that gets migrated is enterprise-critical; 1Data audits the entire data synchronization process for missed data and incorrect updates. Two-way sync. The logical architecture enables data synchronization from the MongoDB database back to the source database. We used MongoDB, Confluent Kafka and Debezium to implement this initial version of 1Data: The technical architecture is cloud agnostic, and can be deployed on-prem as well. We’ll be customizing it for key cloud platforms as well as fleshing out specific architectures to adopt for common data sync scenarios. Conclusion The 1Data solution accelerator lends itself to multiple use cases, from single view to legacy modernization. Please reach out to us for technical details and implementation assistance, and watch this space as we develop the 1Data accelerator further.

October 15, 2020
Developer

Legacy Modernization Leveraging MongoDB and Infosys iDSS

Large enterprises need to refresh their digital strategies to handle the growing demand, emerging competition, and ground dynamics. Infosys and MongoDB started working jointly to help these large organizations reach their full potential by focusing on data modernization. With inputs from MongoDB, Infosys teams have developed a framework called Infosys Data Services Suite ( iDSS ), which allows users to move data from various siloed and legacy relational systems to MongoDB, the most popular document-based modern data platform in the world. By working with iDSS, enterprises can maintain business continuity while seamlessly adapting to the modern demands of analytics and data growth. iDSS is a no-code framework specifically designed and developed to move data from legacy relational systems to MongoDB in an efficient and cost-effective way to ensure predictability, reduced total cost of ownership (TCO), increased productivity, and faster time to market. Because the relational and document model are two opposing database models, you may logically ask How is it possible to map the rows and columns of a relational database to the JSON-like structure of MongoDB? Agreed. Moving data from a relational database management system (RDBMS) to MongoDB is not a trivial task, and that is why experts from Infosys designed iDSS. The tool works like any modern-day extract, transform, and load (ETL) product that can extract data from the RDBMS, transform the data from the relational data model to the document model in a no-code way, and then load the data into MongoDB. Demo iDSS helps you efficiently carry out data modernizations while reducing ETL development effort by as much as 37%. Infosys Data Services Suite (iDSS) Key Features iDSS Features for Migration to MongoDB RDBMS to MONGODB Transform and Load : RDBMS-to-MongoDB migrations with structure changes/transformations/filters RDBMS to JSON Transform and Load: RDBMS-to-JSON file generation with structure changes/ transformations/filters MongoDB to MongoDB: Support for MongoDB-to-MongoDB data migrations, with structure changes/transformations/filters, using MongoDB aggregation framework Other Features Summary Data profiling, business data validation, data quality checks Diversified connectors for enterprise packaged systems such as SAP ECC and S/4HANA, SuccessFactors, Microsoft Dynamics, and Salesforce Enabled for major cloud platforms such as GCP, AWS, and Microsoft Azure Business reviews, approvals/signoff Change data capture (CDC) and incremental data loading Data comparison/ata reconciliation Easy custom development for client requirements Dedicated expert support for tool deployments Learn More Here: http://mongodb.com/modernize

October 1, 2020
Developer

Ready to get Started with MongoDB Atlas?

Get Started