MongoDB Blog

Articles, announcements, news, updates and more

How Accenture’s Data Modernizer Tool Accelerates the Modernizing of Legacy and Mainframe Apps to MongoDB

As companies scale their MongoDB estates and apply them to more and more use cases, opportunities for deeper insights arise—but first come issues around data modeling. The Challenge It’s one thing to start modeling for a green field application where you have the ability to apply modeling patterns . But it’s a different matter entirely to start from an existing relational estate and take into account multiple data sources and their business requirements in an effort to create the ideal MongoDB data model based on explicit schema (table and relations) as well as implicit schema (access patterns and queries). Unfortunately, this is a fairly common obstacle. Perhaps an application has been around for a while and documentation hasn’t kept up with changes. Or maybe a company has lost talent and expertise in some areas of a monolithic app. These are just a couple of examples that demonstrate how time consuming and error-prone this effort can be. The Solution MongoDB and Accenture are excited to announce a modernization tool which enables faster MongoDB adoption by accelerating data modeling. As part of its mainframe offload and cloud modernization initiatives, Accenture’s Java-based data modernizer helps companies arrive at a recommended MongoDB document data model more quickly and efficiently. Using this tool results in a faster operational data layer for offloading mainframes and faster implementation of MongoDB Atlas for migrating applications to the cloud while also transforming them. Accenture and MongoDB are long-standing strategic partners, and this investment is another addition to our joint customers’ toolkits. “This modernizer is the right tool for teams who are focused on changing business requirements as they rethink their applications for the cloud. With on average 50% faster data modeling and 80% data model accuracy straight out of the tool, developers can work on modernizing instead of spending weeks figuring out the right data model to simply meet existing business requirements that have not been explicit.” (Francesco De Cassai) How it Works The modernizer analyzes access logs and relational schema from Oracle and DB2 databases, whether they’re powered by mainframes or not. This allows teams to obtain critical insights before kicking off development activities, thereby better informing the process. Additionally, the modernizer combines the implementation of modeling patterns , development best practices such as document sizing (best, average, worst case), and naming conventions, while also providing confidence levels to predict the accuracy of the model it’s suggested. The tool combines Accenture’s deep expertise and lessons learned from numerous successful projects. Our customers are investing in a new approach to managing data: Data as a Service & Data Decoupling. This strategic initiative focuses on consolidating and organizing enterprise data in one place, most often on the cloud, and then making it available to serve digital projects across the enterprise. MongoDB Atlas unlocks data from legacy systems to drive new applications and digital systems, without the need to disrupt existing backends as companies modernize and migrate to the cloud. With this tool and MongoDB capabilities, MongoDB and Accenture allow organizations to get the most out of their data. Find out more about the MongoDB & Accenture partnership here . Learn more about MongoDB’s Modernization Initiative .

January 21, 2021
Home

Built with MongoDB: Spectro Cloud

Recently named one of the hottest Kubernetes startups , Spectro Cloud has been making waves across the tech ecosystem. An enterprise cloud-native infrastructure founded by three startup veterans, Spectro Cloud makes Kubernetes manageable at scale for enterprises that need manageability and flexibility. Spectro Cloud’s cluster profiles automate cluster deployment and maintenance across the enterprise and help operations prioritize the needs of the applications teams and simplify infrastructure administration. The company has raised $7.5 million in seed funding and has 36 team members. In this edition of #BuiltWithMongoDB, we talk to CTO/Co-founder Saad Malik and Vice President of Product Tina Nolte about Spectro Cloud’s deeply technical team and why they #BuiltWithMongoDB. Siya Raj Purohit: What makes you want to work at Spectro Cloud? Tina Nolte: Most of the team comes from Cisco, where the founders previously worked, so we already had good rapport and were truly friends. We believe that culture is something you build from Day 1, and once it exists, it’s hard to change. For that reason, we have a strict no-jerks policy. How that plays out is that we provide a lot of autonomy to the team and help support goals that individuals have. For example, we encourage everyone on the team to write blog content about things they are interested in to share their knowledge and build their personal brands. It’s something that our engineering managers even nudge junior engineers about: “So you haven’t shared any wisdom with the world recently — why don’t you write a post?” If you—as a developer—experience some kind of issue, or it took you time to understand something, it’s likely that someone else on the planet has experienced that issue too. So we encourage our engineers to help them out, and it’s good for our engineers to have an external presence, too. See Spectro Cloud’s post: Kustomize your way to MongoDB ReplicaSet Finally, we talk about Spectro Cloud as a family. We’re pretty confident that other people have our backs whether it’s personally or professionally, and that kind of connectivity is pretty special. SRP: How did you decide to start using MongoDB? Saad Malik: Our application is not very data heavy in terms of transactions or relations; it’s very document based. Although we are running a SaaS platform, we didn’t want to be in the business of doing backups, managing policy, and storing configurations. We wanted to use a platform that managed these things for us. So we were looking at Amazon’s DocumentDB or MongoDB Atlas. We realized we would have to use an on-premises version of our platform, so obviously if we were to be running an on-premises version of MongoDB, it would make sense to use Atlas. Our team also had experience with MongoDB so it was a clear choice. And it’s been fantastic — we have been very happy with the performance, and so far, it’s been scaling very well for us. SRP: What has it been like to scale with MongoDB? SM: It’s been fantastic — we haven’t had any outages with Atlas. We obviously have our notifications configured so if there are any outages or things going wrong with the MongoDB clusters, we can catch when one of our clusters is misbehaving. That visibility and getting the monitoring upfront is very helpful to us, because we’re able to figure out which of our application issues is causing the problem. When we were getting started, we had a technical advisor from MongoDB provide us with a one-day seminar on best practices for utilizing the platform and how to optimize queries. From then on, the online documentation has been sufficient for us to problem solve and scale. SRP: What does the database infrastructure look like today? SM: On the database side, we have three different environments: dev integration, stage, and production. All three of them run an Atlas version from a database that is completely separate. The stack that sits on top of it is Kubernetes application, all using Golang, DB drivers for MongoDB, and accessing the application on there. SRP: What advice do you have for developers who aspire to someday become CTO? SM: The number one thing I look for is someone who is very curious. When I was an early-career engineer, my mentors would tell me to always be very curious and focused in terms of what you’re doing. Understand how things are working, not just at the library level, but keep on digging down until you understand not just how, but why something works a certain way. If you understand the nuances, you’ll be able to identify true game-changing opportunities. Building something cool with MongoDB? Check out our developer resources , and let us know if you want your startup to be featured in our #BuiltWithMongoDB series.

January 20, 2021
Applied

Add Interactivity to Your Embedded Analytics with Click Events

MongoDB Charts’ data visualizations can now become more interactive, so users and stakeholders can dive deeper into the insights they care more about. That’s possible with a new feature currently in beta with support for most Charts types: click events. A click event in the Charts embedding SDK is simply a notification that a user clicked on a chart. That click could be anything: They might have clicked on a bar in a bar chart, a chart’s legend, or even empty white space within the chart. Web developers can subscribe to these events and create different functionality depending on where the user clicked. Why might you want to enhance your app or embedded analytics workflow with click event data? Click-event data opens up a wide range of possibilities. Here are a couple of examples, inspired by various Charts users who’ve been telling us how they’d like to use click-event data. Open up another chart, based on a user clicking on data within a chart: A logistics company has a bar chart that shows pending orders at an aggregate level per region, and they want to see more detail on pending orders for a specific region. They create a click event handler in their application that opens up a new chart with pending orders per supplier, based on the region selected in the aggregate chart. Filtering the other charts on a dashboard when a series or data point on a single chart is clicked: A retail clothing company has a dashboard with various shopping cart information such as sales, orders processed, and returns, for their portfolio of products. The head of outerwear sales only wants to see data for the “outerwear” category of products, so they click on the “outerwear” series within a bar chart. The rest of the dashboard can adapt so that it shows only information relevant to outerwear. The example below is created from one of our sample data sets. We created two charts in a single app, tied to the sample movie data set that every Atlas user can access. On the left is a stacked bar chart with category level data that includes genre and decade. On the right is a table chart that shows each individual movie within a category. Clicking on a specific category in the bar chart updates the movies shown in the table chart. How can you get started with click events of embedded charts? If you haven’t yet used the embedding SDK for MongoDB Charts, you’ll want to familiarize yourself with the docs , consider watching this video tutorial , and access the SDK via the Charts GitHub repository . Regardless if you’re new to using the SDK or have experience with it, it’s important to note that you will need to use the @mongodb-js/charts-embed-dom@beta tagged version of the SDK to have access to the click events functionality while it’s in beta. There are two examples specifically for click events in the repository: click-events-basic and click-events-filtering . If you just want to explore and test out the functionality, you can play around with them in the following sandboxes using codesandbox.io: Click events basic sandbox Click events filtering sandbox Here’s a snapshot of the data surfaced in a click event that is available for developers to use in their apps. In this example I clicked on the yellow section of the top bar in the Movie Genres chart above. Note how it includes details of the click coordinates, the type and role of the clicked mark, and the corresponding chart data for the mark. { "chartId": "90a8fe84-dd27-4d53-a3fc-0e40392685dd", "event": { "type": "click", "altKey": false, "ctrlKey": false, "shiftKey": false, "metaKey": false, "offsetX": 383, "offsetY": 15, "clientX": 403, "clientY": 99, "pageX": 403, "pageY": 99, "screenX": 756, "screenY": 217 }, "data": { "y": { "label": "Genre", "value": "Drama" }, "x": { "label": "# Movies", "value": 3255 }, "color": { "label": "Decade", "value": "2010 - 2020", "lowerBound": 2010, "upperBound": 2020 } }, "target": { "type": "rect", "role": "mark", "fill": "#F0D175" }, "apiVersion": 1 } Whether you’re an avid user or new to MongoDB Charts, we hope you consider taking advantage of the new click event capability to increase the interactivity of Charts. It’s in beta because there is more functionality still to come. It has yet to be released for a few chart types: geospatial, table, top item, word cloud, and number charts. On that note, we’d love to hear your thoughts through the MongoDB Feedback Engine . If you haven’t tried Charts yet, you can get started for free by signing up for a MongoDB Atlas and deploying a free tier cluster.

January 20, 2021
Developer

Flowhub Relies on MongoDB to Meet Changing Regulations and Scale Its Business

The legal landscape for cannabis in the United States is in constant flux. Each year, new states and other jurisdictions legalize or decriminalize it, and the regulations governing how it can be sold and used change even more frequently. For companies in the industry, this affects not only how they do business, but also how they manage their data. Responding to regulatory changes requires speedy updates to software and a database that makes it easy to change the structure of your data as needed – and that’s not to mention the scaling needs of an industry that’s growing incredibly rapidly. Flowhub makes software for the cannabis industry, and the company is leaping these hurdles every day. I recently spoke with Brad Beeler , Lead Architect at Flowhub, about the company, the challenges of working in an industry with complex regulations, and why Flowhub chose MongoDB to power its business. We also discussed how consulting from MongoDB not only improved performance but also saved the company money, generating a return on investment in less than a month. Eric Holzhauer: First, can you tell us a bit about your company? Brad Beeler: Flowhub provides essential technology for cannabis dispensaries. Founded in 2015, Flowhub pioneered the first Metrc API integration to help dispensaries stay compliant. Today, over 1,000 dispensaries trust Flowhub's point of sale, inventory management, business intelligence, and mobile solutions to process $3B+ cannabis sales annually. Flowhub in use in a dispensary EH: How is Flowhub using MongoDB? BB: Essentially all of our applications – point of sale, inventory management, and more – are built on MongoDB, and we’ve been using MongoDB Atlas from the beginning. When I joined two and a half years ago, our main production cluster was on an M40 cluster tier, and we’ve now scaled up to an M80. The business has expanded a lot, both by onboarding new clients with more locations and by increasing sales in our client base. We’re now at $3 billion of customer transactions a year. As we went through that growth, we started by making optimizations at the database level prior to throwing more resources at it, and then went on to scale the cluster. One great thing about Atlas is that it gave us the metrics we needed to understand our growth. After we’d made some optimizations, we could look at CPU and memory utilization, check that there wasn’t a way to further improve query execution with indexes, and then know it was time to scale. It’s really important for usability that we keep latency low and that the application UI is responsive, and scaling in Atlas helps us ensure that performance. We also deploy an extra analytics node in Atlas, which is where we run queries for reporting. Most of our application data access is relatively straightforward CRUD, but we run aggregation pipelines to create reports: day-over-day sales, running financials, and so forth. Those reports are extra intensive at month-end or year-end, when our customers are looking back at the prior period to understand their business trends. It’s very useful to be able to isolate that workload from our core application queries. I’ll also say that MongoDB Compass has been an amazing tool for creating aggregation pipelines. EH: Can you tell us some more about what makes your industry unique, and why MongoDB is a good fit? BB: The regulatory landscape is a major factor. In the U.S., there’s a patchwork of regulation, and it continues to evolve – you may have seen that several new states legalized cannabis in the 2020 election cycle. States are still exploring how they want to regulate this industry, and as they discover what works and what doesn’t, they change the regulations fairly frequently. We have to adapt to those changing variables, and MongoDB facilitates that. We can change the application layer to account for new regulations, and there’s minimal maintenance to change the database layer to match. That makes our development cycles faster and speeds up our time to market. MongoDB’s flexibility is great for moving quickly to meet new data requirements. As a few concrete examples: The state of Oregon wanted to make sure that consumers knew exactly how much cannabis they were purchasing, regardless of format. Since some dispensaries sell prerolled products, they need to record the weight of the paper associated with those products. So now that’s a new data point we have to collect. We updated the application UI to add a form field where the dispensary can input the paper weight, and that data flows right into the database. Dispensaries are also issuing not only purchase receipts, but exit labels like what you’d find on a prescription from a pharmacy. And depending on the state, that exit label might include potency level, percentage of cannabinoids, what batch and package the cannabis came from, and so on. All of that is data we need to be storing, and potentially calculating or reformatting according to specific state requirements. Everything in our industry is tracked from seed to sale. Plants get barcodes very early on, and that identity is tracked all the way through different growth cycles and into packaging. So if there’s a recall, for example, it’s possible to identify all of the products from a specific plant, or plants from a certain origin. Tracking that data and integrating with systems up the supply chain is critical for us. That data is all tracked in a regulatory system. We integrate with Metrc , which is the largest cannabis tracking system in the country. So our systems feed back into Metrc, and we automate the process of reporting all the required information. That’s much easier than a manual alternative – for example, uploading spreadsheets to Metrc, which dispensaries would otherwise need to do. We also pull information down from Metrc. When a store receives a shipment, it will import the package records into our system, and we’ll store them as inventory and get the relevant information from the Metrc API. Flowhub user interface EH: What impact has MongoDB had on your business? BB: MongoDB definitely has improved our time to market in a couple of ways. I mentioned the differences of regulation and data requirements across states; MongoDB’s flexibility makes it easier to launch into a new state and collect the right data or make required calculations based on data. We also improve time to market because of developer productivity. Since we’re a JavaScript shop, JSON is second nature to our developers, and MongoDB’s document structure is very easy to understand and work with. EH: What version of MongoDB are you using? BB: We started out on 3.4, and have since upgraded to MongoDB 4.0. We’re preparing to upgrade to 4.2 to take advantage of some of the additional features in the database and in MongoDB Cloud. One thing we’re excited about is Atlas Search : by running a true search engine close to our data, we think we can get some pretty big performance improvements. Most of our infrastructure is built on Node.js, and we’re using the Node.js driver . A great thing about MongoDB’s replication and the driver is that if there’s a failover and a new primary is elected, the driver keeps chugging, staying connected to the replica sets and retrying reads and writes if needed. That’s prevented any downtime or connectivity issues for us. EH: How are you securing MongoDB? BB: Security is very important to us, and we rely on Atlas’s security controls to protect data. We’ve set up access controls so that our developers can work easily in the development environment, but there are only a few people who can access data in the production environment. IP access lists let us control who and what can access the database, including a few third-party applications that are integrated into Flowhub. We’re looking into implementing VPC Peering for our application connections, which currently go through the IP access list. We’re also interested in Client-Side Field-Level Encryption . We already limit the amount of personally identifiable information (PII) we collect and store, and we’re very careful about securing the PII we do need to store. Client-Side Field-Level Encryption would let us encrypt that at the client level, before it ever reaches the database. EH: You're running on Atlas, so what underlying cloud provider do your use? BB: We’re running everything on Google Cloud. We use Atlas on Google Cloud infrastructure, and our app servers are running in Google Kubernetes Engine. We also use several other Google services. We rely pretty heavily on Google Cloud Pub/Sub as a messaging backbone for an event-driven architecture. Our core applications initially were built with a fairly monolithic architecture, because it was the easiest approach to get going quickly. As we’ve grown, we’re moving more toward microservices. We’ve connected Pub/Sub to MongoDB Atlas, and we’re turning data operations into published events. Microservices can then subscribe to event topics and use the events to take action and maintain or audit local data stores. Our data science team uses Google BigQuery as the backend to most of our own analytics tooling. For most uses, we migrate data from MongoDB Atlas to BigQuery via in-house ETL processes, but for more real-time needs we’re using Google Dataflow to connect to MongoDB’s oplog and stream data into BigQuery. EH: As you grow your business and scale your MongoDB usage, what's been the most important resource for you? BB: MongoDB’s Flex Consulting has been great for optimizing performance and scaling efficiently. Flowhub has been around for a number of years, and as we’ve grown, our database has grown and evolved. Some of the schema, query, and index decisions that we had made years ago weren’t optimized for what we’re doing now, but we hadn’t revisited them comprehensively. Especially when we were scaling our cluster, we knew that we could make more improvements. Our MongoDB Consulting Engineer investigated our data structure and how we were accessing data, performance, what indexes we had, and so on. We even got into the internals of the WiredTiger storage engine and what optimizations we could make there. We learned a ton about MongoDB, and the Consulting Engineer also introduced us to some tools so we could diagnose performance issues ourselves. Based on our Consulting Engineer’s recommendations, we changed the structure of how we stored some data and reworked certain queries to improve performance. We also cleaned up a bunch of unnecessary indexes. We had created a number of indexes over the years for different query patterns, and our Consulting Engineer was able to identify which ones could be removed wholesale, and which indexes could be replaced with a single new one to cover different query patterns. We made some optimizations in Atlas as well, moving to a Low CPU instance based on the shape of our workload and changing to a more efficient backup option. With the optimizations recommended in our consulting engagement, we were able to reduce our spend by more than 35%. MongoDB consulting paid for itself in less than a month, which was incredible. I had to develop a business case internally for investing in consulting, and this level of savings made it an easy sell. The knowledge we picked up during our consulting engagement was invaluable. That’s something we’ll carry forward and that will continue to provide benefits. We’re much better at our indexing strategy, for example. Say you’re introducing a new type of query and thinking about adding an index: now we know what questions to ask. How often is this going to be run? Could you change the query to use an existing index, or change an existing index to cover this query? If we decide we need a new index, should we deprecate an old one? With the optimizations recommended in our consulting engagement, we were able to reduce our spend by more than 35%. MongoDB consulting paid for itself in less than a month, which was incredible. Brad Beeler, Lead Architect, Flowhub EH: What advice would you give to someone who's considering MongoDB for their next project? BB: Take the time upfront to understand your data and how it’s going to be used. That’ll give you a good head start for structuring the data in MongoDB, designing queries, and implementing indexes. Obviously, Flex Consulting was very helpful for us on this front, so give that a look.

January 19, 2021
Applied

Imposter Syndrome and Public Speaking: 5 Tips for a Successful Tech Talk

I've done hundreds of presentations at large and small events. I’ve delivered keynotes in front of a thousand people. You probably think I enter the stage each time with full confidence that I’ll give a solid and engaging presentation. You’re wrong. I’m an imposter when it comes to public speaking, and I'm not the only one. Every time I accept a speaking gig I think there must be people in the audience with much more experience and who can deliver my talk more eloquently. I feel like a fraud, and I know lots of other speakers that have the same feeling. That’s imposter syndrome. Studies have shown that actually 70% of us have experienced this kind of anxiety at some stage in our lives. But none of this stops me from speaking, and it shouldn’t stop you. You’ll get so much out of being brave enough to tell your story in front of a small or large audience: people open up and share their experiences with you, you end up helping others by inspiring them with your story, and you meet a lot of interesting people. Here are five tips for how to overcome your imposter syndrome and deliver a successful tech talk with confidence. 1. Be Better It helps to watch a lot of talks from conferences and note down what you like and don’t like about them. When preparing your talk, have that list present and avoid including things you dislike. This will help you develop better practices in delivery, story telling, slide design, or whatever area you have identified as important. When I put together my first presentation for an event in Munich, I watched some talks from the previous year. I was bored when speakers started with five minutes of talking about themselves and what they’ve done, so I decided to always begin my talks with telling a personal relatable story. It helped me to have the impression that my talks had a much better intro than all others (which might not be true, but I felt that way). 2. Be Authentic If you pretend to be someone you’re not, you’ll definitely suffer from the imposter syndrome. Be honest with the audience. Share your experience about a specific topic. Audiences love to hear real stories and want to learn from your successes and mistakes. As I mentioned before, I always start talks by sharing a true and relatable story. That serves a few purposes: The talk has an interesting start, and I have people’s attention from the first minute. When I share my own struggles and successes, it helps make the audience feel that I’m one of them. I don’t have to pretend to be someone that I’m not. It's my story and my experience. Being authentic helps you feel less like a fraud. 3. Grow Confidence Get feedback at every stage of your talk development, from writing your conference submission , putting together your talk , and crafting your slides to practicing and refining. Ask coworkers, friends, or other speakers to give you feedback. I know this is very hard for us imposters. We’re afraid to get caught being a fraud by people we care about. On the other hand, it’s a safe environment for receiving candid and honest feedback. I always test my new talks in front of a couple of coworkers. They know me and don’t want to hurt my feelings, but they also do not hold back if they don’t like parts of my presentation. It always makes my talks much better and gives me confidence that if they like it, the conference audience might like it too. If you’re just starting with public speaking, it might be an option to hold your presentation first at a smaller community gathering. These events normally are held in a much more relaxed atmosphere and have a smaller audience. It’ll boost your confidence if those user groups like your presentation, and you’ll feel more prepared for larger stages. Start small, grow confidence, and then go big (if you want). 4. The Audience is Your Friend Every speaker I know is nervous before the talk, no matter how many presentations they’ve given. Always remember: the audience wants you to succeed. They came to learn something. They don’t want you to fail. The audience is full of friends that want to support you. Bring some friends. It’s always good to see a friendly, familiar face in the audience (even if your talk is being delivered virtually). Talk to your friend right before you go on stage so your mind is not focused too much on the challenge ahead of you. Find friendly faces. Look around in the audience and find some friendly faces. You can use those attendees later in the talk to get visual confirmation that you’re doing great. If you’re giving your talk in-person, talk to people at the entrance or in the first row. Have a casual conversation, or just say hi. Connecting to the audience helps you to not think of them as strangers that are going to raise critical questions. 5. Use the Imposter Syndrome to Your Advantage Imposter syndrome gives you the feeling of being a fraud talking about a specific topic. You think there must be experts at this conference that know much more about the topic than you do. Use that feeling as a catalyst to really double down on learning more about the topic. I once submitted a talk about software development automation to an event in Norway and got accepted. I knew a few concepts but wasn’t an expert. The next day I went into deep-learning mode, read tons of articles on the internet, and scheduled a dozen interviews with people from various organizations who were responsible for running development tools, were tech leads, or just were innovative developers. I dug up so many interesting stories. I was scared as hell when giving the talk for the first time because I still did not feel like an expert on the topic, but people loved the stories I uncovered and as a result, I got invited to give the talk as a keynote at another event in Italy. Investing time in your research helps you to build confidence, so use the imposter syndrome as an accelerator. How can you start putting together your tech talk abstract and getting ready to submit it to your next conference? At MongoDB University we’ve put together a free course on this topic. At MongoDB, we believe that everyone has an interesting story to share, and we want to help you bring it to life. MongoDB.live 2021, MongoDB’s biggest conference of the year, is back, and we’re currently looking for speakers to inspire and equip attendees with new technologies, ideas, and solutions. Submit your talk today .

January 19, 2021
Home

Modernize data between siloed data warehouses with Infosys Data Mesh and MongoDB

The Data Challenge in Digital Transformation Enterprises that embark on a Digital transformation often face significant challenges with accessing data in a timely manner—an issue that can quickly impede customer satisfaction. To deliver the best digital experience for customers, companies must create the right engagement strategy. This requires all relevant data available in the enterprise be readily accessible. For example, when a customer contacts an insurance company, it is important that the company has a comprehensive understanding of the customer’s background as well as any prior interactions, so they can orchestrate the best possible experience. Data is available in both BI (Business Intelligence) systems, like Enterprise Data Warehouses, and OI (Operational Intelligence) systems, like policy and claim systems. There is a need to bring these BI and OI systems together to avoid any disruption to the digital functions that may delay synchronization. Data removed from an operational system loses context. Re-establishing this domain context and providing persona-based access to the data requires domain-oriented, decentralized data ownership, as well as architecture. Ultimately, organizations seek to use data as a key to fueling the products and services they provide their customers. This data should minimize the cost of customer research—but the data needs to be trusted and high quality. Companies need access to these siloed sources of data in a seamless self-service approach across various product life cycles. The Challenge of Centralized Data Historically, businesses have handled large amounts of data from various sources by ingesting it all into a centralized database (data warehouse, data lake, or data lake on cloud). They would then feed insight drivers, like reporting tools and dashboards as well as online transaction processing applications, from that central repository. The challenge with this approach is the broken link between analytical systems and transactional systems that impedes the digital experience. Centralized systems, like data warehouses, introduce latency and fail to meet the real time response and performance levels needed to build next-generation digital experiences. What is Infosys Data Mesh? Data Mesh helps organizations bridge the chasm between analytics and application development teams within large enterprises. Data Mesh is an architecture pattern that takes a new approach to domain-driven distributed architecture and the decentralization of data. Its basic philosophy is to encapsulate the data, its relationships, context, and access functionality into a data product with guaranteed quality, trust, and ease of use for business consumption. Data Mesh is best suited for low-latency access to data assets used in digital transformations that are intended to improve experience through rich insights. With its richer domain flavor — distributed ownership, manageability, and low latency access — Data Mesh is best positioned as a bridge between transactional (consuming applications) and analytical systems. This diagram depicts the high-level solution view of Data Mesh: Data Mesh Key Design Principles Domain-first approach. Data as a product. Data Mesh products share the following attributes which maximize usability and minimize friction: Self-described: metadata is precise and accurate Discoverable and addressable: products are uniquely identifiable and easy to find Secure and well-governed: only those who are granted access have it Trustworthy: proper data quality conrtols are applie, SLA/SLOs are maintainted Open standard and interoperable: data formats — XBRL, JSON Build new products easily. Any cross-functional team can build a new, enterprise-level product in an existing domain and/or fro existing products Simplified access for multiple technology stacks. Polygot data and ports, cloud and non-cloud. Common infrastructure and services for all data pipelines and catalogs. Platform Requirements for a Data Mesh To build a data mesh, companies need a database platform that can create domain-driven data products that meet various enterprise needs. This includes: Flexible data structures — to accomodate new behaviors An API-driven construct — to access current data products and build new domain-data ones Support for high-performance query on large-scale data structures A shared, scalable infrastructure Why MongoDB is the Optimal Platform for Infosys Data Mesh MongoDB is the best platform for realizing the Infosys Data Mesh architecture and powering analytics-driven enterprises because it provides: A flexible document model and a poly-cloud infrastructure availability so teams can easily modify and enrich flat or hierarchical data models MongoDB Realm Webhooks to create service APIs which connect data across products and enable consumption needs based on business context A scalable, shared infrastructure and support for high-performance querying of large scale data Service APIs for constructing Infosys Data Mesh Two use cases: table, th, td { border: 1px solid black; border-collapse: collapse; } Case 1: Case 2: A wealth management firm offers a variety of products to its customers — things like checking and savings accounts, trading, credit and debit cards, insurance, and investment vehicles. Challenges: Each product is serviced by a different system and technology infrastructure Internal consumers of this data have different needs: product managers analyze product performance, wealth managers and financial advisors rely on customer-centric analytics, and financial control teams track the firm’s revenue performance Solution: Using the Infosys Data Mesh model, the firm’s data owners create domain-data products categorized by customer and product, and then curate and publish them through a technology-agnostic, API-driven service layer. Consumers can then use this service layer to build the data products they need to carry out their business functions. The Risk and Finance unit of a large global bank has multiple regional data lakes catering to each region’s management information system and analytical needs. This poses multiple challenges for creating global data products: Challeges: Technology varies across regions ETL can becomes less advantageous depending on circumstance Regulations govern cross-regional data transfer policies Solution: To address these challenges, the bank creates an architecture of regional data hubs for region-specific products and, as with Case 1, makes those products available to authorized consumers through a technology-agnostic, API-driven service layer. Next, it implements an enterprise data catalog with an easy-to-use search interface on top of the API layer. The catalog’s query engine executes cross-hub queries, creating a self-service model for users to seamlessly discover and consume data products and to align newer ones with their evolving business needs. Enterprise security platform integration ensures that all regulatory and compliance requirements are fully met. How Businesses Overall Can Benefit Data and Insights become pervasive and consumable across applications and personas Speed-to-insights (including real time) enable newer digital experiences and better engagement leading to superior business results Self-service through trusted data products is enabled Infosysy DNA Assets on MongoDB Accelerates the Creation of Industry-Specific Domain Data Products table, th, td { border: 1px solid black; border-collapse: collapse; } Infosys Genome Infosys Data Prep Infosys Marketplace Creates the foundation for Data Mesh by unifying semantics across industries Guides consumers through the product creation process with a scalable data preparation framework Enables discovery and consumption of domain-data products via an enterprise data catalog Download our Modernization Guide for information about which applications are best suited for modernization and tools to get started.

January 14, 2021
Developer

Legacy Modernization with MongoDB and Confluent

In many organizations, crucial enterprise data is locked in dozens or hundreds of silos that may be, controlled by different teams, and stuck in systems that aren’t able to serve new workloads or access patterns. This is a blocker for innovation and insight ultimately hampering the business. For example, imagine building a new mobile app for your customers that enables them to view their account data in a single view. Designing the app could require months of time to simply navigate the internal processes necessary to gain access to the legacy systems and even more time to figure out how to integrate them. An Operational Data Layer, or ODL, can offer a “best of both worlds” approach, providing the benefits of modernization without the risk of a full rip and replace. Legacy systems are left intact – at least at first – meaning that existing applications can continue to work as usual without interruption. New or improved data consumers will access the ODL rather than the legacy data stores, protecting those stores from new workloads that may strain their capacity and expose single points of failure. At the same time, building an ODL offers a chance to redesign the application’s data model, allowing for new development and features that aren’t possible with the rigid tabular structure of existing relational systems. With an ODL, it’s possible to combine data from multiple legacy sources into a single repository where new applications, such as a customer single view or artificial intelligence processes, can access the entire corpus of data. Existing workloads can gradually shift to the ODL, delivering value at each step. Eventually, the ODL can be promoted to a system of record and legacy systems can be decommissioned. Read our blog covering DaaS with MongoDB and Confluent to learn more. There’s also a push today for applications and databases to be entirely cloud-based, but the reality is that current business applications are often too complex to be migrated easily or completely. Instead, many businesses are opting to move application data between on-premises and cloud deployments in an effort to leverage the full advantage of public cloud computing without having to undertake a complete, massive data lift-and-shift. Confluent can be used for both one-time and real-time data synchronization between legacy data sources and modern data platforms like MongoDB, whose fully managed global cloud database service, MongoDB Atlas , is supported across AWS, Google Cloud, and Azure. Confluent Platform can be self-managed in your own data center while Confluent Cloud can be used on the public clouds. Whether leaving your application on-premise is a personal choice or a corporate mandate, there are many good reasons to integrate with MongoDB Atlas. Bring your data closer to your users in more than 70 regions with Atlas’s global clusters Address your most intense workloads with one-click, automated sharding for scale out and zero-downtime scale up Quickly provision TBs of database storage, all on high performance SSDs with dedicated I/O bandwidth Natively query and analyze data across AWS S3 and MongoDB Atlas with MongoDB Atlas Data Lake Perform full-text search queries with MongoDB Atlas Search Build native mobile applications that seamlessly synchronize data with MongoDB Realm Create powerful visualizations and dashboards of your MongoDB data with MongoDB Charts Off-load older data to cost effective storage with MongoDB Atlas Online Archive In this video we will show one time migration and Real time continuous data synchronization from a Relational System to MongoDB Atlas using Confluent Platform and the MongoDB Connector for Apache Kafka . Also we will be talking about different ways to store and consume the data within MongoDB Atlas. Git repository for the demo is here . Learn more about the MongoDB and Confluent partnership here and download the joint Reference Architecture here . Click here to learn more about modernizing to MongoDB.

January 7, 2021
Developer

Built with MongoDB: Coursedog

Nicholas Diao and Justin Wenig had just joined the undergraduate program at Columbia University and were excited to take their first computer science class. They registered and, to their delight, were accepted into the course. When they showed up to the first class, however, they were greeted with distressing news: the class was double-booked. As they quickly discovered, their situation was hardly unique. Universities, such as their own, often lack the software to manage the complexities of class scheduling. They decided to fix that. What started as an undergraduate project to solve a local problem turned into Coursedog , a Y Combinator backed curriculum success platform for higher education. Coursedog works with more than 70 universities to modernize the way they propose, schedule, and publish their classes to students. Coursedog has raised $4.2M in funding and been building with MongoDB since Day 1. Both founders were recently named to Forbes 30 under 30. In this edition of #BuiltWithMongoDB, we talk to Nicholas about being a student founder, building prototypes to find product-market fit, and growing with the MongoDB platform. Siya Raj Purohit: Coursedog started while you were still in college. Let’s talk about how you came across the problem your platform is built to address. Nicholas Diao: My co-founder Justin and I both wanted to be in a specific CS class. On the first day of class, we had our textbooks, our coffee, and our snacks, and were ready to go - and then we realized the professor wasn’t there. It turns out the professor had been double-booked. As aspiring CS majors, we thought “how can this be a problem of the 21st century?!” We assumed there was some automated system that ensured smooth scheduling for university classes. But it turned out that what we thought would be an automated system was a couple of overworked administrators with Microsoft Excel spreadsheets in a dark back room where they had to build the schedule themselves. And, of course, that process is error-prone and not the best use of time for university administrators. Justin and I spoke to around 400 to 500 university administrators to better understand the scheduling process, which is far more complex than it appears and has an immense impact on students and their ability to get the courses they need to learn and graduate. We realized this was a problem that was mostly ignored. Columbia Law School joined us as a design partner, and we started building a simple prototype to address this problem and make the system better for all universities. SRP: What was your initial prototype like? ND: We used simple HTML, CSS, JavaScript, and Node.JS server with a MongoDB database. Part of the reason we chose MongoDB is its ability to be really flexible, because we were learning and iterating day after day. A few months later, we ended up signing our first official school contract at Brigham Young University based on good references from Columbia Law School. In the winter of 2019, we entered Y Combinator. After we graduated from YC, we raised $4M in Series A funding, and now we’re working with more than 70 institutions and have released three additional products focused on curriculum management, event management, and catalog management. We have a team of more than 40 people across three countries. SRP: What does your tech stack consist of? ND: It's the MEVN stack: MongoDB, Express.js, Vue.js, and Node.js. We use AWS for architecture. SRP: When you started building Coursedog, you were still a CS student. How did you decide to choose MongoDB? ND: There are a couple of reasons why we started building with MongoDB. Early on, we wanted to quickly build demos that our customers could provide immediate feedback on. We knew we were tackling a complex problem and we didn’t know what our ultimate data structure would be, so we wanted a database that could be as flexible and iterative as possible and that ended up being a NoSQL database that was cloud-hosted. MongoDB came to the top of that list. It was a great decision, because we made many modifications to our data model and MongoDB managed the complexity of our solution while being easier than any other database. What got our team to really fall in love with MongoDB was the power of what we were able to do with it . There are relationships between different data objects (for example, for an economics class, managing all the components you need to know: room size, class size, department, and so on). We’re able to do very powerful joins in the SQL database and complex filters. We can build out everything we need in an iterative fashion, and MongoDB has all the functions to enable us to build more complicated features over time. We once spoke with a MongoDB technical advisor too: aggregation pipelines were new to us, and our conversation gave us great footing for getting started. From there on, the MongoDB documentation has been detailed enough to help us navigate scaling challenges. SRP: Now you’re used by more than 70 universities. How has your experience been scaling up with MongoDB? ND: We’ve found it really easy to scale because of the seamlessness and flexibility of the product and its easy communication. We have a clear sense of how much data we’re using and what the performance metrics are, and we get timely notifications. We really appreciate that if you hit 80% of a specific metric, MongoDB will send you a notification. This has been hugely helpful to our DevOps and infrastructure folks for monitoring. We’ve actually taken some cues from that: if you have 80% of a room booked, we send a notification to our users. As we’ve scaled and worked with an increasing amount of data (typically there are between 150 and 200 data types for each school), we have found that MongoDB has the flexibility and customizations we required. This is sometimes a contrast to AWS, our architectural tool. Doing the same things in AWS is much harder. For example, to change notifications (or even set them up), you have to read AWS’s incomprehensible online documentation, and then there are about 30 different places you can go to make that change on the site. In contrast, MongoDB makes it so easy to manage the back-end so we can focus on building the business. SRP: What advice would you have for college students who aspire to build their own company and move into the CTO role for that company? ND: The most important skill to pick up during college is collaboration. The way schools evaluate students is all content-focused (test, papers, problem sets) but what really matters in your career is your ability to collaborate with other people. I wouldn’t have gotten anywhere if Justin and I were not able to build a strong partnership and work with other smart, hard-working people to get Coursedog off the ground. For college students, I would say that in the long run, it doesn’t matter what grade you get on that problem set or lab; it matters if you’re able to work well with the people around you. My second piece of advice is when building solutions, start small, start local, and start trying to solve problems for the people around you - as we did with Coursedog. The best companies come from solving personal problems and then building it out from there. Building something cool with MongoDB? Check out our developer resources , and let us know if you want your startup to be featured in our #BuiltWithMongoDB series.

January 6, 2021
Applied

Finding Inspiration and Motivation at MongoDB University

For many people, across the globe, 2020 was a strange and challenging year. The new year has brought the hope of healthier and more prosperous times ahead, but inspiration to stay positive can still be tough to find. For MongoDB Certified Developer Kirk-Patrick Brown, the past months presented obstacles, but with perseverance he also experienced growth and even found ways to give back to his local community using what he learned at MongoDB University . Kirk-Patrick sat down with us virtually, from his home in Jamaica, to talk about his passion for MongoDB, getting certified through MongoDB University in the middle of the pandemic, and staying motivated. Can you tell us about yourself and your approach to software development? I’m Kirk-Patrick Brown, a senior software developer at Smart Mobile Solutions Jamaica. I consider myself an artist. I have a history in martial arts and poetry. I medaled in the Jamaica Taekwondo Championships and received the certificate of merit in a creative writing competition hosted by the Jamaica Cultural Development Commission. It was only natural to bring those artistic traits when moving into software development. For me, software development is also an artistic pursuit. It gives me a canvas to create and bring ideas to life, which in turn brings business value. When did you begin building with MongoDB? I had my first hands on-experience with MongoDB in 2018. I realized it was one of those rare gems that you see, and you're immediately curious about how it actually works, because it’s not like what you’re used to. In Jamaica there are a lot of organizations that run on some form of relational database. But once I learned about MongoDB and NoSQL I became a self-motivated evangelist for MongoDB. I understand that organizations may have used relational databases in the past, and that is understandable because there is a longer history and at one time that was the main type of database for your typical workload, but things have changed drastically. In this era there is more demand for data and all different types of unstructured data. With the advent of big data, systems that were designed years ago may not be able to provide optimal storage and performance. MongoDB is a better alternative for such use cases and enables built-in features such as auto-sharding to horizontally scale and aid in the efficient storage and retrieval of big data. MongoDB keeps being so innovative. The other day I was preparing for a multicloud accreditation with Aviatrix, and it was so funny--at the very same time, MongoDB came out with multicloud clusters. It was just beautiful. You don’t want to get locked into one cloud provider for your deployments. Even though these cloud providers offer availability zones for increased fault tolerance, things can still happen. Becoming multi-cloud allows you to become more resilient to disaster. Being in multiple clouds also lets you bring some of your replica sets closer geographically to your customers. By leveraging regional presences across multiple clouds, you can reduce in-network latency, and increase your ability to fulfill queries faster. That’s one of the main features of MongoDB replication--the ability to configure a member to be of higher priority than others, which could be driven by the location in which most of your queries originate. Multi-cloud clusters enable high availability and performance, and I think it was amazing of MongoDB to create such a feature. You call yourself a “self motivated evangelist” for MongoDB. We’re flattered! What has your experience been? I’m actively trying to get organizations to appreciate NoSQL. Recently I presented to a group of developers in the agile space. I spoke to them about replication, sharding, indexes, performance, and how MongoDB ties into advanced features of security in terms of authentication. I’m primarily pushing for developers and organizations to appreciate the Atlas offering from MongoDB. Right out of the box you can instantly have a deployed database out there in Atlas--with the click of a button, pretty much. You can get up and running immediately because MongoDB is a cloud-first database. Plus there's always customer support, even at the free tiers. You don’t feel alone with your database when you’re using MongoDB Atlas. There has been some resistance, because NoSQL requires a bit of a mental shift to understand what it can provide. But we live in a world where things continually change. If you are not open to adapting I don’t even have to say what’s going to happen, you know? You became MongoDB Certified through MongoDB University in the middle of the pandemic. Can you tell us about that experience? Even before the pandemic started I was studying courses at MongoDB University, and traveling 100 kilometers to go to work every week, while also caring for my family and three year-old son back at home. There were some delays, but I was able to become MongoDB-certified in July 2020. Becoming MongoDB-certified has impacted me in positive ways. I’ve met people I did not know before. It has also given me a level of confidence as it relates to building a database that is highly available, scalable, and provides good data reads via the different types of indexes and indexing techniques provided by MongoDB. I can create the database, perform and optimize CRUD operations, apply security and performance activities alongside a highly available and scalable cluster, all thanks to the knowledge provided by MongoDB University. The courses at MongoDB University covered those aspects very well. There is enough theory but also a great amount of practical application in the courses, so you leave with working knowledge that you can immediately use. What is the project you worked on during the pandemic that you’re most proud of? One of the things I’ve worked on intensely during the pandemic is helping to develop a video verification application for a local company and building out most of the backend functionality. For that project, there was a great deal of research needed into the technological tools and implementation to support recording verification videos of customers. I felt like it was my contribution to society at a time when it was dangerous for people to come into that local business. If I can develop something that allows even one person not to need to come into that physical location, that could be the difference between someone contracting the virus or not. A virus that has taken many lives and disrupted a lot of families this year. What advice do you have for other developers who are struggling right now with motivation to advance themselves and their careers? Don’t ever give up. In anything that you do. There is nothing that you’ll do that’s going to be both good and easy. Being a developer, you experience different problems that you have to solve but you have to keep moving forward. I don’t believe in failure, because in anything you do, there is always a win. You have your experiences and those experiences can guide your decision making. It’s just like machine learning. Machines need a lot of data and you can’t give the machine all positive data. It needs some negative data for it to become a good training model. You need bad experiences as well as good ones. If we had all good experiences our brains would not have the training models to make those correct decisions when we need them. Each day I make one definite step or positive decision. And that may be as simple as going onto the MongoDB University site and saying “I’m going to complete this one course.” You just have to keep going at it. You plan for a lot of things in life, but things most of the time don’t happen when you want them to. There's going to be some delay or something. But you can’t give up. Because if you give up then everything is lost. As long as there is time and there is life then there is opportunity to keep doing this thing. And it may take a little bit to get there but eventually you will. But if you give up, you definitely won’t!

January 6, 2021
Developer

Ready to get Started with MongoDB Atlas?

Get Started