In a recent webinar, MongoDB Technical Services Engineer Daniel Coupal presented on how you can use MMS for performance tuning and monitoring. He explains which metrics to examine when optimizing your MongoDB deployment. Daniel and the MongoDB Technical Services team work with thousands of MongoDB customers and community users each year, and drawing from those experiences Daniel also provided several real-world examples of diagnosing and debugging performance issues with the MongoDB Management Service. You can watch the full video below.
Monitoring is completely free with MMS. Why not give it a try? Create your free account today.
The Leaf in the Wild: Wearable Sensors Connecting “Man’s Best Friend” - Tractive & MongoDB
Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects. I had the opportunity to sit down with Michael Lettner, CTO of Hardware & Services and Bernhard Wolkerstorfer, Head of Web & Services at Tractive, to discuss how they use MongoDB at their Internet of Things startup. Tell us a little bit about your company. What are you trying to accomplish? How do you see yourself growing in the next few years? Tractive is a cool 18-month old startup designed for pet owners. We extend the concept of the “quantified self” to the quantified pet, enabling owners to monitor their beloved companions through wearable sensor technology. Our first service was the GPS Pet Tracking device that attaches to the pet’s collar and enables the owner to receive real time location-based tracking on their iOS or Android device. Users can also define a safe zone that acts as a virtual fence - whenever the pet leaves the safe zone, a notification is sent to the owner’s device. We have extended our products to include Tractive Motion that tracks a pet’s activity. Owners can compare how much exercise their pet is getting to other owners with the same breed. The Peterest image gallery enables owners to share images and activity with other members of their social network, and Pet Manager can be used to record veterinary appointments, allergies, vaccination schedules and more. Tractive is currently available in over 70 countries, mainly across Europe and the Middle East, and is now rapidly extending worldwide with our first customers recently added in the USA, Asia, Australia and New Zealand. Please describe your application using MongoDB. MongoDB is our primary database - we use it to store all of the data we rely on to deliver our services - from sensor and geospatial data, to activity data, to user data and social sharing. Image data is stored in AWS S3 with its metadata managed by MongoDB. We also use MongoDB to log all data from our infrastructure, ensuring our service is always available. Why did you select MongoDB for Tractive? Did you consider other alternatives? We initially came from a background of using relational databases, but we believed that these were not appropriate tools for managing the diversity of sensor data we would rely on for the Tractive services. In addition, we knew we would be rapidly evolving the functionality of our apps and were concerned the rigidity of the relational data model would constrain our creativity and time to market. We knew the way forward was a non-relational database, and many would give us the flexible data model our app needed. Beyond a dynamic schema, we had additional criteria that guided our ultimate decision How easily would the database allow us to store and query geospatial data? How well could the database handle time-series and event-based data? What sort of query flexibility did the database offer to support analytics against the data? How easily and quickly could the database scale as our customer base and data volumes grew? Was the database open source? There are a multitude of key-value, wide column and document databases we could have chosen. There were many that could ingest time-series data quickly, but they lacked the ability to run rich queries against the data in place – instead forcing us to replicate the data to external systems. Only MongoDB met all of key criteria – easy to develop against, simple to run in operations and without throwing away the type of query functionality we had come to expect from relational databases. Please describe your MongoDB deployment We run our MongoDB cluster across three shards with each shard configured as a three-node replica set. This architecture gives us the resilience we need to deliver always-on availability, and enables us to rapidly add shards as our service continues to grow. The cluster is deployed in a colocation facility with an external service provider. Our backend is primarily based on Ruby and currently running MongoDB 2.2 in production. We are planning a move to MongoDB 2.6 to take advantage of some specific new capabilities: Aggregation framework improvements such as cursors Geospatial enhancements Index intersection with the ability to use more than one index to resolve a query Can you share best practices you learned while scaling MongoDB? For best results, shard before you have to. Get a thorough understanding of your data structures and query patterns. This will help you select a shard key that best suits your applications. If you follow these simple rules, sharding in MongoDB is really simple. It’s automatic and transparent to the developer. Scaling is of course much more than simply throwing hardware at the database cluster. So we got a lot of benefits from MongoDB tooling in optimizing our queries. During development, we used the MongoDB explain operator to ensure good index coverage. We also use the MongoDB Database Profiler to log all slow queries for further analysis and optimization. For our analytics queries, we initially used MongoDB’s inbuilt MapReduce, but have since moved to the aggregation framework , which is faster and simpler. Are you using any tools to monitor, manage and backup your MongoDB deployment? We rely heavily on the MongoDB Management Service application for proactive monitoring of our database cluster. Through MMS alerting we identified a potential issue with replication and were able to rectify it before it caused an outage. For backups, we currently use mongodump, but are evaluating MMS Backup as this has the potential to extend our disaster recovery capabilities. For overall performance monitoring of our application stack, we use New Relic which is implemented in the drivers we use. What business advantage is MongoDB delivering? As a startup, time to market is key. We could not have got to market as quickly with other databases. MongoDB’s flexible document model and dynamic schema have been essential not only in launching the original service, but now as we evolve our products. Requirements change quickly and we are always adding new features. MongoDB enables us to do that. As we add more products and features, we add new customers. We need the ability to scale our infrastructure fast. Again MongoDB provides that scalability and operational simplicity we need to focus on the business, rather than the database. What advice would you give someone who is considering using MongoDB for their next project? We came from a relational database background and were surprised how easy it was for us in development and ops to transfer that knowledge to MongoDB. That helps us get up and running quickly. MongoDB schema design is new concept and requires a change in thinking - from a normalized model that packs data into rows and columns across multiple tables to a document model that allows embedding of related data into a single object. Developers need to move on from focusing on how data is stored, to how it is queried by the application. You need to identify your queries and build your schema from there. The good news is that there is a wealth of documentation online. The MongoDB blog is a great resource to learn best practices from the community. An example is the awesome post on MongoDB schema design for time series data - this will help anyone managing this type of data in IoT applications. The MongoDB University provides free self-paced training for developers (in multiple languages), administrators and operations staff. There are also some really useful tutorials covering every step of MongoDB replication and sharding . Our recommendation would be to perform due diligence during your research - ensure you understand your requirements, then download the software and get started in your evaluation. Wrapping Up Mike and Bernhard - I’d like to thank you for taking the time to share your experiences with us!
How DataSwitch And MongoDB Atlas Can Help Modernize Your Legacy Workloads
Data modernization is here to stay, and DataSwitch and MongoDB are leading the way forward. Research strongly indicates that the future of the Database Management System (DBMS) market is in the cloud, and the ideal way to shift from an outdated, legacy DBMS to a modern, cloud-friendly data warehouse is through data modernization. There are a few key factors driving this shift. Increasingly, companies need to store and manage unstructured data in a cloud-enabled system, as opposed to a legacy DBMS which is only designed for structured data. Moreover, the amount of data generated by a business is increasing at a rate of 55% to 65% every year and the majority of it is unstructured. A modernized database that can improve data quality and availability provides tremendous benefits in performance, scalability, and cost optimization. It also provides a foundation for improving business value through informed decision-making. Additionally, cloud-enabled databases support greater agility so you can upgrade current applications and build new ones faster to meet customer demand. Gartner predicts that by 2022, 75% of all databases will be on the cloud – either by direct deployment or through data migration and modernization. But research shows that over 40% of migration projects fail. This is due to challenges such as: Inadequate knowledge of legacy applications and their data design Complexity of code and design from different legacy applications Lack of automation tools for transforming from legacy data processing to cloud-friendly data and processes It is essential to harness a strategic approach and choose the right partner for your data modernization journey. We’re here to help you do just that. Why MongoDB? MongoDB is built for modern application developers and for the cloud era. As a general purpose, document-based, distributed database, it facilitates high productivity and can handle huge volumes of data. The document database stores data in JSON-like documents and is built on a scale-out architecture that is optimal for any kind of developer who builds scalable applications through agile methodologies. Ultimately, MongoDB fosters business agility, scalability and innovation. Key MongoDB advantages include: Rich JSON Documents Powerful query language Multi-cloud data distribution Security of sensitive data Quick storage and retrieval of data Capacity for huge volumes of data and traffic Design supports greater developer productivity Extremely reliable for mission-critical workloads Architected for optimal performance and efficiency Key advantages of MongoDB Atlas , MongoDB’s hosted database as a service, include: Multi-cloud data distribution Secure for sensitive data Designed for developer productivity Reliable for mission critical workloads Built for optimal performance Managed for operational efficiency To be clear, JSON documents are the most productive way to work with data as they support nested objects and arrays as values. They also support schemas that are flexible and dynamic. MongoDB’s powerful query language enables sorting and filtering of any field, regardless of how nested it is in a document. Moreover, it provides support for aggregations as well as modern use cases including graph search, geo-based search and text search. Queries are in JSON and are easy to compose. MongoDB provides support for joins in queries. MongoDB supports two types of relationships with the ability to reference and embed. It has all the power of a relational database and much, much more. Companies of all sizes can use MongoDB as it successfully operates on a large and mature platform ecosystem. Developers enjoy a great user experience with the ability to provision MongoDB Atlas clusters and commence coding instantly. A global community of developers and consultants makes it easy to get the help you need, if and when you need it. In addition, MongoDB supports all major languages and provides enterprise-grade support. Why DataSwitch as a partner for MongoDB? Automated schema re-design, data migration & code conversion DataSwitch is a trusted partner for cost-effective, accelerated solutions for digital data transformation, migration and modernization through a modern database platform. Our no-code and low-code solutions along with cloud data expertise and unique, automated schema generation accelerates time to market. We provide end-to-end data, schema and process migration with automated replatforming and refactoring, thereby delivering: 50% faster time to market 60% reduction in total cost of delivery Assured quality with built-in best practices, guidelines and accuracy Data modernization: How “DataSwitch Migrate” helps you migrate from RDBMS to MongoDB DataSwitch Migrate (“DS Migrate”) is a no-code and low-code toolkit that leverages advanced automation to provide intuitive, predictive and self-serviceable schema redesign from a traditional RDBMS model to MongoDB’s Document Model with built-in best practices. Based on data volume, performance, and criticality, DS Migrate automatically recommends the appropriate ETTL (Extract, Transfer, Transform & Load) data migration process. DataSwitch delivers data engineering solutions and transformations in half the timeframe of the existing typical data modernization solutions. Consider these key areas: Schema redesign – construct a new framework for data management. DS Migrate provides automated data migration and transformation based on your redesigned schema, as well as no-touch code conversion from legacy data scripts to MongoDB Atlas APIs. Users can simply drag and drop the schema for redesign and the platform converts it to a document-based JSON structure by applying MongoDB modeling best practices. The platform then automatically migrates data to the new, re-designed JSON structure. It also converts the legacy database script for MongoDB. This automated, user-friendly data migration is faster than anything you’ve ever seen. Here’s a look at how the schema designer works. Refactoring – change the data structure to match the new schema. DS Migrate handles this through auto code generation for migrating the data. This is far beyond a mere lift and shift. DataSwitch takes care of refactoring and replatforming (moving from the legacy platform to MongoDB) automatically. It is a game-changing unique capability to perform all these tasks within a single platform. Security – mask and tokenize data while moving the data from on-premise to the cloud. As the data is moving to a potentially public cloud, you must keep it secure. DataSwitch’s tool has the capability to configure and apply security measures automatically while migrating the data. Data Quality – ensure that data is clean, complete, trustworthy, consistent. DataSwitch allows you to configure your own quality rules and automatically apply them during data migration. In summary: first, the DataSwitch tool automatically extracts the data from an existing database, like Oracle. It then exports the data and stores it locally before zipping and transferring it to the cloud. Next, DataSwitch transforms the data by altering the data structure to match the re-designed schema, and applying data security measures during the transform step. Lastly, DS Migrate loads the data and processes it into MongoDB in its entirety. Process Conversion Process conversion, where scripts and process logic are migrated from legacy DBMS to a modern DBMS, is made easier thanks to a high degree of automation. Minimal coding and manual intervention are required and the journey is accelerated. It involves: DML – Data Manipulation Language CRUD – typical application functionality (Create, Read, Update & Delete) Converting to the equivalent of MongoDB Atlas API Degree of automation DataSwitch provides during Migration Schema Migration Activities DS Automation Capabilities Application Data Usage Analysis 70% 3NF to NoSQL Schema Recommendation 60% Schema Re-Design Self Services 50% Predictive Data Mapping 60% Process Migration Activities DS Automation Capabilities CRUD based SQL conversion (Oracle, MySQL, SQLServer, Teradata, DB2) to MongoDB API 70% Data Migration Activities DS Automation Capabilities Migration Script Creation 90% Historical Data Migration 90% 2 Catch Load 90% DataSwitch Legacy Modernization as a Service (LMaas): Our consulting expertise combined with the DS Migrate tool allows us to harness the power of the cloud for data transformation of RDBMS legacy data systems to MongoDB. Our solution delivers legacy transformation in half the time frame through pay-per-usage. Key strengths include: ● Data Architecture Consulting ● Data Modernization Assessment and Migration Strategy ● Specialized Modernization Services DS Migrate Architecture Diagram Contact us to learn more.