MongoDB customers and community members are the people who realize GIANT ideas. We are excited to begin highlighting some of our community members, our MongoDB Giants, who are tackling challenging problems and bringing solutions to life with MongoDB.
March’s Giant of the Month is Mike Grayson, Senior MongoDB DBA at PayChex, a provider of payroll, human resource, and benefits outsourcing solutions for small to medium-sized businesses. Mike has been involved in many aspects of the MongoDB community since he started using the database in 2014. He received numerous internal awards from PayChex for his dedication to operationalizing their new system and educating his teams on the new database technology. In his own words, “with lots of help from Ops Manager, integrating MongoDB in to our ecosystem has been a great and painless process.”
Mike was also instrumental in the MongoDB 3.2 release and participated in Beta Testing for The Encrypted Storage Engine and MongoDB Compass. Paychex is a member of MongoDB’s customer advisory board and through their involvement Michael provides input into MongoDB’s product development.
In his spare time, if he’s not spending his time on the Advocacy Hub or reading about MongoDB and other databases, you can find Mike playing video games, rooting for Bayern Munich, or any of his favorite Philadelphia sports teams with his wife, three girls, and two dogs in the beautiful Finger Lakes Region of New York.
Have your voice heard in the MongoDB community. Join our Advocacy Hub and start getting involved today.
Using MongoDB, Kafka and Spark to Build Infrastructure for India’s First Affordable Smart-Homes Project
By Gautam Rege, Co-Founder of Josh Software and Co-Founder of SimplySmart Solutions In Sheltrex , a growing community about two hours outside of Mumbai, India, we’re part of a project that will put more than 100,000 people in affordable smart homes. To make those homes truly smart we’re building infrastructure that streams data from millions of sensors in near real-time. Citizens can then access the data through a mobile application that allows them to better manage their home. It’s a fantastic example of how technology can improve our lives, but building scalable and fast infrastructure is not simple. In this blog, I want to highlight how my team at Josh Software , one of India’s leading internet of things and web application specialists, is overcoming those challenges by using a stack of interesting data tools like Apache Kafka, Apache Spark and MongoDB . Of the planned 20,000 homes in Sheltrex, more than 1,500 have already been completed. Many people people are already living on site. The pilot is a proving ground for a whole host of smart township technologies. From mobile connected security to smart-meters monitoring power consumption. Along with the mobile application for individual citizens we’ve also built software that will aggregate this data for the entire community. This gives the township the ability to negotiate more competitive rates from India’s electricity providers. Sheltrex affordable home project in Karjat, India To provide homeowners and the community with accurate and timely utility data means processing information from millions of sensors quickly, then storing it in a robust and efficient way. The Smart City Application communicates with our stack APIs to make business sense for residents and the township management. The entire solution is split into two “universes.” Universe One is where we stream all the sensor data that is flooding in from the homes in real time. This could include data points like temperature or energy usage. The sensor and smart-meter data is first ingested into a messaging system powered by Kafka (an open source, high-throughput, distributed, publish-subscribe platform that can quickly process real-time data feeds at a large scale). Through Kafka the data is dropped into Spark , a large-scale data processing engine that is basically a much faster and simpler alternative to MapReduce. It’s in Spark, using Java and Python, that we do the processing and aggregation of the data - before it’s written on to our second “universe.” Universe Two is where the smart home data is stored and accessed by the mobile application. We need something fast, flexible and robust, so we turned to MongoDB. It is the primary database for all storage, analysis and archiving of the smart home data. This includes time-series data like regular temperature information, as well as enriched metadata such as accumulated electricity costs and usage rates. To connect the analytical and operational data sets we use the MongoDB Connector for Hadoop . We’ve found that the three technologies work well in harmony, creating a resilient, scalable and powerful big data pipeline, without the complexity inherent in other distributed streaming and database environments. Both in development, where it’s relatively simple to integrate them, and in production where the data flows smoothly between each stage. Smarter, faster I’ve been using MongoDB since the beginning, in fact, I’ve written a couple of books on the subject . It’s been great to see how the database itself has matured and kept adding the right features at the right time. Another big advantage for us is how much more productive MongoDB makes developers and operations staff. The devops team is continuously delivering code to support new requirements, so they need to make things happen fast. MongoDB’s ease of use means we can accelerate our development process and get new features integrated, tested and deployed quickly. Right now we’re operating across eight Amazon Web Services instances in the same zone. As the project expands and more citizens move into Sheltrex we expect to see huge growth. That’s why it’s been so important for us to leverage technologies that operate efficiently at scale. Sheltrex affordable home project in Karjat, India So far the pilot has been incredibly successful and we’re pleased with how our infrastructure is steadily increasing it’s capacity as thousands of new homes come online. But what we’re doing in Sheltrex is only the beginning. Housing is a volume game, as more people live in smart affordable homes the greater the effect will be for the community and the environment. I believe this type of affordable and intelligent housing should become standard across the world. Minor initial costs lead to massive efficiencies over the lifetime of the building. These are not simply monetary - consider the wasted water and electricity that we could save. To get there it will take political will and, of course, considerable funding, but from my point of view the technology is ready to go today. By building our giant idea on modern and mature technologies like MongoDB, we’re ready to change the world. About Josh Software & SimplySmart Driven by enthusiasm and passion, Josh is India’s leading company in building innovative web applications working exclusively in Ruby On Rails since 2007. The company thrives only on three basic needs - disruption, innovation, and learning. It builds products for customers who are able to fulfil at least two of these needs. Details are available at www.joshsoftware.com . Due to the diverse nature of building smart solutions for townships, Josh has incorporated another company called SimplySmart Solutions that builds and implements these solutions. As the name suggests SimplySmart Technologies relies on simple solutions for making things smarter. Details are available at www.simplysmart.tech . Who else runs on MongoDB? Find out: Who else uses MongoDB?
How DataSwitch And MongoDB Atlas Can Help Modernize Your Legacy Workloads
Data modernization is here to stay, and DataSwitch and MongoDB are leading the way forward. Research strongly indicates that the future of the Database Management System (DBMS) market is in the cloud, and the ideal way to shift from an outdated, legacy DBMS to a modern, cloud-friendly data warehouse is through data modernization. There are a few key factors driving this shift. Increasingly, companies need to store and manage unstructured data in a cloud-enabled system, as opposed to a legacy DBMS which is only designed for structured data. Moreover, the amount of data generated by a business is increasing at a rate of 55% to 65% every year and the majority of it is unstructured. A modernized database that can improve data quality and availability provides tremendous benefits in performance, scalability, and cost optimization. It also provides a foundation for improving business value through informed decision-making. Additionally, cloud-enabled databases support greater agility so you can upgrade current applications and build new ones faster to meet customer demand. Gartner predicts that by 2022, 75% of all databases will be on the cloud – either by direct deployment or through data migration and modernization. But research shows that over 40% of migration projects fail. This is due to challenges such as: Inadequate knowledge of legacy applications and their data design Complexity of code and design from different legacy applications Lack of automation tools for transforming from legacy data processing to cloud-friendly data and processes It is essential to harness a strategic approach and choose the right partner for your data modernization journey. We’re here to help you do just that. Why MongoDB? MongoDB is built for modern application developers and for the cloud era. As a general purpose, document-based, distributed database, it facilitates high productivity and can handle huge volumes of data. The document database stores data in JSON-like documents and is built on a scale-out architecture that is optimal for any kind of developer who builds scalable applications through agile methodologies. Ultimately, MongoDB fosters business agility, scalability and innovation. Key MongoDB advantages include: Rich JSON Documents Powerful query language Multi-cloud data distribution Security of sensitive data Quick storage and retrieval of data Capacity for huge volumes of data and traffic Design supports greater developer productivity Extremely reliable for mission-critical workloads Architected for optimal performance and efficiency Key advantages of MongoDB Atlas , MongoDB’s hosted database as a service, include: Multi-cloud data distribution Secure for sensitive data Designed for developer productivity Reliable for mission critical workloads Built for optimal performance Managed for operational efficiency To be clear, JSON documents are the most productive way to work with data as they support nested objects and arrays as values. They also support schemas that are flexible and dynamic. MongoDB’s powerful query language enables sorting and filtering of any field, regardless of how nested it is in a document. Moreover, it provides support for aggregations as well as modern use cases including graph search, geo-based search and text search. Queries are in JSON and are easy to compose. MongoDB provides support for joins in queries. MongoDB supports two types of relationships with the ability to reference and embed. It has all the power of a relational database and much, much more. Companies of all sizes can use MongoDB as it successfully operates on a large and mature platform ecosystem. Developers enjoy a great user experience with the ability to provision MongoDB Atlas clusters and commence coding instantly. A global community of developers and consultants makes it easy to get the help you need, if and when you need it. In addition, MongoDB supports all major languages and provides enterprise-grade support. Why DataSwitch as a partner for MongoDB? Automated schema re-design, data migration & code conversion DataSwitch is a trusted partner for cost-effective, accelerated solutions for digital data transformation, migration and modernization through a modern database platform. Our no-code and low-code solutions along with cloud data expertise and unique, automated schema generation accelerates time to market. We provide end-to-end data, schema and process migration with automated replatforming and refactoring, thereby delivering: 50% faster time to market 60% reduction in total cost of delivery Assured quality with built-in best practices, guidelines and accuracy Data modernization: How “DataSwitch Migrate” helps you migrate from RDBMS to MongoDB DataSwitch Migrate (“DS Migrate”) is a no-code and low-code toolkit that leverages advanced automation to provide intuitive, predictive and self-serviceable schema redesign from a traditional RDBMS model to MongoDB’s Document Model with built-in best practices. Based on data volume, performance, and criticality, DS Migrate automatically recommends the appropriate ETTL (Extract, Transfer, Transform & Load) data migration process. DataSwitch delivers data engineering solutions and transformations in half the timeframe of the existing typical data modernization solutions. Consider these key areas: Schema redesign – construct a new framework for data management. DS Migrate provides automated data migration and transformation based on your redesigned schema, as well as no-touch code conversion from legacy data scripts to MongoDB Atlas APIs. Users can simply drag and drop the schema for redesign and the platform converts it to a document-based JSON structure by applying MongoDB modeling best practices. The platform then automatically migrates data to the new, re-designed JSON structure. It also converts the legacy database script for MongoDB. This automated, user-friendly data migration is faster than anything you’ve ever seen. Here’s a look at how the schema designer works. Refactoring – change the data structure to match the new schema. DS Migrate handles this through auto code generation for migrating the data. This is far beyond a mere lift and shift. DataSwitch takes care of refactoring and replatforming (moving from the legacy platform to MongoDB) automatically. It is a game-changing unique capability to perform all these tasks within a single platform. Security – mask and tokenize data while moving the data from on-premise to the cloud. As the data is moving to a potentially public cloud, you must keep it secure. DataSwitch’s tool has the capability to configure and apply security measures automatically while migrating the data. Data Quality – ensure that data is clean, complete, trustworthy, consistent. DataSwitch allows you to configure your own quality rules and automatically apply them during data migration. In summary: first, the DataSwitch tool automatically extracts the data from an existing database, like Oracle. It then exports the data and stores it locally before zipping and transferring it to the cloud. Next, DataSwitch transforms the data by altering the data structure to match the re-designed schema, and applying data security measures during the transform step. Lastly, DS Migrate loads the data and processes it into MongoDB in its entirety. Process Conversion Process conversion, where scripts and process logic are migrated from legacy DBMS to a modern DBMS, is made easier thanks to a high degree of automation. Minimal coding and manual intervention are required and the journey is accelerated. It involves: DML – Data Manipulation Language CRUD – typical application functionality (Create, Read, Update & Delete) Converting to the equivalent of MongoDB Atlas API Degree of automation DataSwitch provides during Migration Schema Migration Activities DS Automation Capabilities Application Data Usage Analysis 70% 3NF to NoSQL Schema Recommendation 60% Schema Re-Design Self Services 50% Predictive Data Mapping 60% Process Migration Activities DS Automation Capabilities CRUD based SQL conversion (Oracle, MySQL, SQLServer, Teradata, DB2) to MongoDB API 70% Data Migration Activities DS Automation Capabilities Migration Script Creation 90% Historical Data Migration 90% 2 Catch Load 90% DataSwitch Legacy Modernization as a Service (LMaas): Our consulting expertise combined with the DS Migrate tool allows us to harness the power of the cloud for data transformation of RDBMS legacy data systems to MongoDB. Our solution delivers legacy transformation in half the time frame through pay-per-usage. Key strengths include: ● Data Architecture Consulting ● Data Modernization Assessment and Migration Strategy ● Specialized Modernization Services DS Migrate Architecture Diagram Contact us to learn more.