Real-Time Energy Monitoring for Smart Buildings with MongoDB and HiveMQ

The Internet of Things (IoT) has ushered in a new era of energy efficiency, enabling the deployment of energy-efficient sensors for energy conservation and resource utilization. With over 1.5 billion connected IoT devices already installed in commercial smart buildings in 2022 and a projected surge to 3.25 billion devices by 2028, the volume of data generated is staggering. To put it in perspective, an average home in 2020 would generate approximately 4.7 terabytes of data annually. However, managing and harnessing this immense amount of real-time streaming data poses a significant challenge for developers. In smart buildings, where a multitude of IoT sensors continuously gather event streaming data, developers often grapple with integrating disparate technologies and investing significant time into data streaming integration.

In this blog, we present a simple yet powerful solution to this challenge. We will demonstrate how you can effortlessly move IoT data using standard protocols such as MQTT to MongoDB Atlas using the HiveMQ Enterprise Extension for MongoDB. By optimizing smart buildings with real-time energy monitoring through the seamless integration of MongoDB and HiveMQ, we unlock the potential for efficient energy management and a sustainable future.

Let’s get started!

Dream team: HiveMQ + MongoDB

In a world where energy conservation and efficient resource utilization are essential, let’s go through Figure 1 and see how simple it is to use MongoDB, HiveMQ’s MQTT broker, and the Enterprise Extension for MongoDB to enable real-time energy monitoring for smart building.

Figure 1: Combining HiveMQ and MongoDB process data in real-time

Step 1: Data transmission

Using MQTT-based IoT devices deployed throughout the building, electricity consumption, temperature, and occupancy data is collected and sent to the HiveMQ MQTT broker. The MQTT broker acts as a central hub, efficiently and securely handling the communication between devices and backend systems. The HiveMQ MQTT broker also ensures reliable message delivery. It also provides MQTT-specific features like quality of service, session management, and topic-based message routing.

Step 2: Data ingestion

The HiveMQ MongoDB extension seamlessly integrates with MongoDB, allowing for persistent storage of the MQTT data in a highly scalable and flexible manner. The fully customizable templating system allows MQTT data to be stored according to the building’s specific operational requirements. MongoDB's document-based model accommodates the varying data formats and structures generated by different IoT devices.

Step 3: Data visualization and analytics

Once the MQTT data is securely stored in MongoDB, using its powerful in-app analytics, building managers can gain deep insights into energy consumption patterns, identify anomalies, and optimize energy usage. By leveraging MongoDB's rich query support and aggregation framework, building managers can make data-driven decisions promptly, reducing costs and enhancing sustainability. In cases where data needs to be exported to an ML/AI engine, MongoDB Spark and Kafka connectors can be used. Users of MongoDB Atlas can leverage Atlas Device Sync and Realm to send real-time alerts and messages to mobile devices. Data can be visualized using MongoDB Atlas Charts or through a third-party Business Intelligence (BI) tool connected via MongoDB BI connector or Atlas SQL interface.

Conclusion

By seamlessly integrating HiveMQ's MQTT broker with MongoDB, developers can efficiently handle data transmission, ingestion, and storage. This integration enables building managers to gain valuable insights into energy consumption patterns, make data-driven decisions, and optimize energy usage.

To learn more about MongoDB’s role in IoT, please visit our IoT webpage. You can also try the HiveMQ platform now with the Enterprise Extension for MongoDB for free.

Thank you Ainhoa Múgica for her contributions to this blog.

← Previous

How Telcos Drive Mission-Critical Innovation and Cost Savings Through Automation

Modern telecommunications customers don’t only expect flawless network performance, they demand it. For global enterprises that hold the key to human connection, the industry standard is nothing less than a fully integrated, customer-centric approach to service delivery. How are telecommunications firms tackling these customer expectations today? By embracing AI and machine learning capabilities, developing new decisioning models, and ensuring network security and optimization with a 99.995% uptime SLA. Why automation is vital in the telco modernization journey AI adoption is growing at a break-neck speed, and the telecommunications industry has a close eye on the way that automated decision-making can improve operational efficiency and reduce costs. In the latest TM Forum Digital Transformation Tracker 7 survey , 73% of respondents agreed that operational efficiency and cost reduction were very important drivers of CSPs’ digital transformation journeys, compared with 69% in 2022 and 65% in 2018. By eliminating manual tasks, and reducing errors introduced by manual intervention, automation is improving end-to-end performance and reducing handoffs and touchpoints. In the case of automated decision-making, it’s possible to leverage large volumes of data that already exist within telco organizations, alongside data science and machine learning techniques, to generate data-driven insights and inferences to better serve customers and develop cost savings. Three automation use cases in the telecommunications industry Here are three ways that MongoDB’s customers in the telecommunications industry can innovate with automation: Service assurance processes: Telcos can proactively identify issues impacting customers, or even predict them before they occur, utilizing automated processing of large amounts of diverse data. Network automation techniques can then step in to automatically remediate the situation, and output intent that can be processed by intent-based network automation tooling. Network automation: While service assurance processes can make decisions around what needs to be done, network automation tooling takes responsibility for effecting change. What’s more, the demands of 5G networks will force operators to open up traditional closed systems to third parties via network APIs. For example, these APIs can allow automated provisioning and configuration of 5G network slices. Customer expectations here will be that these operations are self-serve, and will happen in real time, making automation key. Customer issue management: We’ve all come across chatbots used in B2C customer service experiences. As chatbots become more sophisticated, potentially leveraging modern generative AI techniques, many more customer care issues will be automated. This change will not only reduce the cost of call centers, but speed up mean time to resolution for customer care issues. The future of AI/ML-based automated decision-making in telecommunications We’ve established the importance of automation for service assurance, network automation, and customer experiences. By utilizing the power of AI and data science, telcos have the opportunity to take these technologies further into the realm of network security and fraud mitigation. However, getting to automation with unstructured data is no simple task. Studies show that more than 50% of data scientists’ time is spent wrangling data, and more than 80% of all essential data is unstructured. Figure 1: Data processing operations for machine learning models. To build out a new AI/ML-based system, both data processing and ML capabilities must be in place. These two solutions are typically provided by different systems with clear integration points. Any AI/ML-based solution requires large amounts of historical data to train the model. The storage and feeding of this data is usually the job of a traditional data warehouse system. This gets complicated when decisions need to be made in real time with “live” data, such as in anti-fraud use cases. To achieve real-time results, it’s required to integrate an operational database into the architecture to stream real-time data and requests into the model and to persist the model output. In this hybrid system example, we have both operational and analytics data requirements co-existing; this interaction adds to the overall architectural complexity of the system. It’s important to note that raw data cannot be used by AI models “as it.” First the data must be cleaned, potentially deduplicated and turned into features. Standardized techniques are required to do this, including binning, normalization, standardization, and one-hot encoding. The aggregation pipeline provides powerful data processing capabilities that assist with this process. MongoDB Atlas, the developer data platform, is capable of handling each of the above requirements from a single platform. Its analytical nodes and Data Lake allow for massive amounts of historical data storage, and service to the model for training purposes. Figure 2: MongoDB's role in machine learning pipeline. Real-time data can also be ingested and served through MongoDB Atlas via change streams, triggers, and integrations. The powerful aggregation framework of MongoDB is capable of transforming raw data into usable features. Lastly, integration patterns based on Spark, Kafka, and HTTP are supported out of the box, which greatly reduces overall architectural complexity. Once decision-making models produce the new data output, it can be persisted back into the transactional database automatically actioned by additional automation tooling. Decision-making models will constantly evolve and will require new facets of data to be used. MongoDB’s document model and dynamic schema naturally supports the ingestion of new types and formats of data without the need for complex schema changes. > Join MongoDB, Hansen Technology and TM Forum for a live discussion on the future of AI in the telecommunications industry.

July 10, 2023
Next →

Building AI With MongoDB: Integrating Vector Search And Cohere to Build Frontier Enterprise Apps

Cohere is the leading enterprise AI platform, building large language models (LLMs) which help businesses unlock the potential of their data. Operating at the frontier of AI, Cohere’s models provide a more intuitive way for users to retrieve, summarize, and generate complex information. Cohere offers both text generation and embedding models to its customers. Enterprises running mission-critical AI workloads select Cohere because its models offer the best performance-cost tradeoff and can be deployed in production at scale. Cohere’s platform is cloud-agnostic. Their models are accessible through their own API as well as popular cloud managed services, and can be deployed on a virtual private cloud (VPC) or even on-prem to meet companies where their data is, offering the highest levels of flexibility and control. Cohere’s leading Embed 3 and Rerank 3 models can be used with MongoDB Atlas Vector Search to convert MongoDB data to vectors and build a state-of-the-art semantic search system. Search results also can be passed to Cohere’s Command R family of models for retrieval augmented generation (RAG) with citations. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. A new approach to vector embeddings It is in the realm of embedding where Cohere has made a host of recent advances. Described as “AI for language understanding,” Embed is Cohere’s leading text representation language model. Cohere offers both English and multilingual embedding models, and gives users the ability to specify the type of data they are computing an embedding for (e.g., search document, search query). The result is embeddings that improve the accuracy of search results for traditional enterprise search or retrieval-augmented generation. One challenge developers faced using Embed was that documents had to be passed one by one to the model endpoint, limiting throughput when dealing with larger data sets. To address that challenge and improve developer experience, Cohere has recently announced its new Embed Jobs endpoint . Now entire data sets can be passed in one operation to the model, and embedded outputs can be more easily ingested back into your storage systems. Additionally, with only a few lines of code, Rerank 3 can be added at the final stage of search systems to improve accuracy. It also works across 100+ languages and offers uniquely high accuracy on complex data such as JSON, code, and tabular structure. This is particularly useful for developers who rely on legacy dense retrieval systems. Demonstrating how developers can exploit this new endpoint, we have published the How to use Cohere embeddings and rerank modules with MongoDB Atlas tutorial . Readers will learn how to store, index, and search the embeddings from Cohere. They will also learn how to use the Cohere Rerank model to provide a powerful semantic boost to the quality of keyword and vector search results. Figure 1: Illustrating the embedding generation and search workflow shown in the tutorial Why MongoDB Atlas and Cohere? MongoDB Atlas provides a proven OLTP database handling high read and write throughput backed by transactional guarantees. Pairing these capabilities with Cohere’s batch embeddings is massively valuable to developers building sophisticated gen AI apps. Developers can be confident that Atlas Vector Search will handle high scale vector ingestion, making embeddings immediately available for accurate and reliable semantic search and RAG. Increasing the speed of experimentation, developers and data scientists can configure separate vector search indexes side by side to compare the performance of different parameters used in the creation of vector embeddings. In addition to batch embeddings, Atlas Triggers can also be used to embed new or updated source content in real time, as illustrated in the Cohere workflow shown in Figure 2. Figure 2: MongoDB Atlas Vector Search supports Cohere’s batch and real time workflows. (Image courtesy of Cohere) Supporting both batch and real-time embeddings from Cohere makes MongoDB Atlas well suited to highly dynamic gen AI-powered apps that need to be grounded in live, operational data. Developers can use MongoDB’s expressive query API to pre-filter query predicates against metadata, making it much faster to access and retrieve the more relevant vector embeddings. The unification and synchronization of source application data, metadata, and vector embeddings in a single platform, accessed by a single API, makes building gen AI apps faster, with lower cost and complexity. Those apps can be layered on top of the secure, resilient, and mature MongoDB Atlas developer data platform that is used today by over 45,000 customers spanning startups to enterprises and governments handling mission-critical workloads. What's next? To start your journey into gen AI and Atlas Vector Search, review our 10-minute Learning Byte . In the video, you’ll learn about use cases, benefits, and how to get started using Atlas Vector Search.

April 25, 2024