AnnouncementIntroducing MongoDB 8.0, the fastest MongoDB ever! Read more >
NEWSLearn why MongoDB was named a leader in the 2024 Gartner® Magic Quadrant™ Read the blog >
AnnouncementIntroducing Search Demo Builder, the newest addition to the Atlas Search Playground Learn more >

SOLUTIONS

Power Smart Meter Analysis with MongoDB

MongoDB provides a scalable platform for smart meter systems, enabling real-time data collection, time series analysis, and anomaly detection.
Start Free
Illustration of a spinning wheel in motion with a dollar sign in the middle of the arrow pointer.
Solution Overview

With smart meter adoption expected to reach 90% in the U.S. and Canada by 2027, MongoDB provides a comprehensive platform for managing and analyzing smart meter time series data at scale. The solution enables utility companies to collect high-frequency meter readings, process streaming data in real time, and store historical data efficiently using MongoDB's time series collections. Powerful aggregation pipelines and change streams allow for sophisticated anomaly detection, helping utility companies quickly identify potential issues like voltage spikes, power outages, or unusual consumption patterns.

As energy consumption continues to rise globally, the amount of data being generated by smart meters also increases. Our solution leverages MongoDB's integration with IoT messaging protocols such as MQTT for reliable data ingestion from smart meters’ head-end units, while MongoDB Atlas's native time series support enables efficient storage and querying of temporal data. Utility companies can seamlessly scale from thousands to millions of connected meters while maintaining performance and data accessibility. Advanced features like data tiering and Atlas Online Archive help manage historical data cost-effectively, while built-in analytics capabilities enable real-time monitoring and operational insights through customizable dashboards and reports.

Reference Architectures

There are two main components of this solution:

1. Smart meter data storage

To use smart metering, power utility companies need to deploy a core smart metering ecosystem that includes the smart meters themselves, the meter data collection network, the head-end system (HES), and the meter data management system (MDMS). Smart meters collect data from end consumers and transmit it to the data aggregator via the Local Area Network (LAN). The transmission frequency can be adjusted to 15 minutes, 30 minutes, or hourly, depending on data demand requirements. The aggregator retrieves the data and then transmits it to the head-end system. The head-end system analyzes the data and sends it to the MDMS. The initial communications path is two-way—signals or commands can be sent directly to the meters, customer premise, or distribution device.

In our solution, MongoDB is the backend database for the MDMS system. Once the data reaches the HES, it is transmitted to MongoDB using a MQTT provider. Leading MQTT broker providers (such as HiveMQ, EMQX, and Cedalo) have developed MongoDB integrations that can enable seamless data transfer and efficient management of the information collected from smart meters.

This solution employs the Cedalo Pro Mosquitto MQTT broker to facilitate effective message transmission. By utilizing the MongoDB bridge, it creates a seamless connection with MongoDB, ensuring a continuous and efficient flow of data between the two systems.

Subsequently, MongoDB stores the smart meter data, which includes information such as meter ID, timestamp, and measurements (energy, current, voltage, etc.).

2. Next generation meter data management system with MongoDB

Once the smart meter data is stored in MongoDB Atlas, it can be analyzed for anomalies and insights. This solution leverages three key MongoDB features to create a next generation MDMS: change streams, time series collections, and the powerful MongoDB Query API.

MongoDB change streams enable real-time capturing and processing of database changes, such as insert operations, providing immediate insights into energy consumption patterns, system anomalies, and potential faults. This real-time capability is crucial for smart meter systems, as it allows for timely analysis via the MongoDB aggregation framework, detecting anomalies and preventing costly damages and service disruptions.

The analyzed data is stored in time series collections. These specialized collections provide high compression, enhance query performance of time series data, and enable fast operations.

End-to-end smart meter management system architecture
Figure 1: End-to-end smart meter management system architecture
Data model approach

The solution uses two separate time series data models. The first model is designed for raw smart meter data with meter_id as the metaField.

The second dataset includes anomalies detected in various measurements based on predefined criteria. In this solution example, anomalies are defined as readings that are over three standard deviations from the mean values. We are using meter_id as the metaField here.

Building the solution

This smart meter data management solution demonstrates an architecture that leverages MQTT broker and MongoDB Atlas to address the complex challenges of modern energy infrastructure data collection and analysis. While the reference implementation showcases Cedalo's MQTT Broker, the solution is designed to be broker-agnostic, supporting integration with various MQTT providers.

Key solution components
  1. MQTT broker (data ingestion layer)
    • Set up your MQTT broker for reliable message handling
    • Configure broker integration with MongoDB
    • Establish secure data streaming pipelines
    • Enable real-time data collection from smart meters
    • Implement broker management and monitoring

  2. Environment configuration (configuration layer)
    • Configure environment variables for:

      • MQTT broker connection details
      • MongoDB connection URI
      • Database and collection names
      • Analytics engine configuration
      • Atlas Charts integration

  3. Database configuration (data processing layer)
    • Create MongoDB database with required collections
    • Set up time series collections for:
      • Anomalies data
      • Metrics data
      • Transformed data
    • Configure data transformation pipelines
    • Enable real-time data processing capabilities

  4. Application deployment (analytics and visualization layer)
    • Deploy application components
    • Implement real-time monitoring capabilities
    • Set up anomaly detection algorithms
    • Create customized dashboards
    • Enable interactive data exploration

For a complete, hands-on implementation, check out the detailed repository: https://github.com/mongodb-industry-solutions/Leafy-Smart-Meters

Key Learnings
  • Data management is critical: Smart meters generate massive volumes of time series data, presenting significant challenges in data interoperability, privacy, and efficient processing. Effective solutions require flexible, scalable databases that can handle high-frequency, complex data streams.

  • MQTT enables efficient IoT communication: The lightweight publish-subscribe protocol is crucial for smart meter ecosystems, providing efficient, secure, and reliable data transmission between IoT devices, aggregators, and management systems.

  • MongoDB offers comprehensive IoT data solutions: With features like time series collections, change streams, aggregation frameworks, and the flexible document model, MongoDB addresses key challenges in smart meter data management, including anomaly detection, real-time analytics, and scalable data storage.

  • Integrated approach is key: Successful smart meter implementations require a holistic solution that combines robust messaging protocols (like MQTT), advanced database technologies, and comprehensive analytics capabilities to transform raw meter data into actionable insights for utility companies.

Technologies and Products Used
MongoDB developer data platform:
Partner technologies:
  • Cedalo MQTT Platform
Authors
  • Dr. Humza Akhtar, MongoDB
  • Rami Pinto, MongoDB
  • Sebastian Rojas Arbulu, MongoDB
  • Diego Canales, MongoDB
Related resources
general_content_developer

GitHub Repository: Leafy-Smart-Meters

Create this demo for yourself by following the instructions in this solution’s repository.

general_action_read

Saving Energy, Smarter: MongoDB and Cedalo for Smart Meter Systems

Discover how MongoDB and Cedalo enable smart meter systems.

cloud_iot

MongoDB for Internet of Things (IoT)

Learn how MongoDB’s developer data platform supports the entire IoT data life cycle.

Get started with Atlas today

Get started in seconds. Our free clusters come with 512 MB of storage so you can experiment with sample data and get familiar with our platform.
Try FreeContact sales
Illustration of hands typing on a laptop in the foreground and a superimposed desktop window and coffee cup in the background.