Audit Event Logging with Real-Time Reporting Deployed Across Continents with Write-Local and Read-Anywhere MongoDB Cluster
Security and compliance is a top-of-mind issue for executives in all enterprises around the world. Enforcing robust security controls to your most critical digital assets and systems through fine-grained auditing is a key step towards defending against potentially costly breaches.
I had the opportunity to sit down with Addison Chappell, Enterprise Architect at Ogilvy & Mather, to discuss an innovative auditing application built on top of MongoDB.
Can you start by telling us a little bit about your company?
Ogilvy & Mather (O&M) is one of the largest marketing communications companies in the world. We comprise industry leading business units operating across a range of disciplines including advertising, public relations, branding, promotions and market analytics. O&M services Fortune Global 500 companies as well as local businesses through its network of more than 500 offices in 126 countries.
Please describe how you are using MongoDB
MongoDB is being used for our core auditing application, capturing authentication and authorization activities of all users as they access O&M’s systems. From events written to MongoDB, we are able to build an audit trail of system access, which is used by our support, compliance and security teams.
From the auditing application, our teams have fine-grained visibility into who did what and when, what privileges each user has, and how each account is configured. The teams can enforce security policies such as password resets; resolve application access issues; monitor application usage by user, business unit and region; and much more.
What are the key characteristics of the auditing application?
The application is write-heavy, with MongoDB ingesting tens of gigabytes of data every day, from tens of thousands of users distributed around the globe.
MongoDB is used for both data ingest, and in generating real time analytics. We are using the MongoDB aggregation pipeline to roll up key metrics, such as snapshots of how many users are active on our system at any one time.
Did you start out on MongoDB, or were you using another database?
We originally built the application on a relational database, but it just was not able to keep up with the increasing size of our data set.
We also needed to ensure users got the same consistent low latency wherever they were located around the world. Creating a distributed database environment that spans continents proved extremely challenging using relational technology. It was hard to both capture data reliably, and query it within the performance SLAs demanded by the business.
What led you to MongoDB?
We discovered MongoDB back in 2013 while looking for alternatives to our SQL-based solutions. We choose MongoDB based on technical maturity, the size of the community actively using it, and the quality of support.
Please describe your MongoDB deployment
We have deployed a sharded MongoDB cluster in three data centers across two continents. We have active/active data centers in North America and on mainland Europe both serving read and write traffic, with a third data center in London housing replica sets arbiters to protect against network partitions causing divergent copies of the database.
O&M’s globally distributed MongoDB cluster: supporting local writes, with read-anywhere access
As illustrated in our architectural diagram, each active data center is provisioned with two shards, each of which is configured as a replica set with a primary and local secondary node, and then two remote secondaries in the other data center. This way, we can achieve continuous availability in the event of a regional data center outage.
We use MongoDB zones (also known as Tag-Aware sharding) to partition our database according to user location. With MongoDB zones, we ensure audit event data is written to nodes local to the user, thereby minimizing the effects of network latency, and then we can read that data globally for centralized analytics and reporting.
Do you use MongoDB’s commercial subscriptions?
We have built the auditing application on MongoDB Enterprise Advanced, providing us with access to the expert, proactive technical support required for this mission-critical application.
Through MongoDB Enterprise Advanced, we also get access to the Connector for BI for advanced analytics and data visualization, and Ops Manager for advanced operational tooling. We will be exploring both of these options in the future.
I understand you are planning to upgrade to the latest MongoDB 3.2 release later this year. Can you share your motivations for the upgrade?
There are three drivers behind the upgrade:
- The new default WiredTiger storage engine. We expect document level concurrency control to deliver higher performance for our write-intensive application, while compression will reduce storage overhead as our data set continues to grow.
- The MongoDB Encrypted storage engine will enable us to further protect our most sensitive data.
- MongoDB aggregation pipeline enhancements will help us build richer analytics natively within the database, reducing application-side code.
What have been the major benefits to you of switching from your previous relational database to MongoDB?
- Scalability to handle a data set growing at tens of GBs every day
- The ability to generate near real-time analytics against live operational data, even in a write heavy app like ours
- True cross-continent geo-distribution to support the performance and availability requirements of a global audience
What advice would you give to someone consider MongoDB for their next project?
MongoDB is very easy to get started with, but that doesn’t mean you shouldn't plan and architect your application carefully:
- Think about your data model, and how you plan to query against it. There is a decent introduction to MongoDB data modeling in the documentation.
- Also consider shard key selection, and make sure it is appropriate for both current and future anticipated application requirements.
Addison, thanks for sharing your experiences with the MongoDB community.
If you are struggling with your traditional database, download our:
About the Author - Mat Keep
Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.