Making a user to tail the oplog with Automation
Update 2/25/2016: The new UI has changed the way this process would look (putting the users & roles under the “More” menu on the Deployment page), but the idea is the same. Feel free to open a ticket or chat us with any questions you may have about this.
A question we are asked a lot is how to create a user that can tail the oplog using Cloud Manager Automation. This is a feature needed by Meteor users if they want to use MongoDB authentication to protect their database servers. Here’s how:
- Head to your Authorization & Roles page
- Create a new role (I called mine “oplogger”) that has permissions to read the local database
- Once you save this role, you can go to your “Authentication & Users” tab:
- Then you can create a user with the “oplogger” role (and any other roles you may want) and save it with a password you know
- Push your changes via “Review & Deploy” and then “Confirm & Deploy”
Once you configure your Meteor installation (
MONGO_OPLOG_URL) to connect with the new credentials, your app should work as expected, providing you live tracking of changes.
Securing MongoDB Part 3: Database Auditing and Encryption
Welcome back to our 4-part blog series presenting the best practices and controls available in MongoDB to help you create a secure, compliant database platform. In this installment, we’ll be discussing database auditing and encryption. As a quick recap, in part 1 , we took a look at the general requirements for data security and regulatory compliance, and then in part 2 , reviewed MongoDB access control enforcing authentication and authorization. In part 4 , we’ll wrap up with environmental control and management. If you want to get a head-start and learn about all of these topics in one installment, just go ahead and download the MongoDB Security Architecture guide . MongoDB Auditing The auditing framework provided as part of MongoDB Enterprise Advanced logs all access and actions executed against the database. The auditing framework captures administrative actions (DDL) such as schema operations as well as authentication and authorization activities, along with read and write (DML) operations to the database. Administrators can construct and filter audit trails for any operation against MongoDB, whether DML, DCL or DDL without having to rely on third party tools. For example, it is possible to log and audit the identities of users who retrieved specific documents, and any changes made to the database during their session. **Figure 1**: MongoDB Maintains an Audit Trail of Administrative Actions Against the Database Administrators can configure MongoDB to log all actions or apply filters to capture only specific events, users or roles. The audit log can be written to multiple destinations in a variety of formats including to the console and syslog (in JSON format), and to a file (JSON or BSON), which can then be loaded to MongoDB and analyzed to identify relevant events. MongoDB Enterprise Advanced also supports role-based auditing. It is possible to log and report activities by specific role, such as userAdmin or dbAdmin – coupled with any inherited roles each user has – rather than having to extract activity for each individual administrator. Auditing adds performance overhead to a MongoDB system. The amount is dependent on several factors including which events are logged and where the audit log is maintained, such as on an external storage device and the audit log format. Users should consider the specific needs of their application for auditing and their performance goals in order to determine their optimal configuration. Learn more from the MongoDB auditing documentation . MongoDB Encryption Administrators can encrypt MongoDB data in motion over the network and at rest in permanent storage. Network Encryption Support for SSL/TLS allows clients to connect to MongoDB over an encrypted channel. Clients are defined as any entity capable of connecting to the MongoDB server, including: Users and administrators Applications MongoDB tools (e.g., mongodump, mongorestore, mongotop) Nodes that make up a MongoDB cluster, such as replica set members, query routers and config servers. It is possible to mix SSL/TLS with non-SSL/TLS connections on the same port, which can be useful when applying finer grained encryption controls for internal and external traffic, as well as avoiding downtime when upgrading a MongoDB cluster to support SSL. The TLS protocol is also supported with x.509 certificates. MongoDB Enterprise Advanced supports FIPS 140-2 encryption if run in FIPS Mode with a FIPS validated Cryptographic module. The mongod and mongos processes should be configured with the "sslFIPSMode" setting In addition, these processes should be deployed on systems with an OpenSSL library configured with the FIPS 140-2 module. The MongoDB documentation includes a tutorial for configuring TLS/SSL connections . Disk Encryption There are multiple ways to encrypt data at rest with MongoDB. Encryption can implemented at the application level, or via external filesystem and disk encryption solutions. By introducing additional technology into the stack, both of these approaches can add cost and complexity. With the introduction of the Encrypted storage engine in MongoDB 3.2 , protection of data at-rest becomes an integral feature of the database. By natively encrypting database files on disk, administrators eliminate both the management and performance overhead of external encryption mechanisms. This new storage engine provides an additional level of defense, allowing only those staff with the appropriate database credentials access to encrypted data. **Figure 2:** End to End Encryption – Data In-Flight and Data At-Rest Using the Encrypted storage engine, the raw database content, referred to as plaintext, is encrypted using an algorithm that takes a random encryption key as input and generates ciphertext that can only be read if decrypted with the decryption key. The process is entirely transparent to the application. MongoDB supports a variety of encryption schema, with AES-256 (256 bit encryption) in CBC mode being the default. AES-256 in GCM mode is also supported. The encryption schema can be configured for FIPS 140-2 compliance. The storage engine encrypts each database with a separate key. The key-wrapping scheme in MongoDB wraps all of the individual internal database keys with one external master key for each server. The Encrypted storage engine supports two key management options – in both cases, the only key being managed outside of MongoDB is the master key: Local key management via a keyfile. Integration with a third party key management appliance via the KMIP protocol (recommended). Most regulatory requirements mandate that the encryption keys must be rotated and replaced with a new key at least once annually. MongoDB can achieve key rotation without incurring downtime by performing rolling restarts of the replica set. When using a KMIP appliance, the database files themselves do not need to be re-encrypted, thereby avoiding the significant performance overhead imposed by key rotation in other databases. Only the master key is rotated, and the internal database keystore is re-encrypted. The Encrypted storage engine is designed for operational efficiency and performance: Compatible with WiredTiger’s document level concurrency control and compression. Support for Intel’s AES-NI equipped CPUs for acceleration of the encryption/decryption process. As documents are modified, only updated storage blocks need to be encrypted, rather than the entire database. Based on user testing, the Encrypted storage engine minimizes performance overhead to around 15% (this can vary, based on data types being encrypted), which can be much less than the observed overhead imposed by some filesystem encryption solutions. The Encrypted storage engine is based on WiredTiger and available as part of MongoDB Enterprise Advanced. Refer to the documentation to learn more, and see a tutorial on how to configure the storage engine. MongoDB Atlas Encryption As discussed in Part 2 of the Securing MongoDB blog series, MongoDB Atlas is a database as a service for MongoDB, providing all of the features of the database, without the operational heavy lifting required for any application. MongoDB Atlas has been engineered to deliver robust encryption controls. Data managed by the MongoDB Atlas service can be encrypted on the network and on disk. Support for TLS/SSL allows clients to connect to MongoDB over an encrypted channel. All data transfers across the cluster are also encrypted. Data at rest can be protected using encrypted data volumes. Note that this uses the cloud provider’s native volume encryption solution, rather than the MongoDB encrypted storage engine. Review the MongoDB Atlas documentation for more information on configuring the in-built security controls. Getting Started with MongoDB Security With comprehensive controls for user rights management, auditing and encryption, coupled with management controls, MongoDB can meet the best practice and requirements discussed in this blog series. MongoDB Enterprise Advanced is the certified and supported production release of MongoDB, with advanced security features, including Kerberos and LDAP authentication, encryption of data at-rest, FIPS-compliance, and maintenance of audit logs. These capabilities extend MongoDB’s security framework, which includes Role-Based Access Control, PKI certificates, Field-Level Redaction, and SSL/TLS data transport encryption. In the final part of this blog post series, we will dive into environmental control and database management. You can learn about all of these capabilities now by reading the MongoDB Security Architecture guide. If you want to try them for yourself, [download MongoDB Enterprise](https://www.mongodb.com/download-center?#enterprise), free of charge for evaluation and development. MongoDB security architecture About the Author - Mat Keep Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.
Congratulations to the 2023 APAC Innovation Award Winners
I’m thrilled to announce the nine winners of the 2023 MongoDB APAC Innovation Awards . The MongoDB Innovation Awards honor projects and people who dream big. They celebrate the groundbreaking use of data to build compelling applications and the creativity of professionals expanding the limits of technology with MongoDB. This year, we have broken the awards down regionally to celebrate organizations in APAC, from startups to industry-leading enterprises, across a wide variety of industries, who are delivering big results. We are delighted to announce the winners below: 2023 MongoDB APAC Innovation Award Winners: Positive Impact Open Government Products Open Government Products (OGP) is an in-house team of engineers, designers, and product managers, who is a part of the Singapore Government, and is responsible for building technologies for the public good. OGP used MongoDB’s developer data platform, MongoDB Atlas to create its digital form builder, FormSG. Used by the Singapore government and public healthcare institutions, FormSG securely collects data from residents and businesses and helps public officers to create digital government forms in minutes. It eliminates the use of paper forms and the manual process of transcribing physical documents, which had raised concerns around data privacy and protection. During the pandemic, FormSG enabled public officers to collect more than 100,000 daily temperature declarations nationwide. Today, FormSG has served more than 120,000 public officers from 155 agencies and it has created more than 500,000 digital forms to help the government collect data on travel and health declarations by visitors to the country, applications for COVID-19 swab tests, and applications for financial assistance. Organization Transformation Bendigo and Adelaide Bank Bendigo and Adelaide Bank is one of Australia’s largest banks, with around 7,000 employees helping more than 2.2 million customers achieve their financial goals. The bank has been on a multi-year journey of transformation using MongoDB's developer data platform to improve efficiency and deliver a better customer experience as they fulfill their vision to become Australia’s bank of choice. Recently, the cloud team launched Ready-Set-MongoDB (or RSM). This event-driven framework allows developers to streamline the consumption of internal or external APIs, and applies data transformations and storage automatically within a MongoDB collection of their choice. Using MongoDB Atlas Search, the bank also enabled developers to gain insights across its multi-cloud deployments, identifying cost savings, and providing inventory information to account owners and technical stakeholders. Within the first 18 months of launching these programmes, the automation had saved the organization more than 1,100 developers days. It also helped reduce human involvement, removed stale data, and allowed engineers to focus on the things that matter. The development of Ready-Set-MongoDB is ongoing and improving, as new Bendigo multi-cloud challenges arise and new MongoDB products are released. The application is a perfect representation of how Bendigo's Technology Department is using modern technology, rapid development, and innovation-led problem solving to drive organizational transformation. Heroes in Health Redcliffe Lifetech Private Limited Over the last few years, Redcliffe Labs has become India's fastest growing technology-driven diagnostics service provider. Redcliffe Labs is on a mission to serve 500 Million Indians by 2030 with fusion of technology and world- class laboratories. The company already serves thousands of people daily, with more than 73 labs and close to 1500 walk-in centers across 180 cities. Redcliffe Labs has relied on MongoDB Atlas’ flexible document model to power its innovative Smart Health Report, a patient resource that provides a number of indicators and trackers to gauge holistic health. The MongoDB developer data platform's best in class security, compliance, and privacy controls allows Redcliffe's team to confidently handle even the most sensitive patient data. MongoDB Atlas takes care of many of the traditional database management challenges, which means that developers can spend their time building diagnostics for patients, rather than managing databases. Redcliffe Labs is focusing on incorporating next-generation technologies in the diagnostics space with an AI platform that will make Interactive Diagnostics reports, Advanced Health Profiling and more detailed Diagnostics and Health Alerts. Industry Disruptor Cathay Pacific Cathay Pacific , Hong Kong’s home carrier operating in more than 60 destinations worldwide, has been on an impressive journey to become one of the very first airlines to create a truly paperless flight deck. Until recently, a flight from Hong Kong to New York would require a crew to review more than 150 pages of finely printed text and charts before their flight and make ongoing updates throughout the trip. In 2019, Cathay Pacific conducted the first zero paper flight, removing 50kg of manuals, charts, maps, and flight briefing paperwork. They achieved this enormous feat with the help of one seamless and highly customized iPad application: Flight Folder. Built on MongoDB Atlas, Flight Folder is designed to improve the pilot briefing experience. MongoDB helped consolidate dozens of different information sources into one place, and made it possible for flight crews to easily share their experiences with others. It also included a digital refueling feature that helps crews become much more efficient with fueling strategies – saving significant flight time and costs. The use of MongoDB Device Sync enables seamless syncing and no data loss even when the app goes on- and offline mid-flight. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. In addition to the greatly improved flight crew experience, flight times have been reduced, and digital refueling saves eight minutes of ground time on average. All these efficiencies have helped the company avoid the release of 15,000 tons of carbon. From Batch to Real-Time Adani Digital Labs Adani Digital Labs is the India-based digital innovation arm of the larger Adani group. The lab’s team's mission is to create one single platform – a SuperApp called AdaniOne – to empower a billion stories in India. To address several use cases and the huge scale that will be required by the superapp, the Adani Digital team selected MongoDB Atlas as its the main transactional database that will further enhance the application. A key component of the app is how it can bring together disparate data in order to provide a single view of activity across the application. In the first process, developers had taken out the data in batches and sent it to their database However, this was too slow and unpredictable as far as business requirements are concerned. Also, the consolidated view of customer history, orders, inventory, and supply chain network updates was likely to impact their customer's ability to generate revenue. Therefore, in order to find a better solution, Adani Digital Labs built a more modern architecture in line with MongoDB. Using MongoDB's Change Streams and the data platform's native Kafka connector, they created an event-based architecture that pushes the data out in real-time for analysis. Adani Digital Labs is still in the early phases of the SuperApp's rollout and collaborating with MongoDB as its developer data platform continues to help the firm to grow and deliver insights in real time. Industry 4.0 Dongwha Founded in 1948, the Dongwha Group has evolved from a singular focus on the wood and timber industry into a global leader across a number of sectors including building materials, chemicals and media. As part of its wider digital transformation strategy, Dongwha required smarter factories that would improve and optimize their production efficiency. Dongwha built an innovative Smart Factory Software platform that collects and analyzes data to enhance quality and production management capabilities. Originally, the platform was built with the community version of MongoDB. However, in order to scale and adapt, the team recently migrated to MongoDB Atlas in the cloud. This enabled them to store large volumes in the fastest and most secure way, optimize their solution for time series data, and make it easy to run machine learning across their data. Dongwha completed the migration seamlessly, without any disruption or downtime to their factories, and it has now been launched across five different sites. Over the last year, the application has significantly increased its availability and reliability while performance has improved by as much as 6x . As they look to the future, Dongwha plans to roll out the software to more of its international factories. Digital Native myBillBook India is home to more than 60 million small and medium-sized businesses (SMBs) but only a small portion of those SMBs are taking advantage of digitization and many still operate using pen and paper. In addition, many businesses in India still struggle with fluctuations in internet services, outages, and latency. FloBiz is on a mission to change that with myBillBook , a one-stop solution that helps SMBs create professional invoices, manage stock, collect payments, automate reminders through smart banking, engage with their customers, manage staff attendance and payroll and generate more than 25 business reports for accounting and decision making. The app is also mobile-first, so businesses can access them from their mobile devices and allows users to manage billing and inventory in both online and offline environments. The myBillbook app is powered by MongoDB Atlas, providing the flexible and scalable foundation for the business to do everything from building new features to performing complex analytical queries. In addition, MongoDB Realm, the mobile database within the data platform, supports offline usage and syncing to ensure there is never data loss or functionality for users due to poor internet connection. Because of its success in supporting customers with business critical operations, more than 6.5 million business owners in India are now using myBillbook for their billing, accounting, collection and business growth. Customer Focused KASIKORN Business-Technology Group Established in 1945, Kasikornbank (KBank) is one of the largest and oldest banks in Thailand. Their mission is to strive towards service excellence and empower every customer’s life and business. One of KBank’s subsidiaries, KASIKORN Business-Technology Group (KBTG) , developed a mobile banking application – MAKE by KBank. MongoDB Atlas’ flexibility and ease of development enabled MAKE’s development team to choose the best type of database for its tasks, to automate data tiering with Atlas Online Archive, and to reduce hours spent on operational maintenance. With more time to focus on delivering new innovations to customers, they created unique features like Cloud Pocket which can allocate funds into unlimited customizable pockets for separate usage. They also built Pop Pay, a feature that allows users to easily search for nearby friends and transfer money by clicking their profile picture as well as “Expense Summary" a spending analysis services that helps inform and manage users’ financial habits. As of January 2023, MAKE has acquired more than 1 million users, and increased the number of transactions in MAKE from 900,000 to more than 7.5 million in a span of one year. Massive Scale China Mobile China Mobile provides mobile voice and multimedia services via its nationwide mobile telecommunications network across mainland China and Hong Kong. It is the world's largest mobile network operator by total number of subscribers. The telecommunications leader is using MongoDB to support one of its largest and most critical push services, which sends out billing details to more than 1 billion users every month. Prior to MongoDB, the tech team relied on Oracle, but as the user numbers increased, performance degraded. Despite large investments, it was still taking too long to do basic requests like finalize and deliver bills to users. In 2019, after comprehensive testing, China Mobile migrated to MongoDB. By taking advantage of MongoDB's native sharding, they were able to improve performance by 80% and go from 50 Oracle machines, to just 12 machines for the same workload. The service now handles all current requirements and is set up to scale with future growth. With the support of MongoDB, China Mobile is growing steadily,with more than 168 million monthly users and has one of the highest customer satisfaction scores in the China Mobile group.