GIANT Stories at MongoDB

Automating Adobe AEM on MongoDB Atlas and Microsoft Azure

Jason Mimick

Cloud
Update (August 31, 2018)
Please note that this post uses a version of MongoDB (3.6.6) which is currently not officially supported by Adobe AEM. While future versions of AEM will support this version of MongoDB we recommend following all of the official Adobe guidance on this topic. Such information can be found directly in the AEM documenation: aem-with-mongodb and technical-requirements. The specific versions used here are for testing and development purposes only. It is also recommended to follown the best practices as documented by Adobe.

Introduction

Given the overwhelming trend towards automation in the technology industry we thought it would be a great idea to showcase how you can use MongoDB to do some awesome automation with two of MongoDB's premiere partners, Adobe and Microsoft.

Adobe Experience Manager (AEM) is a leading digital experience management solution. Enterprises like Manulife, Nissan, and Philips rely on AEM to deliver rapidly adaptable experiences across their customer journey. With MongoDB Atlas behind AEM, customers get a comprehensive solution integrated with a highly available, globally scalable, and enterprise-level database. Running all this on Microsoft's Azure cloud infrastructure ensures everything will run smoothly, meet your SLA's, and not break your budget. All this together makes delivering great digital experiences that much easier and allows you and your team to focus on your core missions.

This blog post will run through a straight-forward end-to-end automated deployment of an Adobe AEM Author node using MongoDB Atlas all running on Microsoft Azure for a development or testing environment. Production-level deployments require additional steps, such as nodes for additional AEM components (publish, etc) and high-availability for the AEM nodes for example. We hope you can leverage this example in your own automated workflows.

AEM released version 6.4 in March 2018. One of the exciting new features is ability to support cloud-based deployments for MongoDB. For customers looking to move to the cloud, MongoDB is easy to deploy and even easier when leveraging MongoDB Atlas for a full DBaaS - thus minimizing operational overhead, costs, and additional risk of maintaining your own databases. Atlas is our cloud-hosted MongoDB service engineered and run by the same team that builds the database.

This blog post presents a step-by-step example which automates the deployment of your own standalone AEM Author node along with a MongoDB Atlas replica set in Microsoft Azure. We'll use "development-level" cloud services to keep the costs low, but the steps for production-grade deployments are the same. Simply select beefier servers for your database and AEM instance. And, guess what? We'll actually be able to migrate to more substantial resources without requiring any downtime! How is this possible? MongoDB Atlas supports live cluster migration and AEM supports the ability to host multiple Author nodes.

Getting Ready - Cloud Services Access

Step 0 - 5 minutes

To start let's ensure we have accounts on all the services we need to use. There are 3:

Adobe

Use the Adobe Licensing Website to access the AEM software.

Microsoft Azure

Microsoft Azure will host our AEM server in the cloud.

MongoDB Atlas

For deploying the backing MongoDB replica set. Open MongoDB Atlas and click the green "Get started free" button. You'll actually need to configure your Atlas account payment settings as we'll be deploying a MongoDB cluster that is not free.

If you haven’t already, please use the links above and follow the directions to create your accounts. If you already have accounts for these services, then you’re all set.

Provisioning - Spinning up Cloud Resources

Step 1 - MongoDB Atlas API Access - 5 minutes

We'll start by provisioning our database for AEM to store its content. The goal is to configure a 3 node MongoDB replica set using the M20 Cluster Tier.

Sign into MongoDB Atlas to start managing multiple cloud-based MongoDB deployments. Atlas uses the concepts of Organizations and Projects to help manage these deployments. Each organization can have multiple projects. You can also create teams with various levels of access to each project. The automated deployment presented below requires a set of credentials for API access. Please follow the instructions for configuring API access in the MongoDB Atlas documentation. This is the only manual Atlas step required.

If this is your first visit to Atlas, you'll need to create an organization and a project within that organization. For these, you'll just need to name them something. I used AEM-ON-AZURE and aem-on-azure-db. Note your project name, you'll need to update the deployment script with that name in order to associate the AEM MongoDB database with that group.

Step 2 - Get automation tools and scripts

For this demonstration, we'll be showing some of the great automation capabilities of Azure (MongoDB Atlas also has a rich API along with some open source automation tooling.) To get started, download and install the Azure CLI 2.0: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest Mac users with brew can just run:

brew update && brew install azure-cli

A script which automates launching an AEM Author node can be found in the aem-on-atlas-and-azure Github repository.

git clone https://github.com/jasonmimick/aem-on-atlas-and-azure
cd aem-on-atlas-and-azure

The spin-up-demo-aem-azure.sh script will:

  1. Provision a MongoDB Atlas replica-set.
  2. Create an Azure resource group.
  3. Launch an Ubuntu VM.
  4. Create a public IP and firewall rule for the AEM Author HTTP service.
  5. Upload your AEM jar file and license.properties.
  6. Start the AEM Author node.

We've leveraged an excellent blog our partner Adobe released on running a minimal setup AEM6 on MongoDB for the commands to launch AEM. We're also doing a non-interactive launch which allows us to set the admin password without user intervention (https://helpx.adobe.com/in/experience-manager/6-3/sites/administering/using/security-configure-admin-password.html).

Using your favorite editor, update the variables in the spin-up script.

It's a best practice to use the same region for both the Atlas and Azure components, the script contains 2 variables which define the region (fixed values between Atlas and Azure for "US East" are slightly different formats).

Step 3 - Run spin-up script

You'll just need to login to your Azure account and run the script.

az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AW84CG7T5 to 
authenticate.
./spin-up-demo-aem-azure.sh aem-demo      # DEMO_NAME="aem-demo"

The very last step in the spin-up script starts the AEM author node. Once this is running we can ssh into the new Azure VM and do some inspection. A quick grep in the AEM logs does indeed validate that AEM is connecting to Atlas…

$# Note - put your DEMO_NAME for the 'resourceGroup'
$ssh aem@$(az network public-ip list --query="[?resourceGroup == 'my-aem-demo-61h'] | [].ipAddress" --output tsv)
aem@my-aem-demo-61h-aem-vm:~$grep mongodb ./crx-quickstart/logs/error.log
26.06.2018 16:11:40.133 *INFO* [cluster-ClusterId{value='5b3265bb19c9cf6b565256a1', description='MongoConnection for Oak DocumentMK'}-srv2-shard-00-00-n1oxl.mongodb.net:27017] org.mongodb.driver.connection Opened connection [connectionId{localValue:3, serverValue:46593}] to srv2-shard-00-00-n1oxl.mongodb.net:27017
26.06.2018 16:11:40.137 *INFO* [cluster-ClusterId{value='5b3265bb19c9cf6b565256a1', description='MongoConnection for Oak DocumentMK'}-srv2-shard-00-00-n1oxl.mongodb.net:27017] org.mongodb.driver.cluster Monitor thread successfully connected to server with description ServerDescription{address=srv2-shard-00-00-n1oxl.mongodb.net:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 5]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3391937, setName='srv2-shard-0', canonicalAddress=srv2-shard-00-00-n1oxl.mongodb.net:27017, hosts=[srv2-shard-00-00-n1oxl.mongodb.net:27017, srv2-shard-00-01-n1oxl.mongodb.net:27017, srv2-shard-00-02-n1oxl.mongodb.net:27017], passives=[], arbiters=[], primary='srv2-shard-00-01-n1oxl.mongodb.net:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, lastWriteDate=Tue Jun 26 16:11:37 UTC 2018, lastUpdateTimeNanos=5447761739832831}
26.06.2018 16:11:40.169 *INFO* [FelixStartLevel] org.mongodb.driver.connection Opened connection [connectionId{localValue:4, serverValue:96728}] to srv2-shard-00-01-n1oxl.mongodb.net:27017

Checking the Atlas database with the mongo shell shows that our new AEM Author node is indeed able to connect and write data. Note the use of the mongodb+srv format connection string. This feature allows MongoDB replica set nodes to be discovered through DNS SRV records.

$mongo "mongodb+srv://srv2-n1oxl.mongodb.net/test" --username aem
MongoDB shell version v3.6.3
Enter password:
connecting to: mongodb+srv://srv2-n1oxl.mongodb.net/test
2018-06-28T14:58:40.036-0400 I NETWORK  [thread1] Starting new replica set monitor for srv2-shard-0/srv2-shard-00-02-n1oxl.mongodb.net.:27017,srv2-shard-00-00-n1oxl.mongodb.net.:27017,srv2-shard-00-01-n1oxl.mongodb.net.:27017
2018-06-28T14:58:40.327-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to srv2-shard-00-02-n1oxl.mongodb.net.:27017 (1 connections now open to srv2-shard-00-02-n1oxl.mongodb.net.:27017 with a 5 second timeout)
2018-06-28T14:58:40.333-0400 I NETWORK  [thread1] Successfully connected to srv2-shard-00-00-n1oxl.mongodb.net.:27017 (1 connections now open to srv2-shard-00-00-n1oxl.mongodb.net.:27017 with a 5 second timeout)
2018-06-28T14:58:40.690-0400 I NETWORK  [thread1] Successfully connected to srv2-shard-00-01-n1oxl.mongodb.net.:27017 (1 connections now open to srv2-shard-00-01-n1oxl.mongodb.net.:27017 with a 5 second timeout)
2018-06-28T14:58:40.690-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to srv2-shard-00-01-n1oxl.mongodb.net:27017 (1 connections now open to srv2-shard-00-01-n1oxl.mongodb.net:27017 with a 5 second timeout)
2018-06-28T14:58:40.765-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] changing hosts to srv2-shard-0/srv2-shard-00-00-n1oxl.mongodb.net:27017,srv2-shard-00-01-n1oxl.mongodb.net:27017,srv2-shard-00-02-n1oxl.mongodb.net:27017 from srv2-shard-0/srv2-shard-00-00-n1oxl.mongodb.net.:27017,srv2-shard-00-01-n1oxl.mongodb.net.:27017,srv2-shard-00-02-n1oxl.mongodb.net.:27017
2018-06-28T14:58:42.015-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to srv2-shard-00-00-n1oxl.mongodb.net:27017 (1 connections now open to srv2-shard-00-00-n1oxl.mongodb.net:27017 with a 5 second timeout)
2018-06-28T14:58:42.015-0400 I NETWORK  [thread1] Successfully connected to srv2-shard-00-02-n1oxl.mongodb.net:27017 (1 connections now open to srv2-shard-00-02-n1oxl.mongodb.net:27017 with a 5 second timeout)
MongoDB server version: 3.6.5
MongoDB Enterprise srv2-shard-0:PRIMARY> show dbs
admin       0.000GB
aem-author  0.320GB
config      0.000GB
local       0.936GB
MongoDB Enterprise srv2-shard-0:PRIMARY> use aem-author
switched to db aem-author
MongoDB Enterprise srv2-shard-0:PRIMARY> show tables
blobs
clusterNodes
journal
nodes
settings
MongoDB Enterprise srv2-shard-0:PRIMARY>

Wrap up

The aem-on-atlas-and-azure repository also contains a script to clean things up. The tear-down-aem-on-atlas-and-azure.sh bash script will attempt to read the latest configuration from the log file created by the spin-up script and then delete the corresponding Atlas and Azure resources.

This post reviewed an automated procedure for provisioning a complete AEM Author node hosted in the cloud. We hope you've found this informational and are able to apply these techniques in your own AEM deployments. An additional benefit to these kinds of deployments is that they are automatically integrated with the Atlas and Azure built in monitoring and alerting systems. Are you currently or planning to provision AEM in the cloud within your organization? If so, head over to our MongoDB community Slack, join the #aem channel, and let us know!

MongoDB Enterprise Server for Pivotal Cloud Foundry goes GA

Jason Mimick

Technical

Last fall we launched MongoDB Enterprise Server for Pivotal Cloud Foundry (PCF) in beta. Earlier this year, we released v1.0.1, our first GA version of the integration. This week, we're happy to announce the next version, v1.0.3. In this post will outline all of the new post-beta features we've added to our PCF tile. For information on how the tile works and more background, please check out my colleague's excellent post.

New Features:

  • MongoDB 3.6 and MongoDB Ops Manager 3.6.2+ are now supported
  • Support for Organizations/Projects in MongoDB Ops Manager
  • Stemcell now follows all published MongoDB production best practices
  • Backups can now be enabled by default
  • T-Shirt sizing for PCF cluster VM specifications
  • TLS/SSL for MongoDB deployments using Bosh DNS

Speed: Devs and users care about getting features out quickly. They are reorganizing their teams to work on Strategic Initiatives like microservices. They need a database that supports traditional use cases as well as newer ones. These new features together with PCF help address these concerns. The MongoDB Enterprise Server for PCF Tile allows customers to leverage two powerful solutions for database management simultaneously to enhance their DevOps processes and accelerate development.

Standardization to mitigate risk: The PCF PaaS delivers features such as machine provisioning and initialization, and MongoDB Ops Manager provides runtime database management. Together these solutions allow modern enterprises the ability to ensure consistent configuration, security policies, backup, and monitoring are applied across the board to development, test, and production deployments.

Ease of use: This powerful combination additionally affords application developers the capability to deliver out-of-the-box cloud-ready solutions without needing to worry about complex infrastructure details. If you haven't already, download MongoDB Ops Manager and the latest MongoDB Enterprise Server for PCF Tile today!

MongoDB 3.6 and MongoDB Ops Manager 3.6.2+ are now supported

You must use MongoDB Ops Manager 3.6.2 or later for this version of the tile. The reason for this is the shift from a strict single level hierarchy of "Groups" in MongoDB Ops Manager to the more versatile "Organizations/Projects" structure. Refer to the Ops Manager documentation for more information.

Support for Organizations and Projects in MongoDB Ops Manager

The beta versions of the tile generated a cluster (group) name which wasn't very human-friendly. This version resolves this issue by allowing users to specify a cluster (project) name for their deployment at service provisioning time.

Stemcell now follows all published MongoDB production best practices

We have integrated all of the production best practices defined in the production notes. These best practices ensure that PCF tunes your MongoDB clusters with the optimal operating system settings for maximum performance and least risk.

Backups can now be enabled by default (v1.0.3)

Certain organizations enforce strict policies around backups due to regulatory restrictions. MongoDB Ops Manager offers the ability to backup any MongoDB deployment with point-in-time restores and queryable backups. This version of the tile supports the ability to enable backups by default for all MongoDB clusters. Alternatively, backups can be disabled by default and enabled for a cluster during service provisioning. Organizations no longer need to manually configure backups to ensure all deployments are always backed up for disaster recovery and governance scenarios.

Figure 1: MongoDB Enterprise Server Tile for PCF configuration in PCF Ops Manager

T-Shirt sizing for PCF cluster VM specifications (v1.0.3)

Beta versions of the tile only allowed for a single "VM type" (CPU/RAM/Disk) to be defined for all deployed MongoDB clusters. This version provides the ability to define three different "VM types". These types (Small, Medium, and Large) allow PCF operators to define categories of cluster types and then limit which types specific PCF users are allowed to deploy. The ability to support a wide variety of cluster types and sizes will enable an enterprise to support self-service scenarios and provide MongoDB as a service right out of the box.

TLS/SSL for MongoDB deployments using Bosh DNS

PCF Operators can now deploy MongoDB clusters with TLS/SSL enabled. The tile configuration in PCF Ops Manager now includes a new security tab in which one can enter the appropriate certificates and private PEM key files for database servers and Certificate Authorities. These will then be automatically distributed to each MongoDB server, deployed into known locations which can then easily be entered into the security settings for the corresponding MongoDB Ops Manager project for the deployment.

Conclusion

In our beta releases we focused on core functionality, and now with our GA and subsequent release, we have included some key usability features and other helpful additions. Please download and install the tile, wire it up to your MongoDB Ops Manager instance, and take it for a drive. To get started watch this demo video and refer to our documentation.

Also – check out a recent webinar we delivered with Pivotal for a hands-on look at using the MongoDB Enterprise Service Tile for PCF to refactor legacy monolith applications into microservices: How to Overcome Data Challenges When Refactoring Monoliths to Microservices.

MongoDB Enterprise Running on OpenShift

Jason Mimick

Cloud

Update: May 2, 2018
Our developer preview of MongoDB Enterprise Server running on OpenShift now includes a simple OpenShift Template. The mongodb-openshift-dev-preview.template.yaml template file reduces the complexity and additional requirements of running OpenShift with the --service-catalog enabled and deploying the Ansible Service Broker (not to mention needing to install the apb tool on your development system in order to build and deploy the Ansible Playbook Bundle). Currently the template can provision multiple pods each running an automation agent configured to the same MongoDB Ops Manager Project. You can complete the deployment of a MongoDB replica set with a few quick clicks in the Ops Manager user interface. We hope the removal of these additional dependencies helps you and your organization quickly adopt this modern, flexible, and full featured way to deploy and run MongoDB Enterprise on your OpenShift clusters. And, stay tuned! This is the tip of the iceberg for support of your cloud native workloads from MongoDB.


In order to compete and get products to market rapidly enterprises today leverage cloud-ready and cloud-enabled technologies. Platforms as a Service (or PaaS) provide out-of-the-box capabilities which enable application developers to focus on their business logic and users instead of infrastructure and interoperability. This key ability separates successful projects from those which drown themselves in tangential work which never stops.

In this blog post, we'll cover MongoDB's general PaaS and cloud enablement strategy as well as touch upon some new features of Red Hat’s OpenShift which enable you to run production-ready MongoDB clusters. We're also excited to announce the developer preview of MongoDB Enterprise Server running on OpenShift. This preview allows you to test out how your applications will interact with MongoDB running on OpenShift.

Integration Approach for MongoDB and PaaS

Platforms as a Service are increasingly popular, especially for those of you charged with building "cloud-enabled" or "cloud-ready" applications but required to use private data center deployments today. Integrating a database with a PaaS needs to be done appropriately to ensure that database instances can be deployed, configured, and administered properly.

There are two common components of any production-ready cloud-enabled database deployment:

  • A modern, rock-solid database (like MongoDB).
  • Tooling to enable telemetry, access and authorization, and backups (not to mention things like proactive alerting that integrates with your chosen issue tracking system, complete REST-based APIs for automation, and a seamless transition to hosted services.) For MongoDB, this is MongoDB Ops Manager.

A deep integration of MongoDB Ops Manager is core to our approach of integrating MongoDB with popular PaaS offerings. The general design approach is to use the "separation of concerns" design principle. The chosen PaaS handles the physical or virtual machines, CPU and RAM allotment, persistent storage requirements, and machine-level access control, while MongoDB Ops Manager controls all aspects of run-time database deployments

This strategy enables system administrators to quickly deploy "MongoDB as a Solution" offerings within their own data centers. In turn, enterprise developers can easily self-service their own database needs.

If you haven't already, download MongoDB Ops Manager for the best way to run MongoDB.

MongoDB Enterprise Server OpenShift Developer Preview

Our "developer preview" for MongoDB on OpenShift can be found here: https://github.com/jasonmimick/mongodb-openshift-dev-preview. The preview allows provisioning of both MongoDB replica sets and "agent-only" nodes (for easy later use as MongoDB instances) directly through OpenShift. The deployments automatically register themselves with an instance of MongoDB Ops Manager. All the technical details and notes of getting started can be found right in the repo. Here we'll just describe some of functionality and technology used.

The preview requires access to an OpenShift cluster running version 3.9 or later and takes advantage of the new Kubernetes Service Catalog features. Specifically, we're using the Ansible Service Broker and have build an Ansible Playbook Bundle which installs an icon into your OpenShift console. The preview also contains an example of an OpenShift template which supports replica sets and similar functionality.

A tour and deploying your first cluster:

Once you have your development environment ready (see notes in the developer preview Github repository) and have configured an instance of MongoDB Ops Manager you're ready to starting deploying MongoDB Enterprise Server.

Clusters can be provisioned through the OpenShift web console or via command line. The web console provides an intuitive "wizard-like" interface in which users specify values for various parameters, such as MongoDB version, storage size allocation, and MongoDB Ops Manager Organization/Project to name a few.

Command line installs are also available in which parameter values can be scripted or predefined. This extensibility allows for automation and integration with various Continuous Integration and Continuous Delivery technologies.

A future post will detail cluster configuration and various management scenarios, such as upgrades, performance tuning, and troubleshooting connectivity, so stay tuned.

We're excited to introduce simple and efficient ways to manage your MongoDB deployments with tools such as OpenShift and Kubernetes. Please try out the developer preview and drop us a line on Twitter #mongodb-openshift or email bd@mongodb.com for more information.


Be a part of the largest gathering of the MongoDB community. Join us at MongoDB World.