2 results

Leveraging MongoDB Atlas in your Internal Developer Platform (IDP)

DevOps, a portmanteau of “Developer” and “Operations”, rose to prominence around the early 2010s and established a culture of incorporating automated processes and tools designed to deliver applications and services to users faster than the traditional software development process. A significant part of that was the movement to "shift left" by empowering developers to self-serve their infrastructure needs, in theory offering them more control over the application development lifecycle in a way that reduced the dependency on central operational teams. While these shifts towards greater developer autonomy were occurring, the proliferation of public clouds, specific technologies (like GitHub, Docker, Kubernetes, Terraform), and microservices architectures entered the market and became standard practice in the industry. As beneficial as these infrastructure advancements were, these technical shifts added complexity to the setups that developers were using as a part of their application development processes. As a result, developers needed to have a more in-depth, end-to-end understanding of their toolchain, and more dauntingly, take ownership of a growing breadth of infrastructure considerations. This meant that the "shift left" drastically increased the cognitive load on developers, leading to inefficiencies because self-managing infrastructure is time-consuming and difficult without a high level of expertise. In turn, this increased the time to market and hindered innovation. Concurrently, the increasing levels of permissions that developers needed within the organization led to a swath of compliance issues, such as inconsistent security controls, improper auditing, unhygienic data and data practices increased overhead which ate away at department budgets, and incorrect reporting. Unsurprisingly, the desire to enable developers to self-serve to build and ship applications hadn't diminished, but it became clear that empowering them without adding friction or a high level of required expertise needed to become a priority. With this goal in mind, it became clear that investment was required to quickly and efficiently abstract away the complexities of the operational side of things for developers. From this investment comes the rise of Platform Engineering and Internal Developer Platforms (whether companies are labeling it as such or not). Platform engineering and the rise of internal developer platforms Within a developer organization, platform engineering (or even a central platform team) is tasked with creating golden paths for developers to build and ship applications at scale while keeping infrastructure spend and cognitive load on developers low. At the core of the platform engineering ethos is the goal of optimizing the developer experience to accelerate the delivery of applications to customers. Like teaching someone to fish, platform teams help pave the way for greater developer efficiency by providing them with pipelines that they can take and run with, reducing time to build, and paving the way for greater developer autonomy without burdening developers with complexity. To do this, platform teams strive to design toolchains and workflows based on the end goals of the developers in their organization. Therefore, it’s critical for the folks tasked with platform engineering to understand the needs of their developers, and then build a platform that is useful to the target audience. The end result is what is often (but not exclusively) known as an Internal Developer Platform. What is an IDP? An IDP is a collection of tools and services, sourced and stitched together by central teams to create golden paths for developers who will then use the IDP to simplify and streamline application building. IDPs reduce complexity and lower cognitive load on developers - often by dramatically simplifying the experience of configuring infrastructure and services that are not a direct part of the developer's application. They encourage developers to move away from spending excess time managing the tools they use and allow them to focus on delivering applications at speed and scale. IDPs enable developers the freedom to quickly and easily build, deploy, and manage applications while reducing risk and overhead costs for the organization by centralizing oversight and iteration of development practices. An IDP is tailored with developers in mind and will often consist of the following tools: Infrastructure platform that enabled running a wide variety of workloads with the highest degree of security, resilience, and scalability, and a high degree of automation (eg. Kubernetes) Source code repository system that allows teams to establish a single source of truth for configurations, ensuring version control, data governance, and compliance. (eg. Github, Gitlab, BitBucket) Control interface that enables everyone working on the application to interact with and manage its resources. (eg. Port or Backstage) Continuous integration and continuous deployment (CI/CD) pipeline that applies code and infrastructure configuration to an infrastructure platform. (eg. ArgoCD, Flux, CircleCI, Terraform, CloudFormation) Data layer that can handle changes to schemas and data structures. (eg. MongoDB Atlas) Security layer to manage permissions in order to keep compliance. Examples of this are roles-based compliance tools or secrets management tools (eg. Vault). While some tools have overlap and not all of them will be a part of a specific IDP, the goal of platform engineering efforts is to build an IDP for their developers that is tightly integrated with infrastructure resources and services to maximize automation, standardization, self-service, and scale for developers, as well as maximizing security whilst minimizing overhead for the enterprise. While there will be many different terms that different organizations and teams use to refer to their IDP story, at its core, an IDP is a tailored set of tech, tools, and processes , built and managed by a central team, and used to provide developers with golden paths that enable greater developer self-service, lower cognitive load, and reduce risk. How does MongoDB Atlas fit into this story? Developers often cite working with data as one of the most difficult aspects of building applications. Rigid and unintuitive data technologies impede building applications and can lead to project failure if they don’t deliver the data model flexibility and query functionality that your applications demand. A data layer that isn’t integrated into your workflows slows deployments, and manual operations are a never-ending drag on productivity. Failures and downtime lead to on-call emergencies – not to mention the enormous potential risk of a data breach. Therefore, making it easy to work with data is critical to improving the developer experience. IDPs are in part about giving developers the autonomy to build applications. For this reason, MongoDB’s developer data platform is a natural fit for an IDP because it serves as a developer data platform that can easily fit into any team’s existing toolstack and abstracts away the complexities associated with self-managing a data layer. MongoDB’s developer data platform is a step beyond a traditional database in that it helps organizations drive innovation at scale by providing a unified way to work with data that address transactional workloads, app-driven analytics, full-text search, vector search, stream data processing, and more, prioritizing an intuitive developer experience and automating security, resilience, and performance at scale. This simplification and broad coverage of different use cases make a monumental difference to the developer experience. By incorporating MongoDB Atlas within an IDP, developer teams have a fully managed developer data platform at their disposal that enables them to build and underpin best-in-class applications. This way teams won’t have to worry about adding the overhead and manual work involved in self-hosting a database and then building all these other supporting functionality that come out of the box with MongoDB Atlas. Lastly, MongoDB Atlas can be hosted on more cloud regions than any other cloud database in the market today with support for AWS, Azure, and Google Cloud. How can I incorporate MongoDB Atlas into my IDP? MongoDB Atlas’ Developer Data Platform offers many ways to integrate Atlas into their IDP through many tools that leverage the MongoDB Atlas Admin API. The Atlas Admin API can be used independently or via one of these tools/integrations and provides a programmatic interface to directly manage and automate various aspects of MongoDB Atlas, without needing to switch between UIs or incorporate manual scripts. These tools include: Atlas Kubernetes Operator HashiCorp Terraform Atlas Provider AWS CloudFormation Atlas Resources Atlas CDKs Atlas CLI Atlas Go SDK Atlas Admin API With the Atlas Kubernetes Operator, platform teams are able to seamlessly integrate MongoDB Atlas into the current Kubernetes deployment pipeline within their IDP allowing their developers to manage Atlas in the same way they manage their applications running in Kubernetes. First, configurations are stored and managed in a git repository and applied to Kubernetes via CD tools like ArgoCD or Flux. Then, Atlas Operator's custom resources are applied to Atlas using the Atlas Admin API and support all the building blocks you need, including projects, clusters, database users, IP access lists, private endpoints, backup, and more. For teams that want to take the IaC route in connecting Atlas to their IDP, Atlas offers integrations with HashiCorp Terraform and AWS CloudFormation which can also be used to programmatically spin up Atlas services off the IaC integrations built off the Atlas Admin API in the Cloud environment of their choice.. Through provisioning with Terraform, teams can deploy, update, and manage Atlas configurations as code with either the Terraform Provider or the CDKTF. MongoDB also makes it easier for Atlas customers who prefer using AWS CloudFormation to easily manage, provision, and deploy MongoDB Atlas services in three ways: through resources from the CloudFormation Public Registry, AWS Quick Starts, and the AWS CDK. Other programmatic ways that Atlas can be incorporated into an IDP are through Atlas CLI, which interacts with Atlas from a terminal with short and intuitive commands and accomplishes complex operational tasks such as creating a cluster or setting up an access list interactively Atlas Go SDK which provides platform-specific and Go language-specific tools, libraries, and documentation to help build applications quickly and easily Atlas Admin API provides a RESTful API, accessed over HTTPS, to interact directly with MongoDB Atlas control plane resources. Get started with MongoDB Atlas today The fastest way to get started is to create a MongoDB Atlas account from the AWS Marketplace , Azure Marketplace , or Google Cloud Marketplace . Go build with MongoDB Atlas today!

January 4, 2024

Introducing: Atlas Operator for Kubernetes

The MongoDB Enterprise Operator serves to automate and manage MongoDB clusters on self-managed infrastructure. While this integration has provided complete control over self-managed MongoDB deployments from a single Kubernetes control plane, we’re taking it a step further by extending this functionality to our fully-managed database—MongoDB Atlas. We’re excited to introduce the trial version of the Atlas Operator for Kubernetes. The Atlas Operator will allow you to manage all your MongoDB Atlas clusters without ever having to leave Kubernetes. Keep your workflow as seamless and optimized as possible by managing the lifecycle of your cloud-native applications from where you want most. With the trial version of this Atlas Operator, you can provision and deploy fully-managed MongoDB Atlas clusters on the cloud provider of your choice through Kubernetes. This provider is especially important for those seeking to unlock the power of multi-cloud with unique tools and services native to AWS, Google Cloud, and Azure without any added complexity to the data management experience. With this new Atlas Operator, you get the best of all clouds with multi-cloud clusters on Atlas , coupled with the freedom to run your entire stack anywhere, all while managed in one central location. The “trial version” simply means it has all the core functionality to provision fully-managed Atlas clusters, but the bells and whistles are yet to come. In addition to encapsulating core Atlas functionality, it ensures Kubernetes Secrets are created for each database user which allows for easier management of sensitive data. The Atlas Operator also allows you to create IP Bindings so your applications can securely access clusters. If you’re interested in using the trial version of the Atlas Operator today, follow our quickstart guide below to get started! Quickstart Below you’ll find the steps to create your first cluster in Atlas using the Atlas Operator. Note that you need to have a running Kubernetes cluster before deploying the Atlas Operator. Register/Login to Atlas and create API Keys for your Organization. This information together with the Organization ID will be used to configure the Atlas Operator access to Atlas. Deploy the Atlas Operator kubectl apply -f \ https://raw.githubusercontent.com/mongodb/mongodb-atlas-kubernetes/main/deploy/all-in-one.yaml Create a Secret containing connection information from step one. This Secret will be used by the Atlas Operator to connect to Atlas: kubectl create secret generic mongodb-atlas-operator-api-key \ --from-literal="orgId=<the_atlas_organization_id>" \ --from-literal="publicApiKey=<the_atlas_api_public_key>" \ --from-literal="privateApiKey=<the_atlas_api_private_key>" \ -n mongodb-atlas-system Create AtlasProject Custom Resource: cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasProject metadata: name: my-project spec: name: Test Atlas Operator Project projectIpAccessList: - ipAddress: "" comment: "Allowing access to database from everywhere (only for Demo!)" EOF Create AtlasCluster Custom Resource cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasCluster metadata: name: my-atlas-cluster spec: name: "Test-cluster" projectRef: name: my-project providerSettings: instanceSizeName: M10 providerName: AWS regionName: US_EAST_1 EOF (You'll have to wait until the cluster is ready - "status" field shows "ready:true":) kubectl get atlasclusters my-atlas-cluster -o=jsonpath='{.status.conditions[?(@.type=="Ready")].status}' True Create a Secret for the password that will be used to log into Atlas Cluster Database kubectl create secret generic the-user-password \ --from-literal="password=P@@sword%" Create AtlasDatabaseUser Custom Resource (references the password Secret) cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser metadata: name: my-database-user spec: roles: - roleName: "readWriteAnyDatabase" databaseName: "admin" projectRef: name: my-project username: theuser passwordSecretRef: name: the-user-password EOF Shortly the Secret will be created by the Atlas Operator containing the data necessary to connect to the Atlas Cluster. You can mount it into your application Pod and read the connection strings from the file or from the environment variable. kubectl get secrets/test-atlas-operator-project-test-cluster-theuser \ -o=jsonpath="{.data.connectionString.standardSrv}} | base64 -d mongodb+srv://theuser:P%40%40sword%25@test-cluster.peqtm.mongodb.net Stay Tuned for More Be on the lookout for updates in future blog posts! The trial version of the MongoDB Atlas Operator is currently available on multiple marketplaces, but we’ll be looking to make enhancements in the near future. For more information, check out our MongoDB Atlas & Kubernetes GitHub page and our documentation .

April 8, 2021