Enhancing the MongoDB Atlas Go SDK with Automated Code Generation

The MongoDB Atlas API Experience team is committed to offering a seamless developer experience to customers who build and automate against the MongoDB Atlas developer data platform. We offer various programmatic tools, such as the Atlas CLI and the Terraform Atlas Provider, and maintain the Atlas Admin API with key features, including versioning and Open API specification.

Our latest offering, the MongoDB Atlas Go SDK, empowers developers to manage Atlas using Go. It eliminates the need for low-level Admin API calls by utilizing higher-level abstractions. New SDK versions are automatically generated on each Atlas update, ensuring access to the latest Atlas administrative capabilities.

In this post, we’ll discuss the advantages of using an SDK over direct API calls, explore our decision for code generation over manual development, share insights from engineering challenges, review our adoption experience, and discuss some next steps for the Atlas SDKs.

Benefits of using an SDK over direct API calls

Direct API calls offer flexibility and fine-grained control but become cumbersome for complex interactions. Developers manually handle tasks like authentication, error handling, and response parsing, which can be time-consuming and error-prone. Additionally, keeping up with API updates can require regular low-level code updates.

The Atlas Go SDK simplifies development with higher-level abstractions. Pre-built functions and structs encapsulate API interactions, reducing boilerplate code. This frees developers to focus on core application logic rather than API integration intricacies.

The SDK automates authentication elements, error handling, and response parsing, reducing boilerplate code and errors. Plus, staying up-to-date becomes low-effort, as new SDK versions are auto-generated with each Atlas release. While leveraging new functionalities might require some code updates, the SDK significantly simplifies integration compared to direct API calls.

Moving from a manual client to an auto-generated SDK

Previously, MongoDB’s bespoke Go Client for MongoDB Atlas powered internal tools and customer applications. However, maintaining a manually written client increased toil and impacted the team’s productivity. Keeping code quality high required constant effort, leading to a backlog of unsupported Atlas features and a struggle to keep pace with new functionalities.

As Go-based Atlas applications proliferated, the limitations of manual maintenance became clear. We needed a more scalable solution.

We had previously built a mechanism to automatically generate the Open API specification for the Atlas Admin API. This machine-readable document details how to interact with the Atlas functionality through its REST API, accessible through the “Download” option in the MongoDB Atlas Administration API portal. This led us to explore leveraging that specification for client code generation as well.

Building on this foundation, the Atlas Go SDK leverages the Open API spec and a robust delivery pipeline to streamline client generation. The pipeline scans for changes in the spec and triggers the generation of a new SDK version upon detection, producing a pull request as an intermediate output. Following our review and merge, the pipeline requires no other manual steps to continue working on the release process, ultimately publishing a new SDK version.

Code generation has improved output consistency and speed over manual development, freeing up resources for other impactful projects. To showcase the time savings, adding support for a new resource in the old manual Go Client could take roughly two engineer days. The new SDK reduces that time to a less-than-an-hour code review, as all other steps are automated.

Major challenges and solutions

Transitioning to an auto-generated SDK wasn’t without its hurdles. Selecting the right tooling was crucial, and we explored both commercial and open-source solutions for Open API-based code generation. We opted for openapi-generator due to its open-source nature. The choice prioritized flexibility and control, ensuring long-term project autonomy. It also allowed us to open-source the Go SDK generation codebase, fostering community contributions and improvements.

Leveraging the Atlas Open API specification, originally intended for documentation, presented unique challenges. While this approach offered a single source of truth, we encountered discrepancies between the spec and actual API behavior. Notably, minor, non-breaking API changes sometimes resulted in significant, breaking changes in the generated client. For instance, marking a field as optional in the API (non-breaking) turns the equivalent Go model property into a pointer (breaking).

This discovery necessitated a two-fold solution. We addressed the root cause by improving the Open API specification, ensuring better alignment with code generation requirements. Then we developed an automated transformation process to optimize the spec for generating code. This transformed version remains optimized for code generation while the original spec continues to serve its original purpose of live API documentation.

Having a pull request review as part of the release process has helped us ensure quality. In some early cases, it allowed us to catch issues, apply SDK-wide improvements, and add linting rules to prevent reoccurrence.

Maintaining compatibility with the existing Go client was crucial for an easy migration experience. However, achieving perfect compatibility wasn’t always feasible due to inconsistencies in the handwritten client. We meticulously evaluated each change, balancing compatibility with the benefits of code generation. We also created a migration guide and best practices to aid with migrating existing applications.

Finally, to minimize confusion and client bloat, we decided that each Go SDK release should target a single Admin API version. That simplified integration but created a versioning challenge: Go’s semantic versioning uses major bumps for breaking changes. We needed to communicate the targeted Atlas API version within the SDK version while simultaneously denoting breaking SDK changes unrelated to API updates. Our solution was to develop a unique major versioning scheme for the Go SDK. It incorporates the Admin API’s date as a prefix, with additional digits signaling breaking SDK changes for that specific API version. While unconventional, it adheres to semantic versioning and keeps developers informed.

Adoption experience

Our Atlas CLI tool was the proving ground for the new Go SDK. In 2023, it became the first application to migrate from the manual Go client by leveraging pre-release versions of the Go SDK. That staged adoption approach yielded significant benefits. By migrating early, we uncovered valuable improvement areas for the SDK, directly influencing its architecture. This early feedback loop also provided crucial insights into effective SDK generation.

The migration wasn’t without its complexities. It addressed cross-cutting issues across three distinct layers: the Open API spec, the SDK generation, and the CLI integration, each presenting unique development challenges. However, this comprehensive approach ensured that all layers benefitted from the process, and we raised quality across the board.

Following the successful CLI integration, we confidently expanded the SDK’s reach across our product portfolio. Now, it serves as a prominent and trusted middleware solution, not only for the Atlas CLI, but also for several Atlas DevOps tools. Recent additions include the Atlas integrations for AWS CloudFormation and CDK. The Terraform Atlas Provider and the Atlas Kubernetes Operator are also on the immediate roadmap, further solidifying the SDK as a core component within our DevOps ecosystem.

External adoption of the Atlas Go SDK is also gaining momentum. Over 30% of customer-developed Go-based interactions with the Atlas Admin API now leverage the new SDK, a trend we see increasing monthly.

Looking ahead

As Atlas continues to scale, the pace of features will only continue to accelerate. Our primary focus for the Go SDK is to continuously expose new Atlas features as they are released in the Admin API while introducing quality-of-life improvements for developers at every opportunity. We strive to reduce boilerplate code, improve Error Handling, simplify Authentication / Authorization, and to enrich documentation with high-quality examples.

The success of code generation has us exploring what else can be automated. Our team is looking at which other offerings could benefit from automation to free up development time for impactful projects that might be less readily automatable.

Finally, we understand that our developer community thrives on various languages and technologies. Not using Go? Let us know what other languages you’d like to see supported (e.g. Python, Java, TypeScript) by sharing your feedback.

Conclusion

The Atlas Go SDK empowers Go developers to streamline interactions with the Atlas cloud platform. By leveraging code generation and a focus on developer experience, the SDK offers several advantages over manual API calls.

  • Reduced complexity: pre-built functions and higher-level abstractions simplify development by tackling authentication, error handling, and response parsing.

  • Improved maintainability: auto-generation of updated SDK versions with each Atlas release ensures access to the latest functionalities, minimizing manual code changes.

  • Enhanced reliability: Tailored API models promote code reliability and catch potential errors at compile time.

Our in-house adoption experience was a valuable proving ground, influencing the SDK’s development and uncovering key optimization points. Today, the SDK is a cornerstone within our DevOps ecosystem, accelerating the development of downstream DevOps tools.

This successful transition proved the value of code generation for developer productivity and code quality. It also opened up the possibility of generating SDKs for more programming languages in the future.

The Atlas Go SDK can be a valuable asset for Go developers who build solutions on Atlas. To get started with the SDK, see our Docs Page. We also welcome community contributions, so visit our Contributing Guidelines for more details.

We invite you to try it, and we would love to hear your feedback. Go build with the new Atlas Go SDK today!

← Previous

What’s New From MongoDB at Microsoft Build 2024

This week, thousands of engineers, database administrators, and developers are gathering in Seattle for Microsoft Build , Microsoft’s annual developer event. In addition to being on site for meetings and learning sessions, MongoDB is excited to showcase our latest innovations for building generative AI apps and more. First, we’re happy to announce that MongoDB now offers dedicated Search Nodes on Microsoft Azure . We offer both compute-optimized nodes for text or application search workloads, and memory-optimized nodes for vector, semantic search, or gen AI workloads. Search Nodes enhance performance and availability through workload isolation while reducing architectural complexity. The availability of Search Nodes on Azure is the latest example of how the partnership between MongoDB and Microsoft helps organizations of all sizes boost developer productivity and build modern applications faster. Keep reading for more on how MongoDB’s capabilities and integrations with Microsoft are helping customers create, innovate, and scale applications. Integrating services and technology to speed AI development The last year of AI innovation set a clear imperative for every organization—to meet customer expectations, they need to modernize their applications. However, many companies aren’t sure where to start with AI, so MongoDB recently announced the launch of the MongoDB AI Application Program (MAAP) alongside industry-leading AI partners. MAAP will provide customers with strategic advisory, professional services, and an integrated end-to-end technology stack from MongoDB and key partners like Microsoft. We’ve also made several technology announcements to enable building gen AI applications, including native support for MongoDB Atlas Vector Search in Microsoft Semantic Kernel , and a dedicated MongoDB Atlas integration for OpenAI’s ChatGPT Plugin . With the new integration, developers can seamlessly and securely enhance the power of large language models from OpenAI, Azure OpenAI, and Hugging Face with proprietary data in Atlas to build powerful retrieval-augmented generation applications using Python or C#. Developing faster with intelligent tools and frameworks In addition to helping developers build powerful gen AI applications through services like Atlas Vector Search, we’ve been working to enhance developer productivity, making it easier than ever to build applications with MongoDB. For example, we’ve introduced intelligent features to first-party tools like MongoDB Compass and Atlas Charts that support natural language. We also recently announced AI-powered SQL query conversions in Relational Migrator to help teams easily move their workloads to MongoDB. MongoDB is expanding the use of AI to enhance its integration with the world’s most popular integrated development environment, Visual Studio Code. We’re excited to announce the MongoDB Participant for the Github CoPilot chat experience, designed to empower developers to generate queries from natural language, understand collection schemas, and instantly access MongoDB documentation. Sign up for the private preview here . MongoDB also supports a variety of programming frameworks to improve productivity and accelerate application development—while ensuring data consistency and quality. Now generally available, the MongoDB Provider for Entity Framework Core (EF Core), encourages C# developers to build their next project on MongoDB. This new offering helps C# developers—39% of whom use EF Core—unlock the full power of MongoDB using the EF Core APIs and design patterns they already know and love. Streamlining comprehensive data analysis For years, MongoDB and Microsoft have facilitated the large-scale analysis of application-generated data to aid business development. Tools like Microsoft Power BI provide a comprehensive view of business intelligence data for developers and analysts with complex data estates using relational databases alongside MongoDB. MongoDB’s Power BI Connector for Atlas —previously supporting Import Mode—now also supports DirectQuery, which we announced a few weeks ago at MongoDB.local NYC . This allows for real-time querying of MongoDB data and is ideal for large datasets. To further enable customers working in the Microsoft ecosystem, we’ve recently made Atlas Data Federation and Atlas Online Archive generally available on Azure . These services enable users to query, transform, and create views across multiple Atlas databases and Azure cloud storage solutions, like Blob Storage and Data Lake Storage Gen2, simplifying data management and archiving within the Azure ecosystem. Building the future together MongoDB's partnership with Microsoft has made developing modern applications faster and easier. We're thrilled to announce these new capabilities at Microsoft Build 2024 , and look forward to helping our joint customers build amazing things together this year. “MongoDB’s relationship with Microsoft has never been better, and with these latest integrations, our momentum continues to grow,” said Alan Chhabra, MongoDB’s EVP of Worldwide Partners. “Already, many of the largest enterprises and most advanced startups in the world run MongoDB Atlas on Microsoft Azure. These latest innovations will empower even more of our customers to take full advantage of their data to build truly transformational generative AI-powered applications.” MongoDB’s partnership with Microsoft sets projects up for success today and tomorrow by delivering robust, integrated solutions that cater to developers' needs. MongoDB and Microsoft are pushing the boundaries for innovation and service for the developer community. To learn more about our recent announcements and for the latest product updates, visit our What’s New page. And head to our campaign page to learn more about how to build smarter and develop faster with MongoDB Atlas on Microsoft Azure.

May 21, 2024
Next →

AI-Powered Media Personalization: MongoDB and Vector Search

In recent years, the media industry has grappled with a range of serious challenges, from adapting to digital platforms and on-demand consumption, to monetizing digital content, and competing with tech giants and new media upstarts. Economic pressures from declining sources of revenue like advertising, trust issues due to misinformation, and the difficulty of navigating regulatory environments have added to the complexities facing the industry. Additionally, keeping pace with technological advancements, ensuring cybersecurity, engaging audiences with personalized and interactive content, and addressing globalization issues all require significant innovation and investment to maintain content quality and relevance. In particular, a surge in digital content has saturated the media market, making it increasingly difficult to capture and retain audience attention. Furthermore, a decline in referral traffic—primarily from social media platforms and search engines—has put significant pressure on traditional media outlets. An industry survey from a sample of more than 300 digital leaders from more than 50 countries and territories shows that traffic to news sites from Facebook fell 48% in 2023, with traffic from X/Twitter declining by 27%. As a result, publishers are seeking ways to stabilize their user bases and to enhance engagement sustainably, with 77% looking to invest more in direct channels to deal with the loss of referrals. Enter artificial intelligence: generative AI-powered personalization has become a critical tool for driving the future of media channels. The approach we discuss here offers a roadmap for publishers navigating the shifting dynamics of news consumption and user engagement. Indeed, using AI for backend news automation ( 56% ) is considered the most important use of the technology by publishers. In this post, we’ll walk you through using MongoDB Atlas and Atlas Vector Search to transform how content is delivered to users. The shift in news consumption Today's audiences rarely rely on a single news source. Instead, they use multiple platforms to stay informed, a trend that's been driven by the rise of social media, video-based news formats, and skepticism towards traditional media due to the prevalence (or fear) of "fake news." This diversification in news sources presents a dilemma for publishers, who have come to depend on traffic from social media platforms like Facebook and Twitter. However, both platforms have started to deprioritize news content in favor of posts from individual creators and non-news content, leading to a sharp decline in media referrals. The key to retaining audiences lies in making content personalized and engaging. AI-powered personalization and recommendation systems are essential tools for achieving this. Content suggestions and personalization By drawing on user data, behavior analytics, and the multi-dimensional vectorization of media content, MongoDB Atlas and Atlas Vector Search can be applied to multiple AI use cases to revolutionize media channels and improve end-user experiences. By doing so, media organizations can suggest content that aligns more closely with individual preferences and past interactions. This not only enhances user engagement but also increases the likelihood of converting free users into paying subscribers. The essence of leveraging Atlas and Vector Search is to understand the user. By analyzing interactions and consumption patterns, the solution not only grasps what content resonates but also predicts what users are likely to engage with in the future. This insight allows for crafting a highly personalized content journey. The below image shows a reference architecture highlighting where MongoDB can be leveraged to achieve AI-powered personalization. To achieve this, you can integrate several advanced capabilities: Content suggestions and personalization: The solution can suggest content that aligns with individual preferences and past interactions. This not only enhances user engagement but also increases the likelihood of converting free users into paying subscribers. By integrating MongoDB's vector search to perform k-nearest neighbor (k-NN) searches , you can streamline and optimize how content is matched. Vectors are embedded directly in MongoDB documents, which has several advantages. For instance: No complexities of a polyglot persistence architecture. No need to extract, transform, and load (ETL) data between different database systems, which simplifies the data architecture and reduces overhead. MongoDB’s built-in scalability and resilience can support vector search operations more reliably. Organizations can scale their operations vertically or horizontally, even choosing to scale search nodes independently from operational database nodes, flexibly adapting to the specific load scenario. Content summarization and reformatting: In an age of information overload, this solution provides concise summaries and adapts content formats based on user preferences and device specifications. This tailored approach addresses the diverse consumption habits of users across different platforms. Keyword extraction: Essential information is drawn from content through advanced keyword extraction, enabling users to grasp key news dimensions quickly and enhancing the searchability of content within the platform. Keywords are fundamental to how content is indexed and found in search engines, and they significantly influence the SEO (search engine optimization) performance of digital content. In traditional publishing workflows, selecting these keywords can be a highly manual and labor-intensive task, requiring content creators to identify and incorporate relevant keywords meticulously. This process is not only time-consuming but also prone to human error, with significant keywords often overlooked or underutilized, which can diminish the content's visibility and engagement. With the help of the underlying LLM, the solution extracts keywords automatically and with high sophistication. Automatic creation of Insights and dossiers: The solution can automatically generate comprehensive insights and dossiers from multiple articles. This feature is particularly valuable for users interested in deep dives into specific topics or events, providing them with a rich, contextual experience. This capability leverages the power of one or more Large Language Models (LLMs) to generate natural language output, enhancing the richness and accessibility of information derived from across multiple source articles. This process is agnostic to the specific LLMs used, providing flexibility and adaptability to integrate with any leading language model that fits the publisher's requirements. Whether the publisher chooses to employ more widely recognized models (like OpenAI's GPT series) or other emerging technologies, our solution seamlessly incorporates these tools to synthesize and summarize vast amounts of data. Here’s a deeper look at how this works: Integration with multiple sources: The system pulls content from a variety of articles and data sources, retrieved with MongoDB Atlas Vector Search. Found items are then compiled into dossiers, which provide users with a detailed and contextual exploration of topics, curated to offer a narrative or analytical perspective that adds value beyond the original content. Customizable output: The output is highly customizable. Publishers can set parameters based on their audience’s preferences or specific project requirements. This includes adjusting the level of detail, the use of technical versus layman terms, and the inclusion of multimedia elements to complement the text. This feature significantly enhances user engagement by delivering highly personalized and context-rich content. It caters to users looking for quick summaries as well as those seeking in-depth analyses, thereby broadening the appeal of the platform and encouraging deeper interaction with the content. By using LLMs to automate these processes, publishers can maintain a high level of productivity and innovation in content creation, ensuring they remain at the cutting edge of media delivery. Future directions As media consumption habits continue to evolve, AI-powered personalization stands out as a vital tool for publishers. By using AI to deliver tailored content and to automate back end processes, publishers can address the decline in traditional referrals and build stronger, more direct relationships with their audiences. If you would like to learn more about AI-Powered Media Personalization, visit the following resources: AI-Powered Personalization to Drive Next-Generation Media Channels AI-Powered Innovation in Telecommunications and Media GitHub Repository : Create a local version of this solution by following the instructions in the repository

June 13, 2024