ESGDS’ AI platform slashes data processing time by 98% with MongoDB Atlas

INTRODUCTION

ESG Data & Solutions (ESGDS) is a fast-growing Indian technology company. It builds tools to enable banks, investors, and other financial groups to track and analyze a company's performance on Environmental, Social, and Governance (ESG) issues.

With a vast range of covered topics, and multiple providers employing different types of methodologies and taxonomies, ESG data sets are notoriously difficult to work with. 

Because these analyses guide critical research and investment decisions, ESGDS developed ESGSure—a bespoke research platform built on MongoDB Atlas—to address the challenge.

THEIR CHALLENGE

Overcoming the relational model limitations to unlock AI scale

ESGSure collects  points from over 20,000 companies and investors—these include  annual reports and corporate filings, news, and client-specific questionnaires. The platform also tracks a range of other publicly available sources, such as news articles, compliance records, sanctions lists, and more. And these come in various formats: videos, PDFs, transactional data in APIs, and more. 

Before moving to MongoDB Atlas, ESGDS relied on several other databases including relational databases such as PostgreSQL and Pinecone for vector search workloads. As the use cases and data sets expanded, ESGDS encountered limitations. 

“Our platform needs to process massive, diverse, and unstructured data sets, so we can then use a combination of large language models (LLMs), real-time data, and vector search capabilities to deliver AI-driven granular, personalized, and actionable insights for investors,” said Arun Doraisamy, Co-Founder and Chief Technology Officer at ESGDS. “We needed more flexibility, to reduce complexity, and do that at scale. This meant moving away from a relational model and onto a database model that fit our needs.”

Several limitations drove ESGDS to seek a new database:

• Lack of flexibility and scalability: Rigid legacy relational databases lacked the schema flexibility required to dynamically store and update ESGDS's rapidly evolving datasets. This resulted in inconsistent insights that hindered analysts' and investors’ ability to make timely and accurate data-driven decisions. Additionally, a lack of elastic scalability throttled ESGDS’ ability to handle continuous data growth, compromising its ambitious expansion plans. 
• Delayed data insights: Stale data is a significant challenge for the ESG data analysis industry—by the time it is collected and analyzed, ESG data can be up to a year old. To add to this challenge, manual ESG data review in ESGDS’s legacy database took an average of 2 to 3 days per company. ESGDS wanted to automate these processes to provide investors with real-time insights.
• Complex security and compliance: ESGDS manages sensitive, private datasets for its clients. Ensuring secure storage, data encryption, and compliance with ESG frameworks and regional requirements, such as GDPR, had become increasingly complex. With expansion into highly regulated countries on its roadmap, ESGDS knew this challenge would become acute.
• Limited global portability: ESGDS needed a data platform that would easily and efficiently power growth plans across Europe, Asia Pacific, and North America. It had to support a reliable, multi-cloud, and multi-region infrastructure. 

“We needed a modern, flexible model with built-in AI capabilities that could meet our complex needs, and keep evolving to support our ambitious growth and diversification goals,” said Doraisamy.

OUR SOLUTION

MongoDB powering AI and data analytics at scale

In 2020, ESGDS migrated its legacy databases to MongoDB Community Edition, breaking free from rigid architectures and fragmented systems. 

“From the start, we chose MongoDB for its unique flexibility,” said Doraisamy. “With MongoDB, we can quickly change data structures and schemas on the fly, eliminating worries about unstructured data and diverse taxonomies.” 

To power the launch of its ESGSure SaaS offering ESGDS moved to MongoDB Atlas, hosted on AWS, in 2023. A fully managed service with well-defined and proven operational best practices, Atlas offers the elastic scale needed to handle seasonal spikes in data processing, and further geographic expansion. Furthermore, by leveraging MongoDB’s time-series capabilities, ESGDS eliminated stale data. 

With MongoDB’s built-in , ESGDS doesn’t need any bolt-on vector databases, and can perform hybrid queries that combine vector and text search. A foundational part of ESGDS’s AI capabilities, MongoDB Atlas Vector Search efficiently retrieves relevant datasets, which are subsequently sent to LLMs such as Anthropic's Claude. 

MongoDB’s dedicated search nodes help improve search retrieval performance under heavy workloads. This ensures a seamless, consistent user experience during bulk data processing. For example, MongoDB’s vector capabilities can  return sub-second results for every search prompt. Such performance is made possible by ‘’ the data to optimise it for rapid LLM retrieval accuracy and processing.

“We looked at other options such as Pinecone and OpenSearch, but choosing MongoDB Atlas Vector Search over a dedicated vector database eliminated the need for complex and costly ETL (Extract, Transform, Load) processes, as the data is vectorized directly within MongoDB,” said Doraisamy. “This is a real advantage to allow performant LLM workflows.”

MongoDB Atlas delivers enterprise-grade security capabilities and the ability to spin up dedicated clusters for clients when needed. It also enables ESGDS to adopt multi-cloud capabilities for greater resilience and deployment options.

OUTCOME

AI doubling ESGDS’s data ingestion capability

Thanks to MongoDB Atlas, ESGDS delivers granular, and highly customizable ESG data at scale and in near real time. Within a year, ESGDS was able to vectorize nearly 400,000 papers on the ESGSure platform. Crucially, MongoDB’s flexible document model accommodates varying client taxonomies and allows for the easy ingestion of diverse data types.

ESGSure has reduced manual data review time for ESG data sets from multiple days to minutes. Additionally, ESGDS delivers nearly twice the volume of ESG datasets as traditional manual workflows. 

By combining near real-time data ingestion with built-in AI capabilities, MongoDB Atlas enables ESGSure to provide accurate, timely datasets—such as carbon emissions calculations for EV companies—without performance bottlenecks. Atlas’s elastic scalability ensures that ESGDS is prepared for continuous growth, seasonal spikes, and geographical expansion beyond India. 

The architectural flexibility provided by MongoDB Atlas and AWS, enables ESGSure to scale globally, adapt to client preferences, and future-proof against shifts in both LLM landscape and ESG regulatory frameworks.

Building on the success of ESGSure, ESGDS launched Scalabl AI to power the next phase of its growth. Built on the same MongoDB-driven architecture and delivery model, Scalabl AI is an AI orchestration platform that automates operational and data workflows. It is used by organizations across a wide range of industries operating in complex, high data volume environments—from supply chain and certification companies to the banking and financial sector—and that need reliable automation, traceability, and faster decision cycles from distributed data.

“When it comes to our growth and geographic expansion, MongoDB’s multi-cloud and multi-region capabilities have been key,” said Kodancha. “The ability to spin off in any region and on any cloud platform at the click of a button is great; it has allowed us to seamlessly expand into Europe, Australia, and the US, with more countries to come.” 

Critically, MongoDB’s flexibility to be hosted on ESGDS customers' preferred infrastructure is a key differentiator. 

“The ability to offer our clients the option to host the platform on their preferred cloud provider—or even on -premise—really makes a difference, particularly with highly regulated sectors like financial services,” explained Kodancha. “With MongoDB Atlas, we can guarantee high security standards, but also compliance with strict requirements such as GDPR.”

The confidence gained from scaling ESGSure on MongoDB enabled  ESGDS to  expand into Scalabl AI, an ROI-first AI execution and orchestration platform. Scalabl AI delivers measurable efficiency and faster turnaround across high-volume workflows—these include financial services, supply chain and compliance data, certification and benchmarking operations, and professional services research and analysis. 

“MongoDB is a core part of our offering. Without MongoDB Atlas, we couldn’t have built a performant, scalable AI platform reaching global markets and multiple industries,” said Kodancha.