Real-Time ESG Data Management

Wei You Pan
February 14, 2023 | Updated: June 26, 2023

ESG (Environmental, Social, and Governance) data collection and reporting has become a corporate priority, with over 96% of S&P 500 companies publishing sustainability reports in 2021, according to research from the Governance and Accountability Institute.

There are several factors driving the adoption and use of ESG data; ranging from consumer preference for companies with positive ESG information, to employees, who increasingly believe environmental, social, and governance metrics are important indicators when choosing an employer.

Many government bodies and regulators either have, or are considering, mandatory ESG data collection and ESG data reporting requirements for corporations under their jurisdiction. The European Union is taking the lead here, with several key pieces of legislation either already enacted, or coming soon. In the US, the SEC has also announced proposed rule changes for securities reporting, mandating companies make detailed climate-related disclosures in their filings.

In addition to companies that report on their own data, financial firms, including the private equity industry, use ESG data and research to weigh risks and identify opportunities for the companies they invest in.

Faced with growing scrutiny around ESG reporting and scoring, companies are struggling to meet ever more detailed and comprehensive reporting requirements. At the heart of the problem is the sheer volume and variety of data companies are expected to ingest and analyze to produce the scores that investors, consumers, and government entities demand. And with real-time data making its way into reports, ESG data management is becoming even harder.

ESG data collection and analysis

The volume and variety of ESG data makes collection and analysis difficult. The data collection problem can be broken down as follows:

Variety

Unlike financial datasets, which are mostly numerical, ESG metrics can include both structured and unstructured datasets, like an email or a media report. If a company wants to analyze satellite data to derive their own climate dataset, they may even need to analyze images and videos. Given these variables, companies need to employ a data model that can support many different types of data.

Velocity

As companies increasingly integrate real-time data sources into their ESG scoring systems, the velocity of data collected and analyzed increases exponentially. One example is loan due diligence in the financial sector. As customers demand faster loan approval turnaround times, financial institutions that currently rely on quarterly ESG data to make those decisions now need the information in real-time to instantly approve loans in an ESG compliant manner.

Volume

The increased variety of data sources, coupled with the growing velocity of data being collected leads to an increase in the sheer volume of data requiring analysis. Currently, ESG ratings and scores are derived from a blend of human judgment and model driven quantitative rating. But as the volume of data increases, along with the need for instant analysis of that data, real-time analytics and an increased use of AI/ML tools will become an ever greater part of ESG ratings and reporting.

On top of this, there are also no universally applicable ESG standards, leaving companies having to deal with multiple different standards, with different data requirements, depending on which jurisdictions they operate in.

Real-time ESG data analytics

Companies are increasingly incorporating real-time data into their ESG analysis, reporting, and scoring. Harnessing technologies such as cloud computing, AI, and machine learning, those that utilize real-time data can, for instance, instantly parse breaking news stories for ESG-related data on their investments, or incorporate up-to-the-minute satellite data into reports on a firm’s environmental impact.

The financial services industry in particular is taking a lead on integrating real-time ESG data into investment decisions.

Asset and fund managers use real-time data platforms that allow them to calculate up-to-date ESG scores to aid investment decisions and risk calculations. For example, a bank looking to invest in an electric vehicle company would be alerted to a breaking news story about a hazardous accident at the manufacturer’s battery plant, with follow up data from social media or analyst reports quantifying the size of the public reaction and the level of negative market sentiment around the accident.

MongoDB and ESG data management

MongoDB Atlas is an ideal data foundation for ESG platforms. MongoDB Atlas uses the document data model, giving users the ability to ingest data from almost any source, consolidate data from a number of siloed data sets, enable the easy search of that data, and with a few clicks, create customized views of the data without the need for additional ETL operations to other databases or tools.

MongoDB Atlas also future-proofs your ESG data platform with a flexible data schema that can easily adapt to rapidly changing ESG requirements and standards.

See why Hydrus chose MongoDB Atlas as the basis for its ESG reporting platform.

FAQ

ESG data definition

ESG (Environment, Social, and Governance) data comes from a growing list of sources, all of which help “score” a corporation based on how well positioned it is to handle the risks and opportunities presented by the environment, societal stakeholders, and corporate governance.

Environment - What are a company's greenhouse gas emissions? How about its stewardship over natural resources? And how well positioned is it to weather physical climate risks, like global warming, flooding, drought, fire etc.

Social - How does a company measure up against prevailing fair wage and employee engagement metrics? What impact does a company have on the communities where it operates?

Governance - How well is a company managed? How responsive is a company to shareholders? How accountable is leadership? What safeguards are in place to ensure transparency?

The growing interest around ESG data science and data analytics has prompted the rise of a new industry of ESG data companies and ESG data management software vendors.

What are the different ESG data sources?

ESG data may come from two primary sources; 'inside-out' and 'outside-in'.

Inside-out data is supplied by companies, used for analysis, and usually lags 6-12 months due to annual ESG-related disclosures.

Outside-in data is more regularly updated, sometimes even in real time. Most financial institutions, including banks who often have access to a lot of financial and company data from their customers, do not rely solely on their own data. ESG data analysis requires a broad range of inputs and data that the bank does not possess or can obtain even from their customers.

For example, a bank may want to assess the risk of flooding for a chip manufacturing company that has factories in several provinces in China. The bank would need to collect the flood data from the different operating locations in order to score the risk.

As banks don’t typically collect flood data themselves, the bank would purchase data from third-party climate data vendors. At this nascent stage of climate risk assessment within the banking industry, it is likely that the bank would not even attempt to collect the raw climate data and create the risk models to score the risk, relying instead on third-party risk scoring vendors.

The bank would then make use of these scores and combine in models which they have strong competencies eg. credit risk to come up with flood risk-adjusted credit risk scores for loan approvals.

Why is ESG data essential for investors?

ESG data is used by asset managers and investors for market analysis, supporting asset allocation and risk management, and in providing insights into the long-term sustainability of investments in various corporations.

← Previous

Honoring Black History Month: How These MongoDB Employees Defied the Odds

February is Black History Month. It’s a time to reflect on and celebrate the struggles and triumphs of the black community and remember the importance of elevating black voices. Each year at MongoDB, we ask members of our employee resource group BEAM (Black Employees At MongoDB) if they’d like to share a personal story about their experiences and what this month means to them. This year, hear from Administrative Assistant Rita Henderson and Regional Director Daniel Hawthorne to learn more about their journeys into tech. Rita Henderson: Breaking Down Barriers and Owning Technology for Social Justice As we celebrate Black History Month, I am grateful for those who have paved the way for us to have a voice and fight for our rights. I am reminded of the struggles and achievements of black leaders throughout history. The fight for equal rights and justice is ongoing, and technology plays a crucial role in this fight. It is important to empower and uplift underrepresented communities in the tech industry to create a more inclusive and equitable future. I am a proud member of the Afro-Latinx community from North Philadelphia. Growing up in a neighborhood called Badlands, I witnessed first-hand the impact and struggles of poverty, high crime rates, and drugs. I am the youngest of six children, with parents who worked two jobs to make ends meet. Despite my parents' hard work and dedication to provide for their children, life was still a struggle for my family. At the age of 17, after completing my junior year of High School, I became a teen mom. Unfortunately, society tries to shame young mothers, especially teen moms of color. Many people reminded me that teen pregnancy is closely linked to single parenthood and that growing up in single-parent families remains the largest factor in increased poverty among children. Me (middle) and my sibling with our dad. Yes, I photoshopped myself in. As a teen mom, I was determined to break through the barriers society placed on me. With $200 in my pocket, I moved my daughter and I to western Pennsylvania and enrolled in Indiana University of Pennsylvania. There, I earned my bachelor's degree in Criminology and studied the school-to-prison pipeline in black communities. After the murder of the young unarmed black teenager, Mike Brown, and the Ferguson uprising, my sister and I collaborated with organizers in the Ferguson community to launch a free technology program to empower community organizers, educators, and youth with skill sets to create technology tools for social and economic justice. Graduating from Indiana University of Pennsylvania. Pictured with my firstborn London Rae and my mom. I am influenced by the work of the Black Panther Party; specifically, the 10th Point of the party’s 10-Point Platform, “Community Control of Modern Technology”. For 45 years, the Black Panther Party included the right to learn, access, and control technology as a right. Huey said, "Knowing how to struggle is the essence of winning. Recognizing ills is fundamental; recognizing how to overcome ills is mandatory." That is why I believe it is critical for black and latinx people to understand the role technology plays in our society and the economy if we want to understand social justice and create tools for liberation. When I hear people talk about technology in black, latinx, and working-class communities, they often use it as a scare tactic. The fear of data and control and the feeling that technology is too advanced and that we lack the knowledge and tools to participate can be overwhelming. However, it is crucial for our community to claim our place in the tech world. We need to change our thinking and know there is a place for us, just like there is for anyone else. I am grateful for MongoDB's value "embrace the power of differences" and creating a platform where underrepresented communities can share their stories, bring their ideas to the forefront, and be heard in the tech industry. As we celebrate Black History Month, I am grateful for those who have paved the way for us to have a voice and fight for our rights. I am also thankful for the opportunities I have been given to make a difference in my community and empower others to do the same. With education and technology, we can continue breaking down barriers and striving for equality, justice, and liberation. In 2022, my partner and I welcomed our baby girl Lara Sky. Daniel Hawthorne: Building a Career as a Black Man in Tech Sales I was brought into the world with the odds against me, a black boy born in South Central Los Angeles in the 80s. However, I never felt that I was on my own. Throughout my entire life, God has choreographed my every step. At a very young age, my parents decided to move us to Austin, Texas where my grandparents were moving their church ministry. I was raised in Austin along with my two older brothers (Dante and Derrell), my younger sister (Amber), and my younger brother (Joseph). My siblings called me the “golden child” because I was a mama’s boy and kept to myself. The elementary and middle schools that I attended in Austin were fairly diverse, and I seldomly experienced racism. In the 7th grade, my family moved to a suburb of North Austin that wasn’t as diverse, and racist experiences became much more frequent. It was then that I began to acknowledge that being black brought different treatment. There were moments I embraced my blackness, but others where I was more focused on adapting myself into someone I thought those in my non-diverse environment wanted me to be. In middle school, the place to hang out was the Rec Center. I would run into kids from other schools, and we’d have the basketball gym to ourselves for a bit. Eventually, the older guys would take over the court, but I was good enough that I typically got to play with them. I remember observing them as they entered the gym. They’d be dressed in nice work clothes with Dell badges hanging from their shirts - the Rec Center was only five minutes from the Dell HQ - and that became an early image of what success looked like for me. In high school and college, I started my career in sales with a few small gigs. I enjoyed it because I was typically one of the top sellers no matter what I sold. I even sold women’s shoes at one point! After graduating with my M.B.A, I had no idea what my next move would be. But then, that image of success popped into my head. I focused my attention on getting a sales job at Dell. Despite not having any experience in tech, I knew I could excel. Who knew that 10 years after my days on the Rec Center courts, I would land my first job in tech. I joined the inside sales development team at Dell, and it was one of the most pivotal moments of my career. The job was intense. After a week of training, it was clear that I was the least technical in every room. But, I was determined to not let anyone outwork me. We were required to make over 100 outbound calls per day, but I quickly figured out how to achieve the true objective (10 scheduled virtual demonstrations in a week) in fewer calls. Through my efficiency, I helped form new standards and began to make a name for myself. Being in sales development wasn’t my end goal. I knew I wanted to get into outside sales, so I began building relationships with some of the Dell outside sellers I worked with. During a coaching session with one of my mentors, who was also a minority, he shared some guidance that I wasn’t ready for. He told me that if I truly wanted to be in outside sales, I needed to lose my earrings because professional men didn’t wear them. Even though he and I understood that earrings didn’t define me, his guidance was that being a person of color meant I was already playing from behind, and that I should exhaust all things within my control to create as level a playing field as I possibly could. This theme would continue throughout my career. Similar to when I was a kid in the non-diverse suburbs of Austin, as a black man in tech, I’ve felt heavy pressure to be a certain way to appease others. When I was first getting started, I hardly encountered sales folks that looked like me. I’d attend internal trainings and events where there might be one or two other black sellers out of 200+ people. In many ways, I felt that I was on an island and had to live through trial and error. I had a fear that being ‘too black’ would put me at an even greater disadvantage. I walked the line and was careful about what I said or did. I hardly engaged in extracurricular activities with co-workers, and when I did, I kept my guard up. So much of my energy and effort was exhausted into protecting my brand and trying to avoid negative stereotyping because of the color of my skin. I often think about how much more successful I could’ve been had I not felt obligated to focus on the things that never should’ve mattered. My wife and our two daughters at the apple orchards outside of St. Louis, Missouri. As I stated before, God has led my path in life. Numerous times when I was unsure of the next turn to make, He introduced someone to provide direction. I’m truly grateful for the people who may not have looked like me, but provided me with valuable coaching that helped guide my career in tech. I joined MongoDB to help customers with their data transformations, but I didn’t expect that I would go through a transformation myself. I’ve never felt more empowered to just be myself, and through that, I’ve reached new levels of individual and team accomplishments. I was a direct seller for my first two years with the company, and after receiving coaching from peers and leaders around me, I stepped into management a year ago. This wasn’t necessarily a milestone or goal that I had set out for myself, but I came to the realization that there was tremendous value in helping other sellers (and their families) achieve new levels of success. What better company to step into leadership than at MongoDB. Every company has employee resource groups nowadays, but the intentionality behind those groups at MongoDB is different. Our leadership team has leaned into those difficult, vulnerable discussions, sometimes simply to listen because they knew they didn’t have the answers. Even in those scenarios, they’d come up with relevant action that they could personally be responsible for. Despite the comfort zone I had created over the past 10+ years of watering down my blackness, our Sales team encourages individuality and has brought out the best version of me. It’s helped lift a giant weight off my back. I know I’m no longer starting from behind, and I don’t fear that folks are going to judge me. As I wrap-up my first year in sales leadership, I’ve noticed significant transformation in my personal development, and I’m excited that I get to continue taking on new challenges that will bring discomfort, but instill confidence that I can persevere. As we celebrate Black History Month, I think about the opportunity I have to expose other members of the black community to a profession in sales. Our experiences and our perspectives are highly valued and necessary in order to build a better tech-centric future. We’re passionate about cultivating a culture where people of all backgrounds, identities, and experiences feel valued and heard. Find your next career opportunity at MongoDB.

February 14, 2023
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024