MongoDB and Apache Spark at China Eastern Airlines: Delivering 100x Performance Improvements

Mat Keep
July 14, 2016 | Updated: January 20, 2017
#scale#Business

New MongoDB Connector for Apache Spark Enables New Fare Calculation Engine, Supporting 180m Fares and 1.6 billion Queries per Day, Migrated off Oracle.

As one of the world’s largest airlines, China Eastern constantly explores emerging technologies to identify new ways of improving customer experience and reducing cost.

Earlier this year, I interviewed its engineering leaders to learn more about the migration of China Eastern’s flight search platform from Oracle to MongoDB. As a result of the migration, China Eastern achieved orders-of-magnitude performance improvements, and enabled the delivery of application features that had never before been possible.

The next stage of China Eastern’s initiative to transform user experience has been to address airline fare calculations, and serve them reliably, at scale, to the migrated search application.

At this year’s MongoDB World user conference, Chong Huang, Lead Architect at China Eastern Airlines, presented details on their new fare calculation engine built with Apache Spark and MongoDB to New MongoDB Connector for Apache Spark Enables New Fare Calculation Engine, Supporting 180m Fares and 1.6 billion Queries per Day, Migrated off Oracle support more than 1.6 billion queries per day.

The Challenge

Based on averages used across the airline industry, 12,000 searches are needed to generate a single ticket sale. China Eastern currently sells 260,000 airline seats every day, and is targeting 50% of those to come from its online web and online channels. Selling 130,000 seats equates to 1.6 billion searches and fare requests per day, or just under 20,000 searches a second. However, the current fare calculation system built on the Oracle database supports only 200 searches per second. China Eastern’s engineers realized they needed to radically re-architect their fare engine to meet the required 100x growth in search traffic.

The Solution

Rather than calculate fares for each search in real time, China Eastern decided to take a different approach. In the new system, they pre-compute fares every night, and then load those results to the database for access by the search application.

The flight inventory data is loaded into an Apache Spark cluster, which calculates fares by applying business rules stored in MongoDB. These rules compute fares across multiple permutations, including cabin class, route, direct or connecting flight, date, passenger status, and more. In total, 180 million prices are calculated every night! The results are then loaded into MongoDB where they are accessed by the search application.

Why Apache Spark?

The fare engine demands complex computations against large volumes of flight inventory data. Apache Spark provides an in-memory data processing engine that can be distributed and parallelized across multiple nodes to accelerate processing. Testing confirmed linear scalability as CPU cores were added to the cluster. As a Java shop, Apache Spark’s Java API also provides development simplicity for the engineering team.

Why MongoDB?

China Eastern had already implemented a successful project with MongoDB. They knew the database would scale to meet the throughput and latency needs of the fare engine, and they could rely on expert support from MongoDB engineers. The database’s flexible data model would allow multiple fares to be stored for each route in a single document, accessed in one round trip to the database, rather than having to JOIN many tables to serve fares from their Oracle database.

Figure 1: Fare Engine Architecture
Why the MongoDB Connector for Apache Spark?

Just as the project was starting, MongoDB announced the new Databricks-certified MongoDB Connector for Apache Spark.

Using the Connector, Spark can directly read data from the MongoDB collection and turn it into a Spark Resilient Distributed Dataset (RDD), against which transformations and actions can be performed. Internally the Connector uses the splitVector command to create chunk splits, and each chunk can then be assigned to one Spark worker for processing. Data locality awareness in the Connector ensures RDDs are co-located with the associated MongoDB shard, thereby minimizing data movement across the network and reducing latency. Once the transformations and actions have been completed on the RDD, the results can be written back to MongoDB.

The Implementation

Based on performance benchmarking, a cluster of less than 20 Red Hat Linux servers (comprising app servers, and the Spark and MongoDB cluster) are required to meet the demands of 180 million fares and 1.6 billion daily searches. China Eastern is using Apache Spark 1.6 against the latest MongoDB 3.2 release, provisioned with Ops Manager for operational automation. Testing has shown each node in the cluster scaling linearly, and delivering 15x higher performance at 10x lower latency than the previous Oracle based system.

Next Steps

In his MongoDB World presentation, Mr. Huang provides more insight into the platform built with Apache Spark and MongoDB. He shares detailed steps and code samples that show how to download and setup the Spark cluster, how to configure the MongoDB Connector for Apache Spark, the process for submitting a job, and lessons learned along the way to optimize performance.

View the slides from MongoDB World

Download our new whitepaper for examples and guidance on turning analytics into real-time action with Apache Spark and MongoDB.

About the Author - Mat Keep

Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

MongoDB Security Basics For Your Deployments in AWS

Jay Gordon is a Technical Account Manager with MongoDB and is available via our chat to discuss Cloud Manager at cloud.mongodb.com Securing your MongoDB deployment can be done quite simply thanks to using AWS and Cloud Manager. Let’s discuss the process of determining if your AWS deployment has an enhanced security configuration to reduce exposure. If it’s not, we’ll provide you with some tips to properly secure it and ensure your data is not available to the world. We will start with your Cloud Manager deployment in AWS Cloud. The most important part prior to deploying MongoDB is properly configuring your security group. Security Groups are AWS virtual firewalls that allow you to isolate network environments similarly to a Software or Hardware firewall on your EC2 instances. Our documentation page for Firewall Configuration has a list of ports and IPs that need access. Let’s highlight some key points you should hit to improve your network security posture: Outbound Firewall Rules We must allow all outgoing HTTPS (port 443) traffic to pass in order for your Cloud Manager agents to function. Without this, the agents will not be able to communicate back to Cloud Manager. Typically, outbound traffic is not filtered by default. All security groups come with a default 0.0.0.0/0 rule for all outgoing traffic. Prepping your Inbound Firewall Rules This is the part where things can get complicated. Without properly configuring this, we will not be permitted to either access, configure and secure our systems. An empty security group provides you no real access. Before you go to production, you should consider what other systems and people aside from MongoDB Cloud Manager are going to require access. This could be as simple as determining IP addresses of your office VPN, local computer or application servers. By limiting access to your MongoDB deployment to trusted connections only, you reduce your attack surface that can be exploited by remote attackers. Important IPs for Cloud Manager Please note, these IPs can change without notice, so always check our documentation on provisioning and restores for the latest addresses. Auto-Provisioning If you use Cloud Manager provisioning, you must grant SSH access to the following ranges at minimum: 4.71.186.128/25 4.35.16.128/25 Backup SCP Restores If you use SCP as the delivery method when restoring a snapshot, you must grant SSH access to the following ranges at minimum: 4.71.186.128/25 4.35.16.128/25 TCP Ports to Consider All MongoDB processes in a deployment must be accessible to all Cloud Manager agents managing or monitoring processes in that deployment. Therefore, all MongoDB ports must be open to every server that hosts an Automation, Monitoring or Backup Agent. For example, if you are running MongoDB processes on 27000,27017 and 27020, then those three ports must be open from all servers that are hosting an Agent. This example is a minimum ingress ruleset for AWS provisioning and SCP restores. We have also configured our internal network to permit the 27000 - 28000 ports to talk to each other. You can assign multiple security groups to your instances if you’d like to create a separate Security Group to handle your authentication for any external access to the MongoDB systems via ssh or mongo shell. Alerts MongoDB Cloud Manager comes with a monitoring and alerting system. Part of this alerting set will notify you if your MongoDB deployment is exposed to the internet : In the case of a situation like this, Cloud Manager will notify you via whichever integration, email or SMS, you’ve selected. This is just another benefit of using Cloud Manager, the idea that your total overhead can be reduced by allowing our software to replace the typical requirements around monitoring and alerting. If your organization already uses some other form of alerting, you can leverage our Public API which can allow you to hook these alerts into any system you may already be using. User Authentication and SSL Cloud Manager has many other security features built in such as SSL and user authentication which are extremely valuable. Our Automation Agent has the ability to enable these features for you with a few clicks, saving you time and operational overhead. Authentication Once you enable user authentication, we begin the process of automatically managing passwords for your application role users for backups and system management on your behalf. You also have the ability to use roles to create users with different sets of permissions in logical groups. Authentication methods available include SCRAM-SHA-1, Kerberos, LDAP and X.509 client certificate. SSL Some applications can support the ability to connect to a running MongoDB deployment that utilizes SSL to encrypt data in-transit between the application and the database. Tools must provide users the ability to connect with or without an SSL certificate. Users are able to supply a CA or self-signed certificate and connect to the MongoDB deployment. MongoDB security best practices can assist in protecting your data from those who may seek to access it without your permission. Some of these tasks could include support overhead for your team during the configuration process if choosing to do manually. Thankfully Cloud Manager exists to help you implement these best practices with simplicity and speed.

July 13, 2016
Next →

Women Leaders at MongoDB: Lena Smart Discusses Clarity and Goal Setting

March is Women’s History Month. Our women leaders series highlights MongoDB women who are leading teams and empowering others to own their career development and build together. Lena Smart, Chief Information Security Officer, explains why words matter, shares her thoughts on leadership, and discusses MongoDB’s internal mentorship program “MentorHER”. Tell me a bit about your team. My team is responsible for all aspects of security of MongoDB’s global offices and employees. Within my organization I have Governance, Risk and Compliance (GRC) and InfoSec (security engineering, physical security, etc.) under me. We are dedicated to making every effort to protect customer data, including continually improving security processes and controls. On top of that, we are committed to delivering the highest levels of standards conformance and regulatory compliance as part of our ongoing mission to address the most demanding security and privacy requirements of our customers. My goal is to build the best security and GRC team in the world. What characteristics make a good leader? In my opinion, good leaders are decisive and leave little room for ambiguity. They are understanding and know that people are depending on them for their careers, dreams, and aspirations. They make their work matter every day, are focused on continuous learning, and do not “rest on their laurels.” What has your experience been like as a woman growing your career in leadership? My experience as a woman leader has depended on the environment. It was very difficult as a CIO and CISO in the power industry. Every day I felt like I was being undermined by my peers, who (because they were all “power industry engineers”) felt they were the experts in everything to do with security (they were not). It was exhausting. I finally left and joined a FinTech company. That was better, but I still felt I could find an environment where women were actively encouraged to lead. Hence my move to MongoDB. I could not be happier here and love working with all our teams. Tell us about the biggest lesson you’ve learned throughout your career. The biggest lesson I’ve learned is that words matter. As a leader, people can interpret your words in many ways. Be clear in your message. I have a mantra on our team: “one voice, one message”. I encourage my team members to have all the internal discussion they want, but we do not ever air our dirty laundry in public. We stand with “one voice, one message”. People in general like clarity, and we try hard to enforce that idea within my team by encouraging each person to own what they do. What’s your advice for building and developing a team? I believe cultivating a supportive and positive team culture stems from the top down. I really embrace and follow MongoDB’s company values . Build Together is the main value I follow because, to me, people are everything. You need the right people in the right places owning what they do. I am also a huge advocate for people taking the initiative and building upon their own careers. I make sure to set aside a budget for training and certification programs for my team. This allows them to enhance their knowledge and helps them grow and develop into even stronger security and GRC professionals. I also started the Security Champions Program at MongoDB almost four years ago, a volunteer-based program that allows anyone who has an interest in security to join monthly meetings to learn more. Can you tell us a bit about the MentorHER program at MongoDB? I am honored to be the Executive Sponsor for a very important internal program called MentorHER. MentorHER aims to create diverse teams, develop female leaders, drive organizational changes, and enhance MongoDB’s reputation as an employer of choice. I’ve had a couple of mentors who made a positive impact on my career. I cherished our time together and made sure to have a clear understanding of the mentoring program I signed up for. There were goals, regular meetings, and a lot of positivity generated by mentoring. I hope we can replicate that at MongoDB with our MentorHER program. We have a very strong team leading the program, and I feel very confident that we will meet our goals and embrace the different experiences and perspectives of the women around us. What is your advice to women looking to grow their careers as leaders? My advice to other women is this: be clear and honest in what you want from a leadership role. At the C-suite level you will be pulled in many directions. Control that, from the start, where possible. It’s important to be intellectually honest and have clear goals that will help your team grow and mature, and the business flourish. Join a team that builds together every day. View open career opportunities at MongoDB.

March 27, 2023