Lessons from the Delta Outage: See Who is Learning from Costly Airline Disruptions

Before the dust of the massive disruption to Delta Air Line’s global service had fully settled, Reuters published a somewhat pessimistic look at the IT infrastructure of today’s top airlines.

As reported around the world, Delta suffered a power loss in their Atlanta data center, causing service to be cancelled across their network of over 15,000 daily flights. In the technology community, the fact that a seemingly insignificant occurrence (which turned out to be a small fire) could trigger total system failure was an unpleasant reminder of how fragile these aging systems can be.

However, this was not an isolated occurrence. Just weeks ago, Southwest Airlines was left with no alternative than to cancel over 2,000 flights in the wake of a network failure. In 2015, a malfunctioning router halted United Continental flights for two hours.

News outlets reporting on these outages are quick to place blame on ageing infrastructure. Reuters claims that most major US airlines rely on Transaction Processing Facility, an IBM mainframe operating system created in the 1960s for core processes like reservations. While this is certainly the case, lack of infrastructure investment is not negligence as they may imply, but a symptom of the "razor thin” profit margins over the past decades.

More recently, however, falling oil prices have proven to be a boon for those margins. With more cash on hand, airlines will start looking to invest in more wholescale technology overhauls, rather than smaller projects that have a more immediate return on investment like passenger mobile apps, check-in kiosks, and luggage tracking.

Signs of this overhaul are already visible. After decades of outsourcing its core reservations and passenger services system, Delta only brought its core infrastructure in-house in 2014. Southwest Airlines announced it was overhauling its reservations system early this summer by partnering with Amadeus, an IT solutions provider for the travel industry and MongoDB customer.

The second largest airline in China, and the world’s 8th largest carrier by passenger volume, recently replatformed their fare calculation engine with MongoDB. Originally built on an outdated relational database, the old system struggled to provide up-to-date fare data to newer customer-facing systems. Not only were they able to service thousands of more customers every second and deliver near real-time information to passengers at scale, China Eastern Airlines was able to achieve fault resilience by using MongoDB replica sets, which distribute the database across multiple nodes to eliminate a single point of failure.

However, massive system overhauls have been stiffly resisted by some out of fears that upgrades will break more modern systems built on top of the 1960s technologies, or that maintenance work will take mission critical systems offline for too long. Newer database technologies that were designed for distributed, always-on deployments may provide the answer to these concerns.

Rather than rebuilding platforms from scratch, some in the industry are opting to solve the issues beleaguering legacy systems by designing parallel systems using more modern technologies. The 5th largest airline in the world recently achieved a “single view of their customer” by extracting 100TB of data from their legacy systems into a single data lake, and loading it into MongoDB using a common data model. This allowed for a consistent schema for analytics and advanced personalization to draw from, all while leaving legacy systems intact.

The fact remains, as any frequent flyer will tell you, flight schedules can be derailed despite an air carrier’s best efforts. The world’s 3rd largest airline is now better-poised to handle these unexpected disruptions after rebuilding its seat re-accommodation app on MongoDB. With its legacy Oracle application, it could take 5 minutes to provide rebooked seats to passengers from cancelled flights. MongoDB’s flexible data model allowed for additional passenger information to be stored in the database, eliminating the need for feeds from other systems. Now, seat reassignments take seconds.

In a highly competitive industry prone to all manner of existential threats (from severe weather to terrorism to data center failures), it appears that the cost-benefit analyses that have for so long been stacked against technology upgrades may be slowly shifting to favour more modern, agile, and fault tolerant systems.

To learn more about how a database can make a difference in your business, download our white paper:

Quantifying Business Advantage: The Value of Database Selection


← Previous

Avoiding the Dark Side of the Cloud: Platform Lock-In

Date Your Cloud Provider….But Don’t Marry Them The rise of cloud computing is indisputable – driven primarily by the promise of agility in bringing new applications to market faster, and by more closely aligning expense with actual business usage. But moving to the cloud is not without risk. Many surveys, such as the MongoDB Cloud Brief discussed later, point at the fear of platform lock-in as one of the top inhibitors to on-going cloud adoption. Enterprises are turning to open source software to throw off the shackles of proprietary hardware and software. But they are also concerned about exposing the business to a new level of lock-in. This time from APIs and services of the cloud providers themselves. In this blog, we explore the drivers and inhibitors of cloud adoption, as well as which factors are driving the fear of cloud lock-in. We’ll then discuss the steps users can take to get the best of both worlds – the business velocity provided by the cloud, without the risks of locking themselves into a specific vendor. Growing Cloud Adoption So how quickly is the cloud growing? Recent analysis of the Infrastructure-as-a-Service (IaaS) market by IDC (1) provides some interesting statistics: Spending on public cloud platforms is expected to reach $23bn by the end of 2016, representing just under 20% growth over 2015. Private cloud spending is expected to reach $13bn over the same period, representing 10% growth. If we contrast this with spending on “traditional” IT infrastructure, we see a forecast decline of 4.5% through 2016. Now is not a good time to be a peddler of premium IT hardware. By 2020, IDC expects total IaaS cloud spending to hit just under $60bn, making revenues (almost) as large as the traditional IT infrastructure sector. Of course the cloud is a natural home for startups building their businesses. I’m old enough to remember when early seed funding was dedicated purely to financing your own Sun hardware and Oracle software licenses so that you could actually demo your new concept. The thought of doing this today is laughable. But it’s not just startups that are driving cloud growth. Research from RightScale (2) concluded that 17% of enterprises now have over 1,000 Virtual Machines (VMs) provisioned to public cloud providers, up from 13% of enterprises in 2015. Private cloud showed even stronger growth with 31% of enterprises running more than 1,000 VMs, up from 22% in 2015. Here at MongoDB, we’ve conducted our own research as we polled over 2,000 members of the MongoDB community. This research found that 82% of respondents were strategically using or evaluating the cloud today. This, and a multitude of other fun facts and insights are available in our MongoDB 2016 Cloud Brief . So What are the Top Drivers, and the Top Inhibitors of Cloud Adoption? As the Cloud Brief shows, the number one driver for cloud adoption is agility – the need to rollout new applications faster. This desire was reinforced at a recent meeting I had in London with developers from a leading global financial institution. They complained it takes three months for hardware supporting a new project to be procured, installed, racked, and stacked. Clearly unacceptable in today’s hyper-competitive market governed by agile development, continuous integration, and elastic scaling. This need for application agility was the top cited reason for cloud adoption across organizations of all sizes – from those with less than 50 employees to enterprises with more than 5,000. It was also the top reason for cloud adoption cited across all job titles – from the CIO through to architects, developers and DBAs. The Cloud Brief shows another interesting statistic. The majority of respondents use more than one cloud provider. This was primarily driven by the need to take advantage of specific features offered by one provider over another, and this clearly demonstrates the need to remain flexible in your cloud choices. Hitching your wagon to one provider could present serious competitive disadvantage if another cloud vendor introduces something that your rivals can take advantage of, but you can’t. What is that “something”? It could be a specific service or feature, region, instance type, pricing schedule, performance kick. The list goes on. When our survey respondents evaluated the leading inhibitors of cloud usage, security and data privacy came out on top, followed closely by cloud vendor lock-in. We did see more bifurcation in the responses to this specific question: Security was the top inhibitor in medium and large-sized enterprises. Lock-in was the second top inhibitor. Lock-in was the top concern among smaller enterprises. What did company size have to do with the differences in response? Small organizations have increased freedom to innovate quickly and are less likely to be tied to legacy software. Maintaining maximum flexibility as they build their apps means avoiding vendor lock-in that can present restrictions on this ability to evolve. Larger organizations are more likely to have mature contracts with software vendors and are therefore less sensitive to the loss of flexibility caused by long-term vendor agreements. Concerns over data security resonate more for large enterprises as high profile attacks and data breaches are a substantial threat to a large brand. However, lock-in was the second top concern for these larger organizations, ahead of the technical expertise needed to run on the cloud, or concerns about maintaining performance and availability SLAs for workloads running in the cloud. So Where Does the Fear of Lock-in Come From? As discussed in the introduction to this post, many organizations have been burnt by lock-in in the past. The use of open source software and commodity hardware has provided an escape route for many, but they have concerns that by moving to the cloud, they trade one form of lock-in for another. What form does that lock-in take? It’s not about the hardware, operating systems and software of the past, instead it’s about APIs, services, and data. The underlying IaaS components made up by compute, storage and networking are pretty much commodity and can be exchanged between cloud providers. But as we move up the infrastructure stack, so the APIs and data these services exchange become much less portable. Specifically, we need to think about security, management, continuous integration/continuous delivery (CI/CD) pipelines, container orchestration, serverless compute fabrics, content management, search, databases, data warehouses, and analytics, to name just some of the key friction points. And it’s those services that manage our data that cause particular concern. You may have heard of the term “data gravity”. It was (presciently) coined a few years ago, but has a real resonance today. In the same way that as the mass of an object increases so the strength of gravitational pull against it increases as well, in the case of data gravity the more data you have in a specific location, the harder it is to move. An article (3) in the UK’s Computing tech publication illustrates this point. Comparethemarket.com, the largest price comparison site in the UK, made the switch from managing its own on-premises infrastructure to Amazon Web Services (AWS). As a part of that move, the IT team considered the AWS DynamoDB NoSQL database service. However, concerns around exposing itself to excessive AWS control made comparethemarket eliminate DynamoDB as an option. The company has since standardized on MongoDB as the operational database for its microservices-based architecture . There is an important take-away in all of this: It’s fine to date your cloud provider….but don’t marry them. MongoDB and the Cloud We’ve just launched our shiny new MongoDB Atlas database as a service, providing all of the features of MongoDB, without the operational heavy lifting required for any application. So isn’t this new service also presenting the risk of cloud lock-in? The answer is “no”, for two important reasons. The first is that MongoDB Atlas is designed to run on multiple public cloud platforms – so you can spin it up on your vendor of choice. It is available on AWS today, with Azure and Google Cloud Platform coming soon. Eventually we plan to offer MongoDB Atlas across clouds, so you can stretch your MongoDB deployment across providers to take advantage of, for example, specific pricing schemes, regions, or platform features. Secondly, MongoDB Atlas is running the same software you can download yourself from the MongoDB Download Center . This means MongoDB can run on your laptop, on your own local servers, in your chosen co-location facilities, or on your own instances on any public cloud provider. It is quick and easy to migrate existing databases into MongoDB Atlas, and to get it back out again, as we demonstrate in this MongoDB Atlas migration blog . What is also really helpful in mitigating lock-in is that if you decide you want to bring operations out of MongoDB Atlas and back under your control, it is easy to move your databases onto your own infrastructure and manage them using the MongoDB Ops Manager and MongoDB Cloud Manager tools. The user experience across MongoDB Atlas, Cloud Manager, and Ops Manager is consistent, ensuring that disruption is minimal if you decide to switch to your own infrastructure. *Figure 1: Consistent operational interface, where-ever you run MongoDB* The reality is that if you try to achieve this type of flexibility with any of the public cloud vendors database services, you’ll soon hit a wall. Next Steps We all know the cloud is great. But it doesn’t come without risks. If you are looking to move your databases to the cloud – whether for new apps, or migrations of existing on-premises apps, then go ahead and check out MongoDB Atlas. Providing freedom from database cloud lock-in. Try MongoDB Atlas (1) http://www.informationweek.com/cloud/infrastructure-as-a-service/cloud-spending-will-top-$37-billion-in-2016-idc-reports/d/d-id/1326193 (2) http://www.rightscale.com/blog/cloud-industry-insights/cloud-computing-trends-2016-state-cloud-survey (3) http://www.computing.co.uk/ctg/news/2411982/making-movies-how-comparethemarketcom-got-meerkat-movies-up-and-running-in-months

August 15, 2016
Next →

Using MongoDB Skill Scanner to Build Better Training Programs

Technology leaders know that transformation is about more than just adopting modern technologies like MongoDB. The entire organization has to rally behind change — which is no easy task. The skills that modern development teams need are evolving faster than ever, and hiring to fill skills gaps can be too time-consuming and expensive of a process for many organizations. So it’s imperative that we plan for how we want to bring our people with us on our modernization journey, and proactively upskill them on the technologies we’re betting on. Because what happens if you choose MongoDB, but your developers don’t know how to use it? CIOs know that training programs are easier said than done. EY reported that 30% of CIOs acknowledge that their training programs are ineffective, and that they’re struggling to retain talent because of it. These leaders come to us to help them build and execute their MongoDB training programs , and seek advice on two extremely common yet critical challenges: How do we get away from the less effective one-size-fits-all approach? How do we measure the ROI of our training program and connect it to business impact? How we use MongoDB Skill Scanner to overcome training challenges Our Professional Services team uses a tool called MongoDB Skill Scanner to address both of these challenges. This tool helps us provide these three benefits to our customers looking to build a training program: Improve MongoDB proficiency: Teams can use Skill Scanner to quickly and easily assess the MongoDB skill gaps of their team members and gain a comprehensive understanding of their team’s MongoDB skills baseline. Increased productivity and accuracy: When team members have a comprehensive understanding of MongoDB, they are able to work more quickly and accurately on projects, leading to increased productivity and a higher quality of work. Save time and money with targeted Training: Using Skill Scanner, customers can avoid wasting time and money on trial-and-error learning. Instead, they can focus on improving their skills in a more targeted and efficient way with right-sized training plans. By leveraging this data, our customers’ engineers can engage in the right training at the right time, targeted for their job role and specific skill shortages. When a training program is built this way, engineers maximize their knowledge retention and minimize time away from their projects. Skill Scanner includes three role-based assessments, one for developers, database administrators, and DevOps respectively. Through a series of multiple choice questions, Skill Scanner provides customers with a clear understanding of their level of expertise across a set of technical skills that are critical for success in their role. After submitting the assessment, engineers will get results in each skill area outlining if they are beginner, intermediate, or advanced. Why data-driven training programs matter We’ve learned that it’s not enough to just tell teams to go watch training videos or webinars on their own, or to place everyone in the same one-size-fits-all program. Skills gaps vary from team to team, and individual to individual. The one-size-fits-all approach of some programs may not address individual learners' needs, wasting time and making it difficult for them to acquire new skills. By using Skill Scanner, we’re able to interpret this data to help determine which training courses your team should take. But we don’t only capture this data before doing training; we use Skill Scanner again after training programs are completed to see where immediate improvements have been made. This helps technology leaders prove the impact and ROI of their training, and gives them the confidence that their teams are ready to be successful with MongoDB. Developing a Precision Learning Program To go even further, our team can work with you to build a Precision Learning Program, where we use Skill Scanner data to build learning schedules that are unique to each individual. These schedules include a variety of short, blended, learning events such as classes, technical workshops, self-paced exercises, and project coaching. We’ve seen PLP lead to higher knowledge retention and of course, measurable project results. A customer who recently concluded their PLP saw a 43% increase in knowledge retention. Getting started building a personalized training program Skill gaps aren’t a novel problem IT leaders are facing. But with new digital courses, training, and technologies, the resources to close these gaps are at your fingertips. Skill Scanner and Precision Learning Program have been specifically designed to empower teams by offering targeted training that enhances their understanding of MongoDB. These short training events are carefully crafted to close skill gaps without compromising developer productivity. We’ve seen a variety of customers use this tool to help train their team’s individual needs, from needing to upskill new hires on their teams, projects with new MongoDB products, migrating to MongoDB Atlas, and more. It also saves your business the hours developers would've wasted searching for answers (and developers don’t want to spend their time that way, either). “We need help getting from point A to point B and feel MongoDB is uniquely positioned to help” — CTO at large insurance firm If you're interested in trying out MongoDB Skill Scanner or want to explore the MongoDB Precision Learning Program further, you can reach out to your account representative or contact us directly .

June 7, 2023