Event
{Event}  MongoDB is going on a world tour! Gather your team and head to your nearest MongoDB.local. Learn more >

How Verizon Uses Disruptive Developments for Organized Progress Transcript

OK, thank you so much for being here in the business track. For those of you who were here before, we just heard a fantastic presentation about how government is learning how to transform itself. I'm sure you'll be pleased with the next two sessions we have. Right after this, we have a presentation on the Gap and how they're using MongoDB to change their supply chain.

And up now we have a very interesting approach, I'd say-- a title that in and of itself should make people want to be here, which is "How Do We Use Disruptive Technology to Actually Drive Organization?" We have two gentlemen from Verizon to talk to that about with us. One is Matt Bullard, who's one of their senior technical people. He'll stand up and set the stage for us and then will be followed by Shivinder Singh, who's actually quite an interesting character himself, with four patents on what it is to drive database administration and cost savings-- four patents on that. That in itself is innovation, so if nothing else we're glad to have you gentlemen. Thank you for joining us. Please turn off your phones if they're on, and don't forget to use the surveys as we go through this process.

Thank you. My name is Matt Bullard. I'm a senior technical member of Verizon staff. Today's presentation will be covering three areas. First I'll be discussing a little bit about what change looks like in a Fortune 50 company. And then I'll hand the conversation over to Shivinder, and he'll be discussing some of the experiences we've had working with MongoDB as well as what change actually looks like on the ground floor.

A little bit about Verizon Wireless-- Verizon Wireless is the nation's largest and most reliable 4G LTE network, and the nation's first 3G mobile broadband network. Verizon Wireless currently maintains about 103 million retail customers and just about 1700 retail locations. Last year Verizon Wireless generate revenues in excess of $81 billion, and Verizon Wireless, is a 4G network, covers over 97% of all Americans.

In an organization the size of Verizon Wireless, the business needs are constantly growing and changing. These forces drive us to explore new and innovative ways to process manage our data. In today's environment, we're looking to extract greater value from data analytics, shortened time to market, and, overall, bring additional value to our customers. In order to accomplish these goals we must introduce new technologies into highly establish IT ecosystems. Introducing change is always difficult in any organization, therefore the simpler a solution is to implement the greater the value it is to the greater organization.

Whenever you introduce something new into your IT ecosystem you want to make sure it's a positive contributor to that environment. Oftentimes this means blending technologies into existing infrastructure, not necessarily replacing individual elements. Any introduction you make will cause some form of disruption, and the question is-- How will these disruptions affect individual elements in your ecosystem? In a few minutes we'll be discussing how we brought MongoDB on board into the Verizon Wireless ecosystem and some of the ways we address these challenges.

When considering introducing a new technology some fundamental questions need to be addressed. The technologies first need to reduce operating cost or complexity or increase additional value to the customers. For Verizon, this means creating a unified customer view across all lines of businesses, improving our asset utilization, and/or bringing solutions to market quicker. In short, we ask ourselves-- What value will this bring the overall enterprise? And, in terms of new technologies, Verizon's looking to MongoDB as an OLTP engine and Hadoop for managing larger data sets.

Change, however, is rarely a simple endeavour. From an organizational perspective, we need individuals that will champion the change, to take on the status quo. We also need to ensure we have models in place to effectively manage and evaluate ROI. Change tends to always come with a healthy dose of risk. We also need to make sure we effective procedures in place to manage that risk and to manage any type of disruptions to our business.

Whenever we bring in new technologies to our superiors, the first questions here is-- Who's going to be supporting this technology? What happens when there's an outage? As an industry we are all very familiar with the cost of scaling an RDBS solution. We take on or change how they establish procedures. We want to make sure we have solid answers to these questions, and Shivinder will be sharing with you some of our experiences working with Mongo.

Good afternoon, everyone. Thank you for joining us here today. I hope all of you are still awake. Thanks Matt, that was an excellent overview of our organization. So I'm Shivinder Singh. I'm essentially the person who's the go-to person for any of the Verizon Wireless big data initiatives from an infrastructure standpoint. I've had a long working relationship with Verizon, and together me and I team members, we have designed some of the most industry-wide recognized best solutions, and for that, as Becky mentioned, we have received some patents, too. So with that, let's get into the presentation.

What I really want to talk is that you have a change. How does it affect people at the ground level? So, the organization where I come from, it's essentially a infrastructure and operations organization. We are responsible for keeping the lights on 24 by seven. We support all the online assets for Verizon Wireless-- verizonwireless.com, My Verizon, Access Manager, [INAUDIBLE] portal, b2b portal, so on and so forth. Now, as you can imagine, it's a multibillion dollar corridor which we have, so we have very tight SLA demands around this corridor.

I'll give you an example which really shows the criticality of what we do. So there's a unprecedented event in the IT world which takes place every few years. Anyone know what that is? OK. It's basically launch of iPhone. So, when that happens, the carriers who are launching the device, they get hit by a huge tsunami off users in the middle of night at 12, when the doors open. And to have your infrastructures scaled to meet that need, which happens once every few years, is quite challenging. And to be modest enough to say that, to date, no carrier has been able to successfully managed their launch except when iPhone came to Verizon. So, until date, Verizon has only been the carrier which has been able to manage a successful iPhone launch, and we are the team behind it which manages that.

So, since we are able to do all of this in our IT ecosystem so successfully, one of the very few questions whenever we are introducing a new technology into our IT ecosystem is-- Why do we really need this thing? We have an already established system that's running fine. Are we getting a headcount increase? Do we get trained? What happens if there's an outage? We have so many and such tight SLA demands. And when I'm talking to my team members about these disruptive technologies I basically look at the history. So how has history shaped our answers to these tough questions? And the way you need to really look at it is you have to follow the trend of IT. I distinctly divide the IT revolution into two phases. The first phase was when the IT boom had just started and the business had started looking into opportunities where they could reach out to their customers in a much more effective way. And the way they did that? They built a relational model around a unique idea for customer which captured all their touch points. So whether you're in a store, online, or IVR, your ID could be related to a unique ID, and that's how you were to identify.

The second phase of IT revolution-- that's all data bank. There are social feeds, people posting on Twitter, reviewing iPhone from Verizon versus what-- [INAUDIBLE] the [? buying ?] experience [? going on? ?] So the touch points which an organization has with the customers have increased manifold, and at times it becomes really hard to capture all of those touch points in a single traditional technology. So that is why, with the evolution of data, the technologies need to evolve. And that is where the players like Mongo fit in. So that is how we're trying to answer this question.

Now let's take a look of how we have evolved with this evolution. So, back in 2004 and 2005, we were essentially an Oracle RDBMS show. And as you see the timeline has progressed. We have embraced all these new technologies. Now this is not something new. Many of you and your organization would be having a similar timeline, but the reason I have this slide up over here is for a slightly different cause.

One of the biggest dilemmas which most organizations face, including us-- we have faced also this-- that where does the new technology stock live? And why that is happening is because the lines between the technology arena have become very faded, and they are fading with time. Back in the past few years you have a distinct team who would do system administration, but also someone who would do a DB-related work. Also someone who would do middleware administration. But then with time those lines seem to be blurring. And it becomes a very ambiguous question. Where do I put my technology stock? If I introduce this new animal into my IT ecosystem, how do I make sure that it's a constructive member and not causing a disruption?

So what we did was that we looked at the history, and that's how we decided-- we had a brainstorming session-- who were the major players who had been successful in implementing change. And we came up with a certain set of organizational prerequisites. So if you are implementing this change in your organization you need to keep a track, or need to look at the history, of which organization within your company, and which people, have a track record in delivering change successfully. And basically at time it comes down to people. You need to have passion in the people who go out of the way, and maybe even on their personal time, to embrace this new technology. And obviously, if you are bringing something new, you are bound to have risks. So you need to take risks while you're evaluating this.

Now if we at the scale of Verizon can implement this change successfully, I'm pretty sure that this model can be replicated no matter what the size of your organization is. So what was our experience, or how did MongoDB come to Verizon? And that's an interesting story. You know, what really had happened was that we had this application, which is essentially an employee portal-- it is a business critical application. And it is basically the homepage of anyone who works for Verizon. And the development team which designs this, they were looking into a new functionality which could be put on the website where you would capture social feeds from [INAUDIBLE] Twitter, and Facebook and display it specific to the user.

So somebody in the development team did some looking around and came across Mongo, and it started as a POC. And one fine day that POC became production, and on another fine day the production server crashed. They had issues recovering that server, bringing it back on, so they came to my boss and they said, you know that we have this new environment which is in place, this new technology. Would you be willing to support us? And my boss in turn talked to me. And I said, well, if we are bringing something new we had better make sure that it meets our standards. We have some high standards set for any technology which comes to us, so let me look into it.

And I started learning Mongo, and the way I started doing it was I took some online trainings. And I think within a period of two days I was at a level where I could comfortably manage Mongo. And in a period of two weeks I had re-architectured our entire development set-up to be in a replicated cluster versus a stand alone cluster and all and tested, broke the cluster, brought it back up, tested recovery, failovers, all the good features. So at times the point really is that not having a choice is the best catalyst to move forward, at least that was our experience in evaluating this technology.

So what was our experience while evaluating it from a technology perspective? And you know, that's another interesting story which I'm sure not many people within my organization also know, and not the people at MongoDB also know. So the way I look at it is that, whenever you are trying to buy a new car, you obviously want to take the car for a ride. So we have built this environment in production, and now we were at a level where we thought we should get some kind of a professional agreement with Mongo going on.

And I'm not sure if in the audience today have Brendon Noon. Right there. And Dan Whitmeyer. So these were the two gentlemen I started working with. And we were ready to sign a support agreement with them. But, before doing that, I wanted to take you for a ride. And the way I did that was I created an [? outage ?] and a secondary data center. And I called them. I said, guys we are in a situation where our production environment is down. Can you help us out? So they gathered their engineers and got them on a call with us, and they were able to get past us from the [? outage ?] in a short period of time. So you might think, you know, it's Verizon. It's a big company. If they call someone, they are bound to listen. But let tell you, during my initial interactions with them I was not using my official e-mail ID. I was going with my Yahoo e-mail ID. So that was our experience with them.

And, from a technology perspective, when I started evaluating Mongo I could pretty much relate it to the food industry. Many of us might have driven past McDonald's and seen a billboard over there-- over 99 billion served, right? And I'm no expert at food industry, but taking a conservative estimate let's say that maybe they have 100,000 specialized cooks who actually make the recipes. Still, to serve 99 billion meals with that amount of a work force seems quite impossible to me. So what the food industry did back in the day was that they came up with a concept of a assembly line, where you remove the specialized cook and run the operations in a assembly line. Essentially that is what I am seeing is happening in the IT industry today.

Earlier, back in the day, with the traditional RDBMS technologies which we had, if we had to design a system with BR enabled, over the WAN replication, ASN, multi-master, or master-master, so on, and all the good features, with a specialized cook, or a DBA, it would easily take about two weeks. And with Mongo we were able to do that in two hours. And the good part on top of it-- it's all open source. So you can download it. You can get it up and running, and once you have a comfort level you can think of getting professional support. So that was our experience while we were evaluating Mongo.

So what are some of the technology challenges which we saw with a technology like Mongo? Those I'll divide I into two category-- the hardware and the software technology challenges, and then the people related challenges. Now these technologies are no doubt very innovative, but how you can turn these innovations into products which are monetized is totally up to you. You can get as creative with your use cases and have them supported in a database flavor like Mongo. But at the same time, while you're doing that, from operations perspective, you need to have a focus on your system performance.

So if you have a system which is running fine on a 10,000 user base, you really need to make sure that when that user base scales up to 100,000 or 10 million your system performance does not degrade, it actually scales with performance. So that is something which we look into. And the other challenges which we saw was that, if you are planning to put your critical data into any of these new technologies, you need to make sure that you have a very robust backup technology. And then there are some challenges around securing your cluster if you're putting your critical data.

From a people perspective, I think the main issue which we have seen is unlearning the old scales assimilating the new ones. The traditional RDBMS technologies, the SQL technologies, they have been the flag bearers of the IT revolution. They have done a great job doing that. They have shaped the mindset of an entire generation. When you talk to people even today of databases they think in terms of drawings, stored procedures, packages, so on and so forth. So at times it becomes hard to unlearn those skills and assimilate the new ones. That is one of the biggest challenges which we saw. And while you're trying to do that, you also need to keep a focus on process engineering. So if you have designed a certain set of processes which are there in your IT ecosystem to support your technology stack, you need to make sure that, when you are embracing these new technologies into your fold, your process match with these technologies, or these technologies fit in and blend into the fabric which you have built around.

So what are the future use cases, and where do we see Mongo headed at Verizon? Well, as you would see in the conference today, you might have read big data slash Hadoop is the buzzword these days. So Verizon has been a very early adopter of Hadoop. We have some of the biggest clusters, which are running which are growing at phenomenal volumes. And it will be very interesting to see that how we can couple Mongo and blend it into our Hadoop architecture.

The second POC which we are currently working on is for our online log management. So as you can think of it, Verizon has some huge servers, some huge clusters, and all of them generate a huge amount of log. And at times it becomes very costly and very expensive to store all of that information in a traditional RDBMS system. So we are looking at Mongo to be a potential replacement for that.

So with that I would like to conclude my presentation by saying that if this change is happening-- so if we at Verizon, given our scale, can successfully assimilate this change-- I'm sure that this model can be replicated at any level of the industry. The old database technologies, they are not going away. They are here to stay, but what is happening is that the database arena is becoming more leveled out. There are more players in the market who are coming with more innovative products. And the way the data is changing, in terms of user generated data versus machine generated data, these technologies will be here to stay. So, the sooner you adapt to this change, the better your organization would be. At least that's what we figured out in our case, and I'm sure, if you are looking into these technologies, just start with them. So with that, thank you MongoDB for giving us a chance to speak. Thanks Matt for joining. Thanks all of you for staying here. [APPLAUSE]

Thank you very much. Thank you, gentlemen, for those wise words of what it is to start adapting this into your organization. Not having a choice is the way to go. Does anyone have any questions that we can--

What's the border between the Exadata and Mongo? Is Exadata more for speed, and Mongo's more for complexity?

The way I look at it is Exadata serves the purpose of asset transactions. Mongo in terms, you could have similar kind of data over here also, but, again, the two key features which I highlighted from the challenging perspective-- having robust backup technologies and securing the system-- that is, I think, something which needs to evolve in time with these technologies. That's what we saw.

We have one here.

What percentage of your new use cases is Mongo appropriate for?

Quite a lot, I would say. Wherever we have a chance to use NoSQL, given that it runs on commodity hardware, nothing specialized, we are trying to fit and see the comparison of what works better, like a traditional RDBMS versus a NoSQL technology. So almost all cases we are looking at with both the technologies in mind.

[INAUDIBLE] there to develop the applications on MongoDB?

Again, some of the challenges might be that your people might not be familiar, so there's a learning curve which takes over there, especially with the development organizations. So that is one of the challenges which we have seen in developing. The other one, from a technical perspective, which I see as what MongoDB does not support a multi-master write, so your writes are essentially directed to one server. In our case, we have systems running in a multi-site [? mode way, ?] read or write simultaneously happen to multiple servers. So how you develop your application to fit that model is one of the challenges which I see.

Thank you.

Can you elaborate on the point you said where you wanted to get MongoDB more integrated so you could use Hadoop with that?

Yes, so MongoDB has a connector for Hadoop where you can run your discoveries and your analytics on the traditional Hadoop platform-- Hive or [? HBS. ?] And then, for your OLTP needs, it would be interesting to see how you can couple your Mongo with Hadoop and get value out of it. That is something which we have started looking into. Any more questions?

Are you thinking about using your log data with Hadoop as well? [INAUDIBLE]

Yes, it's again case to case. If the data is something which your system generated, and which in turn is huge, Hadoop might be a better candidate for that. But if it's a data where you have use cases of capturing customer clicks or social feeds, then it would make more better sense to have a framework like Mongo over there.

Any last questions? OK, thank you again gentleman very much for your time today and sharing your story. We didn't get the answer to the four patents, but OK. We'll ask you in person maybe later. So I want to remind you please to fill in the surveys. If you don't know how to find the survey please come ask me. And top of the hour at 4 o'clock we have our last presentation in the business track today. It's going to be a great presentation about the Gap and how they're reinventing their supply chain. Please join us for that. And then at 5 o'clock we'll have a few more keynote sessions before this evening's dinner. Thank you.