Thank you so much for staying with us. I promise this will not be the least presentation. This is going to be the best, because here we are at the end of the day. We've heard about transformations across the board. Thank you, Pete, for joining us. And we've heard about transformations across the board. We've heard that it takes a lot of smart people and people open to innovation to continue to drive change in organizations. And that brings us to a good point to introduce Ryan. He's here to speak about how MongoDB is working at the Gap. And he's going to talk about how that works in this process.
And Ryan comes as one of the senior architect leaders within ThoughtWorks, a consulting agency that he's going to talk all about. He brings over 15 years of experience in software development, architecture, and systems. He's done that not only in the US, but also throughout Eastern Europe and India. So he has many different approaches and thoughts about what this looks like. He has a lot to share.
Please make sure you turn off your telephones for this last session. Don't forget, you have to fill in the survey at the bottom of each one of the sessions within the MongDB world application. At the end, he's going to take questions. Ryan, thank you so much for joining us and taking this graveyard shift today.
Thank you, Rebecca, for the introduction. And thanks very much to Mongo for having me here today. I am here to talk a little bit about how we're using MongoDB and Gap. As Rebecca noted, I'm not from Gap. I'm actually a principal consultant at ThoughtWorks. And so first, I just want to say a little bit about who ThoughtWorks is.
We are a global software delivery consultancy. We're about 2,500 people in 12 countries. And what we're really focused on is expertise in the industry and how to drive our customers to innovate and be better at delivering software. So if you go and check us out on the website, you'll see we're passionate about innovation. We do everything from large scale architectural transformations for companies in all industries, travel, fashion, retail, all the way down to doing pop up innovation labs in stores with people. We've had developers on the ground inside retail operations building software with our customers' customers.
So I'm really excited to share this side of the work we do, which is large scale architecture work and transformation and how we're working at Gap. Now, I don't think I probably need to say much about Gap, because I'm sure everybody is here because of Gap. I don't think they came to see just me. And Gap is the face of American fashion in the world.
So they have about 3,000 stores that they operate. They've got a number of franchisees as well. They're operating in 19 countries. And again, they design their own clothes. And they really bring the American style up to the world. So what I want to cover today, I want to cover, basically, the context of the talk is about a purchase order system and how we got started working with Gap. Mongo's found its way now into many other contexts. And I'm really happy to talk more about whatever I don't get to, if anybody wants to talk offline. I want to talk about why MongoDB, of course, and why we thought it was such a good choice for the type of work we were doing.
And I want to highlight that this talk isn't about big data and MongoDB. Quite the opposite. It's about rich domains and making teams more effective with the technology like MongoDB. I don't think we hear people talk enough about that. So we come to some organizational challenges, talk about some of the challenges we've had in adoption and how we surmounted those. A little bit more tales from the field, some of the good the bad, and especially, lessons learned. Like many people have talked about today, everything, isn't always smooth. And I think there's a lot to learn from the mistakes that people make.
So I want to start off just by grounding us a little bit in the what is a purchase order. I'm sure everybody is familiar with the structure of an order. What's interesting about an order is, again, this is a rich business object. An order contains various pieces of information. In this case, it's an order out to a vendor to produce clothing. So we have the vendor information. We have shipping information, reshipping by air or by sea. We have other information, of course, the line items, which are the individual SKUs that are going to be produced. So this shirt in small, whatever it happens to be, and the quantity that needs to be produced. The business thinks about that as a single entity. If you talk to anybody in the business, they don't think about line items or vendor lines. But if you look at the diagram-- and I want to give credit to my colleague Martin Fowler. Some of you might know, he's a very prolific writer and blogger in the industry. We tend to take these cohesive business concepts, like a purchase order, and we disaggregate them. We split them out into all of these little pieces when we develop a traditional relational technology. And why do we do that?
Well, we actually do that because relational technology has baggage that led us to need to be efficient and normalize. But again, the business doesn't think that way. They think about the object, the purchase order, as a whole. So one of things that I want to point out is for a developer who's working in this environment, the developer is forced now to shift away from the business concept-- what is a purchase order and what are the business rules and capabilities around it-- and shift gears into a, how do I make this technology work for me and help me solve a business problem? And that destroys flow.
Because the business is the thing we want our developer solving, the business problems. We don't want the developer having to work with the impedance mismatch between the business concept that they're trying to solve for and the technology they're using to solve it. Yet, our developers probably spend way more time solving technical problems than they do business problems. And so when we embarked on this custom PO system rewrite at Gap, one of the reasons that I was really excited to use MongoDB was I felt that we could reduce this impedance mismatch of being able to take business concepts and think through the business solution, without having to spend so much time thinking about the technology. And that's turned out to very much play out for us.
Another aspect of the solution that we created was we use what we call a microservices architecture. Now, I know this is a business track. And so I'm probably getting a little technical. And I apologize for that. Try to stay with me. If we look at this diagram, again borrowed from my colleague Martin, the thing I want to point out where the red circles are. Our traditional model for building applications has typically been as large monoliths. And what we mean by that is the PO system was one, big, large, code base that did everything about a PO, whether that was handling demand that was coming from the planning systems and creating those purchase orders, whether it was managing the rich life cycle once a purchase order came into existence, updates, and other business rules, or handling how the purchase orders actually integrate to other systems and get down to the vendors. All of those things are actually fairly independent of each other, but the code basis would be big and monolithic.
So what we wanted to do is bring this concept of the microservice, a service dedicated to one business capability. And what happens is each service-- and in the examples I just gave, let's imagine that handling the planning system, telling the PO system, it's time to go. I got to buy these products. Or handling communicating out to the vendors by EDI or whatever technology. Each one of those things is a separate service.
And what you'll see if you look on to the left is MongoDB is effectively bound to that single service. That means at least some collections, if not an entirely dedicated Mongo server directly lines up to some business function. And what this means is developers who are working on this part of the application don't live in a code base that's tangled with all other aspects of the purchase order system. What they focus on, all the way from the business capability down to the database system and back up again, is one smaller set of concepts.
And so this type of model is extremely valuable, again, for keeping developers focused and also reducing the kind of problems that happen when we have monolithic code bases. Developers bang into each other. There are either synchronization points, where people can't work because they're afraid to touch something that might impact their colleagues. Or worse, they spend time trying to undo the tangle that gets created and discovered later. So this is a architectural style I'm very passionate about.
And what I want to point out is that Mongo turns out to be a very interesting technology to support this. Because Mongo is simple and it's lightweight. It's very easy for me to spin up MongoDB on my laptop. I can have a fully, high available cluster five nodes if I want. And it spins up in a second.
But more importantly, it's something that developers can drop quickly and it gets out of the way. And so now, my developer's focusing again on just that part of the application stack. And this leads, again, better flow, better communication around the business capability they're trying to build.
So I've been talking already a bit about what we liked about MongoDB and why I was so passionate about trying to bring it into this context of a purchase order. I want to talk about a couple of other examples. Now, flexible schemas, everybody understands that flexible schemas are a major component of what MongoDB does.
And I think that the interesting thing for us about flexible schema, there are two real reasons. And one of them is a bit classical. It's, I have data that comes in different shapes. And I want to be able to store it quickly and transparently in my database, of course.
But what's interesting to me is that this was an extreme enabler for Agile development, an iterative development. Because what I'm able to do is I'm able to really forget about the storage layer that's underneath me and just get work done. As I evolve, the concept of a purchase order as an aggregate concept, I'm adding fields to it. I'm changing behavior as I learn and as I develop and as I release and I talk to my business.
MongoDB gets out of the way. I drop a collection, start up new code over that database, and MongoDB accepts what I throw at it. Again, developers don't have to stop, break the context of solving the business problem, and get back to what they're doing. They simply get to focus on the business problem. And so as an Agile enabler, as an enabler of developers to work fast and smart, MongoDB do is extremely useful. Again, I'm also really interested in how the flexible schemas can solve business problems. An example for us was we were in production with our PO system in about 2 and 1/2 months. That was an MVP. That's still an amazing turnaround time for an entire purchase order system. So we were extremely excited. And that was a functioning purchase order system. Granted it wasn't the biggest system in the world. It was a working system that supported an international business for about 2 and 1/2 months. And of course, we're Agile. So we continued to evolve that system. So about five months in, we were presented with a business requirement as a team. The business wanted to track in a dashboard style the life of a purchase order. What were the changes, the versions that it had gone through? Typical business requirement. With MongoDB, that business requirement turned out to almost require no development effort. We were able to drop copies of purchase orders and copy them and say, this was the old copy. Let me make another one. Throw them all in the same collection and keep moving. We were able to show the business what that feature might look like intraday, because of MongoDB's flexibility.
Now, that doesn't mean that that was the end of the development. There was more complexity that we had to deal with. But the interesting thing, again, about enabling teams to be productive is we were able to do that with almost no friction, initially, and give that to the business and start getting feedback. And this is where Agile development is really critical.
As we got feedback, we actually descoped a lot of things the business thought they needed and were able to get them the right version of that future in weeks, instead of potentially months, because we were able to show them something and interact with them very quickly. So while we have to solve some of the challenges of dealing with changing schema over time in production with that same feature, MongoDB gave us options. And it gave us speed to market for the first version, which meant we learned. And it meant we actually could cut scope.
And so these things are very, very hard to estimate. I'd love to sit here and tell you we saved 27 months of development time. I can't. Hopefully though, what I can give you is this passionate plea for why this is so important to your developers on the ground. The thing that I want, I hope you'll take away from this is in today's changing business climate, software and technology are increasingly at the heart of being successful.
And great software is about people. We need to enable our people to be productive and to be effective, where effective means building the right software. And I don't think we talk enough about how MongoDB enables that. So to carry on with that, another really interesting idea that I had about MongoDB was very soft skills related. Developers really are interested in MongoDB. And we started this process a couple of years ago. MongoDB has come a long way in that time. It wasn't quite as well known then, but many developers were excited about this new technology.
It seemed to have this great blend of things between relational with the indexing and being able to manage complex queries, but at the same time all the lightness of other NoSQLs. In fact, lighter than most other NoSQLs. If anybody has worked with Cassandra or [? Redis ?], some of the other tools, actually, Mongo's amazing. And so the development team was excited to use this technology.
And that can't be underestimated, because an excited team, a motivated team is a productive team. People who want to come to work and love the technology they work on are much more productive than people who are constantly fighting with the technology they work with. And we fight with enough technology in a complex enterprise architecture that I can show you, when people are working on work that directly involves Mongo in the core stack, very productive, very excited. When they have to go to integrate with awful legacy things, the pace of thought slows down.
And so I wouldn't underestimate the power of MongoDB to excite people, to bring the developers. There's the old adage we all know. Nobody's ever been fired because they bought IBM. Well, no developer has ever wanted to come to work because you bought them IBM. And I wouldn't underestimate this, because I think it makes a big difference.
So the last thing that I do want to say about why is the business-- this is the one thing I didn't predict when I was thinking about technology selection for this project. I didn't think about how cool the flap structure of BSON or JSON really is. Now again, I know this is business track. But hopefully, a little BSON doesn't scare anybody.
MongoDB stores data in this really cool, simple format. It's compact and it's human readable. And this gave us a surprise. Because we discovered that business users actually understand their data very well. We as technologists like to laugh and think we have to save the business users from the technology. But the truth is we don't have to save them from their data. They know it really, really well.
They use Excel. They know how to manipulate it. Excel's its own little programming environment. Business users are very savvy. And what happened was we gave them the BSON because Mongo made it easy. They can sit down with the Mongo console-- see at the bottom, there's a Mongo query. Very simple to substitute in a few attributes. They didn't need to be Mongo experts. They could learn that. Some of them used the GUI sometimes.
But what happened was they would start to fish data out of the system, side by side with the development team. All of a sudden now, the business was coming and sitting with the team and experimenting and thinking together. The fidelity of that conversation is incredible compared to passing things over documents. The fidelity, the real-time nature of that conversation is invaluable.
We have production support issues, where the business was sitting next to a production support engineer and saying, let me see if I can see what's wrong with that purchase order. Let me see why that's not interacting with the rest of our infrastructure directly. And they sometimes would solve the problems before the engineers, because they understood the implications of the data that they could read.
So again, I can't point to this and say, oh 10 weeks, $10,000 saved. But I can say passionately that this is an incredible experience, to sit with the business, to have them work with us and get all these results. And the way that sped up our development cycle was not to be underestimated.
However, everything wasn't simple to do this. I've talked a lot about what I chose to do. But what I haven't talked about is how we actually kicked the project off. Now, just have a question. I know we're here at the Mongo conference. But a show of hands, anybody out here think they would have trouble selling MongoDB in their organization today? People have doubts? OK, not too many hands, but I'm glad to see some.
Because I had a big road when I came to Gap and looked at this problem space and said, I think this is a perfect fit. And they said, whoa. We're in the middle of consolidating our technology. We've got every database under the sun. We're trying to get rid of databases. Are you nuts? You want us to have a new database, and one that's really just on the breaking horizon?
And so we spent a lot of time evangelizing. And again, it comes back to people. We spent a lot of time talking to architects. And I personally ran a bit of a grassroots organizing campaign so that we could eventually roll up to the CTO, this cohesive voice that said, we actually think as a group, as a bunch of Gap employees, not just me, that this is worth taking a chance on because it looks like it solves our problems in an important way.
So I would urge you all to not underestimate. Technologists are not always the best at selling why things are valuable. Help them. Make sure we understand how to speak the language of the business. Make sure we speak the language of, what is the value proposition that this technology has to getting software done faster?
And as we heard a little bit earlier, proof of concepts help. So we took a PoC approach. We said, look, this actually is not a big, hairy project. It's actually a relatively contained project, especially, that we're delivering in a very Agile manner.
We're going to try to get something out in a few months. We can easily build into scope what it would be like to bring in RDBMS into this and make sure that you can see the impact of that is quite small. We think we're going to go much faster with this technology. But if anything goes wrong, we have a plan. And that was a great calming factor. In fact, I actually think at the time, some senior leaders were hoping to kill this thing, but they really didn't want to stop the groundswell of support that they had seen. So they said, fine, do your PoC and we'll talk.
I was smart enough to know that nobody remembers those conversations. And once you're in production, that's it. Unless you actually have a serious problem, you've got it. And we succeeded in just that very way. How about transactions, show of hands? Anybody worried about Mongo not having transactions? Does it scare anybody in the business anymore? OK, I didn't think-- again, most of you.
I think we've got 2 and 1/2 years ago again, this was a big problem. Mongo doesn't have transactions. It's not asset. It's not normal. And it was, but around that single document concept. And so another thinking model that was really valuable for us was we don't have the traditional transaction. If we go back to that slide on the PO, so much of what transactions are used for is keeping the state of a single business concept consistent, because the database made us tear it apart. It's like it solved it's own problem. But it's a problem I don't want to have.
So with Mongo, what we do get to think about though is we're solving this database aggregate problem now. I don't have to think about transactions. But I do have to think about cross document or cross entity or cross business concept transactions. And what Mongo helps us do is realize eventual consistency is how the real world works. The real world isn't very transactional.
How often do you exchange stuff with people really in a split second? There's always something that's going on. Your bank, the classic example of transactions is a bank. They're not transactional. Most operations at a bank, you get debited first because the bank, if they lose it, they want you to lose the money, not them. And then they credit it, especially if that's another bank that they're transferring money to. That's not transactional, really.
But they have lots of really reliable processes to make sure that that process rolls forward quickly, as well-- it's a bank. It may not be that quick. But they definitely have processes to make sure your money gets there eventually. And that's the way the real world works.
So I really enjoy pushing the developers to think about, what is the right order of business operations? What is the safe order of business operations that we can carry forward in an eventually consistent way? Because eventual consistency can reduce complexity from a cognitive point of view. But it can certainly reduce it from a technical one.
We had a real example in our system that was when demand comes in from planning systems and becomes a purchase order, there are lots of cases where that demand might be an update to an existing purchase order. But in many cases, if its important characteristics have changed, maybe the vendor has changed, or more likely, other shipping or other aspects change, it causes the purchase order to often have to either shift part of the demand to another order or possibly remove a purchase order and add another one. These are not simple operations.
And what's important for developers is if you use transactions arbitrarily on the database, you may have one of two problems. You either span too wide and your system doesn't scale and falls down when you're no longer processing 20 transactions a day and it becomes 200,000. Or worse, you get into what we call the lost update problem, where you actually read the database. You forget to lock it properly, because you've got a transaction when you save.
The problem is what you save may have changed from underneath your feet. And when you don't have the crutch of transactions, developers actually have to think about how to do this step wise and in a safe manner. Now again, this can come with perils, because developers have to think about that. They can get it wrong. But I think it's really valuable, again, to push our teams, to build better software that more closely mimics the real world. And so Mongo can help us to do that.
A couple more stories from the field that I want to tell quickly about why Mongo is so lightweight and interesting. You've heard this story before right? High availability is built into Mongo. And it truly is. It was actually an amazing experience.
We have at Gap an operational readiness audit that we do before a new system goes into production. And it's very important they check that your disaster recovery plan is in place, that you have failover, backups, all these sorts of things. And because we are this new Agile-- we're in the middle a bit of an Agile development transformation in the part of Gap in the supply chain where I was working at. So some of the old bets were off. We weren't doing as much upfront planning, but we still had this review process.
And thanks to a calendaring error, we thought we had several days before we had to have our production site ready to be audited. And we came in on a Monday morning and discovered we had the audit in one hour. And so for a lot of reasons, political and otherwise, we didn't want to reschedule.
So a colleague of mine and I, we sat down. And of course, I had some fair amount of Mongo experience. I wasn't a newbie. But we sat down. And in 37 minutes, we had a five node cluster set up, three nodes for high availability, two disaster recovery nodes, and a separate data center. Ended up running in 37 minutes.
The auditor came. And we were sure that we had screwed something up, started knocking down our servers. We saw failover happened. We had a delayed replica set node that we had set up. I don't know if you all are aware, but Mongo has us subtly interesting feature where you can make a replica set node be delayed for some period of time compared to the primary. We had all these things set up.
They were shocked, not only that everything worked, but that we had features that they didn't know how to use in Oracle. They didn't know how they could do that. How could you have a delayed replica set node in two seconds? There's one property in a config file in Mongo.
And what do you do with a delayed node? Let's say somebody fat fingered some data. You have a literally real-time backup. You don't need to go to tape from the night before. If I went to the DBA organization in Gap and asked them to do that on Oracle, it would have taken months for them to figure out how they could offer a feature like that. And as it turned out, that feature saved the day.
Again, team was new to Mongo. I knew that we might make some mistakes. And one of the reasons I set that feature up was I was really worried that at some point, somebody would fat finger something. Truth is when you're adopting new technology, and we'll talk a bit more, there are practices that sometimes get lost. That DVA group is good for making sure that they've locked everything down the right way and that everything is properly supportable. So in our particular case, I got a phone call. I was off site. I was at Google talking about some other technology with somebody. And I get this call from the chief architect of the client who was my peer.
He said, I can't believe it. I don't know how somebody got the credentials, but somebody managed to drop the whole production dataset. This was about five, six months into the project. A moment of gasp and panic. But wait a second, I thought about this happening. So go look at the run mode on the wiki. And he, of course, had been coming up to speed, but had forgotten we had this feature.
I said, go look at the run book. And there you should see how you can check the outlog, find out if there had been any write activity. Because a purchase order system, especially in its infancy, doesn't have constant write activity. Certain moments in the week are more active as people are editing POs and shipping them out. And we were lucky enough that in the 15 minutes since the event had happened, there had been no write activity whatsoever. And he was able to trivially take that disaster recovery node that was 30 minutes delayed and turn it into a primary and rebuild a replica set around it, just following simple commands that were in the outlog.
And so in far less than an hour, probably in 30, 40 minutes, he had recovered from this production disaster, lost no data. And no business users were aware that it had ever happened. And again, it's a terrible story in one sense, because we were in a situation where someone could fat finger the production data. And I'm embarrassed by that. But it's an incredible testament to the technology, that you can set up something with the one parameter that can take five minutes and can solve a problem that awful.
And there are many other use cases that have come up since then. In the supply chain today, we use Mongo for some logistics data. We use Mongo go for just tracking shipping, tracking customs documentation. We use Mongo underneath these microservices. Again, there's a broader transformation agenda at foot there that have been driving around microservices and having lots of small systems that are focused on one business problem. And so Mongo underlies a lot of those.
We're using it also for some infrastructural patterns. We do a lot of asynchronous architecture, where we publish feeds of events from one aspect of the architecture and let other parts of the architecture react to that, rather than having lots of tightly orchestrated, tightly coupled processes. If anybody's interested in microservices and whatever, we can talk later. But that's opened up new opportunities for Mongo.
High volume event feeds, which we create as a component that different teams can consume, we use Mongo underneath that. So we're using Mongo in lots of different, both rich business domains and some of its more traditional, larger scale, high volume writes. And again, so far, it's been extremely interesting for us as a development tool, because it just stays out of the way and it lets teams get things done.
But it wasn't all roses, as inevitably is the case. And we've heard-- yeah, sorry. Sorry to break you the news, but we make mistakes. And I think the interesting thing about Mongo is it's a very sharp knife. And that's why it's awesome, because it's simple and it hits very important use cases that developers need solved. And it solved them well.
The downside of that, though, is you really have to focus on education. We've seen several presentations today in the track that talk about that. James at GDS, where-- ThoughtWorks is actually working with them as well at GDS. We've noticed the same patterns happening in many places where if you take Mongo and you bring it into a team that's reasonably well versed in the technology, then they can do great things. But some other team will see that success and they'll think they can replicate it without understanding it. And Mongo, as simple as it is, of course, it can't read your mind.
So one of the things that we've noticed that falls by the wayside quite commonly are the things that that traditional DBA group is quite good at, which is the operational monitoring backups, those parts of the process. Developers get so focused on getting the business work done, on delivering the right business solutions, that often they forgot. But Mongo is a different model. The development team really has to own Mongo top to bottom.
You could probably have a Mongo DB, DBA group at some point. But a lot of us started these by introducing Mongo go into a development team and letting it run really fast. And so typically, we got into some problems where operations planning or monitoring wasn't being done adequately enough. And so those cases, of course, reared their ugly heads at the worst possible time. And they really can sully the reputation of the technology.
If somebody makes a mistake working with Oracle, which happens all the time, no one blames Oracle, because it's Oracle. It's obviously the team. But what I've noticed is the exact opposite happens with Mongo. Whenever something goes wrong in the vicinity of Mongo, it's Mongo's fault. It's that Mongo thing. And so education is a critical component to making things successful. And we have been really pushing hard and not always succeeding at getting Gap themselves to recognize the importance of a community of practice, developer advocates.
There are many mechanisms. Training is not the only thing. Trainings are moments in time. And you should use training. Mongo has great support. They've been very supportive of Gap when we've finally engaged them properly. But I think you also need an ongoing relationship with the new technology and your development communities. We're trying to set up what we will probably call a community of practice.
And some organization that is not an organizational silo, which is a kind of anti-pattern that we wanted to use Mongo to break, we wanted to get out of that someone else owns my database. I want the database to be part of the application stack. I want it to be fully understood by my developers, but I still need some kind of matrix ability to make sure the right things are happening and that people are learning. So be very careful and very intentional. Talk to Mongo. Reach out to me if you're interested about how you can socialize these things.
The other thing is data modeling. Again, Mongo super, super easy. Throw anything at it, serves it right back up. Very simple, right?
But the problem is in an Agile development life cycle, you take heavy advantage of that at the beginning, because you're like, we're starting with a small part of our feature and we're going to build it as we go. The problem is, where you go. And we've seen a couple of cases where, for example, in the logistics domain, you have a shipment, which is lovely to treat as an aggregate until you start thinking about shipment events, which are the individual every time a box moves somewhere in the world. Your third party logistics providers are sending you information about that.
Sometimes that's like 80 fields of information about the temperature and the age of the son of the person who picked up the box. It's amazing, the richness of these events. But they're high volume. And so you don't want to have an aggregate a shipment, which should reasonably describe the shipment as a thing and also have an array inside that's 500,000 events long. But we've had that happen.
And again, I think there's an extra high risk of that with Mongo, because it's so easy to get started. And we've heard several folks talk about that's a common anti-pattern that people fall into. It's now I've got an aggregate. Let's aggregate everything. We just wrap our arms around the whole enterprise. And that can really hurt you. So you have to keep up with the evolving needs of your application, whether that be purely volume or, in our case with these rich Agile development cycles.
We're doing more. We're bringing more capability in. And you don't want your documents to just become dust bins where everything gets thrown, because that will hurt you. Similarly, you need to really make sure that people who, I think from the Verizon talk we heard, make sure that the old guard, the folks that have been doing RDBMS for a long, long time, make sure that if they're going to be involved in these projects-- or people move around in projects. And we've seen cases where a project started with a team that really understood Mongo and they understood the way and the patterns that should be used. But over time, as people turnover and they move, sometimes people show up who are more from the older school. And they don't realize how sharp the knife can be until it cuts them. And then again, there's the, well, if I had Oracle, I would know what to do right now.
And that's not a failure of the technology. It's a failure of the organization, obviously, to not see that coming and support them. So again, I would stress, very important to make sure that you really look at the organizational support that you bring to the technology, to make sure that it's being used correctly and to make sure that you're actually supporting the people who need to come along.
So in summary, I just wanted to say again, the key benefits that we saw, a PO system delivered into production in 2 and 1/2 months. The team was convinced that Mongo was a big part of our early ability to focus on the business, get the work done. Weeks of work certainly were eliminated by not having to worry about constant schema migrations as we're evolving the business concept. Surely, setting up development environments on developer laptops that exactly mirrored production was incredibly valuable to us, catching things earlier in our development life cycle.
We practice continuous delivery practices. Everything is automated. And that automation with Mongo is super easy. We spent no time doing it. And it paid off many, many times when we found problems way before we got to production. The HA cluster set up in 30 minutes, 40 minutes, just compared to days and days of work with more complicated technology is very, very important to helping us also cement Mongo as a reliable, useful tool. Clearly, it's an open source product. People always talk about that. And that's always a benefit. You've got a whole ecosystem.
But what I really think-- I want to leave you with this-- that it's priceless, the energy and flexibility-- that the flexibility brings to the team and that the developers really work in a completely different pace when they can stay focused on the business and don't have to worry about fighting the impedance of technology. So I would say if you have data in your enterprise that's natural as an aggregate, not necessarily big, but complex, rich things, consider MongoDB as a very serious tool for enabling your teams to be more productive.
If you fundamentally have scale, then obviously you can consider Mongo as well. They're not exclusive. But if you would benefit from your developers moving faster in those complex environments, and I don't know who would say no to that, I would consider Mongo. But if you're in a traditional organization, especially, and you're trying to break down some silos with Mongo as a new way of thinking, make sure that you engage MongoDB for support and assistance in doing that. And make sure you focus on your culture. Because those things can hurt what is an initially very positive trajectory as things scale and they get out of control.
So engage Mongo. They've been fantastic with us at Gap. Absolutely responsive, unbelievably. We heard from Verizon very similar things. They're a great, great company to work with.
And again, I really stress, great software is all about people. And so my MongoDB can be a great enabler of your people. I thank you very much. If anybody wants to talk more about what ThoughtWorks does, again, very happy to talk about microservice architectures, more about what we do at Gap.
But again, thank you for staying through the afternoon session. I appreciate it. Have a great conference. [APPLAUSE]
It's really hard to pick, Ryan, which of the things that you said was my favorite, I confess. 2 and 1/2 months to build an order processing system. Larry, can you tell me if you could build a new order processing system at Saks Fifth Avenue in 2 and 1/2 months? What do you think? It's a pretty exciting statement. 37 minutes to spin up an entire 5 node cluster.
But my favorite thing, I have to confess as a business person, not a technician, is the way you described intraday that you could take something you're working on, the PO system, and they want versions. I can put it in front of the business person and say, what do you think? No friction to build out a system that everybody likes. The business sees it as you develop it and can change it on the fly. To me, that's pretty amazing. Thank you for sharing that story with us. Surely, someone else has questions for Ryan. All right. here we go. Behind you.
Could you just mention very briefly how you deal with the reporting side of the application?
Sure. So this is a constant evolution. It's a complex topic, so I'll try to talk just briefly. Operational reporting and analytical reporting, I think of as slightly different. We tend to drive operational style reports directly off of the Mongo database if they need to have sensitive data. For larger scale reporting, we're actually trying to move towards a more microservices-based approach, where individual business context, one or two or three reports about the same thing, are also managed by a separate MongoDB. And we want to use that event strategy to allow the reporting context to pull data off of the main database and then present it.
But I'm trying very much to steer away from the big, massive data warehouse approach. So MongoDB tends to be a really great tool on both sides of that fence, on the OLTP side and also on the reporting side. But I can't say that everybody at Gap necessarily agrees with me about that. And so there is obviously a big data warehouse group that has other ideas about how that's going to work. They probably will still, though, take advantage of the idea of the eventing and letting the application expose changes as events that they can source in real-time.
We don't do any kind of batch things. And we don't do any database based integration. It's very important to me. Nobody should touch my database. In fact, one of the bad things about MongoDB, again, is that it's got this catchy name.
And so everybody wants to reach in at the Mongo. And I'm like, you don't get to do that. Just stay there. Or they complain that their software of choice doesn't know how to talk to Mongo. And I say, you shouldn't. You should talk to my service layer. So we try to virtualize the service layer, use events to allow real-time data transfer into a purpose build reporting environment. Yup. 20 years ago, I built a system for advertising purchase orders. This is the first day I've encountered Mongo. I may be asking a completely idiotic question. [? Dot ?], presumably, has to pay that vendor they're issuing the purchase order to.
They may have a traditional, relational master file of vendors' information about how credit worthy that vendor is. Was that a problem that you had to encounter? Does it cause any wrinkles in terms of MongoDB database design? So again, it goes back to my architectural philosophy, is that databases aren't integration tools, not anymore. They've been used for too long that way. And the tangle, again, gets in the way of developers being able to be productive. And so we were very rigorous on this project and on several others in saying whatever those external dependencies are, they stay external. That data comes into our application or is fetched in real-time.
We're trying to expand the service footprint. So a lot of that legacy data becomes available as services so that new applications get to live in a pure service world. But I think to your question about Mongo, never would that reality touch Mongo. Mongo is very tightly controlled in a very tight business context. And we use service-based integration to deal with most of the legacy problems. Yes sir.
So since-- I'm wondering if you actually ended up changing the schema and how you ended up governing that schema. Yeah. So we don't use a lot of traditional governance. We leave it to the team to keep the application functioning. And we are heavy believers in test-driven development. We have huge automated test suites. When I say huge, appropriately large, whatever that size is. And so we tend to rely, again, on service-based integration.
So what we govern is the service interfaces that other applications will interact with and how data fields are managed and named, et cetera at that level. The database is not something anyone else should touch. So only my application code needs to understand it.
I think that's an important enabling practice to keep others out of your database. Because again, when others want to touch your database, you now aren't so clear about what your constraints are. And so you become fearful and you don't change things. So that's one of those places where I push governance up to the service level. And that way, the developers own the database with the application. And they can evolve it if the pacing's right, taking advantage of features in Mongo, like online updates.
We do schema migration on production data all the time, because Mongo, again, it may be schema agnostic, but your application isn't. Your application cares what your data looks like. But we get strategies with Mongo. I can do that at deployment time. I can do at run time. All that, though, says that the schema at any point can be fuzzy. And only the application can understand that.
And if I were to expose that and I would apply governance to it, I'd lose a lot of the benefits that Mongo gets me in terms of speed. So I think layering your architecture and providing governance at the right layer is critical. But letting the core be more malleable is very important to developer productivity. Anything else?
Any other questions? Yes.
Can you tell us a bit about how did they even get into MongoDB from the data sources?
Sure. Sure. So again, we're very strict about service-based architecture in this case. So we do not pump data into MongoDB except through the application itself. So we expose a service. And that service becomes an endpoint that's either pulling or receiving push. And then that service is responsible for putting the data in MongoDB, which guarantees, again, that the relationship between my application code in Mongo is owned by that development team. And there's no risk that I get into constraints I don't understand. Obviously, that could be an efficiency issue for some people. I would still urge a service-based approach as the only way to successfully leverage Mongo's incredible flexibility without getting into the traditional problems. And you can make very efficient service layers on top of your Mongo. I would absolutely urge people to stop doing database based integration. Mongo's a great time to break that habit. Anyone else?
I was wondering, do you manipulate inventory data within your POS system? And how do necessarily award overselling during peak times?
So we don't. The purchase order system and a lot of the supply chain systems we're talking about, the supply chain is upstream of the inventory. So POs are being written against vendors to produce the goods. So that doesn't deal directly with the overselling. That's a different set of systems.
That said, Gap has a virtualization system and approach for having a cohesive view of inventory from all their various distribution centers and stores around the world. . And that does not currently use MongoDB. But if I have my way, I may just influence that. But that's a different story.
So that's not a current problem we deal with with MongoDB. But we do have a cohesive system that does track inventory movements and tries to keep a real-- it's not real-time, but intraday counts change. Mind if I go here first? Just real quick. What systems were you using prior? And who are you with in the PO system to reengineer? Were you using SAP--
Sure. So the PO system, there's--
Can you repeat the question?
Sorry? Can you please repeat the question?
Yeah. So he was saying, what systems were we using for purchase orders before we decided to do a custom build? And so we were originally-- we still do have a large legacy system, AS 400, I think. I'm not completely sure, CICS transactions, a lot of stuff, a lot of the business still runs on that.
And what we were actually doing was a what we call a strangler pattern. We were trying to build new capability that would eventually retire the old capability, but we were using them in parallel. The new PO system was serving international markets. And so the arc of things was, legacy is still there and it's still doing many parts of the business. The international is running on this PO system.
And in the interim, they actually had attempted to implement a package. I don't remember the name, to be honest with you. But failed in an implementation of 18 months because Gap's supply chain is quite complex. And it has quite a lot of permutations that are out of the ordinary. And so if you buy a package, you really need to conform to the way the package thinks.
And Gap is improving their mentality around if we buy, we're going to conform. But at the time, they were still trying to do lots of custom things. And that particular vendor couldn't deal with the customization to their package. And so they realized custom build was probably more--
Could you see that legacy infrastructure being replaced at some point?
Yes. There's a clear need to do that and a desire to do that for many reasons, including economic ones. But it's a very large organization. And it's very hard to get an explicit mandate to take on such a huge task. So we nibble at it from many different directions, which also can be very ineffective. Did you still have a question?
Yeah. Thank you. Not that I speak from experience, but it is theoretically possible that a PO system could be densely, densely entangled in a snarl of direct database integrations.
Oh.
Did you run into that with this? And what's the plan to disentangle those and provide a new interface? Because all those consuming systems are going to have to conform to the new service interface, right?
Right. And so this PO system was effectively a greenfield build, because we were basically looking at an international expansion and building a PO system to support that and looking then to bring it back. And so as things have gone and we've had to come back more towards the legacy integrations, we've been extremely clear about Mongo not ever being connected directly to any of those systems. So in a sense, we are not undoing the tangle from our side. We are pushing our model toward the rest of the world.
And what we're doing is we're basically-- this is a part of the architectural strategy that I'm driving-- is encapsulating finding the business capabilities that are currently tangled. Because it's not the PO system, but lots of other things are very tangled. So we're driving what I call a capability-centric architecture.
What is the business capability that could be independent of another one? Create a service facade around that. And then dip back into the legacy technologies that are supporting that. And slowly, slowly behind the scenes, we untangle. But what that gives us is a clean facade for the future. So as new consumers come, they consume uniformly from the service interface. And bit by bit, that lets us hide some of the skeletons from new development and slowly untangle.
So for example, if we have POs for some consumers should be the same whether they're international or whether they're domestic. They just need PO information, merchandise payables matching, these kinds of things. So we want to put a facade in front of the old PO system and the new one so that we get common interface to that for almost anybody who doesn't totally care where the details are. And over time, as things shift from one system to the other, it will be transparent. This requires a heck of a lot of architectural discipline though. Anything else? Well, thank you for the questions.
OK, thank you so much for your time today. Thank you everyone who managed to get through with us. Tomorrow morning, we'll meet again here at 10 o'clock. We have eight great more business tracks, Reinventing Your Business using MongoDB. At 5 o'clock, there's two more presentations. And 6 o'clock tonight, start the busses from here to the Intrepid. Hope we see you all later. Thanks for staying. [MUSIC PLAYING]