When Coinbase launched in the summer of 2012 to provide users with the easiest way to buy and sell Bitcoin, its founders were worried that they were too late to the game. At the time, the Bitcoin network had already been around for 3 years and there was no shortage of competitors vying for market share. Today, Coinbase is the most trusted and easiest to use suite of tools and services for consumers, investors and institutions to buy, trade, store and utilize cryptocurrency in more than 33 countries. Coinbase stewards more than 20 million accounts and more than $220 billion assets have been traded on the Coinbase platform.
So without having the first mover’s advantage, how did Coinbase become synonymous with cryptocurrency in many parts of the world? In part, the tools that Coinbase uses, like MongoDB and Amazon Web Services, played a big role in helping them scale to accommodate the influx in traffic.
“[The founding team] optimized for technologies that would allow them to launch fast,” explains Luke Demi, Software Engineer, Coinbase, “Those initial technology decisions turned out to be extremely important because the cryptocurrency ecosystem has evolved dramatically. MongoDB is important because it allowed us to rapidly adapt to those changes.”
The Power of Data Model Flexibility
Coinbase started out as a Bitcoin wallet. A trading platform was added not long after, along with support for several other cryptocurrencies including Ethereum, Litecoin, and Bitcoin Cash. Later, products were released for advanced traders and institutional investors, such as Coinbase Pro, Coinbase Prime, and Coinbase Asset Management. Each time new products are added, MongoDB’s flexible document data model allows the team to quickly update existing data structures as necessary.
Today, Coinbase uses MongoDB to power
Systems of engagement enabling all the user actions and events that occur during a typical user session
Systems of record storing user identities, account information, transactions, payment methods, and more
Systems of insight used to support and improve the overall customer experience
MongoDB 4.0 adds support for multi-document ACID transactions, making it the only open source database to combine the speed, flexibility, and power of the document model with ACID guarantees.
“The addition of multi-document ACID transactions has the potential to greatly speed [up] our engineering workflow.”
- Niall O’Higgins, Engineering Manager, Coinbase
MongoDB’s data model also accelerates development at Coinbase, a big Ruby on Rails shop. Usage of MongoDB removes the object-relational impedance mismatch commonly associated with leveraging tabular databases in modern development environments, and translates to more productive and satisfied engineers.
According to Demi, “Flexibility goes hand in hand with another benefit of MongoDB, which is keeping developers happy and working on the features that matter ... We’ve prided ourselves on our ability to allow developers to ship code to production in days, not weeks.”
Don’t Get Hacked
Since 2011, there have been over 50 attacks targeted at cryptocurrency exchanges, initial coin offerings, and other digital currency platforms. High profile cryptocurrency hacks such as the attack on the Mt. Gox exchange in 2014 — which saw 7% of the world’s supply in Bitcoin stolen — not only impacted customers but also had implications on the trajectory of digital asset adoption.
“Up until some time in 2017, our number 1 strategic priority, everything we talked about as an engineering team, was ‘Don’t get hacked’,” says Luke Demi, “We have 20 million user accounts that we need to protect to be able to uphold our reputation as the most trusted cryptocurrency platform.”
MongoDB helps the Coinbase team address security challenges by providing a robust set of security controls at the data layer including end-to-end encryption, granular role-based access controls, and more. The database’s flexibility also allows rapid integration with 3rd party security vendors; for example, Coinbase was able to integrate a new identity management service into their platform in under 2 weeks to further protect against fraud.
Building Reliable Systems to Address Crypto Mania
In mid-2017, public interest in digital assets exploded, in part due to the skyrocketing price of Ethereum. Prior to the corresponding surge in trading volumes, Coinbase’s platform experienced surprisingly consistent traffic patterns, around an average of 15,000 backend API requests per minute (RPM). At the time, the team estimated that the upper threshold where they would expect their platform to experience issues, their redline, was 100,000 RPM or about 6-7 times their typical load. But around June 2017, out of nowhere and literally overnight, traffic blew past this threshold.
The engineering team responded in kind, elevating “build reliable systems” alongside their long-standing strategic priority of “don’t get hacked”. Below are just a few of the steps the Coinbase team took to scale to meet increasing demand.
The team upgraded to more recent versions of MongoDB, including switching over to the WiredTiger storage engine and taking advantage of document-level concurrency control.
Analytical workloads were moved to secondary replicas to take load off primaries serving reads and writes.
Advanced monitoring strategies were implemented, which helped identify a flaw in a fingerprinting algorithm. This was resolved by changing the relationship between users and devices from a many-to-many model to a one-to-many model. Learn more on Coinbase’s blog.
After these changes, the red line was shifted to 200,000 RPM. But demand only rose. Their work was not done.
Next, the team identified high read throughput on certain collections and being under the gun, quickly implemented a query-caching layer in Memcached written at the ORM and driver level. This allowed them to simultaneously roll out the query cache to multiple clusters experiencing issues. The red line shifted once again, this time to 400,000 RPM.
Afterwards, Coinbase embarked on scaling out and sharding their MongoDB clusters through a data-driven approach that involved evaluating all queries and ensuring that they were passing shard keys appropriately. This move had the most dramatic impact on how much their system could handle.
In hindsight, the “explosion” of interest in mid 2017 was merely a blip compared to December and January when the market cap of cryptocurrencies were at their highest and Coinbase became the number 1 app in the App Store. The platform experienced 1.2 million requests per minute (that’s 80 times their average volume from only 6 months earlier!), but due to the work the team had already done, including upgrading, scaling out their MongoDB clusters, and implementing better monitoring and instrumentation, there were no issues.
By working closely with MongoDB, Coinbase has improved the speed at which they can scale — from hours to minutes — and their overall platform resilience by 10x, better preparing them for the future.
“Coinbase’s mission is to create an open financial system for the world,” said Niall O’Higgins, Engineering Manager, Coinbase, “MongoDB’s technology is enabling us to scale globally and we’re looking forward to partnering with the company along our journey to become the most compliant, reliable and trusted crypto-trading platform in the world.”