When Coinbase launched in the summer of 2012 to provide users with the easiest way to buy and sell Bitcoin, its founders were worried that they were too late to the game. At the time, the Bitcoin network had already been around for 3 years and there was no shortage of competitors vying for market share. Today, Coinbase is the most trusted and easiest to use suite of tools and services for consumers, investors and institutions to buy, trade, store and utilize cryptocurrency in more than 33 countries. Coinbase stewards more than 20 million accounts and more than $220 billion assets have been traded on the Coinbase platform.
So without having the first mover’s advantage, how did Coinbase become synonymous with cryptocurrency in many parts of the world? In part, the tools that Coinbase uses, like MongoDB and Amazon Web Services, played a big role in helping them scale to accommodate the influx in traffic.
○ Systems of engagement enabling all the user actions and events that occur during a typical user session
○ Systems of record storing user identities, account information, transactions, payment methods, and more
○ Systems of insight used to support and improve the overall customer experience
“The addition of multi-document ACID transactions has the potential to greatly speed [up] our engineering workflow.”
- Niall O’Higgins, Engineering Manager, Coinbase
In mid-2017, public interest in digital assets exploded, in part due to the skyrocketing price of Ethereum. Prior to the corresponding surge in trading volumes, Coinbase’s platform experienced surprisingly consistent traffic patterns, around an average of 15,000 backend API requests per minute (RPM). At the time, the team estimated that the upper threshold where they would expect their platform to experience issues, their redline, was 100,000 RPM or about 6-7 times their typical load. But around June 2017, out of nowhere and literally overnight, traffic blew past this threshold.
The engineering team responded in kind, elevating “build reliable systems” alongside their long-standing strategic priority of “don’t get hacked”. Below are just a few of the steps the Coinbase team took to scale to meet increasing demand.
The team upgraded to more recent versions of MongoDB, including switching over to the WiredTiger storage engine and taking advantage of document-level concurrency control.
Analytical workloads were moved to secondary replicas to take load off primaries serving reads and writes.
Advanced monitoring strategies were implemented, which helped identify a flaw in a fingerprinting algorithm. This was resolved by changing the relationship between users and devices from a many-to-many model to a one-to-many model. Learn more on Coinbase’s blog.
After these changes, the red line was shifted to 200,000 RPM. But demand only rose. Their work was not done.
Next, the team identified high read throughput on certain collections and being under the gun, quickly implemented a query-caching layer in Memcached written at the ORM and driver level. This allowed them to simultaneously roll out the query cache to multiple clusters experiencing issues. The red line shifted once again, this time to 400,000 RPM.
Afterwards, Coinbase embarked on scaling out and sharding their MongoDB clusters through a data-driven approach that involved evaluating all queries and ensuring that they were passing shard keys appropriately. This move had the most dramatic impact on how much their system could handle.
In hindsight, the “explosion” of interest in mid 2017 was merely a blip compared to December and January when the market cap of cryptocurrencies were at their highest and Coinbase became the number 1 app in the App Store. The platform experienced 1.2 million requests per minute (that’s 80 times their average volume from only 6 months earlier!), but due to the work the team had already done, including upgrading, scaling out their MongoDB clusters, and implementing better monitoring and instrumentation, there were no issues.
By working closely with MongoDB and migrating to MongoDB Atlas, the fully managed cloud database, Coinbase has improved the speed at which they can scale — from hours to minutes — and their overall platform resilience by 10x, better preparing them for the future.