Imagine you are a financial advisor or investment broker who has had a tough day. You’ve missed on all of the big bets you’ve made that day and need to figure out a better way to stay on top of the markets.
There is a solution that can help. Nerve Solutions is a fintech and regtech consulting and development firm that specializes in real-time risk management and automated trading products for financial markets. Nerve Solutions wrestles with massive volumes of critical data in real time to help financial services professionals make critical decisions based on the freshest data. It also provides developer tools and APIs allowing brokers to offer tech-enabled products to their end clients.
Whereas some financial sites experience noticeable slowdowns or even crash on high-volume trading days, Nerve Solutions’ n.Prime product can perform more than 900,000 data runs per second while letting customers monitor complex options portfolios in a multi-client, multi-dealer, and multi-broker setup. Its n.Feed product lets authorized brokers consolidate feeds from across an entire market over a single connection to reduce complexity and help them focus on the most important market movements. n.Image is the company’s proprietary low-latency API that helps compute margin requirements. n.Tasc offers plug-and-play APIs for Strategy Creation, Scenario Analysis, and more to help with building an end-to-end Automated Trading and Analytics Platform.
Nerve Solutions Founder Mihir K. Malani has been an RDBMS expert for most of his career, and that experience taught him that an RDBMS was no match for the volume, velocity, and variety of data required to help some of the world’s most important financial services firms stay a step ahead of global financial markets. The relational model was never designed to manage today’s Web and mobile streaming data, because it was conceived and designed decades earlier, before those data sources even existed.
So a few years ago Malani made the move to MongoDB — and improved his company’s database performance, the time it takes to access market data, by 500%.
Malani discussed a simple use case that illustrates both the challenges and the far more complex feats of data handling that Nerve Solutions and MongoDB perform every day: modeling stock data, such as the data appearing in a stock analyst’s candlestick chart for Apple (Nasdaq: $AAPL) in Figure 1.
Another challenge: in the relational world, users have to map features into columns and every new data point needs to be stored in a row. A typical table for financial markets might contain hundreds of thousands of rows. All rows must contain the same number of columns, and if any rows vary from that, you have to divide them out into a separate table. Malani explained, “Embedded data sets? A table inside a table? In the relational world, that’s heresy!” Plus, creating JOINS to tie together all of that data scattered across all of those tables kills performance. “With most RDBMS community editions, as soon as your data starts to grow you start experiencing performance issues. That was one of the major reasons for us to move from RDBMS to MongoDB,” he added.
To obtain the real-time insights based on massive volumes of data, Malani said you must unlearn what you’ve learned over the years dealing with the challenges and limitations of relational databases. In short, he exclaimed: “Think different!”
Mihir K. Malani, Founder, Nerve Solutions
|How it stores data||Data, even for a single stock, |
is not stored together
|All data for the same stock is stored together in a single document, and all data for all stocks is stored together in a collection|
|What you must do to delete a stock from the database||Delete multiple records — |
which can impact other records
|Delete a single document|
|Average fetch time for a single record across two million records||10.3 milliseconds (ms)||2.4ms|
|Data set capacity||Inability to manage a data set |
containing 10 million records
|Ability to work with data sets|
in excess of 500 million records
By using MongoDB, Nerve Solutions accesses stock data 500% faster than it could with a relational database. That adds up to a quantum leap in performance across its business because it enables Nerve Solutions clients to move faster to capture areas of opportunity and avoid risk in the market. The ability to quickly access and, if necessary, delete individual records also enhances a company’s security & compliance readiness for things such a Right to be Forgotten request under the terms of the GDPR.
“In addition to the speed advantage we've gained, MongoDB has given us the added advantage of being able to increase the size of our data sets exponentially. The speed and retrieval issues of our previous database made it impossible to work with data sets consisting of even 10 million records. Moving to MongoDB has allowed us to work with data sets in excess of 500 million records with no signs of performance issues. This has greatly enhanced the scale and coverage of our products,” said Malani.
Mihir K. Malani, Founder, Nerve Solutions
“For many developers and DBAs who, like me, have been dealing with the issues and constraints of relational databases, the door is open to a better way of working with data. To walk through it you need to avoid the tendency to simply replicate RDBMS design in MongoDB and follow data modeling strategies like the ones discussed here.” Malani added, “To obtain the drastic performance improvements you seek, think ‘Embed’ instead of ‘Reference.’ Spend some time with some of the great learning resources MongoDB offers. And most importantly: keep an open mind!”
Moving forward, Nerve Solutions plans to explore using MongoDB in its products that require complex computations on large scale in-memory datasets, said Malani. “We also intend to move remaining areas of our infrastructure from RDBMS to MongoDB, which we feel will help us greatly in working with one common data fabric for all our products. This will help us reduce the number of moving parts in our solutions, thereby making the product on-boarding process much easier for our clients.”