DataDock Powers Institutional Investors’ High-Stakes Decisions with Real-Time Data and Analytics from SingleStore on AWS

In the virtual deal rooms and trading floors dominated by Wall Street power brokers, every transaction is high-stakes, and any advantage is sought with an almost religious fervor. Top-tier banks and broker-dealers are increasingly looking to DataDock Solutions, a SaaS firm offering what it calls “a revolutionary approach to leveraging institutional sales and trading data,” for curated data sets and analytics that unlock insight into every dimension of potential transactions and trades.

Helping the World’s Largest Hedge Funds Structure Products and Execute Trades

“Our customers are structuring financial products and executing trades for their customers, the world’s biggest hedge funds,”  said Kumaran Vijayakumar, CEO and Co-Founder, DataDock Solutions. “When you’re putting together a complex transaction and want to place the opportunity with the right customer, it’s impossible to figure out which hedge fund that should be, at what price, using the very basic data most banks’ enterprise customer relationship management (CRM) systems contain. When you have a relationship with a hedge fund that produces millions of dollars in revenues, the data captured in the CRM system is usually too simplistic to capture all the dimensions of what that relationship is really worth.”

Kumaran Vijayakumar, CEO and Co-Founder, DataDock Solutions

BI for Capital Markets: Data Analytics at Scale

“We are taking a business intelligence [BI] approach for capital markets and the financial industry,” added Martin Adamec, Chief Technology Officer, DataDock Solutions. “We are bringing BI to slice and dice data like a Tableau or Sisense, but at a massive scale, with the customer’s own data on our platform and DataDock analytics on top.“

Martin Adamec, Chief Technology Officer, DataDock Solutions

Challenges/Goals

When DataDock was founded in 2018, its solution used two datastores:

• MariaDB as a primary column store database
• MongoDB to capture loosely structured data

“We chose MariaDB mostly because of its combination of columnstore functionality and a MySQL relational database, which was compatible with our staff’s MySQL-style skills,” Adamec explained. MongoDB was easily deployed in its Community version.

“When we were purely presenting data daily, this database arrangement  was enough,” explained Adamec. But DataDock experienced significant problems with database stability in the second year, when it became clear that the columnstore function in MariaDB was not meeting its requirements. While stability issues had not yet become noticeable by DataDock’s customers, the company could see that the situation would soon be problematic.

“MariaDB’s de-emphasis of columnstore was a move away from what we needed,” Vijayakumar said. “We were a small customer in a minor corner of their business, which is not a good place to be.” While making a transition to a new database provider is usually not easy,  Vijayakumar and Adamec knew  the company had to make a change, and they searched for a new database provider.

Technology Requirements 

DataDock’s requirements for a new database were simple yet significant. First, the company needed to store “huge chunks of data, tables with billions of records in production,” Vijayakumar described. “We need access to that data with single-digit millisecond response times.”

Second, support for JSON data was also paramount. “We knew we needed support for loosely structured data within the same database,” said Vijayakumar. "Our previous two-datastore approach using MongoDB to store JSON data separately was unstable and unsustainable.”

Why SingleStore

Here, SingleStore’s combination of in-memory architecture and columnstore, along with SingleStore’s Bottomless Storage, met DataDock’s requirements. 

“We chose SingleStore Helios because we like to cook our own soup,’’ said Adamec. We buy all the ingredients from fine stores but cook the soup ourselves, with our  top-notch team of  engineers and administrators. This allows us to avoid vendor lock-in with any single cloud services provider.”

Reliability is extremely important for DataDock’s customer base. “Once traders start executing millions of dollars in trades on any platform, being out of service for 15 minutes is a huge and unacceptable problem,” said Vijayakumar. “SingleStore will allow our trader chat room to meet the ‘five nines’ service level agreements (SLAs) that Wall Street banks require.”

Solution

SingleStore Helios powers DataDock’s current financial data intelligence solution for capital markets, and is at the heart of an upcoming offering: a modern chatroom platform for traders. Of the chatroom, Vijayakumar said, “It’s a very structured and defined platform where users can watch the progression of trades together, helping to speed their reaction time tremendously. They can make calculations on the fly, and adjust their bids and prices in real time. You can think of it as a StockX for complex financial services.”

Underpinning DataDock’s enterprise data analytics, reporting, trading and intelligence software, SingleStore powers the company’s highly efficient and scalable data architecture, delivering comprehensive, curated data aggregates without legacy overhead.

DataDock currently has three suites of applications in production on SingleStore, running on a high-availability cluster comprising four nodes and two aggregator nodes. “That’s plenty of horsepower for the performance we require,” said Adamec. “The relational database is central to the architecture of our applications, and we use columnstore in a traditional way. We are using SingleStore mostly in a transactional manner thus far; when there is new data to enter, we enter it as a transaction, updating tables with new data daily.”

Other highlights of DataDock’s SingleStore environment include: 

• Using SingleStore pipelines for batch loading of data from AWS S3, also with Kafka event streaming
• Handling up to a billion rows in columnstore tables 
• Implementing DataDock’s own Change Data Capture (CDC) approach on top of SingleStore, to ensure that when any user makes any change, all other users see it in real time. DataDock uses SingleStore to update all logged-in users.
• Moving toward using tables with heavily populated JSON fields

Flexibility to Meet Banks’ Requirements

SingleStore also affords DataDock customers the flexibility to incorporate DataDock into their technology environment the way they want, in any configuration of cloud and on-premises. “We actually don’t want to give our customers a turnkey solution,” Adamec said. “Wall Street firms are not just the most sophisticated users of technology on earth, they are among the most secretive about how they use it to achieve a competitive advantage. They want to do the heavy lifting.”

SingleStore Support Is a “Big Distinguishing Factor”

Adamec appreciates the quality of support DataDock receives from SingleStore, particularly compared to what the company experienced as a small MariaDB customer. “I’m sending a big shout-out to SingleStore’s support people. They are very responsive and involved in every single aspect of our questions,” he said. “When we need assistance the SingleStore team is very educated, helpful, and offers not just solutions to problems but other suggestions. They educate us on how your technology helps to resolve issues. It’s a big distinguishing factor.”

Moving Toward an Intra-Day Future

Adamec explained the importance of SingleStore’s ability to handle unstructured data. “Shifting JSON structures to SingleStore gives us a long-term solution for unstructured data,” he says. To that end, DataDock is using SingleStore as the data foundation for its forthcoming internal structured chat platform for trading, which allows participants to collaborate in access, analyze and share data in real-time, in an agile fashion. “Even if we’d tried to do this with MongoDB technically, the licensing structure and cost are too expensive. SingleStore allows us to scale infinitely and reliably.”

“Over the first year of using SingleStore we came to realize it was a very good move for several reasons, including the reliability of the platform,” said Vijayakumar. You can do all sorts of obscure things to SingleStore and remain highly performant, moving ever-bigger chunks of  data back and forth – without any hiccups.”

Outcomes

“At first we thought SingleStore was ‘too good to be true,’ but it delivered performance exactly as advertised,” said Vijayakumar. “Today, SingleStore allows us to give a business answer to a business question. Our secret sauce to simplifying BI in a complex environment  is that we use SingleStore to clean, curate and cut  down massive quantities of raw data to the important parts relevant for decision-making.”

“SingleStore allows us to give our customer base of big banks and broker-dealers many options on how to use DataDock solutions,” continued Vijayakumar. “We have a very large set of potential clients with lots of ideas; we are providing them curated data, and we want to make sure we can deliver it in different ways as needed. SingleStore gives us the flexibility to do that.”

Vijayakumar said the best benefit DataDock can provide to its customers is being invisible, to be one of the things that performs exactly as expected and doesn’t go wrong. “SingleStore gives us the ability to make the exactly right data and analytics available to our customers at the right time, in real time."

Supporting a Growing Business

DataDock’s business has tripled since the company moved from Maria DB to SingleStore. In addition, “the stability and structure that SingleStore provides has saved us 20% in development and operational costs,”said Vijayakumar. 

Superior Data Handling

“We are dealing with transactional data and market data,  so both current/real-time and historical data need to be available for instantaneous usage,” said Adamec. “We are taking in, curating and storing what we need, and discarding the rest. We are not in the long-term storage business; rather, the data we keep long-term is all client transaction data. SingleStore allows us to handle large volumes of data with ease.”

Future-Proof with Low-Latency, Real-Time Performance

With DataDock moving toward its trading chat room platform, the speed of data handling and query latency will become even more important. Vijayakumar is confident SingleStore can handle any increase in performance demands with ease. “When we are taking in all options data, a ‘firehose’ of data, we will be able to ingest that into SingleStore easily, and then process a few gigabytes of data down to hundreds of megabytes, dropping the rest."

Query response times will be critically important in the trading chat room, as well.  “As we move into streaming and connect to live trading systems, query latency will be crucial,” Vijayakumar continued. “We will have thousands of concurrent users accessing the application in the future, and are confident that SingleStore will continue to deliver single-digit millisecond responsiveness.”

“It’s analogous to ‘five nines’ availability, the classic gold standard of data center operations,” Adamec summarized. “Stepping up from four to five ‘nines’  is complex. The same equation applies to latency – but SingleStore has already got us covered.”

---

SingleStore is helping companies compete and win across every vertical. Learn more →