Leading Institutional Crypto Data Provider DAR Lowers TCO 50% and Expands Client Base 600% with SingleStore

Cryptocurrency offers a wealth of opportunities for institutional investors, but getting clean and accurate market data to make the right trading decisions can be a challenge for institutional investors. Market manipulation, biased solutions, and lack of standardization in this sector have kept many institutional participants on the digital asset sidelines. 

Digital Asset Research (DAR) wanted to change this landscape for asset managers, banks, custodians, family offices, fintech firms, fund administrators, hedge funds, and venture capital firms by providing ‘clean’, objective pricing and verified volume data for 3,100+ digital assets. 

DAR serves flagship clients such as Bloomberg, FTSE Russell, and Wilshire with expertise in traditional finance and the digital asset space and delivers a range of solutions to navigate the cryptoverse:

• Clean intraday and close pricing data

• Indicators such as market cap, free float, and volume statistics

• Qualitative and quantitative data on digital asset exchanges

• Token data including data on technology practices, security practices, and developer activity

• Market structure data, including price discovery analysis

• Comprehensive calendar information about events affecting the digital asset ecosystem

Challenges/Goals 

Separating the signal from the noise in digital asset pricing is a complex process. In order for DAR’s platform to do this, it needs to be able to vet assets, analyze trades, and produce a rich data set to tie it all together in a fast-paced, 24x7 market. 

The platform it would build needed to handle both real-time and historical data analysis, with 60 billion historical trades and two billion historical aggregated prices available alongside the latest market data. Some digital asset data sets, such as Bitcoin, have more than a decade of data. 

Performing network analytics of all transactions on a chain could involve tens of billions of transactions to create the one-second price and everything in between for an asset. The Digital Asset Research platform collects data from 450+ exchanges in duplicate so it can run multiple processes that are not reliant on each other.

Along with pricing data, DAR also aggregates data for a Crypto Events Calendar to provide a centralized view of events, technical developments, policy changes, token and product launches and listings, and other important news in the crypto world. With our previous database, it was hard to keep track of all of this in a consistent way.

Powering this type of solution requires a massive amount of data. Here’s a look at DAR by the numbers:

• 3,100+ crypto assets priced 
• 140 million trades per day
• 450+ exchanges evaluated
• 450+ SEDOLs assigned to digital assets

Technology Requirements 

The technology requirements to support such a data-intensive platform included: 

• A performant analytics engine with columnstore to democratize data access 
• Real-time analytics for very fast reads and quick ingestion
• A single, unified data store 
• Low total cost of ownership (TCO), as DAR is a startup with a lean engineering team 
• Scalability to support its accelerated growth trajectory in the digital asset market

The team reported that when it was running the system on AWS Aurora (MySQL), it had a hard time deduplicating and allowing other users to query data. If there were six pricing engines running against it as it was updating, there were all kinds of exclusive locks. The database would get backed up and go down every few days. Aurora also had limited analytical capabilities and was cost-prohibitive to run at the scale DAR required. A new database solution was needed to get DAR to the speed and scale it needed to achieve its aggressive goals.

Why SingleStore 

The team knew pretty quickly that AWS Aurora wasn’t going to be a great solution for its needs, so it started experimenting with Snowflake and SingleStore at the same time. It also worked with SingleStore partner Data Sleek to get its MySQL more stable and help it come up with a better environment. 

Price performance became the key factor when choosing between the two technologies. The team reported that with Snowflake, their pricing transparency just wasn’t there and they experienced sticker shock the first month they went to use it, so that became a nonstarter. While Snowflake had better query speeds than Aurora and a high compression rate, it didn’t deliver the exceptional performance DAR was looking for. SingleStore stood out from the start with its cost-effectiveness, and brought the necessary high performance capabilities along with it. Then when they saw that costs would actually go down with SingleStore, it was an easy decision to migrate everything.

SingleStore features and benefits that stood out the most to DAR included: 

• Columnstore capabilities with high compression rates for DARs massive data sets 
• Streaming data and analytics that dramatically minimize time to insights with real-time streaming ingest and blazing fast queries
• Effortless scaling of applications and data sources with a tiered-storage model that offers cloud elasticity and infinite scalability
• A cloud-native managed service offering with AWS support 
• High concurrency
• A robust, user-friendly pipelines tool for setting up data ingestion in minutes
• Unbeatable price performance 

Solution 

DAR opted to deploy SingleStore Helios. DAR had 120 AWS EC2 instances and a platform built around MySQL. With SingleStore, for most of these instances, all the team had to do was change the connection string to get it working. The deployment took only three weeks, as DAR did a lift-and-shift to start and brought historical data over later. In only a month, DAR was completely migrated off AWS Aurora to SingleStore.

The ability to leverage SingleStore as a managed service was very helpful. Unlike a lot of the other markets, the crypto space is 24x7 and the team simply can’t have any downtime. Its real-time prices need to run all the time and that was challenging to do with AWS Aurora.

During the process of optimizing performance on SingleStore, DAR moved to a message-based architecture for processes that need the highest performance, and for that it uses Confluent Cloud and AWS. DAR also uses shard keys and columnstore to further drive performance improvements.

Outcomes

With SingleStore, DAR has massively expanded its customer base, cut its TCO in half, and improved productivity. The platform can do everything the team needs it to in less than a second.

1,000X Faster Data Ingest = Real-Time Data for DAR Clients

The 1,000X faster data ingestion it gained with SingleStore provides real-time data for DAR’s clients and speeds time to market for DAR itself. The team found that SingleStore pipelines are especially useful, and even managers actually write the pipelines themselves because they’re pretty straightforward to use and within minutes they can load a pipeline into production.

“SingleStore handles this extremely well. We’re able to consume data in duplicate, deduplicate it, normalize it, load it into the database, and feed it to our pricing engine in under one second.“ 

Modern Architecture Enables a 50% Reduction in TCO

SingleStore’s modern architecture, AWS support, and price-performance capabilities decreased DAR’s TCO by 50% compared to AWS Aurora MySQL and Snowflake. 

Processing 4-7X More Data — with Extreme Performance 

DAR scaled from 20-30 million daily orders to 140 million after switching to SingleStore, with plenty of room to grow. The performance was also a breath of fresh air for the company. Some batches are running in about seven milliseconds, whereas on AWS Aurora, there would be an operation on the database that would take an exclusive lock that took minutes. SingleStore enables DAR to be in the order of thousands of times faster.

1-Second Price Refresh Helped Make DAR the Digital Asset Pricing Provider on the Bloomberg Terminal 

Before SingleStore, DAR had pricing that refreshed every 15 seconds on its old infrastructure. Now with SingleStore, pricing refreshes every second—which has enabled DAR to become the pricing provider on the Bloomberg Terminal. SingleStore also enabled DAR to start consuming order books to provide the best bid and ask on the prices provided. It took DAR just 10 minutes to ramp up with SingleStore’s pipelines to model the table, look at the JSON object, and begin consuming the batches for this process.

Expanded its Client Base by 600%

SingleStore allowed DAR to easily add features here and there to better cater to its institutional clients. It grew from four clients a year ago to more than 25 clients today, representing more than 600% growth.

Improved IT and Workforce Productivity

 With SingleStore, the team no longer has to worry about emergency outages at one o’clock in the morning or spending 10 hours to recover 30 minutes of lost real-time data.

SingleStore is helping companies compete and win across every vertical. Learn more →