Existing financial data offerings are not built for modern technology. They can’t keep up with the scale and speed needed to support innovative financial applications. Smaller developers are priced out of the market due to solutions that are too expensive, restrictive, and complicated.
The financial industry has long lagged behind other markets when it comes to technology modernization. Pre-pandemic, only 55 percent of financial services organizations took steps to implement their digital transformation plans. The pandemic helped accelerate the process for many companies, but they still had to contend with legacy systems and inefficient operations.
IEX Cloud, an IEX Group company, tackled this problem with an innovative financial data delivery platform that makes quality financial data and services accessible to everyone.
The parent company, IEX Group, was founded in 2012 with a simple mission: build fairer markets. It started by creating a new stock exchange, called the Investors Exchange, focusing on transparency and access to data to level the playing field for long-term investors, such as mutual and pension funds, that represent the savings of millions of people.
IEX Cloud was founded in January 2019. Josh Blackburn, Head of Technology at IEX Cloud, said, “IEX Cloudʼs vision is to create a vibrant data ecosystem that delivers best in class data to the individual developers and the investor community, as well as those at Fintechs and enterprises large and small. The industry has long been held back by barriers that limit access to the data they need to innovate.”
Pioneering a Better Way to Build and Scale Financial Applications
IEX Cloud’s priorities were to:
The platform provided powerful functionality for developers, with a flexible, scalable pricing model based on the data used. Customers are not locked into long-term contracts and substantial upfront payments. Instead, they get open access to a range of institutional-grade datasets with one subscription, split between core and premium data.
Core data includes:
Premium data comes from a growing community of curated partners, such as:
IEX Cloud was designed to be developer-friendly, making it as easy as possible to integrate data into their projects.
Data creators can use IEX Cloud as a turn-key distribution platform to distribute and commercialize their data. The platform takes care of the complete data delivery process by:
These customers can integrate their data into IEX Cloud and gain instant access to a growing global user base or distribute to their pre-existing customers.
How IEX Cloud Customers Use the Platform
IEX Cloud’s flexibility and accessibility supported a wide range of use cases, such as:
Startups gain particular value from IEX Cloud. Dave Hannibal, Partnerships Manager at IEX Cloud, explained the platform’s advantage for this customer segment.
“We help them scale. They can come in and go from ideation through being a full-fledged fintech with millions of users.”
Hannibal continued, “We can work with them, and shepherd them all the way through that process, and scale with them, instantaneously. They never have to do a thing with us - as we continue to add more data on the platform, they can just grab and go. We are built in a way that helps them scale and then deliver on their value proposition. We have Fintechs all around the world who are building with us, who are integrating us, who are leveraging us for more and more data. It continues to grow across new startups and incumbents looking for new ways to redo old processes.”
Finding the Right Database to Help IEX Cloud Scale Beyond the Startup Phase
IEX Cloud had ambitious goals for its platform, but it needed to solve many technical challenges before it could offer the necessary speed and scale for modern financial applications. The company also had only three months to get the beta up and running.
Blackburn’s goal for the platform’s database was, “I wanted everything consolidated into one store, not a bunch of databases all over the place. Typically, you would split your data into different databases.”
He continued, “Amazon Redshift or Snowflake are great for storing large amounts of historical data and doing analytics. But neither database is going to meet a real-time service level agreement (SLA) for web developers, websites, or financial data. We want a delivery platform that scales to any number of users with real-time use cases, and any amount of data under the hood.”
IEX Cloud started out lean to get up and running with a few data sets. A managed MySQL database was selected initially because it was widely supported, quick to get running, and cost-effective for an early-stage startup.
The platform quickly outgrew that and ran into performance issues. As the team started adding more data and moving past the start-up phase, they needed something more web-scale.
Free Databases Are Not Free
The team chose Clickhouse to alleviate the immediate pain. It was a free solution that had high write speeds for overnight ETL processes.
However, free is not free. Clickhouse optimization was consuming a large percentage of the Site Reliability Engineers’ (SRE) time. Clustering didn’t work as advertised, and would fall over with too many concurrent reads. It was built for large analytical queries and few users.
IEX Cloud’s user base was growing, and Clickhouse wasn’t able to support millions of small reads. While it was great for time series data speed, it was too rigid. At this point, the team had data stored in MySQL, Clickhouse, and Redis.
Searching for Their Permanent Database Technology
After IEX Cloud’s first year and with revenue in the door, it was time to look for a database solution that would meet their needs and scale with the business. The team didn’t want to migrate databases again.
Financial data is complex, comes in many forms, and can change after the write. The database technology for IEX Cloud needed to be up for the challenge.
“We're trying to standardize any data set in the world, into one format and one API signature for our users. It's easier to learn. As we add data, it becomes readily available in the system,” explained Blackburn.
“We want to be able to provide dynamic metadata to our users, to inform new data on the system. If we have library maintainers or developers that access the API, they do it in one standard way. All dates are the same, all request formats are the same.”
Other technical requirements for IEX Cloud included:
Benefits of SingleStore DB for Financial Organizations
IEX Cloud evaluated many databases for its impending move, including KDB, Postgres, CitrusDB, BigQuery, MySQL Enterprise, Yugabyte, Vertica, Sybase, Cassandra, and Cockroach. However, each option fell short in solving for all the use cases the team wanted to offer, except for SingleStore.
SingleStore, a unified database for fast analytics on any data, anywhere, had been on IEX Group’s radar for seven years when the company was developing the Investors Exchange. The familiarity with this database technology led IEX Cloud to revisit this solution for its new platform, halfway through 2020.
SingleStore DB is the cloud-native, unified database built for speed, scale, and agility. It's designed to scale for today's most demanding hyper-scale applications and analytical systems. With ultra-fast ingest, super low latency, high concurrency, and the ability to get started fast anywhere, it offers powerful functionality for innovative financial organizations.
The SingleStore DB distributed database allows financial technology companies to easily capture, process, analyze, and act on data to thrive in today’s insight-driven economy. You can run this solution on any public cloud, hybrid cloud, multi-cloud, or on-premises with commodity hardware, making it easy to add capacity at a fraction of the cost of legacy providers. Getting started is simple by design, as it’s built with familiar, developer-friendly SQL.
SingleStore features that help financial services companies modernize their technology include:
How SingleStore Solved All of IEX Cloud’s Use Cases and Powered Its Growth
“The financial markets produce a lot of real-time data. We need to store all that, and serve at the same time. In database land, that's almost impossible,” said Blackburn. Luckily, SingleStore made the impossible, possible for IEX Cloud.
SingleStore was the only database that both met all of the requirements for IEX Cloud and was cost-effective for the startup. It also supports MySQL wire protocol, which allowed the team to use all their existing code and libraries to save months of migration work.
The initial integration wasn’t instant magic, as is the case with any software. IEX Cloud had unique requirements, and SingleStore provided responsive and knowledgeable support to get this unique build up and running quickly.
The data set at the time was small, measured in terabytes, but the team’s roadmap has them exceeding 100s of billions of records daily. They wanted a foundation capable of scaling automatically and horizontally to petabytes as the platform grew.
The team also found that the database’s deployment and administration process was mature, and SingleStore was responsive in assisting them with the rollout. They chose Google Cloud Services (GCS) for the implementation due to its interface and extremely fast network. SingleStore’s native support for Kafka and GCS pipelines allowed IEX Cloud to automate its ETL processes, so all the team needed to do was point at the GCS bucket.
Blackburn went into detail about the complexity inherent in working with financial data. “The interesting thing about finance is when you’re doing analysis of a point in time. You want to know what affected the market and why something happened. When you have hundreds of datasets that could have impacted that, the power is in the more datasets you have to analyze.”
“We’re not in the business of selling storage, we’re in the business of helping deliver the data where you need it to go and having that economic model on top.”
“Looking at another platform like Snowflake or AWS, our delivery platform is a bolted-on feature. It’s not a deliberate focus on, “How do I standardize my delivery?” or “How do I work on authentication, permissions, and all the complexity and metadata in the financial space?”
“It’s not just a financial data problem. Many people have problems standardizing and delivering their data. How do you monetize or deliver that data? We need to be able to standardize all our data to make it efficient and to address that other point of making this accessible to people who don’t have a large amount of money. SingleStore solved for all those use cases.”
The Collaborative Potential of a Symbiotic SingleStore Partnership
IEX Cloud’s partnership with SingleStore goes beyond using the database technology. Hannibal discussed the collaborative and complementary nature of this relationship and the future possibilities.
“Where SingleStore provides a database, we've used it in a unique way to create a whole distribution platform that is a value-add for SingleStore’s clients, too.” Customers can use IEX Cloud for data distribution and SingleStore for their master data projects.
“You quickly see the symbiosis through what SingleStore and IEX Cloud has. That's different versus the other players in the space. So, from a business perspective, that was very interesting and appealing to me.”
What IEX Cloud Achieved With SingleStore DB
IEX Cloud has grown significantly following the adoption of SingleStore DB for its financial data delivery platform, resulting in:
“This system upgrade was a critical step for IEX Cloud, and not just because we expanded our immediate data offering. Behind the scenes, we've solidified partnerships with best-in-class data creators that will enable us to continue enhancing the quality, coverage, and timeliness of our data. Meanwhile, weʼre able to continue providing valuable datasets at an accessible price point. The result: an entirely new business model for financial data that meets the needs of individuals and businesses across the board,” said Blackburn.
What's Next for IEX Cloud?
IEX Cloud has an ambitious roadmap ahead of it for 2021. The ability to onboard data and get it out through the API faster expands the data and coverage available on the platform, improving the end-customer experience.
Upcoming features and financial data types include a comprehensive reference data model, global indices coverage, IPO calendar data, 147 global corporate actions event types, global futures end-of-day pricing and reference data, and historical fixed income data.