Case Studies
Webinar Recap: The Future of Data Management: A Thought Leadership Chat
Organizations today face challenges in staying current with the technological advancements that keep them agile and competitive — something that’s particularly true when it comes to data management. It’s no secret that data is an organization’s most important asset. A survey from Forrester₁ posed a key question to businesses: “What have been the primary benefits your organization has experienced as a result of using data, data management, and analytics?” The top responses centered around improved customer experiences (28%), data quality and consistency (27%), and customer insights (27%). SingleStore recently sat down with Noel Yuhanna, VP & Principal Analyst at Forrester for a webinar, “The Future of Data Management: A Thought Leadership Chat.” The session dove deeper into how the right data management architecture and strategy delivers the right data, at the right time. Assessing Common Data Management Challenges “Data management is the core for any successful analytical strategy,” says Yuhanna. “Without data, you can’t do AI, you can’t do business intelligence, you can’t do any of the blockchain, data lakes, database stuff. So data is important — but also, data management. Data management means data ingestion, transformation, governance, security and preparation.” Yet as Yuhanna goes on to explain, more and more businesses tend to stitch various data management solutions together, each attempting to solve for its given function. “Today, organizations have a wide variety of data management tools.… I’m sure in your organization you have a dozen or two dozen tools,” says Yuhanna. “And why is that the case? Data ingestion is separate, data catalog is separate, transformation is separate, governance and security is separate.” The result? These platforms and tools don’t talk to each other — leading to a lack of standards and proprietary metadata, and siloed policies that leave businesses in a lurch. Even more, most organizations are behind the curve when it comes to data management architectures that support modern, real-time use cases, including: IoT Analytics Fraud DetectionRisk Analytics Customer 360Data-Intensive Applications Finally, distributed data spread across multiple clouds amplifies issues like performance, latency and quality. Fortunately for businesses, the outlook isn’t all grim. According to Yuhanna, adopting a global data management strategy and operations can help move organizations away from these issues to ensure the integrity, quality and reliability of their data. How to Implement Successful Data Management Strategies With SingleStoreDBReal-time analytics to flatten the curve of COVID-19The notion that users expect data to be up-to-date and at their fingertips is nothing new — and this is true not only for consumers, but enterprises as well. Take truedigital, the digital arm of Thailand’s leading communications company. In the early days of the COVID-19 pandemic, the company wanted to get ahead of tracking the virus to prevent further spread. That meant real-time monitoring, geo-fenced areas, population density alerts and more. The sheer amount of data truedigital needed to sort was massive; 30 million phones with 1 million movements/second. Their existing data management architecture couldn’t process the data quickly enough (or at scale) to deliver the output they needed. As SingleStore VP of Product Rick Negrin shares in the webinar, truedigital had the necessary data mesh and data fabric pieces in place, and simply needed a better strategy to make them work together.By taking a closer look and adjusting at how they were ingesting, managing and distributing their data, truedigtal was able to provide real-time, interactive geospatial analysis and alerts to COVID-19 hotspots. Get the full story: truedigital Powering Customer360 insights for global field dashboardsCustomer data platforms are often the first go-to solution for organizations looking to be more data driven. But this data has to be readily available and actionable for employees that rely on the information to make day-to-day decisions. A great example of this is Dell Technologies’’ Customer360 platform. Spanning across several geographies, Dell was dealing with multiple data silos, touch points, product SKUs and sales people. These disparities meant Dell had a very limited view of what customers were doing. With a better data management strategy that included single-view dashboards with low latency streaming and high concurrency to support multiple users, the issue would be remedied. By pushing all of their data into SingleStoreDB, Dell was able to build that real-time, single-view — empowering their sales team to have much better insight into their customers. Simplifying data management architectures to power cybersecurityThe world’s data is growing at an accelerated rate because it doesn’t just originate from a few operating systems. It’s coming from nearly every device available, all of which send that data to the cloud or backup servers. Security company Armis takes all of this available data from devices and aggregates it in a central location, analyzing and processing the information for enterprises to identify any gaps in their security. Armis manages around 100 billion events per day, with some customers that have over 30 terabytes of data. Armis wanted to power a customer experience that combined speed of ingest and the scale of data in real time, making it easy for users to drill down and get the information they needed to safeguard against security threats. Their previous data management architecture consisted of stitched-together, open-source technologies, which couldn’t stand up to rigorous data requirements — not to mention the added complexity and expense. Using SingleStoreDB, Armis simplified their data management stack, building a system that allowed them to deliver on even their strictest real-time SLAs. Get the full story: Armis Security Don’t Miss the Future of Data Management To hear more about the trends driving the next era of data management — and what your data architecture needs to stay competitive — watch our on-demand webinar, “The Future of Data Management: A Thought Leadership Chat.”
Read Post
Case Studies
How Three Developers Powered Their SaaS Apps with SingleStoreDB
Jack, Josh and Gerry all had a similar problem — they were struggling with legacy single-node databases that could no longer keep up with the demands of their business.
Although each developer’s company serves a different industry, they all have something in common: their applications qualify as data intensive, meaning they need to offer fast, interactive experiences for thousands (or hundreds of thousands) of users on-demand, everywhere and in real time.
Most organizations start with first-generation, open-source single-node databases (like MySQL, PostgreSQL or their commercial variants) to power their SaaS applications — quickly running into several challenges:
Bottlenecks with ingesting, processing and analyzing the fast-moving streaming data necessary to power modern, interactive SaaS applicationsLagging query performance as data or concurrency demands growBuilt on single-node architectures and struggles to scale as your business or users grow Offers little-to-no analytical capabilities to drive fast, interactive user experiences.
How did Jack, Josh and Gerry overcome these obstacles to effectively and efficiently supercharge their SaaS apps? As highlighted in our latest eBook, “A 5-Step Guide to Supercharging Your SaaS Apps,” it starts with finding the right database.
Building the World’s Fastest Website Analytics Platform
Read Post
Case Studies
Webinar Recap: Supercharging SaaS Applications With Foodics
SaaS innovator Foodics revolutionizes the way companies and restaurants handle real-time reporting, inventory management, order submission and employee operations.
Headquartered in Riyadh and available across 17 countries, Foodics is growing rapidly around the world — meaning their need to process data at faster speeds and deliver real-time insights is growing, too. Using their application dashboard, users can quickly glean insights on how their business is performing. This performance can be analyzed per branch, payment methods, products and more. And, users can easily apply filters, group different metrics together and change analysis dimensions to find the exact metrics they’re looking for.
We sat down with Foodics to hear more about their database journey in our latest webinar, “Supercharging SaaS Applications: The Foodics Story.”
Foodics’ Journey to SingleStoreDB
As Foodics grew, so did its user base — effectively moving the company into a new SaaS category: they were now dealing with a data-intensive application. Unfortunately, Foodics’ data architecture at the time wasn’t able to meet those requirements.
“Like any tech company, we started with MySQL,” said Mohammed Radwan, head of engineering at Foodics. “We started with MySQL because it was compatible with what we had, and was easy to use. The thing is, with MySQL, it lacked analytics capabilities. It’s more of a transactional database, not for analytics. It did fill its purpose for a while, but then we needed to grow and expand — and that’s why we chose PostgreSQL.”
It’s a journey not unfamiliar to many SaaS companies and app providers — starting with single-node, open-source databases to get their platform off the ground. Yet as Foodics soon discovered, their existing service provider came with a series of challenges that directly impacted their end users, including:
The inability to scale up and scale out Ongoing service instability A lack of data integrity Constant re-balancing of data Low concurrency that only supported 200 users A lack of AWS ecosystem support
Read Post
Case Studies
How Three Companies Use SingleStore for Data Warehouse Augmentation
Across industries, nearly every business is in the data business—which creates enormous dependencies on popular data warehouses like Teradata, Snowflake, Google BigQuery and Amazon RedShift.
Unfortunately, whether built with ported-over legacy technology or born in the cloud, these data warehouses typically depend on rigid, batch-oriented ETL/ELT technologies to capture, ingest, cleanse and transform data before it is available for analysis and reporting.
Sure, these traditional workflows can strong-arm data into a structured format that fits a predefined schema. But they are poorly suited for the data-intensive applications that form the heart and soul of today’s enterprises. Specifically, traditional data warehouses suffer from four bottlenecks:
1 – Limited support for streaming ingest: Data warehouses were not architected for parallel, high-throughput ingestion of streaming, real-time data.
2 - ETL batch windows: Batch windows inject significant delays into the data flow; dashboards and reports reflect past events rather than present conditions, displaying data that is hours or days old.
3 - Query latencies: Data warehouses were not optimized to handle the low-latency queries required for fast analytics applications and interactive dashboards.
4 – Concurrency limitations: Traditional data warehouses break down under high concurrency workloads supporting large groups of users and can be expensive to scale.
Blowing the walls out
To all companies that depend on data warehouses, SingleStore poses a bold opportunity: What if you could achieve 100x performance improvements over traditional data warehouses and associated data pipelines, running data-intensive analytic workloads, and achieving significant cost reductions? Let’s take a look at three companies that are using SingleStore for data warehouse augmentation (DWA) to achieve improvements of this dramatic magnitude—literally blowing the walls out of their data warehouse constraints.
Customer #1: Leading mobile phone manufacturer delivers real-time data visibility to executives
Situation: Senior executives at this fast-moving electronics manufacturer rely on a Tableau dashboard to monitor the real-time sales and market movements of mobile devices, which requires visualizing data by device, region, price point, product attribute and many other dimensions. Teradata, which powered this executive dashboard, couldn't scale to handle the data growth and concurrency requirements of 400+ queries per second.
Challenge: Slow and lagging performance of the executive dashboards meant users had to wait many hours to obtain fresh insights. These delays adversely impacted product launches, marketing campaigns and supply chain operations. For example, managers could not quickly determine how much raw materials were required to satisfy fluctuating consumer demands. Additionally, the company had to ingest 4 billion rows of new data each day, leading to delays of up to 10 hours to process and display the latest data in the dashboard.
SingleStore Results: Augmenting Teradata with SingleStore enabled the electronics manufacturer to transform day-old analytics into real-time insights, boosting data ingestion rates to 12 million rows per second. SingleStore significantly improved performance, delivering query responses in less than 100ms, even with high concurrencies of more than 160,000 queries per second. Finally, SingleStore’s native connection to Tableau made it easy to populate fastboards (real-time dashboards) via the MySQL wire protocol, enabling a direct Tableau-to-SingleStore interface.
Read Post
Case Studies
GoGuardian’s Education Technology Platform, powered by SingleStoreDB Self-Managed, scales to serve over 18,000,000 students
Over the past year, the pandemic has drastically altered the educational landscape. 55 million K-12 students in the United States suddenly found themselves in online or hybrid learning environments. Educators needed new ways of managing their classrooms, keeping students engaged, and tracking their performance.
Some school districts took an ad hoc approach to the software needed to support digital learning. Educators had to learn and log in to multiple applications, adding complexity and inefficiency to an already harrowing time. These solutions also often fell under general-purpose categories and were not always well-suited for the education market.
GoGuardian Empowers Educators in Digital Learning Environments
GoGuardian is a unified education technology platform specializing in moderating student web activities by using machine learning to facilitate a better learning environment. The software suite combines dynamic data inputs to calibrate student engagement and help educators draw conclusions about their class sessions. They can access all of their filtering, classroom management, student safety, and device inventory needs from a single platform.
The individual products included in the GoGuardian suite include:
GoGuardian Admin: Content filtering and monitoring for any school device or operating system.GoGuardian Teacher: Remote classroom management and video conferencing software.GoGuardian Beacon: To identify students at risk of suicide or possible harm to others.GoGuardian Fleet: Chromebook device management tool for tracking, assigning, and reporting on a school district’s technology deployment.GoGuardian DNS: DNS filtering solution built specifically for education.GoGuardian Parent App: Mobile app that allows parents direct access to their children’s device activity and additional content filtering controls.
Giving School Districts and Educators Everything They Need for Online Learning Environments
GoGuardian covers a lot of ground when it comes to managing digital education, and with that comes complex requirements to make everything function as intended. The platform processes events from more than five million students daily.
Handling this event volume is challenging, and the GoGuardian team had a number of priorities they kept front and center when developing the platform.
Data security: Users’ data must be kept secure.Data fidelity and retention: Any gap in data was considered a functional failure, as the platform needed to reconstruct web activity.Scalability and availability: The system had to scale to meet the needs of millions of concurrent users.Queryability: Data that is not accessible to users is useless, and needed to be avoided.
To meet these priorities, GoGuardian needed to overcome the following technical challenges.
Cyclical Data Generation
GoGuardian’s data generation was cyclical. Since most users are students, schools had little reason to keep the products enabled when classes were not in session. The data generation rate outside of school hours is drastically lower than when school is in session. The difference between the peak and trough traffic is quite large.
High Data Throughput
High data throughput was needed to keep up with the real-time traffic of more than five million students. Every event translated into multiple writes across different tables, as they’re a collection of web clicks or navigation.
Data Duplication
Handling data duplication: A piece of data at time T0 may be updated and reappear at T10 + t aggregated. This data is not identical, but they share the same key with expanded or updated information. The updated event has old and new data. GoGuardian could save on row count and storage for many tables by storing only the most up-to-date version of each event. This approach would also reduce overall compute time.
Mutable Event Data
Event data is mutable and may need to update rather than insert. Databases that are not mutation friendly have problems with this. While workarounds are possible by redesigning data generation to support immutable inserts only, that requires retaining the entire payload of generated data. This approach increases row count, which leads to expensive reads.
Read Query Pattern
Reads are dynamic over many dimensions. The data is grouped, aggregated, and filtered by time, classrooms, student, school, URL, and other factors. 94 percent of queries in GoGuardian require ranking or aggregation over a dimension, while only 6 percent are at the row level.
Pre-calculating requests was not a feasible approach, as the team would need to reduce the degree of dimensions and how dynamic the queries could be. Ultimately, this would have required removing features from the products.
Exploring Database Options Capable of Supporting GoGuardian’s Software Suite
The GoGuardian team tested many types of databases to try to find a solution that met their development priorities and addressed the many technical challenges they faced.
SQL
GoGuardian started on a single SQL database, but once they reached a certain scale, they couldn’t rely on a single instance. They moved to an SQL sharding solution to split writes across multiple databases based on a key. Each database held a subset of GoGuardian’s data.
These sets of relational databases handled high-throughput browser events, with large quantities of rows per table on each SQL shard. Queries without indexes have unreasonable latency on a table of this high-scale. Each query needed to be carefully crafted, particularly when joining with other tables. Without this step, the databases would lock up and impact users.
The relational, row-based SQL databases handled writes relatively well, but reads were problematic. While adding additional SQL databases and resharding would help, it wouldn’t keep up with GoGuardian’s growth and introduced maintainability issues.
JK Kim, a GoGuardian Senior Infrastructure Engineer, said, “We are often too focused on latency, performance, and cost, but rarely ever talk about maintainability.”
The GoGuardian team found that shards were not very maintainable, due to the resharding process and the need for a shard router. The process is not as simple as adding a database cluster. With resharding, they had to:
Register the shard with the shard router.Load the data for the keys it was now serving.Ensure even distribution of the load.
“The dull, mundane, and time-consuming nature of that particular work was something we were not thrilled about having to do,” explained Kim.
The shard router itself presented another problem. Operating SQL shards was dependent on the shard router service knowing which shard is responsible for which key. GoGuardian used this stateful mapping because not all keys were equal in the platform’s traffic load, and the degree of variance was high.
The team allocated shards based on the expected traffic. Database uptime and performance was dependent on the shard router service, which was not ideal. While the SQL shard database offered fast writes and a fast, simple index-based fetch, it fell short on aggregate query performance and was resource-intensive to manage.
Druid
GoGuardian’s next database was Druid. This columnar database was adopted to complement the sharded SQL databases, as it offered aggregations across vast amounts of data and dimensions at high speeds and was distributed by nature for easy scaling.
Because of the way GoGuardian generates data, it requires updates of individual rows.
However, Druid doesn’t offer mutation at the row level. The only way to have data mutation was to run an index replacement job, which replaces an entire data block in a batch process.
To work around this limitation, GoGuardian’s team needed to create non-trivial logic to eliminate the duplicated data during the time threshold when the newer, more correct data could show up. Once these data are finalized, it would trigger a batch job to replace the duplicate data with the finalized, condensed data.
By implementing Druid, GoGuardian removed the requirement for a resource-intensive shard router, but added another outside component to their workload: an index replacement job. It also had many moving parts, including Hadoop, SQL databases, Zookeeper, and various node types that contributed to the complexity. Other roadblocks with this database technology included a lack of joins and, at the time, no basic support for SQL.
The team had to drastically change the way they thought about data and designed tables and queries to make Druid work for the platform. They considered redesigning how the data was generated so updates would not be required, and changing their Druid schema and services to adapt to such data.
Ultimately, Druid didn’t work for GoGuardian’s upsert use case. The time, cost, and engineering effort to make this happen would have been significant. To achieve this without causing downtime for GoGuardian’s users, they would have to replicate data ingestion and storage for an extended period. The team also wasn’t convinced that insert-only data generation was a better choice than their original approach.
Phoenix
Phoenix is an Apache project that adds a layer on top of HBase to allow SQL queries and joins. The GoGuardian team was excited about the potential of this database technology, but the testing process didn’t go as planned. The database bugged out and no amount of restarts or configuration changes could bring it back into a functional state.
GoGuardian needed a production database that is so resilient and versatile that no operation should be able to bring the database into an unintentional, unrecoverable, and inoperable state.
BigQuery, Presto, and Amazon Athena
GoGuardian explored these three products due to their excellent parallel wide queries, although they had non-ideal write throughput. They also had query engines that were decoupled from storage.
Of these databases, BigQuery offered the most optimal write throughput when writing to native BigQuery storage rather than using a schema on top of files. However, the team would have still needed to redesign their data generation to deal with the limitations in write throughput.
Spanner
Spanner is Google’s proprietary, geographically distributed database offered through Google Cloud Platform. This relational database offered strong consistency across multiple regions. The GoGuardian team found Spanner fast and exciting to work with, and it was one of the most impressive candidates during the testing phase. However, the cost projection for the platform’s usage was higher than that of the existing legacy systems.
How SingleStoreDB Solved GoGuardian’s Challenges Without Compromise
After testing so many database solutions, GoGuardian’s team was still struggling to find the right fit for the platform’s use case.
Kim explained, “When we sat down and looked at our incoming data and query patterns, we were at an all-too-familiar place for an engineering team: we needed the best of both worlds with fast writes of row-based databases and the fast aggregate reads of columnar databases. Normally this is where we engineers begin to use our “trade-off” and “expectation management” skills.”
Despite trying many types of databases, partition strategies, and schemas, GoGuardian’s team wasn’t able to come up with a confident solution that didn’t result in transition difficulties or compromises in business requirements. That changed when they discovered SingleStoreDB.
“We first learned about SingleStore in the crowded vendor hall of AWS re:Invent 2017. Another engineer and I started sharing our problems with one of their representatives and ended up talking about data throughput, consistency, high availability, transaction isolation, and databases in general. It was one of the most exciting and enlightening conversations I’ve had, and it changed how we served data at GoGuardian,” said Kim.
SingleStoreDB Self-Managed is a cloud-native unified database designed for speed, scale, and agility. Multiple aggregator nodes serve as the brains of the operation and multiple leaf nodes serve as data storage. The database is coordinated by a single master aggregator. Through the simplicity of its design, SingleStore is able to achieve complex operations at low latency. Several benefits drew GoGuardian to this database technology.
Row and Column Data Stores in the Same Database
GoGuardian’s team was excited about the possibilities offered by a unified database architecture.
“SingleStore supports both types of storage, defined at table creation time. Perhaps most importantly, it allows unions and joins across row-based and columnar tables. I cannot stress enough how important this feature is to us, as it fundamentally changed how we served data by giving us the best of both worlds: the fast writes of a row-store and the fast aggregate reads of a column-store.”
“I don’t know of many database solutions that can support both row and columnar storage types. These features allowed us a degree of flexibility we never had previously.”
High Availability
GoGuardian’s engineers put many preventative measures in place to prepare for inevitable machine failure and plan for incident mitigation. SingleStoreDB helps the team maintain high availability by replicating every write into both a master and secondary partition. If the original master fails over, the secondary partition becomes the master.
Speed
“It’s fast. There are some databases that, by design, cannot get better than 500ms latency, regardless of how small or simple the data being queried is. With SingleStore, we are able to see some queries under 30ms when using proper indexes and partition keys.”
Friendly Support
“Ever since our conversation on the crowded floor of AWS re:Invent, all the way through the time when we dropped the multiple legacy databases that were replaced by SingleStore, we have always enjoyed their assistance and friendliness. It has definitely been a pleasant working experience.”
“For example, when we told them that we needed the ability to backup our data to S3, one of their engineers sent us an unreleased version with that feature to start testing with. We were able to establish a line of communication with their engineering team to round out the feature and even iron out a bug on their end. This line of communication increased our confidence in adopting SingleStore.”
Supporting GoGuardian’s Upsert Use Case
SingleStoreDB allowed GoGuardian to write to row-based tables and pay a read latency penalty for the short period of time while the data is volatile. It then unions that table with a columnar table that holds the final data past the volatile period. The unions and joins across row and columnar tables are seamless in this unified database. Support for SQL made it easy for the team to begin testing this database technology.
GoGuardian’s stream processor continuously writes data in real-time into a row table. Periodically, a batch process dumps stable data into a columnar table with the same schema. When the platform reads the data, queries are run against a view that is the union of the row and columnar tables.
After configuring the appropriate partition keys to minimize data skew, GoGuardian’s team found that SingleStoreDB offered stellar speed and efficiency for their use case.
“Previously, we couldn’t do joins at all in our columnar storage, and heavily frowned upon it in our row storage due to inefficiency. However, because SingleStore allows us to do both, we can now have the properly organized and normalized data we data engineers dream about.”
What GoGuardian Achieved With SingleStore's Help
GoGuardian’s education technology platform now serves over 10,000 schools, 500,000 educators, and 18,000,000 students. The performance delivered by SingleStoreDB during proof-of-concept testing and production deployment includes:
An average throughput of 2.49 million events per minute.Write throughput of 750,000 per second.Read throughput of 400 per minute, due to the dynamic nature of the reads.< 30ms latency for some queries.A cost savings of \$30,000 per month compared to legacy solutions.
“At the end of the day, all databases have their own place in this world. The question always comes down to what it takes to make it work for your company’s specific use cases. At GoGuardian, we found SingleStore to be the path of least resistance. Now that everything is up and running smoothly, I’m both proud of what we have accomplished and thankful for the product and the support we have received from the SingleStore team. I have more confidence than ever before in our infrastructure and its ability to handle strenuous loads during peak hours.”
What’s Next for GoGuardian?
GoGuardian continues to help educators adapt to online and hybrid learning environments through a comprehensive suite of powerful tools and continually updated features.
Are you ready to explore the possibilities that a unified database brings to the table? Contact us to learn more about SingleStoreDB or get started for free.
Read Post
Case Studies
IEX Cloud Processes Over 2.5B API Requests Daily with an 8ms Average Response Time Using SingleStore
Existing financial data offerings are not built for modern technology. They can’t keep up with the scale and speed needed to support innovative financial applications. Smaller developers are priced out of the market due to solutions that are too expensive, restrictive, and complicated.
The financial industry has long lagged behind other markets when it comes to technology modernization. Pre-pandemic, only 55 percent of financial services organizations took steps to implement their digital transformation plans. The pandemic helped accelerate the process for many companies, but they still had to contend with legacy systems and inefficient operations.
IEX Cloud, an IEX Group company, tackled this problem with an innovative financial data delivery platform that makes quality financial data and services accessible to everyone.
The parent company, IEX Group, was founded in 2012 with a simple mission: build fairer markets. It started by creating a new stock exchange, called the Investors Exchange, focusing on transparency and access to data to level the playing field for long-term investors, such as mutual and pension funds, that represent the savings of millions of people.
IEX Cloud was founded in January 2019. Josh Blackburn, Head of Technology at IEX Cloud, said, “IEX Cloudʼs vision is to create a vibrant data ecosystem that delivers best in class data to the individual developers and the investor community, as well as those at Fintechs and enterprises large and small. The industry has long been held back by barriers that limit access to the data they need to innovate.”
Pioneering a Better Way to Build and Scale Financial Applications
IEX Cloud’s priorities were to:
Break down these barriers and constantly reinvent to better serve the entire ecosystem.Challenge the status quo in the financial data industry by delivering high-quality data to their community that is easy to use and available at a reasonable price.Provide a powerful set of capabilities to help developers build efficiently and iterate quickly.Simplify legal, modernize access, and enable flexible pricing.
The platform provided powerful functionality for developers, with a flexible, scalable pricing model based on the data used. Customers are not locked into long-term contracts and substantial upfront payments. Instead, they get open access to a range of institutional-grade datasets with one subscription, split between core and premium data.
Core data includes:
Real-time and historical stock pricesCorporate actionsFull U.S. market coverageCryptocurrencyForexCommoditiesEconomic dataReference dataIEX Investors Exchange data95,000+ securities20+ exchanges100+ currency pairsAnd much more
Premium data comes from a growing community of curated partners, such as:
Wall Street HorizonFraud FactorsAudit AnalyticsValuEngineStocktwitsAnd much more
IEX Cloud was designed to be developer-friendly, making it as easy as possible to integrate data into their projects.
All data is accessed through a single API. This API is based on REST, has resource-oriented URLs, returns JSON-encoded responses, and returns standard HTTP response codes.Support for C++, Go, .NET, R, Node.js, Python, Java, PHP, and Ruby.On-demand and streaming data availability.A no-code Rules Engine that automates serverless data analysis through custom event-driven alerts and data-driven conditions.Unlimited sandbox testing gives developers an environment to try free API calls and experiment with this financial data.The included cloud-based caching offers a fully managed service that automates data storage, scaling, high availability, monitoring, and administration.IEX Cloud provides a platform that can act as a developer’s backend, as some startups lack scalable infrastructure.Automated signup flow so users can get started immediately.
Data creators can use IEX Cloud as a turn-key distribution platform to distribute and commercialize their data. The platform takes care of the complete data delivery process by:
Managing and scaling the APITracking usageBilling customersHandling all other overheadData normalization and symbology integration
These customers can integrate their data into IEX Cloud and gain instant access to a growing global user base or distribute to their pre-existing customers.
How IEX Cloud Customers Use the Platform
IEX Cloud’s flexibility and accessibility supported a wide range of use cases, such as:
Front-office traders needing real-time data.Building out trade blotters and order management solutions.Understanding the market data and its risk in attribution implications on one’s portfolio.Supporting back-office operations and reconciliation.Keeping IR teams up to date about what’s going on in their company and near-competitors’ companies through news coverage, events, and streams of data delivered seamlessly through IEX Cloud’s API to their Business Intelligence (BI) solution of choice.Helping IR teams adjust legacy workflows to the work-from-home environment through a flexible platform.
Startups gain particular value from IEX Cloud. Dave Hannibal, Partnerships Manager at IEX Cloud, explained the platform’s advantage for this customer segment.
“We help them scale. They can come in and go from ideation through being a full-fledged fintech with millions of users.”
Hannibal continued, “We can work with them, and shepherd them all the way through that process, and scale with them, instantaneously. They never have to do a thing with us - as we continue to add more data on the platform, they can just grab and go. We are built in a way that helps them scale and then deliver on their value proposition. We have Fintechs all around the world who are building with us, who are integrating us, who are leveraging us for more and more data. It continues to grow across new startups and incumbents looking for new ways to redo old processes.”
Finding the Right Database to Help IEX Cloud Scale Beyond the Startup Phase
IEX Cloud had ambitious goals for its platform, but it needed to solve many technical challenges before it could offer the necessary speed and scale for modern financial applications. The company also had only three months to get the beta up and running.
Blackburn’s goal for the platform’s database was, “I wanted everything consolidated into one store, not a bunch of databases all over the place. Typically, you would split your data into different databases.”
He continued, “Amazon Redshift or Snowflake are great for storing large amounts of historical data and doing analytics. But neither database is going to meet a real-time service level agreement (SLA) for web developers, websites, or financial data. We want a delivery platform that scales to any number of users with real-time use cases, and any amount of data under the hood.”
IEX Cloud started out lean to get up and running with a few data sets. A managed MySQL database was selected initially because it was widely supported, quick to get running, and cost-effective for an early-stage startup.
The platform quickly outgrew that and ran into performance issues. As the team started adding more data and moving past the start-up phase, they needed something more web-scale.
Free Databases Are Not Free
The team chose Clickhouse to alleviate the immediate pain. It was a free solution that had high write speeds for overnight ETL processes.
However, free is not free. Clickhouse optimization was consuming a large percentage of the Site Reliability Engineers’ (SRE) time. Clustering didn’t work as advertised, and would fall over with too many concurrent reads. It was built for large analytical queries and few users.
IEX Cloud’s user base was growing, and Clickhouse wasn’t able to support millions of small reads. While it was great for time series data speed, it was too rigid. At this point, the team had data stored in MySQL, Clickhouse, and Redis.
Searching for Their Permanent Database Technology
After IEX Cloud’s first year and with revenue in the door, it was time to look for a database solution that would meet their needs and scale with the business. The team didn’t want to migrate databases again.
Financial data is complex, comes in many forms, and can change after the write. The database technology for IEX Cloud needed to be up for the challenge.
“We're trying to standardize any data set in the world, into one format and one API signature for our users. It's easier to learn. As we add data, it becomes readily available in the system,” explained Blackburn.
“We want to be able to provide dynamic metadata to our users, to inform new data on the system. If we have library maintainers or developers that access the API, they do it in one standard way. All dates are the same, all request formats are the same.”
Other technical requirements for IEX Cloud included:
Horizontally scalable to account for a dynamically changing traffic profile and volatile marketsMassive read and write speedFast bulk data loadingEasy to manageGood supportDatabase technology based on standards with wide community adoption and a deep talent poolOn-demand querying for specific data pointsServer-Sent Event streaming for real-time updatesFiltering data for the Rules EngineMeeting real-time SLAsSupport for many traditional and alternative financial data sourcesStoring and serving real-time and historical data at the same timeData delivery at scale for any amount of data and usersAbility to evaluate thousands of data points per secondCost-efficient to support the viability of the business modelImmediate availability of the dataSupport for fast ETL operations for hundreds of data sets per day, scaling up to thousands long-term
Benefits of SingleStoreDB for Financial Organizations
IEX Cloud evaluated many databases for its impending move, including KDB, Postgres, CitusDB, BigQuery, MySQL Enterprise, Yugabyte, Vertica, Sybase, Cassandra, and Cockroach. However, each option fell short in solving for all the use cases the team wanted to offer, except for SingleStore.
SingleStore, a unified database for fast analytics on any data, anywhere, had been on IEX Group’s radar for seven years when the company was developing the Investors Exchange. The familiarity with this database technology led IEX Cloud to revisit this solution for its new platform, halfway through 2020.
SingleStoreDB is the cloud-native, unified database built for speed, scale, and agility. It's designed to scale for today's most demanding hyper-scale applications and analytical systems. With ultra-fast ingest, super low latency, high concurrency, and the ability to get started fast anywhere, it offers powerful functionality for innovative financial organizations.
The SingleStoreDB distributed database allows financial technology companies to easily capture, process, analyze, and act on data to thrive in today’s insight-driven economy. You can run this solution on any public cloud, hybrid cloud, multi-cloud, or on-premises with commodity hardware, making it easy to add capacity at a fraction of the cost of legacy providers. Getting started is simple by design, as it’s built with familiar, developer-friendly SQL.
SingleStore features that help financial services companies modernize their technology include:
Memory-optimized database for delivering operational analytics at enterprise scale.Ingestion of millions of events per second with ACID transactions while simultaneously analyzing billions of rows of data in relational SQL, JSON, geospatial, and full-text search formats.Ad hoc data analysis.Workload management to automatically deliver reliable database performance while under high concurrency.Distributed storage to maximize resilience and performance.Support for petabytes of data.Robust parallel execution engine for read and write queries delivering ultra-fast performance.In-memory and on-disk tables.Comprehensive security for sensitive financial data without compromising database performance.
How SingleStore Solved All of IEX Cloud’s Use Cases and Powered Its Growth
“The financial markets produce a lot of real-time data. We need to store all that, and serve at the same time. In database land, that's almost impossible,” said Blackburn. Luckily, SingleStore made the impossible, possible for IEX Cloud.
SingleStore was the only database that both met all of the requirements for IEX Cloud and was cost-effective for the startup. It also supports MySQL wire protocol, which allowed the team to use all their existing code and libraries to save months of migration work.
The initial integration wasn’t instant magic, as is the case with any software. IEX Cloud had unique requirements, and SingleStore provided responsive and knowledgeable support to get this unique build up and running quickly.
The data set at the time was small, measured in terabytes, but the team’s roadmap has them exceeding 100s of billions of records daily. They wanted a foundation capable of scaling automatically and horizontally to petabytes as the platform grew.
The team also found that the database’s deployment and administration process was mature, and SingleStore was responsive in assisting them with the rollout. They chose Google Cloud Services (GCS) for the implementation due to its interface and extremely fast network. SingleStore’s native support for Kafka and GCS pipelines allowed IEX Cloud to automate its ETL processes, so all the team needed to do was point at the GCS bucket.
Blackburn went into detail about the complexity inherent in working with financial data. “The interesting thing about finance is when you’re doing analysis of a point in time. You want to know what affected the market and why something happened. When you have hundreds of datasets that could have impacted that, the power is in the more datasets you have to analyze.”
“We’re not in the business of selling storage, we’re in the business of helping deliver the data where you need it to go and having that economic model on top.”
“Looking at another platform like Snowflake or AWS, our delivery platform is a bolted-on feature. It’s not a deliberate focus on, “How do I standardize my delivery?” or “How do I work on authentication, permissions, and all the complexity and metadata in the financial space?”
“It’s not just a financial data problem. Many people have problems standardizing and delivering their data. How do you monetize or deliver that data? We need to be able to standardize all our data to make it efficient and to address that other point of making this accessible to people who don’t have a large amount of money. SingleStore solved for all those use cases.”
The Collaborative Potential of a Symbiotic SingleStore Partnership
IEX Cloud’s partnership with SingleStore goes beyond using the database technology. Hannibal discussed the collaborative and complementary nature of this relationship and the future possibilities.
“Where SingleStore provides a database, we've used it in a unique way to create a whole distribution platform that is a value-add for SingleStore’s clients, too.” Customers can use IEX Cloud for data distribution and SingleStore for their master data projects.
“You quickly see the symbiosis through what SingleStore and IEX Cloud has. That's different versus the other players in the space. So, from a business perspective, that was very interesting and appealing to me.”
What IEX Cloud Achieved With SingleStoreDB
IEX Cloud has grown significantly following the adoption of SingleStoreDB for its financial data delivery platform, resulting in:
Over 2.5 billion API requests processed daily with an 8ms average response time.200+ financial data sets.Internal data handling 400,00-800,000 ops per second.Intraday ingress of 60-100 MB/sec and egress of 120-160 MB/sec.After only 2 years, they gained 150,000 users in 120 countries, representing 20 million unique users downstream.10-15x speed improvement over the previous database.More cost-effective than scaling MySQL to meet performance and functionality requirements.The equivalent of 2 full-time-employees’ worth of time saved out of a team of 20 when switching from MySQL to SingleStoreDB.ETL process execution time dropped from days to minutes.
“This system upgrade was a critical step for IEX Cloud, and not just because we expanded our immediate data offering. Behind the scenes, we've solidified partnerships with best-in-class data creators that will enable us to continue enhancing the quality, coverage, and timeliness of our data. Meanwhile, weʼre able to continue providing valuable datasets at an accessible price point. The result: an entirely new business model for financial data that meets the needs of individuals and businesses across the board,” said Blackburn.
What's Next for IEX Cloud?
IEX Cloud has an ambitious roadmap ahead of it for 2021. The ability to onboard data and get it out through the API faster expands the data and coverage available on the platform, improving the end-customer experience.
Upcoming features and financial data types include a comprehensive reference data model, global indices coverage, IPO calendar data, 147 global corporate actions event types, global futures end-of-day pricing and reference data, and historical fixed income data.
Are you ready to harness the power of the world’s fastest database for operational analytics? Contact us to discuss ways we can collaborate or get started with SingleStore DB for free.
Read Post
Case Studies
How dailyVest Powers 401(k)s with SingleStoreDB Cloud and Improves Application Performance by 30%
Tax-advantaged retirement savings plans, such as 401(k)s, are commonly offered as a job benefit for employees. The employer sponsors the plan and may offer incentives such as contribution matching to encourage participation. These plans act as an important tool for attracting and retaining talent, but only if they perform well.
The typical account access website for these plans has outdated tools and technologies and lacks insight into plan performance on an individual and company-wide level. This experience can lead to poor plan health, a lack of plan participation, and employees who are not prepared for retirement.
dailyVest is a FinTech company looking to improve investment and performance reporting for recordkeepers, plan sponsors, and plan participants. Its executive management team has a long history in the financial services industry. Co-founders Daniel Benson and Peter McNellis previously founded and built an industry-respected retirement products software company, Viga Technologies, and wanted to transform how retirement and 401(k) plans were managed by employers.
As of December 31st, 2020, dailyVest had:
\$569 billion assets analyzed12.3 million plan participants analyzed16,200 employer-sponsored retirement plans analyzed\$3.3 billion in transactions analyzed\$28 billion in contributions year to date\$32 billion in distributions year to date
dailyVest’s first project in 2003 was a commercial engine to deliver a personal rate of return, which calculated a plan participant’s investment performance based on their own transaction history and resulting cash flows. This first product was designed to exclusively run on-premises.
As dailyVest developed this engine, the team recognized a bigger opportunity to enable plan sponsors to get the most out of their 401(k) data through a white-label dashboard and intelligence tool.
dailyVest needed easy access to update and improve the initial dashboard product, but the on-premises legacy software sat behind a firewall and made it logistically difficult to move forward. Achieving this goal would require a shift to the cloud.
dailyVest’s Core Product and Expansion Modules
The core product that dailyVest needed to move to the cloud was EnterpriseROR. This solution offers an investment performance analytics and reporting system capable of gathering and analyzing vast amounts of investment account data in real or near real-time. The analytics and intelligence are presented as a series of highly customizable and interactive dashboards. Its capabilities can be further expanded through modules, which include:
planAnalytics: A plan sponsor reporting tool that offers robust monitoring and analysis functionality. Plan administrators and sponsors use this solution to measure how well employer-sponsored retirement plans are working, model plan-level investment behavior, and determine the soundness of these plans.Personal Investment Performance (PIP): This real-time investment performance calculation engine and reporting system integrates with financial services firms’ account access websites.Account Statement-on-Demand: This module delivers customizable, on-demand, and batch-generated investor account statements. These dynamic account statements empower plan participants with actionable plan and account information to help them make better investment decisions on an individual level.Print Batch Output: This high-performance Print Batch processing solution generates personal rates of return and other investment content for all plan participants. It’s used for generating personal rates of return and performance charts for printed account statements and archival purposes.
Surfacing Plan Health Data for Plan Sponsors
Plan sponsors, such as Boeing, use dailyVest’s platform to inform many decisions, such as:
Understanding low plan participation: Plan sponsors can review reasons for low enrollment and create a strategy for encouraging participation based on employee profiles.Addressing low retirement contribution rates: Plan sponsors can revise their retirement education strategy to emphasize the benefits of increasing contributions and the impacts it can have on retirement income goals.Fixing inappropriate asset allocation: Employees with inappropriate allocations of asset classes for their ages can receive proactive communications about the risks of this approach, and an investment strategy that better matches their personal financial picture.Flagging poorly performing funds: Plan sponsors can track the performance of funds and determine whether poor performers should remain available to participants.Sending alerts for unusual money movement: Increased fund inflows and outflows, excess trading, and market timing may indicate undesirable participant behavior. Plan sponsors can act on this information to maintain plan health.
The Challenges of Choosing the Right Cloud Database to Power the EnterpriseROR Engine
The sheer size and scale of this financial data required a robust, cloud-based infrastructure that could keep up with dailyVest’s current and future analytics demand. It took two years to convert the plan health data analysis engine to cloud-compatible software, and dailyVest chose Microsoft Azure SQL for its cloud platform and used it for two years.
However, Azure SQL couldn’t effectively scale to meet the team’s performance and cost expectations as the transaction volume increased. Common operations such as copying or restoring a database took an hour and tasks such as generating all key performance metrics for the latest month took more than four hours to complete.
While their customers weren’t complaining about the speed, yet, dailyVest wanted to offer a better user experience and user interface (UI) performance. The increased storage costs for long-term recovery backups from month to month also increased several hundred dollars per month.
Overall, the team needed a database that could address the technical challenges currently standing in EnterpriseROR’s way and control the rising costs. The ideal solution would:
Provide enough compute power and capacity for cached data generation and to speed up the ingestion processDeliver superior performance to support complex queries that generate large amounts of dataBe compatible with the Microsoft Azure cloud environment to avoid a change from a security standpointHandle a transaction growth rate of 36 percent per yearOffer columnstore for better compression and read performance, allowing dailyVest to handle ad hoc queries against large data volumesEnable dailyVest to continue using SQL due to developer familiarityOffer C# to avoid rewriting code for a different cloud
dailyVest evaluated many database technologies as the team developed the cloud version of EnterpriseROR and launched it in the first quarter of 2019. At the time, they ruled out several options:
Redshift: Stored procedures were not supported at that time, and it underperformed compared to Vertica.Vertica: Nested queries were not available during the evaluation.MariaDB: dailyVest was already committed to Microsoft’s Azure cloud, and the database did not support it.
Other databases under consideration included KDB+, ClickHouse, PostgreSQL, MonetDB, and Infobright. They did extensive proof of concept (PoC) analyses to compare the various options and ultimately, none of these other offerings supported the features the team needed or they were too cost-prohibitive.
Benefits of SingleStoreDB Cloud for FinTech Companies
dailyVest first encountered SingleStore years ago, but in its early days when it was just an in-memory database without columnstore. The team looked into it again in 2019 after running into the performance and pricing challenges of Azure SQL. dailyVest discovered that SingleStore now offers columnstore, along with the fast in-memory rowstore and other features and benefits that met its technical and business requirements.
SingleStore offers a unified database for fast analytics on any data, anywhere. It supports deployment on Azure cloud and is optimized for fast analytical queries across large, dynamic datasets. Parallel, high-scale streaming data ingest addressed dailyVest’s ETL performance goals. SingleStore’s ability to process billions of events per second and millions of real-time queries formed a foundation for managing the short and long-term needs of the platform.
dailyVest chose SingleStoreDB Cloud, a fully-managed, on-demand, and elastic cloud database. McNellis explained, “We have the talent to stand up, administer, monitor, and fine-tune the data infrastructure ourselves, but that's not where our value is. We could do that, but it would take time away from what we do really well, which is analyzing data and keeping it safe.”
With SingleStoreDB Cloud, dailyVest gained access to SingleStore’s best-in-class speed, scale, and agility without the headaches of installing, configuring, and maintaining software. The processes for creating and deploying clusters, monitoring cluster health, backups, administration, and optimizing and tuning queries are all streamlined with this fully managed cloud database solution.
Implementing and Optimizing SingleStoreDB Cloud
dailyVest moved the EnterpriseROR platform from Azure SQL to SingleStoreDB Cloud in 2020. The bulk ingested data goes from customer data centers to the SingleStore database and powers the platform’s real-time interactive dashboards and APIs.
Read Post
Case Studies
How Stream Hatchet Powers a Real-Time Esports Analytics Platform with SingleStoreDB Cloud
The video game industry and esports connect companies with a demographic that’s difficult to reach: the millennial male with a higher than average income. Traditionally, companies used real-world sports to reach this demographic, but brands now turn to sponsoring competitions, advertising on live streams, and other non-traditional strategies to tap into this market.
This shift occurred for a few reasons:
Some esports events, such as the League of Legends championships, actually draw larger audiences than the Super Bowl.2.7 billion people worldwide play video games.The video gaming industry is a $159 billion market.The pandemic disrupted traditional sports and entertainment media, leading to weekly live streaming hour increases of 81 percent over the past year, with tens of billions of hours watched. 355 million of these hours were sponsored live streams.
One thing that has held game publishers and brands back from getting the most out of live streaming and esports was a lack of data. They need to know the best live streaming platforms to reach their target audiences, when they should stream, what games and genres to select, how long the stream should be, and dozens of other choices that make a big difference in ROI.
Two gamers, who were software developers by trade, founded Stream Hatchet right after university to fill this gap in the market.
Building a Leading Video Game and Esports Analytics Solution
Stream Hatchet started in 2015 in Barcelona, Spain. Eduard Monsterrat, now CEO, and Albert Alemany, now CTO, created the platform as a passion project and turned it into a successful B2B data intelligence company.
The first iteration of Stream Hatchet was developed as a tool for individual streamers to monitor their streams and better interact with viewers, as Twitch’s built-in dashboards were lacking. They didn’t know if they were gaining viewers, followers, or subscribers. The main goal of the original dashboard was to promote real-time interaction.
A few years after Stream Hatchet’s founding, the company had gone through an accelerator program, acquisition, and an expansion into the United States. When the US team came on board, the platform transitioned from B2C to B2B gaming and esports live streaming analytics. A new set of tools was introduced to Stream Hatchet, and the most important of these was the esports dashboard.
Alemany explained, “There are plenty of tournaments being played every day, such as competitive League of Legends and Fortnite. We have this tool that allows our customers to check this data. Which teams are playing? What channels are broadcasting? What viewership are these tournaments getting?”
“They can monitor how they’re doing and better optimize schedules around when it’s best for them to stream those leagues. Is it on Mondays at a specific time, or on one afternoon on the weekends? They can see what other leagues are running at the same time to optimize around that data.”
The Stream Hatchet platform delivered a comprehensive range of purpose-built esports and live streaming analytics features and benefits to its users, including:
Standardized cross-platform measurement data between different live streaming platforms.Five years of historical data to analyze past trends and performance across all genres, games, events, and channels.Brand lift analysis to rank category performance.Influencer targeting, ROI analysis, and strategy.Sponsored campaign analysis.Independent third-party source for live streaming data.Competitive analysis of other influencers.Enhanced audience data.White-labeled and custom business intelligence dashboards.Aggregated dynamic data.Data from millions of global streaming channels and 20 of the top streaming platforms.API access for customers to integrate trusted live streaming data into their own dashboard solutions.
The Difficulties of Processing Highly Complex Live Streaming Queries
The technology supporting Stream Hatchet needed to do a lot of heavy lifting to deliver the expected functionality and analytics performance. The team started with MySQL on AWS RDS, but over the past two years, the real-time functionality wasn’t there. While the data ingestion scale wasn’t large, the queries were highly complex and ran into major speed issues.
“It was getting challenging to satisfy the use cases that our clients wanted to achieve, especially our bigger clients like Riot Games and Activision Blizzard. It wasn’t working for their data scientist teams to sit for one or more minutes each time they applied different filters,” said Alemany.
Hundreds of customers ended up complaining about slow queries in their custom esports dashboards. The poor dashboard performance threatened Stream Hatchet’s potential growth and impacted its revenue.
The challenges of working with this esports data came in many forms:
A single query may aggregate many gigabytes of data.Some database solutions are poor fits for working with time series data, which is predominant in esports.Both real-time and historical data were needed to provide the full live streaming context.Customers wanted to compare macro-level trends by game, publisher, league, and region across multiple streaming platforms.Minute-by-minute granularity was necessary to provide the full context of the live streaming data.
The front-end esports dashboards were created with Javascript Meteor, and the API was built with Express on AWS. The Stream Hatchet team experimented with many other solutions to power the back end, including:
AWS AthenaAWS TimestreamElasticsearchMongoDBMultiple SQL versionsMySQL on Google Cloud Platform
“We were surprised to see that not even some in-memory versions of MySQL were able to do the aggregations we were doing,” said Alemany.
Stream Hatchet also used AWS EC2 for enriching the esports data, and MySQL on AWS still powered other services that had lower performance requirements. The platform had an increasingly complex infrastructure that still wasn’t meeting customer demands, even as it took more resources to operate.
“The tool was not functional anymore, so we started looking for solutions,” said Alemany.
The new database solution for Stream Hatchet needed to fulfill the following technical requirements:
Wire compliant with SQL, so the team didn’t need to change the entire back end or redesign the APIsCapable of working with 1,000s of Time Series data points in complex aggregated queriesFast analytics speedSupport for 10s of concurrent users globallyMultiple table lookups and joinsAccept writes from external applicationsSupport for data sourced from third-party fact tables and manual metadata entry
Increasing Query Speed and Reducing Infrastructure Complexity with SingleStoreDB Cloud
Stream Hatchet found SingleStore by searching for in-memory SQL engines and databases on Google. SingleStoreDB Cloud is the fully-managed, on-demand cloud database service for fast analytics on any public cloud environment. This Cloud Database as a Service solution delivered instant, effortless access to SingleStore’s best-in-class speed, scale, and agility.
Stream Hatchet selected SingleStoreDB Cloud over the self-managed database as it was the easiest solution and had a competitive database cost. Alemany explained, “It didn’t make sense for us to maintain infrastructure when the cost SingleStore was offering was very competitive. It was a clear decision for us.”
SingleStore features that make it an excellent choice for video game analytics include:
Unified database designed for fast analytics on any data, anywhere.Fast analytical queries across large, dynamic datasets with high concurrency.Accelerated speed of dashboards to create Fastboards.Super low-latency, blazing-fast queries.No need for multiple purpose-built data engines.
Implementing Esports Analytics on SingleStoreDB Cloud
SingleStore’s SQL Wire Compliance made it easy for Stream Hatchet to set up a proof-of-concept. The team used a subset of all esports data and took approximately two months to move into production following an intensive testing phase. However, they saw performance improvements starting at the two-week mark. The implementation was smooth, and SingleStore’s team assisted with configuring queries, setting up data ingestion, and helping Stream Hatchet explore potential use cases.
Stream Hatchet configured SingleStoreDB Cloud to handle real-time and historical esports data for powering internal and customer dashboards, query data, and report generation.
Ingested data was split between manual, automated, and third-party sources:
Manual metadata: The Stream Hatchet team captures and inputs esports event data, such as the channels broadcasting, match schedules, and the events taking place. The metadata is enriched with audience data from live streaming platforms.Automated: This system looks at live stream titles and can detect possible esports events based on the information in the title. The platform automatically records this data for later review.Third-party data: Stream Hatchet works with a data provider that covers esports event and stream data, such as the exact time stamps teams are playing each other.
The schema design, cluster size, and infrastructure that Stream Hatchet used included:
4 primary fact tablesStreams of 2 million records20 static reference tables with 5,000 records eachSSMS S0 with 4 vCPU, 32 GB RAM, 500 GB storage
What Stream Hatchet Achieved With SingleStoreDB Cloud's Help
“Before SingleStore, we had to work with different services on the backend for our weekly esports reporting. That created a lot of challenges to pull this data from different places and integrate it. Now that we’re only using SingleStore, it’s much smoother than before.”
The Stream Hatchet platform saw the following improvements:
2x improvement in esports report generation time.10x increase in esports dashboard speed.Esports query response times are now measured in a handful of seconds at most, rather than minutes.Customers can now drill down minute-by-minute into the data to see what specific channels were doing at a particular minute, and look at the actual trends to understand spikes in viewership.
“We’re able to exploit much more of our esports data and extract more insights. Before SingleStore, we had to look at it from the surface because our back end didn’t allow us to go deeper. Now we can crunch the data in different ways in real-time that allows our users to extract insights and see things that they couldn’t before.”
What's Next for Stream Hatchet?
“Our goal is to expand the team, as we’re still very small, and tackle all the organizations we believe will benefit from esports and live streaming data.”
Now that the esports use case is in production, Stream Hatchet wants to expand SingleStoreDB Cloud usage to its other lines of business. The team can power multiple business units for under $400 per month. The team plans on migrating some of their other data offerings from MySQL on AWS RDS. They expect to see a 2-3x speed improvement with this implementation.
Stream Hatchet also wants to look at data differently in the future, implementing:
ForecastingSentiment analysisMachine learning
“For instance, one of the projects that we have in the pipeline is trying to detect the exposure of brands in streams. We’re trying to see the logos or the overlay banners that streamers set up in their broadcasts, with the goal of being able to say “Logitech was present in that many streams,” or “Was watched that many hours,” said Alemany.
Experience the ultra-high performance and elastic scalability of SingleStore on-demand. Contact us to learn more about SingleStoreDB Cloud or get started now with $500 in free credits.
Read Post
Case Studies
Blink! It’s an Eternity for Data-Intensive Financial Services Applications
You did it again—you just blinked. According to the Harvard Database of Useful Biological Numbers, a blink of a human eye lasts roughly 100 milliseconds. Not a long period of time, but a relative eternity compared to SingleStore use cases such as: (i) A large ride-sharing company executing 40,000 concurrent transactions with an SLA of two milliseconds, (ii) Real-time fraud detection for a top-tier bank, running several analytic queries within their 50-millisecond service level agreement (SLA) at high availability, and (iii) A major internet company doing 12 million upserts per second (1,000 milliseconds).
Those are just a few examples shared by SingleStore’s chief product officer, Jordan Tigani, at a recent DATAcated virtual event. His presentation, “Data Intensive Applications in Financial Services,” dove into the history of the need for speed in the financial industry, and how SingleStore is uniquely positioned to execute data-intensive tasks at almost impossible-to-imagine speeds. Here’s a recap of his talk.
From “Flash Boys” to today’s data-intensive norm.
Jordan kicked off his talk with a reference to Michael Lewis’s 2014 book, “Flash Boys,” a great take on the need for speed in the financial industry. The first part of the book chronicles how a company called Spread Networks spent \$300M to build a fiber-optic route between a datacenter in New Jersey to the Chicago Mercantile Exchange. Their goal was to reduce the delay by four milliseconds—a money-minting competitive advantage.
The ability to make decisions about data faster than the competition is at the heart of the trading game. Data constraints don’t just show up in algorithmic trading, however. There are a wide range of problems in financial services that boil down to fast, reliable access to data for making decisions. These include:
Building rich customer experiences that provide real-time recommendationsFraud detection during a card transaction, “on-the-swipe”Portfolio management customers want to see not just what is going on in their account, but what is going on in the market right now.
All of these applications share a common characteristic: they are data-intensive applications, defined by Martin Kleppmann in his great book of the same name as “applications where the primary bottleneck is data,” because the data size is either too large, too complex, or moving too quickly.
Jordan explained how this is exactly the type of problem seen over and over in financial services apps, where decisions must be made quickly, or information delivered to users quickly. More importantly, he said, this phenomenon is becoming widespread. “Data-intensive applications show up everywhere,” he said. “If you consider most of the rich, modern applications of the recent era, most of them derive much of their value from their ability to incorporate large, complex, fast-moving data to provide their core user experiences.”
The last part is key; data intensity isn’t just an add on, it is essential to organizations being able to provide the level of service that is at the core to their main product.
What do data-intensive applications require?
Jordan broke down the infrastructure requirements for data-intensive apps as follows:
They need to be able to get data in quickly and reliably. If you need to make a decision, you had better be operating on the latest data, or you’re going to be in trouble.Even if you’re not doing high-frequency trading, latency is important. If you have a user waiting for information, they can detect latency of tens or hundreds of milliseconds. For years, industry studies have shown that even small increases in response times have a big impact on user attention and ultimately revenue.Requests don’t come in uniformly throughout the day, so you need to be able to handle a lot of requests at once. When there is a big event in the market, you don’t want your data systems to get bogged down.
The systems traditionally used to process data are not well suited for data-intensive applications; transactional databases have trouble with analytics at scale, and data warehouses have trouble achieving the low latency needed for online serving. Both of these structures have architectural limits.
SingleStore Overview
In his talk Jordan explained the backstory of SingleStore, which was designed as an in-memory database that could do extremely low latency, sub-millisecond transactions. “Over time, we added an analytics-optimized storage layer, which added the ability to scan trillions of rows per second,” he said. “We combined the two into a single store, which is how we came up with the name ‘SingleStore’ for the company.”
SingleStore is one of the only databases that, on its own, can meet the needs of the largest data intensive applications. It can scale to handle myriad concurrent users because it is incredibly efficient with resource usage; one SingleStore customer found that it uses less than 1% of the CPU cores of a major cloud data warehouse. Other examples include the three at the beginning of this blog.
Oh, and “Flash Boys”? The main story in that book is about a company called IEX that aimed to build a new type of exchange. Their data services arm, IEX Cloud, is using SingleStore to provide richer data analysis and reporting capabilities to consumers. You can learn more by watching this webinar and reading the eBook, “Ludicrously Fast Analytics: A 5-Step Guide for Developers of Modern Applications.”
Watch the recorded version of the webinar here.
To keep up with how SingleStore is unlocking value in a wide range of industries by enabling data-intensive applications, follow us on Twitter @singlestoreDB.
Read Post
Case Studies
Nucleus Security and SingleStore Partner to Manage Vulnerabilities at Scale
Cybercrime damages in 2021 are expected to reach \$6 trillion, with organizations of all sizes and industries exploring ways to protect themselves better. Potential security vulnerabilities come in many forms, from cross site scripting to improper privilege management. Over 17,000 new security vulnerabilities were discovered in 2020 alone. IT security teams are responsible for securing infrastructure that’s continually increasing in complexity, so it’s challenging to respond quickly to newly discovered exploits or confirm that a known vulnerability is addressed. It doesn’t take long for an organization to end up with a substantial vulnerability backlog. On average, over six months, organizations end up with a backlog exceeding 57,000 and fail to mitigate 28 percent of these vulnerabilities. A pre-emptive, holistic, and scalable vulnerability management approach is needed to keep systems safe.
Addressing a Critical Vulnerability Management Process Gap
The founders of Nucleus Security, Stephen Carter, CEO, and Scott Kuffer, COO, had decades of experience providing vulnerability management services to United States government agencies. They used a variety of vulnerability scanning tools for this purpose. Whenever they found a vulnerability, they had to manually enter a remediation request into a ticketing system, such as Jira or ServiceNow. The gap between the identification of vulnerabilities and the creation of tickets created an efficiency bottleneck. All of the vulnerability reports needed to be normalized across a range of formats used by different tools and prioritized by the user. The response then needs to be automated for greater speed, manageability, and scalability.
Carter and Kuffer created the Nucleus vulnerability management platform to make it faster and easier to provide services to their clients. Their original intent was to have a custom tool for their own operation, but they quickly discovered a major need for this type of platform.
The users most interested in Nucleus Security were:
Large enterprises and government agencies needing to manage security vulnerabilities at scale.SingleStoreDB Cloud Security Providers (MSSPs) looking for a platform that supports their vulnerability management processes with multiple clients, offers client access, and provides a way to expand it with their proprietary functionality.
Handling Unified Vulnerability Management at Scale
Since Nucleus was created by and for vulnerability management experts, the platform solved many problems that stood in the way of discovering and remediating vulnerabilities before they became exploits.
Smart automation and workflow optimization eliminated many tedious and time-consuming parts of vulnerability management through:
Centralizing vulnerability aggregation, normalization, and prioritization in a single platform.Delivering the right information to the right people in the right format by connecting vulnerability, ticketing, alerting, and issue tracking tools together.Increasing vulnerability management speed and capacity from end-to-end.Reducing costs for vulnerability management.Improving the visibility of vulnerability management data and remediation efforts.
How Nucleus Works
The Nucleus application was delivered as a software-as-a-service (SaaS) offering and was designed with a traditional three-tier architecture. “There’s a web application, which is customer-facing, and a job queue that processes data. Then there’s a database on the backend, serving both,” says Kuffer.
Users log in to the customer-facing Nucleus web application.They set up integrations with the tools they’re using from a list of more than 70 supported applications.The users create rules to ingest data and trigger alerts, reports, and tickets.They can access a centralized vulnerability management dashboard, perform searches, and analyze their scan results.A separate backend job queue ingests and processes the data on the user-specific schedule from the selected data sources’ APIs.A database powers the frontend and backend operations.
The product licensing is based on the number of assets that an organization is working with. In Nucleus, an asset is defined as an item that the user scans. Each asset can have multiple vulnerabilities.
The Challenges of Pivoting to a Commercially Viable Vulnerability Management Platform
The decision to launch a separate company, Nucleus Security, had not originally been the founders’ plan. Kuffer explains, “We actually just went to a conference called Hacker Halted in Atlanta, Georgia, and we had Nucleus as basically a line item in a data sheet for our old company. We just got basically hammered with leads at that point.”
“We were not prepared at all for any leads whatsoever, and so none of us had any sales experience, we didn’t have a company, we had nothing. We didn’t really have much of anything, other than a product that didn’t scale for all of these needs.”
Nucleus had many leads coming in that were bigger names and companies, which helped to expedite the decision to fork it off into a separate business at the end of 2018. However, the founders needed a way to scale the platform for these Fortune 500 companies, and they needed it fast.
Finding the Right Database Solution
When Nucleus Security began architecting the Nucleus platform, they explored several database options, including document-oriented databases, graph databases, and traditional relational databases.
Carter says, “All of the options had their strengths and weaknesses, and we felt that several database technologies could work well for the initial proof of concept. However, it quickly became apparent that a high-performance relational database system was a hard requirement to support our data model and the features we were planning to bring to market.”
They started out developing the solution they would have needed when they were working directly with government agencies. These clients are highly focused on compliance and run scans weekly or monthly, based on the regulatory requirements. The database solutions that Nucleus tried often took a long time to process vulnerabilities, but it wasn’t a big issue at a weekly or monthly cadence.
Struggles with MariaDB
The Nucleus prototype used MariaDB, which is on many federal government-approved software lists. “MariaDB comes bundled with the government-approved operating system we were using, which is Red Hat Linux,” explains Carter. “For the prototype, this worked just fine. But when we started to onboard larger customers, we hit some ceilings performance-wise, and some pretty major performance issues.”
“As a workaround, we were trying to precalculate a bunch of stuff, to show it in the UI. But if you’re scanning every hour, and it takes an hour and a half to do the calculations, then you get a huge backlog,” says Carter.
The team spent a lot of time tuning queries to squeeze out every bit of performance that they could to keep up with the capacity needed for their beta customers. However, even with the clustering and load-balanced configurations available, it was clear that a different database solution would be needed for Nucleus to support very large enterprises, which have hundreds of thousands of devices and tens of millions of vulnerabilities.
Commercial clients scan for vulnerabilities multiple times daily or weekly. They wanted to see the results of a scan in minutes, not hours or longer. Over time, the backlog built up and hit a ceiling where the Nucleus application couldn’t process jobs fast enough. Batch processing worked with federal agencies, but enterprises demand real-time vulnerability management.
“It was the database that was the bottleneck all along,” says Kuffer. “We looked at some NoSQL solutions. We looked at Percona for clustering, but we would have had to rewrite a lot of our code – and all of our queries.” Nucleus Security also investigated other SQL solutions based on PostgreSQL core, such as Greenplum.
The relational database for a commercially viable version of Nucleus needed:
Real-time processingSupport for 10s of millions of vulnerabilities and 100s of thousands of devicesHigh scalability
The Benefits of SingleStoreDB for Vulnerability Management Platforms
Nucleus Security started looking into alternatives to MariaDB that were better suited for its vulnerability management platform. They found Percona first, but the initial tests indicated that it wouldn’t help with their use case. The scaling focused more on being a high-availability cluster. While they could add to the cluster and use load balancing schemes with different nodes, it was an extremely manual process. It also required a minimum of three servers.
SingleStore came up during Nucleus Security’s search for a database and impressed them from the start. “SingleStore was a great option, because SingleStore is not only relational; it also supports the MySQL wire protocol, which of course is inherent in MariaDB,” explains Carter. “It was almost a drop-in replacement for MariaDB, and it’s way less complex, and also much easier to maintain than the Percona cluster solution that we were looking at.”
SingleStore is The Single Database for All Data-Intensive Applications powering modern applications and analytical systems with a cloud-native, scalable architecture for maximum ingest and query performance at the highest concurrency. It delivered many powerful capabilities, such as:
Optimization to run anywhere from bare metal to the hybrid cloud with commodity hardware, including multi-cloudSimple to deployBetter performance than legacy RDBMS and NoSQL databases for high-velocity big data workloadsMemory-optimizedLow total cost of ownership by integrating with existing systemsIngest millions of events per second with ACID transactions while simultaneously analyzing billions of rows of data in relational SQL, JSON, geospatial, and full-text search formatsData shardingLatency-free analyticsSeamless horizontal and vertical scalingWorkload managementCompressed on-disk tables
Switching to SingleStoreDB from MariaDB in Nucleus
Nucleus Security started with the free tier of SingleStoreDB Self-Managed for the proof of concept early in 2019. The founders wanted to determine how hard it would be to migrate the application to the new database. They imported 80-100 gigs of data from MariaDB that included several tables with several 100 million rows to test their slowest queries. It only took an afternoon to migrate the development database to SingleStoreDB Self-Managed and get Nucleus working without any architectural changes.
Carter says, “It was dead simple to get set up. Whereas, I’ve got experience getting Oracle database clusters set up, and those things can be nightmares. And our experience with SingleStore was very good. We do not spend a lot of time maintaining it, troubleshooting it.”
Following the successful proof of concept, the Nucleus application moved to a three-node cluster along with several single server instances for customers in Australia, the United States, and the European Union. It took 3-4 weeks to get Nucleus tested, deployed, and entirely in production on SingleStoreDB Self-Managed. Nucleus got its first keystone client, the Australian Post Office, in March 2019, shortly after the migration. They had 2,000 code repositories and required approximately 50,000 scans per day.
Kuffer says, “They paid us a lot of money upfront on a multi-year deal, and plus they had the brand name and it allowed us to transition that into a lot of later success. We definitely wouldn’t have been able to support the scale that they had without SingleStore.”
No Architectural Changes
The Nucleus app brings in data directly from the APIs of vulnerability scanning tools used by customers, and interacts directly with their job scheduling systems, such as Jira or ServiceNow, directly. There’s no need, at this time, to use Kafka or other streaming technologies.
Nucleus did not need to make any architectural changes to their application to move to SingleStore; it has served as a drop-in replacement for MariaDB. Since MySQL wire compatibility is shared by both, making the move was easy. By replacing MariaDB with SingleStore, Nucleus customers can now support MSSPs with full scalability.
Read Post
Case Studies
SingleStore for Fastboards
Introduction
Over the last many years, analytical dashboards have proliferated enterprises from the boardroom to the manufacturing line. As businesses have become increasingly more reliant on analytics, end users have started to demand snappier performance from their dashboards. Gone are the days that visualizations are used simply for historical data. Data professionals are now using them for predictive and prescriptive analytics on petabytes of information, expecting the most real-time insights to tell their data stories.
Many organizations across consumer products, healthcare, fintech and other verticals have been met with the challenge of slow moving BI applications in recent years, due to the growth of data and many new types of users. Backend platforms are often retooled for each use case and still are unable to handle the growing concurrency demands of the business. This leads to applications that take too long to render and users that are faced with the dreaded “loading” pinwheel.
The Pain
The challenge of slow-running dashboards is not limited to legacy systems. Users experience slow dashboards when their applications are backed by legacy architectures as well as by more modern database systems. The pain points that enterprises face with legacy architectures start with the original intent of those systems: to support historical reporting and batch-oriented dashboards. Many databases from vendors like Oracle were built around a different era of business demands — when operational decisions were not made through data visualization. Over time, other (now legacy) systems like SQL Server emerged to help with transactional processing of data. This spread of new database systems purpose-built by workload introduced put even more stress on ETL technologies. The Hadoop era aimed to solve this, but ultimately consumers were left at the mercy of too many different systems for data to reside in and excessive data duplication across those silos. This made data storytelling very difficult.
Read Post
Case Studies
Infosys and SingleStore: Working Together
Infosys and SingleStore are now working together, identifying opportunities to implement SingleStore – a fast, scalable, relational database, renowned for fast analytics, streaming support, and solutions in areas such as financial services, digital media, telecom, and IoT – with the speed and effectiveness for which Infosys is rightly renowned, worldwide.
Infosys is a global leader in next-generation digital services and consulting, with clients in 46 countries. With over three decades of experience in managing the systems and workings of global enterprises, Infosys expertly steers clients through their digital journey. Key to their work is enabling the enterprise with an AI-powered core which helps prioritize the execution of change at each company. They seek to enable an agile digital strategy across the business. They then deliver a learning agenda, driving continuous improvement through building and transferring digital skills, expertise, and ideas from the Infosys innovation ecosystem.
SingleStore is an up-and-coming database provider gaining increasing recognition for fast, scalable relational database offerings. The SingleStore database delivers lightning-fast analytics and compatibility with ANSI SQL, standing out from the NoSQL crowd. SingleStore is also gaining attention with customers that include many of the Fortune 500, including half of the Top 10 North American banks.
SingleStore and Infosys have now partnered to help clients across verticals. The key to the partnership is the ability of SingleStore to offer uniquely powerful solutions, with outstanding price-performance, to a variety of database-related issues, especially in analytics.
Infosys contributes a gimlet eye for opportunities which can best be realized by the use of SingleStore. Infosys can also lead the systems integration work needed to quickly stand up SingleStore, and put it to work, within a complex operating environment.
The solutions jointly offered by the SingleStore and Infosys blur the boundaries of two categories which are commonly used to describe technology offerings: painkillers and vitamins. A painkiller solves problems – reports that take days to run, and dashboards that take hours to update. Slow-running ecommerce sites that inadvertently send customers to the competition. Applications that refuse to scale, requiring doublings and redoublings of back-end investment for meager performance gains.
Vitamins help companies take advantage of opportunities – the new report, new dashboard, new ecommerce site, or new applications that helps a large company step ahead of smaller competitors, or helps a government agency offer services faster, more effectively, and at less cost. And, when technology is used in just the right way, it can do both at once: fix things that are broken, and open up competitive advantage.
SingleStore provides raw power for such solutions; Infosys offers finesse in identifying opportunities and implementing solutions, helping to fix problems the right way, the first time, maximizing opportunities, and minimizing time to market. With the description given here, you’ll be able to identify opportunities in which Infosys and SingleStore may be able to help in your organization.
What Infosys Offers
Infosys is a global organization, nearly 40 years old, with nearly a quarter of a million employees and with a market capitalization, at this writing, of more than \$40 billion. Infosys offers digital services and consulting, with Infosys Consulting operating as a business within the overall Infosys organization. Slightly more than half their business is in the US, with the rest spread around the globe.
Infosys partners closely with its clients, and offers a broad range of services that adapt to their needs through its vast suites of services, with platforms spread across different industry verticals.
Infosys acts as a trusted advisor to its clients. Part of their value proposition is their ability to identify valuable emerging technologies, use these technologies in a few situations for which they may be particularly well suited, then share their findings across the Infosys organization worldwide. This creates a portfolio of proven technologies that all Infosys clients can adopt with confidence.
What SingleStore Offers
SingleStore offers a fast, scalable, relational database. SingleStore combines transaction and analytics processing in a single database, like other databases in the emerging NewSQL category, also referred to as hybrid transactional and analytical processing, or HTAP – a term coined by Gartner nearly a decade ago, shortly after SingleStore was founded.
SingleStore is a private, venture capital-funded company based in San Francisco. SingleStore has grown to more than 200 employees, including a large proportion of engineers who continue to develop the platform. At its current stage of development, SingleStore is especially well suited for powering fast analytics that blend historical and real-time data, with fast ingest, high concurrency, and with breakthrough responsiveness to SQL queries, whether simple or complex. This field is known as operational analytics, and is used heavily in financial services, digital media, telecom, IoT, and other areas.
SingleStore usage is sometimes held as something of a secret among its customers, who gain tremendous competitive advantage in applications including credit card fraud detection, predictive maintenance, ad marketplace sales, and other applications. With an increasing range of business partnerships, however, and with the Infosys partnership as the most prominent, the secret is increasingly out.
How Infosys and SingleStore Work Together
Infosys and SingleStore work together with both new and existing customers. Infosys looks for opportunities where the SingleStore database adds unique value, due to its speed, scalability, SQL compatibility, and other features. SingleStore looks for opportunities where advising is needed at a broader level than whether the SingleStore database is the right fit for a specific use case – where an organization is looking at significant change within IT, such as moving a large proportion of their application portfolio from on-premises servers to the cloud.
Infosys sees SingleStore as a key enabler, and well-suited for use cases such as:
Real-time business intelligenceOffloading OLTP databasesPerforming operational analyticsBuilding real-time apps for highly-concurrent read access (such as for dashboards and internal enterprise apps)Performing real-time analytics, such as anomaly detection to spot fraud and customer segmentation
Here are a few examples of recent implementations of SingleStore in Infosys-led projects:
A large consumer electronics company is moving internal data warehousing workloads from Vertica to SingleStore. Infosys is working closely with SingleStore to set up and manage SingleStore infrastructure to support the former Vertica workload, migrate applications out of Vertica, and point dashboards to run on SingleStore.A global music company is using SingleStore to power real-time business intelligence (BI) solutions. Tableau is being used as a visualization layer for data extract from a range of sources and displayed in dashboard. Formerly, the dashboard was having inconsistency and latency issues. Infosys introduced SingleStore as an intermediate, low-latency layer, moving data sets from Hadoop to SingleStore.A leading bank is offloading a transaction processing workload from the leading “old SQL” database to SingleStore. Offloading the transactional database to SingleStore resulted in cost savings and better customer experience.
Learn More
This blog post may have left you curious about using the SingleStore database; about working with Infosys; or about working with Infosys and SingleStore as partners. You can contact Infosys askus@infosys.com ; contact SingleStore; or try SingleStore for free.
Read Post
Case Studies
SingleStore Powers the Rapid Rise of Thentia Regulatory Technology
The following announcement is being released by SingleStore today to announce the adoption of SingleStore as the core operational database for cloud-based regulatory licensing, assurance, and enforcement technology from Thentia, a global leader in regulatory software.
SingleStore Powers the Rapid Rise of Thentia Regulatory Technology
As More Regulators Seek Digital Tools Amid COVID-19, There’s a Greater Need for Speed, Scale
SAN FRANCISCO – AUGUST 25, 2020 – SingleStore, The Single Database for All Data-Intensive Applications for operational analytics and cloud-native applications, announced today that Thentia, a global leader in regulatory software, is launching SingleStore as the core operational database for its cloud-based regulatory licensing, assurance, and enforcement technology. SingleStore’s speed and scale will enable Thentia to achieve fast response times and automate a historically paper-based regulatory compliance process. At a time when scamming is on the rise, the solution helps to bring greater confidence and responsiveness to health and safety systems.
Thentia offers cloud-based applications that independent and government regulators in sectors such as healthcare use to manage tasks digitally and securely. Demand for Thentia’s technology has exploded amid the pandemic, as a growing number of regulators look for modern, digital solutions that enable them to work efficiently, remotely, and safely to protect public health.
“We’re seeing incredible growth at Thentia, and our relationship with SingleStore enables us to support and build upon that growth,” said Julian Cardarelli, CEO of Thentia. “With SingleStore, we can process billions of events and do real-time analytics, providing our customers a level of analytical power they have never before experienced.”
Harry Cayton CBE, a world-renowned expert on regulatory practices and a recent addition to the Thentia board of directors, explained, “This is important now, because it enables regulators to do more with less at a moment in which COVID-19 makes work especially challenging and many governments are streamlining operations, looking to work smarter.”
“Data is more important than ever, especially for individuals and organizations that are charged with protecting public safety,” said SingleStore co-CEO Raj Verma. “We are grateful to have the opportunity to work with Thentia to enable regulators, and all the people they serve, to benefit from the power of real-time data delivered over a scalable and secure architecture.”
SingleStore makes it possible for Thentia to build a wealth of reports for its customers, who can pull these reports to get detailed, in-the-moment information about such metrics as complaint increases or decreases, complaint handling times, and number of licensees. With SingleStore, Thentia also will be able to leverage machine learning and predictive algorithms to generate insights. This is far more effective than the alternative, attempting to gain insights from Excel documents, which are less engaging for users, and which become outdated almost immediately.
About Thentia\ Based in Toronto, Canada, Thentia is an industry leader in using proprietary technology to help regulatory bodies efficiently fulfill their regulatory obligations. Thentia services a wide variety of clients throughout Canada and the United States using cutting-edge software and industry-leading expertise in regulatory standards.
About SingleStore\ SingleStore is The Single Database for All Data-Intensive Applications, powering modern applications and analytical systems with a cloud-native, massively scalable architecture. SingleStore delivers maximum ingest, accelerated transaction processing, and blisteringly fast query performance, including AI integration and machine learning models, all at the highest concurrency. Global enterprises use the SingleStore distributed database to easily ingest, process, analyze, and act on data, to thrive in today’s insight-driven economy. SingleStore is optimized to run on any public cloud or on-premises with commodity hardware. Visit www.singlestore.com or follow us @SingleStoreDB.
Contact
Gina von Esmarch\ 415-203-4660\ gvon@singlestore.com
Read Post
Case Studies
How We Reverse-Engineered The Chicago Mercantile Exchange
The exchange is at the heart of our economic system. From the Rialto in Venice (fourteenth century), the Grand Bazaar in Turkey (seventeenth century), the Amsterdam Bourse in Holland (seventeenth century) and the New York Stock Exchange (twentieth century), markets have purposely been built to be a place for buyers and sellers of goods and services to meet and transact. In this blog post, we explain how we built an exchange, with SingleStore at the core.
After the invention of computers, and the widespread adoption of the Internet, the marketplace went online. The buying and selling that used to take the spice trader in the Dutch East Indies – now modern-day Indonesia – about a month, with price matching provided by the Amsterdam Bourse, now only takes a fraction of a second using automated computerized trading systems; a better price discovery method for the buyers and sellers had evolved.
Building such a centralized, computerized trading platform is not an easy task. We thought that since the matching engine is at the core of many economic activities, there must be a step-by-step tutorial to build a trading system from scratch available somewhere on the Internet. Our assumption turned out to be completely wrong.
Building an exchange requires knowledge spanning from multiple distinct fields, such as game theory (mathematics), algorithmic and data structure (computer science), and market microstructure (finance). Inspired by Cinnobar (now part of Nasdaq), and LSE, which operates the Oslo Bourse exchange, our team decided to rebuild the foundations from the ground up: we set our sights on building a financial exchange from scratch.
The tutorials that you do find on the web are geared towards high-frequency market making, or proprietary buyers and sellers on a financial exchange. In financial lingo, the tutorials are for the buy side/sell side, and not on the exchange side – how you listen to price movements, make a decision, and send an automated order (either to buy or sell a financial instrument) to the exchange. We never found any information regarding how you would receive orders from clients, match them, and conduct proper clearing mechanisms after a trade has happened – that is, the clearing process. Also, after a trade happens, how do you make sure each market participant listens correctly to the price changes that have happened during the trade – that is, the price data feed.
We call this the centralized exchange problem and, truth be told, this has been solved efficiently by large financial groups such as the National Association of Securities Dealers Automated Quotations Exchange (NASDAQ), the New York Stock Exchange (NYSE), and the Chicago Mercantile Exchange (CME) group. One particular implementation of a financial exchange intrigued us: the CME. The reason the CME is different is because the platform not only has standardized spot trading, but it also has standardized options, futures, and spread markets with implied orders functionality.
Many technology groups have solved the spot market pretty easily. You might have heard about several cryptocurrency exchanges appearing around the globe, such as BitMEX, Binance, Bitfinex and Coinbase. You can easily trade, using leverage in the range of 1-100x, several financial products such as perpetual inverse swaps (which are basically spot on leverage) and/or directly buying/selling cryptocurrencies, such as Bitcoin and Ethereum.
Building A Spot Exchange
Before explaining our solution, let’s cover how a typical spot exchange operates. You have multiple clients that send a sequence of orders, each with a quantity and price. The order then goes to a journey as follows:
Receive order in gateway, timestamp the orderSend the order to the risk management system; calculate if client has enough balanceIf they do, send the order to the matching engineThe matching engine then spits out events such as trade event, order event, depth eventThe events then gets passed to several other parts of the exchangeThe market data feed engine takes the trades and depth and blasts it to the clientThe spot clearing system takes both sides of trades and debits and credits the account of each trader.
(This sequence will also determine the round trip time (RTT) of an exchange; more on that later.)
Now here, bear in mind that the clearing system at the end takes the counter-party position of the client’s trade and delivers the financial instrument. Say a prop trader buys 3000 shares of Goldman Sachs ($GS) @ $207.36 (in US dollars), and another hedge fund in New York sells 3000 shares of $GS @ $207.36. The clearing system will then debit $622,080 from the traders account and credits the trader with 3000 $GS shares – and also vice versa, credit $622,080 from the hedge fund and debits 3000 $GS shares. The spot exchange clearing engine solves the centralized clearing problem; e.g the clearing engine becomes the seller for every buyer and the buyer for every seller. Thus making sure market integrity exists and ensuring trades happen.
If you add leverage to the equation, this becomes a little bit more complicated. Here, for most equity centralized exchanges, they do not lend money to individual/institutional traders. To ensure its clearing objective, the clearing house would need to ensure the brokerage house that lends the trader could always pay the order value sent by the buyer. This is also true for a leveraged seller (short seller).
Now imagine if you don’t have intermediaries like brokerage houses. For a leveraged trade to happen, now the clearing house is exposed to credit default risk of the leveraged buyer and or short seller. During each tick, the clearing house needs to listen for price changes and compute the leverage buyer’s/seller’s position and maintain that. In the event of a drastic price change, the centralized clearing house should take over the position and liquidate it on the open market. The level in which a leveraged buyer/seller gets liquidated is called the liquidation price.
Building a Derivatives Exchange
Now, imagine this matching system but for derivatives. A derivative is a financial contract that is tied to an underlying price. For example an option on Goldman Sachs ($GS) equity is a right to purchase (or sell) Goldman Sachs ($GS) at a specific price predetermined now (called the strike price) at a later date (for example in a month).
The main difference between a spot financial exchange and a derivative financial exchange is its ability to conduct mark-to-market computation of all positions of all traders in the exchange. For a computer scientist, this is a simple matrix computation problem, but imagine, for every single tick of market data.
Revisiting the spot market order matching process, process 1-7 looks similar but with few extra steps.
Receive order in gateway, timestamp the orderSend the order to the risk management system, calculate if client has enough balanceIf they do, send the order to the matching engineThe matching engine, then spits events such as trade event, order event, depth eventThe events then gets passed to several other parts of the exchangeThe market data feed engine takes the trades, and depth and blasts it to the clientThe derivative clearing system does a couple of things
Take the latest trade, compute volatility projections, price bands (moving average) and several other metrics and recompute every single value of the portfolio of each trader on the exchangeThe traders that have positions that are too risky or are not contained within the respected margin balance, will get liquidated.On liquidation, the position will be taken care of the independent clearing house, and be sold immediately.On delivery date, the derivative clearing house will make sure the seller of the contract delivers the financial contract which depends on a cash/physical settlement.
Now algorithmically this becomes one step harder. To have step number 7 be efficient, one could utilize parallel computation methods such as GPGPU computing.
Building Derivatives Pipeline for Implied Markets
If you have traded on the Chicago Mercantile Exchange’s Globex system, you might as well admire the beauty of how the whole architecture works. Especially around a feature called “implied markets” (for more on implied markets see https://www.cmegroup.com/education/videos/implied-markets-video.html).
According to the CME’s confluence website (https://www.cmegroup.com/confluence/display/EPICSANDBOX/Implied+Orders) there are several rules of a quantity of an implied order sent to the market. These are:
Implication requires at minimum two orders in related markets in the proper combination.Implied bids do not trade against implied offers.Implied bids can exist at the same or inverted price levels with implied offers. When this occurs, CME Globex ceases dissemination of the implied bid; however, the bid is still calculated and can be traded against.Implied OUT quantity will not disseminate when the leg value of the included spread is in a ratio greater than one. The price and quantity are still calculated and can be traded against.Implied quantity in futures markets does not have time priority.Implied quantity is not available:
When the market is not in a matching state (e.g. Pre-Open)When implied calculation has been suspended
On an implied market there are two different calendar futures, say since right now we are in July there would be the N20 (July 2020) and Q20 (August 2020) futures delivery dates. You will then have another implied market which is the delta differential between the two order books N20 and Q20 which is called the N-Q20 calendar spreads.
To solve the implied market problem, one would need three instances of the matching engine that are asynchronous and event driven that listens to not only incoming orders from the risk checking system but also makes sure that the calendar spread order book is updated too.
Read Post
Case Studies
SingleStore Ramps Up the Cadence of Financial Dashboards for an Industrial Machinery Leader
According to analysts, “the growth rates of insights-driven businesses makes them an economic tidal wave” – one that will earn \$1.8 trillion dollars by 2021. How does a long-established, Global 500 company, listed in the Top 100 global brands, become an insights-driven business? A leading producer of industrial machinery is taking bold steps in this direction, powered by SingleStore.
Industrial machinery is a broad and varied business. It includes components that go into planes, trains, and automobiles – and the machines that make and power those components. The largest industrial machinery companies are global and have hundreds of thousands of employees. These might seem to be classic “old economy” businesses – primarily involved in making things, not software or services, and requiring a great deal of capital and people to get things done. Yet leading companies in this segment are well-represented on the cutting edge of technology, and are heavy users of SingleStore.
As with other cutting-edge businesses, industrial machinery companies need speed, across their global operations. For the SingleStore customer profiled here, the immediate need for speed was in analytics. Like many large organizations, the company has an internal financial reporting application, which we’ll refer to here as Cadence.
The Need for Speed in Cadence
Cadence is a high-visibility product within this global company. In fact, given that the company has hundreds of thousands of employees, along with a large network of suppliers, customers, and other stakeholders, Cadence is a widely-used and well-known application by any standard.
Cadence supports direct SQL queries from a wide range of users, and from widely used business intelligence (BI) tools such as Dundas BI, Looker, Microsoft Power BI, and Tableau. Cadence users include personnel at all levels of the company, including the very top. Users were eager to get the data Cadence provided, and dashboards and custom applications made the data easily accessible and actionable – at first.
Read Post
Case Studies
Middle Eastern Technology Company Partners With BCT on Effort to Limit COVID-19 Spread using SingleStore
The following press release has been issued by SingleStore today – Wednesday, July 28th, 2020 – to announce the use of SingleStore technology for COVID-19 tracking by SingleStore partner Bahwan Cybertek (BCT).
Middle Eastern Technology Company Partners With BCT on Effort to Limit COVID-19 Spread using SingleStore
Joint Endeavor Highlights the Power of Digital Transformation, Massively Scalable Architecture
SAN FRANCISCO – July 29, 2020 – In an effort to help a government in the Middle East to minimize the spread of COVID-19, Bahwan Cybertek (BCT) will deploy technology from SingleStore, The Single Database for All Data-Intensive Applications for operational analytics and cloud-native applications. The effort is led by a large technology company in the region.
The client company offers digital services and information and communications technology solutions in several categories, including cybersecurity, digital media, financial technology, IT and telecommunications. For this project, it is collecting data from other telcos in the region to present the relevant government ministry with a unified view of the coronavirus situation and enable it to maximize the impact of the solution.
This project requires the ingestion of data from 1.2 billion location points to be joined to 5 million location tiles, each 100 meters square, to specify the location of mobile phones. The location tiles are aggregated into cities, districts, and regions to provide an understanding of point-in-time population density.
At first, the company attempted to use technology from Teradata for this joint process, but a test query never returned. When Hadoop was tried, it took more than a day to return results. SingleStore replies to the same complex query in 20 minutes. Using SingleStore, the complete workflow process was built in just three weeks.
The client company obtained the SingleStore technology through its work with BCT, the leading systems integrator in the Middle East, which also has significant operations in India. BCT specializes in digital experience, digital supply chain management, and predictive analytics.
Leveraging its international network of industry experts across a range of industries, BCT has delivered digital transformation solutions to more than 1,000 customers in banking, government, oil & gas, power, retail, and supply chain management (SCM)/logistics across Africa, Asia, the Far East, the Middle East, and North America.
“SingleStore is excited to be working with BCT as a systems integration and reseller partner to serve this client, which evaluated several market-leading products and chose SingleStore for its ability to manage workloads at a massive scale, pull in data from a very diverse set of systems with high concurrency, and to answer SQL queries at amazing speeds,” said Raj Verma, SingleStore co-CEO. “This customer achieved more than three million transactions per second within the first few weeks of using SingleStore.”
BCT has deep relationships with its customers and helps its clients – some of whom are also SingleStore clients – get the maximum value from their technology investments. BCT is building deep capabilities related to SingleStore technology, which it is combining with its other domain and technology experience to help customers succeed throughout their digital transformation journeys. Meanwhile, BCT, SingleStore, and the client company are now launching multiple additional projects, reducing the client company’s dependence on Oracle and Teradata, and augmenting its existing Hadoop environment.
“Companies that invest more in digital transformation actually outperform their peers over time,” said S. Durgaprasad (DP), co-founder, director, and group CEO of BCT. “These companies are more prepared for disruption, and better able to monetize new digital channels and build bigger user bases. What’s more, this phenomenon exists regardless of industry vertical.”
About SingleStore
SingleStore is The Single Database for All Data-Intensive Applications, powering modern applications and analytical systems with a cloud-native, massively scalable architecture. SingleStore delivers maximum ingest, accelerated transaction processing and blisteringly fast query performance, including AI integration and machine learning models, all at the highest concurrency. Global enterprises use the SingleStore distributed database to easily ingest, process, analyze and act on data, to thrive in today’s insight-driven economy. SingleStore is optimized to run on any public cloud or on premises with commodity hardware. Visit www.singlestore.com or follow us @SingleStoreDB.
Contact
Gina von Esmarch 415-203-4660 gvon@singlestore.com
Read Post
