Author
Floyd Smith
Director, Content Marketing
Data Intensity
How Infiswift Supercharged Its Analytics for IoT & AI Applications
Infiswift uses AI to help meet real-world challenges, largely through Internet of Things (IoT) deployments. They optimize the operation of physical devices using connectivity and data. The company’s innovative platform makes it easy to connect and manage any number of endpoints at scale, with security, and to build solutions that open new services or improve existing ones. Infiswift empowers its customers to be data-driven in industries such as renewable energy, agriculture, and manufacturing. Infiswift has chosen SingleStore as the real-time insights engine of its platform to deliver fast, reliable analytics, and to power constantly updated machine learning (ML) models. We’ve updated this blog post with fresh insights from our recent interview with Infiswift CTO Jay Srinivasan.
Read Post
Case Studies
SingleStore Ramps Up the Cadence of Financial Dashboards for an Industrial Machinery Leader
According to analysts, “the growth rates of insights-driven businesses makes them an economic tidal wave” – one that will earn \$1.8 trillion dollars by 2021. How does a long-established, Global 500 company, listed in the Top 100 global brands, become an insights-driven business? A leading producer of industrial machinery is taking bold steps in this direction, powered by SingleStore.
Industrial machinery is a broad and varied business. It includes components that go into planes, trains, and automobiles – and the machines that make and power those components. The largest industrial machinery companies are global and have hundreds of thousands of employees. These might seem to be classic “old economy” businesses – primarily involved in making things, not software or services, and requiring a great deal of capital and people to get things done. Yet leading companies in this segment are well-represented on the cutting edge of technology, and are heavy users of SingleStore.
As with other cutting-edge businesses, industrial machinery companies need speed, across their global operations. For the SingleStore customer profiled here, the immediate need for speed was in analytics. Like many large organizations, the company has an internal financial reporting application, which we’ll refer to here as Cadence.
The Need for Speed in Cadence
Cadence is a high-visibility product within this global company. In fact, given that the company has hundreds of thousands of employees, along with a large network of suppliers, customers, and other stakeholders, Cadence is a widely-used and well-known application by any standard.
Cadence supports direct SQL queries from a wide range of users, and from widely used business intelligence (BI) tools such as Dundas BI, Looker, Microsoft Power BI, and Tableau. Cadence users include personnel at all levels of the company, including the very top. Users were eager to get the data Cadence provided, and dashboards and custom applications made the data easily accessible and actionable – at first.
Read Post
Case Studies
Tapjoy Achieves 10X Performance Gains in Move to SingleStore
Tapjoy is an early and enthusiastic customer of SingleStore. They described their move to SingleStore on the Tapjoy engineering blog. This blog post gives some of the high points of Tapjoy’s experience, spelling out why Tapjoy moved – and what they now get from SingleStore. (This blog post is updated from the original version, which appeared several years ago. It includes more details, from several sources, including recent updates to SingleStore.)
Tapjoy is a leader in advertising and app monetization. They use video ads, offers, and rewards to help app publishers grow their user base and monetize mobile apps. They have been very successful, with their SDK embedded in more than 30,000 mobile apps. Tapjoy reaches more than 800 million active users a month and has more than a dozen offices worldwide.
Tapjoy’s engineering team is also global. They have built up a robust data processing backend, which processes billions of data points a day. The backend includes Kafka for data streaming, Amazon’s RDS, MySQL, PostgreSQL, and Google BigQuery. They’ve established solid procedures for problem-solving, vendor assessment, and change management.
The company faced performance issues with a workload running in MySQL. They carried out a careful assessment of alternatives, tested them, then made their move – the move to SingleStore.
What Tapjoy Needed
Tapjoy’s growth often leads them to improve their stack with new components that scale better. Several years ago, the company had a key workload, including financial data, in MySQL. This workload was an exception to a key TapJoy rule: no applications of large size and scale on MySQL. Widely known problems with MySQL include:
Not scalable for performance. (See the link, and the MySQL vs. SingleStore comparisons below.)The database is owned by Oracle, which is not transparent about the MySQL roadmap.MySQL is suffering defections from many users.
Any key workload that stops performing well causes Tapjoy business risks similar to those of any digitally-driven company: inability to grow the business efficiently, greater need for technical personnel, difficulty meeting service level agreements (SLAs), and lower quality of service.
The team at Tapjoy assessed alternatives. They wanted to move fast, so they used a strict set of criteria:
ACID-compliant. The new product needed to be a relational database – either a traditional RDBMS or a NewSQL contender. NoSQL was not an option.Scalable. The system needs to be designed for scalability. While Tapjoy has a strong DevOps team, they weren’t going to be asked to support a system that wasn’t ready to scale.10x-capable. In particular, any new system needed to scale to meet demand at least an order of magnitude greater than was already occurring at the time.Drop-in replacement. The new solution needed to “swap in” smoothly for MySQL; the change needed to be quick, easy, and as code-compatible with MySQL as possible.
The need for scalability was where MySQL fell down, and that’s not only a MySQL issue; RDBMS alternatives tend to lack the ability to scale to the extent Tapjoy needs.
What Tapjoy Considered
With all these constraints, Tapjoy only found a few alternatives worthy of serious consideration:
Sharded MySQL. Many organizations meet scalability demands by sharding MySQL. Neither engineering nor Ops wanted to take on this challenging task, nor to stay with a system that was already failing to do the job.AuroraDB. The AuroraDB option in AWS Relational Data Service (RDS) is wire-compatible with MySQL, a big advantage. However, when tested, it didn’t clear the bar for performance. AuroraDB is scalable by design, but it did not meet Tapjoy’s needs for scalability, in practice.SingleStore. SingleStore was already in use by the Tapjoy data science team. This team described SingleStore as ACID-compliant and fully scalable, offering stellar performance. And, like AuroraDB, it’s MySQL wire-compatible.
SingleStore is designed from scratch to meet demanding requirements like Tapjoy’s.
SingleStore Offers Huge Performance Gains
As part of their consideration of alternatives, the team compared SingleStore to alternatives, and here’s where SingleStore really stood out: the legacy system was far slower than SingleStore.
Sean Kelly, who wrote the Tapjoy engineering blog post about the move to SingleStore, sets the stage: “SingleStore had everything we wanted, and after an initial pilot, we confirmed it would meet our stated requirements. But the proof is in the pudding. We set out to build a cluster, shadow-write data to it, and put it through a battery of tests to make sure it would hold up to our use cases.”
The tests included putting SingleStore under 20x stress test loads and running it with twice the amount of historical data available as the previous database solution, while that database was run under existing cluster load.
Summing a financial data point for a single partner – an operation that needs to be repeated over and over, given Tapjoy’s huge reach in gaming – took more than a minute (68 seconds) on the old solution. It took 1.23 seconds – 55 times faster – on SingleStore.
Read Post
Case Studies
(v)WeCare/Telgoo5 Uses SingleStore to Give MVNOs Ultra-Fast Connections to AT&T, T-Mobile, Verizon, and More
“In our business, you innovate, or you die. We had to find a company that innovates. We found that with SingleStore." Sandip 'Sandy' Mehra, CEO, Telgoo5 and (v)WeCare Technology
Imagine a business that provides customer service-as-a-service to a wide range of clients worldwide. It develops a robust, in-house digital infrastructure to serve a wide range of customer needs. Eventually, the digital infrastructure becomes so capable it decides to offer this as a product in its own right; the new offering is a huge success, and it runs on SingleStore.
No, not AWS (a SingleStore partner, and one of the public cloud platforms for SingleStoreDB Cloud). We’re talking about (v)WeCare Technology/Telgoo5, a long-standing and highly innovative SingleStore customer.
It all began at Vcare, which started out providing call center support as a service to a wide range of customers. Now Vcare has spawned a new business: a real-time charging platform for mobile telecom operators. The system, Telgoo5, provides mobile virtual network operators (MVNOs), who sell mobile access to their customers, with ultra-fast connections to the largest communications service providers (CSPs) including AT&T, T-Mobile, Verizon, and more. It also keeps (v)WeCare Technology ahead of its competitors such as CSG, NEC/NetCracker, and Oracle. Global research firm MarketsandMarkets forecasts the MVNO market to grow to nearly $90B by 2024.
What makes these processes data-intensive is that an MVNO must verify a multitude of things FAST to ensure its customers' calls connect when they place a call. It must verify that a customer has sufficient balance available to make a phone call, send a text message, or use apps, and it must also verify background data requests from various apps including messaging, mapping, GIS, and navigation. If verification does not occur instantaneously, in 250 milliseconds or less, the data connection fails, calls don't connect — and MVNO customers churn.
“In the current pandemic crisis, the two sides of our business are behaving differently,” said Sandip 'Sandy' Mehra, Chief Executive Officer, Telgoo5 and (v)WeCare Technology. “On the Vcare side, where we provide customer support services, we are waiting and watching, to see how customer demand changes during the crisis. But on the prepaid mobile side, we anticipate steady demand, or even growth. Communication is more and more important, and prepaid mobile is a flexible alternative that customers like — especially with the performance that SingleStore is helping us to provide.”
Telgoo5 used to rely on a MySQL-based solution running in five data centers around the world. CSPs sharded the database manually; partitioning problems and the need to reorganize data frequently caused downtime. The system was often missing or barely meeting the 250ms window, causing connections to drop. This created unhappy MVNOs and MVNO customers, and reflected poorly on Telgoo5; and while this problem was shared by competitors, Telgoo5 was missing a major opportunity to stand out from the crowd.
Solution: SingleStore
Four years ago, SingleStore pitched Mehra and team on a SingleStore-based solution and today they are running SingleStoreDB Cloud on AWS. Telgoo5 has discontinued using those five physical data centers, each of which was a potential source of downtime and other problems. Missed connections and downtime have gone from “all too common” to “almost unheard of” as data connections often occur in as little as 35ms, more than 80% faster than required.
Telgoo5 uses its new infrastructure to run machine learning models and AI programs monitoring customer success and suggesting interventions, such as special offers and discounts on additional service levels, to help keep customer loyalty high. (Yes, as in our joint blog post with Fiddler.ai, Telgoo5 is using AI to reduce churn.) Also, as 5G deployments grow, the Telgoo5 system smoothly uplevels to the much higher data volumes and much shorter response times needed by 5G, with nary a hiccup.
Telgoo5’s telecom customers have more than tripled their business volume with no impact on uptime or performance. Telgoo5 simply scales its infrastructure smoothly as business volume grows, and confidently reaches out to new customers, growing its solutions both horizontally and vertically to provide virtually unlimited charging capability.
Telgoo5 is facing different challenges in its two lines of business. In the call center and related support services, (v)WeCare Technology's original business, disruption caused by the coronavirus upended business planning worldwide, and the need for call center support was reduced. On the online charging side, however, businesses are seeing an ever-greater need for mobile communications. Prepaid mobile service, which is what Telgoo5 supports, provides a flexible and cost-effective alternative to subscription plans. With its backbone running on SingleStore and AWS, Telgoo5 is well equipped to maintain and grow its share of this competitive market.
The Online Charging Systems Challenge
Online charging systems are used to provide data services for prepaid mobile users – the most common form of mobile phone billing in the developing world, and increasingly popular in economically developed countries. When a phone call is made, or a data connection is requested, metadata about the request – not actual voice or data content – goes out to the online charging system.
The online charging system has a brief window, typically 250ms, to either make the voice or data connection, or fail. A mobile phone may instantly make 60 simultaneous requests for data. Requests come from mapping apps such as Google Maps, messaging programs such as WhatsApp and Skype, email, built-in text messaging, and many others.
The data requests are made in 5MB chunks. For each request, if the connection is not made in time, or if a voice call or data transfer fails in progress, the phone service has failed the customer.
It doesn’t take many failures for customers to demand refunds, cancel service, switch suppliers, etc. At worst, a customer can develop the belief that prepaid services in general simply don’t work well enough, taking them entirely out of the market that Telgoo5 serves.
Read Post
Case Studies
SingleStore Improves Financial Operations for a Major Oil and Gas Company
A major oil and gas company works in a volatile market, where IoT data floods in at the rate of gigabytes per second – and where up-to-date financial projections, generated by a .NET application, serve as a major competitive advantage. After considering MongoDB and Oracle Exadata, they replaced two NoSQL data engines at the core of their analytics architecture — Hadoop and ElasticSearch — with SingleStoreDB. After a fast implementation, they now run financial forecasts multiple times a day, with much greater performance, with many fewer servers, and at much lower cost than with competitors. The company now also uses SingleStoreDB for a wide range of financial forecasting applications, land contracts analysis, ERP reporting, and more.
Introduction
Finding, extracting, and delivering oil and gas is one of the most challenging businesses in the world. And it is also, in normal times, one of the most lucrative businesses in the world. But the recent fluctuations in the price of oil, along with a collapse in demand caused by the coronavirus epidemic, have exerted unprecedented pressures on this industry
But oil and gas has always been a tough business. One of the biggest challenges, and opportunities, is financial. Well-run companies use mountains of data to help them manage their reserves of available capital. With good access to capital, when things look good, these companies invest in turning on existing supply, and exploring for new reserves. When times are tough, they reduce production and hold tight until the next opportunity.
Players which are poorly run, or which are too small to compete effectively, tend to miss out on some of the peaks and get caught short in the valleys, driving them to sell out, or go out of business. The assets of the weaker company are then purchased by one of their better-run competitors, which thereby grows larger, and – to the extent that the company is run well – more resilient.
A large oil and gas company has built an extensive Internet of Things (IoT) deployment across its physical plant of wells, drilling rigs, and pipelines. They also use a wide range of external data feeds, and mix this internally generated and market information to make smart decisions. However, a few years ago, their IT infrastructure was not up to the task.
They replaced the databases at the core of their infrastructure with SingleStoreDB. The primary use case for this newly powerful engine is to constantly adjust the company’s financial forecasts. Management is in constant touch with everyone from crews in the field, to lenders and analysts in financial capitals, while scrutinizing roughly 100 major variables at a time. With all of this input, they are constantly creating and stress-testing potential new budgets. All of this forms a company-wide risk exercise, with outsize rewards for success, and severe business consequences for failure.
In this case study, we’ll describe in detail the business problem caused by the previous architecture; what alternative solutions were considered; and how SingleStore helped solve the problem. Now, using SingleStoreDB, the company stands out from competitors, stays ahead of the market and grows, through the best and the worst of times.
This is likely to be of interest to other oil and gas companies; other companies that use IoT in daily operations; companies that need financial analytics and budgeting flexibility, across large and changing data sets; and any company that needs to speed up the flow of information, to meet the current wave of business challenges.
Company Profile
Several years ago, the oil and gas company described here, which is based in Texas, approached SingleStore. This Fortune 500 oil and gas company is publicly held, with tens of billions of dollars in annual revenues and thousands of employees. They actively explore for hydrocarbons – oil and gas – predominantly from shale. Shale sources are notable for being on the expensive side to develop, but are also more easily turned off and on for production than traditional sources of oil and gas. This means they can be actively managed, differently than traditional sources – and elicits the need for constant flows of information to get the best use from the shale resources.
The company uses the data from sensors across the drilling process for real-time well production analysis. They combine this with a wide range of live, always-on, internal and external data feeds to embed themselves in financial and informational data flows. The company can use these massive data flows for a wide range of purposes:
To quickly repair problems with rigs and other infrastructureTo predict when maintenance should occurTo decide when to run which rigs, as prices and customer demand changeTo work with financiers to manage cash and access to capital
The company uses these data flows to reduce non-productive time (NPT) and cut costs – and also to tap capital markets and take advantage of opportunities. But the company’s ability to bring in and process all of this data was severely handicapped by slow and limited data processing operation.
Because of this, decisions that could have saved the company millions of dollars a week – or made the company even more money – went unmade, as the data was not available when and where it was most needed.
Existing resources were weak at every point:
Ingest capability could not handle the massive flows of internal and external data in a timely fashion.Processing capability could not integrate live, streaming data with historical data, costing the company the ability to make decisions using all of the data that it has access to.Analytics were slow; in a business where time is money, waiting for answers was intolerable. The company needs to be able to constantly generate and test new budgets, on a region-by-region basis; slow analytics drastically limited flexibility.Concurrency for analytics was poor;the company could not support all of its current and desired users at once, to include ad hoc queries, business intelligence (BI) tool queries, and API calls from apps, all hitting the database(s) at the same time. Adding more analytics or BI tool users, or a new app, slowed down results for all users.
The business saw that its weakest point was financial forecasting. If they could integrate all this data to generate solid budgets, and test them against different scenarios so as to best manage risk, they could get ahead of aggressive competition.
After careful study, SingleStoreDB was chosen, to speed ingest, processing, and the delivery of results, with increased concurrency. The new system was scoped, deployed, and brought into production in just seven months. Costs are down; flexibility, manageability, and profitability are up.
The Business Problem
The company is constantly looking for new customers and negotiating with existing ones. Like any public company, they need to meet detailed financial reporting requirements, and to accurately forecast revenues, costs, and profits and losses – per region, and then roll these budgets up for the company as a whole.
None of this is easy. Shale oil company operations are heavily dependent on prices of conventional oil and gas, as production costs must be below the commodity price for a well to be profitable. These costs include drilling and completion costs, delivery costs, operating expenses, taxes, and more. Companies use decline curves to look at the rate of production decline over the life of a well, and work to determine what the estimated ultimate recovery (EUR) will be for a given well. Getting the economics right – not quarter by quarter, but day by day – is a necessity for these companies, given the impact of changing commodity prices.
Read Post
Trending
A HOAP-ful New Perspective from 451 Research
Analyst firm 451 Research has come up with new research that sees a bright future for HOAP – hybrid operational and analytical processing. The report is titled 451 Perspective: A HOAP-ful future for hybrid operational and analytical processing.
This new type of processing has received several different names, from different analyst firms and consultancies:
HTAP (Gartner)Translytical processing (Forrester)HOAP (451 Research)Operational analytics (common amongst other analyst firms)
By any of these names, this new style of data processing – which unifies transactions and operational analytics, including many data warehousing-type functions – is widely believed to have a bright future. And SingleStore is right in the middle of it.
What HOAP Replaces
In a previous report, 451 Research identified HOAP as an emerging category. Now, they see HOAP experiencing broad adoption. HOAP seeks to unify two formerly separate data processing categories, and to largely eliminate the need for a third:
Online transaction processing (OLTP). Transaction processing systems have various needs for reliability, with requirements which begin at strong – “nearly always works” – to absolute -“must work every time, across time zones and disparate data centers, safeguarding against serious financial and reputational consequences for data loss or significant downtime.”Online analytics processing (OLAP). Analytics processing systems, which include data warehousing systems, data marts, and data shoeshine stands (just kidding), typically work on copies of existing data. They must be fast, reliable, scalable to multiple apps and users, and affordable, with SQL support.Extract, transform, and load (ETL). Systems which transform and format data from ingest or OLTP systems to OLAP systems have become a separate category of their own. Using an ETL system reduces the requirements for the OLTP and OLAP systems that an ETL product connects to, but adding a third system to the mix increase cost and complexity, as well as ensuring significant end-to-end latency.
One of the advantages that 451 Research cites for OLTP systems, and their use of rowstore tables, is the ability to handle complex queries with joins. And they also cite architectural advantages to the separation of transactions, given that these are well suited to rowstore tables, and analytics, which usually benefit from the use of columnstore tables.
SingleStore, however, fuzzes over these distinctions – and, with Universal Storage, is on track to nearly eliminate them. SingleStore not only supports both table types in a single database; it supports joins, and other operations, for rowstore tables, columnstore tables, and across table types.
Read Post
Trending
Cloud Database Trend Report from DZone Features SingleStore
A new Trend Report from DZone highlights the move to cloud databases. The report paints a bright picture for cloud database adoption, with roughly half of developers asserting that all of their organization’s data will be stored in the cloud in three years or fewer. You can get a copy of the report from SingleStore.
DZone has issues a new trend report on cloud databases. In the report, leaders in the database space focus on specific use cases, calling out the factors that help you decide what you need in any database, especially one that’s in the cloud.
The advantages of cloud databases include flexibility to scale up and scale back, easier backups of data, moving database infrastructure out of house, and offloading some database maintenance.
SingleStore is a database that runs anywhere Linux does, on-premises and on all three major cloud providers – AWS, Google Cloud Platform (GCP), and Microsoft Azure. SingleStoreDB Cloud is a managed service with the SingleStore database at its core. SingleStoreDB Cloud is available on AWS and GCP, with Azure support to follow soon. The SingleStore Kubernetes Operator gives you the flexibility to manage this cloud-native database with cloud-native tools.
SingleStore is also a fast, scalable SQL database that includes many features that are normally claimed only by NoSQL databases: easy scalability, fast ingest, fast query response at volume, and support for a wide range of data types, especially JSON and time series data.
Between SingleStoreDB Self-Managed (the version you download and run on Linux), and SingleStoreDB Cloud, the advantages of cloud databases – scalability, easy and reliable backups, and moving both infrastructure and maintenance out-of-house – are readily available, on a solution that’s also identical on-premises.
The report points out several interesting facts:
Slightly more than half of organizations that have a cloud database solution in place have had one for two years or less.More than two-thirds of cloud database users either use multiple clouds (40%) or are seriously considering doing so (26%).Analytics is the #1 reason for moving databases to the cloud, with modernization of existing apps and becoming cloud native also ranking highly.The database as a service (DBaaS) model, represented by SingleStoreDB Cloud and many other options, has a slight lead over those who use a self-managed database.About half of respondents believe all of their data will be in the cloud in three years or fewer.
Read Post
Product
SingleStoreDB Cloud Technical Overview
Solutions Engineer Mihir Bhojani presents a 20-minute technical overview of SingleStoreDB Cloud. In this webinar recap, we present the key points of the interview, and give you a chance to review them at your leisure. You can also view the SingleStoreDB Cloud Technical Overview.
In this webinar, we’ll cover what exactly SingleStoreDB Cloud is and how it compares with self-managed SingleStore, which you download yourself, provision, and run in the cloud or on-premises. With SingleStoreDB Cloud, SingleStore provisions the hardware in the cloud and runs SingleStore itself; you just set up tables and manage your data. After describing SingleStoreDB Cloud, I’ll have a hands-on demo that will show you the whole end-to-end process of getting started with SingleStoreDB Cloud.
Read Post
Product
Best Practices for Migrating Your Database to the Cloud – Webinar Recap 3 of 3
This webinar concludes our three-part series on cloud data migration. In this session, Domenic Ravita actually breaks down the steps of actually doing the migration, including all the key things you have to do to prepare and guard against problems. Domenic then demonstrates part of the actual data migration process, using SingleStore tools to move data into SingleStoreDB Cloud.
About This Webinar Series
This is the third part of a three-part series. First, we had a session on migration strategies; broad-brush business considerations to think about, beyond just the technical lift and shift strategy or data migration. And the business decisions and business strategy to guide you as to picking what sorts of workloads you will migrate, as well as the different kinds of new application architectures you might take advantage of. And then, last week, we got down to the next layer, talked about ensuring a successful migration to the cloud of apps and databases in general.
In today’s webinar we’ll talk about more of the actual migration process itself. We’ll go into a little bit of more detail in terms of what to consider with the data definition language, queries, DML, that sort of thing. And then I’ll cover one aspect of that, which is the change data capture (CDC) or replication from Oracle to SingleStore, and show you what that looks like.
Database Migration Best Practices
I’ll talk about the process itself here in terms of what to look at, the basic steps that you are going to be performing, what are the key considerations in each of those steps. Then we’ll get into more specifics of what a migration to SingleStoreDB Cloud looks like and then I’ll give some customer story examples to wrap up, and we’ll follow this with a Q and A.
Read Post
Product
Ensuring a Successful Cloud Data Migration – Webinar Recap 2 of 3
This webinar describes what’s needed to successfully migrate your data to the cloud. What does a good cloud data migration look like? What are the strategic decisions you have to make as to priorities, and what are the challenges you’ll face in carrying out your plans? We will describe a tried and tested approach you can use to get your first wave of applications successfully migrated to the cloud.
About This Webinar Series
This is the second part of a three-part series. Last week we covered migration strategies; broad-brush business considerations to think about, beyond just the technical lift and shift strategy or data migration. And the business decisions and business strategy to guide you as to picking what sorts of workloads you will migrate. What sorts of new application architectures you might take advantage of.
In today’s webinar we’ll go one level deeper. Once those decisions are mapped out, what does a good cloud data migration look like? And in the final session, we’ll go a layer deeper, and we’ll get into migrating a particular source to target database.
Challenges of Database Migrations
So you’re looking at migrating an on-premises, so-called legacy database. You have particular IT responsibilities that span lots of different areas, lots of different infrastructure, lots of different layers. You’ve got the responsibility of all of these things. But a lot of this work doesn’t provide any differentiation for you in the marketplace or for your business.
So when you look at what’s possible in moving to a cloud database, the main thing that you get to take advantage of is a lot of that infrastructure is taken care of for you. And so any cloud database is going to greatly reduce that cost of ownership in terms of the infrastructure management.
So the general value proposition of a cloud database, or any SaaS service, is that it allows you to reduce all of this work and focus on the business differentiation of your application.
Read Post
Product
Migration Strategy for Moving Operational Databases to the Cloud – Webinar Recap 1 of 3
This webinar describes the benefits and risks of moving operational databases to the cloud. It’s the first webinar in a three part series focused on migrating operational databases to the cloud. Migrating to cloud based operational data infrastructure unlocks a number of key benefits, but it’s also not without risk or complexity. The first session uncovers the motivations and benefits of moving operational data to the cloud and describe the unique challenges of migrating operational databases to the cloud. (Visit here to view all three webinars and download slides.)
About This Webinar Series
Today starts the first in a series of three webinars:
In this webinar we’ll discuss in broad strokes, migration strategy, cloud migration, and how those strategies are influenced by larger IT transformation or digital transformation strategy.In our next webinar, we’ll go into the next level of details in terms of database migration best practices, where we’ll cover processes and techniques of database migration across any sort of database, really.In the final webinar, we’ll get specific to the technical nuts and bolts of how we do this in migrating to SingleStoreDB Cloud, which is SingleStore’s database as a service.
In this webinar, we’ll cover the journey to the cloud, a little bit about the current state of enterprise IT landscapes, and some of the challenges and business considerations that go into making a plan, making an assessment, and choosing what kind of workloads to support.
Next we’ll get into the different types of data migrations that are typically performed. And some of the questions you need to start asking if you’re at the beginning of this kind of journey. And finally, we’ll get into some specific types of workloads along the way.
Any sort of change to a functioning system can invoke fear and dread, especially when it comes to operational databases, which of course process the critical transactions for the business. After all, they’re the lifeblood of the business. And so, we’ll start to peel the onion and break that down a little bit.
If you’re just starting your journey to the cloud, you’ve probably done some experimentation, and you’ve spun up some databases of different types in some of the popular cloud vendors. And these cloud providers give guidelines oriented towards the databases and database services that they support. There’s often case studies which relate to transformations or migrations from Web 2.0 companies, companies like Netflix, who famously have moved all of their infrastructure to AWS years ago.
But in the enterprise space, there’s a different starting point. That starting point is many years, perhaps decades of lots of different heterogeneous technologies. In regards to databases themselves, a variety of different databases and versions over the years. Some that are mainframe-resident, some from the client-server era, older versions of Oracle and Microsoft SQL, IBM DB2, et cetera.
And these databases perform various workloads and may have many application dependencies on them. So, unlike those web 2.0 companies, most enterprises have to start with a really sober inventory analysis to look at what their applications are. They have to look at that application portfolio, understand the interconnections and dependencies among the systems.In the last 10 to 15 years especially, we see the uptake of new varieties of data stores, particularly NoSQL data stores such as Cassandra or key-value stores or in-memory data grids, streaming systems, and the like.
Note. See here for SingleStore’s very widely read take on NoSQL.
Introduction
In companies that have just been started in the last 15, 20 years, you could completely run that business without your own data center. And in that case, your starting point often is a SaaS application for payroll, human resources, et cetera. In addition to new custom apps that you will build, and of course, those will be on some infrastructure or platform as a service (PaaS) provider.
So some of this is intentional, and that large enterprises may want to hedge their bet across different providers. And that’s consistent with a traditional IT strategy in the pre-cloud era, where I might have an IBM Unix machine, and then an HP Unix machine, or more recently Red Hat, Linux, and Windows and applications.
Read Post
Data Intensity
Capterra Captures Positive Reviews of SingleStore
Capterra describes itself as “the leading online resource for business software buyers,” boasting more than one million verified software reviews across hundreds of software categories. Those one million reviews include many positive takes on SingleStoreDB Self-Managed, including this comment from a big data manager: “Best Distributed RDBMS Ever!”
Capterra is growing fast, with more than five million monthly users, and the number of verified reviews on the site nearly doubling between 2018 and mid-2019. Capterra research asserts that reviews reduce purchase cycles by more than six months. More than four in ten small and medium business (SMB) users surveyed use reviews in their purchase process.
And these users, by and large, love SingleStore. It’s a “great product” with “similarity to MySQL” and “more scalability.” It’s “very fast,” “easy to operate,” and “works perfect on Linux.” “If you are looking for something for real-time analytics/dashboard, this is the go-to option.”
The Capterra site is focused on SMB users. For enterprise users, we have a roundup of comments and an update with newer comments from reviews site G2 Crowd, which is more focused on enterprise users. And we’ve captured highlights from reviews site Gartner Peer Insights, which also focuses on the enterprise. (Gartner owns both Gartner Peer Insights and Capterra, which it purchased for more than \$200M in 2019.)
Together, these review sites can give you a fair picture of SingleStore’s suitability for your needs – and, hopefully, shorten your purchase cycle, as Capterra does for many of its users.
Most Helpful Reviews (Nearly) Say It All
The most helpful reviews show at the top of the list for Capterra’s software reviews. Several of the most helpful reviews for SingleStore include an awful lot of the best features of SingleStore, as seen on Capterra, and across all three of the reviews sites we’ve described:
“One solution for streaming analytics on big data,” says a senior manager for data engineering. He’s focused on machine learning and AI, and he describes the software as “super simple.” His shop runs “multi-petabyte S3” stores with “huge Kafka clusters.” They see “sub-second response on a test case with 1000 concurrent heavy API calls (scanning billions of rows).” SingleStore is “incredibly fast,” “fantastic partners” who offer “access to their core engineering team.”A director of IT infrastructure at a large oil and energy company has built a “simplified data lake system” around SingleStore. He sees “large amounts of IoT data (trillions of rows)” that “can be queried in milliseconds.” Processes that “took hours to run” are now “running sub-second on SingleStore.” The software offers “amazing performance” and is “highly and easily scalable.”A senior architect for IT and services at a large company calls SingleStore a “supersonic DB” that “aces every database” that he has worked with. SingleStore is “the database of the new generation” with “huge potential for both on-premises and cloud.” It features “high compatibility,” “resilience,” and “scalability.” SingleStore is “highly recommended to any organization wanting to get rid of old-fashioned databases.”
Many of the comments offer real insight. One big data manager lists pros which include “JSON support and full-text search,” “drop-in replacement to the famous MySQL,” and “in-memory tables for high OLTP workloads and on-disk columnar storage for OLAP workloads.”
Users are able to “ingest millions of documents every day and run sophisticated dashboards against them.” They achieve a “huge performance win,” see SingleStore as “easy to connect to Kafka” and “easy to set up on Kubernetes.” SingleStore is a “great replacement for Hadoop for a fraction of the cost,” with aggregation times dropping from over 2 hours to less than 20 minutes.
And “You can seamlessly join both row and columnar tables and query across it.”
A few more adjectives, from these and other reviews: “elegant”; “excellent”; “amazing”; “the go-to option” for real-time analytics and dashboards; “great support”; “blazing fast”; “good engineering principles”; “fast implementation” (in a one-day hackathon); “too easy to set up.”
Read Post
Engineering
Building Fast Distributed Synchronous Replication at SingleStore – Webinar Recap 1 of 2
This is the first part of a two-part blog post; part two is here. The recent release of SingleStoreDB Self-Managed 7.0 has fast replication as one of its major features. With this release, SingleStore offers high-throughput, synchronous replication that, in most cases, only slows SingleStore’s very fast performance by about 10%, compared with asynchronous replication. This is achieved in a very high-performing, distributed, relational database. In this talk, available on YouTube, Rodrigo Gomes describes the high points as to how SingleStore achieved these results.
Rodrigo Gomes is a senior engineer in SingleStore’s San Francisco office, specializing in distributed systems, transaction processing, and replication. In this first part of the talk (part two is here), Rodrigo describes SingleStore’s underlying architecture, then starts his discussion of replication in SingleStore.
Introduction to SingleStore Clusters
Okay. So before I start, I’m just going to give some context and a quick intro to SingleStore. This is a rough sketch of what a SingleStore cluster looks like.
Read Post
Engineering
Building Fast Distributed Synchronous Replication at SingleStore – Webinar Recap 2 of 2
This is the second part of a two-part blog post; part one is here. The recent release of SingleStore 7.0 has fast replication as one of its major features. With this release, SingleStore offers high-throughput, synchronous replication that, in most cases, only slows SingleStore’s very fast performance by about 10%, compared with asynchronous replication. This is achieved in a very high-performing, distributed, relational database. In this talk, available on YouTube, Rodrigo Gomes describes the high points as to how SingleStore achieved these results.
Rodrigo Gomes is a senior engineer in SingleStore’s San Francisco office, specializing in distributed systems, transaction processing, and replication. In this second part of the talk (part one is here), Rodrigo looks at alternatives for replication, then describes how SingleStore carries it out.
Considering Replication Alternatives
First, we should define what the goals are. What are the objectives? We have a primary and secondary, as before, two nodes – and we want the secondary to be a logically equivalent copy of the primary. That just means that if I point my workload at the secondary, I should get the same responses as I would on the primary.
What is highly desirable is performance. You don’t want replication to be taking a very long time out of your system, and you don’t really want to under-utilize any resources. So this goes hand in hand with performance.
Read Post
Data Intensity
Gartner Peer Insights Applauds SingleStore
Gartner Peer Insights features technology product reviews from enterprise users, with more than 300,000 reviews covering nearly 6,000 products in more than 330 categories. Upholding their reputation as top industry analysts for enterprise technology, Gartner sees to it that the reviews are, in their words, “rigorously vetted,” with “no vendor bias.” SingleStore has nearly two dozen reviews, with an overall rating of 4.5 stars. Reviews cover highlighting key points of what the software does for users. For those who want to know more about SingleStore, these reviews are a stellar resource. And, for those who are already SingleStore users, you can post a review today.
Each rating includes dozens of areas for comment, in several distinctive areas: evaluation & contracting, such as reasons for purchase; integration & deployment, such as listing other platforms and products the software will be integrated with; service & support, such as the quality of support; product capabilities, such as the variety of data types supported; and additional context.
Read Post
Data Intensity
New Year, New FAQs
As interest in SingleStore increases, we get many questions about how SingleStore works, how to get started with it, and more. Performance reviews website G2.com has a SingleStore Q&A section where potential customers can ask questions. Here are some of the questions we hear most often – with answers – lightly edited, for context and clarity.
Q. What is the advantage of SingleStore over other distributed databases?
A. Compared to relational databases – those which support SQL – we believe that SingleStore is the fastest, most efficient SQL database. SingleStore features full, linear scalability, unlike competitors. We also handle both transactions and analytics in one database, as described by customers in the reviews on G2.com.
Compared to NoSQL, SingleStore is at least as scalable, far more efficient with machine resources, and of course, unlike NoSQL databases, we have full ANSI SQL support. SingleStore also supports data types, such as JSON and geospatial data, that may otherwise only be supported by NoSQL databases.
Q. How to simplify scaling of a SingleStore cluster? We would like our microservices to use-in memory processing and storage for analytics purposes.
A. This question does seem particularly pertinent to microservices, as you are more likely to have multiple data stores. There are several parts to the answer:
This tutorial describes how to scale your cluster for optimal performance.You can use Kubernetes, specifically the SingleStore Kubernetes Operator, to scale clusters flexibly.With SingleStoreDB Cloud, our elastic managed service in the cloud, you simply send a request, and SingleStore will quickly rescale the cluster for you.For more specifics, please use the SingleStore Forums to give a more detailed description and get a more detailed answer – or file a support ticket, if you have an Enterprise license.Alternatively, contact SingleStore directly for more in-depth information.
Q. Is SingleStore a row-based or column-based store?
A. We are happy to report that the answer is: Yes. SingleStore supports both rowstore and columnstore tables in the same database (or separate databases), with the ability to efficiently run JOINs and other SQL operations across both table types. We also support new capabilities, under the umbrella of SingleStore Universal Storage, which will gradually unify the two table types for most workloads; see the description of Universal Storage-related changes in SingleStoreDB Self-Managed 7.0, below. And see the SingleStore review comments on G2.com for more information about rowstore and columnstore tables, and also contact SingleStore directly.
Read Post
Data Intensity
The G2 Crowd Has (Much) More to Say About SingleStore
SingleStore is garnering a lot of positive attention on business solutions review site G2.com (formerly known as G2 Crowd). As Walt P, a data engineer at an enterprise company, puts it: “We have loaded data in volumes that have historically stopped other database technologies in their tracks.” Since our look at the site last July, dozens of new reviews have been posted, citing SingleStore’s speed, high capacity, and SQL compatibility, among other features. However, the most recent review was in December; now, some lucky SingleStore user has the chance to register the first comment of the new year. (Also, as most would say, of the new decade, though – as xkcd points out – this is a matter of some controversy.)
G2.com features information and frank user reviews of business solutions – software and services – across a wide range of categories. Software categories include CRM software, demand generation software, BI tools, AI development and delivery tools, marketplace platforms, CAD/CAM tools, and many more. Services categories include lawyers, tax people, sales training, graphic design, website design, staffing services, channel partners, and much more.
It’s easy to link from a product to its overarching category (such as Relational Databases Software, for SingleStore and related software, or to interact with a G2 Advisor, to help you narrow your search with help from a live person. (Not a bot – at least, not last time we checked.) You can quickly share your comments on social media, ask a vendor question, or request a demo from a vendor.
Using G2.com will help you see whether a product is a strong player in the market; assess its strengths and weaknesses; look at the competition; and get ready to answer intelligent questions in a vendor call. Your review comments encourage the vendors of products you use to keep up the good work and to fix any problems.
Note: Be sure to hit the Show More link on G2.com reviews. Otherwise, you might miss comments about recommendations to others, the problems being solved, benefits being realized, and other valuable information.
What SingleStore Users on G2.com Say About SingleStore
The recent comments on G2.com have yielded a star rating of 4 out of a possible 5 points. The major positive areas cited in include speed, capacity, and ease of implementation. (Some of the comments have been lightly edited for spelling, syntax, spelling out acronyms, and so on – the original comments are on the G2 site.)
Speed
The major focus of many comments, and a common thread through nearly all of them, is SingleStore’s speed.
SingleStore achieves its high speed through a distributed, lock-free architecture. New features in SingleStore Universal Storage improve speed further, with smaller data size (equals faster loading times) for rowstore tables, and faster seeks and faster updating, using secondary hash indexes, for columnstore tables.
Read Post
Engineering
The Write Stuff
Why did the world need SingleStore? In this blog post, updated from a few years ago, early SingleStore Product Manager Carlos Bueno explains why SingleStore works better, for a wide range of purposes, than a NoSQL setup. (Thanks, Carlos!) We’ve updated the blog post with new SingleStore product features, graphics, and relevant links. To wit, you should also see Rick Negrin’s famous blog post on NoSQL and our recent case study on replacing Hadoop with SingleStore.
Tell us if this sounds familiar. Once upon a time a company ran its operations on The Database Server, a single machine that talked SQL. (Until the advent of NewSQL, most relational database systems – the ones that support SQL – ran on a single core machine at a time. – Ed.) It was tricked out with fast hard drives and cool blue lights. As the business grew, it became harder for The Database to keep up. So they bought an identical server as a hot spare and set up replication, at first only for backups and failover. That machine was too tempting to leave sitting idle, of course. The business analysts asked for access so they could run reports on live-ish data. Soon, the “hot spare” was just as busy – and just as mission-critical – as the master server. And each machine needed its own backup.
The business grew some more. The cost of hardware to handle the load went way up. Caching reads only helped so much, and don’t get us started about maintaining cache consistency. It was beginning to look like it would be impossible for The Database Server to handle the volume of writes coming in. The operations people weren’t happy either. The latest semi-annual schema change had been so traumatic and caused so much downtime that they were still twitching.
It was then that the company took a deep breath, catalogued all their troubles and heartache, and decided to ditch SQL altogether. It was not an easy choice, but these were desperate times. Six months later the company was humming along on a cluster of “NoSQL” machines acting in concert. It scaled horizontally. The schemas were fluid. Life was good.
For a while, anyway. It turned out that when scaled up, the NoSQL cluster worked fine except for two minor things: reading data and writing data. Reading data (“finding documents”) could be sped up by adding indexes. But each new index slowed down write throughput. The business analysts weren’t about to learn how to program just to make their reports. That task fell back onto the engineers, who had to hire more engineers, just to keep up. They told themselves all this was just the price of graduating to Big Data.
The business grew a little more, and the cracks suddenly widened. They discovered that “global write lock” essentially means “good luck doing more than a few thousand writes per second.”
A few thousand sounds like a lot, but there are only 86,400 seconds in a day, and the peak-hour of traffic is generally two or three times the average – because, people sleep. A limit of 3,000 writes per second translates to roughly 90 million writes a day. And let’s not talk about reads. Flirting with these limits became as painful as the database platform they’d just abandoned.
Tell us if this sounds familiar. I’ve seen a lot of companies suddenly find themselves stuck up a tree like this. It’s not a fun place to be. Hiring performance experts to twiddle with the existing system may or may not help. Moving to a different platform may or may not help either. A startup you’ve definitely heard of runs four – count ‘em, four – separate NoSQL systems, because each one had some indispensable feature (eg, sharding or replication) that the others didn’t. That way lies madness.
Read Post
Case Studies
Case Study: SingleStore Replaces Hadoop for Managing 2 Million New-Style Utility Meters
SME Solutions Group is a SingleStore partner. An SME customer, a utility company, was installing a new meter network, comprising 2 million meters, generating far more data than the old meters. The volume of data coming in, and reporting needs, were set to overwhelm their existing, complex, Hadoop-based solution. The answer: replacing 10 different data processing components with a single SingleStore cluster. The result: outstanding performance, scalability for future requirements, the ability to use standard business intelligence tools via SQL, and low costs.
SME Solutions Group (LinkedIn page here) helps institutions manage risks and improve operations, through services such as data analytics and business intelligence (BI) tools integration. SingleStore is an SME Solutions Group database partner. George Barrett, Solutions Engineer at SME, says: “SingleStore is like a Swiss Army knife – able to handle operational analytics, data warehouse, and data lake requirements in a single database.” You can learn more about how the two companies work together in this webinar and in our previous blog post.
Introduction
A utility company had installed a complex data infrastructure. Data came in from all the company’s systems of record: eCommerce, finance, customer relationship management, logistics, and more.
Read Post
Product
Webinar: Time Series Data Capture & Analysis in SingleStoreDB Self-Managed 7.0
With the SingleStoreDB Self-Managed 7. 0 release, SingleStore has added more special-purpose features, making it even easier to manage time series data within our best-of-breed operational database. These new features allow you to structure queries on time series data with far fewer lines of code and with less complexity. With time series features in SingleStore, we make it easier for any SQL user, or any tool that uses SQL, to work with time series data, while making expert users even more productive. In a recent webinar (view the recording here), Eric Hanson described the new features and how to use them.
The webinar begins with an overview of SingleStore, then describes how customers have been using SingleStore for time series data for years, prior to the SingleStoreDB Self-Managed 7.0 release. Then there’s a description of the time series features that SingleStore has added, making it easier to query and manage time series data, and a Q&A section at the end.
Introducing SingleStore
SingleStore is a very high-performance scalable SQL relational database system. It’s really good for scalable operations, both for transaction processing and analytics on tabular data. Typically, it can be as much as 10 times faster, and three times more cost-effective, than legacy database providers for large volumes under high concurrency.
We like to call SingleStore the No-Limits Database because of its amazing scalability. It’s the cloud-native operational database that’s built for speed and scale. We have capabilities to support operational analytics. So, operational analytics is when you have to deliver very high analytical performance in an operational database environment where you may have concurrent updates and queries running at an intensive, demanding level. Some people like to say that it’s when you need “Analytics with an SLA.”
Now, I know that everybody thinks they have an SLA when they have an analytical database, but when you have a really demanding SLA like requiring interactive, very consistent response time in an analytical database environment, under fast ingest, and with high concurrency, that’s when SingleStore really shines.
We also support predictive ML and AI capabilities. For example, we’ve got some built-in functions for vector similarity matching. Some of our customers were using SingleStore in a deep learning environment to do things like face and image matching and customers are prototyping applications based on deep learning like fuzzy text matching. The built-in dot product and Euclidean distance functions we have can help you make those applications run with very high performance. (Nonprofit Thorn is one organization that uses these ML and AI-related capabilities at the core of their app, Spotlight, which helps law enforcement identify trafficked children. – Ed.)
Also, people are using SingleStore when they need to move to cloud or replace legacy relational database systems. When they reach some sort of inflection point, like they know they need to move to cloud, they want to take advantage of the scalability of the cloud, they want to consider a truly scalable product, and so they’ll look at SingleStore. Also, when it comes time to re-architect the legacy application – if, say, the scale of data has grown tremendously, or is expected to change in the near future, people really may decide they need to find a more scalable and economical platform for their relational data, and that may prompt them to move to SingleStore.
Here are examples of the kinds of workloads and customers we support: Half of the top 10 banks banks in North America, two of the top three telecommunications companies in North America, over 160 million streaming media users, 12 of the Fortune 50 largest companies in the United States, and technology leaders from Akamai to Uber.
If you want to think about SingleStore and how it’s different from other database products, you can think of it as a very modern, high-performance, scalable SQL relational database. We have all three: speed, scale, and SQL. We get our speed because we compile queries to machine code. We also have in-memory data structures for operational applications, an in-memory rowstore structure, and a disk-based columnstore structure.
Read Post
Case Studies
Case Study: Moving to Kafka and AI at a Major Technology Services Company
How does a major technology services company equip itself to compete in the digital age – and provide services so outstanding that they can significantly advance the business prospects of their customers? All while reducing complexity, cutting costs, and tightening SLAs – in some cases, by 10x or more? For one such company, the solution is to deliver real-time, operational analytics with Kafka and SingleStore.
In this company, data flowed through several data stores, and from a relational, SQL database, into NoSQL data stores for batch query processing, and back into SQL for BI, apps, and ad hoc queries. Now, data flows in a straight line, through Kafka and into SingleStore. Airflow provides orchestration.
Before SingleStore: Custom Code, PostgreSQL, HDFS, Hive, Impala, and SQL Server
At this technology services company, analytics is absolutely crucial to the business. The company needs analytics insights to deliver services and run their business. And they use their platform to provide reports and data visualizations to their customers. (We’re leaving the company unidentified so they can speak more freely about their process and their technology decisions.)
The company’s original data processing platform was developed several years ago, and is still in use today – soon to be replaced by Kafka and SingleStore. Like so many companies at that time, they chose a NoSQL approach at the core of their analytics infrastructure.
Data flowed through the analytics core in steps:
A custom workflow engine brings in data and schedules jobs. The engine was written in Python to maximize flexibility in collecting data and scheduling data pipelines.The data is normalized and stored in PostgreSQL, one of the leading relational databases.Data then moves into HBase, the data store for Hadoop – a NoSQL system that provides the ability to version data at an atomic (columnar) level.In the next step, data moves to Apache Hive, the data warehousing solution for Hadoop. Then new, updated Parquet tables are created on Cloudera’s version of the Apache Impala Hadoop-to-SQL query engine.Data then moves to SQL Server, another leading relational database, where it can be accessed by traditional, SQL-based business intelligence (BI) tools.
Read Post
Case Studies
Case Study: Medaxion Brings Real-Time Decision-Making to MedTech
Medaxion has found a solution to the analytics problems that plague companies across the health care sector and beyond. John Toups, CTO of Medaxion, puts it plainly. “By combining the ease of presentation and abstraction of Looker for analytics, with the technical prowess of SingleStore as the database behind Looker, we now have simply the best analytics in healthcare.”
Medaxion has changed the working lives of anesthesiologists, its core customers. These highly skilled professionals must deliver absolutely critical medical care, day after day, while also meeting demanding business requirements. These men and women, who once generated a blizzard of paper at every turn, now “record their anesthetics in Medaxion Pulse,” according to Toups, “and through Looker and SingleStore, that data is now actionable.”
Today, surgical teams and patients who sometimes waited hours for an anesthesiologist to arrive are now helped much faster, with the aid of predictive analytics. “If it’s cold and snowy, some elective surgeries might get cancelled because it’s hard to get around,” Toups shared. “At the same time, ER admissions might spike because of hazards on the road and the outdoors. We can monitor the data, minute by minute, and help put anesthesiologists where they’re needed most” – getting care to more people, faster.
Looker Exposes a Slow Database
Looker is an analytics tool that’s designed to make analytics fast and easy. Looker integrates deeply with multiple data sources, rapidly converting user requests to live results. Looker also provides an abstraction layer, LookML, a modeling language that keeps users from having to learn SQL or other code.
With Looker’s speed, and the ease of use provided by LookML, demand for analytics at Medaxion rose sharply. But increased demand meant more queries coming into Looker. And, while Looker itself could handle the volume, MySQL – Medaxion’s previous underlying database – couldn’t keep up.
SingleStore Solves the Problem
Toups and Medaxion had a problem. How did they solve it? Simple: by leaving MySQL and moving to SingleStore.
“Much of what my customers want from analytics is real-time operational information, and there is enormous interest in monitoring and improving quality of care,” said Toups. For instance, anesthesiologists need to be near patients who will be needing surgery. It’s a dispatching problem that is, in a way, similar to the issue faced by Lyft and Uber: have the right person, providing the right service, in the right place, at the right time. But for Medaxion’s anesthesiologist clients, and the patients who need them, the stakes are higher.
With Looker running on MySQL, Medaxion experienced significant problems. “At first, our analytical reporting was incredibly slow,” said Toups.
“When we first started implementing Looker, a couple of years ago, we did a traditional ETL ‘lift and load’ into a MySQL reporting warehouse,” reported Toups. “This resulted in about 600GB of data. Now, on SingleStore, I use columnstore with compression and SingleStore’s native sharding.”
“The underlying data includes measurements such as systolic and diastolic blood pressure, along with heartbeat and respiration and a variety of medical history information, all of which are strongly correlated to one another,” Toups continued. “I use associative techniques to compress that 600GB down to a 20GB dataset. Simply unprecedented compression. We always had similarity on our data, but we couldn’t take advantage of it in the MySQL environment.”
“It was taking 30 or 40 minutes to generate retrospective quality data on the MySQL platform,” Toups said. “Because we couldn’t easily cache the entire data set in memory on MySQL, it was constant disk thrashing. But on SingleStore, the same analysis runs in less than a minute, and most queries return in under a second.”
Toups summed up Medaxion’s progress: “We replaced the middle of the analytics chain with SingleStore. Looker is the body of the rocket ship that carries the information my customers – and their patients – need. SingleStore is the engine of that rocket ship.”
Read Post
Data Intensity
Case Study: Emmy-Winning SSIMWAVE Chooses SingleStore for Scalability, Performance, and More
SSIMWAVE customers – from film producers to network engineers to media business executives – work to some of the highest standards in the world. They demand to work with the best. SSIMWAVE also works at that level, as the company’s 2015 Emmy award for engineering achievement demonstrates. They also ask the same high standards of their technology vendors/partners. For SSIMWAVE’s rather comprehensive analytics needs, only one database makes the grade: SingleStore.
SSIMWAVE has unique technology and unique analytics needs. SSIMWAVE mimics the human visual system, enabling the software to quantify the quality of video streams, as perceived by viewers, into a single viewer score. Video delivery systems can then be architected, engineered, and configured to manage against this score. This score correlates strongly to what actual human beings would perceive the video quality to be. This allows SSIMWAVE users to make informed trade-offs among resources and perceived quality, automatically or manually, and all in real time.
SSIMWAVE Cracks the Code
According to Cisco, video data accounted for 73 percent of Internet traffic in 2017, a share that is projected to grow to 82 percent by 2022. Maximizing the quality of this video content, with the least bandwidth usage and at the lowest cost possible, is one of the most important engineering, business, and user experience issues in the online world.
The barrier to balancing video quality against compression has been that only human beings could accurately assess the quality of a given video segment when it was compressed, then displayed on different devices. Further complicating the picture (no pun intended) is the fact that people, when asked to rate video quality, give different answers with varying levels of consistency over time. This has meant that a panel of several people was needed to render a useful assessment. As a result, a software engineer or operations person wanting to process and deliver data within acceptable levels didn’t have a reliable, affordable method for knowing how much was just enough, without serious compromise to the viewer’s experience.
Read Post
Company
Community Stars Light Up SingleStore Forums
The SingleStore Forums are seeing more and more community contributions – not just timely and incisive questions, but answers from the community, and valuable content contributions as well. We have two new community stars for the summer, and some valuable Q&As flying back and forth.
What Makes an Online Community Work?
Participation in any online community is mostly optional. People always have a lot of ways they can spend their time, so for an online community to take off, it has to offer a lot to people.
So it’s notable that the SingleStore Forums are seeing more and more answered questions, and solid contributions, from customers and users – while SingleStore employees continue to help out as well.
The Forums are also important to a key SingleStore initiative, offering free use of SingleStore for small and, in our humble opinion, even medium-sized deployments. Only Enterprise users have direct access to SingleStore’s highly regarded, responsive support team, so the Forums are a crucial source of help.
Summer Community Stars
The first Community Star, named in June, and featured in a previous blog post, was Ziv Meidav. Now we have a second, and a third Community Star.
The July Community Star, Brandon Vincent, has all the tools needed to play the game, as they say in baseball. Not only does he dig in to help other users with complex technical questions; he recently posted an important piece of documentation, the excellent Columnstore Key Guidelines. Covering both shard keys and columnstore table keys, the Guidelines have a lot of important answers – and a couple of in-depth questions as well.
The August Community Star, Mani Gandhi, is part of the broader SingleStore community, out beyond the Community Forums. Mani has amassed more than 14,000 karma on Hacker News, a good chunk of that while making insightful comments about SingleStore. (Do go read his posts, but don’t upvote him because of this reference, as Hacker News frowns on that.)
Fun on the Forums
There’s a lot of serious stuff on the Forums, of course – discussions of potential errors in a SingleStore Pipeline, a question about GROUP_CONCAT sorting (coming soon, in SingleStoreDB Self-Managed 7.0), and a question and answer about unwanted increases in memory usage.
Read Post
Product
CREATE PIPELINE: Real-Time Streaming and Exactly-Once Semantics with Kafka
In this presentation, recorded shortly after SingleStore introduced SingleStore Pipelines, two SingleStore engineers describe SingleStore’s underlying architecture and how it matches up perfectly to Kafka, including in the areas of scalability and exactly-once updates. The discussion includes specific SQL commands used to interface SingleStore to Kafka, unleashing a great deal of processing power from both technologies. In the video, the SingleStore people go on to describe how to try this on your own laptop, with free-to-use SingleStoreDB Self-Managed.
Introduction to SingleStore: Carl Sverre
I want to start with a question. What is SingleStore? It’s really important to understand the underpinnings of what makes Pipeline so great, which is our SingleStore distributed SQL engine.
There are three main areas of SingleStore that I want to talk about really briefly. The first area of SingleStore is that we’re a scalable SQL database. So if you’re familiar with MySQL, Postgres, Oracle, SQL Server, a lot of our really awesome competitors, we are really similar. If you’re used to their syntax, you can get up and running with SingleStore really easily, especially if you use MySQL. We actually have followed their syntax very similarly, and so if you already used MySQL, you can pretty much drop in SingleStore in place, and it just works.
Read Post
Case Studies
Why Do Banks Need Real-Time Transaction Processing?
A new report from RT Insights describes the benefits of real-time transaction processing in banking and financial services and shows how traditional database architectures interfere with real-time data movement. In order to get the benefits of real-time transaction processing, such as improved portfolio management, fast credit card fraud and acceptance checks, and others, banks and other financial services institutions need to use a translytical database, combining the best of transaction and analytical data processing capabilities in a single, fast, scalable system.
A Real-Time Database for Banking
What is a real-time database? And why would you need one for banking and financial services companies?
A real-time database is a database that can support real-time processing. According to Wikipedia (as of the publication date), “Real-time processing means that a transaction is processed fast enough for the result to come back and be acted on right away.” That certainly sounds like something banks could use – when you go to the ATM machine, or use a credit card, or apply for a home loan, you certainly want the systems you’re using to return the right answers, right away. (These functions are also good examples of the use of machine learning in financial services, another SingleStore specialty.)
Indeed, accounting and banking are two of the areas where real-time databases are said to be most useful. The RTi report cites many important applications for “faster and more intelligent decision-making”: fraud monitoring; dynamic portfolio analysis; regulatory compliance; and protection from cyberthreats.
Read Post
Case Studies
Case Study: Fraud Detection “On the Swipe” For a Major US Bank
This case study was originally presented as part of a webinar session by Mike Boyarski, Sr. Director of Product Marketing at SingleStore. It’s been updated to include additional information and references. In the webinar, which you can view here, and access the slides here, Mike describes the challenges facing financial services institutions which have decades’ worth of accumulated technology solutions – and which need to evolve their infrastructure immediately to meet today’s needs. In this case study, which was also described in the webinar, Mike shows how a major US bank created a new streaming data architecture with SingleStore at its core. Using SingleStore enabled them to move from overnight, batch fraud detection to fraud detection “on the swipe,” applying machine learning models in real time. He presents a reference architecture that can be used for similar use cases, in financial services and beyond.
This case study presents a reference architecture that can be used by leading retail banks and credit card issuers to fight fraud in real time, or adapted for many other real-time analytics use cases as well. In addition, it describes how SingleStore gives fraud detection services an edge by delivering a high-performing data platform that enables faster ingest, real-time scoring, and rapid response to a broader set of events. A similar architecture is being used by other SingleStore customers in financial services, as described in our Areeba case study.
This case study was originally presented as part of our webinar series, How Data Innovation is Transforming Banking (click the link to access the entire series of webinars and slides). This series includes several webinars, described in these three blog posts:
Real-Time Fraud Detection for an Improved Customer ExperienceProviding Better Wealth Management with Real-Time DataModernizing Portfolio Analytics for Reduced Risk and Better Performance
Also included are these two case studies:
Replacing Exadata with SingleStore to Power Portfolio AnalyticsMachine Learning and Fraud Detection “On the Swipe” For Major US Bank (this case study)
You can also read about SingleStore’s work in financial services – including use cases and reference architectures that are applicable across industries – in SingleStore’s Financial Services Solutions Guide. If you’d like to request a printed and bound copy, contact SingleStore.
Real-Time Fraud Case Study
This application is a credit card solution. The SingleStore customer was looking to deliver a high-performance, agile fraud detection platform using standard SQL, with challenging performance requirements. And so I’ll talk about what that means around agility and some of the sort of performance demands they have.
The customer has a time budget of one second from the time the card is swiped to the approval or refusal. There’s a very sort of sophisticated set of queries that need to be run in a very short window of time. They have about a 50 millisecond budget to work with to run a number of queries. In this application they are looking at about a 70-value feature record. And so we’ll spend a little bit of time on how that looks.
Read Post
Data Intensity
Case Study: Replacing Exadata with SingleStore to Power Portfolio Analytics and Machine Learning
This case study was presented as part of a webinar session by Rick Negrin, VP of Product Management at SingleStore. In the webinar, which you can view here, and access the slides here, Rick demonstrates how a major financial services company replaced Oracle Exadata with SingleStore to power portfolio analytics, with greatly increased responsiveness for users and the ability to easily incorporate machine learning models into their applications. In this case study, we’ll emphasize the bank’s digital infrastructure using Exadata, then present their implementation of SingleStore as a reference architecture that you can consider for your own organization’s needs.
This case study was originally presented as part of our webinar series, How Data Innovation is Transforming Banking (click the link to access the entire series of webinars and slides). This series includes several webinars, described in these three blog posts:
Real-Time Fraud Detection for an Improved Customer ExperienceProviding Better Wealth Management with Real-Time DataModernizing Portfolio Analytics for Reduced Risk and Better Performance
Also included are these two case studies:
Replacing Exadata with SingleStore to Power Portfolio Analytics (this case study)Machine Learning and Fraud Detection “On the Swipe” For Major US Bank
You can also read about SingleStore’s work in financial services – including use cases and reference architectures that are applicable across industries – in SingleStore’s Financial Services Solutions Guide. If you’d like to request a printed and bound copy, contact SingleStore.
Case Study, Before: The Previous Architecture, with ETL, Exadata, and RAC
This case study describes an asset management company. It’s a fairly large asset management company, with about a thousand employees and probably just under a half a trillion in assets under management. They have been in business for several decades. They’ve invested heavily in a lot of different technologies, primarily in some legacy database technologies. Things were working pretty well for awhile, but as new requirements started to come in and more users are using the system, they’ve started running into trouble.
So this is what their architecture looked like. And it should be fairly familiar to most of you. They have a variety of data sources, obviously their own internal operational systems, mostly legacy databases. Combined with some external third-party data and partner data that they would bring in. As well as behavioral data from how their users are using the system, both on the web and with mobile. And all of that data was moved, via standard extract, transform, and load (ETL) processes, into a traditional data warehouse.
Read Post
Data Intensity
Webinar: Modernizing Portfolio Analytics for Reduced Risk and Better Performance
In this webinar Rick Negrin, Product Management VP at SingleStore, describes the importance of portfolio analytics, enhanced by machine learning models, to financial services institutions – helping them to meet customer needs and edge out competitors. He shows how SingleStore speeds up portfolio analytics at scale, with unmatched support for large numbers of simultaneous users – whether connecting via ad hoc SQL queries, business intelligence tools, apps, or machine learning models. You can view the recorded webinar and download the slides. He also describes how a major US financial services institutions implemented Kafka, Spark, and SingleStore, replacing Oracle and widespread use of cumbersome extract, transform, and load (ETL) routines, in this separate case study.
The business problem Rick discusses is the need to modernize portfolio analytics for reduced risk and better performance – both for the customer managing their portfolio, and for the institution offering portfolio management tools to customers. Institutional investors want smarter portfolio management services that deliver optimal returns while reducing their exposure to any one industry, currency, or other specific source of risk.
Portfolio managers want guided insights to help them avoid sudden or dramatic rebalancing of funds that can drive up costs and reduce confidence and customer loyalty. SingleStore powers a number of portfolio dashboards and what-if analysis, leveraging live market data for the most up-to-date view of the market. The separate case study shows how a major financial services company used SingleStore to solve these problems, supporting their leadership position in the market.
This webinar was originally presented as part of our webinar series, How Data Innovation is Transforming Banking (click the link to access the entire series of webinars and slides). This series includes several webinars, described in these three blog posts:
Real-Time Fraud Detection for an Improved Customer ExperienceProviding Better Wealth Management with Real-Time DataModernizing Portfolio Analytics for Reduced Risk and Better Performance (this webinar)
Also included are these two case studies:
Replacing Exadata with SingleStore to Power Portfolio AnalyticsMachine Learning and Fraud Detection “On the Swipe” For Major US Bank
You can also read about SingleStore’s work in financial services – including use cases and reference architectures that are applicable across industries – in SingleStore’s Financial Services Solutions Guide. If you’d like to request a printed and bound copy, contact SingleStore.
The Role of the Database in Digital Transformation
Digital transformation remains the top priority by far for banks. This is confirmed by a Gartner study from 2019, but we at SingleStore hear it anecdotally in all the conversations that we have with financial institutions. This is because of the opportunity that digital transformation provides. When you to take advantage of new technologies, you can create new sources of revenue, and you can drive down your costs with new operating models, allowing you to deliver digital products and services that just weren’t possible before.
To make this happen, you need to have an architecture and operating platform that supports a new set of requirements. One need is to drive down latency: the time from when a new piece of information is born to the time you’re able to gain insight and take action on it. The effort is to get that as close to zero as possible.
When you do that, you can make faster data-driven actions in your business. So when something’s going on in the financial markets, the customer wants to understand it, to know what’s going on, as quickly as possible. And to be able to take action on it, in order to either reduce risk in a portfolio or perhaps take advantage of some new opportunity that’s come up.
Read Post
