Product
Deploy SingleStore with Mesosphere
SingleStore is now certified for deployment on the Mesosphere DCOS (Datacenter Operating System). With a few simple commands, users can launch a SingleStore cluster on DCOS.
The Mesosphere DCOS (Datacenter Operating System) is a commercially supported product built on top of Apache Mesos that serves as a datacenter abstraction layer. DCOS deploys distributed applications with a few simple command-line interface (CLI) steps, and handles application provisioning, resource management, and fault tolerance seamlessly. It also centralizes management of a distributed system in a single interface for easy deployment.
SingleStore is a relational, in-memory database optimized for multi-machine deployment. Its distributed nature makes it a good match for DCOS.
The package needed to deploy SingleStore on DCOS can be found in the Mesosphere Universe repository, which allows all DCOS users to install SingleStore anytime simply by running `dcos package install memsql` from a DCOS console. SingleStore leverages the Marathon scheduler to pull a SingleStore Docker image to deploy the SingleStore software, which is tightly integrated with DCOS.
SingleStore supports a variety of deployment topologies, and the Mesosphere DCOS adds another option to this growing list. SingleStore can be deployed on:
bare metal machines in your datacentervirtual machines in your datacenterDocker containersAmazon EC2, Microsoft Azure and other public cloud platform providersMesosphere DCOS
Additional Links
Read the full press release here: Mesosphere Launches Developer Program, VIP Partner Program and SDK for Building Distributed Datacenter-Scale Services on Mesosphere DCOS
Try SingleStore today! Run ``dcos package install memsql`` on your DCOS system, or download the free SingleStore Community Edition with unlimited scale and capacity.
Read Post
Product
New SingleStore Ops Comes with Database Speed Test
SingleStore Ops 4.0.31 is now available for download! For this release, we have maintained a focus on providing the smoothest possible on-ramp for developers to quickly get productive with SingleStore. We believe enterprise-class software should be easy to install or use.
Here are the new features of the latest SingleStore Ops release, available today. Note that aside from features, this release includes bug fixes, and is a free upgrade for existing SingleStore Ops users.
Read Post
Product
SingleStoreDB Self-Managed 4: Market and Strategy CEO Q&A
SingleStoreDB Self-Managed 4 represents a leap forward in relational database innovation with new capabilities such as real-time geospatial intelligence and high-throughput connectivity to Apache Spark. It also includes a free forever, unlimited capacity Community Edition, enhancements to the Optimizer for distributed joins, and a new version of the SingleStore Ops management framework.
Start Using SingleStore Community Edition Now
Unlimited scale and capacity. Free forever.
Download Community Edition →
As Big Data abounds, understanding every company and product can be tricky. To make easier, here are a few questions and answers about SingleStore.
The database landscape is big. Where does SingleStore fit?
SingleStore is the leading database for real-time transactions and analytics. That means we’re operational by nature, much more akin to SQL Server, Oracle, or SAP HANA than Hadoop. SingleStore is also:
In-Memory
Providing the utmost performance for today’s demanding workloadsDistributed
Enabling cost-effective, horizontal scale-out on-premises or in the cloudRelational and multi-model
Allowing companies to use in-house SQL tools, applications, and knowledgeWith JSON and Geospatial data formats supportedSoftware
Designed to run on commodity hardware for costs savings
You talk about transactions and analytics. Can you explain in more detail?
To meet real-time demands, companies must be able to capture information across millions to hundreds of millions of sensors or mobile applications. They also want to analyze that data up to the last transaction. So with real-time operations you don’t have the luxury or the pain of ETL. You need to bypass ETL by transacting and analyzing in a single database designed to support these concurrent workloads. It boils down to analytics on changing datasets, today’s critical capability.
SingleStoreDB Self-Managed 4 includes the SingleStore Spark Connector and the SingleStore Loader for HDFS and S3. How should we think about SingleStore with Spark and Hadoop?
SingleStore and Spark work well together as they both have memory-optimized, distributed frameworks. Spark is a processing framework that enables real-time transformation and advanced analytics, but Spark itself does not have a storage or persistence ability. By storing data permanently in SingleStore, customers get an easy way to build operational applications, and the ability to take operational data and ‘round-trip’ it to Spark for advanced analytics.
With Hadoop, customers frequently build simplified Lambda architectures using SingleStore. All data can go directly to HDFS for long term archiving. Simultaneously, data can go directly into SingleStore, bypassing Hadoop for the real-time path. Should historical data be needed for analysis, SingleStore can import that data from HDFS using the SingleStore Loader.
Many folks say not everything needs to be in-memory. How do you respond?
We agree! While in-memory computing remains critical for many applications, the pace of data growth still eclipses memory-only solutions. This is exactly why SingleStore ships with an integrated column-store optimized for disk and flash. Now customers can create tables entirely in-memory, or across a combination of memory and disk and flash. And starting with SingleStoreDB Self-Managed 4, we license the software based on DRAM capacity so the use of disk or flash storage in the column store is unlimited at no additional cost.
We are big believers that customers will see the benefit of placing data in a structured format from the beginning, and the SingleStore column store will let them do that affordably.
You have always focused on SQL. All the while NoSQL has received a lot of discussion. Where is SingleStore in this?
SQL is the lingua franca of the data processing world, having been well adopted over its decades-long history. Companies can build SQL applications quickly, and then immediately use in house analytics tools and practices to derive insights.
Prevailing wisdom used to be that SQL could not scale. That is untrue and companies are now discovering the powerful combination SingleStore delivers of a relational, in-memory, distributed database with speed, scale, and simplicity.
The future is multi-model and SingleStore also includes JSON and Geospatial datatypes to fulfill this promise.
The Community Edition is freely available but not open-source. What strategy is SingleStore pursuing?
Our strategy is to build a scalable software business. This means running on all the platforms customers want to use, including public and private clouds, in the way they want to use them. Community Edition is a way to allow more people to use SingleStore without limitations on time or capacity.
There is a wide spectrum of commercial database offerings. On one extreme there are proprietary hardware / software combinations. Our belief is that proprietary hardware is a down elevator. On the other end of the spectrum there are open-source projects with businesses built around them. But that is essentially selling consulting hours, not software. There are many successful businesses in the middle of the spectrum, such as MySQL’s dual commercial / open-source licensing, and we maintain open-source projects, including SingleStore Loader for Hadoop and S3, a large number of general-purpose Python packages, and the SingleStore Spark Connector.
At SingleStore, we want to offer performance that trumps legacy vendors, deliver it a fraction of the cost, and provide complete deployment choice across hardware, data centers, and clouds. In doing so we hope to help more companies achieve their goals to become real-time enterprises.
Read Post
Product
Launching Our Community Edition
We started SingleStore with a vision to enable every company to be a real-time enterprise, and to do that by delivering the leading real-time database for transactions and analytics. Since then, the forces shaping our digital economy have only added wind to our sails. The world is more mobile, interconnected, interactive, and on the cusp of several industry transformations such as the Internet of Things.
Real-time processing is the secret to keeping up, and in-memory solutions are the foundation. Yet existing options have been too expensive or too complex for companies to adopt.
That changes today with the release of SingleStoreDB Self-Managed 4 and our new Community Edition, a free unlimited capacity, unlimited scale offering of SingleStore that includes all transactional and analytical features.
By sharing SingleStore capabilities with the world, for free, we expect many developers and companies will have a chance to explore what is possible with in-memory computing. As the pace of business advances, SingleStore will be there.
Start Using SingleStore Community Edition Now
Unlimited scale and capacity. Free forever.
Download Community Edition →
We hope you enjoy working with our Community Edition. Please feel free to share feedback at our Community Edition Help page.
Eric Frenkiel, CEO and co-founder, SingleStore
Nikita Shamgunov, CTO and co-founder, SingleStore
Community FAQ
Read Post
Product
Celebrating SingleStore Availability Two Years In
Today, I couldn’t be more excited to mark the two year anniversary of SingleStore general availability! SingleStore began with a simple idea to build a new database that would give any company the ability to operate in real-time and make their business more data-driven, responsive, and scalable. Since releasing SingleStore, it’s been an amazingly fun journey as the company has grown by leaps and bounds every quarter.
To celebrate our second birthday, I wanted to take a brief moment to reflect on what we’ve been able to accomplish in the two years since releasing SingleStore.
People
SingleStore started in the Y-Combinator winter class of 2011 with just two people – co-founder and CTO Nikita Shamgunov, and myself. Since then, we’ve grown the company to more than 50 people who bring great experience, energy, and passion to the company. We’ve also added database visionaries like Jerry Held and Chris Fry to our executive team to help us see our vision come to fruition.
Customers
We’ve added 40+ enterprise customers over the past 2 years, including top brands like Comcast, Samsung and Shutterstock. It’s been incredibly rewarding to see our customers use SingleStore in ways we never imagined, truly pushing the boundaries of what is possible in Big Data.
Product
Since launching with general availability in 2013, we’ve expanded the SingleStore platform to scale with growing market demand. Major additions to the platform include:
Going beyond memory by including a flash-optimized column store that is closely integrated with the in-memory row store to provide a single database for real-time and historical analyticsWorking with Apache Spark by shipping a bi-directional connector to operationalize Spark models and resultsIncorporating real-time geospatial intelligence to help customers build location-aware applications and analytics
What’s Next?
The most exciting times are still ahead!
Big data has been traditionally thought of as a mechanism for extracting insights from yesterday’s data. We seek to change that way of thinking, empowering businesses to be more responsive by operating with real-time data in the here and now. As demand for real-time and in-memory databases increases, we plan to be there helping customers achieve phenomenal results.
Read Post
Product
Harnessing the Enterprise Capabilities of Spark
As more developers and data scientists try Apache Spark, they ask questions about persistence, transactions and mutable data, and how to deploy statistical models in production. To address some of these questions, our CEO Eric Frenkiel recently wrote an article for Data Informed explaining key use cases integrating SingleStore and Spark together to drive concrete business value.
The article explains how you can combine SingleStore and Spark for applications like stream processing, advanced analytics, and feeding the results of analytics back into operational systems to increase efficiency and revenue. As distributed systems with speedy in-memory processing, SingleStore and Spark naturally complement one another and form the backbone of a flexible, versatile real-time data pipeline.
Read the full article here.
Get The SingleStore Spark Connector Guide
The 79 page guide covers how to design, build, and deploy Spark applications using the SingleStore Spark Connector. Inside, you will find code samples to help you get started and performance recommendations for your production-ready Apache Spark and SingleStore implementations.
Download Here
Read Post
Product
Operationalizing Spark with SingleStore
Combining the data processing prowess of Spark with a real-time database for transactions and analytics, where both are memory-optimized and distributed, leads to powerful new business use cases. SingleStore Spark Connector links at end of this post.
Data Appetite and Evolution
Our generation of, and appetite for, data continues unabated. This drives a critical need for tools to quickly process and transform data. Apache Spark, the new memory-optimized data processing framework, fills this gap by combining performance, a concise programming interface, and easy Hadoop integration, all leading to its rapid popularity.
However, Spark itself does not store data outside of processing operations. That explains that while a recent survey of over 2000 developers chose Spark to replace MapReduce, 62% still load data to Spark with the Hadoop Distributed File System and there is a forthcoming Tachyon memory-centric distributed file system that can be used as storage for Spark.
But what if we could tie Spark’s intuitive, concise, expressive programming capabilities closer to the databases that power our businesses? That opportunity lies in operationalizing Spark deployments, combining the rich advanced analytics of Spark with transactional systems-of-record.
Introducing the SingleStore Spark Connector
Meeting enterprise needs to deploy and make use of Spark, SingleStore introduced the SingleStore Spark Connector for high-throughput, bi-directional data transfer between a Spark cluster and a SingleStore cluster. Since Spark and SingleStore are both memory-optimized, distributed systems, the SingleStore Spark Connector benefits from cluster-wide parallelization for maximum performance and minimal transfer time. The SingleStore Spark Connector is available as open source on Github.
SingleStore Spark Connector Architecture
There are two main components of the SingleStore Spark Connector that allow Spark to query from and write to SingleStore.
A `SingleStoreRDD` class for loading data from a SingleStore queryA `saveToSingleStore` function for persisting results to a SingleStore table
Read Post
Product
Run Real-Time Applications with Spark and the SingleStore Spark Connector
Apache Spark is one of the most powerful distributed computing frameworks available today. Its combination of fast, in-memory computing with an architecture that’s easy to understand has made it popular for users working with huge amounts of data.
While Spark shines at operating on large datasets, it still requires a solution for data persistence. HDFS is a common choice, but while it integrates well with Spark, its disk-based nature can impact performance in real-time applications (e.g. applications built with the Spark Streaming libraries). Also, Spark does not have a native capability to commit transactions.
Making Spark Even Better
That’s why SingleStore is releasing the SingleStore Spark connector, which gives users the ability to read and write data between SingleStore and Spark. SingleStore is a natural fit for Spark because it can easily handle the high rate of inserts and reads that Spark often requires, while also having enough space for all of the data that Spark can create.
Read Post
Product
Full Speed Ahead
Nearly ten years ago I received a phone call about a startup in Silicon Valley solving application performance problems with memory.
From my data center infrastructure experience, I knew the days of mechanical disk drives were limited. I had to get on the memory train, so I went.
That experience led me to meet the co-founders of Fusion-io and ultimately join them in 2010. When Fusion-io went public in 2011 revenue was on its way from $36 million to $197 million annually. The time was right for flash memory and Fusion-io had the products to deliver.
Companies like Facebook and others jumped at the opportunity to supercharge their databases and infrastructure, going so far as to deploy all solid-state data centers to meet the needs of a globally connected population interacting with data and images around the clock.
During the next several years I watched customers deploy solutions for Oracle, SAP HANA, Microsoft SQL Server, and MySQL to achieve great results. Ultimately however, these solutions remained available to only a portion of the population. They were driven by adding remarkable hardware to good software.
What if we could change the equation so customers could work with remarkable software and hardware that didn’t break the bank?
My excitement for SingleStore comes from this very premise. Far beyond making the databases of yesteryear look good, SingleStore has rethought the database itself.
With a ground-up focus on memory and DRAM, distributed systems, and the ability to deploy anywhere from bare metal to a cloud container or VM, SingleStore has designed a product for today’s interconnected and interactive world. It scales out, handles the most torturous workloads without breaking a sweat, delivers analytics in the midst of massive data capture, and preserves the SQL goodness that has served as the enterprise analytics lingua franca for decades.
The SingleStore team has made phenomenal progress in the last few years, delivering a solid product with incredible market opportunity. A real-time world awaits as we experience the growth of data, applications, and touch points in our daily lives.
The SingleStore path is flanked by customers generating new revenue, driving down solution costs, and innovating with data-driven solutions in ways that had not been possible before.
I’m thrilled to be a part of it!
Read Post
Engineering
SingleStore and Cisco Work Together to Make Real-Time Performance on Hadoop a Reality
While Hadoop is great for storing large volumes of data, it’s too slow for building real-time applications. However, our recent collaboration with Cisco provides a solution for Hadoop users who want a better way of processing real-time data. Using Cisco’s Application Centric Infrastructure including APIC and Nexus switch technology, we’ve been able to demonstrate exceptional throughput on concurrent SingleStore and Hadoop 2.0 workloads.
Here’s How It Works
Cisco’s new networking technology automatically prioritizes smaller packet streams generated by real-time workloads over the larger packet streams typically generated by Hadoop. This enables impressive throughput on clusters running simultaneous SingleStore and Hadoop workloads.
At the Strata + Hadoop conference last week in New York, Cisco demonstrated the solution on an 80 node cluster running both SingleStore and Hadoop. Without additional network traffic, the cluster can serve 2.4 million reads per second from SingleStore’s in-memory database. Without packet-prioritization, the database’s performance drops to under 600 thousand reads per second when a simulated Hadoop workload is added to saturate the cluster’s network. With packet-prioritization, the performance recovers to 1.4 million reads per second, more than doubling the throughput.
Why Does It Matter?
This advance provides the ability to collocate SingleStore, for real-time, mission critical data ingest and analysis, with Hadoop workloads that are less time-sensitive and executed as large batch jobs on historical data.
By combining Hadoop’s storage infrastructure with SingleStore’s real-time data processing ability, businesses get the best of both worlds: real-time analytics with Hadoop scale workloads. As an added bonus, the solution allows businesses to save on hardware costs by running SingleStore and Hadoop together on the same cluster.
If you want to learn more, contact a SingleStore representative at sales@singlestore.com or at (855) 463-7660.
Read Post
Product
SingleStore Announces Strategic Investment from In-Q-Tel
Today SingleStore is excited to announce an investment from In-Q-Tel (IQT), the strategic investment firm that identifies innovative technology solutions to support the missions of the U.S. Intelligence Community. SingleStore is used by enterprise companies such as Comcast, Shutterstock, and Zynga, and IQT’s recent investment will allow us to generate market opportunities within the government space. The investment from IQT further validates SingleStore’s leadership in the distributed in-memory database market, and we are pleased to be able to work with IQT to help bring deep insights to the U.S. Intelligence Community. Read the full press release on SingleStore’s website.
Read Post
Product
SingleStoreDB Self-Managed 2.5 Ships Today with JSON Datatype, Online Alter Table, and More
We are excited to announce that SingleStoreDB Self-Managed 2.5 is now available for download. Of the many new features and performance improvements, one of the most exciting is support for JSON analytics. With native support for the JSON datatype, SingleStore delivers a consolidated view across structured and semi-structured data in real-time.
Read Post
Product
Unsexy Database Features That Matter: Part 2
In the previous post, we talked about how you should focus on how people derive value from your product. Today we will focus on database management — your goal should always be 24/7 100% uptime.
How People Manage A Database Product
Deployment (time to get the system up and running)
How much time does it take to get your product up and running? People have talked about barriers to entry for a long long time. The worst one is, “I couldn’t install your product so I gave up.” This is simply unforgivable.
Modern platforms offer wonderful turn-key deployment tools: rpm/deb for linux, virtual machine images and AMIs, Amazon Marketplace, cloud formation, docker containers. Pick your poison and do it right.
Security
Enterprise demands high security products. Kerberos authentication, password management policies. It’s unsexy, it boring, but it matters. Without it you can’t get into some of the most important enterprise accounts.
Online Operations
You would think that every database has a backup story, until you add one word to it: online. Online means that the backup doesn’t takes read or write locks for the duration of the backup and that the database state is consistent with regard to a specific point in time. Of course commercial databases have had this feature for a long time and some of the open source counterparts (Postgres) have it as well, but even MySQL and MongoDB don’t have a turn-key solution for this.
It’s relatively easy to have offline alter table that blocks all the reads and writes to the database. Online alter table is very hard to build, but it allows production systems to have zero downtime in the face of schema changes. We bit the bullet and built it. Needless to say, it’s a tremendous customer delight.
High Availability
Commodity hardware tends to fail. Non-commodity hardware tends to fail just as much. Your system must survive that with minimum downtime. As usual, it’s easier said than done, but you must have it, otherwise no one would want to run it in production.
Conclusion
When you build a database server, it’s easy to get carried away with sexy things like exotic data structures, trendy new distributed designs, new query languages, etc. But you have to always invest in fundamentals just as much as you are investing in what makes your product unique and special to your customers.
Read Post
Product
Unsexy Database Features That Matter: Part 1
As enterprise customers start to use your product in many different ways, new feature requests will start to flow into the product team. These features aren’t sexy features by any means, but every enterprise customer cares about them deeply, and we try to include as many customer feature requests as possible into every release, sexy or not.
To really understand these unsexy features, we must examine two areas:
How users derive value out of your product.
How users manage a database product.
In this two-part blog series, we will explore these topics and why they matter in today’s database market.
Part 1: How People Derive Value
Tools Compatibility
At the end of the day, people connect to a database via a suite of tools and apps such as SequalPro, Tableau, Looker, ORMs, database explorers, etc. Ideally, database developers are able to use existing drivers/network protocols that people already understand.
Query Surface Area and Query Optimizer
Super successful products “just work.” A major component of “just working” is having SQL query surface area that supports everything an unsophisticated user can throw at it (e.g. full set of data types, builtin function, subqueries, etc). This can be a hard pill to swallow, but it’s also what makes your business defendable from the barrier to entry standpoint.
This also applies to a query optimizer (QO). The “just works” idea goes out the window if the QO chooses a disastrously slow execution plan. In a way, the job of a QO is to avoid disastrously slow plans rather than finding perfect execution plans. Building a great QO is a hard problem, and SingleStore is constantly improving our QO in every release to ensure that queries run as efficiently as possible. And by the way, from the engineering perspective building a QO is quite a sexy project.
Data Loading
I guarantee that anyone who deals with a lot of data has one major pain: ETL. ETL sucks! You are moving data across different systems and it’s never smooth or fast. Date formats are different, data types don’t perfectly match, NULLs, character escaping, etc. It is all the little things that start to get really annoying.
SingleStore supports a lot of ways of getting data into the database. Obviously we support CSV load via LOAD DATA, as well as a variety of other ways to push data into SingleStore, including out-of-the-box streaming of data from collectd, and statsd to enable machine data performance analytics. Our goal is to help customers avoid ETL as much as possible.
The Little Things
Working on error messaging, language support, character encodings, data type conversions is brutal, as it requires tremendous attention to detail. We write a ton of tests and churn a lot of code that is not groundbreaking, but it has to be done. This improves the lives of our customers and lets them iterate on their applications faster. Precise error messages make application development easier. Character encoding support broadens the reach of the product.
The little things provide what we call a “delight” moment when working with SingleStore is easy and intuitive.
Stay tuned for part two as we talk more about how people manage their databases.
Read Post
