Author
Ankur Goyal
Vice President of Engineering
Data Intensity
Build Real-Time Data Pipelines with SingleStore Streamliner
SingleStore Streamliner is now generally available! Streamliner is an integrated SingleStore and Apache Spark solution for streaming data from real-time data sources, such as sensors, IoT devices, transactions, application data and logs.
The SingleStore database pairs perfectly with Apache Spark out-of-the-box. Apache Spark is a distributed, in-memory data processing framework that provides programmatic libraries for users to work with data across a broad set of use cases, including streaming, machine learning, and graph data processing. SingleStore and Spark share many design principles: they are in-memory, distributed, and data-centric. Spark provides an amazing interface to the unique functionality in SingleStore: fast and durable transactions, a real-time hybrid row/column-oriented analytics engine, and a highly concurrent environment for serving complex SQL queries.
The SingleStore Spark Connector, released earlier this year, allows Spark and SingleStore integration, facilitating bi-directional data movement between Spark and SingleStore. The connector generated a lot of interest from users who saw the benefits of using Spark for data transformation and SingleStore for data persistence. A consistent theme in the use cases we saw was the desire to use Spark to stream data into SingleStore with Spark Streaming. SingleStore Streamliner is the result of our work to productize this workflow into an easy, UI-driven tool that makes this process dead simple.
Let’s review the thinking behind some of the decisions we made as we were developing Streamliner.
Early work with Pinterest
Pinterest showcased a Kafka+Spark+SingleStore solution in Strata+Hadoop World last February, which was a collaborative effort with SingleStore. See the Pinterest blog post and the Pinterest demo to learn more. The Pinterest solution leveraged Spark Streaming to quickly ingest and enrich data from Kafka, and then store it in SingleStore for analysis.
Read Post
Engineering
How We Hire Remarkable Engineers
Read Post
Engineering
Seven Databases in Seven Weeks: SingleStore at CMU
Last week I gave a presentation as part of the “Seven Databases in Seven Weeks” seminar series designed to showcase new technologies available for front-end application developers. Hosted by Carnegie Mellon University, the series features leading developers of NoSQL and NewSQL database management systems.
Watch the presentation below to learn about Hybrid Transactional and Analytical Processing (HTAP) and SingleStore’s key innovations that enable us to deliver on being a powerful, in-memory, HTAP-driven database.
Read Post
