Jumpstart SingleStore Kai™ With MongoDB® Replication Pipelines

MongoDB®, a robust transactional database, has long been favored for its flexibility.

However, when it comes to running analytics or aggregations on vast volumes of data, it faces inherent limitations. This is why we introduced SingleStore Kai™, a fast, easy and powerful API to drive up to 100x faster analytics on your MongoDB® applications — without any query changes, application migration or data transformations.*

The beauty of SingleStore Kai lies in its simplicity. Just update your connection string and harness the power of SingleStoreDB with your existing MongoDB® tools and drivers.

But to fully experience Kai's capabilities with your existing MongoDB® data, a crucial step is required: how to get data from MongoDB® into SingleStoreDB.

Setting up a data replication pipeline between MongoDB® and SingleStoreDB to keep both data stores in perfect sync can be fraught with challenges. From managing infrastructure and associated costs to handling the inherent complexity of the tooling, the traditional approach can be a cumbersome ordeal.

Moreover, when it comes to enabling real-time analytics, you cannot afford latency in data synchronization from MongoDB® to SingleStoreDB.

This is why we are happy to introduce the public preview of our groundbreaking native replication for applications built on MongoDB®, powered by SingleStore Pipelines. There are several advantages of this new powerful capability:

• Lightning-fast ingestion. With SingleStore Pipelines seamlessly integrated into the database engine, you’ll experience fast, reliable data ingestion.
• No additional cost. Experience seamless data replication between MongoDB® and SingleStoreDB, without incurring extra costs.
• Effortless setup. Whether you prefer SQL or MongoDB® commands, the setup process is a breeze.
• Get started quickly. Dive into the world of Kai, and get up and running quickly without delay.

Let's see how we can use this new capability to replicate data into SingleStore and kickstart your SingleStore Kai journey.

So, how does MongoDB® replication work?

The following diagram shows the setup of a MongoDB® application that is using a MongoDB® database for its transactional capabilities. The Kai shell provides a path for users to connect to the database, and query the data.

You can power your application using SingleStore Kai by augmenting or replacing MongoDB®. The following image shows the scenario of augmenting MongoDB® where it continues to be the operational database, while SingleStoreDB becomes the analytical one.

Collections of JSON that require analytics are replicated into SingleStoreDB using MongoDB® replication capabilities, and the application runs ultra-fast analytical queries against SingleStoreDB with the same MongoDB® queries.

Replication is achieved by one-time data replication and ongoing CDC (Change Data Capture) from MongoDB® collections.

Replace MongoDB® with SingleStoreDB

If your primary goal is to perform fast analytics while minimizing data replication, SingleStoreDB can replace MongoDB® as your primary database to become the sole database for both operations and analytics.

In this case, you will move all collections and data into SingleStoreDB, running all operations against it as the primary database and analytical engine. You can enable the data migration by setting up the pipeline and doing one-time data replication from MongoDB®.

How to setup the MongoDB® replication

There are two ways to set up this feature to bring in data from MongoDB® clusters into SingleStore:

Method #1: MongoDB® wrapper commands through the SingleStore Kai endpoint

This method is possible only on SingleStore Managed Services for a workspace with Kai enabled.

Step 1

Use your Mongo shell or any other Mongo Client (eg. VS code) you prefer, establish a connection to your SingleStore workspace, and find the connection string in your workspace connection options.

Step 2

Create a database in your workspace and create a link that connects to the remote Mongo cluster. It can be MongoDB® Atlas or self managed Mongo clusters.

Step 3

Create a collection using the link created in the previous step. This step can be used to select which collections need to be replicated and which ones to skip. Refer to the detailed steps in our documentation here.

Example

Let’s say we have a MongoDB® Atlas cluster with a collection custs in the retail_sales database and it has ~200k documents in it.

After running the commands, you can check that the destination collection/table is created in SingleStoreDB and the pipelines set up are replicating the one-time snapshot that has ~200k documents and the further change streams.

Method #2: SQL commands

Alternatively, you can also use SQL interface to replicate data from MongoDB® clusters into SingleStoreDB. You can find the SQL commands to run in the documentation here. 

Monitoring the replication

Note that internally there are regular SingleStore Pipelines that complete the replication process — you can debug them and get the status of the replication using the information_schema table. To check the status of the replication, run the following SQL command on the target database in SingleStoreDB.

SELECT SOURCE_PARTITION_ID,
   EARLIEST_OFFSET,
   LATEST_OFFSET,
LATEST_EXPECTED_OFFSET-LATEST_OFFSET as DIFF,
UPDATED_UNIX_TIMESTAMP
FROM information_schema.pipelines_cursors;

The replication status is in progress when the DIFF is 1 and is up to date when the DIFF becomes 0.

Summary

In summary, SingleStore Kai offers a smooth experience to power analytics on MongoDB® data without any application code changes or data transformations. And now with native MongoDB® replication capabilities, setting up data replication from MongoDB® into SingleStoreDB seamless and quick. Harness the power of real-time analytics for your MongoDB® apps and unlock new possibilities.

Interested in trying it for yourself? Get started with SingleStore Kai today.

*SingleStore is not a MongoDB® partner. SingleStore Kai™is an API that offers compatibility and enhanced performance for applications built on MongoDB®.