Your Phone Already Has Bottomless

SingleStore has a key new feature in our 7.5 Release called bottomless. In this blog we use something familiar to everyone, photo apps, as a metaphor to explain bottomless and its relevance to databases. The things consumers get by syncing their photos to the cloud can benefit real data applications too: we call it separation of storage and compute. Learn why, for data intensive applications, separation of storage and compute in your data store just makes sense.

I take tons of pictures on my phone, most of which are blurry pictures of my dog. However, it’s been literally years since I have had to think about my phone’s storage. I take photo after photo with reckless abandon, never deleting or curating, and act like it's my god given right to walk around with every picture I’ve ever taken on my person at all times. I’m not an expert in smart phone photo storage technology, but I think I can figure out what they’re up to.

Photos taken on your phone are synced with the cloud. I don’t remember when they started doing this, or even if I consented to it, but by 2021, it's an expectation. And while it might be a bit strange to think that, some time in the last decade, my personal pictures were copied to a datacenter in the far reaches of who-knows-where, the little conveniences we all get seem to be worth it, even if we barely notice.

For one thing, I don’t have to think about storage at all! I don’t look at most of my photos most of the time… I mean, I got my first smart phone for high school graduation and have been accumulating pictures ever since. But it's really not a big deal, the local versions of the pictures can be deleted. If I ever do want to look back, those photos will just take slightly longer to load as they seamlessly download from their cloud backups.

I also never have to worry about taking backups or losing photos. I hate upgrading technology, so I drive my phones into the ground but when my phone inevitably dies and I have to buy a new one, all my pictures are already “there”. I don’t even have to wait for them to download, since they can just download lazily when I go to look! Not only do I not even have to take backups, but restoring from the backup is fast, since I’m ready to go as soon as I download the metadata. It's as if the cloud and the phone itself are one system.

All these things that consumers can get by syncing their photos to the cloud can benefit real data applications too: it’s called separation of storage and compute. Only store the data you’re querying! Don’t worry about backups, all the data is stored in an object store with 11 nines of durability! You can provision more compute to run over the same data in the same way as you can buy a new phone and already have your photos. For data intensive applications, separation of storage and compute in your data store just makes sense.

There is one subtle question, almost an implementation detail, that you can ask: when does a photo get uploaded to the cloud? It would be absurd, for this application, for someone not to be able to take a photo unless they could immediately upload it! Could you imagine the annoying, unpredictable lag you’d get if every time you pressed the button your phone had to upload an image to the cloud? Of course that's not how it works, I can go on a hike away from service and snap all the pictures I want; the photos I take are just stored in the phone until you get back to service.

Separation of storage and compute is mostly found in cloud data warehouses, which are good for analytics but don’t prioritize transactional capabilities.  Obviously, cloud data warehouses have a completely different set of trade-offs than consumer smartphones, so our competitors made the opposite decision: in the analogy, the camera app would hang until the photo was uploaded.  For an analytics-only system, this tradeoff makes sense because transactional latency doesn’t matter to them.

This fundamental choice sets SingleStore Bottomless apart from our competitors. If I write one row to SingleStore, that row commits without ever touching remote storage, just like I can take a picture without ever touching the cloud*.  This means transactions have low, predictable latency, which means SingleStore can run apps in addition to analytics.

* The careful reader might think this means SingleStore can lose data. This is false, transactions need to be ack’d in multiple places in the SingleStore cluster to be committed, but this article is really more about the metaphor than the technical details, and the metaphor breaks down because local replication for consumer smartphones doesn’t make sense.

To learn more about bottomless, read this blog post, or for a more technical presentation, watch this video. 

Try the new 7.5 release version of SingleStore and experience bottomless for yourself.