Arcules Scales AI-driven Video Surveillance and Analytics Platform with SingleStore to Support Thousands of Global Users

Consumer-grade security systems made the leap from analog to cloud-enabled relatively quickly, but enterprise-grade technology lagged behind. These organizations found themselves wasting time on applying software updates, maintaining remote access, worrying about risk, and ripping out cameras when it switched systems. 

Arcules has as its primary goal to make the world a safer place. This diverse and passionate team of security and cloud experts shared a desire to create the world’s most user-friendly, affordable, and flexible security surveillance system and insights solution.

The platform Arcules developed offers an intuitive, cloud-native solution that uses AI to unify and make sense of the data from video  surveillance systems for security and beyond. The Arcules platform is device-agnostic, with a plug-and-play experience that uses a small edge hardware appliance to autodiscover and connect supported devices like IoT cameras, sensors, and access control panels to the platform. Users can view all their remote sites in one interface through the web interface or native Android and iOS mobile apps, making it ideal for multi-site distributed services including franchises such as WeWork. 

Arcules views its cloud platform as not only the future of security video, but also an all-inclusive analytics tool offering meaningful operational insights using the data from cameras, sensors, door controllers, and other devices. 

Challenges/Goals 

Many of the typical video surveillance management systems today for enterprises lack the concept of viewing all of its sites in a single, cloud-based package. Arcules caters to multi-site distributed services like franchises, where organizations have many locations without a single pane of control, by providing video surveillance-as-a-service. However, that’s not the only use it has in mind for its platform.

While the main focus of its solution is security and surveillance, Arcules wanted to expand the system to offer analytics from video data to get more value from the data. To do so, it needed all-new architecture. 

Arcules was growing as a company, so it had to do something about scalability of the platform. A lot of what it had built originally as a startup was just to get off the ground and get its proof-of-concept in production. When it started to see adoption of the platform, it had to start web-scaling everything.

Without this scaling capability, the user experience would suffer. The structure of the platform was built with microservices, but the underlying database was struggling. Arcules was already microservices-based, so it was on a good footing, but a lot of its database technologies behind the scenes weren’t ready for the scalability that its microservices provide. As part of a seamless user experience, Arcules needed a video index to allow the system to intelligently decide where the video data should come from. If the data is in the cloud and the user isn’t at the local site, then it can be accessed from the cloud. If the user is on-site, then they can access the local data. Arcules wanted to make the experience a lot cleaner and faster for the user, with the video data source abstracted away so it still feels like an on-premises security system.

“We want our customers to experience very little lag,” said Rowe.

Technology Requirements 

To enable real-time analytic processing of the data and a seamless user experience, Arcules needed a modern database that could deliver speed and scale at a cost-effective price.  

Rowe articulated Arcules’ main concern. “The biggest problem we had was scalability. We’d hit the vertical scaling limits of the databases before.” 

Arcules’ other concerns included: 

• Providing the correct solutions for customers, supporting its needs for fast data and real-time insights.
• Offering the right fit for customers to meet the surveillance requirements for enterprises. 
• Creating a seamless experience for the user from a product perspective, to deliver as-good-as on-premises latency for its surveillance systems.  

The ideal database technology needed to deliver: 

• Real-time analytics and gap and island analysis on all video datasets 
• Ad-hoc requests of certain timeframes to generate data for the user
• Support for Google Cloud (GC) 
• Processing speeds capable of returning results on 30 days of video data within 10 seconds for near-real-time accuracy 
• High availability and multi-region support

Why SingleStore 

When Arcules originally launched its platform, it used MySQL as part of its bootstrapping process. Arcules switched to PostgreSQL for better security and data segregation, using Google Cloud SQL for a fully-managed relational database. It  developed its platform using containerized microservices on Google Kubernetes Engine (GKE), connecting to the data through Cloud SQL Proxy sidecars. However, this setup could not keep up with Arcules’ success, as PostgreSQL had a lack of horizontal scaling.  

The team started exploring other database options four years ago. PlanetScale didn’t work for its use case, and while ElasticSearch solved the use case issue, it was cost-prohibitive. Then SingleStore appeared on its radar.

SingleStore is built for data intensity and provides a robust feature set capable of powering and scaling Arcules’ platform:

• Vector data functionality is built right into the core SingleStore platform so Arcules can power AI-driven applications without a separate, purpose-built vector database
• Run anywhere capabilities support Arcules’ deployment on its on-premises Kubernetes cluster and GC
• Universal storage offers a patented single table type for transactions and analytics, reducing latency by bringing these workloads together
• Compatibility with ANSI SQL for a smooth transition from PostgreSQL
• Fast ingestion for keeping up with continual surveillance video feeds 
• Pure speed for fast transactions, queries, and analytical processing

One of the things the team likes best about SingleStore is that it heavily compresses the data. The raw data ingested into the database versus the actual data that SingleStore stores is radically different because Arcules compress it by at least 90% all the time.

Solution 

Arcules uses SingleStore to power its video indexing capabilities to support real-time video indexing, and fast data to improve the user experience and expand the platform’s capabilities. It opted for the SingleStore Self-Managed solution, hosting it on GC. 

Arcules runs off of three SingleStore production database clusters using four nodes, located in the United States, Europe, and Japan. They are high availability, with active failovers. These databases are supported by several development clusters. 

The data moves from the user’s cameras and IoT devices through the Arcules edge appliance to the Arcules gateway, which is its on-premises Kubernetes cluster. The data gets converted into a specialized format, and the metadata is propagated via Google Bucket in GC. This process auto-triggers to feed the data via a pub/sub model API into SingleStore. The index allows Arcules to intelligently decide where to grab the data from either the cloud or local systems.

Arcules has a data-intensive platform, with 30 terabytes (TB) of data attached to the disks today, with 15-20 TB being utilized on a 30-day basis. A typical user channel has two-and-a-half gigabytes (GB) of video metadata. Users will request the data for specific timeframes. Arcules does gap and island analysis within SingleStore’s query language to generate data for the user, allowing them to quickly navigate through the video content without having to scrub the data. 

As a multi-tenant, global solution serving users in 15 countries, it took a few months for Arcules to determine the right shard key to ensure that the data was distributed properly. When running certain types of queries, such as gap and island analysis, teams try to keep the data on the same node, so they don’t have to jump the nodes and go to the aggregators. That’s where the more intelligent time bucketing concept comes into play. Arcules is using this method quite heavily inside SingleStore to do the gap and island analysis.

The queries that Arcules runs to produce this time bucket analysis are complex and resource-intensive. It’s a full analysis that looks at the patterns within certain time boundaries. It modifies the query to smooth out the video to account for any packet drops. If the time between one video and the next is within a few hundred milliseconds, it gets ignored. Otherwise, the system performs other processes on it, such as generating heat maps, verifying door access, and counting people in a room.

From an ingestion perspective, Arcules averages more than 100,000 inserts per second, but it varies. For example, the ingest and user concurrency are lower overnight because there’s not as much video being recorded. It can also see peaks and troughs in the usage because of SingleStore.

Outcomes

Arcules has gained serious advantages in real-time performance, scalability, and user experience as a result of choosing SingleStore:

Real-Time Performance Gets Video Data to Users On Demand

Users have a seamless experience as though the data is local because of SingleStore’s speed. Ingest speeds range from thousands to millions of rows per second, while the system averages approximately 250,000 inserts per second. Data query results on an hour’s worth of data and resilient ingestion have sub-second response times. For 30 days of data, with 2.7 million records, results arrive within 10 seconds. 

Robust Scalability Supports Thousands of Concurrent Users

For the video feeds, Arcules is ingesting terabytes per day and has 30TB attached to the disks. However, the data it stores per channel is much lower at 2.5GB, thanks to the more than 90% compression rate of SingleStore. As Arcules grows, it’s now able to support thousands of users accessing the application simultaneously.

Off-the-Charts Performance Improvements (20,000%) 

Arcules was one of the first startups to enter the surveillance market with a cloud-native, real-time solution, and the 20,000% performance improvements that SingleStore brought to the system had people shaking their heads in disbelief. Users didn’t believe it was that fast, and with SingleStore, the team has encountered that multiple times. Given that it was one of the first to enter the market with this kind of concept, users thought there was no way that was happening in real time. One of the field CTOs told the team they had to adjust the performance improvement numbers down from 20,000% to 1,000% because users didn’t believe the actual number.

Arcules is looking towards the future for its data infrastructure with SingleStore. It plans to add more analytical data and other metadata types that may tie back to video in these database clusters, using them like a time series database.

Another exciting plan on the horizon for Arcules is incorporating AI for basic array searching. For example, a user could say, “I need a red car and it needs to be within X area of my footage,” and they would receive matching results.

SingleStore is helping companies compete and win across every vertical. Learn more →