Armis Saves 70% on Data Pipeline Cost with SingleStore and AWS, Accelerates its Valuation to $3.4B

Industry analysts predict there will be up to 56 billion connected devices worldwide by 2025. The number of unmanaged devices within the enterprise environment is rapidly growing, and organizations need a way to discover and secure it all.

Armis Security was founded in 2015 to help them do just that.

Armis helps enterprises discover and secure managed, unmanaged and IoT devices, including medical devices and industrial control systems (ICS). Armis provides 100% end-to-end asset visibility through a real-time platform that offers granular, industry-specific device details and behavioral insights. The Armis Platform and Armis Asset Management collect data and draw insights from it, allowing enterprises to harness and act on a myriad of information in real time.

“Google Maps helps you find your way from point A to point B, but also provides a great deal of additional information and context that gives you a deeper understanding about not only specific locations and touchpoints but your overall environment,” said Aviram Cohen, Executive Vice President, R&D, Armis. “Our vision is to be the ‘Google Maps’ of your organization by giving you one place to discover every asset in your environment — armed with instant context around risk, vulnerability, threat assessment and threat detection — then act on those insights to secure your assets.”

Aviram Cohen, Executive Vice President, R&D, Armis Security

Challenges/Goals

Armis originally launched its platform using a PostgreSQL database. Over time, the time-based data set got too large for Postgres to handle. At this point the team migrated this data set from 400+ PostgresSQL databases into a huge Elasticsearch cluster (160 nodes).

"Using Elasticseach seemed like a reasonable choice considering the vast knowledge of the team in that technology and the fact that we needed fast searches and aggregations over a large data set," said Tomer Praizler, Chief Architect, Armis Security. It's just that we didn't realize how big this dataset is going to be, and what would it require from us to maintain it."

The data properties themselves changed: the data in ElasticSearch increasingly became more and more mutable, as product requirements evolved. The Armis use case required a full reindex of the ElasticSearch cluster (close to 100B documents) due to the nature of the data, which is highly mutable. Although Elasticsearch can certainly support this kind of reindexing flow, considering the size of the cluster, it required a lot of infrastructure and protections to make this flow robust. “We used a lot of EKS resources for indexing, a very big Kafka cluster to buffer the indexing to Elasticsearch, and of course an EMR cluster to run periodically and fetch all updated entries from our data lake to be indexed again," said Praizler.

Tomer Praizler, Chief Architect, Armis Security

Armis was unable to scale with ElasticSearch, and worse, it was paying dearly for the privilege: the entire data pipeline including ElasticSearch cost more than $1 million annually.

To support its comprehensive asset management platform and the customers who depend on it, Armis needed to: 

• Scale with high performance to meet growing data demands, as enterprises work with a growing variety of connected devices 
• Adhere to strict standards such as the Federal Risk and Authorization Management Program (FedRAMP)
• Reduce the platform’s complexity and cost 

Technology Requirements 

Armis Security processes 100 billion events per day for its global customer base and 30TB data sets in its largest customer environments. Armis collects data from anything an enterprise may have, including devices, firewalls, IoT, multi-tenant, ServiceNow and network traffic. This means Armis has a massive data pipeline to manage and analyze.

Armis needed a database technology to fulfill its technical requirements, including: 

• Query SLAs of 1.5 seconds across three days’ worth of data; three seconds across seven days of data; and 10 seconds across more than 30 days of data to support real-time analytics
• A solution it could deploy as both a managed service and an on-premises solution to meet FedRAMP on-premises requirements
• The ability to frequently update lookups and effectively perform joins

Why SingleStore

Armis considered using Google BigQuery, but lack of multi-cloud support and absence of self-managed solutions were dealbreakers. “Then, in speaking with our customer base, we learned that most of them are SingleStore customers, and the more we heard, the more we liked,” said Cohen. Armis selected SingleStore Self-Managed for the on-premises portion of its deployment to satisfy FedRAMP compliance, and SingleStore Helios to support its cloud data strategy via AWS.

Other features that helped close the deal included:

• In-memory row stores and column stores in the same database
• Petabyte scale 
• Security features built into SingleStore that Armis government clients require, including ISO 27001 and SOC 2 Type 2

Solution

The Armis Platform, of which now SingleStore plays a significant part, collects various types of raw data (traffic, asset, user data and more) from various sources, processes it, analyzes it, enriches it and aggregates it. This creates a full dynamic picture on all of the assets of its clients, which is accessible within the product by free queries on devices, IP session data, predefined metrics and more. The Armis Platform:

• Manages 100 billion events per day
• Manages 30TB data sets in its largest customer environments
• Delivers 1.5-second query speed across three days’ worth of data

Twingo represents, sells and deploys leading big data technologies. Experts in architectural design, Twingo helped Armis choose the right technology and provides the optimal big data solutions for complex problems. Twingo contributed to the POC for the SingleStore deployment at Armis, helping design the data cluster sizing, redesign queries, and optimize the model, then define and run the POC. Armis now has 32 managed SingleStore units, and each unit consists of 8 CPU cores, 64GB RAM, and a 2TB SSD.

“With ElasticSearch, if a single device was updated, Armis needed to update backwards  three months’ worth of data,” said Golan Nahum, CEO, Twingo. “Armis needed to move to a relational model to work with ultra high scale and at the same time, simplify the modeling. With SingleStore, Armis achieved a substantial reduction in complexity with a significant increase in performance and huge reduction in cost.”

Golan Nahum, CEO, Twingo

“Our partnership with Twingo helped us run a successful POC with minimal resources on our end. Thanks to Twingo’s experts, we avoided mistakes and created an optimized solution,” said Praizler. “They gave us continuous advice on how to progress and develop the system while pushing priorities for feature requests.”

At present, Armis still uses PostgreSQL and Elasticsearch for smaller and transactional workloads, has already moved its largest data set from Elasticsearch to SingleStore and moved all analytical workloads from PostgreSQL to SingleStore.

“Operations-wise, we simplified our pipeline with SingleStore, and things work much better than they did with ElasticSearch,” said  Roy Franco, Data Infrastructure Team Leader, Armis Security. “We ingest this massive data pipeline, analyze streaming data, and allow users to drill down on everything,” he explained.

Much of the data in the Armis Platform is updatable:

• Facts result from the actual aggregations
• Usable data is about the devices themselves and it updates in batch mode every 30 minutes

“We want to make this data real time by streaming those changes into SingleStore,” said Cohen. “As an example, let’s say a user’s Macbook Pro has just updated and is now vulnerable; they need to close that gap immediately.”

Outcomes 

“Queries that would time out completely under ElasticSearch are now processing in less than 10 seconds with SingleStore, and some clock in under 1.5 seconds,” said Cohen. The massive increase in technical performance has led to eye-popping financial performance:

Accelerated Business Growth as Valuation Reaches $3.4 Billion

The scaling and performance improvements offered by SingleStore helped Armis Security substantially grow its business, which has helped it triple its valuation in less than two years. When private equity firm Insight Partners acquired Armis Security in February 2020, it had a $1.1 billion valuation. By YE2021, Armis Security was worth $3.4 billion. 

70% Cost Savings

With SingleStore, Armis can flex its costs up and down with the size of its cluster. “We moved from the entire data pipeline including ElasticSearch costing more than $1 million annually to paying a fraction of that for SingleStore Helios, reducing our data pipeline cost by 70%,” said Cohen.

Faster Performance Improves Customer Device Security

The vastly improved performance Armis Security has realized with SingleStore gives its customers a better view of their device landscapes, allowing them to react faster based on fresher data to keep their environments secure.

SingleStore is helping companies compete and win across every vertical. Learn more →