Silicon Digital Drops MySQL, Chooses SingleStore Over Clickhouse to Achieve Real-Time Analytics in the Roblox Metaverse

Silicon Digital, owned by Gamefam and the creator of RoMonitor Stats, is based in the Isle of Wight, UK. The company develops experiences and technologies for developers, brands, and advertisers on the Roblox Metaverse, reaching millions of players every day. The company works with brands to help bring their intellectual property to the metaverse through branded experiences or in-play advertising.

Silicon Digital is a pioneer in the development of the Roblox metaverse with its tools and experiences, and was the first company in the UK to focus exclusively on Roblox Metaverse development. It has also built a robust advertising software development kit (SDK) with in-depth targeting technology for the metaverse. 

Silicon Digital provides all the analytics and rankings for GameFam’s 52 games in the Roblox Metaverse, along with powering GameFam’s advertising analytics. The team also developed RoMonitor Stats, a widely used analytics platform for Roblox games. It helps developers find up-and-coming games on the Roblox Metaverse, learn tactics that help their games grow, and connect with their player communities.

Challenges/Goals 

The Silicon Digital team wanted to give more data to creators on the Roblox platform through the best analytics for advertising technology. It also wanted to bring value to the parent company by providing analytics around player engagement, allowing GameFam to see if it was building the right games for its audience.

Silicon Digital originally deployed its analytics on Amazon Lightsail, running on a MySQL database. After nine months of running on this platform, performance became extremely slow. The team was forced to export all user data and analytics in real time, with four machines pulling from one server. “We were worried we were going to lose a lot of our analytics and all of our historical analytics. This was in the early days of the platform. We hadn’t built backup plans yet and hadn’t had discussions about disaster recovery, “ said Nicholas Phythian, Founder and Director, Silicon Digital.

Nicholas Phythian, Founder and Director, Silicon Digital

When these performance issues and outages occurred, the Roblox developer community would complain loudly on social channels — and in addition to paying a price on social media, downtime also caused Silicon Digital to lose advertising revenue from brands.

Technology Requirements 

Silicon Digital needed a database that could support a variety of use cases, including: 

• Analytics about ad impressions 
• Data warehousing 
• Data storage 
• Data catalog 
• Data visualization 

The ideal solution would deliver the speed, performance, and reliability Silicon Digital needed so it could offer a quality user experience and avoid downtime.

“We wanted something that could scale. With our business, we have lots of data. With MySQL on Amazon Lightsail, we were limited to 260 gigabytes," explained Phythian. "We had to find a solution or change to Amazon’s more expensive products, and we did not want to experience bill shock later.”

The team also wanted ultra-fast performance to speed up queries, moving towards real-time analytics. Its original solution required complex workarounds to achieve this. “If we wanted to get data, we had to query the database directly and it was slow. The engineer sat there with their laptop, hoping it didn’t go to sleep while they ran the query in the background,” said Phythian.

This approach was also resource-intensive and made it difficult to scale the solution.

“You don’t want to be running live site queries on someone’s laptop and then visualizing a whole data table and sticking it into Microsoft Excel. We don’t want to have to use Excel because our database can’t handle us pulling data in real time,” said Phythian.

“We had issues with scaling and slowdowns during live campaign runs. We reached a point in February 2022 where we said, ‘We can no longer use MySQL. We need to find a better solution,’" said Phythian. 

Why SingleStore 

Silicon Digital started searching and discovered SingleStore. “We're trying to keep it all in-house, or we choose things we know are used by the best of the best. SingleStore is used by the best of the best,” said Phythian. That’s very clear from its Customers Page.”

The team also considered ClickHouse during the evaluation process. “ClickHouse didn’t look appealing to us. We thought SingleStore looked more modern and the website was more professional,” said Phythian.

One of Gamefam’s partners, Dive, also uses SingleStore. “We saw how fast SingleStore was from our partner’s perspective. We thought if it worked for them and they are tracking in-game events on a mass scale, it's going to work for what we need to do," he added.

SingleStore’s drop-in compatibility with MySQL and support for SQL meant Silicon Digital could move forward without making major changes to the codebase or having its engineers face a steep learning curve. “The engineers were very happy with it. We started the SingleStore discussion there, spun up our first server, then initiated the migration process,” said Phythian. 

Disaster recovery was also top of mind for Silicon Digital, as it needs to have zero interruptions to its advertising and analytics services. “We have horror stories about how we survived multiple outages with disc failures on Amazon Lightsail because we didn't scale up correctly and we were handling heavy data,” said Phythian.

Earlier on, Phythian and the team ran OLTP on SingleStore as a MySQL replacement and OLAP queries on TimescaleDB, but that also increased complexity working with an additional database. Now Silicon Digital has unified analytics and transactions on one platform with SingleStore.

Solution 

Silicon Digital selected SingleStore Helios, the fully-managed, on-demand cloud database service, to power its data-intensive applications. “SingleStore is the heart of everything — data, services, and adtech — to create that data processing warehouse,” said Phythian.

Silicon Digital supports 60 apps including: 

• GameFam’s 52 games 
• AdTech portal
• Grafana visualization tool
• RoMonitor platform 
• RoMonitor iOS app
• RoMonitor Discord bot 
• RoMonitor Twitter backend 
• Internal reporting 

Silicon Digital uses three SingleStore clusters: one each for testing, adtech, and RoMonitor Stats. The SingleStore database takes and tracks all impressions from billboards, which drive visualizations on a Grafana dashboard. Amazon Lightsail remains in place to serve the main front end website. Laravel is the core framework for all of the company’s backend projects. Silicon Digital deployed this solution with SingleStore’s valued partner Twingo.

“We like managed services because if billing issues happen, it’s on the provider. They built their infrastructure and know what they’re doing,” he added. “With SingleStore, we’re more relaxed than if we were managing it ourselves. We know security is up-to-date. SingleStore Helios is a great platform for us.”

Since Silicon Digital deployed SingleStore’s fully managed service, it can fully focus on improving its Roblox Metaverse solutions and exploring new innovations.  “Today we’ve got disaster recovery everywhere, with analytics running on two separate SingleStore systems. If something goes wrong, we always have a backup,” said Phythian. “With the managed service we know we are safe if something goes wrong. It's great to have that team at your back, which you don't get with many companies. With SingleStore, we've had that and we've been very happy with the support we've received.”

Silicon Digital’s engineers had an easy time learning SingleStore, thanks to SQL compatibility and the available documentation and training resources. If the company had decided to go to a NoSQL database such as MongoDB, the engineers would have to learn a new way to work with and query data. When the team migrated the single MySQL server to SingleStore, they copied the same table structure and made a few tweaks here and there. However, no major changes to the code base were required.

Phythianl also had high praise for SingleStore’s rolling update system. “When our master aggregator needs to update, it only goes down for a split second and has a minimal impact.” 

In addition to its speed, scale, and agility, Silicon Digital also leverages these other SingleStore features: 

• Unlimited (Bottomless) Storage: Adtech is data-intensive, and Amazon Lightsail’s 260GB data storage limit was insufficient. This SingleStore feature future-proofs Silicon Digital’s growing data requirements.
• Data API: The Roblox Studio app requires SQL commands to go via HTTP, as it does not support raw database services. SIngleStore’s Data API allows Silicon Digital to send these commands directly, which cuts its costs and allows the team to focus on what they do best. 

Data volume is highly variable, and includes seasonal fluctuations, but as Phythian noted, “I'm not worried about data ingest spikes because SingleStore is bottomless, so we don't have to be concerned.”

Outcomes 

By choosing SingleStore Helios, Silicon Digital resolved its performance, scalability, and disaster recovery issues, and gained the agility to continue innovating at the cutting edge of the Roblox Metaverse. “When we migrated away from MySQL to 100% SingleStore, we saw an immediate improvement. Our engineers and users are happy,” said Phythian.

Breaking Free of Scaling Limitations with SingleStore Unlimited Storage

Silicon Digital faced a 260GB data limit on Amazon Lightsail. Given that the company could be dealing with 500 GB of data and 1.7 billion rows in just one table, and the seasonal fluctuations in data volume that created spikes during November and December, that made continuing on the existing infrastructure impossible.

“Now we’re not worried about the holidays, because SingleStore and its Unlimited Storage have us covered,”  said Phythian. “When we add a new game, we no longer have to worry about what it will do to our data infrastructure.”

Dashboards Now Load in One Second Instead of 30 Minutes

Silicon Digital has seen a massive increase in speed across the platform since choosing SingleStore Helios and migrating away from MySQL, both for its internal backends and the user-facing experience. “SingleStore lets us do real-time database querying, which is what we want. We can process all data, visualize it quickly, and not have to worry about it taking 30 minutes to load a dashboard. Now, it’s under one second,“ said Phythian.

Delivering Peace of Mind and Gaining a Competitive Advantage Through a Reliable, Zero Downtime Solution

“What we love about SingleStore is that it alleviates our worries about disaster recovery, scalability, and complexity. Migration, making changes, and even onboarding new developers is simple,” said Phythian.

Silicon Digital competitors are still using data infrastructure that is slow, with frequent complaints about downtime. “If you need to update MySQL, you’ll be offline for ages. With SingleStore it takes just a split second. The only outage is caused by us manually upgrading Laravel. Brands are going to come to us now thanks to this stability and reliability,” said Phythian.

Enabling Hundreds of Thousands of Concurrent Users

“SingleStore has given us the ability to grow — we needed to scale and we have that freedom now,” said Phythian.

With SingleStore’s high-performance capabilities, it’s easy for Silicon Digital to continue adding games without worrying about increases in user concurrency. He continued, “When we onboard a new game into the GameFam portfolio, we don’t need to worry about what that’s going to do to our infrastructure.”

Expanding the Reach of Metaverse Gaming

Silicon Digital is excited to explore the future with SingleStore at its side. “We are trying to get ahead of the curve. Gaming is one of the key parts of the Metaverse. That's something that is being put aside because of other buzzwords like non-fungible tokens (NFTs), Web3, and crypto. Gaming and doing things you normally do in the real world, such as concerts, virtually, is where it’s at. “We’re seeing big brands like Walmart and Amazon launching experiences on Roblox. You can see the growth and now more brands are coming into the Metaverse,” said Phythian. 

The expansion of the Metaverse is only going to continue to grow, linked with continuing mainstream adoption of virtual reality (VR) technology. 

“We know that we are in the right place at the right time. We were very lucky when COVID hit because we're an online business. Retail businesses suffered, while online businesses, and especially gaming, thrived. During COVID millions of people turned to gaming,” said Phythian. “We plan to start tracking more of these platforms and game environments, building our adtech networks and bring in more brands to continue to scale our campaigns.”

---

SingleStore is helping companies compete and win across every vertical. Learn more →