Author

Bryant Xiao

Engineering Manager - Data Engineering

Monitoring A/B Experiments In Real Time

Monitoring A/B Experiments In Real Time

This post originally appeared on the Pinterest Engineering Blog by Bryant Xiao. As a data driven company, we rely heavily on A/B experiments to make decisions on new products and features. How efficiently we run these experiments strongly affects how fast we can iterate. By providing experimenters with real-time metrics, we increase our chance to successfully run experiments and move faster. We have daily workflows to compute hundreds of metrics for each experiment. While these daily metrics provide important insights about behavior, they typically aren’t available until the next day. What if the triggering isn’t correct so that Pinners are not actually logged? What if there’s a bug that causes a big drop in the metrics? What about imbalanced groups? Before the real-time dashboard, there was no way to tell until the next day. Also, any subsequent changes / corrections would require another day to see the effect, which slows us down. The real-time experiment dashboard solves these problems. Here we’ll share how we build the real-time experiment metrics pipeline, and how we use it to set up experiments correctly, catch bugs and avoid disastrous changes early, including: Setting up the real-time data pipeline using SingleStoreBuilding the hourly and on-demand metrics computation frameworkUse cases for real-time experiment metrics Data Pipeline Below is the high level architecture of the real-time experiment metrics.

SingleStore Recognized In

Translytical Data
Platforms Q4 2022

Back to All Articles