Changing face of real-time analytics

With work-from-home, order-from-home, shop-from-home becoming our new normal, we all have a much bigger digital footprint. This means businesses have even more real-time user data streaming in. And what separates the winning businesses on the other side of this pandemic will be how intelligently they use that data to increase user engagement. These are systems of intelligence that Jerry Chen, partner at Greylock describes. “What is a system of intelligence and why is it so defensible? What makes a system of intelligence valuable is that it typically crosses multiple data sets, multiple systems of record. One example is an application that combines web analytics with customer data and social data to predict end user behavior, churn, LTV, or just serve more timely content.”

The challenge with real-time data

1. Real-time data comes in many shapes, from many sources - what your users are clicking on shows up as clickstream data in JSON format via Kafka or Segment, what your users are buying shows shows up in a table in your transactional database, what that user demographic looks like comes in XML format from a third-party source like Nielsen and lives in your data lake.
2. Developers need to test and iterate on new features - Your product roadmap is constantly evolving based on what your users need, and your developers want to personalize, experiment and A/B test quickly. This means new data schemas, new sources and new types of queries pop up every few days. In the world of batch analytics and BI dashboards you knew exactly what data you needed and what type of questions your business was asking - which gave you the luxury of setting up all sorts of pipelines to format the data exactly as needed, create materialized views, optimize the queries. Using that same familiar approach as you go from batch analytics to real-time applications is a failure mode.

Modern real-time analytics and applications

So what does it look like when real-time analytics is used for serving applications and not just dashboards? Gartner defines Continuous intelligence as a design pattern in which real-time analytics are integrated into business operations, processing current and historical data to prescribe actions in response to business moments and other events. Analysts predict that by 2025 more than 30% of data will be real-time in nature, and by 2022, more than half of major new business systems will incorporate continuous intelligence that uses real-time context data to improve decisions.

This means the lines between real-time analytics and real-time analytical applications are now blurring. Live dashboards that alert humans to take actions and online applications that automatically trigger the right actions are both in the realm of real-time analytics. When you are evaluating your real-time analytics solutions, look at not just price-performance but also flexibility to handle new data formats and new types of queries so that you are future-roadmap-proof. To foster real-time innovation, the key is to ensure that your developers are only bottlenecked on their own creativity and not on what their real-time data architecture can do.