Top 5 Reasons for Moving From Batch To Real-Time Analytics

Fast analytics on fresh data is better than slow analytics on stale data. Fresh beats stale every time. Fast beats slow in every space.

Time and time again, companies in a wide variety of industries have boosted revenue, increased productivity and cut costs by making the leap from batch analytics to real-time analytics.

One of the perks of my job is getting to work every day with trailblazers of the real-time revolution, whether it is Doug Moore at construction SaaS provider Command Alkon, Carl Sjogreen at e-learning provider Seesaw, Jake Quist at venture capital firm Sequoia Capital or Yaron Levi at health app maker Rumble. From them, I’ve observed five key benefits repeatedly enjoyed by companies making the move from batch to real-time analytics. I’d like to share them with you here.

Reason No. 1: Building a Data-Driven Culture

Companies make smarter decisions when they are guided by data. Smart companies create a culture in which data-driven insights inform every key decision and operational process. That requires democratizing access to data, taking it from the C-suite and the data scientists training their ML models to every operational employee or customer who would stand to benefit.

You can’t build a data-driven culture relying on batch-based analytics and BI. There is too much latency at every step — finding the data, ingesting it, querying it and representing it. In an age of lightning-fast consumer apps such as Instagram, users won’t tolerate excruciatingly slow analytics experiences. Not your customers, nor even your internal employees. If answering every question takes 20 minutes, your workers simply won’t ask any follow-up questions.

Where extract-based BI tools fail, modern interactive analytics tools and data-driven customer-facing applications succeed, providing users with sub-second response times as they drill down into seconds-old data. One investment firm we work with formerly had data warehouse-based dashboards with 50 to 60 gauges each. Performance was sluggish, as it took two to five seconds for each gauge to update, so employee usage was low.

The firm replaced the dashboards with a web app built on top of Rockset, which accelerated their query speeds by more than 100 times. Now, the gauges take just 18 milliseconds to update with the latest data. As a result, usage of the dashboard has skyrocketed.

Users embrace a data-driven culture when they can ask questions in real time. Being able to explore data for answers, also known as guided decision making, is incredibly powerful. It enables companies to pull off bold and creative moves, which because they are informed by the latest data, don’t come with the normally associated risks. Smart data-driven decisions become a companywide habit. And that can only happen with real-time analytics.

Reason No. 2: Deploying the Power of Business Observability

Running a business has traditionally been equal parts art and science. When devising long-range strategies and operational plans, leaders could rely on historic data and reports. But in chaotic times that required quick decisions, managers and operations personnel only had their instincts to rely on. Without access to fresh data, a corporate war room was little more than a glorified Hollywood movie set.

The irony is that as today’s business operations have gotten more distributed and complex, the number of applications that are monitoring and generating data about those operations has also grown. But until recently, the problems with that data has been multifold. The data was either:

• Too old, and thus useless in a fast-moving tactical situation.
• Too copious and unfiltered, resulting in alert fatigue and information overload.
• Too poorly-analyzed to enable you to understand “Why?”

Sophisticated real-time analytics ushers in a new paradigm called observability. Observability has two aspects. The first and more mature side is infrastructure observability, pioneered by DevOps and site reliability engineers (SREs). It was imperative for them to know what was happening at the moment to debug anything. Strong monitoring and alerting capabilities that poked and prodded them when something was wrong were also essential to avoid the need to be glued to their chairs staring at charts 24/7.

The newer aspect is business observability, which empowers human decision makers with useful, up-to-the-moment, and easily understood business insights generated from the freshest data. Now, every sales lead, product manager, risk officer and operations manager can have deep, drill-down visibility into the real-time state of their operational area, whether it be the effectiveness of their ad spend, quarterly sales goals, payment default risks, or support operations. They can also be alerted whenever numbers exceed specific thresholds.

Here’s an example from a big conglomerate that I know well. As the end of every fiscal quarter neared, the company’s executives wanted to know exactly where its sales and profits were in relation to its forecasts. Unfortunately, reconciling revenue operations and updating earnings data normally took 12 to 24 hours. They could deliver updated forecasts within six hours, but only by making lots of data engineers and analysts work overtime.

Instead, the company turned to real-time analytics to power its real-time revenue and earnings dashboards. Now, the company’s leaders can meet and decide based on current information what incentives to give their salespeople, which customer offers to put into play, and when to deploy them. And they can now make those decisions confidently in real time before the windows of opportunity close, not just in a blind panic.

Or take another company I’ve gotten to know well, the popular online learning platform, Seesaw. Seesaw is used by more than 10 million K-12 teachers, students and family members every month. The company saw a 10x surge in growth during the COVID-19-created shift to remote learning, Seesaw’s co-founder and chief product officer Carl Sjogreen recently shared in a video interview with theCube.

Seesaw’s cloud-native technology constantly generated a wealth of data around how students and teachers used the service. Seesaw built real-time business observability by using Rockset to analyze that data. By using Rockset and the reverse ETL tool Hightouch, insights are delivered straight into Salesforce for easy access by the company’s sales, support and product teams. Now, salespeople can understand which school districts and teachers are succeeding and which ones are a churn risk. Product engineers can now see where performance bottlenecks are emerging and quickly diagnose why. And teachers and school districts can get more information on which students are succeeding and failing and why, enabling them to quickly take action.

Reason No. 3: Managing Risk in Real-Time

In this interconnected world, risks loom all around us, ready to emerge quickly — spiraling pandemics, crashing stock markets, unexpected government policies, sophisticated financial scammers and determined hackers. But risk mitigation has traditionally been a reactive, after-the-alarm-bells exercise.

A passive risk mitigation posture is no longer adequate for today’s businesses. There will be too many fire drills, too many halted operations, lost sales and massive fines. Companies are upping their risk mitigation game with real-time and predictive analytics.

We are working with a large buy-now-pay-later company with 90 million users connected to 250 thousand merchants. The company was doing its risk modeling with a batch-based analytics system that refreshed data every six hours. However, as transactions grew, that was slowing, weakening its protection of merchants. The firm wanted to manage its risk in real time and flag anomalies across multiple dimensions — merchant, country, payment method and more — so that affected merchants could take corrective action faster and save millions of dollars a year.

Moreover, as the payments firm moved up to a real-time analytics system, it didn’t want an alert sent for every transaction that appeared out of the ordinary. That would result in constant false alarms, which would quickly result in alert fatigue for its overworked fraud team.

Using Rockset’s SQL-based Rollups feature, the firm was able to get up-to-the-second accurate real-time metrics across all desired dimensions and effortlessly build all their risk models using SQL analytical functions. Their risk models run every minute, with every invocation taking mere hundreds of milliseconds to run. This lets the payments firm instantly filter out innocuous outliers so it can deploy investigators to chase the most likely cases of fraud, boosting case outcomes and keeping its business risk under control.

Reason No. 4: Real-Time Personalization

To compete effectively today, e-commerce companies must go beyond simply price, selection and convenience. Personalizing their customer experience is a must-have. Statistics show that 80% of shoppers are more likely to buy from brands that offer personalized experiences. Personalization can also increase sales by 20%. And with the wealth of customer data and real-time signals available today, most e-tailers are rushing to take advantage.

One of our customers, the vitamin company Ritual, deployed Rockset on top of its existing Snowflake cloud data warehouse. Within a week, Ritual was able to deploy on-the-fly customer offers such as targeted banner ads, or coupons and bundle offers during checkout. These offers are personalized based on “affinity profiles” that individual customers are linked to based on their prior purchases and product views.

Besides being a key part of Ritual’s website upgrade, real-time analytics enabled Ritual to generate personalized offers in the short window they have to serve their web pages. The company plans to continue expanding the use of real-time personalization by dropping customized offers into its product pages to boost sales.

Customer personalization isn’t just key for online retailers but anyone interacting with B2C or B2B customers over digital channels. Recommendation engines help Netflix serve the movies you are most likely to be interested in. Real-time personalized recommendations drive increased engagement for a popular social media site with whom we work.

Reason No. 5: Providing Proof of Value to Customers

Real-time business observability doesn’t just benefit the company deploying it. It can also be a huge boon for its customers. Think of car-sharing services such as Uber and Lyft that let waiting passengers see on their app exactly where their car is and when it is expected to arrive. Those high expectations have leapt from the consumer software arena into the B2B software space too. Businesses are now expecting real-time web or app updates from their vendors and suppliers, too.

That’s why one of our customers, Command Alkon, moved all of their in-app search, reporting and analytics from batch to real time. The U.S. company’s cloud-based construction software handles millions of material shipping tickets daily, including 80% of the concrete deliveries in North America. And after upgrading to real-time analytics, Command Alkon could give its customers, whether it be material suppliers, shipping companies or builders, a real-time, granular view into every ticket and delivery order.

Late deliveries create delays at construction sites, and that raises costs for builders and creates friction between all parties. In particular, the window for mixing and delivering concrete before it hardens and becomes unusable is especially tight, as short as 90 minutes in the summer heat. Providing up-to-minute, drill-down visibility into their supply chains, including status updates, truck locations, delivery confirmations and more was a key value proposition for Command Alkon and has helped the company grow quickly and win many new customers.

Conclusion

I hope you learned why companies of all kinds are moving from batch to real-time analytics.

If you found the use cases intriguing but immediately wondered if you could repurpose your existing data stack for real time, let me share this. In every case above, the companies invested time or money trying to retool their existing batch-oriented database for real-time analytics. And in each case, they found that deploying a real-time analytics solution instead of abusing their data lake/warehouse would be far less work and cost, and offer much-better price-performance.

The next time you hear someone insist that batch is good enough, please send them this article.

If you are looking to modernize your data stack and want to talk more about real-time analytics, I would love to hear from you. Just write “I would like to talk to Venkat” in this form, and I will get back to you personally.

Rockset is the real-time analytics database in the cloud for modern data teams. Get faster analytics on fresher data, at lower costs, by exploiting indexing over brute-force scanning.