Operational Analytics - The Last Mile In Data and Analytics

Bill Gates has been quoted that we are in the fastest era of innovation. That’s a lot of innovation - let’s remember the wheel, bronze, printing press, automobile, airplane, CPU, to name a few.

Let’s take a look at the last twenty years or so. First there was the internet which democratized data, content and distribution, among other things. Steve Jobs and the rise of mobile. Amazon and the standardization of e-commerce. Around the same time AWS invented, or at least made mainstream, the public cloud, allowing anyone or any company to become a builder and try things quickly, easily, with minimal cost. Open source further extended the trend of democratizing data. The combination of these advances gave developers more freedom than ever before. No longer were they stuck using the language and environment their IT department provided. Developers were now freed to use any thing they wanted. If a new open source project enabled faster streaming or data ingest, developers jumped on it. Whatever worked best was now most likely to become the most highly used, creating a technology meritocracy. The next big wave to come was the advent of microservices which enabled applications to be built even faster and more flexibly. Now we have machine learning and AI, which are in the early days of perhaps reshaping the world as we know it.

As a result of these innovations the world and our expectations have changed dramatically. We expect to have access to everything immediately, whether that’s the delivery of our groceries, the ride to our destination, or any information about anything we want to know about - at that instant!

The common denominator to most of these changes is data. Data has become the fuel for most of these forces. Over the last twenty years there have been major advances in technology that have changed the data and analytics space. However, despite all these advances there remains one major gap - having visibility to what is happening now. Companies are executing faster and faster however their ability to see what is happening is delayed. Most enterprises have at best a 30 minute delay as to what is happening in their business. And that is only possible with tremendous investment in time, tools, people and, ultimately, money. So there is a “last mile” gap. The data is there, often in many places, but accessing it, and then making it consumable / searchable by dashboards and applications, has been missing. This is a major issue for businesses. They have to execute in real-time to remain competitive but they can’t see how they’re actually performing against that until after the fact. Sounds a bit like driving by only looking in the rear view mirror. Yikes!

Why does this gap exist? What is preventing businesses from acquiring this insight? Of course it comes down to technology. It turns out that the solutions that are so effective at providing high speed transactional capability that operate these businesses are also ineffective at being able to report or provide analytics. And systems that are effective at providing analytics and insight can't provide these insights at speed, to large numbers of users. The need to combine both of these concepts is often referred to as Operational Analytics, with a new breed of engine emerging to handle its requirements.

In my nearly twenty years in the data and analytics space there has been one constant, value is difficult to extract from data. One of the primary barriers to that value is the significant time, preparation, tools, people and expense needed to get the data in the state that is consumable by applications and users. Rockset is the first technology I’ve seen that enables users and companies to bypass all that work, time, and expense and get to the value seamlessly. Operational Analytics is here. And the last mile bridged.

Real-time SQL on NoSQL

Related Posts

Real-Time Analytics in the World of Virtual Reality and Live Streaming

An architecture for real-time decision-making and live dashboards on VR data in Kafka, coming from live-streamed events.

From Good to Great: How Operational Analytics Gives Businesses a Real-Time Edge

All businesses today are a series of real-time events. But what separates the good from the great is how they capture and operationalize that data.

Operational Analytics: What every software engineer should know about low-latency queries on large data sets

What are the characteristics of an Operational Analytics processing system, and how does it differ from OLTP, OLAP and other data systems?