Rockset Podcast Episode 3: SaaS Real Time Analytics

About This Podcast

Gio Tropeano:

Welcome to Why Wait? The Rise of Real Time Analytics. The Rockset Podcast, where we talk about the intersection of analytics and data with real engineering leaders from around the world. I'm Gio Tropeano of Rockset, and I'm here with my cohost, Rockset co-founder and CTO, Dhruba Borthakur. This episode we are joined today by Ali Dasdan, the Head of Engineering at Confluence Cloud. Ali and Dhruba, welcome to Why Wait, and thank you so much for being here with me.

Ali Dasdan:

Thank you.

Dhruba Borthakur:

Thanks.

Gio Tropeano:

Before we kick it off, for our listeners and visitors to this content, to this episode, please share your thoughts on our community Slack channel at rockset-community.slack.com. We'd love to hear your thoughts, your ideas, and any questions that you have as you listen to this.

Gio Tropeano:

Ali, you've been at the forefront of the big data revolution from your early days at Yahoo to build the Yahoo search engine. You've built products and also published extensively on data systems. Storing data has become relatively easy nowadays, right? But getting insights is still super hard, relatively speaking, complex, and actually time consuming. Ali, I think we'd love to hear from you. Tell us a little bit about yourself, your history and what you're all about.

Ali Dasdan:

Thank you. Thank you for the invitation to talk about real time analytics. I think it's a very, very important topic and super relevant to many companies. Name's Ali Dasdan. I've been in industry for about 20 years now. Time has passed. Joined after my PhD in computer science. One interesting thing about my career is the variety of experience, actually. I worked in companies of different sizes, multiple startups as well as big companies, different industries. And Yahoo, the example that you mentioned actually, I was tasked to build the internal data team for Yahoo Web Search. That is actually... Dhruba and I worked together. We are first users of Hadoop. Right now I'm at Atlassian. Yeah, that should be short enough, I guess.

Gio Tropeano:

Awesome. And what are you building at Atlassian?

Ali Dasdan:

So I've been leading engineering for a Confluence Cloud for about two years now. So Confluence, as well as our more well-known friend, Jira, these are very common tools that tech teams use, finance teams use, NASA uses; many, many customers. So I'm leading the Conference Cloud engineering at the moment.

Dhruba Borthakur:

Yeah. Talking about Jira, I mean, we have used it when we were doing the Hadoop project a long time back together. But today we're not talking about Hadoop much, but we are going to talk mostly about Why Wait and what is it all about, real-time analytics.

Dhruba Borthakur:

So tell me how you use real time analytics in your product itself? How is your product benefiting, or what kind of data stacks are they leveraging to give additional features and functionality to your Jira product?

Ali Dasdan:

Yeah. Since I have a variety of experiences actually, maybe I can give multiple examples. Though I'll be brief.

Dhruba Borthakur:

Yes. That'd be great.

Ali Dasdan:

Yeah. I think the first one is when I was doing real time advertising. So we were doing it for display advertising, mobile, social and video. The way it works is basically where online users about to see, let's say a web page or a mobile app page, and advertisement has to be found in real time for that user. So when the user gets what they are supposed to be seeing, the whole content already has advertisement embedded in it. This has to be done super fast, in the blink of an eye. Actually faster than the blink of an eye. And this happens millions of times per second across the world.

Ali Dasdan:

So on the one hand you have advertisers, and there's a market in the middle called ad exchanges. That's where the bidding and auctions are happening. And the other sides, of course, the websites or the mobile apps where ads are shown and where the users are, right?

Ali Dasdan:

All of these stakeholders, they want to know whether they are getting their money's worth. Return on investment, right? Because of the speed of the operation, lots of money can be lost in the blink of an eye again. Super, super important that you actually understand exactly how your money is being spent and also adapt to it. Right? So if you feel like it's not going in the right direction, either manually or enabling some automated means, you want to enable it as fast as possible.

Ali Dasdan:

So real time analytics is key to that operation. You cannot, literally, be successful unless you have real time analytics returning those results to you so you can act on it. Also in an understandable manner.

Dhruba Borthakur:

Yeah. I mean, if you talk about online ad placement, that has to be real time, I guess. So there's a lot of data also involved. But when I talk about the real time analytics use cases, isn't it that it's a very complex system? So you need to spend a lot of time in operations in managing the system, or having people manage the system, or having tools to enable real time analytics. Is this a very complex problem for people to solve?

Ali Dasdan:

Yeah. That's actually a really good point. Because you have to serve users across the world, even speed of light is sometimes slow for this operation. So you have to have data centers in multiple places, multiple continents, multiple countries, and you have to make sure that the budget that the advertisers are putting, they are put into all these servers across the world as fast as possible. And if you are not doing a good job on the operation side... Let's say one of your servers are bidding on behalf of an advertiser and you got disconnected. So that bidding will happen and advertisers will lose money. A very simple example on that.

Ali Dasdan:

So operations is a hugely important. Again, real time analytics comes over here, and not only for engineers to act on it, but also for automated action. Again, because of the speed of the operation, you have to make sure that whatever you are putting in place can act as automated as possible. For example, for the bidding example I've given, you have to say what if a disconnect happens and how the system should behave. One way to solve this one is you can say, "I'm going to actually program the budget in such a way that a bidder is going to have a limited budget, and it's going to stop when the budget finishes, and it's going to solve if I get disconnected." There's no way that you can run away and spend all your money. So one example of that.

Ali Dasdan:

But real-time analytics over here, again, comes up not only for engineers and operations people to act on, but also for the automated means and read those signals and act algorithmically to prevent damage. Big damage can happen. Yes. A really good point.

Dhruba Borthakur:

Yeah. I guess real time analytics started in the ad placement business, but then I think you have other experiences you can talk about. Can you give us experiences from other verticals that you have worked with where real-time analytics is useful in a completely different context?

Ali Dasdan:

Yes. Another example, actually, is when I did, for a couple of years, retail in Europe. So the scale over here is huge, right? These companies operate in multiple countries, tens of billions of dollar revenue, and even an improvement being tiny, literally tiny, when multiplied by large numbers, it's huge. So to get that improvement, you have to be super timely. You've got to be timely, maybe because of two reasons. One is for your online users, as well as for your physical users. People don't have any patience now, right? They want to buy, they want to get what they're looking for right away. Online site is one click away. They can go to a competitor right away. Even for physical, right? Next door is another store. In every country, that's the case. So you have to be ready for it. You have to forecast that demand well in advance.

Ali Dasdan:

And the second thing is, again, you've got to do it well in advance because it's not simply you take from place A and put in front of the user. There are lots of steps in between from the place where a product is sourced until the user gets it. So you have to be literally predicting this well in advance. And real time analytics over here is key. Analytics in general, to figure out exactly the demand, but real time in the sense that basically you can act on it right away. If needed, for example, you got to move some products from store A to store B for the users, if demand on that store is going to be more. So this is a retail example. Again, real-time analytics comes super handy for the success of the company.

Dhruba Borthakur:

Yeah. For the retail use case, I keep reading about things where sensors are being placed in different retail stores, right? And they actually power all the data that is used to do some of these real time analytics. Any other use cases you can talk about based on your experience, about how real time analytics might be used in a completely different industry segment?

Ali Dasdan:

One more example I can give from the current company that I am working. So I'm leading engineer for Confluence Cloud at Atlassian. Our mission is to unleash the potential in teams, and that potential has to be released with the products that we have. I personally believe speed is the most important advantage. And this is not speed only in execution, but speed in decision-making, speed in failing fast and acting, agility associated with it. Basically all the way. And our customers are using our products to collaborate, to share, to get tasks done, literally to be productive all in all.

Ali Dasdan:

And real time analytics comes over here, because we have to provide that speed to our users. Imagine you are about to make a decision. You don't want to wait on it for days for somebody to actually provide the data input, for example, for a confluence page. The comment is there and you want to know, "Oh yeah, there was a comment." So that comment, let's say, is a counterpoint or a contribution to what you have written and you want to act on it straight away, so you want to be notified about that. Then real-time analytics comes over there to help you to be far more productive.

Dhruba Borthakur:

That's a great point, actually. Looks like to me that you were saying that you're kind of trying to give us examples from retail and then, say, you'r software as a service industry that you are currently working on. So one difference that I see between these two use cases is that for the retail use cases you gave us examples about, to me it looks like the analytics are being used to make the business processes better, and how can I make it a very core part of my BI fundamentals. Whereas for the other example you gave about the SaaS product that you are building, there it looks like your analytics is actually built into the application that the users are using directly. So it's kind of a direct impact or user-facing impact to your end users who are leveraging power from the analytics. Can you speak a little bit about the differences of these two?

Ali Dasdan:

Yeah. That's a really good point, actually. It shows also the power of real time analytics in different contexts for different stakeholders. So yeah, for the retail example, definitely it improves the benefit for the company. But again, for the convenience of the users. So the users are already in the picture. But for the productivity space, definitely the products benefit from it. But in the end, we want to make our users of our product far more productive.

Ali Dasdan:

And there's a human activity. Humans will collaborate thousands of years from now. And that requires basically that collaboration has to be sped up, because of the developments in the whole world and the technology and otherwise. And whatever we are doing, basically, with real time analytics in this context, again, not only improves our products, but product in this case. The direct benefit is for our users, for that speed in decision making, again, execution and productivity. Yeah. That's an interesting difference, and also shows the widespread benefit of real time analytics all the way from user site to customer site to stakeholder site to all the others. Yeah.

Dhruba Borthakur:

Any other advice you might want to share with engineering leaders or people who are building real-time analytics systems?

Ali Dasdan:

Yeah. I feel really lucky that I've been doing this for almost 15 years. As I said, I started at Yahoo, was tasked to create the internal data team. We did lots of monitoring, for web search systems, as well as data analysis, I worked in development. And not only at Yahoo, but other companies that I was part of. I was a witness to not only benefits of big data, as well as real time analytics in different contexts, as we have just discussed. This is almost like 15 years happening. Again, this is one data point. I am a single person. But I sincerely hope that whoever is still thinking about adopting real time analytics will do so with the urgency that it requires. And they should feel lucky now. When we were doing it, and you remember those days, we had to build many of the tools ourselves. Even in advertising, for example, context, we had to build a Kafka-like system because there was no Kafka. We had to build a Hive-like system because there was no Hive when we started.

Ali Dasdan:

But now not only these exists, but industry has even moved far beyond that. The only thing that the current users need to do, just adopt it. Literally make a decision between A versus B. So I would say, "What are you waiting for?" Right? Start now.

Ali Dasdan:

I would say, basically, not only start now, but focus on the outcomes and delivering value as soon as possible. So sometimes you might go into your model for let's complete our My Platform with all bells and whistles. I think that may not be the good approach, again, given the speed of the operation right now, how fast we have to operate. So start now. Don't wait on it. Start returning value to your customers. And as I said, you're super lucky that many things are readily available. And incrementally, given what's available on the market, as well as what's coming, we can definitely complete your platform for even bigger benefit.

Gio Tropeano:

Well done. Awesome. Thank you so much for your time today, gentlemen, and sharing your experiences with analytics, both real time and otherwise. Right? At different industries. I think it's super helpful. If listeners want to learn more about Atlassian, where do you recommend they go to learn more?

Ali Dasdan:

Yeah. Thank you for the opportunity again, Gio and Dhruha. It was a fun conversation, even though it's just a summary, brief. But I hope that users definitely explore further. And Dhruba and I had actually a tutorial in an IEEE conference. They can check it out. But for Atlassian, definitely our website as well as what we are putting out on the social channels, LinkedIn and YouTube and all that. Lots of more information to come. Our blog provides more information about how we're using data, but what also comes after data, Smarts and all that. So these are the resources, hopefully, they will check out.

Gio Tropeano:

Thank you so much. For our listeners, we at Rockset are building a real time analytics cloud-based platform that can add value to use cases that are similar to those that were discussed today. Thank you, once again, both for joining, and stay tuned for our next episode.

Ali Dasdan:

Thank you.

Dhruba Borthakur:

Thank you. Thanks. Bye.