Rockset Episode 13: Data Engineering & Data Science Trends with Seattle Data Guy, Ben Rogojan

Gio Tropeano:

This episode's guest is Ben Rogojan, also known as the Seattle Data Guy. Ben is a data solutions consultant, helping his clients improve their data infrastructure. He specializes in data science and data engineering, was previously a data engineer at Facebook and Healthentic, and also has a weekly newsletter, SeattleDataGuy.Substack.com. Check that out. He's a longtime friend of Rockset's and has written articles for us that can be found on rockset.com, actually. Ben, welcome to the podcast and we're happy to have you with us.

Ben Rogojan:

Yeah, no, thank you guys. Thank you all. Thank you Nadine, thank you Daniel, and everyone else who's here joining.

Gio Tropeano:

Absolutely. Ben, I'd love to hear from you. I want to hear, since you've gotten your hands dirty and now you're helping a lot of companies, how has the data landscape evolved over the past three years?

Ben Rogojan:

Yeah. I think there's a lot, obviously that has changed a lot, that has stayed the same. To me, it's always interesting how much those are the same. I think the biggest change that I've noticed personally is a lot more consumption-based tools, which has allowed I think a lot of other companies at different sizes especially along the SMB side to come in and start looking at their data more. You don't need a $300,000 license with one of the bigger ETL tools or data pipeline tools. You can pay 50,000 or $40,000 a year for maybe something that's a little more, again, consumption based. So I think that's something that's been interesting. That's both on the data warehouse side, on the ELT side or ETL side. I think that's what I've felt has changed. I think that's why as a consultant, I've noticed a lot more work come up, because a lot more people are trying to dive into the space and I think that's what a lot of that is connected to. It's just the accessibility becoming more there, from a pricing standpoint.

Gio Tropeano:

Let's talk about hype then. We can talk about data trends. Which data trends would you consider to be overhyped and why?

Ben Rogojan:

Yeah. I think the one hard thing for me to answer that question is just because it's always going to land on where a company is at personally in their data journey. Like, what might be an overhyped concept for one company might be where they're at in their maturity level for another company. Machine learning is great if you've got your data in order, if you've got a decent pipeline developed on the backend that developed basically strong infrastructure that you can rely on, but it's not great for a company that's just getting started out. Then it's like, you're paying attention to what companies are doing that are basically flying rockets and you're over here trying to figure out how to walk. So I think that's generally where a lot of hype comes in. The problems that you need to solve are... You're reading an article and you think that's the problem you need to solve, but you're just looking at something that might not answer your need.

Ben Rogojan:

I think a lot of companies, like you look at FANGs and they're all doing a lot of cool things, but depending on even which FANG you're at, you might be doing different work, and depending on what team you're at, you're doing different work. Netflix has switched a lot of their workloads into more streaming kind of processing, but at Facebook we were doing plenty of batch processing, and that just comes down to what problem are you trying to solve? You're focusing less on the tool you're using. It's like, what problem are we solving? Okay. We just need this report once a day, once a week. Okay. That's a different problem versus Netflix that was like, "Okay, our day is getting too big or it's going to take too long a process. Let's do this live," things like that.

Ben Rogojan:

I think that's generally where I view hype. It's always going to be different on a company basis. Everyone's in different stages and setting up solid data infrastructure is always step one. Everything else after that is hype for you, if you don't have that. And then after that, you can start building off that. Start looking for specific use cases for what you need as a company. I think that's usually how I feel about hype in particular.

Daniel Lu:

I wanted to come in and quickly ask: You mentioned that a lot of companies may not be at the maturity that they need in order to take advantage of some of these data trends. Is there a specific data trend or two that a lot of people have a lot of excitement about that you've been hearing a ton about, that may not be overhyped per se, but there's just a lot of excitement and interest and maybe a lot of searches coming to your website or your YouTube videos that people are super-curious about?

Ben Rogojan:

Sure. I guess I would say that things like, again, machine learning and streaming are both interesting aspects and use cases that people want to do more. I remember when I interviewed at Facebook, I think they asked what I wanted to work on or what I was excited to learn at Facebook and I think I talked about distributed computing and streaming. The interviewer ended up asking, "What are some of the downsides to some of these solutions and trying to implement them?" I think that what he was reaching for was essentially that they often take a lot of maintenance or a lot of extra work, or if you don't abstract it correctly it takes a lot of extra effort.

Ben Rogojan:

So, things like machine learning, streaming, all these things, when they have a framework around it that makes it easy to utilize and easy to maintain are great, but when they don't, just become a massive pain for any company to manage. A lot of the big tech companies have solutions to solve this in terms of like, "Yeah, you want to do some sort of A-B testing or machine learning and have all the infrastructure around it. Oh, you just have to really develop the model and then you push it to this one place and it already has all the infrastructure around it and then you don't have to worry about that." I think things like that, people are definitely hyped about it. They're interested about it. They're trying to figure out how to do it, but there's definitely a lot of gaps in terms of getting to that end result, and effectively. Yeah.

Nadine Farah:

I have a question for you, Ben. You create a lot of content for those new to the data engineering space. What is some advice for folks new to the data engineering that you can give? Maybe like top three.

Ben Rogojan:

Okay. That's a broad question. Any advice? Are we trying to get career? Are we trying to get tools? I guess, would be curious where-

Nadine Farah:

I'd say one career and two tooling.

Ben Rogojan:

Yeah. In terms of career, I think breaking into data engineering is always hard. I think if you're trying to break into data engineering, the best advice I'd say is don't get too stuck on either a company or working even as a data engineer to start. There's plenty of other roles that are adjacent to data engineering that you can then parallel into more of a data engine role.

Ben Rogojan:

I often tell people that to some extent, obviously titles matter, but to some extent, just making sure you're doing the work behind the title is often very important. So if you can work with a data analyst and start doing data engineering work, it's much more likely that you can pivot into data engineering, either at your next company or maybe the next time you're talking to a different manager inside of a larger company. I think that's advice there.

Ben Rogojan:

In terms of tooling, I guess even there it's like, what kind of advice? My usual advice is pick the tooling that makes sense for the project. Understand that if you need to manage a thousand pipelines, you're going to need very different tooling than two pipelines. Those are two different solutions, two different really problems. In order to scale a thousand pipelines, whatever infrastructure you pick is unavoidably going to take almost just one person to manage the infrastructure component alone. Even if you pick some more of the managed services out there, there's just likely things that are going to go wrong and then one person to develop on at a minimum, if not possibly more. Whereas with two pipelines, maybe you can get away with still doing something like CRON and a Python script. If that's your only need and you don't have a bigger use case than setting up all this extra infrastructure, it's great, but again, it requires a certain amount of overhead regardless. Trying to pick the right solution for the right problem I think is always important.

Ben Rogojan:

I think the other advice is trying to avoid the shiny new thing too much. A lot of problems can be solved by a lot of basics in data engineering first and then as problems evolve, that's when you start seeing the need for new tooling. Once you have a use case, you've well developed it, you've implemented a solution that you can rely on, then you often can look and see, can I improve this? Can I make it better?

Ben Rogojan:

Again, there's an article about Netflix and how they had to do that with their OLAP to streaming process where they realized, "Oh, okay. What we currently have is going to not work within six months to year. We need to switch." So then they pushed into more of a switch. I think that's what most companies do. You do something, you make sure it works, it's solid. And then eventually, "Okay, we see it's not going to work for X reason. Now it's time to switch."

Ben Rogojan:

Chasing the hype just for the next tool without actually having a need, it's one, very expensive on a company for multiple reasons, right? Whatever that tool's going to cost, whatever the re-engineering, all of it's going to be very expensive. Oftentimes it's better to just see. Like, "Hey, make sure that this tool works," and then if the use case is growing or whatever, then you can expand upon it. The whole MVP to more final solution kind of process.

Daniel Lu:

That's super-interesting. Are there any tools that you think most data engineers should be at least considering or looking at if not implementing? I know there's again, that maturity curve that you mentioned, where step one is pretty basic in getting your house in order before you want to start going after the shiny new things, but are there... I don't want to call them shiny new things, but are there tools that you're thinking, "Oh, not a lot of people know about this," or have a false understanding of what they actually do and the power they have, and maybe you should do a little bit more digging here?

Ben Rogojan:

I don't know if there's any specific tools that I would say aren't necessarily recognized. I think what's interesting to me in the data engineering space is now that I've been in it for longer, I think I have a broader understanding of what exists, but even when I was coming up, I think I started with SSIS more for automated jobs. Didn't really realize that that was connected to ETLs. Just realized this moves data from one point to another.

Ben Rogojan:

Eventually, the next company I worked at did PowerShell and they had developed their own framework around that to manage a lot of the SQL and stored procedures that were in the data warehouse. Then from there, I learned more about Airflow and just progressed through that.

Ben Rogojan:

I think it's sometimes that natural progression, just because depending on the company you start at, you're going to see a different set of tooling. I think if you're just starting out, Airflow is still I think a great place to start. You're going to find a lot of problems. You're going to find a lot of issues. But then once you find all those problems, all those issues, you also find that there's a bigger community around them which helps solve some of those problems.

Ben Rogojan:

Then I think it gives you more of a insight into maybe picking a different tool. Like if you're really against, if you really don't like Airflow, there's at least three to four other, at this point, newer solutions that are coming into the space, trying to approach a lot of problems that Airflow has. I think that's a great place to start though, because it's going to teach you a lot about what we're trying to do, like the basic concepts. Like scheduling, dependencies and all these more complex concepts that expand out.

Ben Rogojan:

When you first start, if you're only doing something on CRON and some Python scripts or some PowerShell scripts, you're not necessarily thinking about dependencies in the same way. You're having to think about it very manually versus very programmatically. That's why I think, I usually say people can start there and it's usually fine.

Ben Rogojan:

Data warehouse, I'm pretty open to most. I built data warehouses on Snowflake, Postgres, BigQuery and a half dozen other tools. I'm not too keen towards one or the other. At this point, it just is going to depend on what your use cases are, how your team is being supported and what processes you're going to develop, because any of those tools can be basically... What's the word? Almost become just a place to put data and not really become a data warehouse. So at the end of the day, it's going to be less about tools and more about best practices and processes that you put in place. So, that's how I look at that.

Daniel Lu:

Yeah. I love that. I actually just was watching your video about Snowflake recently. It was a really good introduction on what it is. I think there's a lot of curiosity and interest in the tool, which is great. I'd like to dig a little bit deeper into data warehouses and you said you had a lot of experience with Snowflake and others. What are your thoughts around Snowflake? The pros and the cons, who is it good for? Maybe some limitations of the technology?

Ben Rogojan:

Yeah. Snowflake I think usually fits really well for people who maybe don't have a huge technical team to manage every aspect of their data warehouse. I think the downside to a lot of tools that make our lives easier is that we maybe forget about process. We maybe forget about best practices, because it's just like, "Oh, let me just put whatever data here into my raw data layer," and then there's no real thought about how it actually gets modeled or processed from there.

Ben Rogojan:

I was actually just reading an article this morning. It was just more of a quick, five paragraphs from Bill Inman where he was doing a little Snowflake critique, and that was kind of his point. I think it's kind of accurate. It's very easy to just dump data into Snowflake and then just have your own little siloed datasets still. There's no real connection between different datasets. That I think always generally comes more into to the process side of things. Like, okay, well, we're putting this data here, but how are we going to connect it all? All these other discussions that you would've had to have prior, or planned out a little more. That's more around process than it is around tool though, to me. Yeah.

Ben Rogojan:

It's more like, some tools are just so easy to use. You combine that with FiveTran or something like that, that process becomes secondary because it's just like, "Oh, we've got the data here. We can answer our very siloed set of questions." Maybe it's just on my HR data, but then once you start asking questions like, "Okay, what about my HR data versus my operations data?" You're going to start finding issues that things don't connect and things don't... There's no magic pill to fix that.

Ben Rogojan:

There's certain tools that can manage certain portions of that like customer, whatever. CDP type tools can manage that side of it, but then internally how do you manage all of this? That was something that I think I would say I was spoiled at with Facebook. Everything is so integrated that there's always some key that connects every system. There's always something, so you can get away with treating things more like what people call a data mesh, because there is some way to connect everything at the end of the day. I don't think a lot of companies have that because in order to do that, you have to basically develop the software itself and in turn, develop the processes around that software.

Ben Rogojan:

Yeah. Snowflake's not going to fix that, right? That's going to come down to people and that's going to come down to processes more than it comes down to technology.

Nadine Farah:

If you could warn or educate your audience about one thing when building data apps, what would that be and why? Maybe it could be something that you've learned as you grew to a data engineer and a consultant.

Ben Rogojan:

Sure. By data apps, do you mean like dashboards, like applications that are built off of...

Nadine Farah:

Like embedded analytics and operational analytics.

Ben Rogojan:

Sure. I guess first if we're just talking more about dashboards, I always like to tell people to keep logic as much in one place as possible. There's a lot of different places you can put business logic or layer in all of that. The more you push it into multiple places, the harder it becomes to track what happens to data.

Ben Rogojan:

Obviously, there's trade-offs for that as well, but the more you can shove it into one place, the easier it is to QA, the easier it is to manage. Obviously, there's some niceties to having raw data or more raw data that analysts can work with and then process and then create further layers from there, but it's also a great way of getting inaccurate data very quickly as people slice and dice in multiple different ways as they're building systems.

Ben Rogojan:

I'm trying to think more on operational analytics. There's some similarities there where it's like, making sure you're processing whatever analytics or segmentation or something like that in one place is often great just for QA.

Ben Rogojan:

I'm trying to think if there's anything else I would add to that. Any advice? Yeah, I think that's my general advice. I'm sure there's other people who maybe speak a little deeper on it, but I think that's my general advice there.

Daniel Lu:

Makes sense. You have such history and a lot of experience doing consulting then. You've run into a lot of different problems, and also just being a data engineer at Facebook and others. I'm curious. What are some of the pain points that you've encountered? Whether it's building data pipelines or whatnot. Earlier, you had mentioned a little bit about understanding the process and making sure you got that figured out, but I'm trying to think about our audience who are perhaps building out some of their data architecture right now and they're looking at you and you're too busy to help them because they've reached out and you don't have the time quite yet. What are some things that you might recommend to them as a starting point, or some easy pitfalls that you think people should be looking at?

Ben Rogojan:

Yeah. I think there, more starting on pain points and talking about pain points that I've seen with customers, I think generally what I've seen there... One of the probably main reasons I end up getting pinged is due to the fact that maybe things were built in the past three or four years ago and then whatever team built it is no longer there. Like, it's been hobbling along for the last two years, year or something, and no one really fully understands what's going on. Maybe it's kind of producing what they want. Maybe it's not producing anything that want. Then I have to come in and play this reverse engineering doctor and for better or worse, try to fix as much as I can. There's a lot of untangling of systems.

Ben Rogojan:

I think generally my recommendation to most people is try to build as simple as possible as you start. Just build a flow that works, make sure the data's accurate, make sure you don't go crazy early on. Pick your few use cases that you really want to build out, build those out. Again, like I said, a process that you rely on where it's like, okay, if I want to get data from point A to point B, here's how I do it. Here's how I make sure the data's accurate. Make sure you have that kind of culture developed and then you can start expanding on it, and make sure you start delivering at least some baseline reporting to whoever your target audience is. Not necessarily so fast that you're wrong, but at a speed that makes sure that they start getting confidence.

Ben Rogojan:

The hardest thing I think today, especially with people wanting data tomorrow or yesterday, is making sure that they understand that it still takes time. We're still in a world where making sure data's accurate is isn't always easy. There's still a lot of business logic that goes in some systems. Some companies are using systems that are still, you're pulling from IBM mainframe. Some companies are using cloud, some companies are using just on-prem systems. It's just so many different levels of types of systems you're connecting to and things of that nature that you need to be wary of the fact that data can still be wrong, even though you've pulled it. You maybe didn't include some logic. You shouldn't assume you understand the data just because, or assume that data's 100% accurate just because you pulled it directly from the data source.

Ben Rogojan:

There's always some kink in most datasets. Maybe how it's appended, maybe how it's updated. If you don't understand those things when you start out, you're going to get wrong data. As much as it's very easy to pull datasets in these days with tools that have direct connections to all of our various sources and then push it to Snowflake, there's still a need for a lot of patience and understanding. You've got to understand the data. You can't just pull it in and be like, "Okay, here's the data. Use it." You still want to make sure it's accurate, reliable, things of that nature.

Nadine Farah:

There's a tweet that Sarah Catanzaro, she's Head of Data at Mattermark, she had a tweet a few weeks ago and she said something to the fact of, it's clear that the modern data stack built on the corporate data warehouse tools for ELT, data modeling, metrics and reverse ELT operationalization is becoming a standard. How is this architecture or are these workflows fundamentally wrong? What is your thought around this tweet? Or do you have a comment on that particular tweet?

Ben Rogojan:

One, I really enjoy most of her tweets. I don't think I'm going to pretend to know fully, like it's like, was this... Partially it's challenged. I think this is something that I've been trying to do too recently. There's been so much in terms of... How would I explain this? There's been so much in terms of siloed thinking and then also thinking, or tools that... I actually posted about this recently where I was kind of making a joke, but there's a lot of tools and best practices recently that are being either pushed forward by VC funding, essentially. Obviously there's incentive for this tool or this process or this system to be accurate because that means someone on the other side makes a great amount of money. Then there's also, maybe there's four articles about this concept that came out two years ago and people are trying to figure out how to do it because they it's a good idea.

Ben Rogojan:

I think we're just in a situation, personally, where we need to... There is this need to start asking, is all of this stuff that's come out in the last three to five years stuff we should be doing? One, a lot of it is like the wheel of... Referenced in Game of Thrones, where it's like this is on top and that's on top. We're constantly just doing the same thing over and over again, maybe giving it new names. I think there's that part of it.

Ben Rogojan:

There's the aspect that a lot of this has just been all pulled apart versions of more complex or bigger tools, like in our Oracle and bigger vendor days. I think it's just, we're all at this point where we realize we need to question if all of this stuff even makes sense, how we're building this makes sense.

Ben Rogojan:

It's about, I think whittling out all the stuff that's just noise and really figuring out, okay, what actually makes sense? What's the real way of building these systems? I think that's personally, maybe what she felt. That's how I think I translated it as. This is something I've been trying to do more and more, personally. It's like, how do we really start questioning everything that we've been told, or even we believe around data from the last five years?

Ben Rogojan:

Prior to that, we had a way of doing data. We've had three decades where people were like, this was the way you did data. I think that's still kind of the same on the enterprise level. Again, I think the big change has been more on the SMP and mid-level market, where now suddenly it's like, oh, we can all start doing this. It's not that expensive until you suddenly sign seven contracts that are all worth $50,000. Then it's a different conversation. But prior, at least at a base level, it's not that expensive and you could possibly at least get started. Yeah, I think that's my own thoughts now. It's like, okay, what actually makes sense here? What doesn't make sense here?

Ben Rogojan:

There's a lot of cool names, there's a lot of cool terms we can use for things, a lot of things that are around marketing, but what actually makes business case sense? This is why I often push people, focus on your data pipelines, your data storage, your reporting, and then some data quality and observability. Other than that, all that other stuff is going to be fluff around that. Until you can get those other four things right, a lot of that other stuff is just adding on expenses, adding on management, adding on maintenance, that you might not even be in a place to think about. Yeah, that's how I feel about that tweet in general.

Daniel Lu:

Any topics Ben that you think we should continue to have to push the data ecosystem forward? I'm seeing your tweets, I'm seeing on LinkedIn you have such provocative questions for the community to try to get people to discuss these things, to reframe the conversation, to really rethink whether it's marketing fluff versus reality and something that's needed. What are some topics that you wish the community should be talking about a little bit more?

Ben Rogojan:

One, if you think my tweets are provocative, there's definitely plenty of other people, or LinkedIn posts. There are far more provocative tweets in LinkedIn posts. I try to stay pretty tame. I'm too much of, I think a friendly people pleaser to want to poke anyone.

Ben Rogojan:

I'm trying to think. I think a lot of things are being sussed out, for example. Even the reverse ETL whole game and that thing. I think a lot of those tools are starting to integrate more with the EL or the ETL tools, or the tools that are more going to almost become this full-sync style tooling. So I think that's something that will slowly start to make more sense, because if you use something like Informatica, that already does things like that. It does both sides. It's not a separate game. So I think things like that will make sense.

Ben Rogojan:

I would love to hear more, I think, personally about the metrics layer and people's thoughts there, like where things should exist. I'd also love to hear more... I think I saw something from Firebolt this morning where they were talking about raw layers of data and making sure people have access to that directly and things of that nature. I think that's something that's at least worth poking at. Yeah. I think those are some things.

Ben Rogojan:

There's some data storage things, but I think again, those are getting sussed out as well, like data manage, data warehouse, data lake house. I think we're going to keep seeing people try to make it work for some companies, and I assume for other companies it won't work. I think that's how I feel about that.

Ben Rogojan:

You could argue that to many degrees, Facebook had this data mesh, but we just never called it that. So that's why it's like... I think it's going to work for some companies. I think it's not going to work for other companies, but I still love to hear different perspectives and hear where it's worked, where it hasn't worked. Yeah. I think there's a lot to glean from everyone.

Ben Rogojan:

I feel bad. I'm going to try to start having some debates about this and I say I feel bad because I was like, oh, this is a great idea. I did talk to one person prior who gave me the idea, but then I realized that... I don't know if you know Prukalpa. She's been doing this for a while already. I think she's got one debate coming out here on the metrics layer. But yeah, I still want to do them, but I was just like, ah, I thought I had a little bit more of an original idea. I think we're all in this space where we're like, okay, we've got a lot of ideas out there. Can we start picking them? Because we're all tired of hearing different versions of different stories. Yeah.

Gio Tropeano:

Nice. Well, we started with hype. Let's end it there on forecasting and trends. Ben, is there anything else, any other thoughts that you want to share with the audience that's top of mind before we close?

Ben Rogojan:

Yeah. If we're talking about trends and where things are going, I think consolidation is going to and is occurring. We're seeing different acquisitions occur right now. We're seeing tools trying to provide more services, more full services. I think it's unavoidable for multiple reasons. It's better for the products themselves because they hopefully reduce churn that way, as well as it's better for the customer who doesn't have to now have 10 contracts.

Ben Rogojan:

It's always a trade-off, obviously when it comes to bundle and bundle whatever, but I imagine it's going to be at least some mini bundling here and there where it's like, okay, this tool's at least a little fuller now. It's not just one tiny small piece. So I'll be curious to see about that.

Ben Rogojan:

I'm trying to think what else. Yeah, I think that's the big thing I see for the next two years. I think we are going to start seeing more discussions and more acquisitions and more consolidation in between companies that are different as well as in between companies that are the same thing. Then, between the need for talent as well as the need for customers and growth, I feel like that's an unavoidable thing. We're just going to see more and more things swallowed up.

Ben Rogojan:

I think it'll also be interesting to see if we'll start seeing more and more outages because that's something I keep thinking. It's like, well, the more and more we have JIRA things that happen, the more and more we're all in one system, the more and more likely someone runs a bad script and suddenly 50% of the world is down on something. I thought again, that was an interesting I think story. I don't know if anyone's read The Pragmatic Engineer, but his takes on that and his breakdowns on that are always great. Yeah, I think that'll be something interesting to see if we have more of, as more things are cloud. It's almost unavoidable. It's just more impactful when these issues happen.

Gio Tropeano:

Ben, thank you so much for your time today and your insight. Nadine, Daniel, you as well. Thank you so much for being with us. For those of you who have joined, you can learn more about Ben and his work by subscribing to his newsletter. That's The Seattle Data Guy at Substack, and you can also follow Ben on YouTube. You can just search for Seattle Data Guy, follow him there. You've got some great videos there.

Gio Tropeano:

Don't forget to subscribe and share this episode along with your thoughts. The Rise of Real-Time Analytics podcast is brought to you by Rockset. At Rockset, we've built a realtime analytics cloud-based database, helping businesses easily leverage their application data in real time. Check us out at Rockset.com. You can try us out for two weeks with $300 in free credits. Thank you again for joining and stay tuned for our next episode.

Ben Rogojan:

Awesome. Thank you, guys. Bye.

Nadine Farah:

See ya. Bye.