Brad Zielke @ Target | Data Science Hangout

Transcript#

This transcript was generated automatically and may contain errors.

Welcome back to the Data Science Hangout. I'm Rachel, I lead Customer Marketing at Posit, and I'm so excited to have you joining us today. If it's your first time joining us, the Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. And we get together here every Thursday at the same time, same place.

But with that, I am so excited to be joined by my co-host today, Brad Zelke, Senior Director of Data Sciences at Target. And Brad, I'd love to have you kick us off here by introducing yourself and sharing a little bit about your role and also something you like to do outside of work.

Thanks, Rachel, for having me. Nice to virtually meet everybody. Really quick, so I'm a Senior Director of Data Science at Target. I work in the operations space, so my team supports various decision modeling capabilities across everything from vendors and our supply chain and transportation providers all the way through to what happens in our stores and also how we operate in our properties world. So when we talk about properties, we talk about construction, we talk about energy optimization, work like that. And so it's really fun space across Target because we actually touch everything that physically happens.

A little bit more about me, I've been with Target 20 years, so I have one job before Target and then I've been here ever since. And I'll get into a little bit of this as we talk more, but I think the variety of work in data science and retail is just fascinating. So across my 20 years, I've had opportunities to work in cybersecurity, work in supply chain, obviously stores. I've been able to work with HR analytics and other things, and it's just a really neat place where you can kind of cross domains in the industry as well.

And then what I enjoy outside of work, so I have four kids. Four kids keeps me busy, but I love coaching sports and anything athletic and outdoor. So I spend my time running. I'll run my sixth marathon this fall. My sons are swimmers and hockey players, so my winters are packed either in a pool or on the ice.

And so we spend, I think, as a leadership team, the majority of our job actually engaging, educating, and partnering with them to teach, train, and then drive kind of the focus on this work versus delivering it.

Measuring physical impact and feedback loops

Yes, so the feedback loop we often look at, right, as a combination of all the virtual records and events and the features that we train our model on and then score and then end up giving us output. But we also look at, like, what are the business outcomes and measurements that subsequently follow. We want to have effects and outcomes. And that's how it turns into the value for Target.

And so an example I'll talk about a little bit I can share. It's actually published in our Target blog, tech blog, if you want to go look at it. But we do something called inventory corrections in our stores. And so we use models that help us predict when something virtually is not there, even though physically we think it is there. And as we look at those events in our stores, what we can see is different things subsequently happen. So we'll take a virtual record, we'll correct it. And then often tasks and workload are generated from that. And so then we'll see, did the team members go get the thing from the back room? Or did they find it?

Sometimes we make mistakes. And so we'll recommend something that's not there that actually is there. And then the team member will go actually do something with it. And then it doesn't fit on the floor, which then means they have rework to do. And that's a cost to target. And it's a really frustrating experience for the store team member. And so looking at not just the precision of the model and the output and what value we're driving from that, but then the subsequent impacts of that has allowed us to connect back and also build trust because we care more about the whole business problem than just even making those nuanced decisions better across the whole value stream.

Just to give you some scale on that, we do that hundreds of millions of times a week, which every team member then has to touch those subsequent impacts. And it's fascinating to see this interaction and the costs and the interplay between the physical and virtual worlds.

Delivering bad news with data

A real life example of that is highlighted often in simulation. So we'll use and work across business teams. And recently, some of you guys probably know Target rolled out Starbucks as part of your drive up order. If you shop at Target, it's an amazing benefit because you can get your Starbucks in your car while you're also waiting for your groceries or anything else. I'm a big fan.

But as we were looking at and modeling out, actually simulating that for stores and the work there, what we found is the data told a different story than what our subject matter experts at headquarters expected to happen when team members or payroll was being used in stores. And I, you know, that wasn't a great story to be like, sorry, I know your process is defined this way and you expect all of this to happen in this way, but you're wrong. But I think the reality is we met them there. We showed them the physical. We actually met with them in stores. We walked through the virtual records, the physical actions and activities. We saw these things changing and evolving.

And what happened is we drew alignment between the physical and the virtual and highlighted those, I'd say perceptions that were incorrect and allowed data to kind of work and tell the story. And so, yes, it was bad news. But I think at the end, because we were all aligned and doing that work together, we were able to deliver it with some empathy and connection, which ultimately led to us advancing the overall solution and simulation for our stores, which helped them transform other decisions. And so they saw it not just as a serving bad news, but being a partner and then unlocking other capabilities that advanced it further.

What to look for when hiring junior data scientists

Yeah, I'd say the biggest thing, regardless of what's on the resume or experience is just the curiosity and willingness to go experiment and do new things. This space has changed so much and it's going to continue to change so rapidly. Their willingness to learn, to dive in with curiosity, build and connect and work with peers. The one thing I really, that's kept me at Target as long as it has, is the diversity of talent around me. And so I think their ability to be curious and then reach out to people and work with others and pull in diverse perspectives. Those are the things I'm often asking about in interviews. That differentiates somebody from what's just on their resume.

And so I think that type of, however you can show that in terms of your experience or the projects you work on or the type of products, models, things you're building, I think that just goes such a long way. Because rarely are you just doing it yourself and rarely are you the one who's done the initial research, algorithm development or model development that was published originally. I think there's a lot of just how do you, you know, do more of the applied side and work with teams and dive in to solve real problems.

Model governance and tooling

No, it's all in-house. I would love to say we have built this full like MLOps ecosystem out and we have, you know, some unbiased ways of measuring, scoring, doing all of that. I would say we do a much better job than we've ever done. We do have a data governance team that helps us at least with the data pieces and making sure, you know, what people are using and consuming as features and things are much more controlled than they've ever been.

But as we think about like model scoring and the outputs, like we do that individually. We often work within our teams to do the reviews. And I think sometimes that's good enough. But I think there's a lot of times where we could do better. And, you know, that's something that's top of mind for Target, top of mind for where we're going with the future of our science platforms, where a lot of those things are, you know, in our desired end state is built right into the patterns and processes that we're following as scientists. And so we're on a journey to do that.

I'd say our tooling, for those who maybe don't know Target, all of our internal tooling is almost all in-house, open source built. And so a lot of what it takes to stand that up is a combination of just the right engineering partners, the right scientists, and, you know, honestly, even the right leadership to build that governance and controls across. And we are committed to doing that, but it is a journey and it's going to continue to evolve.

You know, we use all different languages. It depends what we're building, what stage it's in. I'm a big, you know, the tool has to fit the right problem what you're working on. And so we're not going to bias that as much as possible. I'd say tooling that I'm most excited about we're investing in right now is kind of is the MLOps like platform for our scientists. What is the right things that we build there? What are the right, you know, feature store model manager, all of those things. So I think that's something I'm really excited about. But we will have to unify that for all languages and everything that we do because we do a lot of different work.

You know, my team spends a lot of time in both JVM based languages. So Scala as well as Kotlin are two big ones that we use for like optimization and OR work. But then we do a lot of work especially in our forecasting space. And then we do a ton of work in Python, obviously. And, you know, using all kinds of different libraries and things like that that are accessible to the team. So and then we have our we have internal tooling that's built now for like notebooks and dev practices and all of that. But we we make sure everything ends up in Git. It's just the way that we kind of govern and do manage all code and versioning and all of that stuff as well.

Understanding inputs, outputs, and asking why

I'll start probably even more simplistic. Like, I think the biggest thing I preach to most of my business partners and even team is, do you understand the inputs that are affecting your outputs? And do you really understand what went into those decisions or those variables that led to, you know, the other models or algorithms or all of that? And most of the time, what I find is a lot of hypotheses on what happened within those inputs. Well, somebody else owned that or somebody else knew that or that was already decided and I can't change it.

And in reality, there's so much of just, can you affect the inputs instead of just changing or overriding or, you know, biasing in between what you want to have happen? That I think it starts with, do you take the time to understand those effects of what happened upstream of you? Which regardless of science, I think is a really good behavior. If I start with just what has happened before I ended up having to make this decision or what I'm dealing with right now, I can usually start with a point of empathy to say, what did they deal with? What was constrained for them? What were variables that they weren't able to consider? What was the data they didn't have? And then I can work back even to their inputs and say, wow, if I actually change this other thing or help them understand that other thing, the effects all the way through this can be changed as well.

I would just ask, you know, why? Why is this a problem? Like, you know, I think asking that why and understanding it, there's oftentimes where people will come to us with problems like, we need data science to do this. And when you ask why and then you ask them to define it, sometimes it's not even a data science problem, right? Like you need a binary decision, you know, and sometimes like we can help you build that, like we can do that. But like that doesn't take any data science model. And so I think asking the why and then digging into like, what do you really want to accomplish? Sometimes you can really simplify the solution without even having to do the work.

Recommended books

My favorite leadership book is Team of Rivals. It's about Abraham Lincoln. It's pretty in depth. It's a deep read, but it is fantastic for anybody in leadership. It highlights how he built a team of diverse perspectives, challenging disagreements, even fights and arguments. And I just think it was, it's an amazing read. And I think you just learn a lot from his approach and understanding and it's good for anybody.

The other one I like for business leadership is Radical Collaboration. If you're into that, it talks a lot about how to move diverse teams forward, even through, you know, the mucky messiness of different personalities and other things to you.

I'd say the biggest piece of career advice, you know, I lean into empathy and humility. And so I'd say always assume you're not the smartest in the room. Ask a lot of questions and then be willing to adjust. I think as a leader, I have found more and more often I'm surrounded by diverse perspectives and understanding. And what I might start the team down or shift the team around is usually not where we're going to end up. And so I think it's great when you can step back, let other people lead you, and then you can evolve your perspective as a leader as well.

And so I'd say always assume you're not the smartest in the room. Ask a lot of questions and then be willing to adjust.

Yeah, there's a couple of things. So I talked about us being on a journey to experiment and moving faster. We have something at Target called our 50 days of learning. And so we make sure our teams have the space. If you think about 50 days in a year, it's 20% of your time. We try to make sure the team has that space to explore new ideas and also look at, you know, sharing ideas across the organization and sponsoring those things. So, you know, if you're in, you know, my team on the OR side, but you want to work on maybe more of a deep learning project or something like that, we'll use that 50 days of learning to have somebody go get experience in recommendation systems or in search or some other things like that.

But then also I talked about the product model. I think the product model and having that business partnership and expertise allows you to also index on the value that your team is working against. And so the more value you unlock, the more investment you get, the more creative and space you get to go work on those unique, exciting problems and research as well. And so I think when you blend that empathy and connection to the value they want to unlock, as well as your curiosity and connection to the team, you end up being able to do more independently too.

Absolutely. Thank you so much, Brad, for taking the time to join us today and sharing your experience. Thank you all for so many great questions too.

Brad Zielke @ Target | Data Science Hangout

Transcript#

Evolution of data science at Target

Data science in operations: use cases

Engaging business partners and communicating results

Measuring physical impact and feedback loops

Delivering bad news with data

What to look for when hiring junior data scientists

Model governance and tooling

Understanding inputs, outputs, and asking why

Recommended books

Career advice and sharing ideas