Josh King @ The Knot Worldwide | Creating your data and analytics roadmap | Data Science Hangout

Transcript#

This transcript was generated automatically and may contain errors.

Hello, everybody. Welcome back to the Data Science Hangout. Hope everyone's having a great week. If we haven't had a chance to meet yet before I'm Rachel and I lead our pro community at Posit. So the Data Science Hangout is our open space to chat about data science leadership, questions you're facing and getting to hear about what's going on in the world of data across different industries. And so it happens every Thursday at the same time, same place. So if you are in the future on YouTube, and watching this recording, just know that we'd love to have you join us live too. And you can use the link to add it to your calendar in the comments.

But together, we're all dedicated to making this a welcoming environment for everybody. And so we want to hear from everyone at the hangouts, no matter your level of experience or area of work. And so it's totally okay if you just want to listen in. Maybe you're eating lunch or going for a walk. But there's always three ways you can jump in and ask questions or provide your own perspective on topics too. So you could jump in by raising your hand on zoom, you can put questions in the zoom chat, and you can put a little star next to it if you want me to read it instead, or else I'll just call on you to add some context and introduce yourself. And then lastly, we also have a slide of links where you can ask questions anonymously.

Thank you all so much for being here today. And thank you so much, Josh, for joining. I feel like I wanted you to join us here for a long time. And I'm so excited to have you here today. Josh King is director of marketing data science and analytics at the knot worldwide. And Josh, maybe to get us going, it'd be great to have you introduce yourself and your role and and also something you like to do outside of work.

Sure, sounds good. And sorry that it's taken so long for me to actually land on the show. I feel like we've been playing a game of cat and mouse throughout that. I think it was since like Domino's, maybe. Yeah, it's been since Domino's, which was a couple of years ago. So yeah, I'm the director of marketing data science and analytics, or easier to say DSNA, or another way is saying marketing science. So I've been with the knot worldwide for about seven months, but it kind of really feels like it's going on seven years with just the pace that we move and the amount of cross departmental collaboration we have.

So we have a specific marketing focus in our group that's pretty new to the team and the team is pretty small right now because of that. So we have three analysts on my team, one data scientist. I'm also hiring for another data scientist. So if you're in the market, definitely feel free to reach out. So there's one position on my team focusing on marketing. We have another senior data scientist that's focused on operations based out of our Barcelona office. Mine can be remote anywhere in the US. We also have a product data science manager that we have open and also director of data science, basically my counterpart, but in operations that's also based out of Spain. So we have a lot of open roles right now. Feel free to reach out to chat about those.

And so my team at TKWW has an embedded nature and we partner really closely with everybody who rolls up to our CMO. And this is across the entirety of the world. So we don't just have like a US focus. We're a global company and we have a global purview within marketing data science, which is pretty nice. So I, prior to coming to TKWW, some people internally call it TKWW, which is even shorter and easier to say. Prior to coming here, I had about a year at a series E startup called Policy Genius, where I was basically doing the same function. That was a lot of fun, very fast moving, smaller company. And then prior to that, I had about four years of experience doing very similar thing at Domino's. So it's very different going from trying to sell pizza to trying to work in the business of love, which is pretty nice.

So one thing I like to do outside of work. So I love hanging out with my wife. She's also very close into the R community. She used to use SPSS for a lot of her analysis. She's an academic, she's a professor at Duke. And I got her to transition over to R now. She jokes around that she has a self-written package that she calls helper whenever she needs my help to help debug some of her code, which is pretty nice for her to have me as an asset like that. Also have two kids, a five-year-old son, almost 11-month-old daughter, and love to play hockey outside of work. So I play hockey one night a week in an adult league. Big fan of the Carolina Hurricanes, if anybody else here is in the Southeast. And yes, Louis, I think my kids will use R. My son already says that he wants to be a data scientist at five, which is pretty awesome.

But if you consider fomenting empiricism, it's like within whatever workplace you're in or whatever setting you're in, just driving the thinking of having data and empirical evidence to support your decision making.

I think it really goes a long way for, as I'm talking to my team or talking to our business stakeholders to think through, we may have a gut feeling about something, but how can we think a little more analytically minded and try to challenge ourselves to see what the data would show us and try to drive a bit more value in terms of what's capable there, rather than kind of just taking things status quo.

Yeah, I think generally it's more of a mentality that I try to push with my team, especially as we get requests from stakeholders. So one thing is I try to push an embedded nature, especially within the analytics team. The data scientists team are more focused on like a project basis. The analysts are really embedded within the teams that we're working with. So this could range from brand media, social media, content, SEO, CRM, performance marketing. We work with a lot of different marketing departments and the analysts are basically aligned to some of these different aspects to really be embedded within those cultures and not just an outsider kind of looking in to try to get context after the fact when they have a question. So as they're embedded, it allows for us to think as a team rather than say, hey, I'm a marketer. Hey, data team, I have this problem. Can you solve it for me? It's more, this is a problem that we have as a company. Now, how can we think together to better ideate this overall business problem?

So the analysts really have a seat at the table to have a lot of that ideation to identify that we even have a business problem to begin with and then take that business problem and translate it now ourselves as the data team into a data problem and then build a data solution to solve that problem. But I think a lot of it is just coming back to having this mindset of making sure that we're keeping focused on the end goal that we're looking for and not necessarily say, hey, data team, can you go pull this data for us? The gut reaction is, sure, I can easily do that for you, but what's the intent of the question that you're trying to solve or the problem that you're trying to solve? And let us think a bit more holistically and as a partnership so we can think to try to drive a bit more solutioning and decisioning behind that because we, as data science and analytics, already have that mindset and know a bit more of what data can do.

We received one question early on from one of our stakeholders when I came here, and it was, hey, can you go do a segmentation for us or can you build some clusters for us? And when he's like, I mean, yeah, we could, but what are the hypotheses that you have that you're trying to solve? What is the business problem you're trying to solve? And I didn't really have the context of the business at the time because I was only about a month in, but it really comes back to trying to identify what is it you're trying to solve out of this? If I deliver some segments for you, how are you going to use that to drive decisions? Are you going to have personalization? Is it going to change your audience strategy? Is it going to allow us to do some type of testing to make sure they're relevant? So I think just continuing to have those conversations with stakeholders allows it to be a bit more focused and less prescriptive from their part, and more so thinking a bit more generally about what can be done.

Media mix modeling and privacy

Yeah, so within marketing, the MMM, this is a project that I kicked off with a data scientist that I hired back in November. So we're still pretty early on building out our initial proof concept for a single outcome variable for a single brand. I think as we continue to expand on this one and get a lot more iterative and bring out a bigger scale to a global model and also on different outcome variables, definitely awareness is one that's top of mind for us. So we have a significant amount of marketing spend right now that's focused on really capturing user intent, that they're already in the market and they're searching for something that's very relevant to our company. But considering a lot more, how do we start scaling to really drive awareness across those engaged or pre-engaged couples who will be engaged soon? So that we're already top of mind when they get to that stage rather than after the fact and find us after.

Yeah, so MTA stands for multi-touch attribution, which is a means of aligning all of the marketing touch points that a user could have before they make some action or they don't make an action. And so, yes, definitely privacy constraints are a significant blocker there. Even thinking back to, like, 2018 when we were doing this at Domino's, it was a significant blocker within the social media channels specifically that are very walled gardens. You can't align on an impression level basis who's seeing your ads. So it's definitely a challenge there. One way that we're trying to work around that is by rather than building out a pure, what I would call MTA, which is generally using some type of Markov chain model that is impression level, that we're building out more of a click-based MTA, which aligns for a specific user. Again, this kind of comes down to our identity resolution and how well we can align any specific click to any specific individual for any of the clicks that they're making.

The other aspect there was getting into the MMM. And this is kind of like, I'd say, over the last four or five years, there's been a bit of a resurgence in an MMM where prior to that, everybody was saying, oh, MMM's dead. It's such an archaic method that's being utilized to optimize marketing spend. And like, yeah, it was developed in the 60s, but it's still highly relevant, especially now as privacy concerns are coming out, especially within Europe where privacy laws are already well ahead of what they are in the U.S. So the benefit to an MMM is that it doesn't require user-level data to do this. It's very looking at it from econometrics perspective and building out a model that can be very focused across the entirety and not just within digital channels, also within, like, TV and terrestrial radio and billboards to get a read into how all the marketing channels kind of work together.

They're not separate teams. It kind of depends on what people are used to using. And that's definitely a struggle to consider when you want to do quality assurance on the code basis, but that's being written. If one of the data scientists is an expert in the problem that you're trying to solve or has some knowledge about it, but they know Python and I'm coding in R, then it definitely becomes a challenge. We're currently working on building out our overall data roadmap for our department on what is the appropriate means and scenario here for getting it and being able to have this cross-code capabilities. I think one of the areas to do is just have some means of identifying who would be relevant to look at your code base, even if they're not necessarily on the same team. I've been at some companies in the past that were significant in size where there was basically an internal R user group and an internal Python user group, which allows for a lot of that language-specific collaboration and discussions.

Creating your data and analytics roadmap

Josh, I just picked up on a key word you said there, is your data and analytics roadmap. And we kind of have a similar process. We manage two separate roadmaps. And I'm just wondering how logistically you manage that. Like what goes into creating that? How far out do you go? How far out do you try to plan? Does it cover all types of data work? So is it just more like a data science roadmap? Or do you include data engineering projects, BI, you know, ad hoc things?

Sure. Yeah. So I'm actually in the midst right now, as in like the hour before this meeting, I was doing this on getting a bit more refinement on my roadmap, because we're having a data heads offsite next week in New York to kind of talk through holistically about each of our pillars, as well as the overall data work. So it's top of mind for me. So pretty, pretty relevant timing there. So one of the things I like to come back to is the CTO of my last company, he had this really good framework for kind of thinking through this, and it's to break it out into your mission, your vision, your maturity model, and your strategy. And that kind of allows a bit. And so if you want to look it up, he has a great article on Medium. His name is Dave Kaplan. And he has some good articles on that. But if you search for those keywords and Dave Kaplan, there's a really great posting on it.

This kind of gets into a good framework for building out and identifying like, what are the different aspects of a strategy that you should consider, and having a clear alignment for what that will look like. And generally, a mission is like a single sentence for why your team or your department or your company exists. And your vision is, what does it mean to succeed within that? So like, if I'm thinking through marketing DSNA, I'm thinking through like, what does it mean to have a best in class marketing data science organization within like across relevant brands? And then your maturity model is, where are you along that map from like very basic or early foundational building into like, I'm best in class. And then your strategy is how you get from where you are now to where you want to be in your vision. So this really aligns a good framework for what it is that you're building out.

And to the other part of your question, like, does it cover all data work? So I have data scientists and analytics that report into me, but there's also some of a dotted line from all of our data engineers that focus on marketing problems. So that can include BI, that can include data operations, data warehouse, tracking architecture. And as we're thinking through a bit more holistically in terms of all of our data functions that are supporting marketing, I identify like, what do we need to drive decision-making for our marketing organization? And not only like, how does data science and analytics fit into that, but like, what is the infrastructure we need from a data engineering perspective that would allow us and enable us to be able to do this work? So that it's not just focused on that one, but then yes, it's not necessarily my vision or my strategy that would get us there, but working really closely with our head of data platform so that my vision and his vision are really aligned together and thinking through a bit more holistically as we're building that out. And I think the other question was like, how long is it? So my current focus is on building out a one-year, a three-year, and a five-year roadmap to really have some like intermediate stages along the way that we're getting towards.

One of the things I like to come back to is the CTO of my last company, he had this really good framework for kind of thinking through this, and it's to break it out into your mission, your vision, your maturity model, and your strategy.

Gotcha. That's impressive. We're working on the one to three-year. We haven't got to the five-year yet, but that's pretty cool. I think like it's a bit of a struggle because especially within data science, like it's so rapidly evolving that it's hard to think through like what's even going to be the atmosphere and the environment in five years from now.

The other struggle we seem to see is the need for some of the foundational data engineering work to come first, to enable some of those capabilities, right? We've got a million requests to add chat GPT to something and we're like, well, we need to build some data infrastructure first. So it's not as easy sell, I think, to do that foundational work first.

Yeah, it's definitely a bit challenging in terms of orchestration to make sure that all those pieces kind of fit together. So as I build out my roadmap, there are some things that can be done now based on the infrastructure we have and some things that like, yeah, it'd be great to do it now, but really this would need to be on the immediate roadmap for data engineering to build out what we need in order for us to do that even a year from now.

Experimentation and cross-departmental relationships

Yeah, I guess just asking, are those objectives your team thinks about? Is that kind of what you mean when you're doing media mix modeling? Yeah, so within marketing, the experimentation is really being done somewhat siloed within the marketing channels currently and that like Facebook or Google, they have their own abilities to do a lot of testing. We can do some analysis on the back end to do like more down-funnel experiments. So we're getting into now, this is one of the things on like the next couple quarter roadmap, is to build out more of a geo-based experimentation framework, which allows, like it's not randomized, but allows us to have a good amount of insight for a specific geography or a sub-geography and then have a relevant control set to compare against. I say a lot of those RCTs happening within our product side and getting through people who are actually within our platform to try to drive a lot of that experimentation via A-B testing, and it's not even necessarily A-B testing, like multivariate testing as well, to really drive a lot of that like personalization, what works well as we're making changes to the website.

I know a bit earlier you talked about how you work across, like cross-departmental partnerships, and a lot of times we'll talk about like how you build relationships across the company in this data science hangout format, and I'm just curious what that looks like at The Knot.

Yeah, so I try to make a specific intent to, like especially since as I was going through my first six months, really building out a lot of the relationships with the people that I'd be working with or like partnering generally close with, and a lot of that is just like communicating on Slack to say like, hey, what's going on, or having regular touch bases and one-on-ones. There's also, if you consider within whatever organization you're in, if there's ever like lunch and learns or any aspect to hear from others' perspective, go to those and ask questions. I think like that's one way just to kind of get out there and start to build some of those relationships is by just being there and hearing other people's perspectives, and if they ask, do you have any questions, like try to have some type of question that you're able to ask because I think that's just a good way to try to get your voice heard and start to build a lot of that relationship.

Career advice

A question that I always like to ask Josh is, is there a piece of career advice that you've ever received or maybe that you've given that stands out to you?

Well, I kind of actually already used one of them, which was if you're in a meeting and you're asked, if you have any questions, then ask a question. But I think there is something that's somewhat relevant to that, even though it's not necessarily a piece of advice, but like, just a general mannerism. If you're giving a presentation and you get to that and you say, do you have any questions? It's much easier to get some engagement from others if you reframe that and say, what questions do you have? I think, like, generally, and I've seen this also from experience, granted I didn't set it up as an A-B test, but you get a lot more relevance coming out of it and engagement if you say, what questions do you have? And people are more likely to do that.

Also, if you ask somebody, what questions do you have, or do you have any questions, count to 17 in your head. It's not like one, two, three, four. It's like one, two, three, four, five, six, seven, getting to 17. And that pause is sufficient enough to allow somebody to kind of wrap their thoughts together and build out something so that they can actually have something ready for you. A lot of times somebody will say, do you have any questions? Okay, good, and then see you later. But actually having that specific pause in there and actually doing that mental counting in your head allows for a decent amount of time to actually get there.

If you're giving a presentation and you get to that and you say, do you have any questions? It's much easier to get some engagement from others if you reframe that and say, what questions do you have?

The thought of waiting for 17 seconds there just gave me a little bit of anxiety. Yeah, I've called that out specifically to my team so that if they feel a very long pause for me, that's exactly what I'm doing, because I heard this piece of advice at one point. And so it's not that I'm just like sitting there waiting. I'm waiting indefinitely. It's just I have this specific pause.

Career path and generative AI

What important roles or opportunities have you taken throughout your career that has helped shape who you are today?

So I used to work at Domino's, and I was promoted to director. And then I think like six months later, so my wife was a professor at University of Michigan at that point, but we're both from the Carolinas. And Duke University reached out to her and ended up offering her position. And so we were moving back to the Carolinas to be closer to family. And then I wasn't really finding the right job in an industry that I felt really strong about. But as I was just kind of talking to people that I was meeting through my wife's academic relationships, I met these. At Duke, they had a new program called Messer, an interdisciplinary data science. And within that one, I really loved kind of like the framing that they were doing. And they're looking for somebody who had a lot of industry experience to come in and be faculty to not just like talk about the theoretics behind the machine learning applications that are happening, but they're looking for somebody that had a lot of that like software skill aspect that you can find with a strong data scientist to work through project management and build relationships and ask the right questions and like effective storytelling, et cetera.

So because I wasn't finding the right job in industry, I wanted to go with that one. And sadly, my wife was like, oh, you're going to hate academia. It's not like working in the industry. I'm like, no, no, it's going to be great. And then I got in there and like it was very changing for me. I saw a lot of different perspectives and was able to get a lot of a different like project experience as I was doing that. I did find pretty quickly that like academia is just not for me. But like I still feel like it was something that really changed me for the better as I was working with a lot of these grad students that were really early on in building their career and seeing a lot of these different projects that were being worked on.

Yeah, that's a really engaging question. I think there's a lot of hot takes on that. I think for the immediate future, if nothing else, it's going to allow data scientists to probably code a bit more efficiently. I think a significant challenge, and I haven't asked ChatGPT to go in and build something, like write a piece of code for me. I think my wife has done this, which maybe alleviates my requirement to utilize helper for her. Along those lines, I think it's going to allow data scientists to be more efficient as they're coding. I'd say it's probably five to seven years until I can actually do anything substantially relevant to consider data scientists are only working on framing problems in a way that AI can interpret them, and be able to be embedded on top of your snowflake warehouse, so that you can just ask a question, then it'll go through and build it a model. I think we're a long way away from that.

Josh King @ The Knot Worldwide | Creating your data and analytics roadmap | Data Science Hangout

Transcript#

How The Knot uses data

Seasonality and COVID's impact

Fomenting empiricism

Media mix modeling and privacy

Creating your data and analytics roadmap

Experimentation and cross-departmental relationships

Career advice

Career path and generative AI