Bryan Butler | R in Marketing - Survey Design | RStudio

This is only going to be about a 30-minute talk, so we should have plenty of time at the end for any kind of question, discussion, or whatever we want everybody else wants to talk to after all of this. And this is a project that I've worked on over the years, and it keeps coming back. So that's one of the reasons why I decided to bring this up with a marketing group.

I've been in and out of marketing groups for the past few years, and also worked in marketing research. And so a little bit about me, I am a head manager of analytics data science at a small bank in Boston. I started out in chemistry and then got an MBA, did a lot of work in the insurance industry doing some pretty exotic stuff, got heavily involved in Monte Carlo and pricing, and then got involved in quant banking consulting. So I think, as you can see, like every other data scientist, I just went the exact straight route out of college into data science.

Overview of the project

So what we're going to talk about here is what this project is all about, and I'm going to try to give it to you in a way so that you can take this to your next marketing project. Talk a little bit about what good survey design is, or what I think it is.

There's an interesting bit of intellectual property that the ability to work on called customer quotient, similar in a way to NPS, but also very different. We'll look at some parameters to constraints, then getting involved in the machine learning part of how do you actually do machine learning with survey data.

So the role of this was, how do you take a survey, because that's what a lot of people in marketing research do, and develop a segmentation engine out of it, so that as people complete your survey, you can put them in the right segment for whatever, and your segments can be based on whatever. In this case, this was a sort of a behavioral psychographic segmentation, which means it wasn't so much, you bought product A, B, or you live in this part of the world, or wherever. This is more about how you feel and how you behave.

And so it gives you a little different approach to sort of the different dimensions of marketing. And it was looking to reveal, this is, again, this is part of a much larger project, how people view companies. And the other way to think about it is, at the end of the day, it rates how customer-centric a company is. And that's how the questions were sort of phrased.

Good survey design

So these are some of the things that I've learned about what makes a good survey. And these are just my, these are my takeaways from being involved in surveys and trying to apply machine learning and, you know, how do you get it, how do you ensure you get a lot of respondents, et cetera. So the first thing is, I want a goal of, like, what is the survey design to answer? And then you're going to want to have some supporting questions. So I was thinking structurally, that's how we want to start it. And the goal, you shouldn't have more than three things you're trying to answer. I usually say one to two is good, just in case, you know, you pick up some other ancillary information. And these goal statements are binaries, yes, no's.

My first experience was dealing with a survey and people are like, well, how likely are you to purchase from us again? And so you end up with these neutral, you know, everybody's in the neutral range, and it doesn't really help you out. Or what do you do when you get a handful at the positive and a handful in the negative and a handful in the neutral? Do you drop them? Do you include them with something else? So this sort of resolves that issue. It's a simple yes, no. With that, the supporting questions, you want as few as you can get away with, but still get the information that you want. So, you know, I like to think maybe you're between five and 10 questions. If you start getting over 10 questions, you're just getting too much, you're asking for too much information, and you'll find that people will drop out over time.

The flip side around it is in the marketing research world, you do have the ability to pay people with, pick your incentive, Amazon gift card, PayPal, whatever else, they will stay in for longer, knowing that if they finish it, they'll get some money at the end. And I've done a couple of these, I've done many of both, and there doesn't seem to be a difference in engagement as far as like, you get different answers if you pay people.

And then here's a key thing. I like my supporting questions. So we got our one binary, like now instead of how likely are you to buy from me again, is I have bought from you again, or I will buy from you again, yes or no? And then we're going to ask a bunch of questions that support it. And I think everybody's seen these, strongly agree, agree, neutral, et cetera. I do not like that scale. One, you don't get as much variability in the modeling phase, whereas one to 10 gives you plenty of variability. And I think if you go over more than 10, people don't know what the numbers mean.

And just as a little aside, I worked in the wine industry for a while, and you know, they have their 100 point wine scale. But the funny thing is 99 point, whatever percentage of the wines are ranked between 80 and 100. So what they really have is a 20 point scale. And so I don't know, they disguise it as a 100 point scale versus 20 point scale. Same thing here. 10 points will give you enough variability.

And then the other thing is, you know, people always say, oh, we can ask a question a different way to make sure that we don't have people gaming the system. Maybe if you get this giant survey that are like a personality test or something like that, but that's a whole different animal here. If you're surveying your customers, they're not going to, they're not going to do the duplicates. And if they start seeing the duplicates, it'd be like, this is a waste of time. And it's also a waste of your time, too. Like every question matters.

And I've seen some other indices out there where they have like, they try to decompose like a net promoter score, which into three indices, and it's kind of like, the three indices are so incredibly correlated that you just wasted two questions asking about the other two indices when they're like 90% correlated. So it's kind of like, you really have one index. In the market world, I know anybody who's worked in marketing has heard of net promoter score. It's kind of a hybrid because it does ask on a scale of one to 10. But you can also, but at the end of the day, you can make people binaries out of it. In fact, there is a, you know, people that score you nine and 10 are your promoters, six to eight are your neutrals, and five and below are your detractors. And the score only works with promoters and detractors. So it's a very unique scale. But when you do NPS, you should really ask, you know, get into the whys.

And then a good rule of thumb is to try to get about 1000 responses. Because you can do some really good modeling. I know everybody thinks, this is a world of big data, I need 10,000, I need 100,000, I need a million. No, you don't. You need a well designed survey. And if you get close to 1000 responses, you will get a reasonable amount of information.

You need a well designed survey. And if you get close to 1000 responses, you will get a reasonable amount of information.

What's interesting about surveys, is that if your questions are distinctive enough, then even if you get smaller segments, you will be able to predict them. So for instance, you know, if segment one is questions one to three, and you know, segment two is maybe a question three and question four, but then segment three is this oddball question, question five, even if it comes out small, and you get into this highly imbalanced data scenario, the fact that question five is so unique, it will help even with the prediction in an imbalanced situation, because it's not random.

Customer quotient

So a little bit about customer quotient. This is developed by a colleague of mine at C Space in Boston, Dr. Manel Austin. And it talks and it builds on this cost of this sort of how customer centric as your company, and they ask questions from from the customer's perspective, and you build up the pyramid from, you know, relevance, you're just in the game at that point. Then we have a customer series of questions around customer experience, how well you do, then the better you are, you get into openness, and that begins to really unlock the doors. Are you willing to have an honest and open two way dialogue with your customers?

Empathy is really what changes the game. So when I've seen companies in the segments, companies in the segments, it's not till you do reasonably well at the openness, and get into empathy that you begin to really like, we've grouped different companies and segments and everything else. And you see, when you start rating up on the empathy scores, you begin to run away, and become extremely well. And the last one is what we call emotional validation. Those are the people out there that are selling your product on their recommendations.

What we used to talk about is you have two people, you have three people having a conversation, someone says, I need to get a new smartphone. And the first person says, get an iPhone phone, and the other person really doesn't say anything. So your iPhone is your sort of emotional validation person. And they'll give you five reasons why you should get an iPhone. Or on the flip side, they'll give you five reasons why you should get an Android or whatever it is, but they are selling your product. And it's called emotional validation. Those are your ultimate customers, because they are doing your marketing. And we find that people that score higher up there actually have to spend less in marketing and advertisement over the long term, and they get some better scores. That's just a little bit of this intellectual property area in the space that I worked on here.

For those with traditional marketing backgrounds, where you talk about the four P's and the three C's and all those. So the four P's, product price, place and promotion. The three C's are customer, company and competition. And what these do is sort of map various questions that help with the CQ scoring into these different traditional marketing buckets. So that when we talked about pitching this to different companies, if I had a traditional marketer, I could say, yeah, this is your product piece, this is your price piece. And you can see price, it's all from that customer point of view. The company appreciates my loyalty. And that's literally how the question worked. It's an action word, or I don't feel ripped off. And you can see in most of these questions, and these are sort of paraphrases of it, but we've made it action oriented, and it's from the customer standpoint, not I. Versus if you think about a lot of surveys is the company is asking, do you like us? Do we do a good job at this and things like that? Whereas this whole framework really reverses it around.

Project parameters and constraints

So let's talk about this project that we had. Client specification number one had to be built in Excel. And that happens. I wish I could just do it in Markdown or something easy like that or a web app. But at the same time, it's workable. What it does do is it changes the model processing a little bit. I'm not going to be able to run XGBoost or any of that. But at the same time, I may not need to. And there's a good way that I get around this down the road. Our sample size was 900, not 1,000. But that's fine. And it's like, okay, we're probably only going to get about three or four segments out of this.

And part of this came out of is that a major company had a project with a major consulting company. And they said, well, you have eight distinctive segments. And it was like, okay, really? So we're going to test out whether we really have eight distinctive behaviors by using this survey. And the initial step, and this is one of the things that, like I said, this one did not have the dependent variable. And that's why I added it in early on as a good survey has a binary outcome. So we had to create one or create some sort of segmentation outcome. The nice thing is survey questions on a 1 to 10 scale, easy to work with.

And if you think about it, from the machine learning world, as we go down the road, we're not going to have to center scale or do any of that with the data. It's already prepackaged on a 1 to 10 scale. And we're good. Whether we do any kind of, and we're going to do unsupervised learning or supervised, you are not going to have to rescale all the data, which makes it nice.

And when you do some of these things, and these end ones are my discussion with them, what's good enough actors? Are there penalties for false positives or false negatives? Like if we put the person in the wrong segment, how costly is it to your business? And should we add penalties if they are costly in one way or another?

And that would change a little bit of the modeling too. We would actually add a cost matrix into it down the road if we had to. But in this case, there weren't too many big deals. And they said, you know what? You give me 80% of the way, you give me 80% of the right answers, I'll be happy. I was like, okay, 85, that's what we're shooting for. Hopefully we can do better. That's one of the things for me is I always like to be in the 90s.

The machine learning roadmap

So here's a little roadmap here. And you can use this over and over, like any kind of survey work you do. I've used this format for a lot of NPS modeling that I've done for various companies over time.

Design a nice survey. So we ask for NPS people, we ask that question, but then we ask them questions about customer service, we ask them questions about pricing, other aspects of the business to see what you're trying to do is figure out what drives that answer, what drives that yes, no. So then we're going to do some clustering, I like hierarchical clustering, it's easy for visualization, the scale is easy. So we're going to separate our respondents into some different segments and clusters. And then we're going to create a dummy variable for each one. So you're in segment one or you're not, you're in segment two, you're not, you're in segment three, you're not.

I think a lot of people would try to go straight to sort of a multinomial model, hey, we're just going to predict what segment you're in right out of with one equation. Those require a lot of data. So yeah, if you're in one of these big data worlds, and you've got 10,000s or 50,000, you could probably pull it off with 1000 simple binaries is going to be the way to go. So here's my little my little magic here. What I first do is use the elastic net regression, the glmnet model, a glmnet package by Trevor Hassey, Rob Tertranian. It was designed for genomics, it really will help you reduce dimensionality, and it's extremely fast.

And to give you a little bit more information on this, um, the people that are familiar with ridge, or lasso regression, this is the hybrid. So a ridge will attempt to squeeze down all your coefficients to make them very small. And the lasso will do is if there are multiple variables that are correlated, the lasso will pick one and leave the other ones out. Now the problem with lasso regression is you don't know why it made the decision. Is it, you know, a function of the gradient descent, or a function of the order of the data, you just don't know. And they talk about it in their paper about that, whereas what the elastic net does is, I have three variables that are correlated. If they're statistically valid, we leave them all, we leave all three in, if they're not statistically valid, we pull them all out. And so that's why, how it gets to your dimensionality reduction.

So then this last step is, all right, we got to go into Excel. Now you probably could do an elastic net, maybe not, in Excel, but you can easily do a logistic regression. And so what we're going to do is build out our template. So when we get batches of scores, we just run them through our worksheet with logistic regression in there, and then it's going to use a voting method, and it'll choose which segment you're in.

Clustering and elbow plot

So the first thing we did is, we run this, the elbow, famous elbow plot used for clustering. And it's called within the sum of the squares residuals. And so what we got is number of clusters, they told me they had eight segments, so I went for up to eight clusters, or at least their consulting company did. And what we're looking for is the bend in the plot. And you can see, you get the elbow at about three or four, at which point then it all sort of flattens out. So if I had a lot more data, I probably would shoot for four clusters. Given that it was only 900 respondents, three would probably do the job. You can see that the sum of the squares goes down, obviously, the most between one, two, and three, between three and four, it's very little enough that tails off. So that's why we're looking at that inflection point. So we plot them all out visually, we get three groups.

And this isn't, you know, you can see in the hierarchical clustering, that you get the divisions, almost fairly cleanly between the blue block and the green block, and the green block and the red block. And the other way to interpret this is, the people in the blue block are very different from the people in the red segment. And there's some gradations across the middle. I use this plot a lot when I do text analytics, because it's really fun then. Because what you're seeing is, when you do this with text, is these people are saying one thing, they're saying something similar to these guys. But now it's getting different, and then you get something really weird at the other end. So I like this plot from a visual standpoint. And the other axis is just the distance metric.

And as I mentioned, because all of our questions were a one to 10 scale, I didn't have to rescale anything to be able to do this clustering. And yep, this is small, and it may be difficult to predict, but it doesn't automatically mean that. Because as I said, if there are certain questions that are answered by a handful of people in the group, that makes them marketably different, then you'll still be pretty good at predicting whether or not people are in there.

So when I overlay my cluster analysis across their eight segments, it was kind of interesting. I was like, well, you may have eight segments, but you have three distinct behaviors within each segment. And what I mean is that there is this grouping here, we'll call number one for now, behavior one, that seems to cut across all of them with an exception of this segment. And we also have behavior two. There's people in between, but then this behavior number five stands out. And it appears that this convenience store resellers segment is actually kind of small in general. But looking at those numbers, it's pretty clear that we have, you know, you could describe it two different ways. You can say, we have eight segments with three behaviors. Or you could say, I actually have 24 segments. So granted that I have this social couple, there are actually three forms of social couples.

Now, one of the interesting things about looking at some of these segments is, what about people that would be crossing over some of them? Like, how did the company specifically put them in each segment? And can you move? Or will you move over time? Like how long do you stay a new mom? And, you know, how long do you stay a social couple? Some of these others are a little bit more static, but we got a nice clean picture here. So it's good. Our elbow plot told us we had three or four segments. And most likely three, our hierarchical clustering came up with about three segments, and we can see it here. So the size thing is, so we have three segments, three equations.

Elastic net and logistic regression results

So here's a little bit more on the elastic net. The reason why I use it is it's going to give me my upper bound of performance, or accuracy in this case. Because my goal is to get down to a logistic regression with a couple of questions. But if my elastic net comes in only at 80% accurate on the rough, before we even do any kind of dimensionality reduction, then my logistic regression isn't even going to cross my 80% threshold. So that's going to be problematic down the road. And the way it works is, you know, you think about your logistic regression. It's, you're in segment one or not, you're in segment two or not, and the probability you're in segment three or not. And for each person, you get three scores and the highest score wins. So in this case, the person, this, the data point came back, the probability in segment three was 90%. It beats 55, it beats 21. Therefore, they get assigned a segment three.

Versus I think, when you think about your traditional way of just using a single logistic regression, you generally assign your probability at 50%. And it's greater than 50, it's in that segment. Here, we're using a little, an ensembling type approach, and we're going to, we're going to vote basically. Standard split training testing. So, you know, 900 data points gets me 250 testing and, you know, 650 training. It's not great. It's not bad.

How do we do? So here's the first segment. Segment number one. The elastic net regression. Near perfect. Very little false positives, false negatives. Good split here. So it's not so unbalanced that basically it's an easy guess. So this is nice. 99.26. Really good. And so you see down here, when we did these, I'm not going to sort of show you the same plot three different times.

And another key piece to look at is this, this no information rate, 0.61. That is essentially your data split. If the no information rate in this was 90 or 0.90 or something or higher than that, which when you get an imbalanced data sets, then that, you know, that 95, 99, well, it's a good lift, but it's not as impressive versus we're at 60 on a random guess. And, you know, we're near 100% accurate. And you can see we got 94% on this segment too, and 80 in this small segment. That's, that's not surprising. We knew it was going to be a small segment. We knew it was going to be a little more challenging, but it's still, for the client standpoint, they're quite happy with what they were seeing initially.

This is a nice little variable importance plot with the elastic net model. And you've got your questions ranked order of importance. And when the, when the plots out this way, that means they put you in the segment and you'll see in another plot, they go the other way. They're saying, when you answer this question, it kicks you out of the segment. And now there's others, more sophisticated approaches out there. There's LIME. That's probably the most common now that people are using locally interpretable modeling. But this, one of the things that tells me now is I can begin to remove some of the questions from this equation, right? What I'm trying to do is make us simplify this as much as possible. So I'm going to say for segment one, we're just going to use a couple of questions. So that not only have three variables for this equation, make it perfectly simple for it, especially since we don't have a lot of data.

And so what did model one, what did question one, segment one say? Well, they appreciate my loyalty. I feel proud, sense of belonging, and they believe the company employees use their own products and service. So this is, we think, highly correlated to an emotional validation segment. And so these are your really good customers. You want to take care of them. You want to know who these people are.

In the world of computer vision, I was at a conference back when we used to go to those things in a live person. And someone said, you should run your image recognition about your customers when they come in your bank because they match the face and the balance or something like that. And you can come up and say, wow, this person's like a really big customer. You should do something to take care of them. Interesting portrayal, but someone would say, yeah, but everybody's an important customer. Just another way to think about how do you take care of your loyal customers and how do you show them their loyalty? And that's a question 16 is an important one. And I know when we were doing this, we were presenting something like this to some internal people. My marketing research person used the concept of Amazon pantry. She's like, she said, they get it wrong all the time, but they make it right all the time. So yeah, maybe I order six things and five out of the six are right. And one of them's wrong, but she said, they always make it right. And I think that's that part of that. How do you get, even if you make a mistake, how do you get people coming back to you? And it's all about fixing your errors as a marketer and as a company.

So now when I reduce it down, this is that what I said is we're going to go to a simple logistic regression with just a couple of questions. I lose a little bit accuracy. So rather than almost a perfect with only one off in here, we've actually began to make a little bit more false negatives or false positives here as we think they're in segment one where they're really not, but we're still at 94%. So we're good. And we're not, again, we got our no information rate. So it's, we've got good lift. We now have a happy logistic regression model for segment one.

So segment two, as I said, looking at the variable importance, these, when you answer these questions a certain way, they actually pull you out of the segment. So you need them as a way to separate out segment one from segment two. And if we ran this through Lime, you would actually see a red and green, or if you use the traditional colors, you'll see red and green scoring for these questions that they're important because they will either put you in or push you out of the segment.

And these are a different set of questions we see. So our 16s and 21s, as you can see, are up here, not as important. And here we're imbalanced. And it's a mixed accuracy, right? So yeah, we think we're really good, but most of these people are getting pushed out in the other segment. And this is the logistic regression model for making, and that's where I said the known information rate is 90. We're getting a little bit more lift, not much. And we're making equal errors, false positive, false negatives. But we definitely have these questions that are oriented around customer service. And that's essentially what these four questions point to.

And then segment three, our small one, and this is why it's hard. You got some of the questions that kick you out of the segment, and some of the questions that kick you in the segment, and there's a little bit of both. So you think about it this way, in segment one, these guys were high scorers. So now in segment one, in segment three, you're getting low scorers on here, but now you're getting a different scoring across here. So these questions are pushing you out of the segment. And then in segment two, you're getting a little bit of those questions are pushing you in them into the segment. And the nice thing is, we do okay at a small segment. No information, no information rate is 52%, almost purely random coin toss. And we're able to beat that by almost 30 points, 78. So again, I mentioned it was like, love it at 80%, love to hit 85. But at the same time, given that it's a coin toss, and the data set, then at 80%, we're doing okay.

And obviously, if I use a far more sophisticated model, we could probably do a little bit better. But it goes back to the constraints, the customer wanted it in Excel, they wanted to be able to batch load their own data, and just kind of see the levers. And I think that's the other part of the importance of logistic regression. And same thing with Elastic Net, it's just a logistic regression with a regularization function, is that a business person can say, yeah, if I score better with this question, I'm actually going to increase my loyalty scores or, you know, that that's partial differential part where you say, and this is what I do when I do when I analyze surveys, I said, Look, if you increase your score for this question by one point, the probability that they are one of your promoters or, you know, frequent buyers, and everything goes up by 5%. And if you increase by two, it goes up. So it's a very quantitative approach to being able to say, how much do I need to move the lever?

I said, Look, if you increase your score for this question by one point, the probability that they are one of your promoters or, you know, frequent buyers, and everything goes up by 5%. So it's a very quantitative approach to being able to say, how much do I need to move the lever?

And I'll say in the NPS world, and I was doing this in my last company in healthcare, it was, it's a lot easier to make a person go from a detractor to a neutral, than it is to take a neutral person to make them a promoter. And so as you think about ranking your your actions that you're going to take, and so you're in, in this case, your dependent variables, they're promoter or they're not, and you rank the questions of importance, it'll tell you what's driving your answers. You should focus on ones that are the low hanging fruit, but they're also easy to get them out of the detractor hole. And that's why actually, we did that in a couple places. And it worked, our executives actually had in their bonus plan, that their NPS had to go up by four or five points. And this actually created a bit of a an action plan for them.

Summary and wrap-up

So what did we do here? So we started out with our elastic net. We got the question, we got the survey down from 17 to nine questions. That's good. Obviously, the elastic net is very good. I looked at and I didn't show these here, but we looked at CV, ROC curves, confusion matrix, I showed you confusion matrices, just so we could get the few extra stats out of there. We have a model for each segment, we use our voting approach. Overall, the overall accuracy was over 85%. And they said that was the client said it was acceptable. And they didn't have any, we didn't have to come up with penalties for misclassification. Making a false positive or false negative on a segment is from a market standpoint, it's not the end of the world. Maybe if you're spending lots and lots of money on something, it may become more important. But in this case, it was really looking to understand what's going on within their customer segments.

And that's it. So this, so if we look at it, you can take a survey, make a binary question out of it, apply clustering, maybe like KNN better than hierarchical or something like that. Get your segment labels and then apply some machine learning against it. And if you get about 1000 respondents, as you've seen here, even with 900, it'd be reasonably accurate. And you can do this to all any survey. And this is just I do this on a regular basis, over and over different surveys, try to tease out some really good quantitative aspects of it.

Q&A

Awesome. Thank you so much, Brian. I see there's a few questions that are starting to come in right now on Slido. But just want to let people know if you want to unmute yourself or raise your hand on on zoom, you could ask questions live to

Lee, I see you had asked the question earlier in the zoom chat to Brian, I believe it said, do you reverse code? Or any other item types? Would you want to add any other context to that question? Can you hear me? Okay. Yep, I can. So earlier, when you were talking about item types, you talked about Likert scales and 10 point scales and all that. I'm wondering if you use any reverse coding or Thurstone type items to make sure people just aren't repeatedly filling out the same value over and over when you do your assessments.

I don't like to do that. We used to have a large, I used to have large conversations with people, not so much reverse coding, but it comes straight liners, which I think is what you're getting at, you know, somebody, they give everything a five or everything, you know, whatever it is, I look at it. I don't want to throw it out because especially with a Likert scale. It's a, you don't know, that could be a very valid data point. 10 point scale, you tend to see a lot less straight, what they would call straight lining. But it's always part of the EDA. And just because we sat on that conversation all the time.

Thanks, Brian. I can read some of the anonymous questions. But again, if you want to use Slido too, you can put your name in there. And you can ask it live as well. But the first one was, how are the surveys conducted and collected? I've noticed YouTube does surveys in between videos. How did you do these?

This survey was done through a marketing research platform. Our same, actually, most of them were done that way. So when I, so I said, this one is through a marketing research platform, I don't know who you are. And you just plug it away. And then we get it out on the back end, I think, like SurveyMonkey and something like that. I've done this exact same analysis with data from SurveyMonkey also. That was on InsureTech industry. We did about 2000 respondents from all from different groups.

And some of them are just, a lot of them are generally web based, where you've had a transaction, just like you say on YouTube, you know, hey, how did you find this video, you know, give us a rating one to 10 or sometimes, you know, you'll get a note. At the end of this call, you'll be asked to take a survey. And my health care company that I work for had it that way too. So which was interesting, because now you're punching in numbers on a phone. You can still do one to 10. It's just, it's a little different than click in a box, and you don't get comments as much. So then you, you have to ask good questions, detailed questions.

I was curious, if, if you don't know who is answering the survey, and then you're going and making changes through the business, how do you measure, like, what actually made the change in the survey, or if it was like those actual customers or?

Most of them, most of these types are directed at your, at your customers. So you, so you know, your customers. And in fact, I did one recently at our bank. It was a survey. And so we had all their information. And it was actually a rerun of a service. So they ran a couple pre-pandemic, post-pandemic, wanted to see what was going on. But it was highly skewed by the age of the respondent. And, and, you know, you just have to point that out. Like, you know, I'd actually show a slide to our presidents and say, look, this is our real customer base. This is what they look like. This is what our respondent group looks like by age and demographics. They are different.