Brennan Antone - Democratizing Organizational Surveys with Quarto and Shiny
When gathering data from groups (e.g., surveys), where does it go, and who does it help? How can we deliver value directly back to all survey participants, not just top organizational decision-makers? In this talk, I discuss re-designing how we report on organizational feedback surveys, moving from a top-down to a bottom-up approach to organizational change. Interactive dashboards can make data and feedback accessible to all. This tackles challenges with data quality, privacy, and power - allowing everyone to benefit directly from their data. I examine how Quarto and Shiny enable the creation of "flipped reports", and describe takeaways from implementing them with two Fortune 500 companies. This talk teaches how personalized tools can make data accessible to all, and can alter the power dynamics around how organizations enact change. Talk by Brennan Antone Slides: https://github.com/BrennanAntone/positconf24 Professional Website: https://brennanantone.com/
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hi everyone, I'm excited to be starting off the session. Today I'm going to be talking less about some shiny new tool that you can use, but more about rethinking what we do with these tools and how we use them. In particular, in the context of feedback. Feedback's valuable, it's how we know how to make changes. I'm sure we've all seen feedback forms come up, for example on websites when you're booking a flight, that ask you a bunch of questions.
And this can be an annoying experience. You just want to click away the pop-up, make it go away, and you don't really want to fill it out, you don't really want to provide good information a lot of the time. So a big question is how do we get good feedback? Not just in the case of websites or online experiences, but also once we start to think about organizations. Because feedback's the way we know what is going on in my team and organization, or more often than not, what's going wrong and how can I fix it?
So we have to think through what is the process of getting feedback and what does it feel like for the people providing the feedback. In organizations, sometimes you're doing employee engagement surveys, sometimes you're getting information in other ways, but a lot of times it can still just feel like kind of like putting a card in a suggestion box. You put it in and you just don't know what's going to happen with it, I'm maybe not sure who's going to read it, how they're going to interpret it, is something ever going to come of that feedback, or is it just going to sit there forever?
Challenges with organizational feedback surveys
Even once we've switched to digital surveys, a lot of these same feelings still persist, it still affects how we think about things. So what I want to talk about today is some of the challenges in getting good information from employees or other people in an organization.
So when you have an employee feedback survey, there's a lot of different pain points or challenges that can occur and some underlying ethical questions relating to them. The first one is time and motivation. People are busy, people have different things in their job, and a lot of times people aren't going to want to fill out a feedback survey.
You may have like a million emails in your inbox and just another one is, oh, reminder, please fill out the survey by Friday. It can be frustrating, so you either end up with people not doing the feedback survey, or you get people just filling it out as quickly as possible, sometimes not putting in good or meaningful responses. In both cases, you're bad. Forcing someone to like answer a survey if they don't provide good information isn't really helping anyone at all.
The second point of challenge is privacy or trust. Who's going to be receiving the feedback and are they going to receive it in a positive light? What's kind of the potential impact and what's the potential harm? This is important, especially when you're asking people about their job. No one in a feedback survey is really going to say, oh, I'm doing a poor job, or they may be very hesitant to say something like, oh, I think this is going wrong, when negative feedback about another department, about a project could affect their co-workers and could affect their friends in a negative light. So you have to really think about how can I make people feel safe and feel validated to give honest responses?
Finally, you have to think about impact and the way it can affect things. When you're providing information, you don't necessarily know how it's going to be used. If I say something that even if I think it's a positive light, together with other information, other context, other motives from decision makers, it could have an unexpected effect. So when I'm filling out feedback, I may be hesitant or I may be thinking about like what can go wrong? Do I trust this process?
All of these pain points make collecting data hard. How do I get an honest, high quality view of what's going on in an organization? And sometimes organizations can become really desperate when they're dealing with this. Either they're sending a million emails and reminders constantly bugging people, or in the case of one organization I worked with, they started to force people's responses where it would literally lock employees out of the system if they hadn't completed the feedback surveys on time and they would have to go through and do it. And that's a good way to get 100% response rate, but it's not necessarily a good way to really know what's going on in your organization.
Is there a more organic way we can do this? And that's why today I'm going to be thinking through the impacts of these bad incentives on surveys. They can hurt data quality, they can make it hard to take these responses and actually change organizations, and what can we do about it? How can we get a better experience? And can we use some of these tools like Shiny, Quarto, or even Markdown to start to change the way we design surveys and deliver results to address these pain points? So that's our goal for today, to kind of think through the process.
Research context: organizational network surveys
So to talk through what I did, I want to give you a little bit more context about my own work. I'm an academic, I'm at Grinnell, and I was working on this project with several amazing collaborators at Northwestern. And we study networks, how people are connected to others within a team or an organization. And we were looking at this in several companies in the context of COVID, trying to understand what is going on, how are people seeking advice, providing leadership and connecting in these challenging times. And the way we do this is by asking questions that build these sort of network models of who do people go to for advice, where are these silos in the department, where the whole network is bigger than the sum of its parts.
So this is really reliant on surveys. You have to ask people about connections like advice, leadership, trust. These are all things that only exist in people's heads. They're not something that you can quantify with some like digital trace data. So as we started to conduct research, we were working with different companies, and we would work with the management and leadership in the company.
So what you typically do in this network survey environment is you survey all the employees in the department, you combine all the responses, so all the employees are the individual nodes in that network, and you look at the big picture, and you try to give the management some type of top-level insight, takeaways, and rely on them to make changes. But this is all dependent on getting good data. We only act on what we can see, and if you ask people a question like who's making tasks difficult to complete, who do you enjoy working with, what are the challenges on your team, you may not get accurate answers when you're trying to ask employees that when it's coming from their boss.
And that's the problem is this framing, it's focused on me as the academic or the management or leadership of an organization of taking action. We gather data, we put the data together, we get a good report, and we see what's going on. That kind of framing though isn't putting the participants that are actually taking time to fill out the survey first. We're not thinking about their own user experiences, and that's kind of the big challenge. So how do we rethink this?
Rethinking the approach: bottom-up change
In the model before, if we wanted to improve the organization, fix problems, it's all about sort of top-down change. We have this network map, we give some key insights to management, and management or leadership is going to change something, and they're going to fix something about the organization. They're going to have one cohesive strategy. We started to ask what's the alternative?
What if instead of trying to just help management, we tried to give value directly to the participants in the survey? So instead of the management knowing everything, people being concerned about their privacy and trust, will they use the information well? What if we give every individual their own tiny piece of the puzzle? We give them information about their own local network based off of the responses they provide, and we empower all of them to make changes. Changes are going to come from the bottom up, and hopefully help out everyone in the company.
What if we give every individual their own tiny piece of the puzzle? We give them information about their own local network based off of the responses they provide, and we empower all of them to make changes. Changes are going to come from the bottom up, and hopefully help out everyone in the company.
So that's what we started to do with some of these companies. So we started building this tool called the Personalized Network Report, and the big picture idea that we had for this is, can we take your survey responses, and can we give you back value based off of them? So instead of taking your survey responses, fill this out, you'll never directly hear from us again after it's done, and we're going to give some aggregate information to your boss that you're going to hope does something well with it, and hope your responses aren't harmful to others. Can we take your responses, and can we deliver value to you?
So in some sense, we're just circling back the data that they provide to them, but in some other senses, we're giving them context on how to interpret it, insights on how to look at a network, what does it mean, how do I compare to others on average, and helping them think through the process. And that's the idea, to help every individual make changes, rather than trying to just benefit management and just give information to management. Completely rethinking power balance.
The personalized network report in practice
So what did this look like? When we were running it with organizations, it was moving from something that felt like running a survey, to something more like a professional development workshop. We'd have participants fill out this questionnaire, they would describe their networks, they would get this personalized feedback report, and then we would go into the companies and we would do a workshop with them. We'd talk them through how to interpret the report, how to use this information, how to think about their networks, and make strategies for how to leverage them. So it was all data driven, but we were running it more like a professional development workshop, focusing on helping everyone, rather than just running the survey that's just some task on a big list they needed to complete.
In practice, for the implementation, the questionnaire itself, we were just running something using Qualtrics. We would then take the information from the Qualtrics API, put it into PDF reports in Markdown, and then distribute it to the employees via email. So very basic stuff, there's nothing technically really sophisticated here, but the big picture was thinking through what does this process look like.
So in building this questionnaire, we were kind of walking people through this process of how do I map out my network. I'm picking a context in the workplace, I'm identifying key people, I'm describing network connections between them, and thinking about everyone and how everything's interrelated. And what we saw is that even with this very simple, very straightforward-seeming approach, we were able to get good results. We felt that we were able, participants said they were able to better make changes and better understand their network, and overall, without giving any information to the management or having to do like any direct exchange, we were able to get better quality data for our own research. So we were protecting participants' privacy, and we were doing interventions in the organization from this bottom-up perspective rather than a top-down one.
Why it worked: incentives and reflection
So why did this work? And these are some of the most important points I want to make in my presentation today. The reason participants found value and the reason they provided good information is because we were meeting participants and trying to help them. They were filling out something that was for their own benefit and where we were engaging them with the results. So the incentives were right. When we sent out these reports, we were giving them tools to better understand the data.
So our reports looked something like this. The very first version was just very simple PDFs, but we were helping people map out their networks. We were describing different properties of their networks and giving them their own score compared to other different groups in their organization and other groups in general, helping them understand what do I want my network to look like? How can I strategize about it? What do I want to change? And by taking this sort of expertise that we had and turning it into something accessible to the survey respondents, we were able to empower them to understand, reflect, and make changes.
So a key part was we let participants reflect. A lot of people mentioned that they found value just in that first stage of this process where they were just filling out the questionnaire. And this was surprising to us, but after a while, it started to make sense. Everyone knows in their head that, like, oh, networking is important, building interpersonal connections, and having someone to go to for advice in the workplace is important, but people have busy calendars. You often don't set aside time in your day and say, I'm going to deliberately reflect on what's going on in my organization. I'm going to map everything out. I'm going to think of different strategies and what I should be doing differently. Maybe you do, but I think most of us do not, are not that deliberate in our planning. And having an exercise where participants had time to do this, had time to reflect and answer these questions for themselves was beneficial.
And this was also beneficial in sort of launching organizational changes because participants could bring in their own context. Yes, we're getting quantitative data on the networks, but there's so much more going on in organizations that's not reflected in the data. The people that really know their jobs, that really understand what's going on, what needs to improve, how to make changes, are the people doing those jobs day in and day out. There's so much qualitative and subjective context they can bring in that positions them in a good place to make changes and to interpret the results. Rather than me having to understand all the minutia of their jobs, having to shadow them in order to understand what should they do differently, what's going on, if we give them access to the data, if we walk them through how to interpret network data, they can bring in their own context and hopefully better understand it.
There's so much qualitative and subjective context they can bring in that positions them in a good place to make changes and to interpret the results.
For instance, one of our participants described coming into a new company and when they were looking at their network, they were thinking about these different friend groups within the firm. They were thinking about their position and eventually they were able to make a strategy based off of their desire to connect to that sort of third friend group. But they're pulling in all the subjective knowledge that we as researchers don't necessarily have. And when we give data back to all the participants, it lets them make a lot of changes. So that's the big picture of how we saw value in kind of flipping the survey approach into something user-facing.
Moving to an interactive Shiny and Quarto platform
So that was the first version of our solution. What we quickly realized was it wasn't very engaging and it wasn't very immediate. People wanted the ability to play around with the responses, see what's going on, and get immediate feedback. So that's where after building that Markdown version, we then later switched to a version using Shiny and Quarto to deliver something more interactive. So building a web app that's going to pull in both the question answering components and the immediately getting feedback as you go into one integrated experience. It no longer feels as much like a survey, but it's a good platform to see what do my networks look like, think about what can it change, how do I compare to others, and providing that information and context is very powerful. But definitely rethinking the platform and putting it into that format was very beneficial.
Key takeaways
So overall, what have I kind of talked about today? I've talked about rethinking the way we collect data when we have to ask a human for that data, thinking about empowering the respondents, and thinking about the way we make change in organizations. So rather than just thinking it's going to be a survey, I'm going to fill out a survey, and then someone's going to come up with the answer. Either I'm going to give a dashboard and management's going to come up with the answer, or I'm going to come up with the answer and pass it off to management. Instead, let's give the information to everyone and help everyone make changes based off of what we've learned.
So blurring that divide between surveys, analysis, and interventions into one platform that's going to put them all together and help everyone be engaged in the process. So I hope that based off of this, if you're collecting data yourself, you can start to consider this role of power. Who has it? Who's going to make changes? And how can we make better experiences for the respondents? It's more than just asking people or forcing people to complete a survey.
If we think through this approach, we can get better quality data, we can help people out, and we can create resources that we can use in lots of contexts. So after building this tool, we've used it for teaching exercises, we've used it for leadership development programs and organizations, we've used it as sort of interventions or organizational surveys. But building that kind of platform that helps everyone can be very valuable. If we kind of use interactive tools to move outside that traditional kind of data collection, data analysis paradigm. Thank you so much for listening today. If you're interested in hearing more about what I did, seeing some of the code, or if you're interested in using this type of tool in your own context, definitely reach out to me. I'll also be on the academic job market for positions starting in fall 2025. Thank you for listening today.
Q&A
Thank you. We have time for a couple of questions. It seems like you come to this work with a strong theoretical background. What did you find most surprising about your results?
I think the most surprising thing for me is I thought the value that I thought the value for participants would be really hard to get out of it. Once we started delivering these reports to participants and asking them what was valuable, participants started saying, oh, it was valuable even before the report. Just even once we had the most basic version where we were doing a Qualtrics survey and that would be put into a report, just filling out the Qualtrics survey, some of them said they had value because they didn't do that reflection. And I realized there's so much space to not just think about these surveys as like, oh, I'm getting data, but as more of a guided experience, more of like just thinking through questions can be valuable. And it doesn't all have to be researcher-basing. Thank you.
One last question. Your individual feedback reports were based on networks. Are there areas or types of information that are missed by this approach?
Definitely, there's so much information. I think anytime you've put a survey together on anything, you're missing a ton of information. The best you can kind of maybe do is having like open-ended like text entry questions, but even that's limited. That's one of the big things I kind of the big things I kind of liked about this approach, thinking about it from an intervention angle, is yes, we're giving them network information and that's very limited. But once we ask people, what were your takeaways? What did you change? Like the example with the quote I gave, they were bringing in so much context that was outside of the network information. And both that and the network information together kind of informed their decisions and choices. There's so much richness in what they know. It's way, way more helpful to give the information to them in an accessible format than I think it would be for me to try and understand their jobs or figure out all that context. I just couldn't do it. So I think with the participants being empowered, they can understand that context and additional information so much better. Thank you.
