Developing a Prototyping Competency in a Statistical Science Organization - posit::conf(2023)
Presented by Daniel Woodie The introduction of new tools, methods, and processes can be a struggle within a statistical science organization. Being deliberate and investing in the creation of a prototyping competency can help in accelerating progress. Prototyping allows organizations to quickly experiment with new ideas, reduce the risk of failure, identify potential issues early, and iterate until the desired outcome is achieved. I will highlight the key areas we have focused on accelerating, our framework for developing this competency, how we use Shiny, and the lessons we've learned along the way. Developing a prototyping competency is crucial for statistical science organizations that wish to stay competitive and innovative in today's rapidly changing landscape. Presented at Posit Conference, between Sept 19-20 2023, Learn more at posit.co/conference. -------------------------- Talk Track: Building effective data science teams. Session Code: TALK-1059
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Good morning, everyone, and welcome to this session on building effective data science teams. I'm so happy to see you all here. We'll have a host of fantastic speakers. Each one will have 20 minutes today. That includes questions, if time permits. And it is my great pleasure to welcome the first speaker, Daniel Woody, up to the stage. Please come in.
All right. Thanks, everyone. I have to start this off with a little story.
So actually last week I was having a meeting with my manager, and we were doing like a performance review check-in. And he had lots of positive feedback about, you know, how I'm building this prototyping competency within Lilly, and, you know, I've done a great job with coming up with like an operating model and having metrics for the team and setting the direction. But he gave me some feedback, too, on things to improve on. And the thing to improve on is my presentation skills. And so this is a great opportunity to, you know, put some of that to work.
So that was Tuesday of last week. And then Wednesday of last week, I get on LinkedIn, get on the internet, and I see this. You know, must-haves for this conference is my talk. And then this company put out a top ten and put my talk as number one. And I was like, we need to level set a little bit here, guys.
So thanks, everyone, for coming to the talk today. Here's the rough agenda. So I'll be going over, you know, why to build a prototyping capability. Something I've put together, I'm calling this statistical sciences hierarchy of needs based on Maslow's hierarchy of needs. I'm going to walk through our process a little bit. How we define a prototype. How we operate as a team. And lessons learned. And one thing I'll point out is when I say, like, we as a team, it's me and one other guy. We're a very scrappy operation.
Why build a prototyping capability
So yeah. On the topic of why to build a prototyping capability, you know, they're useful and quickly vetting new ideas. So I work in pharma in the stats department. And innovation can be hard in that group. And a lot of it is there's just a fear of risk taking. And having the prototyping capability allows you to quickly vet, you know, new technologies that you may want to explore. And it really provides an accelerator to the broader strategy for the organization. It's incredibly valuable for elevating the conversation. Not just with your team, but with leadership. Because a lot of times they like to go big and it slows them down. Whereas going small allows them to, you know, test out these new ideas in kind of a risk averse way.
Statistical sciences hierarchy of needs
So the statistical science is hierarchy of needs. So if you've seen Maslow's hierarchy of needs, it's, you know, based on that, obviously. Which is a schema or framework for kind of finding, you know, fulfillment and being healthy. And, you know, when applying it to our organization, this is kind of my latest draft with it. But yeah, for us, we have IT infrastructure at the foundation of it. Which can be kind of foreign, I think, to people that are traditional statisticians. It is kind of the bedrock for everything else that we do.
And having that value and perspective is kind of critical to progress in the other layers. And so, you know, once you have, you know, something in place or strategy in place for moving IT infrastructure forward, then you can start to look at, like, your tools and your capabilities. And so for us, the tools are building R packages, Shiny applications. And on the capability side, it's, you know, we work with drug development teams. And so we're working on clinical trials. And so it's kind of taking those tools and using them for those product teams. And then beyond that, we also have training. And so once you have kind of your tools narrowed down, then you're looking at, okay, how do you upskill the organization? Which is a big task.
And then beyond that, we have the prototypes team. And so with the prototypes team, like I said, we're an accelerator. And so we're working in IT infrastructure. We're working in the tools and capabilities. And we're even testing out new training modalities. And our job is really to make those lower levels kind of work in harmony better. And then at the top, you have self-actualization or, yeah, just everything working in the direction you want it to. So that is kind of our north star, if you will.
Our process
So our process. Very simple. I point this out because a lot of times people can get lost in technology and say, oh, I've got, you know, this new hammer. And then they just go and find the nails. But, you know, that may not be the best way to start out. And so, yeah, obviously find the problem. Once you have the problem, solve it in some way. That could be with some new technology. That's our lean on it. Beyond that, get paid slash reduce cost. And then lastly, refine and repeat. So that's kind of how we stay practical, I guess, with all of our solutions.
One example prototype, you know, walking through or framing it in that same way. So one issue if you've built Shiny applications is that they will freeze when running a long running job. And the application will just stop working. And so the solution to that is to use or to enable asynchronous processing. There's a variety of ways to do that. So at Lilly, there's a great developer, Will Landau, he's made a number of tools. One is crew, which helps to facilitate spinning up independent workers and running a job from an existing R session. And so we put together a basic prototype to see, could this solve our problem? And if so, you know, what does that look like?
It does drive impact by making the applications more usable and broadening our impact. And then lastly, on the step for refining and repeating, it was open sourced and people continued to build on it. And the direction our team has gone has been to enable offloading those independent workers into new compute environments. So that's how we iterate.
Defining a prototype
And so on the topic of what is a prototype, it's important to define what your finish line is. Because you don't just want to blindly be going out and building a prototype and without much direction or aim, kind of just continuing to iterate. And so for us, the finish line is typically, you know, either it's a pre-specified set of features, oftentimes it's a guidance on go or no-go for future development. Or in that same vein, providing the guidance on, okay, if you want to mature the software, what would that take to move it to the next stage?
So this is a scorecard that we've put together as a team to define what those other stages look like. And so in the leftmost column, you have the software maturity stage. So we end up spending most of our time in the POC or functional prototype. But then you can move on to, you know, alpha or beta releases, where you're broadening out that end user group or eventually taking it on to production software, where people are using it to make important decisions. And so going from that leftmost column, moving over, we have things like having end user testing, you know, the documentation, more unit testing, integration testing, and working with IT as well. But, yeah, like I said, we spend most of our time in the POC to alpha or beta release. And then when it comes to moving things on to production, usually we'll simply provide the guidance and start working with the teams for a handoff or something like that.
How we operate as a team
And so moving on to how we operate as a team. So again, it's just me, James, the other guy. We check in daily. We don't have a project manager. We project manage ourselves. Typically, we're working on two to three prototypes at a time. And they take anywhere from a few weeks to a few months to complete. As far as setting goals as a team, we use the objectives and key results framework. And so every quarter, we kind of set some big picture goals. For example, one is to complete ten prototypes in the next quarter as a team. And that allows us it's a good mix of kind of long and short term goals. So we have three months to work on those prototypes. But then we have a lot of flexibility in between and defining what those prototypes are, what teams or groups we're working with.
Yeah, another thing we do is we make our goals widely available. And at those quarterly check-ins, we meet with a lot of the sponsors for the team or the stakeholders and reset the goals. We make our goals widely available so anyone can look at them, provide input to them.
Yeah, and then lastly, this is a fun one, or at least I think it's a fun one. We also do a retrospective, and we do something called a fear setting exercise. I saw it in a TED Talk and tried it out, and we've done a few now. And we did this fear setting. So if you haven't seen it or heard about it, it's kind of the opposite of goal setting, which I think scared my manager when I first pitched it.
So fear setting, it's useful for someone like me. So I'm very optimistic, but I also like taking risks. And so that can be trouble because those anxieties or that voice in your head saying, no, don't do it, can be useful. And so the fear setting kind of facilitates that. So what we'll do is we'll talk about ways that our group, like something could go wrong or one example would be like we start working with a certain team, and they want to use all of our resources, and so we can't really work on the other stuff. So that's a big fear. We don't just want to be a shop for one team.
And so going beyond that fear, it's not just like we talk about them, and then we get paralyzed by them. Then we talk about, OK, what if that did happen? What do we do? What can we introduce maybe to our goals to make sure that that kind of thing isn't happening?
So yeah, and another fun part of that is, like I said, I lean towards more like being optimistic, and there's definitely some people that are on the other side of that spectrum, more cautious, more anxious, a lot of fears, and it's fun to recruit them because I rely on that because I'm like, I don't, you know, honestly, I don't even know what, I have a blind spot. I don't know what my fears should be. And so yeah, relying on them is really valuable.
Lessons learned
And lastly, just to share some of the lessons learned. Like I mentioned earlier, conversation around innovation will be elevated. I think it's in the spirit of organizations transitioning from waterfall to agile, but instead of taking gigantic, you know, multi-year projects that have a lot of risk, and yeah, you can balance that with doing smaller projects, testing the waters, learning more about it, and yeah, lifting up the entire conversation, both on the development side and within leadership.
So you will fail often, but more than make up for it with reducing cost and accelerating learning. Yeah, I think as a team, this is going to sound strange, but I wish we failed more. And I think my manager has said we should fail more. And we do fail, but I think when you're failing more, you're taking, you know, you're pushing the boundaries more. And so that's an important part of it. And beyond failing, you know, obviously, you're learning from those prototypes.
Yeah, I think as a team, this is going to sound strange, but I wish we failed more. And I think my manager has said we should fail more. And we do fail, but I think when you're failing more, you're taking, you know, you're pushing the boundaries more.
As I mentioned earlier, it's a very small team. I really think you can do a lot with a small, scrappy team, especially in that earlier phase for software maturity. When you're talking about production software, there's more important things than moving fast necessarily. You want reliability and quality with a lot of that stuff. It's harder to move faster with a big team. And also with older, more mature software, it's harder to make the updates. So that's where having, yeah, prototypes is really valuable.
Next, I'd say iteration is not just for software. It's really good for metrics as well. So earlier I mentioned one of our metrics or goals is to complete 10 prototypes a quarter. And some of you may have listened to that and said, well, that's not a really good metric. And it's one we work on too. So there's certainly different flavors of what that metric could be. Right now it's a good volume. We're adjusting it. But, yeah, one thing that has come out of this is that those metrics need to be updated. We revisit them on a quarterly basis. But, yeah, it's good to have that flexibility to iterate on metrics as well.
Next, spending some time with your fears is incredibly valuable. Yeah, I think, you know, I shared earlier how it's good for people that lean towards being more optimistic. But I think it's also good for people that spend more time with their anxieties because it kind of forces people to go, you know, from those fears to, like, okay, let's say it does happen. Then what? And you're not just stuck in that, you know, cycle of thinking about what could potentially go wrong.
Experimenting early and often is a huge accelerator. And then lastly, I would say statisticians are becoming software developers. That's a big narrative within our organization. And having the prototypes team exist to bridge that gap is very valuable. I think we're key to testing and vetting the new ideas, new technologies, new processes that the rest of the organization is starting to test the waters with. And we get to be the guinea pigs to share those learnings out.
And then lastly, I would say statisticians are becoming software developers. That's a big narrative within our organization. And having the prototypes team exist to bridge that gap is very valuable.
Well, that is all I have. So thank you, everyone, for coming to the talk today.
Thank you, Daniel. We will have time for a single question. And so it must be really tempting to keep working the prototype to production. How do you decide when enough is enough?
That is a great question. When to decide when to move, like, a prototype to production. So a lot of it ends up coming. So we'll provide, like I said, a go or no-go decision, and then we'll do some evaluation on, like, okay, you want to take it to the next phase? Here's what that would take. And our team really isn't set up for supporting the QA process with IT, monitoring infrastructure, things like that. I think having that conversation early and up front is important. Because that's definitely the direction it would go is and we don't want to be a team that is just supporting the production software. It stops us from accelerating those conversations.