Stephanie Lussier @ Moderna | Data Science Hangout
We were recently joined by Stephanie Lussier, Manager, Biostatistics at Moderna to discuss growing a new data science team at Moderna and developing a pipeline to enable internal clinical trial decision-making at rapid speeds. Speaker bio: Stephanie Lussier is a Manager of Biostatistics at Moderna. She’s a member of the Moderna Specialty Data Analytics team, which focuses on building a seamless analytic platform and capabilities to enable quantitative decision-making in a timely fashion across clinical programs. Her professional interests include designing novel data visualizations, building tools that help clinical trial statisticians, and promoting the use of open-source software. ______ ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co LinkedIn: https://www.linkedin.com/company/posit-software/ To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!) Thanks for hanging out with us!
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Awesome. Then let's jump in here. Well, welcome to the Data Science Hangout, everybody. Nice to see you all back here. And if this is your first time joining us, a special welcome to you as well. It's so nice to meet you. I'm Rachel. I lead Customer Marketing at Posit.
I like to start asking this, but is this anybody's first Data Science Hangout? Say hi in the chat because we'd love to welcome you in and say hello to anybody joining for the first time too. But this is our open space to chat about data science leadership, questions you're facing, and getting to hear about what's going on in the world of data across different industries. So we're here every Thursday at the same time, same place.
If you're watching this recording on YouTube later, the link to add it to your calendar will be in the details below as well. I said every Thursday, but just a heads up, we will be off next week for the Thanksgiving holiday. But at the Hangouts, we're all dedicated to making this a welcoming environment for everyone. Love hearing from everybody, no matter your years of experience, your titles, the languages you work in, or industry. It is totally okay to just listen in here.
Maybe you're walking your dog, you're on your lunch break, or just want to listen in for the first one. But there's also ways you can jump in and ask questions or provide your own perspective. So you can jump in by raising your hand on Zoom, and I'll keep my eye out. You can put questions in the Zoom chat, and feel free to put a little star next to it if you want me to read it, or I can call on you to introduce yourself and add some context. And then we also have a Slido link, which Curtis will share in a second here, where you can ask questions anonymously too.
Just before we get started, I wanted to share if you are hiring, feel free to share any roles in the chat here. That's definitely not spammy to us. The Biogen team actually shared one with me earlier today, so I just wanted to share it in the chat there.
One more shout out, if anybody is in Boston, Kevin and I are hosting Tidy Tuesday meetup tonight. Yes, on a Thursday, but that's at Microsoft in Cambridge. But with that, I am so excited to introduce my co-host for today, Stephanie Lussier, Manager of Biostatistics at Moderna. And Stephanie, I'd love to have you introduce yourself and share a little bit about your role, and also something you like to do outside of work.
Stephanie's background and role
Sure, so thank you for the brief introduction. As you said, I'm currently a Manager of Biostatistics at Moderna, and I sit in our Specialty Data Analytics team. So a little bit about my background, I graduated with a Master's in Biostats in 2017, and then proceeded to work at a CRO, a contract research organization for five years. And during that time, I had what some might call a little bit more of a traditional statistician role within the clinical trial realm, and was doing open source projects, kind of more on the side, like a 20% time kind of deal.
And so in 2022, I kind of got to the point where I really wanted that to be a bigger part of my role. And so that's where I found my current position at Moderna. And so I'm in the Specialty Data Analytics team, and our mission, kind of in simple terms, is to just use clinical data with modern open source technologies to really enable internal decision making.
So that's where I am now, and happy to talk more about it. And outside of work, I kind of collect hobbies like Pokemon cards. And as a result of that, I'm definitely like jack of all trades, master of none, which I'm okay with. And so some of those hobbies include yoga. That's kind of a big one for me. I've been learning to play tennis. I make a lot of sourdough. I'm learning to throw pots on the wheel. And I do like to travel. And when I travel, I do like to scuba dive. So I'm happy to talk about hobbies for an entire hour.
Wait, can you tell me a bit more about what throwing pots on the wheel means? So like using clay to make pottery. And so a lot of that, at my skill level, means making cups and bowls that aren't very symmetrical, but are supposed to be. It's very relaxing. For me, like I can't draw at all or paint or anything of that type. And so this is kind of an art form that I can do a little bit.
So with all due respect to past data leaders, I thought a little bit about what my goal is, being on the Data Science Hangout today. And for those that don't know, the leaders are great, but the chat is where it's at. And so my goal is just for people to have fun in the chat. So go crazy. It has resources. It has debates. It has fun topics. And if you've never participated in the chat before, I'd encourage you to do so if you want to.
Becoming a manager for the first time
Well, Stephanie, while we are waiting for some questions to come in from everybody here, I know something that we talked a bit about is becoming a manager for the first time. And I'm curious if there's something that you've learned or you had to learn throughout this process of becoming a manager that you'd be able to share with us.
Sure. So a lot of pieces of this. In my previous role, like I said, more as like a traditional statistician, you work within clinical teams. And so you end up doing some management of like statistical outputs. So that is kind of like managing statisticians and programmers on a project level. And I think that was really good preparation for being an actual true manager.
And so now I'm on a team. We have seven full time employees and seven contractors and I'm starting to pick up part of like the true management pieces of that. But let's see, as far as what I've learned, there's a lot of paperwork parts of it. And so hopefully anyone becoming a manager has some sort of mentor walking them through that part of it. For example, requesting a laptop is a pretty difficult goal to achieve, apparently. Shout out to our awesome digital team who helps with that. So learning like those little technical pieces of it is like a whole new world to just take on slowly and ask questions.
I think that's the most important thing. And then as far as actual management, I think just forming a relationship with people, working closely together, asking them to ask questions, just really like creating a comfortable space has been really important for me.
Language and tools in clinical trials
The question was, when you graduate or what is the language that you use in your academic program? Did you need to learn SAS when you joined the CRO? Yeah, so in the clinical trial world, the main statistical language that is used right now is SAS. I don't know what SAS stands for. It's fine. But it's a statistical programming language. So during my grad school, I was lucky enough to learn both SAS and R and even some Python. And I think that really guided where I ended up today.
So SAS was taught in certain classes that I had and in other classes, the professors favored R. And I think the reason I really ended up on the R path was that my main thesis project was done in R. And so that really gave me a strong foundation so that when I started my first role, I was definitely expected to use SAS in that more traditional setting. But I was able to connect with the people in my first organization who did use R. And that really kicked off thinking more about developing with R and what R in the clinical trial space looks like.
Visualizations for internal decision making
Something that you and I had chatted a little bit about before is that you've realized people don't want to look at these standard tables, listings, and figures, which I just learned that's what TLFs means when I see that listed or named in pharma. But I'm curious, in building tools for internal decision making, what has been really helpful for you in communicating?
Yeah. So the fun thing too about the TLF abbreviation is that it goes by many abbreviations. So watch out for that. It can be TLF, TFL, TLG, TGL. And it all means the same thing. And why is it like that? I think that's just to confuse people. And I think that's a common theme with abbreviations in clinical trials.
So to actually answer the question, yeah, a big part of my role is creating visualizations for internal decision making. And so we think a lot about what is the best way to do that. I think one of the most important questions is what is the question that someone is trying to answer and who is the end user? And so, for example, a lot of times the end user for us is, say, the executive committee. And so people on the executive committee at Moderna, at least, are highly skilled professionals. They're not looking for a green button or a red button. They want to know what were the results. And they want to see it in a visually pleasing, straightforward display. But they also still want to know some key statistical elements.
And so that's actually part of the reason why a lot of the people on our team have a statistics background is because we are still displaying confidence intervals and risk ratios in our displays. And so that's an example where we know the audience and we know the questions they're going to ask. Another example would be we create, say, visualizations that are going to be used by the clinical team that want to look at data in an ongoing way. And so you might think about how well those people know this data and they want to know every nitty gritty about it. And so that's where something like interactive applications really comes into play so that as they form their own questions, they can answer those questions.
And so that's actually part of the reason why a lot of the people on our team have a statistics background is because we are still displaying confidence intervals and risk ratios in our displays.
That's great. Can you share maybe an example for us or example use case where people were able to answer some of their own questions with an interactive app? Sure. So I'm working, one of the big projects that I work on both internally and externally right now is on patient profiles. And so the whole goal with patient profiles is that you can look at one particular participant in a study and the primary uses there is really for safety. So you could look at, say, their labs, basic demographics information, their what we call adverse events, so any safety signals that happen through the study. And so we had a participant that had an unexpected response in a biomarker. And so they were able to use the patient profile to look at when that particular biomarker uptick occurred, what else was going on for that particular participant.
Shiny apps, interactivity, and the FDA
Was that visualization, was that a Shiny application? Have you developed Shiny-based apps for your internal users? And have you ended up using the same interactive or static visualizations in drill downs when interacting with the FDA?
So the particular example I provided is actually all being powered by non-Shiny interactivity. And there's been a couple of great talks about this topic lately. DeepShaw gave a hilarious talk at PositConf about this. And my mentor, Augustine Calatrone, also had a recorded talk at rFarmer where he kind of touched on this as well. And so kind of with a judicious use of crosstalk, plotly, interactive tables, you can make an interactive HTML document that has a limited but pretty good capabilities. And so that is the current platform for that particular patient profile. So yeah, if you're not familiar with that setup, I do think it's powerful.
In my previous company, we didn't really have a good hosting platform. And so we were able to really utilize that setup to create interactive reports. Like at a high level, the limitation is that the interactivity happens on one level, sort of. And there's no recalculation happening of the data. You're essentially filtering the data. But you can really do a lot with that setup.
So then moving to like Shiny, it's definitely something that our team uses. We have a variety of different visualizations. And so that decision is kind of part of like what we talked about earlier when you get, say, a data visualization request deciding what is the right technology. And so if more and more exploration is the goal, then Shiny, I think, is the way to go. And we definitely utilize that.
And then I think the third question is about sending that to the FDA. And that's not something that we're doing right now. But I think probably a lot of folks are following the R submissions group. And they've done a pilot to look at what a Shiny app to the FDA could look like. And they're going to be doing more utilizing the WebR framework, which is a hot, exciting topic right now, that could apply really well for this concept of Shiny to FDA as well.
Knowing your audience and limiting interactivity
What's the limiting principle for how much interactivity to give people? Because I kind of found that there are two sides to that continuum. One, you can just keep building the app and building the app and adding more functionality. But you want to deliver it. And then the more content you add, the more functionality you add to it, the more possibility someone else pointed out that there's a possibility that people are going to derive insights that aren't necessarily true or meaningful. But then on the flip side, you know that the one thing you don't implement is the thing you're going to get asked for. So where do you draw the line on what to do and what to leave out?
That's a big question. It's difficult to nail down. I think in my particular realm of things, I definitely think about it along the lines of where a study is progressing. So in the early, early phases of a clinical trial, or even preclinical, things can be much more exploratory. And so when exploratory is the goal, hypothesis generating is the goal, you know, go crazy and put a disclaimer that, you know, there are things discoverable here that need to be proven outside of this application.
And then when you work your way down to, you know, questions that are more formulated, a researcher has really provided the question they're trying to answer. That's where I would think about limiting it and trying to really just answer the question that's being asked and limit the back and forth. Because, you know, there's a whole theory behind this. But when you're in phase three of a clinical trial, the question should be laid out and the question should be answerable. And that upholds the entire integrity of the trial.
Getting the right feedback on reports
Similar to you, I wind up sending reports up and stuff. And sometimes the feedback that I'll get is good and sometimes like, eh. What's one thing that the leadership does that's good, that's helpful to you that gets them the report that they actually want?
Yeah, I love that. So what is good feedback and what is less helpful feedback? Good feedback is for them to tell us how and why a report was or wasn't useful so that we can then apply it to future reports. And so the why I think is important. And maybe you have to ask the why. Because if they just say this didn't help, that's not useful. So getting a better understanding of what they were looking for, maybe an understanding of a report answers the question that they asked, hopefully. But if that question changed over time and that's why the report is no longer helpful, a little bit of insight into how that question might have changed or why that question might have changed helps us kind of predict in the future how questions might change or how we can provide supplemental materials that might answer those changing questions.
So yeah, I think the takeaway on that is it's OK to turn it into a conversation and probe a little bit more so that you can move forward and improve for the future.
I'll just make it really quick. I have a lot of prior careers, one of which I'm actually a professional counselor with a master's in the whole thing. So to me, listening is really, really, really critical. I think it's almost more important than the analysis. So just make sure that I really, truly understand what the question is. And if I can give a professional counselor tip for just a second, also find out what it is not. Because when you find out what it is, you might go four miles down that path and find, oh, it's actually something slightly different. I was working on that. No, we don't want that. Is it this and not that? And once you do that, a lot of things clear up, and then it's easy to move forward.
Yeah, I previously did a lot of work on manuscript writing. And so in traditional clinical trials, you have your endpoints that are pre-written, and you answer those endpoints that are pre-specified. And sometimes in a more exploratory biomarker manuscript setting, there can be a lot of back and forth like you're talking about with questions changing over time. And so keeping the conversation flowing was always important to really narrow the scope and try to find the answers that they were looking for.
Yeah, I'll kind of repeat what I said in the chat is one of the things that I learned to ask for is what questions are my leadership being asked that led them to ask me to do something? Because how they're interpreting the questions that they're being asked, and then there's like three levels of information distillation before they ask you for a piece of math or a piece of reporting or a visualization. And so then you make the thing that you think they were asking for, but it doesn't really meet the ask that their leadership asked them for. And so there's a little bit of a cycle there. And so getting plugged into meetings where you can listen to how your leadership talks about these kinds of things or being forwarded emails or things like that can really help you get a broader picture of where your organization fits within the broader thing.
Yeah, one of the comments that you touched on, Catherine, is something that we kind of prioritize in our team. And that's asking to get invited to the meetings, to kind of parse down the amount of distillation that occurs. And so no one wants a calendar full of meetings when there's lots of, you know, deep work to do. But when they're important, asking to get invited so you can hear the whole conversation instead of the summary.
Recyclability of analyses and internal packages
I'm interested in, Stephanie, at Moderna, and with sort of the profile of work you're doing, how recyclable is sort of those analyses of those reports. And if you want a bit of context, I spend a lot of time trying to not do anything multiple times. So I'm curious if the biostats area, or at least, you know, with your team, if there is a lot of opportunity to library your sort of starting points or what have you?
Yes, great question. The short, very short answer is yes, there's a lot of recyclability. And I think a lot of similar teams, as well as mine, are working on, you know, creating internal packages that help with that recyclability. But something that has kind of been on my mind lately is making sure that when you're creating functions, modules, whatever, you're still allowing for customization. Because at least in my experience, what people in clinical teams, they want to add stuff. They want to say, can we look at this too? Can you add this facet? You know, even little details. And so as we go on, you know, making progress in creating like an internal package system, that's something that we're trying to keep in mind is finding that sweet spot between out of the box, boom, here you go, I made you this plot, and allowing for customization.
Advice for recent graduates
Do you have any advice for a recent graduate trying to get her foot in the door? It's a bit hard to do currently. My job search to switch jobs took what felt like a long time. I don't want to like put a number on it. Like I was always told statisticians can get a new job so quickly. And I was really picky in my situation. And so it wasn't quick. And I did have like, fear surrounding that, that I look back on now. And it all makes sense, you know?
So but for thinking about someone with less experience, I saw the chat going this direction last week, which is a topic that I think is really cool to talk about is the presence of like an online GitHub repository. Because I think there's arguments for and against this. But I think in a position of being a recent grad, that's something that I would think about doing is do some little tiny projects and put them on GitHub just to demonstrate your programming skills.
And then in the clinical trial space, if you're having difficulty, I would suggest looking at contract research organizations, I learned so much in that role. And I think they have a lot of positions, hopefully for more entry level. I think sometimes the pharma side of thing does look for more experience. And there are more entry level associate type roles in the CRO world. So that's something to like, look into and read more about.
And then another thing I would think about is the clinical trials world has a lot of clinical trials knowledge. And so trying to find some resources online to become familiar with what a clinical trial looks like. We have data standards called CDISC to do a little bit of research about that. I think that would make you really stand out as a candidate.
Managing meetings and deep work
Can you guys hear me okay? I guess like in the CRO space, you bring up a few good points, but I think for me, it's more like transitioning and towing the line between a statistician or programming role. I've seen a lot more individual struggle going from a programmer to a statistician than from a statistician to a programmer. And so the thing that I keep finding myself wondering is do you find yourself missing some programming aspects where you're kind of just inundated with writing up SAPs and protocols versus actually getting to do some of the work?
Yeah, really great question. It's interesting to see, like I've seen colleagues over time switching between programming and statistics. And that's always kind of telling about a person's passion and how that might change over time. I think it's really cool. I actually think right now in my current role, I have a very good mix of statistics, programming, management, kind of all blended together, which for me works well. I like doing all of those things.
And like to your a couple of things. So I think we're going to see maybe the definition between statisticians and programmers change over time, like where that line is drawn. And that future roles might start looking a little bit different. I think there's already several companies that are kind of calling everyone a data scientist now and blurring those lines. And so I think, you know, within that idea of blurred lines, it's important to know what you like and advocate for yourself to do what you like.
And then another piece of that, that's something I really like to speak about this when I'm interviewing candidates for our team, is to have an open conversation about, you know, like you said, I was seeking out this change from a traditional statistician to what looks a little bit more like a data scientist, you might say. But like fresh grads, to have a conversation about what that means and what that means in terms of your career. I always like to have that conversation because the industry as it stands right now has more of those traditional roles. And maybe it will change over time, but just kind of having an open conversation about it because it might be a new concept for people that are new to the industry.
Open source and Pharmaverse at Moderna
You mentioned that Moderna is kind of open to open source and those tools. So I'm very curious if in your day-to-day work, you're using any like Pharmaverse packages or any other pharma-related open source packages? And like overall, how are you approaching the open source projects at Moderna? Are you going to release some open source yours like as well within your company?
Yeah, great. We love like, I mean, I love open source. I think that's why we're here on this Hangout, because we all love open source. So yeah, your question had a lot of different parts again. So as far as our team using Pharmaverse, so a little background for non-pharma people, Pharmaverse is a collection of packages contributed to by all different pharma companies as well as CROs. And the aim of the Pharmaverse packages is largely to do with submission. So actually putting a drug in front of the FDA for approval. And that is not where my group is using open source software at this time.
We're very much keeping our eyes and ears open to it. And if that's something you're interested in and you haven't yet watched Novo Nordisk's YouTube video, someone can put it in the chat. That's like very groundbreaking, very exciting. It was presented beautifully. And we're very happy at Moderna to see these companies laying the path for us. Very appreciative of those companies as well as all the initiatives that surround that.
So yeah, that's not totally our focus. And so I don't see us using Pharmaverse packages a ton. That being said, we have a safety stats group that are also open source folks, and they have developed similar table packages for what looks more like an FDA reporting output. And again, those are currently being used internally. But as a part of that work, there is a member of that team or several members of that team that do participate in Pharmaverse open source, like the Falcon package. And so in general, I find that leadership is very open to open source collaboration.
Career advice and the most fulfilling project
The question I want to ask is just what's the most fulfilling project you've worked on so far in your career? It, like, gives me a good moment to reflect back. Like, you know, we talk about data science and clinical trials, but clinical trials are like a big deal. They affect people's lives. And that definitely is a huge motivator for me in my day to day. And it's always important to like, think back, like this data is a person giving up their time, giving up their selves to help improve science. And like, I'm giving myself chills. Like, it's just such an awesome thing to be a part of.
Me personally, my answer definitely has to be that in my previous role, I worked on a really big food allergy trial. And it would essentially be the first actual treatment, like a antibody approved to treat food allergy. And it was a behemoth of a study. No one wanted to work on it. It was very complicated. But it was so rewarding to think that you're helping children who can't have lunch in the cafeteria change their lives. And so I was really proud of the work I did on that study. It was an awesome team.
Like, it's just such an awesome thing to be a part of.
Oh, and I'll add to that. I had a colleague. Once I was off the study, I had a former colleague whose child joined the study. So that's super cool to actually like, feel so connected to that study. And so again, hearing from his point of view, oh my gosh, we're at the doctor's all the time. This is so much work. And like, on our side, that's like, this schedule of events is insane. I don't know how to program the visits data set. Like those two things go hand in hand, and our side is easier. Like I think always keep that in mind.
So maybe at some point I'll pull all the answers to these this question together, but I know we've asked it in a few different Hangouts. Is there a piece of career advice that you've been given or that you've given others that stands out to you?
Sure. So when I joined the CRO we've talked so much about back in 2017, I was just very lucky to be scooped up by a very passionate mentor who has 100% guided my career. And he's given me so much advice. And the advice is both solicited and unsolicited. And it ranges a big gamut. You know, there's statistics advice, there's programming advice, there's advice about my mortgage, there's advice about how to plan my wedding. And so it was hard to sift through many, many words of wisdom to answer the question.
I think one thing that I didn't know at the time was going to be so important, but luckily was very kind of like lead by example, is how to be a remote employee that is engaged with a team. I didn't necessarily think I would be a remote employee. And that changed and most of us are now. And I think the single greatest advice is just pick up the phone, like to be a part of the team so that you're not siloed away, form personal relationships, and maybe not everyone wants to form personal relationships, pick up on that, if that's the case. But to call people that you work with and ask them a question about work and ask them a question about their personal life. I think that was one of one of the better lessons I've learned.
Like to be a part of the team so that you're not siloed away, form personal relationships, I think that was one of one of the better lessons I've learned.
So much. Thank you all for spending your Thursday with us. And thank you, Stephanie, for sharing your experience and knowledge with all of us. This has been great. I love the engaging conversation and definitely one of the more lively chats as well. That's what I like to hear.