Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
Will, thank you so much for being here. I am so excited to talk with you today. Will is a Senior Advisor of Innovative Statistics at Eli Lilly, and I would love to know what that job title means because it's one of the coolest job titles I've ever heard. Can you tell us a little bit about what you do and something you like to do for fun?
Sure, and thanks for having me. It's really amazing to be here. I work in a group of several hundred statisticians at Lilly, and our group is a central tool development, method development consulting role. Statisticians are organized by therapeutic area, like cardiometabolic health or neuroscience or autoimmune diseases, et cetera. We're kind of like an everything central group, and we spend more time. We're not necessarily saying that we're better at methods and tools, but that's where we spend most of our time. Whereas one of our colleagues would say, I am developing this simulation to design this autoimmune trial, in our group we say, okay, what are the different kinds of simulations and models that people across the company are developing? Let's standardize and make these implementations rock solid so we can roll them out to everyone so other statisticians have an easier time doing their work.
Senior Advisor is really a fancy way of saying statistician, and those are just in general ranks across the company. We used to have job titles that said senior scientist or things like that, very general, but all it means is statistician in my case. As far as something that I like to do for fun, over the last five years or so, I've been getting oddly into backyard ecology. Just from walking around my local park, and eventually I decided to join a group that walked with the head naturalist there. I started learning about different kinds of common native and invasive plants, and just learning, oh, that over there, that's bush honeysuckle. That grows in monocultures that take over the park and are doing ecological damage to these pollinator pathways and watersheds that we have around here. That lit a fire under me. Now I volunteer with a bunch of people locally, and we have this little meadow that we're working on removing invasive species from the park. We're seeing all kinds of really nice native plants coming up again. We're gradually seeing diversity return to the area, and we're creating this little oasis wherever we can. That's been really fun to get my hands dirty with that, and gradually see the impact build over time.
Will's background and the targets package
Well, you did not mention, I think, the one thing that I really want everybody to know. Can you talk about your relationship with the targets package? Speaking of making things easier for a lot of people to do.
Yeah. That's a great story. I'm glad you brought it up. Because I was, in 2015, like many PhD students, I was struggling to get my dissertation done. Of course, a lot of that is the writing, but for me, it was, and a lot of others probably on this call, it was a computational struggle as well. Because I was trying to fit this complicated Bayesian model on a collection of simulated and real agronomy datasets in genomics. These are datasets that, in terms of big data back then, it was just kind of medium data size around big fitting datasets that were kind of like 30,000 genes by 16 observations or something. And the computation on that was quite intense, though. It was a long, hours-long Markov chain Monte Carlo algorithm that was fitting each time.
And I would struggle going back and forth between sending these off for computation to our local servers, and then midway through the computation realizing, well, something's wrong. There's a bug here. There's something I want to reorganize here to support the writing. And it was just this constant back and forth of trying to keep results up to date and waiting a long time for them to finish, and then getting disorganized and confused really quickly.
And so I got really interested in pipeline tools. And credit to my advisor, Jared Neamey at Iowa State, because he said that, well, a lot of the struggle is needless. You should be using a tool like Make, which of course is a tool from the 70s to manage compilation of C and Fortran programs. But I think then people like my advisor, who were computationally really savvy, were just starting to realize that Make and its main operating model had uses for reproducible data science to both only run results when the rerun is actually necessary, meaning that if a result was up to date, if an analysis is up to date with this upstream code and data, why not just skip that and only rerun the models whose data encoded changed?
And that would have saved me a lot of time then. And there was also the mental model of a workflow, a mental model of a data analysis pipeline. And I think that more than the computation, more people in the statistics community get value out of having a mental model, a graphical mental model for what exactly is going on in a research project or simulation study or anything of that nature.
And so at the time, we had things like Make, but we didn't really have something that was idiomatic to R that fit the particular concerns and sensibilities of our users, a reliance on functions instead of scripts, integration with literate programming, and those things that were really important that just weren't in language agnostic pipeline tools. And so at the time, Rich Fitzjohn, who is also really active in our community in ROpenSci, he's at Imperial College London, I believe. And so he was working on a tool called Remake, which was, in my mind, a first stab of this and had really mind-blowing insights into what that integration with R would really look like. But by the time I became interested in working on this stuff, he had moved on to other things. And with his endorsement, I created a couple of sequence of pipeline tools of my own. First one was called Drake, which lasted a couple of years. But then realizing that I was hitting design constraints, moved on to Targets, which has been around for about five years or so, five or six years, and really pleased to see the adoption.
And internally at Lilly, we use it a lot for clinical trial simulation, really nuanced studies that we're trying to design to figure out what models we should use, what sample sizes the study should have, and things like that. Really diving into specific targeted questions that require a lot of custom code instead of tools that we just write and people can grab off the shelf, pull up a browser, pull up the Shiny app. The problems that we use Targets for are just everyone is different. Everyone has its own complexities. And we also use it for model testing. So if I'm writing a Bayesian model and I want to run a simulation-based calibration checking workflow, that's a tool that I would use to architect the pipeline.
And externally, it's really interesting to see the uses that have come up that are off-label to, let's say, wrangle a pipeline of hundreds of Quarto reports or have a pipeline that ingests data every day and just publishes a dashboard or updates a dashboard on a daily cadence. And just seeing feedback from the community has really opened my eyes to what people use a pipeline tool for. And it's really versatile. And I think Targets is very opinionated, and it's interesting how useful it can be, even if not all of the opinions are universal among its users.
Managing open source alongside a day job
Hi, Will. And thanks again for all the great work that you're doing. I remember someone once saying to me that if you write an R package, you'll never be lonely because everyone will keep sending you support requests and comments and stuff. And so what I see from the outside is that you're doing great work to try to kind of separate out that development support stuff and your day-to-day job and personal time and life like that. How the heck do you do it?
That's a great question. I'm sure just everybody starts their first job, they leave school, they join industry. I had all these ideas in my head that I just wanted to get out at the time. And I was trying to start a new job at the same time. And I compensated by effectively working two jobs. And that was hard. I didn't do it very skillfully in the first few years. I think that now, I think that a lot of things have improved that because I think the company's interests and my interests are more aligned. I had a lot to learn when I joined. And just like anyone, I mean, pharma is a huge and complicated industry. And for the problems it works on, it's critically important to get it right. And it's critically important to absorb everything I can. But as I transitioned to a more capabilities role and targets became more fundamental to our simulation infrastructure, those interests aligned and things naturally overlapped, saved a lot of time on my end.
And I think that as far as responding to people online, I think that there's some of the most fun conversations that I have on a daily basis, because you just dive right into the technically interesting details of a problem. So answering people's questions, although sometimes it's a struggle to find time, but it has been a lot of fun and I have learned a lot from it.
And I think lately I've noticed a bit of a slowdown in the throughput of questions that I've gotten about targets externally. And if I had to guess, it's probably a combination of the fact that targets is pretty mature right now. And also because people are getting a lot of answers from AI as well. So I think they're both of those things.
Targets vs. Nextflow and SnakeMake
Hi, my name is Noor. Can everyone hear me? So I'm about, for context, I'm a bioinformatician, but my background is in math and stats. But as someone who's primarily worked with tools like Snake, Make, and Nextflow, why should I use targets or what are some benefits you would say are using targets over something like Nextflow? I have a sneaky suspicion since lots of my tools are in R, I may see a benefit for using targets versus wrestling with Nextflow. But if you could get some more sort of like context or insight.
That's an excellent question. Really, at least something that I've been thinking about for a while. And although I've never used Nextflow or Snake, Make seriously, I think I've heard enough and read enough about it to say that the main thing that I'd go for, for deciding which tool to use is how integrated is your workflow with R. I think that with that, because targets says, there are hundreds of pipeline tools that are language agnostic and for these other purposes, targets makes a bet that if you really commit to R, then they're going to be just workflow advantages for people who spend most of their time and do most of their work there. And that's, I would say that that's really the overriding critical question of what tool to use in that sense.
Also, traditionally with, I've noticed people who use R also, if they need distributed computing, high-performance computing, then we often go more towards traditional schedulers like Slurm and we use Altair grid engine at Lilly and things like that. I've tried to get into the cloud distributed computing space through crew and its extensions there, but it's just never been as big a part of my day job because there isn't as much of a demand for those kinds of resources. So I guess it also depends on the computing environment as well.
Nextflow is just a completely different conceptual model. It's based on data flow programming instead of make-like programming on the same way, on a directed graph. So Targets uses something called Khan's algorithm for topological sort, which is how it traverses its own graph with a little bit of a twist that you make dynamic updates to the graph sometimes for dynamic branching. And that's very different from Nextflow's model where edges in that graph are first-class citizens. And to understand sort of what problems and areas of flexibility it can handle and what kinds of efficiency gains you have there, that's something that I'd like to better understand.
Gen AI's impact on open source development
Well, I think that as far as the daily impact of open source development and collaboration and things like that, the slowdown of throughput of issues and questions I think is the biggest impact that I've noticed. I don't often get pull requests relative to the number of issues that come my way, but I have heard reports that there's this challenge that a lot more, many more pull requests have been AI generated and to manage that and manage the communication with the human behind them I've heard is challenging for a lot of people.
I think even more of that though, I'm questioning the value proposition of being a package developer. Because writing code has never been an end result anyway. It's always ever been a means to an end. It's always been just a capability to solve the problem and it requires for a full value proposition as for a career, you have to have the domain knowledge and to know where to find those problems and apply them. Now, because code is so much cheaper than I'm thinking, in what ways do we have to reinvent ourselves in the AI age?
Because writing code has never been an end result anyway. It's always ever been a means to an end.
The thing right now is this transition, if there is one, is pretty slow because code is cheap, but good code is not cheap. Humans still very much have value, but I've been trying to transition a lot of my work to strategy and systems in addition to just being somebody who writes statistical models, implements models or implements simulation infrastructure and tools.
I actually just open sourced a guide that I'm working on, a model for developing agents that we're adopting internally to solve targeted problems. I'm calling it a paradigm of trusted mini agents. It really dives into something that Sarah Altman said in her talk with Simon Couch at Gen AI Day this week, where she said that we recommend that it's important when you're designing an agent to design a scaffold so it matters less if LLMs are wrong.
What I've been trying to do for the last several months internally is to really take that idea as far as I can take it to its inevitable conclusions for targeted problems. Fortunately, just right before this hangout, I got permission to open source this guide.
Crew and mirai for distributed computing
Crew came about because I wanted to separate out the distributed computing layer from targets. I wanted to come up with something very general that just had the kind of distributed computing that targets needed and that was versatile enough to work on a bunch of different clusters and cloud computing as well. There's a really important side of this, which is just there's a lot of variability from cluster to cluster. There's a reason to have a mechanism for inter-process communication that's very general and that doesn't depend on the specific configuration of each cluster. A really good assumption is to transact over the local network to send tasks to workers and receive them back. That required mirai, really.
Crew wouldn't have existed without mirai. mirai was just the perfect tool to transact across the network for that. I saw mirai just being discussed in the open source distributed computing infrastructure and R back in early 2023, and I started talking to Charlie that way. We've been collaborating on mirai and Crew for about three years. He's a really amazing guy. At that point, he hadn't been very long in this space. He had an earlier career in investment banking. It's just mind-blowing not only how he transitioned into the space, but just how good he is at what he does and how insightful he is all the time.
He's been super accommodating as far as collaborating on mirai. I bring this arguably very strange use case that a lot of people have a hard time wrapping their heads around. He's just been fantastic in solving this crux of a problem that really allowed targets to really fly on all these different platforms.
Communicating uncertainty and the statistician's toolkit
How do you communicate uncertainty to people who want a single number?
Oh, that's interesting. What a great, broad question. Yeah. I remember working cross-functionally on studies and hearing from medical colleagues very confidently that, you know, we were looking at sort of mean treatment effects from three different studies in the same master protocol, and there was a slight downward trend in them. And what I was hearing from medical was that they were very confident that there was a significant trend going a certain way in the responses. And the communication is difficult, because it has to be done carefully enough to protect the relationship, but also communicate the fact that apparent trends like this could just happen by random chance, and how what that random chance looks like is very difficult to communicate to people.
And in visualizations, we produce, of course, we add uncertainty where we can, where the model justifies it. And to caution people against just looking at estimates and to always look at and interpret the uncertainties as well. And that's always been just really important about just a piece of a statistician's toolkit when it comes to communicating, especially cross functionally. Because that happens really all the time.
A typical day and tools used
Yeah, so I think this varies a lot seasonally, as major projects come and go. There are some things where it's an all hands on deck effort. Let's say if the team wants to use a fancy historical survival borrowing model that's never been done before. And we need to pull together with their team to spin up the simulation pipeline and have results by the end of the week or the end of the week after that. And that's a lot of heads being put together at once.
And so there are times when, you know, all day, every day might be spent on helping with some simulation projects to, for example, compare these new models and to compare them in terms of power and type one error across a variety of scenarios that might break the modeling assumptions or simulate model misspecification. In quieter times, we work on, we spend a lot of time working on general tools. And there are, of course, meetings to share learnings and things like that, both for our group and across the company. And there's, you know, a lot of time is developing internal tools and capabilities meeting with small groups of about three or four people. And there's a culture internally of seminar series and internally, both for statistics and for tooling. And so those are helpful just to spread around knowledge about things that we're learning and conferences that we go to, and the knowledge that we bring back from those.
Targets for agentic LLM workflows
Oh, that's really interesting because that's, and I haven't gotten questions like that specifically for LLM powered workflows, but I mean, targets tries to create a reasonably deterministic scenario in the sense that it assumes that any randomness is produced by, or apparent randomness is produced by random number generator algorithms native to R and that it has really opinionated control over the seeds that it uses for that computation. And if the temperature is anywhere above zero in those LLMs, then you're really going to get different results every time. And depending if part of a pipeline is an individual analysis, then there's a different set of expectations that you might have to have for a pipeline like that.
I'm thinking of something like using vitals, the vitals package as an intermediary to determine whether you've got a binary pass fail for that particular area of the targets pipeline. Yeah. I think that it could be, if you're doing a vitals pipeline to just, for example, check and compare the performance of different versions of models across, on a set of, or multiple sets of test questions, then I mean, yeah, that certainly would fit in a targets pipeline. And occasionally I've talked to the tidy models team in general about potential integrations. And it seems like they have their own workflow solutions. Some of their stuff is compatible with targets in a sense that you might have a target in the pipeline for each workflow in the tidy models ecosystem. And there are some things that they like to do internally, like different parameter tuning algorithms and things like that. But yeah, whether to rely on the tidy models workflow ecosystem versus targets as a tool that is responsible for a lot of the high level stuff in the workflow is really a case by case project by project thing.
Bayesian vs. frequentist approaches in clinical trials
That's an excellent, excellent question. And I think the answer that most people are familiar with who think about these things is do we have prior information? How do we use that prior information? And do we want to be opinionated about incorporating that through a specific or rigorous prior distribution on the parameters? And do we want to have other techniques that may go beyond priors for historical borrowing? That's a use case that a lot of people are familiar with. But we've implemented, I've certainly implemented my fair share of Bayesian models that run most of the time on diffuse priors.
And the reason is because decisions in early phase get a lot easier to quantify. Because we talk about in statistics, right? We talk about statistical significance. Is this result just the result of random chance versus practical significance? Do we have enough evidence of observing a large enough effect size? And because of the flexibility in Bayesian models, it's much easier to, and much more convenient to answer that second question. You can define, for example, you can more easily quantify the posterior probability of observing a large enough effect. And that's incredibly useful in early phase studies where we're taking the first look at, let's say the efficacy of a therapy and for internal decision-making, we want to say, do we have enough confidence in this effect size to move to a registration study?
And it turns out that not only do you not have to have a specific parameter in that model for the way that you're quantifying the treatment effect, you don't need frequentist asymptotics, like asymptotic approximations, like the Delta method to get at the information that you're trying to look for. You can use MCMC or Markov chain Monte Carlo and transformations on posterior samples without using a Jacobian adjustment. And that alone adds just a lot of convenience.
And I guess I would also add to that, that in Bayesian statistics, we take a philosophically different view of probability. And it's really useful just to, probably to use it, to think of it as a quantification, a quantifying a way to quantify the degree of belief in something in general. So this is what my professor in grad school called the epistemic view of probability. It's a measure of how much we believe something, which is more general and flexible and gets arguably more to the point than a frequentist definition of probability, which is just a long run ratio of repeated simulations of an outcome.
Bayesian MCMC tools
What Bayesian MCMC tool do you use? Do you use STAN or the macro typewriters to get to STAN? STAN is the software that uses the Hamiltonian algorithm to solve the MCMC question. I use BRMS because it's just easier for me to formulate the regression model. What do you use?
That's a great question. I think that STAN is a tool that I go to often for these problems, just because of how versatile it is. The advantage of Hamiltonian Monte Carlo is that as long as your model has continuous distributions for the parameters, then there's a whole lot of different models you can fit. And the syntax allows you to express it really easily because all you need to do is define the log joint density of the parameters in the data. And that creates a lot of opportunity. And it is a really fast, really powerful toolkit. I think that I sometimes have run into models that are just too big or complicated to fit efficiently in STAN. And occasionally, I've just implemented my own Gibbs sampler in C or C++. In fact, my grad school dissertation project, I was computing in CUDA, which is directly on top of an NVIDIA GPU.
One tool that I've used for some frequentist modeling really easily is called RTMB. TMB stands for Template Model Builder. And if you're with the MMRM package, Mixed Models Repeated Measures, that's what it uses. There's a layer of R on top of it now called RTMB. And it's just magic because you just straightforwardly write the objective function in R. And you can plug it into after running it through RTMB for a translation step, you can run it in any optimizer. You can fit it with maximum likelihood or restricted maximum likelihood in really any optimizer in R.
And for Bayesians, there's this package called TMB STAN, where if you have an objective function that you've written in R, you can just like magic translate it into a Bayesian model that runs HMC or Hamiltonian Monte Carlo with STAN as the backend. It's not quite as fast as direct STAN code for big hierarchical models according to their journal statistical software paper, but my gosh, that's going to be the first thing that I go to to prototype the next Bayesian model that I write. And it's really, and the integration with R is just magic.
I mean, for those of you who use STAN out there, the syntax is kind of a cross between C++ and its own statistical helper functions and the going back and forth between, for debugging purposes and the compilation steps are a bit tricky. But just the ability to write out your model all in R and have it run just as well as if you had programmed it in pure STAN or anything, it's really amazing. I can't say enough good things about that toolkit.
AI agents and trust in clinical workflows
As a function, we're exploring this as in-depth and as fast as possible. My group and I were more on the clinical design and modeling and simulation space than we are usually on the analysis deliverable space. But certainly we have a lot of visibility into that space and there's a lot of work that's being done there. And on top of that, there's a lot of everywhere, there's a lot of usage of things like cloud code and codex, just general coding agents to try to take a first crack at that problem. And what we're finding is there are a lot of things that can be prototyped very quickly and easily, but trust is the attribute that's just hardest to come by. And trust is the substance and the currency that's just become increasingly rare these days.
Trust is the substance and the currency that's just become increasingly rare these days.
And for specific parts of that and similar problems, my group and I are really pushing this trusted mini-agent pattern to try to break down problems into ones where we can develop an agent that engineers hallucinations out of the system. Again, to Sarah's point from Gen AI Day, that we design the scaffold so it matters less if the LLM is wrong. And we're thinking about a lot of this in the simulation space where, as you'll see in the last chapter in that guide, that Quarto book that I linked to in the chat a while back, it's an example in trial simulation. The statistics in it are way oversimplified, but it gets to the concept where instead of the LLM producing the results or the LLM just independently figure out how it's going to solve this problem, the purpose of the LLM is relegated to a translation layer between the intent and the goals of the user.
Who might say, hey, I want to simulate this trial under a couple different scenarios, one for power, one for type one error, and explore a couple different sample sizes. Translating that intent into formal simulation parameters that can be checked. And then registered agent tools run the computations in the background. And we have, from the engineering, confidence that results only come from those trusted tools. So the question isn't anymore, do we trust these results? The question is, do we agree with the assumptions? And is this actually the workflow that the user intended to simulate? Is this design, as represented in these design parameters, what the user actually wanted to simulate? And I think that that can be accomplished through engineering and constraints without any prompt engineering. Because prompt engineering really is, according to the trust in many agent model, prompt engineering is about convenience, not correctness.
Career advice
The best piece of career advice that I've gotten from my family is, and this was during my early career struggles, was don't run away from a bad thing, run towards a good thing. And this is really important for me because it was a culture shock going into industry. There was a lot of just soul searching. What is my place in this company? How do my interests and the company's interests align? And is this where I see myself? And those are really big questions. And I'm really happy in the role that I carved out for myself, and colleagues have helped carve out for me. It took a lot of work to get there, and it took sticking with it. And I think that I'm really glad that I did because there was just so much overwhelming positive stuff right where I was. And everybody's decision about their career direction is different, but that's just a particular piece of advice that I got at one of the particularly difficult times in my early career. And I reflect on that often, and it's kept me going.
Don't run away from a bad thing, run towards a good thing.
