Ellis Hughes | Approaches to Assay Processing Package Validation | RStudio (2020)
In this talk I will discuss the steps that have been created for validating internally generated R packages at SCHARP (Statistical Center for HIV/AIDS Research and Prevention) and the lessons learned while creating packages as a team. Housed within Fred Hutch, SCHARP is an instrumental partner in the research and clinical trials surrounding HIV prevention and vaccine development. Part of SCHARP’s work involves analyzing experimental biomarkers and endpoints which change as the experimental question, analysis methods, antigens measured, and assays evolve. Maintaining a validated code base that is rigid in its output format, but flexible enough to cater a variety of inputs with minimal custom coding has proven to be important for reproducibility and scalability. SCHARP has developed several key steps in the creation, validation, and documentation of R packages that take advantage of R’s packaging functionality. First, the programming team works with leadership to define specifications and lay out a roadmap of the package at the functional level. Next, statistical programmers work together to develop the package, taking advantage of the rich R ecosystem of packages for development such as roxygen2, devtools, usethis, and testthat. Once the code has been developed, the package is validated to ensure it passes all specifications using a combination of testthat and rmarkdown. Finally, the package is made available for use across the team on live data. These procedures set up a framework for validating assay processing packages that furthers the ability of Fred Hutch to provide world-class support for our clinical trials
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Thank you all for being here today. I appreciate you coming to this talk. So how many of you have written a package before? How many of you have participated in a validation? How many of you have validated an R package?
What is computer system validation?
So before we get going, what is computer system validation? So that is establishing documentary evidence that our software performs some sort of task, activity, process, in compliance with the specification we've laid out with a high degree of assurance. So there's a couple of very important aspects to that. It's documenting the evidence. And then it's compliance with the specifications with a high degree of assurance. So you are sure that it will continue to behave the way you expect it to.
So part of the reason that we do validation is it's required for FDA submission. There's the 21 CFR 11. And there's also some general principles of software validation that was released by the FDA in 2002, giving us some guidelines about how we should be performing our validations. But that isn't the only reason we should be doing validation. Validation has a lot of other benefits as well, such as improved quality and safety with the products that we generate. It results in faster processing, because we already have a system in place. And it promotes trust with our outputs, because we've already tested it in a variety of inputs.
So we already know that it's going to produce good results, or it'll error out if it doesn't. Validation is very different from verification. And often, they're confused. So we're going to go over that a little bit too. So validation is, does the system behave as we expect it to? Will it return the correct value given the input? Or will it throw an error if the values are outside of our expectations? Verification on the other hand is simply do two separate processes on the same data set, for example, give us the same output. So it's similar, but different.
But for small companies, validation can be a high bar. There's a lot of labor hours in validation. And there's tracking the progress. There's writing everything around that, and tracking the documentation, which can feel like a lot of work. You're constantly filling out this paperwork that's outside of your coding.
For validation, it feels a little excessive, because you first have to develop and get sign-off on your specifications. Next, you're going to have to write the actual code following good programming practices and record who wrote which functions. Then you have to document and develop your test cases and get sign-off on that. Next, write your test code based on your test cases and record who wrote that. Manually evaluate and document the test code results. Once again, getting a third party to do that. And finally, review all this documentation that you have and get final sign-off for release. That's a lot of work.
But I'm here to tell you that validation and R can be best friends. They can work together to create an awesome product. Using rmarkdown, testthat, and roxygen2, we can create PDFs that look like this within our packages that track who wrote the specifications, who wrote the functions, who wrote the test cases and test code. Capture our specifications as we wrote them and the test cases within the PDF and finally display all the test results surrounding them.
But I'm here to tell you that validation and R can be best friends. They can work together to create an awesome product.
So my name is Ellis Hughes. I'm a statistical programmer. I have a history working in statistical genetics, and currently I work in HIV vaccine research. I'm also really involved with the community. I'm a Seattle Use R organizer. And this past year I helped run the Cascadia R conference, which was really rewarding. Fred Hutch is known for cancer research, but they also do vaccine development. And that's the group that I work within. It's called SHARP. We were established in 1992, and we've had a worldwide impact on HIV and AIDS vaccine research.
The five steps of validation
So here's the five steps that I would say are part of validation. First you have to have your specifications. Then you write the code. Then you write your test cases. Then you write the actual test case code. And finally you document it all together. I'm also assuming that you're going to be writing a package for this validation.
So first, when you're working on your specifications, you're trying to lay out what are the requirements of your package. What will the package do? What are the expected outputs of your package? And what are the high-level steps that are going to be followed? So this isn't going into detail. This is laying out the groundwork for everything.
So if I were to be writing a specification for my presentation today, I'd say the contents of my presentation will cover my team's approach to validation. It'll be roughly 15 to 20 minutes long, and hopefully it'll be entertaining.
So when I go to record these specifications, I'm going to want to use rmarkdown to document all this for me. I'm going to use a single file per specification and number those specifications within it. In addition, I'm going to use roxygen2 to add details around who wrote that specification and when, and what the specification title is, and then I'm going to put this inside my R package.
So the benefit of putting our specifications within our package and writing it in rmarkdown, it allows us to really quickly update the specifications if any of them change. Additionally, we can change the ownership. If somebody goes and updates the specifications, they are already right there for them to update. By putting it inside the package, it's close to the task at hand, so it's really easy to reference. You don't have to find a third-party document to pull that information in. But also by separating out the specifications in their own files, it forces you to decouple your specifications so they're not reliant on one another, so you can make very clear specifications.
These specifications are Markdown files. We suggest putting them under the vignettes folder, creating a validation folder, and finally creating a folder for all your specifications and dropping all of them in there.
Documenting code ownership
So typically, when you're documenting the code ownership, you're going to be using an external resource, such as Word, Excel, maybe Smartsheets, and this is really cumbersome because it's outside of your programming world. You have to take a minute and step out of it and go and record, oh, I updated this function on this date, and here's what happened. And that really just takes you out of their flow of programming.
What I suggest doing is using the section tag in roxygen and documenting the last updated by and the last updated date. So you can record who updated it and when, and it's right there within the function. You can also do it at function creation. The benefit of using these roxygen tags is it doesn't alter how it would behave when you're trying to compile your manuals, and it actually has the added benefit of recording it in the manual who wrote the function and when they last updated it, so you can track that.
But now somebody came in and they updated my function. They changed how long the function is going to wait. And so this was not done by me, and it was done today. But it is right there in the code. They didn't have to step outside and remember to update it in the Smartsheet or the Excel file, but it's able to do it right there within the function and in the commit, it's tracked if you're using version control.
Writing test cases
Now when you're writing your test cases, so test cases are where you're actually going to draw the connection between the code that you wrote and the actual specifications that you have and how to pair them together. These are distinct from unit tests. They're often confused with one another, but unit tests are for testing as the function author does your function behave the way you're expecting it to. Test cases, on the other hand, are for documenting how you've matched up with your specifications.
Similar but different. A single test case can satisfy multiple specifications, conversely, a single specification must match to at least one test case, so you have to prove that each specification has been met. So the way you go through and document your test cases is first you record which specification is going to be met, specify the required input data for your test case, record the steps you need to be following but without writing any code, then detail the expectations that they'll need to be testing to prove that you've met these specifications. You'll use rmarkdown to perform this task, and you'll use roxygen2 to document all that additional information, such as who wrote that test case and when.
So this is what a test case might look like. Once again, for my presentation, I would first see that I met my specifications by gathering a captive audience of coworkers and probably later my significant other, give them the presentation and ask them, if it was informative, ask them what they learned. I can then time my presentation to see if I was actually between the 15 and 20 minutes, and then I can see if I was entertaining by the number of eye rolls from my significant other or chuckles from my coworkers.
By using this modular test case format, it makes it easy to shift and update your test cases so you can, as you get more information, you can change them or make sure that you're hitting all the specifications. These will also live under the validation folder you created, but create a new folder called test cases.
Test case coding
Now test case coding. This is where you actually implement the new test cases that you just wrote in rmarkdown. You're going to record the results of those test cases, and then you're going to have a third party write this code. It's critical to have this third party do it, because they have no prior knowledge of the way the function should be behaving. The benefits of this is it helps resolve any interpretation errors before you release it. They're able to go through your documentation, because they don't know the way that it's supposed to behave, and see if there's anything that's unclear about your arguments, the way you set that up.
Additionally, they'll be going through your test cases and seeing how that works, and if there's any ambiguity in the way that they're supposed to be processing it, they come back and ask you, and they can improve your test cases, improve your code. They can also potentially identify any places for improvements in your code itself. They could have a preconceived notion with the way how quickly things should be running, and by going through it, and if it runs slower, they can bring that up to you, and you can go back and update it.
The approach that we're going to take with this test code is we're going to use a familiar format called testthat. I think most people that have developed a hack to know that. We're also going to augment these test cases with roxygen2. I don't know if you've noticed, but there's a bit of a theme.
This looks like any other test case that you might run normally, but we've added this roxygen documentation about what the test is, who wrote that test, and when they wrote it. It was not done by me, so it's a third party. They did it today, and they wrote out all the expectations that they were supposed to be doing. In this case, the test case was rather simple. They were just putting in a simple function and seeing what it returned, but this is a simplification of the idea that can be generally applicable.
We're using testthat because it's a familiar framework. Most people, if they've written a package before, already know how to write tests using testthat. They can develop their test code interactively, meaning they don't have to just send it off to batch and have it run and see what the results are, but it does allow for automation, so we can actually execute all of the test cases at once within a file and capture all that information in what's called a reporter object. These are objects output by testthat, and they track each test and expectations.
It will report a success if each expectation passes, and it'll report failure, and for each one of the expectations you have, track all that as well. The benefit is also that we can easily access the contents. This is a standard output of a reporter object that you're probably familiar with if you're building tests and do control shift T in your RStudio IDE, it'll output that, and you could do that if that's acceptable by your organization, but we were required to actually make a nice printed output of a table where we're recording each of our expectations and whether they were as expected, or it'll report the failure mode, and then, additionally, a pass fail surrounding that.
By writing our test code additionally in this modular format, it makes it so that we can update the test code and track who updated that when independently. If somebody works on a function, but they can then still do test code development for functions that they haven't worked on yet, these will live within the validation folder, as I said earlier, this time under test code, and number them from however many test cases you have.
Generating the final document
So, the final document, what we're gonna be presenting at the very end is, so what do we have so far? We first have the documents written in Markdown for the specifications. Next, we have roxygen documentation and all our code that records who wrote what. We have documents in our test case, or excuse me, we have our test cases written in Markdown that we can scrape that as well, and we have our test case code that is written in testthat augmented with roxygen documentation, so you can use roxygen2 to scrape all the lists of ownership from all of our various locations.
We have our Markdown, so we can pull that out of the files and print that out nicely, and we can use that to generate our final document. Finally, we have testthat that records all the results of the code, and we can then print that into a nice table. So with their powers combined, we can generate that PDF that I showed you earlier, but now you have some reference, and each of the specifications and all the documents that I showed you earlier were used to compile this PDF, where I wrote the specifications, but I did not write the most recent version of that function.
Additionally, the specifications of what my presentation must be and what I was going to be testing and the expectations at the very end are all documented there, so you can grow this as many tests as you need to to pass all your specifications. So this is what the final file format might look like. It doesn't look any different than another R package, but it has this ability to have validation stored within it and makes it very simple for you to validate, or makes the process very simple. So validation in R can truly live together forever.
So validation in R can truly live together forever.
So I'd like to take a second to thank my team back at Fred Hutch for helping me to work and develop this idea and this presentation. Thank my wife for listening to me give this presentation a number of times. This presentation is available at thebioengineer.github.io, slash validation. The HTML behind all of the slides, as well as a sample package, lives on my GitHub, slash validation, and then you can find me on Twitter. Thank you.
Q&A
We do have a couple questions here for you, and we have five minutes, so let's go ahead and tackle these. The first one is, how does version control EG get fit into validation? Is that a viable alternative to tags inside the code itself?
So that depends on your org's take on version control and how they want to approach that. This makes it so that we can track this outside of version control and makes it independent of any of that. So, I mean, potentially if your org would allow that, but it's all dependent on your org.
Question two, how do these methods of validating internal packages compare and contrast with how you validate external packages? Well, I guess that would be, once again, dependent on your org, how you want to approach that. So, I mean, eventually it would be really cool if we get all packages to start doing this approach, especially if you're going to use it in pharma, where we can have this document of truth going, this is what I tested, here's what I was testing against, and here's what the results were. I'll also add, there's the R validation hub that's looking at that challenge.
Ever looked at how CI or CD tools could fit into this workflow? I definitely think there's an opportunity to use that as part of your build process, where you can have it build this validation tool for you. What I didn't go into is the idea that we would then print out this PDF at the end of the day, have people manually sign off, because we don't have e-cigs yet, and we can then have a final Word document that's like, this is truth. We've signed off on this manually in wet ink, but it was generated automatically for us.
I got two last questions here we'll tackle real quick. The first one, how do you document co-testing coverage? So there's several tools in R to do that in generic test cases. But with these test cases here that we're documenting, we don't necessarily have a direct way to test that. That would be definitely a good feature to be adding, is tracking which specifications are being met by which, so you can create a matrix. That's definitely a good idea.
