Braulio Cuandon & Ana Alyeska Santos | Using R to Create Reproducible Engineering Test Reports
Talk from rstudio::conf(2020) Engineers at Biosense Webster, a Johnson and Johnson medical device company that specializes in diagnosing and treating cardiac arrhythmias, write multiple test reports to comply with FDA regulatory standards. These intricate reports require 36 hours of an engineer’s time on average, constraining the engineers from completing investigations and studies in a timely matter. Writing scripts in R that create reproducible reports can significantly reduce the time spent by an engineer creating these reports allowing them to do a much thorough investigation with a larger scope. Through Shiny, engineers could conveniently have their parameters and recorded data processed and stored in a database by accessing a web link and filling out the required information within a user-friendly interface. Upon the generation of the report, accurate and properly formatted test reports, compliant to both the company and FDA regulatory standards, are produced through Rmarkdown and knitr knitting all the outputs with complete data analysis tools such as normality plots and process capability measurements to a word document that follows company required headers, footers, and headings. The reproducible report creation shown in this report can be extended to other types of test reports and protocols. The pilot phase that has been conducted has shown that complete report production has been decreased from 36 hours to an hour
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hello all, we're going to be talking about how to create reproducible reports with R. My name is Braulio Cuandon, this is my partner Ana Santos. I'll be going over the scenario, the regulations, the restrictions, and the goals we faced in this project. Ana will be going over the project, the program itself, and the program features.
Just a little background, we work for BWI, and last year, around January, July, we were tasked with creating an R program that can write engineering reports for engineers. A bit more background, so BWI, Biosense Webster, is a leader in the science of diagnosing and treating heart rhythm disorders, specifically AFib. AFib can lead to strokes, heart failures, and blood clots. And since we develop a product that comes in contact with the heart itself, we are heavily regulated by the FDA in the documentation section, specifically our reports.
We have strict guidelines to how our reports are composed and presented to the FDA. BWI itself generates about over 1,000 reports a year. Each report takes about four work days, so that's a total of 30 quick hours. So just doing quick math, 1,000 reports, 32 hours, let's say engineer one makes about $25 an hour. It's a lot of money and time wasted. So we wanted to eliminate all that.
However, due to how strict our procedures are, their methods are, we have very strict regulations when it comes to writing the report itself, so strict that even minor errors or deviations from our templates often result in rejections.
Report format requirements
This is actually an example of a report we generated with our program. As you can see, it looks just like a regular Word document, which is great because that's what we were aiming for. We wanted to pass through our documentation system without any kickback, and we could not do that with HTML or PDF, so we had to rely heavily on the Word format. We did manage to get about two reports into our documentation system without the approvers themselves noticing that this was our generated program.
We do have certain regulations we had to follow. For example, all our reports are required to have a cover page, and in that cover page we are required to have a title, the location of the document, the author's name, and the date it was generated. In our headers, we are required to have our logo, the test study, the title, the number, the revisions, and the pagination. All this is really hard to do with Word, and we had very, very many setbacks with this.
Moving towards the body, we do have regulations on our headers. They must be numbered in sequential order, and certain sections must contain specific content. We also had regulations with our table. All the headers should be highlighted in gray so the reviewer could see the difference between headers and actual data. We also had regulations that came to our stat methods. This example of a process capability, whenever we do histograms or process capability studies, we are required to show the CPK. Anything below 1.3 will generate a rationale on why it's allowed.
Report lifecycle and common errors
Just a quick walkthrough for our lifecycle of a report. Like I said before, it takes about four days to generate a report. After we generate a report, we send it out for approvals. From then, it takes about two weeks to hear back from our approvers. Our approvers are really busy. They're usually higher level engineers. Taking about three hours of the time to review our report is another money pit. If they find anything, they're going to reject it, kick it back, and we have to redo the process.
We have to spend about another two hours of their own time and our time to fix any issues, and then the cycle begins again. You can see after constant rejections how many hours of the day and how much time we waste with just writing a simple report.
These are some kind of errors we usually find. We'll be using the wrong kind of template. For example, someone used a Rev G version of the template instead of the newest Rev H. This is very common since our engineers like to get previous approved reports and modify them for their studies. This is another example. When they revise previous reports, they don't verify if they changed all the product names or if there's any contradicting statements. For example, we're talking about catheter body in the purpose, but in scope we're talking about catheter tip. That's a contradiction that will get us in trouble with the FDA.
We're also talking about 32 samples, but in the acceptance criteria we only have about 30. There's also inconsistency with our lot numbers and our units.
Project goals and restrictions
Some restrictions we faced is our rigid system. We had to follow our template to the T or else it would get rejected and we have to lose more time making a report. The second one is we have to submit it in a Word document. Using our markdown with Word was one of the biggest hurdles for us. A lot of the packages do not work with Word output, so we had to figure out little tricky ways to get around it.
Our overall goal for the project was, one, limit it as much time as possible it takes to generate a report, two, make it super simple that it could be done in one click, three, review the user's input to make sure there's no mathematical errors, no grammatical errors, or any stat errors. Then finally, our last goal was we wanted to make this program flexible as possible to be used for a variety of tests. We did not want a one-trick pony report.
Program approach and design
As you can see, you have found how cumbersome it could be to make the reports. What approach did we do to actually improve this process?
First we specified the reference document within our YAML header. The reference document would serve as the backbone of the rest of the documents that we are going to generate. Within the styles of the headings and the Word document, what we did was to edit the headings. For example, we wanted to edit the font, the color, the size, and the indention. That way we could mimic how the text are supposed to be shown within the specified procedures within our document system. We also specified that Word document, of course, is what we wanted as an output because this is the only type of document that is allowed within our document management system.
As much as we wanted to standardize the report generation procedure, we also wanted to give our users the flexibility to actually be able to customize their reports in their own ways because each engineer does different types of testings, and this would require different types of analysis. For this one, we required them to input a raw data spreadsheet, which includes the specifications of the testing that they have performed. We also have text input sections where we could capture the details of the title of the testing that was performed and who is the author of the report. Also to give them a flexibility when it comes to analyzing, we let them choose from a pool of options of what types of plots they are able to show.
Raw data input and validation
And with that, I mentioned earlier that users are required to input a raw data spreadsheet. Now you're going to ask what are the typical things that are within the raw data spreadsheet. So this is an example of a toned down raw data spreadsheet, and within here, the specifications of the testing is shown, such as the title of the testing that was performed, what are the steps that has been done in order to follow the procedure of the testing, and in here we could see the values that were gathered within the testing that has been performed, and also the bounds on which or wherein the measured data should fall into.
And within this raw data input, what we do is check the data for any types of inconsistencies. Let's say there's inconsistencies within dates. We are trying to catch that early on. That way it doesn't get escalated into a higher level problem. We also look into the errors within grammar and spelling, and also we look for drastic outliers because we wanted this to be a potential problem in the future if it's just a little typographical error.
And also, you know, sometimes our engineers are rushing their reports, furiously typing, and sometimes accidentally they hit the caps lock on, and we don't want it to be shown in their report because what we wanted to show is something in title case as shown in the photo below. So stringer is really helpful to do that. The function string to upper and string to title helps us to be able to control this user inputs.
Technical hurdles and solutions
Now the report generation procedures using R and having it in a reproducible manner was never easy. There's a lot of things that are hurdles along the way. So the first one is that the Word document headers were a little bit difficult to edit, especially if you wanted the text to appear dynamically as the users input them. And there's also hurdles within actually outputting the tables that are following the specific procedures that was outlined within our documents. And also having users with different backgrounds, different locations, be able to access this change within the process of generating the reports. And also, lastly, for users to further be able to customize the reports that they're trying to make.
So how did we edit the Word document headers based off the users inputs? So what we did is we have a reference document wherein we have keywords. As you can see, the second photo has quadruple X. So we wanted to edit the quadruple X with the user input. So packages officer is really helpful for this, specifically the headers replace all text function. What it does is it looks for the keyword. And this keyword is replaced by the input, which, for example, is chapter tip diameter. And then with this, this print function will save this in a specific path. And this will be the reference document that will be specified in the YAML header. That way, going forward, all the documents that will be printed by the user will be as such, because we wanted these tiles and revision levels to appear in the actual output.
So now, as specified earlier, there's specific formatting that has to be followed when it comes to the tables that we're trying to generate. And officer and flex table was really useful for such, because we were able to make beautiful tables and actually output it in Word. And Pandoc is definitely essential for having this done, because it gives us a success to be able to output it in Word. And another feature that we like the most with officer and flex table is the style function, wherein we could add a conditional statement. So within the conditional statement, what we do is the measure data, which in this case is tip diameter. We'll look if it is below the lower specification limit or above the upper specification limit. And if it does, then we wanted to highlight it, because this is probably a potential type of concern that we wanted to address early on. So what we do is we change the text of the color in red and also italicize the text. That way, it catches the user's attention.
Deploying via Shiny
And then going forward, we wanted people within the company with different backgrounds, different skill sets, and different locations to be able to access this type of methodology. So with that, we decided to leverage the Johnson & Johnson Shiny server. And within that, we deployed a Shiny app that has this, our markdown, wrapped. And with this, it allowed for users to access it within a web link, put their inputs. And once they hit generate report, they could automatically have the report properly formatted within their downloads folder.
Also, we were able to add more features for further customizations of the plots that the users want to actually generate. So for example, here, if the user uploads the raw data spreadsheet, the column headers within the raw data spreadsheet is red. And once this is red, the dropdown menus for the scatterplot x-axis and scatterplot y-axis is automatically updated with the column headers. Why do we want this? It's because the raw data spreadsheet column headers could be edited by the users, and we want them to be comfortable as possible and easy for them to be able to generate these plots.
Example output walkthrough
This is an example of a plot that was generated by the user. Now we showed you little snippets of how the report looks like by going over a few of our methodology. Now, what's a nice thing on the cake, we want to show you an example output. Keep in mind that these are not actual data, and it's only used for demonstration purposes only.
So that is the title page with the title and the author name. We have the sections with the subsequent parts of the sections properly indented. We have tables. We have the table for equipment. We have procedures that are shown sequentially. We have the failure highlighted in red within the raw data table. Then we have more observations, as you can see. And then further down the document, we have scatterplots. We have boxplots with the failure highlighted in red, as shown as the outlier. We have normality plots. We have capability histograms, and the table below that shows capability measurements that were also calculated. And also below, we could see, for example, the recommendation, and this is a subsequent paragraph that is in the introduction of the section. It's properly indented. And then the actual recommendation is further indented. So these are the little things that we made sure is also captured.
Key takeaways
Now, what do we want you to take away from this presentation? You may think that the methodology shown might not be as complex. But the key idea here that we learned along the way is we wanted to challenge all of you. We wanted to challenge each one of you as you go back home through your individual organizations. We want you to promote the idea of reproducible data analysis with your individual organizations. Because along the way, we've seen how it is important within a company. We wanted you to be able to eliminate documentation errors. And once you do that, you could actually be able to focus on higher-level issues.
We want you to promote the idea of reproducible data analysis with your individual organizations. Because along the way, we've seen how it is important within a company.
Also, this is just one application of reproducible data analysis. And we know that on your different job functions, you could also implement such methodology. And lastly, once you are able to successfully deploy such, I think that you could eliminate bottlenecks within your own systems. And for us gathered here today, I think if we free up some time within our own day-to-day work functions, we could write more scripts, we could deploy more applications, and I think that is a win-win situation for all of us. It's an investment that we're making for further down the line.
Q&A
We have three minutes for questions here, so let's tackle a few of these real quick. Have you tried out the GT package for custom tables? Have you heard of that package? I don't think it's on CRAN yet, but it is a new package that's out for creating custom tables. No, but we might want to check it out. No, we haven't. But we will check it out.
Second question is, how much iteration has this project required? That is, are reporting requirements subject to change? Yes. It's not as often as a new, as you think. I think every year to half a year, we get a new update in documents. The changes are minor. So we just have to change the code slightly to fit the new template.
How were the process control charts produced? Custom package? For the capability histogram, the library, the package ggqc was used. And then the process capability measures is coming from another package that, I'm trying to think about it, capability tools? I mean, whoever asked that question, if you connect with me, I could get back to you.
Once you put the report in Word, how do you handle changes to the document? How do you incorporate feedback edits to the Word doc back into our markdown? The previous question, I just remembered the package. The package used to calculate the process capability measures is QCC.
Once you put the report in Word, how do you handle changes to the document? So if they're asking about the feedback of the approvers, so basically the approvers would put a note within the document management system, and since the output is in Word, it gives the flexibility for the actual author to be able to edit it. But the users don't have access to our markdown file itself. It's only the programmers who have access, if that answers your question. They do have the capability of emailing us, and we can make those modifications to fit the overall report better. Because what we also wanted to control is users trying to favor themselves in the creation of their reports. That's why we still wanted that control.
