Frank Harrell | R for Graphical Clinical Trial Reporting | RStudio (2020)
For clinical trials a good deal of effort goes into producing both final trial reports and interim reports for data monitoring committees, and experience has shown that reviewers much prefer graphical to tabular reports. Interactive graphical reports go a step further and allow the most important information to be presented by default, while inviting the reviewer to drill down to see other details. The drill-down capability, implemented by hover text using the R plotly package, allows one to almost entirely dispense with tables because the hover text can contain the part of a table that pertains to the reviewer's current focal point in the graphical display, among other things. Also, there are major efficiency gains by having a high-level language for producing common elements of reports related to accrual, exclusions, descriptive statistics, adverse events, time to event, and longitudinal data. This talk will overview the hreport package, which relies on R, RMarkdown, knitr, plotly, Hmisc, and HTML5. RStudio is an ideal report development environment for using these tools
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
The slides for my talk are available on my website fharrell.com, so go there, you can get the slides and inside the slides you'll see the links to the demonstration reports that I'll be showing you parts of in this talk. Also if you want to follow somebody on Twitter who likes to criticize poorly done statistics or poorly done machine learning, follow F2Harrell on Twitter, I'd be glad to have you there discuss those issues and more.
Interactive graphics for clinical reporting
So just a brief mention of interactive graphics and what sort of levels of interactivity. Most of you know this really well, but there's full interactivity using tools such as RShiny where you can change your mind about what variables you're analyzing and how you're analyzing them such as how much smoothing to do in a non-parametric smoother. But what I'm going to be talking about today is partial interactivity.
So what sort of things can you do with partial interactivity? You can zoom and pan, rescale axes, provide extra information using pop-ups such as hover text when you hover the mouse over an area. And then very importantly is you can select which traces of data that you want to show, which elements of a graph do you want to show. And then instead of having legends and explanations, I think it's better to have information available on demand. So instead of devoting space to things to explain what are the elements of a box plot, when you hover over that part of a box plot, just have the definition of which quantile you're showing pop up for you.
So I'm using the Plotly package, which I'm a huge fan of, which is using the famous D3 JavaScript graphics library. This in my mind is the best partially interactive scientific graphics system and it has its own language for the graphs. It's something you have to learn. It's much different from ggplot2's language, but it gives you a lot of flexibility. You can also just convert any ggplot2 graphic to Plotly automatically if you like the way ggplot2 laid out things. But if you want to add things sort of trace by trace, you might want to do what I did was go most straightly to Plotly.
This in my mind is the best partially interactive scientific graphics system and it has its own language for the graphs.
So there's some new Plotly graphics functions I've been adding to my existing packages over the last couple of years and I've been adding a lot of HTML methods as I get more away from LaTeX and PDF into HTML. I've added a whole lot of functions for creating customized HTML output that you can include right in your reports. And there's special HTML methods for the describe function and the summaryM function. SummaryM is like for creating table one in a clinical trial report. HisboxP is for making spike histograms with extra information added to them. PlotP is for making interactive survival plots. And then there's advanced HTML table makers using the HTML table package in R.
Philosophy and structure of clinical trial reports
So let's get to clinical trial reports. So the applications I'm talking about are where you're doing, say, a randomized clinical trial to study a treatment versus some control, or you might be doing a drug study, something in the pharmaceutical industry. And there's two types of clinical trial reports. There's sort of final clinical trial reports and there's interim reports as often used in monitoring the conduct of a clinical trial that you might give to a data monitoring committee, also called a data and safety monitoring board.
And so the task that I started with was developing high-level abstractions for the job of creating reports. Most statisticians, when they create a big report, they sort of start over. They may have some code they used from previous trials, but they do a lot of new coding and it's quite repetitive. So if you can make high-level abstractions, I think you can foster better statistical analysis and better reporting practices and make everything reproducible and really minimize programming.
And the authors of reports about human subjects' research get really tired of reading through tables and they usually fall asleep, they don't pay attention to things, and tables are not that good for showing patterns anyway. And so we really want to get rid of that.
Now there's many standard components to clinical trial reports. These are some of them, a summary of how you were able to accrue subjects into the study, how did the patient flow occur and what sort of exclusions did you find between the accrual or the screening of subjects before you randomized them to experimental treatment, describing baseline variables, longitudinal analysis, adverse event analysis, lab safety parameters, event timing and incidence, and sequential monitoring of event probabilities.
So the philosophy is that tables really do not lead to pattern recognition and graphics are better. If you have more than two numbers, I think a graph is better than a table. In other words, graph is almost always better. And graphs should use features that humans are really good at perceiving, such as position along a common scale. And then we need side posts on the graphics. So when I'm reading a long report, I lose track of where I am in terms of, say, which subjects are currently being displayed or analyzed. And I'll show you what that means. Tables are secondary. They can be in an appendix or hyperlinked. Or what's better, I think, is to have pieces of tables as hover text from a plot.
And then this was probably a little more controversial. Most studies where you're doing an A-B comparison, the way an A-B comparison is really created is that confidence intervals for A are not relevant to the design and confidence intervals for B are not relevant to the design. They're actually inconsistent with a parallel group design. But the design is made to give confidence intervals for differences between A and B. And that's what we want to emphasize. We want to show the entire distribution when possible, favor quantiles over means and standard deviations, and then I personally don't like percentages and I do everything I can to get rid of percentages. That's a personal opinion.
So I have an older package that used the LaTeX and PDF model, and this is a very, very stable package. It's been around for many years, used in production in some really huge multinational clinical trials. And then the newer package, which is not on CRAN yet, which is hReport for HTML report. So in these packages, there's lots of utility functions, high-level report components, unified handling of figure generation and caption. So when you write a long report, you spend a lot of time writing figure captions. So the figure captions are all automatically generated by these packages. And then there's some new graphical elements.
So the high-level functions are accrual report, exclusion report, descriptive statistics, event report, and serve report for time to event analysis. Now when you're writing a package and you want to implement things at a high level, you really have to survey best practices, and it's your opportunity to implement best practices and also to create best practices if people like the package. And so instead of doing one-off reports, if you start saying, what should be in a patient screening report or accrual report, and really nailing that, and then making it so that people can join the project on GitHub and they can extend the report with new options or change the format, it's much better than just coding from scratch for each report.
Number at risk report, and in the future, there'll be a sequential monitoring boundary sort of report. And the format of all these functions, the way you call them, is using the formula language. You have your analysis variables on the left-hand side of the model, separated by plus, and then you have your stratification variables and optional ID variable if you're analyzing something where there's multiple records per subject.
So rmarkdown creates HTML documents, and these allow semi-interactive graphics. So if you're using Plotly, this uses HTML widgets, and those are included in your HTML without you doing anything special whatsoever, and they'll display beautifully on virtually any browser, and they'll automatically resize for viewing on a tablet, and it's very effective to view these reports on a tablet. And then our functions write HTML. They write regular tabular output, hyperlinks, navigation bars, and almost no tables. And then hovering over a part of a graphic will display the relevant portion of the table. So I don't display whole tables anymore, but just portions of tables on demand.
Walkthrough of example reports
So now we get into the two examples, and then if you download the slides, you can click on these links to see them in much more detail.
The first thing you notice is we have a new table of contents function in the HMIS package that it doesn't waste so much space on the left with your floating table of contents, and then it allows you to specify the level. So if you click on three, you'll get a three-level outline, and if you click on two, you just get a two-level outline, and one is a one-level, but the most important thing is to be able to get rid of the outline altogether.
And so the other thing that we have in this, you see that little L? That's a little figure icon for a graph, so that means this is a figure, and this is a short figure caption, and clicking there will go to that figure.
This has a bunch of philosophy stuff, and then notes about figure captions, and these are the signposts that are used in each graph. And so to keep me aware of where I am in a report, I have these signposts below each graph. So what that means is you have a little bar here where the red is telling you how many of the enrolled subjects are included in the current analysis being displayed. So if you saw a bar like that that's full height, that would mean you're analyzing all the enrolled subjects. This would mean you're analyzing three-quarters of the enrolled subjects, but all of the randomized subjects. This would show that you're analyzing one-fourth of the enrolled subjects, and only one-half of the randomized subjects. So the green is the proportion of randomized subjects you're currently analyzing, and then if it's stratified for treatment, you'll get two more, and the last two bars will tell you what proportion of those randomized to one treatment are currently being considered in the analysis, what proportion of randomized to another treatment B are being considered.
And then this is the first output, and you can click on code to see the code for any of these. This is the accrual report. So this actually took one of the longest times to write, because when you have a multi-site clinical trial, that may also be a multinational clinical trial. You have countries, regions, sites, so it's a very hierarchical organization. And to really summarize a multi-site, multinational trial the way that you need to, took a little bit of thought, but this is a trial snapshot. So you get your overall summaries about the trial content, and how many people are passing the screening stage. Then you get your accrual, and this is where Plotly comes in, because you can just hover and see your target numbers of patients you should have accrued by that time, versus the actual number that you've accrued by that time, and you see that's May 1994.
There is the little signpost over here, which you can barely see probably from where you are, and there's some fine details showing you the number of non-missing observations analyzed and the groups being analyzed. You can just zoom in on that to see detailed denominators. This is just a dot chart showing the number of sites, how many subjects were randomized at how many sites. So looking at the randomization frequency over sites, and then we have another graph like that.
Then we start getting to baseline variables, and this is where we start using the features of Plotly more effectively. So this is just showing the proportion of females in treatment A and in treatment B, and you see on the right the legend shows you the color code for treatment A and B, which is used throughout the report, and then if you hover over that, you'll see that this is .222 of the subjects were female in treatment A, and this was for all regions, and you can see the numerator 18 and the denominator is 81. So one thing I learned early on in this is you always give numerators to denominators because somebody's going to ask you. You don't want to leave it to guesswork.
The other element of this graph, and of course you can turn things off by clicking here. So if I didn't want to show the stratified data by region, I can just click on the right, and that will turn those traces off. That's just a nice feature of Plotly, which we'll see more in a minute, but the other design consideration here is when you're showing the proportion randomized to A and proportion randomized to B who smoke, which is these two dots down here, you also see it stratified by region, and you can see that this is north, the north region here. So I think of the stratification by region as something that's subservient to the main stratification. So the main stratification is treatment A versus B, and if I want to further stratify by region, those dots are below the main dots, and they're smaller. So to denote that those are subservient or minor stratification, I'm using a smaller size, and if you click on the legend, you can turn those off.
Displaying continuous and longitudinal data
Now this is the way I've settled as my favorite way to display continuous data is really you need to show the data, and box plots don't really have a good inked information ratio. They just don't show enough. A box plot will not show by modality. It'll completely hide it, but a histogram will show by modality. If you have digit preference in your data, more numbers entering in five or zero, you'll see digit preference very clearly, and so I found the secret formula for how many bins there should be in a histogram. It's either 100 or 200, independent of the sample size, but if you have fewer than 100 distinct values in your data, you would use that fewer number as the number of bins, but you essentially have one bin per possible value up to about 200 bins, and that way you have all the resolution you need, and it scales to huge data sets, but in addition to showing the histogram, which has a lot of information, you can show the elements of the box plot plus more. So you can see that below each histogram, you see the mean is with a dot, the median is with a big vertical bar, and then you see the quartiles in the .05 and .95 quantiles. So you see all the box plot information plus more, but the histograms are more informative, and then if you wanted to turn off the box plot stuff, you just click here, and that's no longer there.
So you can easily turn that off. So that's how I display continuous data. That's stratified by treatment. You see this for a bunch of variables, and you'll see on all these displays, like here in the fine print, you'll see denominators for each variable being displayed. That's a number of non-missing values.
Then there's various displays for longitudinal data. This is for longitudinal binary data, like adverse events. I display that lots of different ways, and then for just showing overall incidents of adverse events, I just use a dot chart where each dot, you're showing the proportion that have that event, and if you hover over it, you'll see this is diarrhea treatment B, 20 out of 169 subjects had that event. But then you have a confidence band for the difference, and if you position that at the midpoint of the two, and it's the half of the width of the confidence interval, that has the property that it touches those two dots if and only if there's no significant difference at that alpha level that you're using for the confidence interval.
And then if you hover over the bars, you will see, no, I can't get that to pop up right now. You're supposed to see the actual confidence intervals for the difference, and you can turn those confidence intervals off if you want by clicking on the right.
Exclusion and survival reports
Lots of ways to display longitudinal data, and then I want to show you just one item from the other report, this is the exclusion report. And so it analyzes what is causing patients to be excluded, and it doesn't ask you what order to analyze those in. It finds the major exclusion, which would be here, no significant coronary artery disease, that excluded 11 subjects from the trial. And then what excluded the next largest number of subjects was pneumonia within six weeks. And then for each exclusion, it reports the incremental exclusion, so how many new subjects are excluded because of pneumonia after you've already excluded those without CAD. And it shows the marginal exclusions, how many people will be excluded because of pneumonia, whether or not they were excluded for anything else.
And then in terms of survival report, this is the kind of output you get. And what I really like about it, besides having a confidence interval for the difference of two Kaplan-Meier curves, or this is one minus this, so this is cumulative incidence curves, is it recognizes that the number at risk is really a continuous variable. So I don't want to just show the number at risk at yearly intervals. I want to show that as a continuous function of the follow-up time. So as you hover over these different areas, you will, you'll see that, you'll see the number at risk at exactly that time point. It's just another way where I think you can use Plotly very effectively.
So I don't want to just show the number at risk at yearly intervals. I want to show that as a continuous function of the follow-up time.
So if you get the slides and go into those clicks, you'll see those, and more information can be attained at these three sites. So thank you very much for listening.
Q&A
So one of the more popular questions, is there a function to make consort diagrams or can you do alluvial plots for patient flows? That was asked, might have been the person asking it now. I asked that on Twitter the other day, and somebody responded to use one of the general figure drawing packages in R, like, I think, I don't know, it's TICC-Z or one that uses GraphVis. But I don't think anybody's packaged those sort of things into a ready-to-use function. I think it would be nice.
Maybe we'll do one more question. How can a staff statistician working in clinical trials, do you think, convince PIs that it's worth it to switch over to HTML reports from Word or PDF? It'll depend on the age of the person on the committee. So I found that every committee has one person that's as old as I am, and they want to see paper, not just PDF. They actually want to print the darn thing. And that's kind of a lost cause. So I would stick to the younger committee members, giving them state-of-the-art, and then letting them embarrass the older committee member.
I'm a big fan of HTML output, so I appreciate that answer, Frank. Thank you guys all for attending. That's the end of this session. Give a round of applause to all of our speakers.
