Joe Cheng | Shiny in production: Principles, practices, and tools | RStudio (2019)

All right, well, I'm really excited to talk to you guys today about Shiny in production. But by way of introduction, I'm going to take a little diversion and take a look at an example app that I've been working on, and we're going to use it to illustrate some of the principles and tools we're going to talk about today.

This is not the example app. That'd be a little weird. I want to talk about cloud.rproject.org, or if you look in the CRAN mirror chooser, it's called zero-cloud. This is a CRAN mirror that you may or may not know is run by RStudio. And it's hosted on Amazon AWS, so it's pretty fast from anywhere in the world. But the thing that makes this CRAN mirror interesting is that the download logs are freely available to download, thanks to Hadley.

This is what the data looks like, just a simple CSV file that you can download, and it tells you every individual package that is downloaded from our servers, and an anonymized IP address. So lots of people have done really interesting things with this data, and what I was interested in is I just had this hunch that every public-facing website that I've ever set up that got a significant amount of traffic always had just a few IP addresses that would act crazy. Just download way too much of whatever it was that we're offering, not in a greedy way, but in a totally WTF way, as you will see in a moment. And I was curious, does this happen on our CRAN mirror as well? Certainly it's a very, very popular service.

So I wrote this application called CRAN Whales, and I'll give you a look here.

So this application is loading right now. It has two parameters that I want to focus on. On the top left, you can pick a date that you're interested in, and then the top N downloaders. So in this case, by default it shows the top six downloaders, the six most active IP addresses. On the right here, you can see there's some summaries about how much total traffic there was. There's three terabytes of data downloaded on the 14th, and however many files were downloaded. And for all those IP addresses that we anonymized, there were 69,000 unique IP addresses.

At the bottom here, there is a plot of the hourly activity for downloads, and you can see that it kind of ebbs and flows during the day. And that highlighted portion at the bottom, that is the fraction of the downloads that the top six downloaders are responsible for. 70,000 downloaders, and six of them are doing that. And depending on the day that you look at, it can vary quite widely.

The second tab will let us look at these whales. I'm calling them whales, the top downloaders. So you can see here, these are the top six, and the height of the bar represents the number of downloads that they made. So PowerfulArgali, these names are randomized, obviously, PowerfulArgali downloaded as much as the next five people put together. But ZealousRabbit is kind of a distant second here.

The next tab will show those same six downloaders, or we can change that number. But now we're looking at their downloads over time. So for each hour, how many downloads did they make? And ZealousRabbit was pretty steady throughout the day, whereas StickyChinchilla... I haven't seen these names. They're different every day. So StickyChinchilla just had kind of a burst in the middle of the day.

And finally, maybe the most interesting view is we have a detailed view where we can actually look at the individual downloads of each of those users. So this is where I go, you know, what the hell is going on here? ZealousRabbit downloaded eight packages 72,000 times. It's probably a Python script gone wrong. 33.4 gigabytes downloaded.

Whereas StickyChinchilla looks like maybe they downloaded all of CRAN, 13,000 packages maybe. They downloaded the current version of CRAN, and then they said, oh, let's go back and get some more. I'm not sure what happened there.

So really weird stuff happening. Strange things are afoot on CRAN, and I'm... If you have any ideas, if you're running one of these scripts that are hammering us, well, number one, maybe stop. But also I would like to know what's going on.

Why this example matters for production

So why am I talking about this? Why am I showing this fun little example app? It's just an example, but there are things about this example that feel a lot like production shiny apps. So it's a useful stand-in.

Number one, this kind of app is very natural to build in shiny, right? Kind of down the middle of what shiny is good for. You're visualizing data that is kind of like in a tidy format. Super easy to build. Number two, a lot of production apps have data that is updated over time, often on a predictable schedule, and that's the case here, because the CRAN logs are dumped once a day.

Number three, there's significant computation happening. When I first brought up that application, it took maybe eight or ten seconds to load. It's loading hundreds of megabytes of CSV data and then doing lots of grouping and filtering to do all the counting of individual downloaders.

And mostly aggregated data is being shown here. We're talking about, you know, millions of rows of data, and we are doing various grouping and counting to show, you know, simpler plots, and that's the case also with most production apps. You're looking at summaries of large amounts of data.

What does "production" mean?

Okay, so before we go any further, let's talk about what production even means. I mean, we throw around this word a lot, right? And I define production here as there's software environments that are used and relied on by real users with real consequences if things go wrong, right?

So real users, not developers and testers that, you know, they understand if something goes down. We're talking about people who really need your stuff to work, and there are real stakes and real consequences if things go down. So the sort of antonym or some not production environments would be proof of concepts, prototypes, sandboxes. These are places where you expect things to go wrong, and in some cases, like testing and staging, you kind of want things to go wrong. That is not the case with production. You're always having a good day if you find bugs and or have outages on production.

So I'm using the word software environments. What does that even mean? Well, it really means the network hardware and software and your code, all of that together when you make a deployment of a Shiny app. So this is just one example of what a production environment might look like. You've got RStudio Connect in the middle running your Shiny application. That could be Shiny server instead. You have Nginx, which is just a web server sitting in front if you like. And this particular Shiny app is running on Connect, and it's connecting to an Oracle database and a Spark cluster, say. So all that together, if that's being accessed by real users, that is your production environment.

And what we generally do if you're following best practices is you have an identical setup for staging. You want it to match as closely as possible to what you're running in production. So you duplicate that entire stack so that when you are using staging, you're really doing kind of like a dress rehearsal. And you want to find all the bugs and work them out in dress rehearsal before it's showtime.

Goals for production environments

When we're running code in production, when we're running production environments, we have several goals. So first goal is to keep it up. We want to keep our production applications up and running. Number two, we want to keep them safe. The data may be private, the functionality might be privileged, and the code might be proprietary. We want to keep all that safe from unauthorized access. Now, that may not be the case for you. Maybe you're dealing with all open data and your code is open source, but you still don't want your server to get hacked, for example.

Number three, we want to keep it correct. We want our applications to be relatively free of bugs. We want them to give answers that are correct. So we want to be sure that we can verify that.

And finally, we want to keep it snappy. When you're working individually with R on your own laptop, the performance might be a little less important. But when you put an application into production, you are probably going to have multiple people simultaneously using your code, which means performance is a much bigger concern than it normally is. So we want to make sure that our applications run fast enough for the traffic that we expect to get and have the ability to scale if we need to.

Can Shiny be used in production?

Now, a question that I have heard many times in the last few years is, can Shiny be used for production? And my answer is, why do you keep asking me that? Of course. So for years now, yes, you can use Shiny in production. People have been using Shiny in production, including RStudio and many, many of our customers. Yes, it has always been possible to use Shiny in production.

That being said, we've worked really hard on the Shiny team and elsewhere in RStudio to make the answer in 2019 more than possible. We're trying to make it really easy to run Shiny apps in production. And I feel like that bar has been achieved, personally.

That being said, I feel like it wouldn't be in RStudio keynote if we didn't kind of tell you about the limitations and the challenges to our own software. And in the case of Shiny in production, there are really three categories of challenges. There are cultural, organizational, and technical challenges if you want to put Shiny apps into production.

Cultural challenges

So talking about cultural challenges first, Shiny apps are developed by R users. Well, I certainly hope so. R users, by and large, don't come from a software engineering background. And if you don't come from that background, if you don't come from a background of deploying production software, you may not even realize that you are dealing with a production situation. You may not think about the fact that people are going to be relying on this code to be running day after day.

And even if you do know what the stakes are, if you realize what the stakes are, you may not know what the best practices are for writing production applications and deploying them.

And even if you do know what those best practices are, often people don't anticipate or budget for the amount of time and effort it takes to productionize and follow those best practices. If it only takes you three weeks to create a Shiny app, and you're getting ready to deploy it, and then now you realize that there's three more weeks of work to do the appropriate automated testing, to do performance and scale testing, to think about security and do those kinds of reviews, it can be really tempting to just cut the corner and just push out to production, especially if nobody's standing there kind of looking over your shoulder. And we have definitely seen people do that, and sometimes they pay the price.

And finally, our users, by and large, I think as a community, we're not a culture that prioritizes runtime efficiency as highly as other communities. Like the other extreme would be like the C community. We're generally more interested in expressing ourselves quickly, iterating easily, having expressive APIs, and while runtime efficiency is important, it is not the most important. And that can be a detriment when we start talking about production applications.

Organizational challenges

There are also organizational challenges. If you're working with an organization that has IT and upper management, they can be skeptical of deploying Shiny in production. I hear this all the time from people that the data scientists want to build Shiny apps and deploy them widely, but then IT and management stands in the way.

Now, IT departments just sort of inherently, their incentives drive them towards conservatism, and I actually think that's appropriate. I worked in IT as my first job, so I have some sympathy here. They're probably going to be skeptical at first of data scientists creating production applications. I mean, it's not, this is not something that they're used to, and there's definitely going to be a little bit of a credibility gap here.

And of course, IT and upper management, they're always going to be skeptical of technologies they haven't heard of, and a couple of years ago, lots of IT departments that we talked to didn't know what R was, much less Shiny. That's changed a lot now. I think way more of the conversations we have now, people come to us knowing about R and Shiny, but it's definitely something that's still to be overcome in some organizations.

And elsewhere in the organization, if there's an engineering department, a software engineering department, they may not be on your side. This is something that they're used to doing, and now data scientists are coming in and saying that they want to do these things too. And they may be particularly biased against R, just having heard that it's a DSL for statistics, not a real programming language, which is one of my pet peeves when people say that about R. I think that's sort of the opposite of the truth.

Technical challenges

And finally, there are technical challenges, and these are my favorite kinds of challenges because they're the ones that I can do something about for you. Now Shiny dramatically lowers the effort involved in creating a web app. I hope that's not a controversial statement. We've made it very easy, but what we haven't done is made it any easier to do the other things that are involved besides creating your app that you still have to do before going to production. We haven't made it particularly easy to test Shiny apps. We haven't made it particularly easy to load test Shiny apps, and we haven't made it particularly easy to profile and deploy Shiny apps until relatively recently.

And finally, R can be slow, and it is single-threaded, meaning that an R process can only do one thing at any given moment. It has to finish whatever it's doing before it can do something else.

So the good news about these challenges is that we can and have done something about the effort involved in doing testing and profiling and deployment, which I'll talk about in a moment, and about the slowness in 90-plus percent of the cases that I've seen, R's not the problem, and we'll talk about that as well.

So the good news about these challenges is that we can and have done something about the effort involved in doing testing and profiling and deployment, which I'll talk about in a moment, and about the slowness in 90-plus percent of the cases that I've seen, R's not the problem, and we'll talk about that as well.

Tools for Shiny in production

So the rest of this talk I'm going to be focusing on the actual tools and software that we've been working on to help make it easier to do Shiny in production. And by the way, I probably should have said this up front. If you are coming from a setting where you are not building Shiny apps for production, that does not mean that these things will not apply to you. Even if you're not putting apps up that people are going to access from all over or whatever, if you care about your app being correct and staying correct, if you care about performance, these tools are still going to be relevant for you.

A bunch of these tools I'm going to not talk about, and then other ones I'm going to focus on. So RStudio Connect, you're going to hear about plenty this week, but it is our way of serving Shiny with push-button deployment. It's an incredible product. We're investing a lot in it. ShinyTest is an automated UI testing framework for Shiny. We did, Winston Chang did a talk on this at last year's RStudioConf, and you can download that video, and there's documentation here as well. That is now on CRAN, which it was not last year.

What I am definitely going to talk about today is Shiny Load Test, which is load testing for Shiny, which is more exciting than it sounds. ProfViz is not new, but it is a profiler for R, and I think it's so important that I want to emphasize it again today. So I'll be doing some demos there too.

Plot Caching is a new feature that we've released in Shiny 1.2, which came out, I think, in December, and we'll talk about that in a moment as well as a way of speeding up plots. And one last thing that I will not be talking about today is Async, which is sort of a last resort technique when your Shiny applications have inherently slow portions, and I did a talk about that last year at RStudioConf. Again, there's a video available, and there's a very, very detailed website at rstudio.github.io slash promises. Unlike every other piece of software I've ever written, I am asking you if you are interested in Async to read the documentation in order. I have numbered the articles to kind of emphasize that. If you go out of order, nothing is going to make sense. So Async is hard. That's why it's the last resort. But if you need it, you really need it, and I encourage you to study carefully.

So we'll be just focusing on these three areas today, Shiny Load Test, ProfViz, and Plot Caching.

Setting performance targets with CRAN Whales

Now we're going to look at these three tools through the lens of Cranwhales. Cranwhales was pretty slow, and I'm just picking some random numbers here to set a target for us just to give a for instance. So let's just say we want to support 100 concurrent users for this application, which would probably be equivalent to an average app at a relatively large organization. And then, again, just picking numbers, let's just say we want to support this on one dedicated server with a 16-core CPU. In a lot of cases, you'd be running multiple servers just for redundancy, if nothing else. But in this case, just to keep it simple, let's say 100 users, 16-core CPU. So it'd be great if each of our processes could support between 10 and 20 concurrent users, although we'll definitely, if it's more, great. But let's just start with 20 as a test, and we'll see what happens.

So our overall high-level view of how we're going to test the performance of this app is we're going to start by using Shiny load test to see if this application is already fast enough. If it's not fast enough, we'll use Profvis to see what is wrong with it, what's making it slow. And then the real work begins of optimizing and figuring out how do we make that slow part not slow anymore.

The first and most common way I think that people should do it is to move the work out of Shiny. Don't do the work when a user is sitting there waiting. Do the work ahead of time. And especially don't do the same work for every user. That's extremely wasteful. I see almost nobody doing this on their first try. Almost everybody just says, I have CSV data, I've got gigs of CSV data, I'm going to load it up into my Shiny app whenever someone connects. Why is this app slow? So that's the reason.

Number two, if you can, make the code faster. There are things that you should avoid in R like mutating data frames in for loops. You should avoid text connections. You should avoid using the apply function to iterate over data frames. Sometimes you can find way, way faster ways of doing those things by vectorizing or using dplyr and things like that. Number three, sometimes you can cache. We're going to talk about that in a moment. And then if you really can't do those other three things, then you turn to async, which is a whole other area of discussion.

And once you've done all that, then you repeat. Then now you go back to Shiny load test and see if it's fast enough. And if it's fast enough, then you're done.

Shiny load test

So we'll start by talking about Shiny load test. What Shiny load test does is it generates large amounts of realistic but synthetic traffic to your application. And it monitors how long it takes that traffic to run, and then we can analyze the latency. So step one is you have to have your app running somewhere. So you can run it locally, or you can run it on RStudio Connect or on a Shiny server. Number two, you use Shiny load test in a browser to record what an average user would do, just whatever you think is representative. So click on whatever inputs you think a user would click on. Take as much time to think as you think an average user is going to take to think. And once you've done that recording, you can play back that recording with some level of concurrency. So instead of one user taking those actions, you can have as many users as you want doing those actions simultaneously.

And then finally, we'll take the results of that step three playback, and we're going to analyze them in Shiny load test. We have a built-in reporting feature, which I'll show you. Or it's just a data frame of events, so you can do your own analysis if you like.

So this is what that actually looks like. To run your Shiny app, we just call runAppLikeNormal. In this case, port 6104. And then we launch Shiny load test double colon record session in a separate R process and just point it at wherever your Shiny app is running. So that URL could be on Connect, it could be on Shiny Server Pro, or it could be local.

Now that we have this recording, recording.log, say, is where it goes. We use kind of a sub-command of Shiny load test called Shiny Cannon. And you give it this recording, and you give it a URL to pound on. And then you tell it how many simultaneous workers and how long do you want to run this test for. In this case, 20 workers for five minutes. And then it prints out a lot of log stuff as it goes to work. When it's finished, then you go back to R, you load Shiny load test, and you call load underscore runs to load the data. That gives you a data frame that you can then pass to Shiny load test report. And when that runs, it gives you something like this.

So it gives you a graphical static HTML file that gives you some indication of how long it's taking these sessions to succeed. In this particular case, what we're seeing is really, really bad performance. At n equals 20, we are in really bad shape.

So what you're looking at here is each row of this plot is a single simulated worker, a single simulated person. And the x-axis is time. The red blocks represent the time it's taking the home page to load. And then the blue blocks each represent a reactive operation. So you changed a tab, or you clicked on an input. You caused some kind of plot or something to be generated. And that's how long they had to wait. And the spaces between the blue boxes are the user's think time, where they're just looking at the screen.

So you probably can't see the x-axis, which maybe is a good thing, because that last value is 250 seconds. So we are talking about something that, in an ideal situation, would take 45 seconds, or maybe 40 seconds, when n equals 1. And when we get to n equals 20, it's taking five, six minutes to complete. So completely unacceptable. This application is way too slow to support 20 users.

So what do we do? Well, we could just throw more R processes at it. That's always an option. And if we run out of processes on a server, we could throw more servers at it. You laugh, but at some point, that is the answer. I mean, Google doesn't run on one machine. It runs on thousands of machines. So we do always have the option of horizontal scaling. But what fun would that be? So let's just make it fast.

Profiling with Profvis

So the next step, and a step that everybody skips, is to use a profiler. When your app is slow, use a profiler. When your R code is slow, in general, use a profiler. Do not guess. Your intuition stinks. I don't even know you, but it stinks. I promise you.

It's so easy to use profvis. All you have to do is go to RStudio. And at the top here, there's a Profile menu. Start profiling. Run your app. Wait patiently. Something interesting happened? Stop profiling. And that will generate a very intricate visualization of what was happening during that time that we were waiting. So that's all you have to do.

I have, in advance, prepared a profile. I'm not going to get too far into how to use profvis. Documentation is very good. But let me just say, once you learn how to read these graphs, the answers are right there in front of you. It is so easy. There's no reason to guess what the time is being spent on.

And what this visualization tells us, in this case, is that it's taking 6.3 seconds to read the CSV file alone. Just the parsing, 6.3 seconds, way too long. When we're calculating our first output, which is all underscore hour, it's taking 620 milliseconds just filtering and aggregating the data. And that's filtering and aggregating. That's going to be the same for every user who accesses that day of data. And then the plotting, just for that one plot, is taking a third of a second. So those are all pretty big numbers when you're trying to put apps into production for a lot of users.

Moving work outside of Shiny (ETL)

So as I said before, the most important optimization that you should be thinking of as a Shiny author is to move your work to happen outside of Shiny. Do work ahead of time. If performance matters, do it ahead of time. It's really tempting to load that raw data in, but you're going to pay the price performance-wise. So ahead of time, do as much filtering and summarizing as you can. And then when you save your data, instead of saving it as CSV, save it as Feather files if it is important for you to be able to read it quickly. Feather is incredibly quick to load data. It's a little bit slower, I think, than CSV if you're writing, but for reading, which is what we care about here, it's the best.

If your data source changes over time, then you're not just going to be doing this once. You're not going to be the one sitting down and just running these filtering, summarizing, and then saving to Feather on your machine. You need it to happen on a schedule. If that's you, then you can use RStudio Connect with scheduled R Markdown reports. That is a really excellent solution that has lots of benefits that I don't have time to get into now. If you don't have Connect or you're using open source Shiny server, you can also do something yourself with the Unix Utility Cron, for example. But any way you execute scheduled tasks, this is the way to do it.

And by the way, this sort of approach of preprocessing your data for downstream applications, you may know it by its acronym ETL, Extract, Transform, Load. And lots of organizations have entire departments devoted to this function. So if that's you, then maybe someone can do this for you.

What I've done is I've modified Cranwhales to use this approach. And the repo for Cranwhales is rstudio slash cranwhales. And I've made a branch called ETL that does some of the processing ahead of time.

So if this is our original performance with the original version, this is what it looks like after just that one change. Our overall time to completion has gone from a max of 250 and an average somewhere definitely north of 200 down to, I don't know, 60 seconds, something like that. So a really big difference. The size of each individual block has not only gotten smaller, but also gotten more consistent. So we're seeing more predictable, more understandable user wait times. And overall, we're looking a tremendous amount better.

So I'm going to pause for a second and bring up. Those are just screenshots, but I'm going to bring up. This is the actual Shiny load test web page that was generated. So this is that screen that we've been looking at. This is the visualization we've been looking at. And we can flip back and forth between these sync and ETL versions of the app.

And this view shows all of the sessions that were run, sort of arranged tip to tip. But this actually all happened over five minutes. So this view shows those sessions running where the x-axis is actual elapsed wall time. So you can see here that these 20 simulated users did not finish very many sessions. Maybe each one finished three sessions or something like that. If we switch to the ETL branch, a lot more sessions are getting complete. So a lot more work was able to be done because it was so much faster.

There's also this event waterfall view. On the left hand side, we're basically looking at the recording script that we generated when we kind of simulated being a user. That first line, you probably can't read it, says get home page. And then there are many lines, maybe a couple dozen lines, of retrieving JavaScript and CSS, which should be really fast and it is.

Everything from that kind of hump where they all go to the right down, those are all reactive operations. So that's the actual work of the application being done. And again, the x-axis here is time. So in an ideal situation, if our app is super fast, these lines would drop straight down. They'd drop straight to the ground. Instead, what we're seeing is as soon as they kind of finish the loading of JavaScript and CSS, these lines kick way out to the right, meaning that the user is waiting a long time to proceed from one step to the next.

If we switch to the ETL branch, these lines are much straighter. So you can see not only are they straighter, but they're sort of smoother and more consistent. So each of the individual sessions are behaving pretty much the same way.

I won't get too far into the other options available here, but there are a bunch of other things you can look at. This one shows the amount of latency for each home page request. And the top facet here is the original version, and then down here is the ETL version. So you can see a huge difference. And then the same thing for WebSocket traffic, which really means reactive computation. So reactive computations took a really long time here. And then for the new improved version, way better.

So let's just summarize how we did. So ETL, clearly, much faster. What we learned from that report was that the median session duration has dropped from 210 seconds to about 50 seconds, so a huge difference. And the maximum wait time was unspeakable, now less than 10 seconds.

So 10 seconds is good, but it's not amazing, right? So can we do better? We can start our second round of questions and just for time reasons, I'm not going to show you the profits this time. But what it basically shows us is that almost all the time you can see is in ggplot plotting at this point, several hundred milliseconds per plot.

Plot caching

So in order to make this faster, we now have to turn our attention to optimizing plots. And that is a particularly difficult thing to do for various reasons. What we've done with Shiny 1.2 is introduce this feature called Plot Caching. And if you're not familiar with the concept of caching, what it means is if you have an operation that is going to be performed multiple times with exactly the same results, maybe don't execute it every time. Execute it the first time, save the result, and in future times, you recognize that, oh, somebody's looking for the same result, and you serve up your already cached result. So we're going to do exactly this with plots. It requires Shiny 1.2, so if you want to play with this, you'll need to install from CRAN.

Now, plot caching is not for every Shiny application, so there are some criteria you need to meet before you can decide that this is going to work for you. Number one, you have to have slow plots. If your plots are super fast, then caching them is not going to make a huge difference. But we do have slow plots. There are several hundred milliseconds each. And those plots have to be a significant fraction of the overall slowness of your app.

If you're taking seven seconds to load CSV data, then speeding up the several hundred milliseconds of your plot, you can do it, but it's not going to have a dramatic impact. But for us, we are highly optimized everywhere else. We're just looking at plots remaining.

Third, and probably most important, because caching only speeds things up if somebody asks for a plot a second and third and fourth time, you have to have the kind of app where that's going to be likely to happen. If everybody's looking at individualized or random data or data that updates every second, then caching may not help you. But for us, anyone who's looking at a given day is looking at the same data. So check there as well. So we have an excellent candidate for plot caching.

Now, we've made this as simple as we possibly can. And I'm really proud of the work that was done here by Winston Chang. This is what a regular plot looks like. And this is what it looks like using the new caching feature. So number one, we change render plot to render cached plot. And number two, it's a little bit trickier to explain, but we have to tell Shiny a little bit about what are the variables that matter when it comes to forming this plot. In this particular case, we're plotting diamonds. We're going to make a scatter plot. And we are going to let the user decide what variable to set the color.

So one user may select Clarity. Another one may select Cut. And we need to make sure that we don't mismatch those two as being equivalent. So if a user asks for a plot by Clarity, we don't want to serve them a previously saved version that was based on Cut. So what CacheKey Expert does is it gives you, the Shiny author, the opportunity to tell Shiny, when you cache, these are the variables that matter. Now, we are actually doing some things under the hood. Like, if you are on a mobile device, we'll cache a different version. If you're on a retina screen versus not, we'll cache a different version. We'll actually change it based on the width of your browser. But the part that you need to care about is CacheKey Expert. But this is all you need to do.

So I did this seven times for my Shiny app. This is the ETL version that we have looked at already. And once we have made those caching changes and run the same test again, holy shit is right, there's nothing left. There's nothing left that you can see.

The little rectangles are still there. And I can see them on my screen. But I guess they're not showing up very well there. But the ending point is somewhere around maybe 20 seconds. So this is running significantly faster than the original one was with no load at all, which is not really that surprising. At this point, 20 concurrent users, what are we even talking about? So we are way, way beyond that.

So again, our original ridiculous version. This is ETL. And with caching, it's hardly anything to see. And I think it's most clear when you look at the latency here. That bottom one, I mean, there essentially is no latency.

Just for fun, I don't have the results here. Oh, actually, maybe I do. I ran it this morning with 100 concurrent users just to see what would happen. If we could do all 100 concurrent users that we want to support in one process.

And it is now visible. The latency is visible now that we've quintupled the traffic. But it is maybe borderline acceptable. I'd run maybe two processes with this branch.

Wrapping up

So that kind of summarizes these tools, Shiny Load Test, Profvis, and Plot Caching. I will make the URL for these slides available. So if you want to see any of these other URLs, you can see it. So when I do Q&A, I'll be showing that.

So with these tools, our goal is to make it much easier to run Shiny apps in production. But these challenges remain, cultural and organizational challenges. That'd probably be a subject for a pretty good talk. But it will not be this talk.

And finally, I want to leave you with this thought that deploying production apps, I hope you've gotten the sense that it's a real skill. And it takes experience to do this well. So be humble when you're deploying production apps. And especially the people in IT that are haranguing you and raising a skeptical eyebrow, they're not the enemy. They might be the enemy. But oftentimes, they're not the enemy. They have your best interests in mind. And they are trying to protect you from pitfalls that many, many companies and organizations and programmers before you have stumbled into. So if you have people in IT and engineering that you can treat as partners, you're a huge step ahead. And definitely lean on those resources if they're available to you.

And especially the people in IT that are haranguing you and raising a skeptical eyebrow, they're not the enemy. They have your best interests in mind. And they are trying to protect you from pitfalls that many, many companies and organizations and programmers before you have stumbled into.

I do want to give credit to the wonderful members of the Shiny team, past and present, and other members of the RStudio team who have helped with all the software that I demonstrated today. The Shiny team, I couldn't ask for a better team. We're really excited about the work that we've done in the last couple of years. And we're really, really excited about the stuff that we're doing this year. So thank you.

So you can download this slide deck at this URL. Oh, actually, if this URL doesn't work, just take off that Shiny in production. I don't remember if I set the slug right. But speakerdeck.com slash jchang5. And I also wanted to call out, Kelly O'Brien has started working on a Shiny in production book. Sean Loth and Kelly O'Brien did an awesome workshop the last couple days on Shiny in production. And they've taken some of their materials and they're condensing it into a book. So that's where the work in progress is located.

And with that, I will take seven minutes of questions. So if you've been to an RStudio conf before, you'll know we'll have these throwable microphones. And you will also know that I am very poor at throwing.

Q&A

Talking about the plot caching, is there a way to do that in a distributed environment to pass it between multiple servers?

So the question is, if you have multiple servers running the same Shiny app, can they share the same cache? Yes. I haven't gone into any detail about where the cached plots are being stored. But it is an extensible mechanism that gives you a lot of control. So you have several ways to go. The simplest way for sort of casual use is you can just cache to the file system. And you could also cache to a shared file system. But what I would recommend instead is there are examples in the documentation for using Redis, which is sort of what all the big boys use to do this kind of caching. Big boys and girls used to do this kind of caching. And you can pretty easily plug that in as the back end for plot caching.

Before I take the next question, I forgot to point out that this slide deck actually has an appendix of best practices. By no means complete. These are just like the random thoughts from my head about some of the different things that you might want to do in R to have your apps be good production apps.

You mentioned preparing data in a scheduled R Markdown document. I'm wondering where you put the files, especially on a clustered RConnect environment?

Yes, great question. So currently with Connect, what you would do is you would set up a folder somewhere on the... I can't, I don't know where you are. With Connect today, what you would do is you would create a folder on disk that both applications have permissions to access. And then you would basically just configure that path into your application.

In the future, what we would really like is to have a more sort of first class representation of shared data in Connect. So the name we've been using for that feature is called data pools. So unless that changes, if you ever see the data pools feature arrive for Connect, that's what that's talking about. And that would give us, once we know more about what your intent is by having this first class feature of a data pool, then we can do things like permission the data pool for you instead of you having to manage those things yourself. But yes, for today, you could go to a database. You could go to a shared file system location. But really, you're sort of on your own. But there are lots of ways to do it.

In academia, a lot of credit is based on publishing in journals. For publishing Shiny apps, do you have a favorite journal? Or would you go a different route for giving credit to the people making the Shiny apps?

You should ask anyone else at our studio besides me. I barely graduated with a bachelor's. So when I hear the word academia, I freeze up a little bit. I have no idea. I really don't know anything about journals. People ask me, like, I want to cite Shiny. How should I cite Shiny? And I don't even understand what that question means. I say cite it however you want. And they don't seem to like that answer.

Is there anything built or being built to help with integration testing, automated integration testing for Shiny?

You make me regret that I cut that whole section from my... Shiny test is the answer to that. A Shiny test hit CRAN sometime late last year. You should be unit testing your R code that you are calling from your Shiny app. But unit testing doesn't help you with testing the app as a whole. So we call that integration testing. Shiny test is specifically designed to help you do that. It is designed not only to do integration testing, but to make it so easy that you might actually do it.

So what it does is, similar to Shiny load test, you record a session of yourself using the app. And you have a special UI on the side of your app that we inject that lets you say, take a snapshot here, take a snapshot here, as you do various things. And then in the future, as your code changes, or as your software configuration changes, as you upgrade packages from the tidyverse, you can run that test again. And then it will automatically give you a visual view of the differences.

So it will give you both sort of a textual representation of the data that's different. And also, what it'll also do is give you the ability to examine, in several different ways, the visual snapshot of the before and after your changes. So yeah, ShinyTest is where you want to go for that.

It's 10.30. One more question?

I was just wondering if you all are thinking about providing caching for other kinds of rendered outputs, leaflet data tables, things like that.

It is as if I seeded these questions. Yes, caching, it's almost backwards that we started with plot caching. Plot caching is, by far, the hardest thing to cache in Shiny. So we decided to start there, because that's the thing that we don't want users to try to do themselves. No offense, but I don't want you to have to go through that effort to do it correctly. Other things are a little bit easier. You can kind of build the caching yourself if you're sophisticated with Shiny.

But we were so happy with the way render cache plot turned out that what we are investigating is there's going to be cache plot, and then there's going to be everything else cache. So there won't be a render cached text or a render cached print. Everything else will get a generic. You'll just pipe your render text or whatever into a cache directive,