Shiny and R to Build Dynamic Dashboards
In a static report, you answer known questions. With a dynamic report, you give the reader the tools to answer their own questions. Unleash the full flexibility of analytic app development with Shiny. In this talk, Winston Chang will discuss how to use R and Shiny to quickly create data dashboards. You’ll also get a glimpse of some new features in Shiny for presenting and interacting with data. He will also demonstrate how you can easily deploy apps to the web via RStudio’s hosting service shinyapps.io. Related blog post: https://blog.rstudio.com/2017/05/18/shinydashboard-0-6-0/
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Alright, yeah, I'm going to be talking about Dynamic Dashboards with Shiny. Welcome everybody, thanks for coming.
These slides are available at this URL, this one at github.com slash rstudio slash webinars, and there's a bunch of directories in there. This is under today's, for today's date, which is 2015-04.
So here's an overview of what I'm going to cover. So first, you know, what is a Data Dashboard? That's an important piece of information to cover. I'm going to talk a little bit about Shiny, but I'm not going to go into depth about Shiny. I'm going to assume a little bit of knowledge of it. And then I'm going to talk about using the Shiny Dashboard package, which is a new package, it's under development, we're going to release it pretty soon, it's still polishing off some edges, some rough edges on it. And finally I'm going to talk about how to deploy a Dashboard that you've created using Shiny.
What is a data dashboard?
So Dashboards have become pretty common on the web these days, and here's one example of it. This actually is a Dashboard for our shinyapps.io service. And you can see it presents, you know, presents the information efficiently so that you can quickly get a sense of what's happening. There's, you know, there's, in this application usage you can see there's high application usage and then nothing, and then it jumps back up again. There's dates when it was deployed and updated and created. And so this, you know, the purpose is to give you all this information at a glance. It's not necessarily to go deep in depth into some statistical analysis of this information.
So as I said, you know, Dashboards, you usually want to convey information efficiently using a Data Dashboard and provide an intuitive user interface for users, because oftentimes the users of Dashboards are not the, you know, they're not statisticians and data scientists and the creators of them. They're people, they're business people who don't necessarily have that level of expertise. And going along with that is that we usually want them to look attractive so that they look professional.
And finally, there's also like a spectrum of Dashboards from just presentation-focused Dashboards to exploration-focused ones. So a presentation-focused Dashboard would just be something that just displays information that doesn't let you interact with it. An exploration-focused one is something that lets you explore the data, slice it, dice it, and get some, find some valuable insights in it.
How dashboards work
Well, there's some basic components that they need to do. So one is that your Dashboard, whatever software is running your Dashboard, it needs to fetch the data somehow. That seems pretty obvious, but you need to get the data, and that can sometimes be hard work. Then you process and summarize the data, and we all know that R is really good for this. And concisely present the process data, often this involves visualization, there's summary tables, and optionally we can provide exploration tools if that's the type of Dashboard that we're working on.
Okay, so fetching the data, when you're, in general when you're creating a Dashboard and you're grabbing the data, it has to be done quickly. If you, you know, if it takes a minute for you to grab your data from your data source, that's generally not an acceptable wait for most users.
So if you have a small data set, relatively small data set, that's usually not a problem. There's many different ways to do this. You can grab it using, from a database, using the DBI package, there's many other R packages for database, interfacing with databases as well. You could schedule some data dumps or data summaries, like nightly, if that's, you know, if that's the frequency that is appropriate for your data, writing it to a file, and then your Dashboard can read in a CSV or summarized, or a summarized data set. Or maybe you have like a web API that you can just visit a web URL and it'll return data in JSON or XML. So there's great R packages for grabbing those as well, JSON Lite and XML.
And Hadley Wickham, our chief scientist at RStudio, is working on a new XML package for working with XML data, which should make it easier and faster. So again, you know, if you've got a really large data set and you need to, and it takes too long to fetch that data, you can summarize it on a scheduled basis and then grab that summarized data in your Dashboard.
So for presenting data using Shiny, there's all sorts of, there's different types of outputs. You can output numbers and text, you can output tables, you can output graphs, and these are some of the function names that are available in Shiny for doing these things.
Finally, well there's components for exploration. So in Shiny, you know, so far I've talked about how data comes out to you, and if you want people to be able to explore their data, they need to have, you need to have inputs. So the user needs to be able to send information to the Dashboard, you know, then their client, which is a web browser, to the server, which is running R. There's a text input box, there's a slider, that's, well that's not totally a standard thing, but we have that. Select inputs, checkboxes, radio buttons, buttons, data inputs, all sorts of things like that that Shiny provides for exploring your data, for getting user input to explore the data.
And we're also working on some new interactive components. These are some R packages, there's one called, we're working on called DT, which provides interactive tables that you can sort and filter interactively, so they both will display data from the server, and they can also accept input from the client. There's Leaflet, which is used for making interactive maps, and we're going to see a bit of that later. And you can, we're also adding features to allow interactions with base R graphics and ggplot2, and we'll see that as well.
Shiny basics
Okay, so, if you're new to Shiny, or if you only know just a little bit about it, I'm going to cover some basics of Shiny right now. This is going to be pretty brief, but hopefully it's sufficient to give you an idea of what it's all about. So what is Shiny? So Shiny is an R package that we're developing at RStudio. It's a platform for creating web applications in R, and it uses a reactive programming model, which makes it really easy for application developers to use their existing R knowledge to create interactive applications, you don't have to know about all sorts of complex underlying stuff like callbacks, and all of that is sort of abstracted away, so you can just, you can write this like you're writing regular R code, or almost like that. And it's free software. It's licensed under the GPL version 3.
Okay, now here's some code for a Shiny application. Every Shiny application has, there's two sides of it. There's the user interface, which is this HTML code for generating a webpage, and there's the server code, which is R code for running your analyses and generating outputs.
Okay so I'm running this application. This is really basic. It doesn't do anything particularly sophisticated. There's a slider that determines how many random points are in this scatter plot here, so I can bump it up to, that's 468, 27, and when I move this slider, this plot updates.
Well, there's, as you saw, the user interface, there's a slider input, which takes in some value, and there's a plot output, which displays the plot on the webpage. And on the server side, there's this render plot, which, this is, the render plot runs the code inside of it, it runs the plot function that's in base R, and it captures that output and sends it to, back to the client.
So N, slider input is N, and when the user updates that value, it causes the input dollar N and the server side code to be updated, and when these values change, Shiny knows that this render plot needs to be re-executed. So it does that, saves the output to the disk plot, and that gets sent back to the client.
So within the server side code, you're almost writing regular R code. And the UI, there's, Shiny has a bunch of constructs for generating these webpages. These three functions, basic page, slider input, and plot output, those generate HTML, and that HTML gets sent to the client.
And then finally, we have the UI and the server, and we put them together in a Shiny app object. Now if you're familiar with Shiny and you're just using a UI.R file and a server.R file, this is a way of creating apps with just a single file, which is, there's some advantages to it, it's, you can just cut and paste the code and run it in the terminal like I did, or in the R console, and it just works. You don't have to go and create a directory and create all these files to make, to run your application.
Shiny Dashboard
Alright, so what is Shiny dashboard? So Shiny was an R package. Shiny dashboard is also an R package, and it's for creating dashboard style layouts with Shiny.
And what Shiny dashboard lets you do is create Shiny applications that look more attractive, more dashboard-like. Now, I should note that, you know, the base Shiny package can create output that is nicer than what I just showed you, that example app, but I just showed you the very bare bones example app, but if you want to create something, a really nice looking dashboard, it can take more work, so Shiny dashboard is designed to make that easier.
Now to really understand this, it helps to know a little bit about the underlying components under Shiny. So Shiny uses Bootstrap for layout, that's a web framework for laying out web pages, and Shiny dashboard uses Admin LTE, which is a theme that's built on top of Bootstrap.
All right, now, if you want to install it, you can just, well, first install DevTools, if you don't already have it installed on your system, and then you can run install GitHub RStudio slash Shiny dashboard. This is, you know, once it's released on CRAN, then you can install it from CRAN using install.packages, but for now, this is how to install the development version.
And there's documentation at rstudio.github.io slash Shiny dashboard. This is, I think it's fairly comprehensive, although I may be biased since I wrote it, I wrote all the documentation, so most of what Shiny dashboard can do.
Okay, so here is another Shiny application. This is one using Shiny dashboard. What we have here is UI is, we create the UI using, instead of basic page, which is what we used before, we have dashboard page, and dashboard page has three components. There's a header, a sidebar, and a body. So if you run this code here, you'll end up with an app that looks like that. So there's a header, a sidebar, and a body. So all these dashboards created with Shiny dashboard need to have those three components.
Okay, and here's a more sophisticated app. There's most of the content in a dashboard is going to be in the body. So this one has dashboard body, fluid row, I'll get to explain that in a little bit, and then a box. A box is the main building block of dashboards using Shiny dashboard.
Now how do those boxes know to be the right width? Well, to understand that, we have to understand the bootstrap grid a little bit. In bootstrap, there's a layout using grids of rows and columns. Now you add rows, it's a row first, you add a row, and then each row has 12 columns, and then HTML elements, like these boxes, can occupy any number of the 12 columns. So the boxes by default take up six spaces, so you can put two of them side by side.
And so, I just want to mention, don't forget your columns. If you have a row and you don't put a column in it, then you'll end up with text that's like, or HTML components that are butting up against the edge there, and that's usually not what you want. With the column, there's a little bit of margin there, so it doesn't touch the edge there.
So we have the body, and in the body we have to put a row, and in the row we put a box with width eight here, and then a box with width four. And when we do that, we end up with this first box is twice the width of the other one.
Okay, and there's other kinds of boxes that are available in Shiny Dashboard. There's info boxes, these are great for just as quickly displaying small amounts of information. They have these nice little icons there that are, you know, can help people quickly understand what they're displaying. Value box is similar, just a slightly different appearance, and these are all documented at this Shiny Dashboard documentation site.
Okay, so that's a really quick overview of putting stuff in the body. In the sidebar, there's generally two types of things we put in there. One is tab items, and the other one is inputs.
So this is a lot of code, so just bear with me. The important parts, well, let me just show you what it looks like. So here we have, again, this dashboard page, there's three parts. There's the header, the sidebar, and the body. In the sidebar, we have a sidebar menu with these menu items, and each one of these can activate a tab in the main body.
So let's see, actually, you know what? I can run this code and just show you what this does. Okay, so this is the first tab, and the second tab. When I click on the first tab, it shows the first tab content, the second tab, it shows the second tab content, and those are linked up by this tab name parameter here.
So menu item tab name is first, and that links up with this tab item with tab name is tab name first, and it shows first tab content there, and same for the second. And there's also these, again, these little icons here. Those are from Font Awesome, which is included with Shiny, and if you look at the documentation of this appearance page, it explains how to use the icons, and that works in all Shiny applications, not just Shiny Dashboard.
Okay, and so now the final part of creating Dashboard is the header. The header can include a title, or message, or notification, or task menus. So on the right, this shows what happens when you click on them. There's a drop-down menu, and you can display information here for users to see. Now I'm not going to, in the interest of time, I'm not going to go into depth with this because this is a less commonly used feature, but it's good to know that it's there.
Leaflet maps in Shiny
So in R, mapping in R, R has a lot of good tools for working with map data, and the map output is decent, but it can be hard to interact with, because usually it just generates a static image, and then you can't click on it, or zoom into it, or anything like that. But maps in the browser, there's a lot of really cool tools for interacting with maps. One of them is called leaflet.js. It's a JavaScript library for interactive maps, and this is the website. Now this map here is just a screenshot I took from that website, and it's interactive. You can pan and zoom, and it works similarly to Google Maps.
Now we're packaging that up in something in our package called leaflet, which provides a nice interface to leaflet.js. So that's available at RStudio, or the documentation is at rstudio.github.io slash leaflet.
So we can load the leaflet library, and then to create a leaflet map, you say leaflet, and then this is a special piping operator, so leaflet, pipe it to add tiles, that adds all those images on there. Set the view to latitude and longitude, in particular latitude and longitude in the zoom level, and then add a popup. And this is, here's the department of statistics at ISU.
And this map will be interactive. So we can actually run this code in RStudio, when we run it there, it pops up right there in the RStudio viewer. So that's one way you can use it, and it's interactive.
You can also use leaflet in a Shiny app, and this is getting back to dashboards. So in the UI side code, you'd say leaflet output, and give it a name, map, and then on the server side, you'd say output dollar map is render leaflet, and then you just put that leaflet code in there. And then that map will just show up in your Shiny application.
Demo dashboard
Okay, so I want to show you how, an example of putting this all together. So I'm going to show you this demo dashboard. It's using data from a company called Pure Health, one of my friends works for them, and they have a cloud platform for supplying data collection and managing health data that comes from mobile apps and wireless devices. And they provide an API for people to be able to grab that data.
So that's what I've been using for this demonstration application here. So this is putting all those parts together, the dashboarding and the Shiny dashboard and leaflet maps, and it fetches map data and creates this dashboard here.
So this is the current location of my friend Jared, who's working at Pro Health, he allowed me to track him today, and we can see where he's been. So today he's traveled seven kilometers, he's, the closest place, well, the closest venue that he's at is home, these red dots are like known locations, they're called venues, and he's four meters from home, which I guess probably means he's inside his house. And this is real, this is real time, and we can, you know, this leaflet map lets us zoom in here, and zoom out.
Now this is sort of a, more of a presentation focused dashboard, but we can also have more exploratory tools as well. So this is using that, switching tabs here, click on location history.
This is showing where he's been in the last several days, so hopefully you can see these, the text on this graph here. So this is from April 11th to April 15th. And he's traveled around the area, this is, we're in Minneapolis, so he's traveled from his home, down to Minneapolis, to down here near the airport.
And with Shiny you can have these filtering tools, so when we interact with this slider, let's say I want to, I just want to see just the data points that are within four kilometers of home. So this is where he's been during this time, that's within four kilometers of home. And you can see that he's been often at these other venues, and on this plot on the bottom, this is generated by ggplot, and this plot on the bottom, this is a distance from home, so you can see that in most of this time he's been close to home, and then he traveled far away, that was probably that trip down near the airport, went back around home, and had a shorter distance trip and stayed wherever it was for a long time.
And this orange dotted line, that's the cutoff point for the distance from home. So anything above that line is not being displayed on the map anymore.
So now we could reset it, and there's, okay, now I'm going to show you this new feature that we're working on in Shiny right now, which is to do this, interacting with a ggplot graph. So I've, this is called brushing of the plots, I've selected this region, and this also does filtering, now the graph is only, or the map is only displaying location data that's within this day, from April 11th to April 12th, and so he stayed close to home that day. We can move it over, from April 12th to April 13th, he stayed, he was at home, and he went down near the airport here, and if we want to find out more about that, we can just, let's see, we could say, okay, I just want to look at this region of time. What is this here? Turns out, during this time, he was near the Mall of America.
What is this here? Turns out, during this time, he was near the Mall of America.
So that's kind of interesting, and then this other day, he was at home, went to work all day in downtown Minneapolis, and then went back home. So with these new plot interactions that we have, it should open up some new doors for being able to take a deep dive into your data, and explore some interesting connections in it.
This is calorimeter data where measuring oxygen intake, carbon dioxide output, and activity. So this person was hooked up to this equipment, and had three exercise sessions where you can see there's a large increase of oxygen and carbon dioxide at those times, and spikes in activity as well. And we can see a correlation here between these variables, oxygen consumption, carbon dioxide output, there's almost a perfect linear correlation, something you almost never see in real world data.
Deploying dashboards
Alright, so let's say we have this dashboard, and we want to deploy it. So, so far, I've just been running it locally on my system, but usually you want to be able to share this with other people, so, and you can't just run it, you know, Shiny on your computer and allow people to connect to it. I mean, you can, but it won't restart properly if your app exits for whatever reason, and you can only run one app at a time, which is probably not ideal.
So in order to deploy it, there's, for real, there's three options. There's Shiny server open source, there's Shiny server pro, and there's shinyapps.io. Now the first two of these, you run them on your own Linux server, so you set up your own Linux server, either, you know, it could be in-house, you could, I guess you could install it in some cloud service like Amazon AWS. And then you install Shiny server, and you can put your apps on there.
The third option is shinyapps.io, this is, I think we came out of our beta testing phase very recently, about a month ago. And this is a hosting service where you just deploy your app to our servers, and we manage all the rest. We make it really easy to deploy it, and we deal with, you know, load management and monitoring and things like that.
Okay, so let me, I'll talk real quick about these different options, some of the pros and cons. The open source version of Shiny server, it's free software, it's licensed under the AGPL. You have to run it on your own Linux server, which can be a pro or a con, but it's more work. But on the other hand, you can keep it internal if you need to. There's no authentication or SSL, so it's not very secure, it's not built for security, it's more for public-facing apps, and there's only one R process per app, so it's better for relatively light traffic.
Now I should note that one, a single R process running Shiny can actually handle multiple connections from users. But if you have a lot of users sharing the same R process, it can get bogged down.
Shiny server pro, it's similar to Shiny server open source, you run it on your own Linux server, there's authentication and SSL, which is an advantage over open source version, and there's multiple R processes per app, so if you get a lot of traffic, it's good for that. And it also provides an admin and monitoring dashboard.
All right, finally, okay, shinyapps.io, we manage this, it's hosted by RStudio and Amazon AWS, it supports SSL and authentication, it allows multiple R processes per app, there's an admin and monitoring dashboard, which I actually showed you right at the beginning of this talk. One thing it doesn't do is there's not persistent storage, so if you want to collect user input, you'd have to send that off, that data to some remote data store, so if that's something that you really need, either you need to do that, or you should, Shiny server or Shiny server pro might be more appropriate in that case.
So first you'd have to install this shinyapps package, you can use dev tools again to do, call install GitHub, RStudio slash shinyapps, okay, next you have to visit the shinyapps.io website and create an account there, and then configure the shinyapps package, there's the getting started guide at this URL, it's very straightforward, there's a number of steps, but it should be pretty straightforward.
Okay, once you've done that, then you can just run shinyapps, deploy app, and then give it the directory that contains your application. So one of the really nice things about it is that the server will use the same version of R and install all the same versions of the packages that you have, so that can simplify deployment a lot. If you've used Shiny server and you've deployed apps there sometimes, and you may have run into problems where packages are, they have different versions on your development machine versus production, and there's some incompatibilities in the code that can happen that way.
So I'll demonstrate this using a simple app, okay, this is a really basic dashboard, so it'll do this stuff, it's deploying to shinyapps, this usually takes about a minute, and when it's done it should pop up a web browser displaying that app, but I'll let this run in the background while I show you another way to do it, which is deploying to shinyapps.io using the publish button, and this might require, I'm not sure, this might require one of the latest development versions of RStudio, these daily preview builds, but those are worth trying out anyway, okay, so the dashboard finished deploying and it just popped up in my web browser, and you can see this URL, winston.shinyapps.io slash simple dashboard, and that's how it works.
Resources and Q&A
Okay, so we've gotten to the end here, so here's some other useful resources for learning about this stuff, there's a lot of stuff here, but you can see the dashboard demo at this URL, winston.shinyapps.io slash activity dashboard, so let's go to that page, so if you want to see what my friend Jared is doing right now, you can go visit this URL, and again this is real-time data, so it is actually what he's doing right now, if you want to learn more about Shiny, we have our whole Shiny site at our studio.com, the Shiny dashboard documentation, I mentioned those links a couple times before, leaflet, Shiny server, shinyapps.io, and admin LTE if you want to learn more about the nuts and bolts about laying out Shiny dashboard.
And that just about covers it. Thanks Winston, so let's take a couple of questions, we've been trying to answer them along the way, but they're coming in fast and furious, then we can handle, but there's sort of a couple of themes might be worth just talking about, there are lots of questions about being able to use different kinds of visualizations or third-party packages, and I think the general answer is anything that can be used with R can be used in the dashboard, but you might want to comment a little further on that, some of the popular additional kinds of visualizations people might use in a dashboard.
Yeah, so any of the, again as you said, any of the plots that can be used in R can be used in Shiny and in dashboards, so but you know, I mentioned ggplot2 a lot, any of the base graphics, base R graphics plots can be, you'll just show up in Shiny if you use them, you can use lattice graphics, and if you need other forms of map output, there's various mapping packages that will display maps for you, so and yeah, so there's many options, anything that R can use, Shiny can use, and Shiny can also use various JavaScript libraries, so we have another package called digraphs, which is for plotting time series, now this is done in the browser, so it's a little bit more interactive, and we're working on building more of those.
Okay, great, and there was another question that came up quite frequently, any recommendations for dynamic resizable widgets, you know, your text boxes that don't, that instead of wrapping might expand as they fill them in, I know it's a bit of a detail, but do you have any recommendation there? You know, I think I'll have to look at that question after this is done, I need to look at the details of that, like exactly the specific use case.
Okay, questions, and I can take one while you keep, while you look here, so there's a question about the Shiny server and its limitation, the open-source Shiny server to one R process at a time, that doesn't mean that only one person or one app can be deployed, you can deploy as many apps and have as many people use that as you need, but it does mean that people are, the application handles users serially, so depending on how long it takes to go and do the activity that the user has instructed the Shiny app to do, will potentially delay the next person, right, because everybody is stacked up, whereas with Shiny server pro and certain of the commercial paid features of shinyapps.io, you'll find that you can deploy multiple instances of R and eliminate that degradation potential.
Let's see, so somebody's asking, is deployment on AWS through RStudio free? So, you know, if you deploy Shiny server or Shiny, well, if you deploy Shiny server on your own AWS instance, that's free, but if you want to deploy to shinyapps.io, which happens to be hosted on AWS, actually, we do have a free tier, don't we, Roger? Yes, we do, on Shiny apps, yes. Yep, on shinyapps.io, there's a free tier, there's a, we put an RStudio branding bar on it, and there's certain resource limitations on it, but it should be fine for low usage apps.
Let's see, somebody else asked, which version of ggplot supports brushing? Well, this is actually a feature, the current version, I'm using the current version of ggplot, but I'm using a special version of Shiny, it's on a development branch, and I'm going to post the code for that app, that demonstration app that I had, I'm going to post that to our GitHub webinar site later today. I just need to make sure that, you know, the API keys are cleaned up, and there's no private, too much private, too private data that's being revealed by it, so that'll happen later today.
Hey Winston, this is a question we get from a few people, both in webinars like this and other places, but for dashboards that are representing a lot of data, you know, any advice on, you know, how to handle large datasets? Yeah, I mean, if they're really large, you know, no matter what you use, be it R or something else, it might take time to crunch that data and to massage it and summarize it in a way that can be displayed effectively, so if that's the case, then you might need to schedule a job to, you know, run that data crunching every hour or daily, and then you can save that output, and then from one of these dashboarding applications, just read in that pre-processed data, so I hope that's what's meant by that question.
So if that's the case, then you might need to schedule a job to, you know, run that data crunching every hour or daily, and then you can save that output, and then from one of these dashboarding applications, just read in that pre-processed data.
Okay, we got a recent one here, is it possible to load new data in real-time and update the visualization? I think the answer is yes. Yeah, in fact, I'll just run that app again, so if you're tracking Jared right now, there's a button here which has refresh location data, and what this does is this just tells it to contact that web API, oops, sorry, this just tells it to, clicking on this refresh location data button tells it to hit that web API again, and it fetches the data and redoes the map, so you might not see much of a change in this location, actually, if we click on it right now, but yeah, he's probably not moving anywhere, but yeah, this will refresh it, you can also set up timers in Shiny using a function called invalidate later or reactive timer that will just tell it to re-execute or refetch data every, you know, every minute or five minutes or whatever specified time you want, so that's another possibility.
And so let me conclude today's webinar with just the request that if you do have any interest in our commercial products, that's how we pay for all this good stuff, we'd love for you to evaluate whether the features of our commercial products, RStudio Server Pro or Shiny Server Pro are right for your organization, the websites are here, we'd love for you to join us at our next webinar where we're actually going to spend more time on the differences between our open-source products and our commercial products.
