Colin Fay | Production-grade Shiny Apps with golem | RStudio (2020)
Shiny is an amazing tool when it comes to creating web applications with R. Almost anybody can get a small Shiny App in a matter of minutes, provided they have a basic knowledge of R. As of today, we can safely tell that it has become the de-facto tool for web application in the R world. Building a proof-of-concept application is easy, but things change when the application becomes larger and more complex, and especially when it comes to sending that app to production—until recently there hasn't been any real framework for building and deploying production-grade Shiny Apps. This is where 'golem' comes into play: offering Shiny developers an opinionated framework for creating production-ready Shiny Applications. With 'golem', Shiny developers now have a toolkit for making a stable, easy-to-maintain, and robust for production web application with R. 'golem' has been developed to abstract away the most common engineering tasks (for example, module creation, addition of external CSS or JavaScript file, ...), so you can focus on what matters: building the application. And once your application is ready to be deployed, 'golem' guides you through testing, and brings you tools for deploying to common platforms. In this talk, Colin and Vincent will present the 'golem' package, first talking about the "why 'golem'?", then presenting the general philosophy behind this framework, and help you get started building your first Shiny App with 'golem'
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
I am pleased to introduce Colin Fay, over here, and Vincent Guider, who work at ThinkR, which is a R consulting services company based out of France, and they're going to talk about engineering production-grade Shiny apps using golem.
So, hi everyone, so my name is Vincent, this is Colin. We are both R hackers, and we love to do a lot of things with R. We work at ThinkR, ThinkR is a French company which works on everything R-related. We do a lot of training, from beginner to expert, we do a lot of Shiny applications, and we love to deploy these Shiny applications in the real world, for the real customer needs. So we put R into production using Shiny.
And we are here today, we are glad to be here to present you a package called golem. So before going deeper into golem, I would like to talk about what is a prod-ready software. A prod-ready software should have metadata to know who to complain with, should be divided into chunks, into small functions. A prod-ready software needs to be tested, and you have to provide a list of requirements to help to deploy this application. And you have to document what you did in the software.
Why do we need this? It is to be sure that this software will be good enough to go into production, and that this software will be good enough to be easily maintainable. So because if you have to create a Shiny application, it could be really easy for everyone to create a quick Shiny app, but if you want to create a real big Shiny, it could be a mess. So you have to follow some rules, some steps, to be sure that the process will be okay to go into production at the end.
And you want to follow rules to work with many people on the same project. So because if you want to put your little Shiny app into production, you will have a lot of issues concerning velocity, security, and maintainability.
Introducing golem
So here comes golem. Golem is an R package. It's a framework which will help you to create a prod-ready Shiny application. This package is for beginner, but also for experts.
So what do you like the most? Okay, so let's start with this Q&A. What do you like the most? Work on building your data manipulation, visualization, and models, or spend 15 minutes checking that you've correctly copied and pasted Shiny module skeletons.
So that's basically the idea behind golem, to provide a framework for doing all this automation, and if you are doing the same thing again and again and again, the best way to do it, and the best way to be sure not to mess up and not to have problems, is to use a framework. We can see it in many languages, even in R. Using a framework allows you to do things more secure.
So why golem? You all know the idea that if you have to copy and paste a piece of code more than twice, you should write a function, and then if you have to copy and paste an infrastructure more than twice, you should write a framework. So that's basically the idea behind golem, is that we've been building a lot of Shiny applications, but we've sent it to production, and in the beginning, it's just like you are building this application, then you are building a second one, and then a third one, and another one, but most of the things you do during the process can be automated, because it's very repetitive, it's some kind of things that you do again and again, so instead of having to copy and paste from an old Shiny application into a new application, or instead of forgetting the steps, we have built golem, which is a framework for a Shiny application that needs to be sent into production.
So why are we using golem? Because we think that automation is gold, that is to say that if you have to do something again and again, it's better to automate it, so everything that can be automated should be automated. For example, the copy and pasting of a Shiny module skeleton is an odd thing, because if you have to copy and paste from somewhere else, you can either have it, I don't know, on the web page, you can have it in another Shiny application, you can have it into your notepad, and then just like copying and pasting it into your application, which is really subject to errors, because you can, you have to copy and paste something, and maybe you are missing a line, maybe something like that. So the idea is really these kind of things that can be automated should be automated.
So the idea is really these kind of things that can be automated should be automated.
We also think that sharing is caring, so we made it open source because we try to give back to the community, so it's very important for us to share these good practices that we've been building for years, and it was important for us to open source this project, and also the idea behind this is to create some kind of collective intelligence, that is to say that if we are using it worldwide, that is to say that if a lot of people use the product, we can all build collective intelligence around it, that is to say, people are using it, they're giving feedback, they want to improve, and with Golem, it's these ideas to build collective intelligence around how to send Shiny to production.
Shiny app as a package
So if we go back to what Vincent said about what is production-ready software, it has metadata, it is divided in function, it is tested, it lists the requirements, and it's documented. And if you think about it, this is what a R package is.
So metadata are the description file, the functions are in the R folder, the tests are in the test folder, the requirements are in the namespace, and the documentation goes into the man and in the vignettes.
So the big idea behind Golem, so the first and the central idea is that a Shiny app should be a package. It's also very helpful to have a package structure if you want to talk with IT, because if you are sending just a file to your IT and say, hey, send it to production, and they are going to say, yeah, but which package do I need, do you have system requirements, where do you want to deploy it, how do you need things like that? But if you are sending a package, a package can be installed and install the dependencies and everything is documented, so it helps deploying the application, because everything is there.
Also, with the documentation, with man and the vignettes, it helps to work on the project on the long run, because if someone comes and wants to take over the maintenance of your Shiny application, you have everything documented with the package structure, and that also means that if you want to deploy it to your company, so if you want many users to use your application, you can install it as an R package and then they can do library your app and launch one function, but will not launch the application.
And the cool thing is that everything you know about how to build a package, it will also work with Golem, so you don't have to learn something more if you know package development, everything you know about it will work, so we try to build on top of everything that exists and all the cool things like use this, like dev tools, like test that, so everything you know, everything that exists and works perfectly, we try to build on top of that.
Golem's structure
So how is a Golem composed? So if you start a new project with Golem, you're going to find this structure, so this is the structure for the upcoming Golem version, but you have standard things, so the description and the namespace and the man and the R folder, which are the standard thing that you have in a package.
There is an app UI and app server, which are the Shiny UI and the Shiny server functions, so everything you have to put in your server goes there and everything that you have to put in your UI goes into UI, so UI server. The runapp.r file is used to launch your app and something that is coming in the next version of Golem is the use of the config package, so you have something called appconfig.r that handles the other files of the Golem config Yammer.
So the inst and the man are standard package folders and the new thing is this dev folder, so the idea is that you have here four files, so the first is start, so it lists a series of steps you can follow to build your application, so the idea is that when you start your application, you have this start script that you can follow, so it lists a series of steps to follow and then you have dev.r file, which are a series of functions that you can call during the process of developing and you also have a deploy.r file that contains functions to create the deployment structures and the rundev is a script that you can launch to relaunch your application while you are developing.
So the idea behind this dev folder is to be sure that you don't forget anything during the process, but it's also to be sure that you are documenting what you are doing while developing, so you will write everything that you are doing and you can keep track of what you have been doing and if you are working, like, with someone else, you can follow what they have done and you can see what they have done. So this is the idea behind the dev folder.
Automation in golem
I know that you said that, I told you that there would be automation, so I've been talking a lot about automation, automation, automation, so here comes the part about automation. We've tried to make it easy to do these repetitive things like creating a module, creating a CSS file, creating a JavaScript handler, using a favicon, so during the building process, if you want to add a new module, you have a function that says add module and what this function does is adding a module in the correct folder.
Same goes for CSS, same goes for the JavaScript handler and the favicon, so if you build a large scale Shiny application, you might be using a lot of modules, so if you just add to copy and paste all the modules and T files, remove lines and things like that, it can get a little bit complicated, but with this function, you just have to give your module a name and it places the module in the correct folder, so all these repetitive things are easier and as they are automated, you are sure that you're not missing anything.
Same goes for deployment, so if you want to deploy to any RStudio product, we have functions to create a file that can be deployed to connect to Shinyapps.io or to a Shiny server and if you want to create a Docker file out of your package, you also have free functions that can be used to create this Docker file and I hope that in the future we can support more deployment platforms.
So for example, we have this template file for creating a module, so if I do golem add module and name, I name it my module, this will be create a file into R mod, so it will create a file in R which is called mod underscore my module dot R, so you don't have to provide the dot R, you can, but if ever you forget to, if ever you add it, it will be removed at the end of the file and if there is no dot R at the end, it will add it, so this kind of thing also helps you, you know, not having trouble copying and pasting things.
So what this function does is creating this skeleton, so I removed the top part, but there are a series of Roxygen skeletons for documenting your module and the same thing that I said before is that you can have a hard time knowing where to put your modules and you have to think about IDs, about things like that and every module skeleton has this little part at the bottom that you can copy in your UI and you can copy in your server, so we are sure that the ID match and things like that, so you have this skeleton and you don't have to worry about whether or not you have the correct skeleton and whether or not you have the correct ID, they are just there because they are scripted and automated, so this is the ID behind add modules.
What's next for golem
What's next with Golem, so all the work in progress and all the IDs are listed on our GitHub repo, so if you want to contribute, you are very welcome to do so, so as I said before we are trying to build like a global intelligence around building Shiny applications for production, so if ever you have any ID or you want to contribute, all the work in progress and IDs are on our account.
Also we, what's next is a new release of Golem, so I hope that it will be released in a few weeks, so we've been working a lot since the summer on a new version of Golem, so the Docker file creation is way more smart and is way quicker than before, we've been integrating the config.yaml, so the config package and future version of Golem, so this version will rely on using config.yaml, we've been working on the way it's documented, on the way module are created, and we've had more JavaScript internal functions, so we can hope that it will be released in a few weeks.
What's next, what you can do is spread the word and share the stickers, so if you come to us, we have a series of, yeah, a handful of Golem stickers that we would be happy to share, so if you want to tweet, blog post, and talk to your friends and family about Golem, please do.
So if you find a bug, if you want to give feedback, if you think, if you find something that you think is weird, you can open issues, if you have ID, feature requests, anything about Golem, I'd be happy to talk about it with you, and you can also open issues on the repo.
So what's next is now it's way easier to send Shiny to production. We've also opened a new website called golemverse.org, so we are using this website to communicate about Golem, but also all the other package that gravitates around Golem, so we have another package called Shiny Zoom, we have things like that that are in the Golemverse, so everything around Golem will be listed here.
And finally, we've been working on this book called Engineering Production Grade Shiny Apps, which is about how to build Shiny application for production, so it's not a book for beginners, it's more for people who have already a little bit of knowledge about Shiny and want to go a step further, so if you've heard about Mastering Shiny by Adly, it's just like a book that comes after that, I'd say.
Q&A
All right, we have time for a few questions, which I will read off of this slide. So our top question is, is Golem the right fit for all Shiny apps? Is there a rule for when a project is big enough to need the framework?
Okay, so you can build any Shiny app with Golem, so even with a small one, I'd say that I think it's easier to start with Golem for a small Shiny app than starting a Shiny app then move to Golem, so if you think that at some point your Shiny application might grow, I think it's easier to start with Golem, and if you already have the templates, it's easier to continue and to build a larger app than to have the app and then turn it into a Golem application.
All right, thanks. Our next question is, this is one I'm very interested in hearing the answer to, after developing and working with Golem in production, what have you learned about Shiny that you think needs to be improved to make it even better?
Oh, a lot of things. That's a good one. I'm very looking forward for working with Shiny on asynchronous things in Shiny, so the Promises package. I know the current implementation is blocking for the user, but I know Joe has been providing some work around to use Promises and I think this is something that I'm very looking forward to, so better integration of asynchronous programming in Shiny.
All right. With our Shiny server, we have problems with different Shiny apps requiring different package versions. How can Golem help us solve this?
In every Golem package, you will have a description file containing every description you need. If you really use Shiny server, it will help you to install this package, but I think you have to provision to have different libpaths and so on to be sure to be able to run each Shiny application in different context, and Golem will definitely help you to do so.
I think we have time for one more. Is converting a Shiny app into a package necessary or crucial to be production-ready? What is the main downside of not having it as a package?
The main downside is that if you provide, so if you think of it as a package, it contains everything needed, so the dependencies and everything, so you can bundle it into a tarball and provide it to the IT and it works. You don't have to provide them the application and the requirements and everything, so as it's in a package, you can rely on the package structure to be tested, so we can use the tests, we can use the dependencies, features and everything that makes our package strength.
Thank you very much, Colin and Vincent. Thanks for the talk.
