Nathan Stephens | Best Practices for Administering RStudio in Production | RStudio (2019)
Most organizations are unfamiliar with the R programming language. As a result they often struggle to onboard and manage R in production. In this webinar we introduce the RStudio Quickstart which makes it easy to try RStudio professional products on your desktop for free. We also outline best practices for using R and RStudio in production. Webinar materials: https://rstudio.com/resources/webinars/best-practices-for-administering-rstudio-in-production/ About Nathan: Nathan has a background in analytic solutions and consulting. He has experience building data science teams, architecting analytic infrastructure, and delivering innovative data products. He is a long time user of R
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
I've been really excited about giving this topic. I've had it on my brain for several months now, and I'm glad to get out some of this information. I just want to praise everyone that showed up for this. The title that includes Best Practices for Administering RStudio in Production is not a very sexy title, so I'm really glad that people showed up for the talk.
I do have a lot of topics to go through. I'll try to go through quickly, give you an overview of a number of different topics I think are important before jumping into the demos. So I work in solutions engineering, and at solutions engineering, our team helps you integrate RStudio products into your system. So things that involve integration are things like authentication, security, servers, cloud architectures, databases, Spark, TensorFlow, Kubernetes, all these things that surround your R environment.
These are the things that we get really excited about. We have a great team. The team members are Sean, Edgar, Cole, Andre, James, Kelly, Chris, and Alex. And we hang out in a variety of places. The best place to find us is in community, and we hope that you'll go to community and interact with us there. We have a GitHub page and our own website, and then we frequently contribute to the support and docs websites.
This webinar is really designed for the R admin. Sometimes I call the R admin a data scientist who wants to do more. They want to do things like pipelines and installation scaling and building these platforms. If that describes you, welcome. The R evangelists, people who are trying to get RStudio and R more widely adopted in their organizations, IT ops, of course. Always happy to have IT ops on board. And anyone who wants to try RStudio professional products, if you don't know anything about professional products and this is your first time here, this is a great way to get introduced to professional products.
About RStudio
So RStudio empowers individuals to be productive with data science. We believe in things like, well, we're strong believers in open source and reproducible research, and we believe in things like APIs and interoperability. We want to have our work be, you know, really usable with clear documentation. We believe in inclusivity and collective success. We really want all of the work that we do to have lasting value.
We build open source and professional software for data science. Our open source software is used in almost every organization that uses R. When organizations benefit from open source software, we think they have an obligation to reinvest in the community that provided it. Organizations can contribute to our open source software in part by purchasing our professional products. But our business does not rely on altruism. Our professional features include important things that customers want, like security, authentication, load balancing, support, and many other things. And these are the products that we sell, RStudio Server Pro, RStudio Connect, and RStudio Package Manager. And then the money that we make from these professional products funds our open source software.
What's the relationship between R and RStudio? Well, we don't own R. We don't package it. We don't distribute it. Our core team has 20 members, and zero of them are from RStudio. We like to say RStudio sits on top of R. So you standardize on R first, and then install R products second. We assume that you have chosen to invest in R.
The challenge of administering R
In fact, last year, I gave a webinar called Professional R Tooling and Integration, where I talked about legitimacy, competencies, and adoption. So legitimacy is about recognizing R as an analytic standard, and competencies are about understanding and managing R tooling, and adoption is about relying on integrated R-based solutions. So today, we're going to be talking about competencies and where those competencies come from.
How do you learn about R? This is a big problem, actually, with a lot of our customers. They don't know how to administer our products in their organization, and there's a couple of reasons why. First of all, R is relatively unknown in most organizations. There's no single place to get all the information you need. It's really hard to see the forest through the trees. There's a lot of pieces involved. A lot of things are changing very quickly. So we see a lot of our customers going through a lot of trial and error.
Some of those include organizational hurdles. For example, the IT department might be completely separate from the data science department, and maybe the communication or the support isn't there. Or there might be resource limitations, like maybe the funding for the headcount is separate from the funding for the servers or the infrastructure, and people aren't able to get the servers that they need.
So today, I want to share some best practices for managing RStudio in production. I want to share the product requirements, go over some tips, and give you a path for getting started. My goal is to give you a big picture view of what success looks like, assuming you're using our products.
Five best practices
So I want to share five best practices for administering RStudio in production.
Number one, keep your system up to date. So you're going to want modern tools for your data science platform. That means modern operating systems and modern browsers. You'll be using the cutting edge tools, you'll be doing cutting edge analysis, you'll be hiring people with very important skills, and you're going to want modern tools for all of those things. You'll need a C++11 compiler so that you can build the R packages on your system, and you'll want to think about your internet access, and that's because all of your packages are coming from the internet. If you don't have internet access in your server environment, a lot of organizations don't, then it's very important that you talk to RStudio because all of our products are designed to run in an offline mode.
Number two, support multiple versions of R. Why do you want to support multiple versions of R? Well, because you want to be able to manage upgrades. You want to be able to test code on a variety of R versions, and you want to support projects that depend on various versions of R. All of our products support multiple versions of R. We recommend that you upgrade at least yearly and that you use a modern version of R, so 3.1 or greater. And then finally, this is very important that we recommend that you build R from source. Building R from source allows you to install multiple versions of R side by side. It's not hard to do. It follows the standard config make install path. Most organizations have a process for building software from source already in place, and we have instructions for how to do that.
Number three, organize your R packages. So R packages rule the nest. Packages will drive your R version, your Linux dependencies, and even your operating system. Data scientists will want to access their most beloved packages. So say you have a data scientist who wants the most recent modeling package, and they try to install it, and it fails because it needs a new version of R. So you want to upgrade that version of R, and then you find that you're going to need some Linux binaries, but those Linux binaries aren't really supported by your operating system. So suddenly you're upgrading your whole operating system to support that package of R. That is not an unrealistic scenario. That's one reason we say keep your platform modern and up to date.
So it turns out that managing packages for a single user is quite easy, like you do on your desktop, but managing packages for an entire platform is really hard. And we should know, RStudio knows a lot about managing packages. So we created a package manager. It's called the RStudio Package Manager, and it solves several problems. One problem is when your environment is disconnected from the internet or you're in an air-gapped environment, you need a way to get packages into that environment, and Package Manager can help with that. Another one is being able to serve consistent environments to Docker containers. So if your Docker container is running one environment, it's probably not the same environment that you're building on, but the Package Manager can help standardize those environments. There's also security control for package repositories. A lot of organizations like to curate the packages that gets delivered to their organization, and then also Package Manager helps with sharing internal packages.
RStudio Package Manager demo
So since Package Manager is brand new, I thought I'd run through a quick demo of how it works. This is the RStudio Package Manager, and it has this nice UI so people can log in and see all the packages that are available to them. And in this case, these are my packages. This is my description. I can see how often my packages are being updated here. So in this case, these packages were updated on March 17th, and 179 packages were added, and 159 packages were archived. You can see the alpha-sim-r package here was archived in version 0.9 because it was upgraded to version 0.10.
Now all the packages in this snapshot are indexed with this identifier 1303. If I want to see all of the in all of the snapshots, I can look at this calendar function and see the history of when my packages were updated. So my most recent version was March 17th, and here I can see this is the specific snapshot 1303 that indexes that specific set of packages. I can copy that URL, and I can paste it into my IDE, and that means that everything that all the packages I pull from then on in the IDE are going to pull from a very specific set of packages. Then I can take that URL, the package repository location, and put that in my Docker container when I spin up the container, and it will pull the same packages I use for my analysis. We call this a shared baseline so that everyone is going to be using the same packages that are pegged to this checkpoint.
We call this a shared baseline so that everyone is going to be using the same packages that are pegged to this checkpoint.
In addition to the indexing the packages, I can also organize multiple repositories. So here you see I have a repository that consists just of validated packages for my group. This repository has packages that were contributed internally from my organization, and this repository has packages that are being pulled down from GitHub. So what this does is what my package manager allows me to do is pull down packages from GitHub and compile them or build them locally so that I can have the most up-to-date packages. I can also see the usage stats for my packages, which is nice. I can see which packages are being downloaded and which licenses are being used. So package manager is really valuable to organizations who are standardizing on R. We hope you'll give that a consideration.
Best practices continued
All right, so number four, use root privileges. So the group in your organization that installs, configures, and manages R in RStudio will need root privileges. So RStudio installs require root privileges, and RStudio products also run as root. RStudio Server Pro runs as the root user in order to create a new R session on behalf of its users, and RStudio Connect runs as a root user in order to isolate applications and processes. I'll also add that system-wide installations of R on Linux oftentimes involve root as well.
So this is very important that the people that have the permissions in your system to install R are able to do so effectively. So that means if that person is you, then you need to know how to install the products and the software sufficiently on that system. If that person is someone else, then you need to help that person understand what this process involves. That means installing R, the R packages, and the products.
Okay, and the last one, five, is securely manage your users. There's two types of users. There's R programmers and there's end users, and the R programmers will need access to R file shares, databases, and probably many other sensitive systems. R processes for the R programmers run as the user under a local account. So users need local accounts on the system to be running these types of analyses. End users, on the other hand, are people that may or may not know R and just consume apps and reports. For these, the R processes typically run under a service account. And if you look at our two products, you'll see that RStudio Server Pro is serving R programmers and RStudio Connect is serving end users.
So how do you hook those up? Well, that's where authentication comes in. Your organization probably has strong opinions on how to authenticate users. This space is only getting more fragmented, not less. So if you look at like a lot of on-prem solutions today, we'll have like LDAP, Active Directory, and then a lot of the web technologies, the cloud technologies have things like Okta, Duo, and Auth, and Ping, and Shibboleth, and many, many more. So we want to support as many systems as possible to work with your authentication system, and we're working hard to do that. And then you will go ahead and configure RStudio with whatever authentication system that your organization has.
Now, if our products don't support your particular authentication system, then we recommend using Proxied Auth. In Proxied Auth, users don't log in through RStudio, they actually log through a proxy that you set up, and all this is in the guides. The supported Auth methods for RStudio Server Pro are PAM, pretty much, and RStudio Connect. The supported Auth methods are typically LDAP, Active Directory, or PAM, and SAML, which is coming up soon. So we're working on SAML. Finally, I want to mention Kerberos, which is an authentication. It's access control, but use Kerberos to connect to databases, and those are supported by our drivers. Our professional drivers allow you to connect to databases using ODBC.
Summary of best practices
All right, so in summary, here's my recommendations. Number one, keep your operating systems and browsers up to date. Number two, plan to support multiple versions of R by building R from source. Number three, organize your R packages for reliability and consistency. Number four, use root privileges to install and run RStudio products. And number five, securely manage your R programmers and end users. So if you stay to these five things as you architect and execute your R platform, you will have a lot more success.
Getting started and the RStudio Quickstart
All right, so now that we have like a path for success, what does it look like to get started? So the first thing is to get the right tools. RStudio makes tools that you put together. There are many ways to assemble R tools, but it's going to be up to you to decide how to put them together. Your configuration depends on what data science means to your organization. Our goal is to make it as easy to install and configure all of our products.
So your solutions might look like a data science lab where all your data scientists work together. Maybe you're building applications in a DevTest prod framework that you're trying to orchestrate and deliver to end users. You could be on-premises or on a cloud or a hybrid cloud. You could have a single server or multi-departmental deployments, depending on how large your organization is. No matter what you're doing, I typically recommend thinking about a crawl-walk-run strategy, where you think about what the platform is going to look like long-term, and then you start with crawl and walk.
Now the number of servers that you use and the size of the servers you use varies across the board. You can put all three products onto the same server. I would call that a sandbox, and that's fine. We do that internally. You could put a different product onto each server, and this is typically what we recommend, is separate the products onto their own servers. We call that a data science lab. You have three servers that are working together. You can double that for high availability, if you'd like to make sure that you're going to have a server to fail over to if there's a problem. And then if you want to extend those out on a clustered approach, you can do that as well.
All right, so once you have your hardware set up, let's talk about recipes for doing the installation and configuration. Recipes are a list of ingredients that make up your platform. They help you organize and automate your work, and they're probably going to be unique to your organization. Now, most recipes will have a structure that will look like code for Linux R and R packages. Then there will be a small section for your installation, and then the rest of it will be about your configuration. So if you've installed R properly, installation will be pretty easy to do. The configuration can be simple or complex.
So I went back and pulled out some of my old recipes, and I wanted to show you one here. This is actually the recipe that I used to build the box that this script is actually running on, and you can see that most of it is around Linux and just getting set up, getting the dependencies. And then I come down here to this section where I show multiple versions of R being built from source, and here's my packages, all the packages I installed in my system library. And then this little section is the install, and that's really typical. If you installed R properly, hopefully our installs go really easily. And then this last section is the configuration, and like I said, your configuration may be complex or simple.
What you really want to do with these recipes is go ahead and automate them. So my colleague Kelly O'Brien gave a great talk at RStudio Conf called Configuration Management and Tools for the R Admin, where she argued for infrastructure as code using things like Ansible, Chef, Puppet, CodeDeploy, SaltStack, or systems like these. And what these do is they organize your recipes so that you can manage your platform more effectively and automatically.
So I definitely encourage you to have those conversations with your team on how to set that up. You need to be very good at deploying R and managing upgrades and helping these platforms scale out. And to do that, you might want to buy or procure a sandbox that allows you to play around with the configuration scripts. So the sandboxes are great places to learn, and if you haven't done this before, it just typically involves reading a lot of docs and trying things out.
It's tax season, and playing with system administration reminds me a little bit of taxes. You just have to read a lot of really boring and detailed documents and try things out. But the general idea is open up the docs, get familiar with the work, and try things out.
All right, so what if you've never used any of these products? Well, that's where the RStudio Quickstart comes in. So the RStudio Quickstart is a virtual machine that runs on your desktop. It includes all of our professional products and includes pre-built assets for you to explore and demonstrate to others. So the reason we did this was so that people could experience RStudio professional products freely and easily. And I want to go ahead and demo the Quickstart to you today.
RStudio Quickstart demo
This is the Quickstart page. You enter your name and email, and the download will take about 20 minutes to download. It's a fairly large payload for your internet connections faster, maybe 10 minutes. And then you'll need to also download and install VirtualBox. And VirtualBox allows you to run this VM on your Windows or your Mac desktops. There's instructions for both. We'll just pull up the appliance and hit the Go button, and you will be good to go.
So once that's ready to go, you'll point your browser to localhost 5000, and it will look something like this. So this is our Welcome page, and we can take a tour, and you can find out all about RStudio Connect and play around with that, and a little bit about RStudio Server Pro as well.
So let's go ahead and get started. We'll just log into RStudio Connect. This is the main UI for RStudio Connect, and you can see there's some assets here that I've already put into the tool, and I'm going to go ahead and show you a couple of those.
This is the Enterprise Ready Dashboards. The great thing about this dashboard is that it runs on Shiny on RStudio Connect, and it connects to a database that we actually installed inside of the virtual machine. And this is done with D3. I can click this, and it will zoom in. If I drill down, it'll open up a new tab. If I drill down again, it'll open up another new tab.
And I can decide who gets to see this dashboard. So for example, right now, anyone can see it, but if I just want to limit it to specific users, I could do that as well. I can also decide how many resources to give the dashboard. Now, this is in a VM, so the resources are going to be limited, but if this were on a server, I could increase the number of connections and serve more users.
So this is a Shiny application, and Shiny applications are really great for interactive analyses. Sometimes you need to run things in batch mode and send snapshots of the work that you're doing, and for that type of work, you want to use an R Markdown document. So here is another dashboard, but it's written in R Markdown. So it might look like a Shiny application, but it's actually not. It's actually a static dashboard. It just has a lot of interactive graphics inside of it. And what's cool about this one is that I can actually make changes to this dashboard through parameters that I've added to my document. So R Markdown can be parameterized.
So here, if I select new settings and I do a new window, run up the acceptance rate, I can run this report, and this will update that report with those settings. When I'm done with that, I can go ahead and send out the email. So I'll go ahead and run this report. While it's running, I'll come over here and look at the email. This email is also inside of the virtual machine, and you can see it has a star because I just sent it today, just now, and this is what the email looks like. So it's got some graphics in it. It's got some information. And I've attached my Excel document here, which allows me to pull up Excel and look at the data.
So this is a great way to share your information. And if you want to distribute your information, you can do that as well on a schedule. I'll hit schedule. I can go ahead and set the schedule and say I want to run this report every Monday, Tuesday, Wednesday. Save that. And I can send myself an email when I'm done. I can send it to collaborators. And that means I can send one report to one group and another version of this report to another group on a schedule. I don't have to worry about it. And if I do have any issues, those are all going to be tracked in the logs here, and I can see the logs. Go back to my code. I can fix the code and publish that information here.
So how do I publish information back in here? Let's take a quick look. I'll come back to the welcome page. And notice I have a lot of other products in here. I've got RStudio Server Pro in here as well. So I'll go ahead and click that. This will deliver an instance of RStudio Pro to me. And I have a simple Shiny application here. I can run this application in my IDE. And you can see that I can play around with this. And I can publish this to RStudio Connect.
Oh, it doesn't like it. Let me go ahead and create a new one. File, new file, Shiny web app, new app. I'll publish this to RStudio Connect. And while it's publishing, what it does is it goes back and it kind of describes all the environment and then sends that information over to RStudio Connect so it can rebuild the environment along with all the code and data assets.
And there we have it in RStudio Connect. And again, I can decide who I want to share this with. I can even open this in its own window. And then that URL will work with anyone that's on my machine, which is me. I'm the only one on my machine. But this is how you would share a Shiny application on a server.
Okay. Now, in addition to the Shiny applications and the R Markdown documents that you can publish, you can also publish APIs. And APIs are basically systems for allowing, you know, one computer to talk to another computer. And in this case, this is a plumber API. And that means that it's which uses REST. And that means I can call my API through a REST call, a REST-based call like through a browser. So, I will show you what this looks like if I go ahead and try out this API. And I say, give me the stock ticker for Facebook.
Right. It gives me all the information for Facebook. And it gives me this URL that I can use to query the API. So, there's Facebook, like do Microsoft, query the API and pull up Microsoft. Now, APIs are extremely helpful for the handoff problem. If you need to hand off your R analyses to someone else, then APIs are a great way to do that because all you have to specify to the other group or the other system is the API itself. You don't have to say anything about R. So, you can keep R running on RStudio Connect and use that to serve information to other systems. It's really nice. They're easy to code. They're easy to deploy.
Now, APIs are extremely helpful for the handoff problem. If you need to hand off your R analyses to someone else, then APIs are a great way to do that because all you have to specify to the other group or the other system is the API itself. You don't have to say anything about R.
So, in addition to RStudio Server Pro, RStudio Connect, you also have the package manager, which I've shown you already, and you can see how this has a curated package list. You can also come in here and take a look at the resource guide, which will give you an overview of getting started and learning about our products. There's this great article here under tips for planning your evaluation that talks about how to explore the value of the products, proving the process works, and testing the configuration.
So, these are all great for getting started. If you have any questions or need any help, you can get that here at the RStudio community forums under the help section. And our team is all over this. The person that put most of this together was Cole Art. This is Cole here, and he would love to hear from you. I would definitely love to make the products better. So, if you end up using Quick Start and you find that you have an issue, please reach out to him.
We hope that you'll use the Quick Start. Try it out. Get a feel for it. It actually comes in handy for many things that I do personally, so I find I get a lot of benefit out of this. We hope that you'll get a chance to try it out and experience some of our products, and we hope that it's an easy and a good experience for you.
Closing summary
All right. So, in summary, we've talked about the happy path, about keeping your operating system and browser up to date, supporting multiple versions of R by building R from source, making sure that you have easy access to R packages, installing products as root, and using a supported operating system. We also talked about recipes and checklists, the crawl, walk, run strategy, and using the sandbox, as well as RStudio Connect or RStudio Quick Start. Again, reach out to us. Please talk to Solutions Engineering. We'd love to hear from you.
We'll be publishing some more material like this to help your teams scale out their products, not just RStudio Server Pro or RStudio Connect alone, but all of the products together. I think we'll be providing more resources around that approach. And again, if you do have any issues, please look us up on Community and share your feedback with us. We'd love to hear from you.
Q&A
What is the best way to dig into the R files that have been hosted on the RStudio Quick Start Connect? Yeah, so there are a number of files on the Quick Start, so that's actually an interesting question. The easiest way for you to get the R files is to come into this view and click on Source Versions and click this and then download the bundle. This bundle will contain a description of the environment as well as all the code that's required to make this run.
All right. You showed how to deploy API and Shiny apps based on R. How does it work when several customers call the service at the same time? So when you deploy a Shiny app or an API, RStudio Server Pro is going to create a bundle and ship that over to RStudio Connect. So RStudio Connect can handle multiple bundles simultaneously, right? So it can handle all these bundles coming into it. And then when it rebuilds that application, it's going to create a container inside of RStudio Connect to run that application. So every bundle is going to get a different container.
Can this setup be done in AWS as well? I'm not sure what that means by this setup, but all of our tools can be done. It can be built on-prem or in the cloud. And this exact setup that you're looking at at Quickstart would be done in AWS.
How does storage work between local and Kubernetes? So we're going to do another webinar about deployments and how deployments work at RStudio and what the best strategies are for doing deployments. Right now, your deployments are being handled kind of on the back end for you. You're not seeing a lot of that work, like when you push a button and you deploy an application or when you start up a container and it just automatically runs for you. You're not seeing the deployment process. But what's happening there is the same thing every time you create an environment, you describe the environment that you're developing in, you bundle up the code and you ship it over, and then you reconstitute that environment in the new location. That's just how deployments work. So the question is, how does data storage work? Well, you can try to ship over that information through environment variables or something like that, or you can reattach that information through shared file storage.
All right, can RStudio Server Pro run Docker R instances? So RStudio Server Pro today does not run Docker R instances. That's the short answer. In the future, we'll have the ability to launch jobs using different container systems.
Does the Plumber API need end users to have R? Yeah, so no. So the end users on a Plumber API are just making a web call, just like you're calling out to a web page. That's the beauty of the API, is that the R process is actually hosted on RStudio Connect. So say you have a scoring equation that you want to run against a database. Your database could actually be configured to call out to RStudio Connect, send it the information that needs to score that model, and R would run on a Connect and deliver the score based on whatever complex analytic model that you've created in R, and then send it back to that system without that system ever having to use R. That's the beauty of the APIs.
What's the best practice to deploy RStudio Connect from dev to prod environment? Yeah, that's another really, really good question, and I didn't get a chance to address it here, but there are ways to set up. I mean, well, first of all, I would say generally that your dev environment is your RStudio IDE, and so you're wondering how do you deploy it to a production system. There's the push-button model that I showed you, but there's also these programmatic deployment functions that allow you to share or publish your work on systems that don't have any information about R, and that's useful for publishing from GitHub, for example, or from some other automated system that your organization uses.
Where Linux systems have shared home directories for users? What are the best practices for running multiple RStudio Pro servers? So, that's exactly right. You're going to want to mount your shared NFS across all of your servers, and there's two locations. One is for your project files. You'll want to put all your project files on an NFS server, and the other one is for your home directory, which has some configuration knobs inside of it for RStudio Server Pro. Both of those should be mounted across all of your servers. You can think of your RStudio servers as compute engines, mostly, right? They ingest information into RAM, and they operate on it, and they spit things back. They don't contain data long-term, and that goes with Connect as well. Connect's more of a compute engine. All of your content should be on a shared file system. All of your metadata should be stored somewhere else, so it's not on those compute nodes.
Are you able to install an older version of packages? I think that's a package manager question, and yes, you can organize those packages any way that you would like.
Can you explain air-gapped environments for package manager? Emilio, I take it that you haven't had to deal with that, because if you did, that would be a painful question. Air-gapped environments are set up for security purposes, and those servers don't have any connection to the internet at all, so the only way you can get information into those is through another process, like some sort of file copy process that the admins control. An air-gapped environment means that if you want to get the latest information, you can't download it. You have to go talk to somebody else to have it transferred over into your setup. If you ever had an old Macintosh circa 1985, that would be an air-gapped environment.
Could you comment on why building R from source is better than, say, installing from Debian or a bunch of packages? Building R from source is better because it builds against your own system, so you're going to be confident that it will run with your system. When you compile it, it's using the libraries on your system. It also, like I said, allows you to put R side by side with other installations of R, so it becomes extremely useful, but I think best practices in general involve compiling that software on your system. What we tend to use the repositories for is identifying the dependencies, so when you build R, you have to go out and find all the things that are needed to build R, and that can be useful. The repositories can be very, very useful for that, but actually getting the binaries from the repositories, I try to steer clear from that.
System-wide libraries versus user-installed. Yeah, that information is coming. There's a lot of work that we're doing right now when it comes to library and package library management, making things reproducible, so your question is a really good one. How do you handle libraries, system-wide libraries versus user-installed libraries? Traditionally, the system admin has control of your system library, and then your users have control of your user libraries, and that tends to be what we recommend today. That's kind of the de facto standard. I think in the future, we'll have better organization around libraries and how they interact with projects so that things become more reproducible.
