There's a new Plumber in town (Thomas Lin Pedersen, Posit) | posit::conf(2025)
There's a new Plumber in town Speaker(s): Thomas Lin Pedersen Abstract: Announcing plumber2! The popular plumber package has gotten a successor. In this talk I'll walk the audience through the "How" and the "Why", what it means for existing plumber APIs, and of course showcase a range of amazing things that are suddenly possible with the new iteration. If you are an existing plumber user I hope to leave you full of excitement about the future of plumber. If you have never used plumber before I hope this talk will show you just how easy it is to create modern and powerful web servers in R. posit::conf(2025) Subscribe to posit::conf updates: https://posit.co/about/subscription-management/
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hello, everyone. My name is Thomas. Thank you all for joining me both here live and if you're there online And welcome to my talk. There's a new plumber in town now the Observant among the audience will quickly figure out that this is about the plumber package at least if you know about the plumber package
Plumber has been with us for a while now It's it's a pretty old package. You may not know it, but the first commit was back in June 10th in 2015 and The statisticians between you will quickly figure out that this means that plumber has a 10 year old anniversary this year, so if you if you read the Title of the slide you might get the sense that this talk will not end well for plumber
But I think we should just like take a brief moment and and recognize What a tremendous success plumber has been both as a package, but also at large for their community I mean it has allowed data scientists with no or very limited knowledge about web technologies To take their R code and share it with the world or share with their organization. I mean that is tremendous
I mean it has allowed data scientists with no or very limited knowledge about web technologies To take their R code and share it with the world or share with their organization. I mean that is tremendous
The case for a rewrite
But it's an old package and I have some experience with old packages. I also maintain a package called ggplot2 which is 18 years old today, so I have I know something about old code and one of the kind of you can call it a Law of thermodynamics of coding is that old code accrue technical depth. There's no way around it
So what do you do once your code evolves into something that becomes hard to maintain? There's different ways to do it deal with this one of them is to kind of just accept the situation There's nothing wrong with that. I mean you can just say well this code is horrible to maintain But on the other hand there's not a lot to do in it, so whenever I need to like delve into it I just have to accept that I have to fight with this code base
This is this is an absolutely Acceptable approach to it because on the other hand the code base is trusted and true It has a lot of mileage under its belt, so there's nothing wrong with keeping it around you just have to kind of Reconcile yourself that it will not be a pleasant experience to to continue coding with it
There's another way around it. You can try to kind of surgically replace some of the the code that that is responsible for the technical depth This is kind of what we do in ggplot2 I think every every time we release ggplot2 we have taken out some of the old code and placed it with something new and this is this is perilous in terms of Maintaining a package in the sense that it's very hard to convince yourself that when you replace something internally That it just continues to work as before outside of the world
So it's certainly not something that that you can do all the time But on the other hand you you have something that continues to evolve and grow without like you can keep The technical depth pretty low Now there's a third option which is to kind of say Goodbye code leave it behind start on a new adventure
For plumber I have decided the last thing so I'm excited to share with you that plumber2 is a new package that we have Created which will Replace plumber in some sense and the rest of the talk Will basically be about me convincing you that that's a fantastic thing
Introducing plumber2
So what is plumber2 well it's a it's a complete rewrite of plumber There is basically no overlap in the code base between the two Now just because it's a complete rewrite doesn't mean that it is complete new in the sense It is built on a lot of battle tested technology So it is built on top of things that have almost been around as long as plumber But the code inside the plumber2 packages is all brand new
Now don't worry if you're used to using plumber It doesn't mean that you have to go out and learn a completely new concept It will feel very familiar if you're already using plumber if you have never heard of plumber or have never used plumber I can just say that it's a it's a framework that is very easy to get into and plumber2 kind of adapts that idea That being said it contains some breaking changes So you cannot just take your plumber applications and just like blindly port them over to plumber2 But it will be very easy And you can you can kind of take all of the knowledge that you have from plumber and apply a lot of it to plumber2 On the flip side then it is filled with amazing new features
Familiarity with the new syntax
But just to start a bit with the familiarity part So if you are used to looking at plumber code you will recognize this I hope So if you don't know anything about plumber it's a way of creating web servers by annotating R functions So it feels very much as writing an R function And if you're used to writing R functions and documentation you're used to writing Roxygen code And plumber2 annotation feels very much like Roxygen code
So this is this is something that you could put on top of your function And it will turn if you put it through plumber it will turn your function into like an end point for a web server And if you're used to this if you're used to doing this with plumber I hope that you can see that this feels very very similar Nothing dangerous here maybe some new things But some of it is completely the same so for instance this line Which defines that the function below this annotation will respond to people going to a URL Which is forecast slash some city
In the example up here the city is Copenhagen that's where I'm from This is exactly the same as you would have in plumber so some of it is completely the same Now there are other things that are different we're in 2025 now So you can actually have multi-line comments in plumber2 which is like crazy But why can we do this well it's because it doesn't just look like Roxygen 2 It's actually powered by Roxygen 2 now Plumber had its own like engine for understanding this annotation But we don't need to reinvent the wheel So plumber2 is now using Roxygen 2 to pass these things So if you're familiar with Roxygen 2 then everything that you write there will be understandable by plumber
It also has some new tags So if you have your function you need to put some inputs into it One of them is the city that we talked about up here in the URL But there are other ways to put input into your end point And one of them is the query string of the URL Which is everything that follows this question mark And in plumber everything was kind of documented under the parent thing But they come different places from And that means that you could kind of mask your input in various ways Well no more because we now have a dedicated place for documenting And taking in query parameters
We also have a dedicated way for things coming from the body of the request So new things but not new concepts I would say The last thing I just want to point out there is of course more to it Is that we have much richer way of documenting both the import type And the output type of the function So here we say that this output of this function when it responds to a request It responds with a 200 which means in HTTP language it means everything is okay Here's what you ask for And we also document that what is returning will be an array of objects And those objects will have an element called day which is a date An element called temperature or temp which is a number And an element called rain which is a boolean And this is just the tip of the iceberg So we have really built out the whole documentation of input and output types in plumber2
New features
The first thing I want to talk about is asynchronicity Now asynchronicity for a web server is potentially important, potentially irrelevant But if you have a function that takes just a bit of time to compute You will end up blocking all the other requests that comes in while this is computing Because R is single-threaded This is not a problem if you only have one user But if you have like maybe 100 users that are constantly pinging to you or even more Then it can be a problem because everyone else has to wait for you to finish the computations This is where asynchronicity comes in You can send off the computations to another server And then pick it up once it's done and then send it back to the user while you're handling all the stuff
It's not that plumber couldn't do this If you returned a promise in a plumber endpoint Then it could actually do the same thing But we have built this out in plumber2 So that we can automatically convert your endpoints into asynchronous functions They are built on Mirai which is one of the newest and best packages For doing asynchronous computations in R But it is extensible as well So if you have your pet asynchronous package that you want to use instead You can actually just create a new plugin for plumber2 To use that for asynchronous computations
And as well it allows you to chain computations So usually when you have some sort of asynchronous computation You want to pick it up and you might want to do something on the mainframe in the end And you can do this with plumber2 with easy tagging And I'm going to show you how So how do you turn something into an asynchronous function? Well use the async tag and this is all it takes So if you pass this into plumber2 Then plumber2 will take care of converting this function into an asynchronous execution And take care of all of it for you basically
Now if you want to do something after this computation has done You can use the then tag following this tag And then do whatever you want to do on the mainframe before Or main thread before returning results to the request
Another shiny new feature in plumber2 is that it supports running shiny Basically what it means is that you can now launch And serve one or more shiny app as part of your plumber2 application Each app will run in a separate process plumber2 will take care of launching them And take care of tearing them down once plumber2 stops
This is actually a subset of a more general functionality That is in plumber2 which is something called a reverse proxy And what this basically means is that Well you might have something that is not shiny that you want to serve through plumber2 Well you can do that because a reverse proxy is simply a communication layer Between your front end and something else on the back end But shiny is kind of a first level citizen Because we can launch them and tear them down as we want to do How does this look? Well, surprisingly simple There's a shiny sack and you can point it to where you want to serve it from Here we have two different shiny apps that we serve from two different paths One is app slash app one and the other is slash app two
Now you don't have to serve them from a particular sub-path of your application You can also just serve it from the root Now you may think why would i even want to do that? That's just the same as launching a shiny app But you might want to do this because there's also another tag that says accept So you can serve shiny from the root But then have the rest of your api application being served from a sub-path of your url So when people visit the root of your page They're served with a shiny app But they can still interact with it with an api from a sub-path
Well, you'll just have a second file that you mount on a different sub-root And there you'll have all the api logic that you're used to have from a plumber
There's also support for Quarto So if you want to serve Quarto from plumber, that's now possible If you have parameterized reports in Quarto or rmarkdown These can be accessed by using the query parameters in the url Those will be funneled into the parameters in your report You can also have support for multiple outputs So if you have a report and you want to serve both a pdf and an html version of it You can just add those extensions to the path that you're querying from Or you can use a content type header to do that And it has built-in caching So it's not going to rebuild the report every time that the user is asking for it
How does that look? Well, kind of like how shiny was looking So we just have a new tag called report And we're just directing it to the path where we want to serve the report from And then underneath it all, where we'd usually have an endpoint logic We'll just point to the path of the file that we want to serve And plumber2 takes care of all that for you
Security improvements
Last feature that I'm going to talk about is security Now, security is quite important when it comes to web technologies It's important if you're just serving plumber inside your own organization It is even more important if you're trying to communicate with the rest of the world And I would say that plumber can do better in that regard And that is why plumber2 is doing better in that regard
So plumber2 is coming with a bunch of plugins that you can add to your Easily add to your web server One of them is just a matter of setting best practice headers This is not like rocket science But a lot of people do not know about what all of the headers in a response should be To confine with best practices And we now take their hands and give them good defaults
There's also, I think this is one of the most requested things in the plumber github page Is something like how do I use course And if you don't know about course, I will show you in a minute And there's also something called resource isolation policies Which is another thing that we now support
So going back to this example that I had with familiarity This is just the same code Consider that we have this weather forecasting service And we are running that from an api and serving it out to the world And then I have another web page I want to use this So I have a web page and some javascript And that javascript is then querying my other api Now this will not work out of the box Because the web has evolved to try to be more secure So that web pages cannot just from javascript ask other web pages for stuff This is something called cross origin resource sharing This is something that you have to turn on And say, well, I want the rest of the world to be able to kind of call what I provide
Now how do I turn this on in plumber2? Well, surprisingly enough, it's pretty simple So we have the course tag And we just say, well, I have this web page My trusted friend I really want to interact with this other api I have And we now turn it on and everything works perfectly
What's next
And then there's the rest Because there are a lot of features in this I don't have time to talk about all of it So this is just a little taster I hope I've gotten you a bit excited about what it can do
Well, I am happy to announce that plumber2 is available on Crayon today Or I would have been happy to announce that if that was the fact The truth is that plumber2 is in the hands of Crayon today So that means that it is probably available on Crayon within a week or so It is very hard to time these things Please use it if you have anything to use it for If you don't have anything to use it for, then just, like, toy with it Have fun with it Please report any issues Please report anything that you feel is missing Any good ideas I'll be happy to add them to it Because the code is now much easier to add stuff to
But also please be mindful of the novelty I'm just saying this If I got you excited, please don't run home And then, like, rewrite all the business logic of your operation in plumber2 And then see it crash horribly It's not that I don't trust my code But it's not been battle-tested yet, right? So start out small and then grow it into your organization
What about plumber? Well, don't worry It will stay on Crayon It is now superseded by plumber2 So it will not receive any new features If you have any requests for features, we will say, well, have a go at plumber2 instead But it will stay there And that means that your current APIs will continue to work as before So you are in no rush to update your packages You can just continue before And then, of course, gradually, I would advise you to move over to plumber2 But in your own pace
You can read much more about plumber2 at this URL And otherwise, I would just say, thank you so much for your attention I hope you will have fun writing APIs
Q&A
Will it work with Vetiver? And if so, when?
Maybe No, this is not something I have dived into yet This is certainly something that we will look into It should work with Connect now It should, of course, integrate well with all our other projects But it is still early days So, I would not try to use Vetiver with it yet But it will come, of course
So, why plumber2? I think you touched on this a little bit in your talk But why plumber2 and not overwrite plumber with the new code And require legacy users to install a legacy version of the package?
So, some of it comes back to It will stay on CRAN You don't have to do anything with the thing that you put in production I think it's nicer of me to not break anything And require users to do something for their business logic So, I'd rather start afresh And allow users to just continue as before But be excited enough to kind of change what they're doing gradually So, I think that is the main thing, really And people should be able to live with having a 2 after it It worked well enough with ggplot2 So, we'll be good Thank you so much
