{plumber2}: Streamlining Web API Development in R - Barret Schloerke
Over the past nine years, the R package {plumber} has simplified the creation of web APIs using annotations over existing R source code with roxygen2-like comments. During this time, the community has gathered valuable insights and identified numerous areas for improvement. To invest in a way forward, a new package called {plumber2} has been created. {plumber2} is designed from the ground up to be highly extensible, enabling developers to easily integrate custom decorators to modify the behavior of their APIs. Furthermore, {plumber2} is built using a modern foundation, leveraging the latest packages associated with the {fiery} framework. This modern architecture is built upon middleware (the ability to introduce custom logic at specific points within the API's request handling process). One of the many fine-grained controls over how your API can behave. By incorporating these improvements and embracing a modern framework, {plumber2} offers a sustainable path forward for building web APIs in R. This new approach avoids the need for short-term fixes and ensures that {plumber2} can continue to evolve and adapt to the changing needs of developers
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Okay, plumber, right? That's what we're here for? Regular plumber? No, plumber2.
Okay, hi, I'm Barret Schloerke. I am a Shiny software engineer at Posit, and today we're going to talk about the transition from plumber to plumber2, and why we're having a new package and it's not just some extra functionality. I am here today presenting, but I am just a messenger. A big thanks to Thomas Lynn-Peterson. He's the one who actually has been implementing all of this, and I'll be thanking him a few times throughout the talk. I just get to present and, you know, with you today, but Thomas is the one doing all the Herculean efforts behind the scenes for plumber2. So, thank you, Thomas.
Terminology overview
Before we get started, I just want to go over a couple of terminologies just to make sure we're all on the same page, but I think if you're here for plumber, you're most likely already familiar with this. So, the plumber package, it took, you would have a script and it would transform those functions with decorators into a web API. A web API, even though we had just a wonderful talk beforehand, web API is a URL interface that abstracts away the underlying data computation. This is nice because, like, you don't need to tell your friends that it's R behind the scenes. It's just a web URL. They can curl it. They can use hitter. It doesn't matter. You're abstracting that away, but the processing is going to be done by R. And the client will send a request and then finally the plumber server will respond back. And a decorator is similar to like a Python decorator. It's just a function that enhances another function and it's attached typically in the lines above the original one that you wrote. And we'll look at these in a little bit. But if you're familiar with any package development, you can almost think of it as just a little roxygen2 in the comments. We're just going to steal those and leverage that functionality.
Why plumber2?
So why plumber2? After using plumber for a while, you'll kind of run across three you'll possibly run across three issues. And we're going to dive deeper into these three issues. One, plumber does a really good job of magically providing parameters. It's very slick when you can just say, oh, I'm going to have a parameter X. And please, let's have a function of X. And it just magically shows up. That in its own right is awesome. So it's very, very powerful there. The other one is you could only really mount a single sub-router. So if you had something like two forms of static docs or static assets and you want to mount it at root, it's not really going to work. You only get one of them. And we'll go into that in more detail later. And then finally, you could create web API using the roxygen2-like comments, but it wasn't actually roxygen2 comments. And we'll explore that a little bit some more later. plumber2, though, if we kind of bounce back and forth, plumber2 will clearly provide those parameters to the request. You can define many sub-routers. And then it's actually doing a web API using roxygen2. So we're not reinventing the wheel. We're using the battle-tested packages to do the heavy lifting for us.
Problem 1: magical parameters
All right. So problem one, let's look at those magical parameters. This is a function that is going to describe just a quick and easy return value where you pass in a message, and it'll say, here's the message, and then the value. Not too complicated. We have a parameter that says message, message to echo, and then we also are saying that this is a post route, and it's located at slash echo slash message variable. So we're going to actually populate message according to the path value. Like, doesn't seem like it would be too controversial, but there actually is a lot of controversy behind it.
So let's say I was going to make a post to that path, and I know it's echo slash my message. Cool. It receives a path parameter message. Did you also know that you could do a query parameter and say MSG equals Barrett, which is not my message? And you can also do a body parameter message, which is not either Barrett or my message. It's something else completely different.
So million dollar, dollar, or single dollar question, which message do you think is the one that's provided to plumber? Has anyone hit this before? Michael, which one is given to the function? Do you think it's the path, query, or body? The body is reasonable because it's a post, but the path is the one that was defined on, but the answer is it's the query parameter that gives it.
answer is it's the query parameter that gives it.
So because this underlying behavior is something that most likely people are expecting, I can't just hose over everyone that's using this behavior. And so now we're starting to say maybe we should have a different function or a different processor for plumber. So now we're starting to push towards it's not just a Band-Aid that we can fix. We need different infrastructure. So the answer in plumber2, these are going to be clearly defined. So instead of just a single at param, we're going to have at param for path parameters, so that still stays. We're going to use at query for any query parameter, and then at body for any body parameter. That way we can distinguish where things are coming from, and it's a one to one relationship. This also helps us into letting us know what should actually be passed into the function. Also, if you're familiar with rec and res, they've been expanded out to request and response. So this is now what the same function would look like in plumber 2 with the extra enhancement, so you can see the difference between the query and body. So we have param message, path message, query message, and body message. If it's a query string, they are no longer automatically added into the function. So the only ones that are added into the function is the path parameter and the special parameters such as request and response. So in this case, message is only going to be the path to access the query parameter message. You can do query message, and same with body. And I think it's very clear, not as super convenient, but it's very clear and precise as to what's actually happening.
Problem 2: routing
Routing. This is the one that got me. So who here has written a filter, possibly? Who has used preempt? Yeah, preempt, something that I still don't understand exactly when it would work, but these were the only two controls that we had for flowing through a plumber API. There was also, you had to use a function called forward, and it just must be called somewhere within your filter. This gets especially tricky when you're doing promises, because it's like, does it have to be returned somewhere? I don't know. It's just a little weird.
And then in addition to all this, because we have these control flows at the front, if you have a mounted API, it does not go to the second plausible mount as well. So let's say I have an end goal of wanting a request of ABC, and so I'm going to take just a C, a sub-router, and I'm going to mount it at AB, so I could have the ABC path. And let's also say I have another router that has support for BC, and I'm going to mount it at route A, so also would support the ABC route. plumber currently says that once you go inside the CPR, you request that path ABC, it will never come back and then go to the next router. That's kind of frustrating, because you want to say, oh, I have support for both of these things. They're both valid. Can I please make both of them work? And it does not happen, and if I now turn that switch, there's going to be a lot of you who are expecting it to fail on that first route, and now all of a sudden it doesn't, and that's a very dangerous move as a package developer. I don't want to do that to you.
So the answer, middleware. If you've done other web frameworks, middleware seems like it should be a first-class citizen, and now we're making that in plumber2. This is because plumber2 is built on a whole set of suite of packages, so it's not just something that has been, like, it's not just wrapping like HTPUV, but instead it's built upon Fiery, it's built upon Router and RecRes, and thank you, Thomas, again, for doing these in the years even before we made the decision to do plumber2.
There's a lot of neat things, so if you are deep in the weeds on plumber ecosystem, I do recommend looking at those packages to see what you can achieve, because when we are having the conversations, it was just like talking with Thomas, and he would say, oh, yeah, we can just glue this, glue this, glue this. So plumber is just the interface, and then all of these packages are the one they're actually supporting it. So thank you again for the insight on preparing those to have this conversation.
But I would like to talk about Router. Within plumber, we made two new objects that you can return. These are next and break, and they feel a lot more natural, and instead of, like, calling them somewhere, they must be returned from the function that you're using. So next will just kind of tell you to move along to the next available function. You know, kind of says it in the name, but if you return it, then you're explicitly stating that this, there is no value, there is no, like, I don't want you to, you know, tell me the number five. I want you to move on to the next plausible route, and after all routes are exhausted, then we finally throw the 404, not at the first mismatch of a route.
And this is really interesting, because before with plumber filters, it kind of felt like everyone had to do the preprocessing with those filters. But now with Routes, they actually prefilter according to the URL first, and then you can do your processing. So, this is really nice for, like, logging or authentication or anything else that is a little bit more advanced, especially when you have things like you only want to have certain parts be available. With before, in plumber filters, you had to do all that URL filtering yourself, and that is annoying, and no one wants to do that, you know, like, in separate locations as well. So, let's look at example.
We're gonna have a parameter, username, nothing special there. The username is from the path through a git request, and it should possibly respond with a 200 with some extra information. Not going to cover that today, but that has been enhanced from plumber with the description for the parameter. And then finally, we have two functions here. I'm showing off a little bit here. If you're familiar with async or with future or Mirai, there's a little tag that you can do there to have that first part of your function be async, and then you'll process the next part. So, that's kind of handy. But what it shows is that eventually I do a server log message, which would print to the screen, and then you set a header for some whatever you're doing, and then we return the value next. So, I didn't have to tell you there was any promises. I didn't have to set up a future to execute. And also, we return the value next, and it's very clear as to what's going on. We're gonna go on to the next matching route. And this would only apply if it's under the user slash username plausible route. So, very exciting.
Problem 3: extending plumber
And then finally, extending plumber. With plumber, we had these routes, but usually if you wanted to make a new one, such as like the package plumber Tableau, wanted a new custom handle, new tag, they had to actually go reimplement a full parser to reprocess the plumber file themselves. And like that's not useful. If you want to do your own security or authentication, you had to do it through a filter rather than just having your own custom little tag. So, no ability to add your own. I feel like this all the time when wanting to do those. Instead, with plumber2, we have add plumber2 tag to help you prepare your API route. So, for example, when we have the param, at param, what we will do is capture the name and the description. And maybe the open API description as well. And that's all that that function will do is it will just capture that part. This is useful because then after all of the tags have been captured, then we will apply some functionality to that block. In the case of like at get and at post, those functions aren't really useful until you know all of the parameters that can be described. And so, it's best if we actually collect all the tags first, tell like, hey, I should create a get function at the end with these parameters. And then when we process it, the param block is actually going to just do nothing. But the get function or the post function will say, oh, please collect the parameters and now I can make my method. And it works out really well to do it in these two stages.
So, given these two methods, we really do need to go back to plumber Tableau and just try to add these two methods. And then we can probably drop maybe like at least half of their code. Because most of it was just redoing that script. Maybe plumber2 Tableau, plumber Tableau 2. But I think that's going to be the hex. So, feature of plumber.
Future of plumber
In my eyes, it's superseded. I know my attention hasn't been on it over the last little bit. But there will be one more release with extended features. But it will be entering into maintenance mode. So, only security issues. It's not going anywhere. It's not really going to break over time. Because it's still doing its good work. But no major feature work will be done. Because the attention is going towards plumber2. And yeah.
