Making Better Error Messages with Rlang and Cli - Emil Hvitfeldt
An important part of writing software revolves around functionality and features. A sometimes overlooked part is what happens when something goes wrong. There are many reasons something can go wrong. Faulty input, user error or deprecation are a few examples. Regardless of what the reason is, we should thrive towards letting the user know as soon and informative as possible so they can get back on track with what they are doing. This talks showcases how to make better error messages with the packages rlang and cli. Emil Hvitfeldt, Posit PBC Emil Hvitfeldt is a software engineer at Posit and part of the tidymodels team’s effort to improve R’s modeling capabilities. He maintains several packages within the realms of modeling, text analysis, and color palettes. Trying to make slidecrafting a well respecting verb. He co-authored the book Supervised Machine Learning for Text Analysis in R with Julia Silge. Working on book titled "Feature Engineering A-Z"
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hello, everybody. I'm here to talk about how we can make better error messages using the cli package. The reason why I'm talking about this is I'm part of the tidymodels team, which is a collection of packages for modeling and machine learning using tidymodels principles. I am one of four full-time engineers working on that project, and we throw a lot of errors.
In fact, I did a count of some of the beta packages we have, and all in all, just the abort calls from rlang used to have over 1,000 just in 12 of our packages, and we have between 40 and 50 packages right now. This covers the majority of the packages we have, but we still have a lot to go through.
I like to think that the user, when using our packages, needs to follow a happy path, and here we see one such path. But an important part about tolling on a happy path is that if you stray away from the path, you're able to get back on it as soon as possible, so you can go back into the right flow. This can be something as simple as having good documentation to make sure that people, if they're stuck, know how to get unstuck, and names and arguments make sense. But another big thing is the use of error messages that come out if they do something wrong or something unexpected happens.
The happy path on this picture we see right here isn't actually that happy, because if you fall to the left, you have a very hard time getting back on the path. I like to think of that as this style of error message that we've all seen that doesn't say anything. It's something that happened, but it doesn't tell you what to do, how to do it better. This is an example of doing all the happy paths as seen in this picture, where we're not helping you at all.
I like to think that helpful error messages that give you everything you need at the right time is the same way as adding cloud rails to our path. This still lets you do everything you need to do from A to B. You have beautiful scenery, but we're trying to help you stay on the happy path, do what you need to do at all times.
I like to think that helpful error messages that give you everything you need at the right time is the same way as adding cloud rails to our path.
Introducing the cli package
I'm using the cli package, and it can do a lot. The cli, short for command line interface, and it gives you a much rich collection of functions and functionalities to create things in the command line interface, so the text output in your console. It can do things like lists and headers and paragraphs, and it can do all these things with styling and theming. We can do rules and trees and all these different things. It can do so much more than just what I'll be talking about today, and I'll just be focusing on the error messages. But we can do it for anything else. We also used it in making really nice print methods for some of our objects.
If you're familiar with rlang that has the abort function, you can essentially replace most of those rlang abort calls with a cli abort call, and they function almost identically. There's a few exercises you might run into.
Passing calls around
One of the things you can do, so this isn't cli specific. This is a thing that came with the rlang idea, is the idea about passing calls around. So here we have a rather simple example. We have the user facing function that takes in ads, then we have an internal chatter that makes sure that the values are all positive or non-negative, and then it takes the square root. It doesn't really matter what it does. The idea is we have data coming in, then we do some chatting on it, and if it passes it, it goes on to do something.
When we normally write this, we get the following error message below, where it says error in internal chatter, and then the error message was sent. But we have the name of the internal chatter in the error message, and that is better than nothing, but it still is a little confusing, especially for new people coming in. They might not know what it is, and maybe we didn't have a good name for the internal function. Here, it's just called internal chatter. It doesn't at all tell you where the error is, especially if you have a really bad infrastructure of functions in functions. It can be an error that happens all the way down somewhere, and you don't quite know what it is.
We see it a little bit in, if you're familiar with the tidymodels, we create a workflow with lesson recipes and some positive models, and then we tune those inside a workflow. Then maybe the error is coming out from somewhere in one of the recipes functions, but it's happening when we're calling a tune function, so it's nice to be able to know where it's from, and we can't do that right now. But we can do it if we pass the call around. The main difference we see here is the internal function. Here, we added a call argument, and to that we passed in the rlang caller_env function. Essentially, what's happening here is when we run into this exception, and it prints out the error message, instead of using the calling environment of the internal chatter, it is using that calling environment, which is the next function up, which here is the user facing function, so we have a nicer interface. And, of course, if we have these nested multiple layers deep, we could pass this call object around, so the calling environment will be the one that the people use.
Glue interpolation in error messages
This is one simple way of adding information to the user that they really need. Another thing we can do with the cli function is the idea of glue interpolation. This is a little bit the same way as pasting things around, but instead of pasting in a comma-separated sequence, we here use these curly brackets. Now, we have updated the error message here, so we say x must be positive, not, and then curly bracket x. So what will happen when this is run is it takes the value and interpolates it in, so the value we put in was minus five, so curly bracket x is being replaced by minus five. So we can not only say something went wrong, we can also say what was wrong. We can say, oh, it needs to be something, something.
Here's a very silly example, but it can become very useful later on to be able to say, oh, the fifth value was negative and it was minus two, and then it's easier for the user to be like, oh, let's go back and find the value in the input that was minus two, and this is what the error was, instead of just saying something's wrong.
The last nicety around the cli is it also has this idea of inline text formatting, where we already had the curly brackets, but if we add this, almost like class names, dot art dot about, there's a lot of different ones you can find in documentation. We can denote the x as an argument and the interpolated value from earlier, we're saying it's a value. So that gives you a standardized values about how to style different things across everything.
Priorities when writing error-throwing functions
The way we write these functions, what we think about errors, is mostly like this. The data comes in at the top and we have a series of checks. It can be one, it can be more than one, and then at the end, we do something. We saw a little bit before, this is the same as this, where if we have a lot of checks, we can pull them out into a standalone checker, so we separate where everything goes. With that, we have some priorities of how we write these functions. The first one is that, ideally, we should throw the error as early as possible. If we need to know it's need to be positive, check it right away, instead of doing a bunch of things and then deal with what's used. That helps the user get back into the swing of things. Maybe you have a function that needs a value, but also needs to connect to a database. If you know the value needs to be a string, check it's a string before you connect to the database, because that takes some time, so you can get back as fast as possible.
The check itself, which is what we have here inside the if statement, everything is an if statement, and then if it's true, we throw the error. The check inside the if should ideally calculate as fast as possible. It should just detect, are we wrong or not? Because then once we're in, we can say, okay, now we know it's wrong. Then we can find out all the different places it's wrong and construct the error that's informative. This allows the call, when it does an error, to run as fast as possible.
Examples from tidymodels
I'll be showing three different examples of how we improved some of the error messages inside the models using the cli framework, which can do a lot more than I'm showing right now. We're seeing a little bit of what we can do. Here we have our old error message. If we run the following code, we get this can't rename variables in this context. It has words, but it doesn't at all say what. It's very unclear why we throw an error right here. With our new error message, we're explaining why. By saying that this function doesn't use the estimate argument, it needs to be specified just in the dot auto-named argument, and we're even saying how to fix the error. Here, we specify the call without the estimate equals.
The way we're dealing with this is somewhere in a call somewhere where we have an if statement, and here we're doing some fancy things. We're getting the call, it doesn't explain the dots, we pull out the dots, and we do the names. Basically, if the dots are named and estimate is in there, which it isn't supposed to be, then we throw the error. What we're passing to the abort thing is here actually a named vector. We have the different elements are named here. One of them is named at, and one of them is named I, and we can see this in the left-hand symbol on the thing. So, at is like the big red at, and the I is this little blue I. There's a couple of shorthands for at, I, and exclamation point. But we don't want everything to be red error all around. It's like, oh, here's the error, and then a little I, how to fix it. We're passing it all through because this is in a helper function.
Another problem a lot of you have probably seen before is the idea of exhausting the memory we have. This is an example where we're using recipes to create dummy variables, but the variable we're creating dummy variables of has 100,000 unique values, but it's just territory representation of the numbers one through 100,000, which would be a really bad matrix. It's like 100,000 times 100,000 values. That's way too bad. But the error we did bad comes from inside a base R function that says vector memory exhausted, limit reached question mark. So, we don't get the error coming out saying, hey, this is the way the function is. If this was inside another function you used, it would be hard to know why this came out unless you remember this error. Another little wrinkle in this error is that it's operating system specific. So, it will look different if you're on Windows than if you're not on Windows.
Our new error uses cli. We say it's coming from step dummy. It comes with, and it now says that it contains too many levels. We're saying how many levels it had, and then we're explaining that it would have resulted in a data frame that was too large.
Here it is a little bit more advanced because we are using model matrix to try to create these dummy variables. If it doesn't work, I want to throw the error instead because I want to reframe it. So, we're using this try-catch statement at the top with the expression in it. If that expression does an error, nothing happens. It's just this pipe. It's just assigned to indicators. But if there is an error, then that error gets pumped into this function right here. What we do is we take this condition, which is the error. The error is an object by itself. I'm doing some regular expression on the message to see does it contain the words better memory or cannot annotate. If that happens, then I pull out the number of levels and then create a call, just as you've seen before. If it can't find that, it will just let the error message through as normal. So, it's foolproof in a way of like, if these words appeared, we replace them with my warning or my error. But if it can't find it, it just moves along like normal.
You might have noticed here, it's just call equals null. It's because Recipes itself is doing something a little bit fancy about how to identify who throws the error. So, we have to specify it here.
Our last example here is also from Recipes. So, the old error right here said you can't prep a tunable recipe, arguments with tune, this threshold, and then a little suggestion. But if you look above, we actually have the threshold, non-tomp, and degrees3 all tunable. So, this error is actually really annoying because if you fixed it and ran it again, then if you remove this tune, it would come again saying, oh, non-tomp doesn't have it. And this would be a lot. So, it's not the right way of handling it. Our new way of handling it is we're listing out the steps. So, we're saying these steps have these arguments set to tune, please fix. And we're doing something again. Again, at the top, number of rows that needs tune is more than one. Very fast calculation to find out if this is a problem or not. And then once we're in here, we're doing some things. We are operating on this needs tuning data frame to find out what it needs.
The message we're creating is in two parts because the way this list right at the bottom has the steps and the columns is hard to do. So, we are creating that using the step message. Another nice thing we have here is we can add the pluralization. So, this ?s in the curly brackets will be added if there's more than one of the quantity we specified earlier. So, if there's only one step, it will say the following step has. So, this is very neat in that way.
All right. And that's everything I have for you. I encourage you to go and look at the cli package. Thank you.
