Workflow Demo Live Q&A - April 24th

Everybody, thanks so much for joining us over here. We'll get started in just a second here. All right. Hey, Julia. Thank you so much. Can you hear me now? I can. Yes. Great.

Okay, perfect. Well, we are just waiting for a few people to come over to the Q&A room. So let's give people two minutes or so here.

Just wanted to remind everybody, if you want to ask questions, you can type them into the YouTube chat here. But you can also use the Slido link, which is shown on the screen. There's a little poll in there that we would love to have you answer just to learn a little bit more about how you're deploying models today. Okay, we're up to 70 people in the Q&A room. So I think we can get started and jump in. Thank you so much, Julia, for the awesome demo and for joining us today.

And I did just want to remind people again, I know I said this in the beginning, but registration is open for PositConf 2024. And so I just want to remind everybody about that. If you enjoyed today's demo, Julia is actually leading a workshop at PositConf as well. And I was thinking, Julia, before we jump in, maybe you could tell us a little bit about the workshop at Conf and what that will cover.

PositConf workshop preview

Yeah, so I am pretty excited about this workshop. I'm teaching it with my coworker Isabelle Zimmerman, who also works on the Vetiver team, working on Vetiver. And actually, this workshop last year when we did it, it was the first time we had ever done a workshop where people were working in R and Python at the same time. Like in the room, we taught it with R and Python together. And it was I feel like it was a really interesting learning opportunity for people who were there.

We had quite a number of people who maybe felt like they were most comfortable with R and they started with R. And then when they saw how approachable these tasks were in Python as well, they kind of like switched over and like people tried both, which was really interesting. It went quite smoothly having both R and Python users together in the room.

So if I were to say what is it covering? We're not covering much about model development. So I think there are some other workshops that are covering that. So when we were watching this video together today, we saw me do EDA, then train a model and then deploy the model. And so the workshop just covers the deployment piece. So we'll talk about how we version models, how we can like think about keeping model artifacts in a reliable way along with the metadata that belongs to them.

We'll talk about model repositories, like what does that mean? What do you want to make sure is in a model repository? We'll talk about deploying models. What are different ways that you can do that? And then we will talk about monitoring models. Like once a model is in production, how do you monitor its performance over time so that you can know when you need to retrain it, know when you're going to have problems with it.

The workshop at Conf is divided, I would say, between using our pro products and not. So, for example, when we talk about how to deploy a model, we'll first show you how to deploy on Connect, because like we talked about in this demo, the experience really can be so seamless and is so well suited to you as a data scientist. But we also show you how do you deploy your models in other ways, like with Docker.

And so one of the things that I love about working on tools like pins and vetiver is that they can be moved around to the different kind of infrastructure that is right for you and is right for your org at its current level of maturity. You know, so like if you start with storing your models on a shared network drive, you can do that and then like move later to Connect. And if that becomes inappropriate, you can move later to an S3 bucket. And you don't need to change much about your approach, but rather you can, you know, scale up and down, move to different kind of infrastructure. If it's right for you to deploy to Connect, you can do that. If it's right for you to deploy to, you know, some other cloud platform using Docker, you can do that as well. So it'll be a fairly like, here's all the different ways you can do it at that workshop. I'm pretty excited about teaching it again.

Q&A: MLflow vs Vetiver

Aaron had asked, how does this process compared to using MLflow? Is one better than the other? Specifically, if the model will eventually live in an environment such as Databricks?

Yeah, this is a great question. So some things that are similar between using Vetiver and using MLflow is that they both provide you ways to version your model, store your model in some kind of like a place where you keep your artifacts together in an organized way and have ways to deploy models. Some things that are different is that if you are going to deploy or even develop like work on your model, how you're going to like, how are you going to go about deploying it? Like you want to run it locally and debug.

When you do that in with MLflow, you have to have the whole MLflow server running. Whereas when you use Vetiver, you're using REST APIs. And so they're much more lightweight. So I would say the the development, the local development experience is easier to get going with on with Vetiver compared to MLflow.

The Vetiver was built explicitly with so that the barrier to entry is as low as possible, like it's appropriate for someone who is deploying their first model. And then it can grow with the person and the org as like needs for complexity and scale grow. MLflow is built not so much for like data science persona, but more for a software engineer persona.

If you are a Databricks customer, that's fully managed MLflow. And so basically the money you're paying to Databricks is for them to do a lot of that management and keeping it up. So the experience is different if you're like a Databricks customer using MLflow there. The issue there is that if you want to deploy models using R, like you train the model using R, unfortunately, the MLflow experience is not great because it is it is not there's not there's not good support for it. We're actually like talking with Databricks about this, like what can we do to improve the situation. So if there are specific kinds of models you want to deploy to Databricks, I recommend you communicate that up to maybe like your Databricks people so that we can have really good information about what the priorities are and how we should invest in say, making the MLflow on Databricks for our experience better. If you're training a Python model, and you're a Databricks customer, it probably makes sense just to go to MLflow.

Q&A: Posit Connect and model metrics

Thank you. I thought it might be just worth mentioning here just because it came up is that Posit and Databricks have a close partnership as well.

David just asked a question a bit ago. Does Posit Connect allow you to compare metrics between different versions of the models?

Yes. So if these are metrics, say you train the model one week with some training data, and you store that as model metadata, then you read it's like, oh, time to retrain the model. Let me go and check the model metadata. So Connect provides you, Connect plus Pins provides you an approach and an infrastructure for building a model repository. Like here are where all my models live along with the metadata that we need to keep for them. Some of this model metadata is automatically extracted out. But then if there are specific things you need to know, which often you do with metrics, often metrics are strongly linked to certain business outcomes. And so it's like a very specific kind of metric that you need to measure, then you can customize that. Like you can customize what goes in there, which is one of the things that we really want to, when we talk to people about their like metrics needs and monitoring kinds of needs, we found that there's quite heterogeneous needs. And so we knew we had to embrace a lot of customization, make it quite straightforward for people to use like a code first approach to be able to specify what kind of metrics they needed to keep.

Q&A: Vetiver API internals

Jeff had asked, I may have missed it, but is the deployed model served with the plumber package? What open API spec renderer does it use?

So vetiver for R wraps plumber with like model aware endpoints, metadata, information. On the Python side, vetiver for Python wraps fast API with the same kinds of like model awareness. You know, we certainly don't need to be like rewriting how to do REST APIs when we have these wonderful general purpose tools for making REST APIs. However, if you were to, you know, like, I need to make a plumber API for my model, like there's a lot of boilerplate best practices kinds of things that vetiver provides, because there are things you need every time when you have a model.

Because it is a plumber API in R or a fast API in Python, you know, defaults work out of the box, you get something that's useful, you know, for maybe like the 80% use case. But you can also extend these and customize these. In R, you can just like pipe and add new endpoints if you need to. You can, you can edit like we actually generate plumber files that you can then go in and edit in the way that you need. Same thing with fast API. In fast API, we have the ability to add custom handlers in a quite straightforward way.

The open API also gets like the that's how you for those of you who don't know, open API specifications are ways that that kind of visual documentation gets generated, and also some of the validation and whatnot that happens. So the vetiver in R and Python automatically generates these open API specifications, because you can you can learn a lot of what you need from the model. So we can automatically extract a lot of that to make this this like nice visual documentation for you to use. And the specific you probably notice it's not swagger, it's not swagger. Rapid Doc, Rapid Doc, that's the name of it. So it's a rapid doc styling of the open API specification. I thought we showed people like these different options and thought that it was a little better fit for the model use case than the than swagger. This is something you can change, like if you want to, you can change the swagger if you prefer it.

Q&A: Storing predictions and best practices

Brendan had asked, what are some best practices when it comes to storing predictions in real time when using the vetiver API? So the predictions that come out are coming out as responses from a rest API. And people use these in different ways. Sometimes people do what's called like batch prediction. So maybe they need to, you know, it's time to generate predictions for this day's customers or the new orders that came in or whatever. And so you say, take everything that happened in the last day, send it to the API for that whole batch, get predictions, and then they'll come back.

And then they, you know, when I say come back, what do I mean? You can mean a lot of different things. It could mean coming back into R, and then we do something in R, some analysis, we make a report. It could mean coming back into Python. We, it gets in some kind of an app. It could mean coming back via like into JavaScript and doing something that we're displaying in a website. Because it's rest APIs, it's very flexible where the predictions end up.

This question, Brendan asked about real time. And sometimes what people, when people mean when they say that is like, I have a new person or customer or order or thing, and it's here, it's ready. And I, I need the prediction right now for this one thing. And so then we, the API says, ha, okay, here you go. Here is the prediction for that one. And then typically, if you need something in real time, you, you aren't going to store it, but rather you're going to take some action with it. Maybe it comes back into JavaScript, into the website, and then you're going to take some action based on it.

Typically if people are storing, I haven't heard about that that much, about like storing it. If you're going to hear about storing predictions more in typically like with, with batch prediction, and often what people do, will do then is write them to, to a database so that we have, say, a database of all the predictions we ever made. And we can go back and look at them or if that's needed for the situation. I guess if you had a use case where you did need to store the prediction that you made, maybe for compliance or auditing reasons, like you need to know later, what did we do? Typically you, you, you would log that in a database. If you're at the kind of scale where you're making like lots of real time predictions, you will typically want to store that into a database. Since it is API, like a REST API that you're interacting with, you, you know, you can do this not with, like if it's part of some larger production system, like you, you're probably not going to be using R to store from a REST API to a database. Like there's lots of very performance scalable tools that you can, you can do that with.

Q&A: Model monitoring and data checkpoints

Somebody had asked, guidance on building checkpoints into models to verify if data sources are being populated. I think they discovered later source files were missing and stats were incorrect. This is a pretty common failure mode is that you, if I'm understanding this question, we, we deploy a model and it's chugging along and it is going to give predictions and then something actually changes in our system such that the data coming in has changed in some way.

So I have two things I kind of want to point you to about this. The first is that Vetiver has some of these checks built in. Vetiver uses what we call an input data prototype. Basically, let me talk about the shape and the characteristics of the input data for the model. Let me record that as metadata, as part of the model, and then we will check it whenever we make a prediction and check that was coming in, like, complies with the input data prototype. This can also be customized. You can actually make stricter checks or you can turn it off if for some reason that's appropriate to you. So the default is a kind of middle road in terms of strictness of this getting checked. So basically what happens here is instead of finding out some data was missing and you're just getting wrong answers, instead it errors. So you immediately know, like you immediately know.

The other thing that's going on here is that this is kind of a question about model monitoring. And I gave a talk last year at PositConf, like a talk that was called Reliable Maintenance of Machine Learning Models. And it walks through the different kinds of monitoring you can do. So I would say, like, what kind of checkpoints are we talking about? We can have checkpoints about making a prediction that the data complies with what I know the shape of it needs to be. Then you say here the stats were incorrect. And, like, that can be a statement about, like, the distribution of the input data or even, like, the output predictions that we get. And so in that talk, which I definitely recommend you take a look at if you're, like, thinking about this thing, we talk about how to build a strategy so that we monitor models over the long run. So in that talk, we talk about monitoring inputs for things like data drift and then monitoring both inputs and outputs for something like things like concept drift.

Vetiver uses what we call an input data prototype. Basically, let me talk about the shape and the characteristics of the input data for the model. Let me record that as metadata, as part of the model, and then we will check it whenever we make a prediction and check that was coming in, like, complies with the input data prototype.

I know some people have to drop at the top of the hour. So I did just want to say here, thank you so much for joining us today. And thank you to anybody watching the recording later as well. As a reminder, we have these workflow demos the last Wednesday of every month. And so if you do want to join us live for the next one, you can add these to your calendar with the link I shared here in the chat. My colleague Isabella is going to lead next month's demo on Quarto dashboards as well. So I just want to give everybody a preview of what's to come next month.

Q&A: PyTorch models with Vetiver

One other question I see that came in in YouTube from Hamed is how easily can we deploy PyTorch models with Vetiver? My own experience in Python is that simpler models are pretty seamless, but I started hitting problems with PyTorch.

So PyTorch is one of the models that the types of models that the Python Vetiver package supports. And so we expect this to work quite well. So far, people that we know about using it have had good experiences with it. We certainly are interested in hearing on specific kinds of problems as you run into them. But I would say we expect it to work. Give it a go. And we would love to hear about any problems you run into.

Q&A: Free vs paid Posit products

Somebody had asked a question a bit earlier about the difference between the paid and unpaid products. So I bet a lot of you are users of the RStudio IDE. And that is a that is not only free, but also open source products. Like you can actually see the code that goes into making RStudio the IDE. There's a desktop app that you like download and use that is, like, free for all use cases, including commercial and all that kind of thing.

Now, there are paid professional products that we that we make that are a little more designed for the enterprise use case. So if we want to specifically talk about RStudio, like we have a product that's called Posit Workbench. And what Posit Workbench is is a product that provides you with multiple different choices of IDEs for your data science work. So you can choose between, like, JupyterLab and RStudio and different IDEs that people like to use for data science work. And so the reason why companies pay for RStudio Workbench is for the, like, enterprise type features that people need. Things like can we use this with single sign on and authentication? Can we use this with our certain security needs that we have around where people do their model development? So we see people choosing to for the pro products for a variety of reasons.

No, I think you've covered that really well. I would add that Posit Connect that you saw today is also one of the paid products. And so Julia showed today deploying a model to Posit Connect, but a lot of our other customers use Connect to deploy dash apps or Shiny applications, Flask APIs, a variety of different data products built in R and Python.

And I guess just to wrap it up, I guess, so the package manager has a there's a there's Posit public package manager, which provides these really convenient binaries that you can use, like when you make Docker containers. But Posit, but package manager also is a paid product that, for example, an enterprise would use internally. And so the use cases for that would be we have internal packages, we need to make really easy for everyone inside of our organization to use. Or we have compliance needs about people using specific versions of packages that have been vetted and put on a, you know, like on a on a allow list, like these are the packages people are allowed to use. And so if you hook up your, you know, your ID to the package manager, you can get those kind of guarantees that you need.

Q&A: Deploying to Posit Connect via Docker

One of the questions we missed from earlier was about Posit Connect specifically, and it was, is there a way to deploy to Posit Connect via Dockerization or containerization?

So it kind of works that way. Now. So when you when I showed you in the demo, that ever deploy RS Connect, and it went and it made a little bundle and put it like the that is a standalone environment that is separated from everything else. So a lot of the reasons you might choose containerization already work with Connect. Now, if you're saying I literally have a Docker container, and I want it to go to Connect, that is a feature request that I believe, you know, Connect is like the Connect team is talking about. So if you're asking, is there a way to do something like it where it you get the benefits of it, that's that is how Connect works already. If the question is, I literally want to take my Docker container, then not as of today, but it's something that I know the Connect team talks about.

Q&A: Logging model inputs and outputs

Sergio asked, is it possible to create a log file of a deployed model where we can find the input data, the output of the recipe and the prediction? This may be useful for the monitoring.

So the answer is, yes, it is possible. We don't do this by default right now. So you don't default get the logging. But since we are talking about FastAPI and Plumber, there are already tools out there that allow you to kind of like add a line of code, and then start producing those kinds of logs. And then say you deployed a model to Connect. We'll say an R model to Connect. It's Vetiver. It's Plumber. You add one of these packages that has like default plus extensible logging for Plumber APIs. And then it starts going into the logs of the Plumber API on Connect. And you see that there. So we didn't want to build these things from scratch because it's a little bit more about general software engineering practices. But we wanted, like with everything, with how we think about Vetiver, we want to make it very straightforward to integrate into these other kinds of tools, depending on what your needs are.

Q&A: Vetiver development lifecycle

Jeff asked, where is Vetiver in its development life cycle today?

Yeah, this is a great question. I think we may still have, we still are calling it experimental, which I think is a little bit maybe not accurate. Maybe we should move that the badge says experimental on it. And I think we should probably bump that up to something like stable. Because the core functionality is quite stable now.

When we have been looking at new features for Vetiver lately, we have been talking about Vetiver on platform. So like Vetiver on SageMaker. We've talked about, so that one exists today. You can use Vetiver on SageMaker. And then when we talk about like what else would we do for further development, the things that we talked about that don't exist as of today would be like Vetiver for MLflow. Like we mentioned that earlier. That's especially relevant to people who use Databricks. We've talked about Vetiver on the Google Cloud, the Vertex AI is what it's called. Like Google Cloud's kind of MLE system thing. Like how can we go there? So those are the things we talk about now in terms of development lifecycle. Where do people want to take their models?

So right now we have options like Connect, of course, very straightforward. Docker, which you can take anywhere. And then how can we make the process easier to take to different kinds of platforms? So today SageMaker works. Where else would we go with it? So I think I would say core functionality, core decisions that we made about what is Vetiver, quite stable. And our forward facing work is like where might we want to take this?

So I think I would say core functionality, core decisions that we made about what is Vetiver, quite stable. And our forward facing work is like where might we want to take this?

Thank you so much. Thank you all so much for joining us too. And I just wanted to add here, your feedback on these workflow demos has been so helpful. And so I would love to learn what other things you would like to see. What could we be doing to make these more helpful for you as well? So if you have maybe 20 seconds to do this for me after today's session, I would greatly appreciate your feedback. I just put a Google form into the chat here. This is really helpful for me as I figure out what do we prioritize here in terms of community events as well.

But thank you so much, Julia, for the awesome demo. I thought this was amazing. And I love that you had a blog post like ready to go right when it launched. Thank you again. Yeah, thank you for having me so much. I saw somebody's comment in the Slido. So I wanted to echo this as well. Julia, thank you for all that you do to educate us all in the content that you put out on your own YouTube too. In the workflow demo, I linked to Julia's own personal YouTube there. And I highly recommend that you go check it out. And again, if you want to join us for another one of these, we have them the last Wednesday of every month. And so Isabella is going to be leading one on Quarto dashboards next month. But have a great rest of the day, everybody. Thank you so much.