Building an MLOps strategy from the ground up - Isabel Zimmerman, RStudio PBC | Crunch 2022

And now, our first speaker of the day, Isabel Zimmerman is a software engineer on the open source team at RStudio, where she works on building open source MLOps frameworks. The title of her talk is Building an MLOps Strategy from the Ground Up. Let her hear you, everybody. Please welcome Isabel Zimmerman.

So, fun fact, my slides are running code behind the scenes, so I'm going to quickly render them. So, I'm running Python scripts right now, and then we get to start.

Hello, everyone. Today, I will be talking to you all about building an MLOps strategy from the ground up. So, if you want to follow along, my slides are at this link. You can, you know, put it in a tab somewhere and forget about it for a few weeks and rediscover it. Also, these slides are clickable and they're running code, so you can kind of interact with them as we go through them. All you have to do is open up this link. They will auto-advance as I advance these slides. It is the magic of this software that I'm using called Quarto.

But, of course, we're not here to talk about slides. We're talking about MLOps today. So, from the ground up, we're going to start with what is MLOps anyway, and how can I start? We also want to keep in mind what should we be considering when we are looking at using tools for MLOps.

But I'll introduce myself first. So, hello, everyone. I am Isabel. I started my career as a software engineer slash data scientist, and I was deploying models using Kubernetes. And if you've ever used Kubernetes, it's not always, like, the most fun thing to do, so I needed a little bit of a de-stressor, and this was coming from my dog, Toast. This is a real-life image here of him.

And I started training him tricks, and my first trick that I trained him was to sit. And if you've ever had a dog, when you start training your dog to sit, you kind of have them come up, and you hold a treat above their head, and you kind of push their bum down, and you say, sit, and you give them a treat. And all is well. And so I was going to show this off to my neighbor, and I was super excited. So, you know, I have Toast in hand, and we're walking. He's right here. And I say, hi, neighbor. And, Toast, sit. And he doesn't sit.

And, of course, the data science in me is like, I overfit my model. Something is wrong here. And I realized that when I had taught him to sit, I had only ever taught him from the front, but when we're walking, you know, he's on my side. So, Toast taught me an important lesson here.

When I was training him in my cozy living room, I only had the information that I knew where I wanted him to sit. But when we were out in the real world, the circumstances changed. He was on my side, and he didn't realize that this was kind of the same thing. He had to sit in either direction.

But I learned a hard lesson. You know, even if you're training for the right outcome, the real world brings a new set of challenges, and things behave differently.

But this is not a dog training conference. So I'll tell you this instead. If you develop models, you can operationalize them. In fact, I'll go so far as to say if you develop models, you should operationalize them. Well, most of them. Some of them.

Every now and then, there's this statistic that goes around that 85% of models don't make it to production. And people use this as a statistic to say we need more MLOps tooling. And I would say, I mean, from the data science perspective, you train a lot of models, and not all of them are production-ready. So I think it's actually a good thing that 85% of models don't make it into production for both quality and cost-saving reasons.

So I gave my dog information. And at home, there was a great response and a reproducible response. But out in the wilderness of my suburban neighborhood, when I gave my dog instructions, it didn't look the same. Things behave differently in different environments. Models work the same way as Little Toasty. When you're training and tuning a model, data scientists, they often live in their Jupyter Notebook. You have your installations of packages that you've had for years, maybe. Hopefully not. But it happens. And when you productionalize your model, sometimes things look different. Maybe the data that's coming in isn't exactly what your model is used to.

This is really important to think about, because the business value of models often comes from when they're in production. And how that model works in the wild. But we're getting a little ahead of ourselves right now. We'll start with, what is MLOps?

What is MLOps?

MLOps is the combination of machine learning and development operations, which I find is not the most useful frame of mind when thinking about, like, what does this word really mean? So I think a better definition is that MLOps is a set of practices to deploy and maintain machine learning models in production reliably and efficiently. And these practices are hard.

When I started deploying models with Kubernetes, I struggled with a lot of the tools that were out there. I felt that as a data scientist, these tools weren't really made for me. They were made for a different persona. So I actually ended up leaving that job, and I started working on tools for data scientists to help them kind of build the skills to develop and deploy models.

So currently I work on a package called Vetiver, and it's helping data scientists deploy models in both Python and R. So this means you can install the package Vetiver in RStudio. You can also install the package Vetiver from PyPI.

The data science workflow

So let's look at this data science workflow right now. So you start by collecting data. You're getting your data from the cloud or the ground or wherever people are getting data these days. And then there are great tools out there to help you understand and clean your data, things like the tidyverse or data table in R or Pandas or NumPy or Suba in Python. Once your model or once your data has been cleaned and you have a good understanding of what's in there, you can train and evaluate your model. Once again, there are great tools out there like TidyModels and Carrot in R and Scikit-learn and PyTorch in Python.

And let's look at the code. I am going to be showing you guys a lot of Python code today, but it's okay. We're in it together. We're going to go step by step. So we're going to do a little bit of, like, data cleaning. This is really just loading data and selecting columns. And this is looking at a dataset that is predicting how many likes an advertisement will get on YouTube, depending on if the data or if the ad is funny or if it shows the product or if it's a patriotic ad or they're a celebrity or danger or animals. So we load in this data, and it looks like this. We have our like count, and everything else is true or false.

Once we have our data loaded, we're going to, you know, train and tune our model. So we're splitting it up into a test and training set. So this is to make sure that we're not giving our model the answers when we're trying to teach it what it's doing. We have a little bit of preprocessing. So you saw that those columns were true and false. Models don't speak English. So we need to translate that into some numbers. And that ordinal encoder is doing that for us. From there, we're going to use a random forest regressor. So we're going to put those numbers in, train the model. We're ready to go.

We're actually going to put these together in a pipeline. And this is important because you want to deploy your feature engineering if you are fitting it and the model together. If we're learning from data, this is all going to be deployed at once.

So we saw the beginning of this cycle. You know, we have really clear tools of what to use for data scientists. But it gets a little bit hazy of where should we go next after we have built our model. And these MLOps practices that we're looking at, Vetiver focuses on three of them. It focuses on versioning, deploying, and monitoring a model.

Versioning models

So MLOps is versioning. So we are amazing data scientists. We have built our model. And we have named it model like things happen sometimes. And once we have trained our model, we're ready to deploy it. So we have our model final. And then we realize maybe we need to do some edits. We need to tune some more parameters. So we have our model final version 2. And then actually there's been some data drift or something. And there's still more to be done with this model. And now we have model final version 2 actually this time. We're ready to deploy.

Okay. So this lacks context. And it's not scalable for one model. So what happens when you have hundreds of models? This is not going to work. So when you're versioning a model, you're going to package up a few different things together. So you're going to include the model object. So the model itself. You're going to have a hash for a really more robust version than this. And then you're going to have some metadata to give this model context.

Managing change in models is what versioning is all about. It's important to think about versioning across time. Because if you have a model version 1, version 2, version 3, you can imagine how the entire team of data scientists looks. It's also important if you have different implementations. Say a staging and a production server. Really, any time you have multiple models, some sort of versioning is going to help you organize this. It's going to make it easier for yourself. And it's going to make it easier for you to communicate and share these models with the rest of your team.

So in Vetiver, we have a secret weapon called pins. This is another package developed by my team. And it's all about versioning. So for pins, you kind of have to think about it as a board. And you're going to take these models and you're going to pin them on the board. So we have our model board set up. It's a temporary board. We have to allow explicitly for pickle models to be read. It's because of arbitrary code. If that last sentence was like, why are you talking about pickles right now, don't worry about it. So our model board is ready. And now we're going to make a Vetiver model.

And this is because when you're training a model, you actually have a lot of context that you can put together into one object. So we're putting our pipeline in here. We're going to give it a name called ads. And we're going to give it a input data prototype called X-train. Or it's just like the X-train data. And this input data prototype is important because like we saw from Toast, sometimes the real world data looks different than what we trained with. And we want this deployable model object of Vetiver to be aware of what the original data is supposed to look like so you can get better errors when things go wrong.

So once this Vetiver model has been created, you're going to pin write. You have your model board. You're going to write this pin onto it. And I promised that this works in R as well. And I think it's important to show you guys kind of how similar this code looks. So in R, you have your libraries of Vetiver and pins. You have your temporary board, your Vetiver model, and your Vetiver pin writing this model to the board. We worked really hard to make sure that each of these packages feels good in the native language. But that it's also easy for teams that are multilingual. If data scientists need to hop between languages, you don't have to learn an entirely new framework.

So I promised you some metadata. Here's some metadata. From just these few lines of code, we have gathered a lot of information here. You can see there's a title. The model's name is ads. It's a pipeline object. It's from scikit-learn. We can see the size of the model. We can see a hash. The very specific version. And we can see that user ptype right there for what the data should look like. We can also see what the required packages are to run this model.

So we were talking about, you know, collaborating with your teammates. And I just showed you a board that is temporary. And actually when you run it at the end of the lines of code, the board will be deleted. So where else can you put these boards? We started with our temporary board. But you could also have a local board. Maybe you just wanted it kind of on a local machine. Or if you wanted to put this on prem, maybe a local board is the right move. But it gets more interesting. So you can also just switch out board to S3. And you can host this on Amazon Web Services. You can host it on Google Cloud Storage. You can pin models to Microsoft Azure Blob Storage as well. You might have to add a few more arguments to, like, give the right credentials. But everything looks and feels the same.

However, when these models are up in Azure or S3 or wherever, you still have to pull them down, load the model, get your data, like put it in the model, make the prediction. And it feels clunky. So we need a better way than just saving models, although this is an important place to start. So we want to deploy the models as well.

Deploying models

And deploying just means putting a model in production, which is kind of a loaded sentence because what is production? It looks very different for different people. But in general, we're going to say that deploying a model is anywhere that is not on your local laptop. This could be inside of an application. This can be just kind of living somewhere or anywhere in between.

So the idea of data science workflows are directly opposite of software engineering best practices. Software engineers are very interested in reusable, testable, like very strongly put together, like super robust deployments. Like they want to be certain that this is going to act the way that they expect it to. And data scientists are a little more experimental. We want to move fast. We want to make lots of different models. We want to be agile when data changes. So there's a little bit of tension there.

When we're thinking of making an application, if you told your software engineers that actually whenever I update my model, we're going to copy and paste the code from my Jupyter Notebook into the application, and it's going to go great, they might kind of look at you funky. We don't want to be copying and pasting code between applications, and we don't want to be emailing model files to each other. So we're going to use something called a REST API. A REST API is just a simple way to have machines communicate with each other. It's standardized, and they're useful because these endpoints can be hosted in computational environments that are completely independent of an application.

So this makes software engineers happy because your endpoint will stay the same even as you update your model, and so the application code won't change, and you can kind of live in your world of your own API if you are a data scientist. And these APIs are also testable, and they are pretty easy to put together.

So we have our Vetiver model from before, still called V. And to set this model up locally in our API, we'll have vetiver-api-v.run. This is built off of FastAPI, which is a very popular framework, similar to Flask, but a little bit more lightweight. So when you're running your application or your API endpoint, you'll get a browser window that looks kind of like this. You can see at the top it's 127.0.0.1. It's on my local machine. This is all a self-documenting API. So it has the name of the model here. It has the pipeline class. We can see where the server is hosted. But if you scroll down, you'll see this predict endpoint. And this predict endpoint is kind of fun. You can interact with it to make sure the model in your API endpoint is behaving as expected. We can see what happens if the model is funny and has animals. We can also look at the response headers and curl to interact with this model, which is useful if you need to pass these off to somebody else on your team.

But this is still local. 127.0.0.1, that's my local machine. So we want to deploy it somewhere else. There is a one-liner for RStudio's pro product. It is deploy-rs-connect. You give it the connect server. You give it the board, the pin name, and the version you are deploying. And these deployments are strongly versioned. So why versioning is so important is if we want to update a model or roll back, we just have to change that version there. So in this one-liner, our model can be on connect.

So this is what it looks like before on my local machine. You can see at the top, 127.0.0.1 is the URL. And here's what it looks like on connect. And it's kind of funny because I can click back and forth and pretty much it looks the exact same. But this is an important piece. Things break very quickly when your development environment looks nothing like what your production environment looks like. So this is really a nod to the reproducibility and stability of APIs. If I test it locally and deploy it, it should look and feel the same, even if it's hosted in a different location.

But I understand that not everyone is a connect user. So I had to build different ways for people to deploy this to other locations. So if you do vetiver write app, give it the board and the pin name, you can also add the version. It will write an app.py file that has all the information to create this API. And a lot of cloud providers will take this app.py file and they'll run it for you so you can bring this to different cloud environments. And for some clouds, this is enough. But others want a Docker file.

So if you've written your app.py file, you can also write a Docker file that's linked to this app.py file, and it will have a Docker file generated that essentially will work out of the box. This is really nice because it makes kind of that deployment struggle a little easier for data scientists to have these tools in hand. You can bring your Docker file and send it off. A lot of cloud providers have a bring-your-own-container platform. And we're also working on specialized Docker containers for different products inside cloud environments, such as AWS Lambda. So in the future, if you look at, like, write Docker Lambda, that might be something we'll be able to have for you.

Monitoring models

So your model is deployed. But a data scientist's work is not done yet. You have to do continuous monitoring to make sure that this model continues to perform the way you expect it to. And that's because data drifts. People's preferences change. If we had a model deployed that was trained on the yellow circles, our model works pretty well. But if preferences change and now we're looking at these purple triangles, our model is not doing very well anymore.

And this is important to track because if your model's performances start decaying, there might not be an error. This is another one of these things where we have to think about software engineering and data scientists. When software engineers are making applications, most of the time, when things fail, they fail very loudly. You get read errors. You get calls at 3 a.m., like, why is everything broken? But models fail silently. They can still run forever and ever with no error, even if your accuracy is 0%. And we keep giving you predictions even if they're horrible. So if you're not monitoring your model in some way, you are oblivious to the decay that may be occurring.

But models fail silently. They can still run forever and ever with no error, even if your accuracy is 0%. And we keep giving you predictions even if they're horrible. So if you're not monitoring your model in some way, you are oblivious to the decay that may be occurring.

So we'll set the scene. Our YouTube-like predictor has been in production for years now. We are really happy with its performance, but we want to keep monitoring and making sure that this is working the way we expect it to. So we're going to start with a few things. We have new data, and this is a data frame that has been collected, however, maybe through data engineering pipelines. And it has a few columns that we have to give to Vetiver, the Vetiver compute metrics function.

And the first column name that we have to say is the name of the date. So this is a very specific flavor of date. A lot of times, models will use time as an input to the model. However, this is actually not the input data that you are predicting on. This is the date that the prediction was made. We also need to know the true value of the like count as well as the predicted value. So we can compare these two to see if our model is still performing well. There are a few other things, like the time delta. So this is how we are aggregating this time. So this is looking at every one week, how is this data performing? And we're going to give it a metric set. Here we have mean absolute error and mean squared error from the scikit-learn metrics. These could just as well be custom metrics as long as they have a y true and y predicted value.

So new data, we're going to have predicted our time delta and our metric set. And then we're going to plot these values. And this kind of has a super simple pipeline. The output of the vetiver compute metrics can go directly into plotting. But it is just a plotly graph. The beauty of open source is that it can be very, very flexible. So we can update the y-axis, we can change the colors, or do whatever else we need to make sure that this graph looks to our liking. So when we show this graph, it'll look something like this. And we can kind of hover over things, see how it's been performing in 2014, 2015. I wouldn't place any bets on how well this model is working. But here's our monitoring so that we know that we can't place any bets, that it's not working.

On the R side, there's also helper functions to get you a template of a dashboard. So this has three tabs that we're going through. The first one is that same line graph that we saw earlier. The second one is a placeholder for a custom plot. So whatever you want to be looking at when you're monitoring your model, you can just continue to add these tabs. Because this dashboard is actually just R code. But as flexible as you can program your R script. The last tab is API visual documentation. It's the same things that we saw a few slides ago.

So this is just to kind of ground people. You know what model you're using, you know where it's living, and you can interact with it in this dashboard.

So what happens when things go wrong? We were monitoring for when things go wrong, but like, oh no, it's happened now. What do we do? The first step is almost always to retrain your model. If your data has drifted and you've chosen kind of the appropriate data or the appropriate model, the data structure kind of looks the same. Retraining, most of the time, works very well and you can continue on with your life. However, sometimes you need to try a new model type. So this is important if things have changed, retraining isn't working, here's another plan of action. However, you always have to remember to version. Because these are strongly versioned APIs, if we want to roll back or move forward in case, you know, performance goes bad again, we need to go back to a different model. This is really important for that stability and those good software engineering practices that we want to integrate into our data science workflow.

Model cards and ethics

So MLOps is thinking about making good models. Now, of course, we want to have statistically good models. That's hopefully a given. But we also want to have models that are giving a nod to ethics. There's no surefire way to avoid bias. But thinking about it is a good place to start. This is a space that my team has been actively investing in and trying to figure out how we can make tools that really give data scientists best practices in not only a statistical sense, but an ethical one.

So we saw this earlier. So vetiver, pinwrite, we're writing this pin to our board. But what I actually had hidden is the fact that this pops up. It says model cards are a framework for transparent, responsible reporting. And it says to use this function if you want to use this Quarto template. So model cards were created from a team at Google. I think they left Google. And they realized that there is a more holistic view of models that maybe isn't captured oftentimes.

So if you run this vetiver model card, you get a partially automated Quarto template that's similar to or same thing as what my slides are. And Quarto is a technical documentation platform. It's pretty easy to install. It kind of feels like a Jupyter notebook, but it's multilingual in a different way. And it can make slides, which is fun.

So we're going to make this model card. And it's partially automated in the places where we were able to automate things. But it also gives a lot of space for data scientists to jot down their thoughts. So you can see I have... It's generated that we have a scikit-learn pipeline model. It's got four features. We can see the time the version was created. And you can put in your email and stuff like that.

If you scroll down further, you can see that there's stuff on training data and evaluation data. We have a printout of what the p-type looks like. This is for a different model. But you can see kind of the title, the default, the type. And we can also look at quantitative analyses. So we'll look at the model performance. We'll look at the disaggregated model performance. So how does this model perform on different groups? We can also visualize the model performance, which is out of the box. There's also a space for you to put custom plots. Once again, this is just code. So you're able to edit this and change things up as much as you need to.

If you scroll all the way down to the bottom, there's a very important piece. And it says ethical considerations and caveats and recommendations. So our model for predicting likes on YouTube, we might not think that it has any ethical challenges. But I'll give two pieces of advice to that thought. One is you should probably check with the people that your model is impacting, especially marginalized groups where this might be impacting differently than the general population. And even if you have done all of the surveys, you've done your due diligence, and you find that there are really no ethical considerations, I would recommend that you don't delete this section.

Even if you have imprecise or imperfect information, just writing down your thought process or what you've thought about for the ethical considerations and recommendations, it's important. So the processes of documenting, they're limited by the data scientist, so this is really only as useful as you want to make it. But I think, so my dad has a quote. He's been telling me this all my life, and I'm going to share it with you guys because I think it's important here. If you haven't written it down, you haven't thought it out. Model cards are a great way to collect your thoughts, to think about your model in a really holistic way that I think is really important.

If you haven't written it down, you haven't thought it out.

Choosing MLOps tools

So from the ground up, what is MLOps? I hope by now you realize that it's a set of practices, and we've gone over a few practices here. So what do we have to keep in mind when looking at tools?

So we're going to fast-forward a week from the conference. We're also going to be looking at my speaker notes right here. So we're going to need a few tools, and we're all scientists, and we're going to do our good research. And our good research is going to include a Google search for what does the MLOps landscape look like. And you're going to get something like this. And you're going to realize this landscape is crazy, and there's no one-size-fits-all tool for what your team needs. I've shown you Vetiver, but there are lots of other options.

So I make data science tools, but I've also thought a lot about what do other tools do and what's important to keep in mind. The first thing is that you and your team are unique. So I've talked to a lot of different data scientists and customers, and for some teams, just having a pinned model kind of up in a cloud somewhere that helps people collaborate with each other is amazing. And that's the ML practices that that team needed. For other people, it's deploying a model, and it's monitoring that model, and that works for them. And there's a great span of things between these two. So think about what your team needs.

You also might want to think about looking for tools that are composable. So you want to have tools that are working in different environments. I showed you Vetiver will work in a local environment. It'll work in RStudio Connect. It'll work in a variety of public clouds. And we also want this to work well with other tools. So if we have maybe different open-source tools that are out there, we want this to be composable to put these pieces together. I think a good example of this is Pandas and Scikit-learn do this amazingly. Pandas and Scikit-learn both use this data structure called a data frame. And it kind of just makes sense. You think about it, and you're like, oh, yeah, like I just put my data frame in Scikit-learn, and my model is trained. But think for just a moment how insane the Python data science ecosystem would look like if those two packages did not agree on a common data structure.

In the same way, you can use Vetiver alone to do the tasks of versioning, deploying, and monitoring, but you can add in other tools that are out there that help you build the system that you need. It's also important to have tools that are composable with themselves. We looked at a pretty straightforward MLOps use case today, but you can actually layer and add in different pieces if you want an API with multiple endpoints, if you want an API that does post-processing. You can put Vetiver together with itself, kind of like Lego blocks, to create a very complex system.

Reproducibility is important. And I can point to about, well, any textbook about machine learning, and they'll tell you that it's important that your machine learning model is reproducible, and the same goes for MLOps. But I do want to highlight the reproducibility of open source. So we saw the landscape slide a few moments ago, and there's a lot of options out in the MLOps space today. From a business perspective, open source is important because open source is forever. It will outlast all of us as long as the Internet is here, also because it's free and companies love that. But this is important that the lifetime of an open source project is so long because even if the company that supports it changes, you can still use the same code.

From a human perspective, open source is for everyone. Once again, it's free. And second, this is a great equalizer for people who need it. As long as you have Internet connection and the courage to open an issue or connect with people, this is an amazing space for people to build skills in high-tech areas such as machine learning operations.

And finally, you should be looking at tools that are ergonomic. Your need here is that you are ready to implement some MLOps practices. Your tool is going to help you do that by some magic of engineering, but we are humans and we have finite capabilities, and we need a tool that feels natural to us to complete this task. Your tool needs to fit the person, and people need to have a tool that feels good to use.

Vetiver is a tool that was made for data scientists in mind. It should feel like a natural extension of the workflow you're already using in Python or R. But not all tools are built for data scientists, and I think that's a good thing. So I think it's important for people to realize that maybe you need to look for a tool that works well for who you are and the persona that you bring, whether that's a software engineer, whether that's a data scientist, or a data engineer. And of course, this goes for any tools, not just machine learning operations tools. And you also want a tool that can do the tasks that you want. It is a broad space, and it covers a lot of ground. And there's different sections that different tools are highlighting. Some MLOps tools are just focused on the pipelines to get data from place to place. Other tools are important because they want to think about experiment tracking. So when you're tuning your hyperparameters, logging every single one of those. So it's important that you have the tools that do the things that you want to do.

So we understand what MLOps is, and we know what to keep in mind for tooling. And as a quick recap, there are practices such as versioning, deploying, and monitoring. We're keeping models in production reliably and efficiently, hoping to lower the cognitive load of people's data science full life cycle. Vetiver can help you with this for R and Python. If you're interested in learning more, you can check out the documentation at vetiver.rstudio.com. You can click on this link in the slides. There is a recent screencast by my colleague, Julia Silke. She deploys a model with Docker in this YouTube video. There's also end-to-end demos from our solutions engineering teams in both R and Python. These links are to the GitHub repo. You can download them, you can run them yourselves, and really just kind of learn about what this space looks like. Also, you can visit these slides. Here is the link to these slides. And I hope you guys all learned something and that you can walk away feeling like you have some of the vocabulary and knowledge to start these conversations about MLOps. Thank you.

Q&A

Thank you so much, Isabel. Great way to start crunch day two. And Slido has been blowing up with questions for you. Let's take the most upvoted one. How do you see Vetiver versus MLflow?

I love this question. So MLflow is another machine learning operations framework. And it is focused kind of in a little bit different ways. So MLflow has invested a lot of knowledge for experiment tracking. And they do such a great job, and it's super fun to play around with. So when you're training your model, it will log every single hyperparameter. I think the difference here is MLflow has put a lot of emphasis on this part of the data science life cycle. And Vetiver is interested in kind of helping the deployment side. So it's only a few lines of code to get from your model to an API endpoint. And I think that's the difference. Vetiver is focused maybe more on deploying, where MLflow has really put a lot of emphasis on experiment tracking.

Great. Thank you. Our next question. What do you think about versioning Jupyter Notebook? How to do it right when trying to develop new model interactively but still have the history?

So Vetiver itself is not necessarily interested in versioning the whole Jupyter Notebook. I'm not quite sure of a reason why you'd want to version the whole thing besides maybe pieces of it. You want to version the data if you're training with data. You want to version your model. But the Jupyter Notebook is kind of just like the means to an end. So I would say to do this right, take out the pieces that need to be versioned, such as data or models, and put them somewhere else that's not inside a Jupyter Notebook. That's my answer for you. Feel free to reach out if that's not the question you're asking, though.

All right. Our next one. It says, no question. This is just an awesome talk. Thank you.

And now the next one. What's the next trick for Toast? So I'll nerd out for a moment here. Toast knows a lot of tricks. He's actually certified as a trick dog now. Right now we're working on having him clean up after himself. So I'll get him, and he can go pick up toys. And now I want to have him go pick them up, bring them somewhere else, and drop them