Kevin Kuo | Introducing mlflow | RStudio (2019)
We introduce the R API for MLflow, which is an open source platform for managing the machine learning lifecycle. We demonstrate each component of the platform–Tracking, Projects, and Models–and describe how they can be leveraged in practical data science workflows. About the Author Kevin Kuo Kevin is a software engineer working on open source packages for big data analytics and machine learning. He has held data science positions in a variety of industries and was a credentialed actuary. He likes mixing cocktails and studying about wine
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
All right, thank you guys for coming to this session and I'm so glad that Javier was the one that ran into AV issues and not me.
So one of the things I learned from giving talks is that there's a very strong positive correlation between the number of memes and pictures you put in your slides and how happy the audience is, you know, from the session reviews I've been seeing. So I'm going to just put a bunch of pictures on here, I won't do too much talking, but at the end I'm going to do some demos because you do need some content in a tech talk.
So we're going to be talking about mlflow and it's an open source platform for managing the machine learning lifecycle, whatever that means, we're going to find out in a little bit and it's a community driven project mostly led by the folks at Databricks with RStudio providing the R interface.
Motivation for MLflow
So let's talk a little bit about the motivation for a project like this. One of the things that data scientists have trouble with sometimes is that you don't remember every model that you tried. So let's say you're building a neural net, you've got a few layers and maybe a residual connection here and there, and you stumble across some hyperparameters that seem promising, so you sort of go into that direction and tweak some numbers, and then, you know, maybe like an hour later you realize that, you know, the performance isn't great, so you want to sort of roll back to the previous version, but then you were so excited about your results you didn't bother, you know, pushing any of your, committing any of your code, so you don't remember what hyperparameters you used.
So, you know, chuckle, please chuckle.
So we're going to talk about how mlflow might, you know, help with that situation by logging your metrics and your hyperparameters. And another thing that sometimes people have issue with is replicating your results, and you might want to reproduce your teammates' numbers or you might just want to reproduce your own numbers from yesterday or last week or last quarter, and sometimes, you know, the code will run and then you'll get some maybe different results, maybe due to some, you know, random initialization of ways or whatnot, but sometimes it just doesn't work, you know, missing libraries, mismatched architectures, and not every data science project ends with a PowerPoint deck and a steering committee meeting.
Sometimes you actually have to take the models to production, and, you know, it's hard to find common ground, depending on where you are, between the folks who prototype the models and the folks who actually need to take the models to production, and this is exacerbated by the fact that there are just so many different machine learning libraries out there, and there are also different deployment targets, right? So there are different libraries you can use to train the models, and then there are different ways to sort of deploy these models. So if you got, you know, the number of ways you can sort of combine the prototyping and the deployment just expands exponentially.
So if you are at some specific companies, mostly out there in Silicon Valley, you might say, oh, you know, we got this all solved, but then what about everyone else? What do you do? So you'll go online, you'll do some Google searches, you'll be like, oh, what's everyone else doing? And then you'll find that, especially for model deployment, there are just so many different things out there, and it's kind of tough.
So over the past couple years, there's been a lot of efforts in the machine learning community and also within the art community to try to solve these problems, or at least attempt to solve these problems. And I personally think that we won't ever come up with one framework that's just going to work for everyone, but I think we can try to segment the problems such that you can have a set of standard practices that sort of will apply to most people.
And I personally think that we won't ever come up with one framework that's just going to work for everyone, but I think we can try to segment the problems such that you can have a set of standard practices that sort of will apply to most people.
MLflow components
So back to the topic, MLflow, it's got three different components. It's got tracking, which is going to help you keep track of your hyperparameters, some nodes and some metrics from your experiments. The project component bundles your project environment so others can reproduce your results. And the model component allows you to serialize and package your model so you can deploy them. So of course, you know, this doesn't make any sense, so we're going to do some demos to show what this thing actually looks like.
Demo: tracking
So we can load a library. By the way, this is on CRAN, so you can just install it off of CRAN. And I'm just going to do something really, really general right now. So we can say MLflow log param, and then let's say we want to log the food param, and then the value happens to be 42. So what is this doing in that background is it's basically launching a server and starting an experiment and actually logging, you know, this parameter.
So this is like the default view of the MLflow interface. So we can go into our experiment run here, and then we see that our parameter field has been logged. So the cool thing is that this is not only for machine learning. If you have other experiments that you want to keep track of, you can also use this mechanism.
So let's say now we want to log a metric. We come back here. We see that we have that metric that's been recorded. And there are other things you can do. There's tags. There are notes. You can write whatever you want. So that's sort of the tracking functionality.
Demo: model logging and serving
And we can now go to model. So Javier wrote a little XGBoost model to try to classify Iris, and then what we have here is a model to try to solve that same problem just using Keras and TensorFlow. So I'm going to run this real quick. So let's go through the code, standardizing values, normalizing values, and then here we can oh, actually, one thing I forgot to mention.
When we log this parameter right here, we started an experiment run, and then after you're done with your experiment run, you need to end the run, and then once we do that, you can see that there's also a duration associated with your experiment run. So that might be helpful when you're trying to compare different ways to train the models. You might spend 10 hours to just get a little bit of accuracy, and that's not worth it.
We can also use this context manager with the width function, and we can train this very simple neural net for a few epochs and see what happens. So you can see real time that the loss is going down, yeah, and the accuracy is going up, so we're doing great.
And then we can go back to our UI, and we see that we have our neural net training run that has also been logged, and because we added a callback in the Keras model fit call, we can see that MLflow has also logged the accuracy for each epoch.
So one other thing we did was we saved the model as an artifact. So you can basically save anything you want and associate it with an experiment. So what we can do right now is we can attempt to serve this model locally just using a web server, so using this function MLflow rfunc serve, and our model directory is the Keras model because that's where we logged it, and then we can provide the run identifier. So we'll do that, and then we got a model that's being served right now. It's port 8090, and then I got this prepared. So we can basically post a bunch of predictor data to this predict endpoint, and then see if we get some results back. So these are like the probabilities of each class.
Demo: running projects from GitHub
You can also run projects from GitHub. So let's just do this real quick, and I can show what's going on there. So basically what this is doing is it's pulling code from GitHub and it's spinning up many environments to execute this code, and then it's going to log the results to a training run, and if we actually go to this URL and our entry point was this trend.r file, we see that we're using a random forest classifier now using Spark to try to classify the flowers. So we see that the run has succeeded.
So we can go back to the available flow interface, and we see that here we have the parameters that were passed to the project run and then also some metrics associated with it, and the cool thing about this is that it records the exact commits on which this code was pulled from, so you can actually, you know, click on this to navigate a repository at that specific point in time.
the cool thing about this is that it records the exact commits on which this code was pulled from, so you can actually, you know, click on this to navigate a repository at that specific point in time.
So I think that's all I have to show you guys, so, you know, definitely think about what sort of applications you can apply this to in your own endeavors.
Q&A
So I think do we have time for questions? Yep, we have some time for questions, if anybody has some.
Kevin, one of the things I was wondering about, do you have to use the MLflow interface in order to retrieve the metrics which are recorded, or is it possible to export it to, you know, a flat file or some kind of log?
Yeah, there's an API that you can use that's currently not exported yet in the R package, but there's a wrapper on the REST interface that you can use to potentially do that. So you don't have to use the interface if you don't want to, and then you can also build your own interface to suit your needs.
What is the state of collaboration between Python and R? So can I have an experiment that has parts of its code in Python, the other part in R, and track them in the same experiment on MLflow?
Yes, yes. So here I demonstrated, for example, this call to pull some code from GitHub, this did not have to be R code, it could have been Python code. And if we actually go back to this run right here, you'll see that there's a run command that's basically a command line command that anyone can use to reproduce your results. So if you're a data scientist working in R, anyone else, even if they have never seen a single line of R code, they can, you know, try to put different hyperparameters into your model and get some results back. So we basically try to abstract away the implementation of the model from the people who want to use the models.
I saw in some of the code that you were making calls to the recipes package, maybe? Is that integrated into MLflow in any way?
So the cool thing about MLflow is that, like in the beginning, I was just logging random stuff. So if you wanted to record specifics about your data preprocessing step, you can definitely do that. So there's really no limit on, like, what you can sort of put in there. So we logged an entire Keras model in there, too. So you can put, like, pictures and, you know, audio file if you want.
