Winston Chang - Building ML and AI apps with Shiny for Python
In the last few months, the best LLMs have taken a big step forward in their ability to help write code. We’ve been building a tool that uses an LLM to help you learn Shiny and build Shiny applications much faster than ever before. If you feel like you don’t have the time to invest in learning Shiny, this might change your mind. You’ll get up and running with Shiny in no time, and you’ll be able to learn from the AI as you go. If you already know Shiny, you’ll be able to dramatically accelerate your development process, and you’ll find yourself trying out new things that wouldn’t have been worth the effort before. Talk by Winston Chang
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Thanks, everybody. So, as we all know, there's a lot of hype around generative AI and LLMs. There's a lot of skepticism and anti-hype as well. Now, I've been playing with these for a while now, and I've seen them, you know, continue to iteratively improve, and I think they've gotten to a point where they can be really helpful for creating things, for helping you learn things, and for making your work more fun by taking some of the tedium out of it.
So, I want to show something today that we've just, um, it just came into existence like less than a month ago. So, it's really new, but it's been a lot of fun, and let me show you what it is. So, it's an assistant for working with Shiny, and we're calling it Shiny Assistant for now, because for lack of a better name. Hopefully, maybe we'll come up with something better in the future. So, this is what it looks like. It's a chat interface. This is built, again, with this chat component that we have currently in Shiny for Python, but it's coming to Shiny for R.
Shiny Assistant demo: answering questions and generating apps
And there's all this text here, which I will not bore you by reading, but instead, I'll just show you what it can do. So, here's one thing you can do. You can ask it a question. In this case, we're going to say, how do I modify my app so that a plot updates only when a button is clicked? Pretty common question. It's going to think. It will answer. This is Shiny for R. It's going to talk about action buttons and eventReactive. I won't go into detail about all that stuff, but it will show you an example application here, which you can, if you've spent the time reading this, you can learn something from this and maybe try out this app. So, that's one kind of question you can ask it.
Another kind of question is this. So, create an app that demonstrates a linear regression. So, it's going to think. It's going to write out some stuff. It's going to generate some app code. And then, you'll notice on the right, if you've used Shiny Live before, you've seen this. This is the Shiny Live interface. So, it generates the code. It loads the code into Shiny Live. And then, it runs that app. Now, we can interact with this a little bit to make sure it's working. Click that button to generate new data. And it's generated some random data. Well, semi-random data. And it's done a linear regression on it.
Now, after you do this, you can twiddle around with these. You can twiddle these controls a little bit. There we go. Just to make sure that it works. And there it goes. It's updating the data and the linear regression for it. Okay. So, this is the first step. But you might be looking at this and you're like, well, this data doesn't look very realistic. It fits the line a little bit too well. It's too evenly distributed. And if you have opinions on this, you can go and ask the Shiny Assistant to make some changes. So, here we'll say, increase the noise by four times. Make the data more normally distributed along x. And increase the number of points by two times.
So, it's going to go and generate some code, update that code. And here's the app. You might actually have noticed that these numbers are larger than what you'd expect by those multiplication factors. That's because I cut out some more iterative steps here in the video. So, don't be too surprised by that. And if we click on generate new data, now we have this app with updated data that looks a little bit more realistic. And we can keep going. So, let's ask it to do more stuff. Add residual lines to the plot. Add a slider to control the slope of the data. And it will update the code. Load that in the editor and it'll run the app. But this time, if we click on the button, there's an error that pops up. So, the Shiny Assistant is not perfect. It doesn't do everything perfectly. And so, sometimes you have to deal with errors like this.
Now, in this particular case, the error is that it couldn't find the pipe. That's the Magruder pipe. And so, you know, I happen to know how to fix this. We can use the base R pipe. That was added in R 4.1. So, we'll just replace that Magruder pipe with the base R pipe. Rerun it. This is Shiny live, so we can just click that button and it'll reload the app. And then click generate new data. And still, the plot didn't come up. And if we look down below, we'll see that there is another error. So, I could go and fix this. I know what the, you know, I don't know what the solution is here. It says error and mutate cannot find function mutate. You probably do too. But if it's an error that you're not familiar with, I'll show you what you can do. You can select the text and just ask Shiny Assistant, like, hey, I got this error. What could be causing it? And it might just answer your question. Or in this particular case, it decided to, you know, update the app and fix the problem, which is that it needed to load dplyr. All right, now when you click it, generate new data, we have the residual lines, and the bug is gone.
Okay, now one other thing that you can ask it, that I'm going to ask it to do here, which is to make the summary look a little bit nicer. This is just the raw output from if you call summary of a LM object. And let's make it look a little bit nicer. So for the summary, instead of displaying plain text output, display the important information and a nicely formatted card. Then go and update the code. And then when it's all done, that card there looks a lot nicer than the raw text output.
Okay, so this is, you know, like, I could have done any of these things individually, but it would have taken a lot longer, right? This is all sort of bread and butter, our programming stuff, doing a linear model, writing a Shiny app with sliders, and so on. Okay, and then after you make something interesting with this, because it's Shiny live, you can just click on the share button. And then you get these links that these URLs that you can copy and paste, and then somebody else can see the app. And if you weren't already aware of this, the whole entire app is encoded in this URL, which is much longer than what you can see here. But that's the whole app.
How Shiny Assistant works
Now, if you look at Shiny Assistant interface here, there's really two important parts. There's the Assistant, which is the chatbot. And then there's Shiny Live. And these two things working together, that's, you know, that together to make the whole Shiny Assistant. For the Shiny Assistant part, it's really useful to think of it really as an assistant. So you can ask it to do anything, and it'll try to do it, but it won't be able to successfully do everything. You can think of it as a helper that knows a lot, and it works very quickly and tirelessly. But it makes mistakes, and it has a limited understanding of complex projects.
It can get you started on a project really quickly. So even if it doesn't give you a finished product, it can give you a scaffold on which to build. And it can answer questions about your app. You know, if it's got questions about how to do something in Shiny, or like, why is this code written this way? It can answer those questions. But, you know, you still need domain knowledge of the problem. You still need to verify and test the code, and you still need to get involved with debugging. So it's here to help you, but not replace you.
You still need domain knowledge of the problem. You still need to verify and test the code, and you still need to get involved with debugging. So it's here to help you, but not replace you.
The other component is Shiny Live. And I'm just going to go over this real quickly. So it's Shiny for R, Python compiled to WebAssembly running in the browser. So the traditional model for a Shiny app deployment is you have R or Python running on a server, and then the client computers connect to the server, but most of the code is running on the server there. With a Shiny Live deployment, the server is dumb, and it just sends out some files, and then you have R or Python running in the browser. So that has some implications for security. So when R and Python is running in the browser, it's in the browser security sandbox, which is very safe. You know, it's been battle-tested on billions of, literally billions of devices over the years. It can't access local files on your computer and has restricted network connectivity. So it makes it pretty safe to run all the code that the LLM generates.
You can't run all packages, because not all packages can compile to WebAssembly, but more and more are coming all the time. And there's no secrets from the user. So with Shiny Live, you know, the code, any code that you have in the data is sent to the user's computer, so they can, you know, they can look at that. So you don't have any secrets from them. And finally, the advantage is that sharing, one advantage is sharing, is that sharing Shiny Live applications is really easy.
Building a RAG app with Shiny Assistant
Now the example that we had before was with R. Shiny Assistant can also write Shiny for Python apps. So there's a little switch in the upper left, and if you notice that, you can just switch it over there, and it'll do Python. Okay, so for the next example that I want to show, we're going to use Shiny for Python. And instead of being this sort of like self-contained application with, you know, the synthetic data, we can actually make an app that talks to the world.
So this is a website that I learned about last week. It's called embedding.io. I don't really know much about it, but it sounded interesting. You can turn any website into a knowledge base for LLM. So it'll take the text from a website and break it up into chunks, and put those text chunks in a vector database. And each chunk will have a vector associated with it, which somehow captures the semantic content of that piece of text. And the purpose of this is that you can ask a question, and that will have a vector encoding as well. And you can pick out chunks of text that are semantically related to that question.
All right, so I thought, okay, this sounds interesting. I'd like to play with it. And on their API documentation, they have this example here. This is how you use it. So it's a curl request. And this is for the writings of, I think this is for Paul Graham's blog. And Paul Graham is the founder of Y Combinator. This is how you interact with it, which is, you know, like I could interact with it this way, but that's kind of awkward and a little bit not very fun to play with. So I thought I'd ask the Shiny assistant to build an app to do this.
Okay, so here's what I'd said. Create an app which queries a web API and prints the result. The curl command is that. I just copied and pasted that. Okay, get the query as an input from the user. So it goes, write some code. And this is the app that it comes up with. And it pre-populated that input with the question that was already there, which is fine. I'll just go with that for now. And when we click the submit button, this is what comes back. This is a JSON array of objects. This is one object, that's one entry. There's another one down here, and there's a whole bunch of these. So that's a nicer way of interacting with it than, you know, via curl and the command line. So I could ask it other questions and it'll get different results for them.
But what if I want to display this in a way that's nicer looking, right? Rather than just like raw JSON. Well, one thing you might want to display is the URL, and the title, and blog post, and the content there. So here's one thing you can do with that. You can select that text and tell Shiny Assistant this. The response is JSON array with objects that look like this. Paste that in there. Okay, display each of these items as a card with the URL, title, and content. Now it's going to go and generate some more code. Run the app, and then when you click the button, the results are displayed there. And with each entry on its own card. And it even decided to link to the blog post, which is pretty cool. I didn't ask it to do that, but I thought that was a good idea.
All right, so now with these vector databases, one of the reasons they're so popular these days is because they all are used for what's called, what's, well, it's called RAG, or Retrieval Augmented Generation. So what it is, is you ask it a question, and they get all these chunks of text. And then you send those chunks of text to an LLM. And you say, you tell the LLM, hey, here's all this information. You know, now given that information, please answer my question.
So I thought, well, can I do that with this, with Shiny Assistant? And in order to do that, we need an LLM. So I decided to use OpenAI. The OpenAI, if you look at the OpenAI API documentation, it has this here. This is some Python code. And the interface is really nice. But unfortunately, this does not work in Shiny Live because some dependencies of the OpenAI package are not available for PyDyed. That's Python and WebAssembly. So I decided, OK, I'm not going to use this. Even though Shiny Assistant could write this code, it won't run in that nice, the Shiny Live interface. Instead, they have, they also have API documentation for using curl. And this is just like basically the raw HTTP request. So I decided to copy and paste this and pay attention here. There's also this thing here. It needs an API key from OpenAI.
All right. So you can take that and ask it, assemble all the contents we had before into a single string. Use that string as part of a prompt to an LLM and the user query has a question to ask about the information. OK, the curl command for the LLM is, paste that in there. And then we'll also tell it to get the API key as a password input from the user. So I don't want to send my API key to, to Claude here, which is what's being used as the backend. So I want to just get that as input from the user. So it goes and generates the code. I have to paste in my API key there, and then I submit. And then this is what it comes up with. So it has the AI response. This is the response from OpenAI. And, but it also includes all these other text chunks here, which is not really what I want. So with the AI response, it's also, there's a few other things that are kind of not right about it. It's truncating it. The response is in Markdown, but it's only rendering it as raw text.
So I can ask Shiny Assistant to do it a little bit differently. So only display the AI response, don't truncate it, and render it as Markdown. So it's going to go and update the code. And again, I'll have to paste in the API key. And when I submit it, this is the summarized, this is the summary that comes from OpenAI. So we just made a RAG app, and that was like in five minutes, which is pretty good.
So we just made a RAG app, and that was like in five minutes, which is pretty good.
Important details and caveats
All right. So there's some things to know about Shiny Assistant. These are important details. So it uses a Claude 3.5 Sonnet from Anthropic, which is, it's a really good model for code generation. It makes mistakes, and you're going to have to deal with those mistakes, and you're going to have to ask it to deal with them. The training data is a little bit behind the times. So for Python, it's not very good with the Shiny Express syntax, because that's relatively new. There's not as much training data for it. So it's better at Shiny for R than it is at Shiny for Python. And again, that probably has to do with there being more training data for Shiny for R.
In Shiny Live, Python is better for doing web requests. So for like talking to these web APIs, in R, it's more difficult to do. There's some workarounds. But I asked George Steg, and he told me that our support for web requests is coming in WebR.
Okay. The code in the editor panel is sent to the LLM, so don't put any secrets in there. Don't put any confidential data, like API keys, in the chat or in the code. If you need to use confidential data, ask the LLM to add a password input, like the one that I had in that last example.
Okay. And I just want to show a real quick some screenshots of some apps that I've made in the last couple of weeks with, along with some other people. I don't have time to describe all of them. But, you know, these apps, they all took I think they all took like 20 minutes or less. Some of them took like five minutes. Some of them took like 20 minutes. And they might not be fully polished products, but they are, you know, it's really satisfying to have an idea and turn it into something that's concrete and tangible in that amount of time. And then you can spend your time refining it afterwards. So I find that to be a lot of fun.
Okay. So a few things here. This is the URL for gallery.shinyapps.io. If you go there, actually, you know what? I'm sorry. I think that might not be enabled right now. I'll do it right after this talk. You'll be it'll direct you to a form that you can fill out. And then you'll be put on a waiting list. And then we'll try to get people off that waiting list pretty quickly.
We also have a Shiny contest, which you probably there's like these cards for it all over these seats. And you can use a Shiny assistant to do the Shiny contest. All right. So the last few details here. Shiny assistant is very new and experimental. We might log the chat sessions for research and model training in the future to keep that in mind. We're paying for the LLM usage, so you don't have to. But if you want to, you can bring your own Anthropic API key. And finally, you know, go forth, make some cool stuff, and share it and have fun with this. Thanks.