"Shiny: Data-centric web applications in Python" - Joe Cheng (PyBay 2023)
Joe Cheng https://pybay.com/speakers/#sz-speaker-7e5324de-3afa-4614-a498-562bd5eb9986 Shiny is a web framework that is designed to let you create data dashboards, interactive visualizations, and workflow apps in pure Python or R. Shiny doesn't require knowledge of HTML, CSS, and JavaScript, and lets you create data-centric applications in a fraction of the time and effort of traditional web stacks. Of course, Python already has several popular and high-quality options for creating data-centric web applications. So it's fair to ask what Shiny can offer the Python community. In this talk, I will introduce Shiny for Python and answer that question. I'll start with some basic demos that show how Shiny apps are constructed. Next, I'll explain Transparent Reactive Programming (TRP), which is the animating concept behind Shiny, and the reason it occupies such an interesting place on the ease-vs-power tradeoff frontier. Finally, I'll wrap up with additional demos that feature interesting functionality that is made trivial with TRP. This talk should be interesting to anyone who uses Python to analyze or visualize data, and does not require experience with Shiny or any other web frameworks. PyBay features the most influential speakers presenting the most crucial technologies to help beginners and seasoned developers alike get up-to-date quickly, in a single-track format. Whether you’re interested in web technologies, data, devops, Python internals, or performance, PyBay will help you stay on top of your game AND network with engineers at companies that are hiring! Working remotely and want to meet your teammates to boost team cohesiveness? Leverage the platform we’ve built. There are great talks, yummy food, fresh air, vitamin D... all the elements developers crave for these days. If there are talks that don’t interest your team, take the opportunity to talk to speakers, create your own team activities or book a tee-time at the adjacent miniature golf course! PyBay is the regional Python conference for the San Francisco Bay Area, bringing together Pythonistas from around the Bay Area and beyond. It is a volunteer-run organization dedicated to building a stronger Python community. PyBay offers deep-dive talks and networking opportunities that aim to enrich and empower the Python community. PyBay is part of BAPyA (Bay Area Python Association). BAPyA member organizations are the SF Python, Pyninsula, and BayPIGgies meetups. Produced by NDV: https://youtube.com/channel/UCQ7dFBzZGlBvtU2hCecsBBg?sub_confirmation=1 Sun Oct 8 18:30:00 2023 at Bungalo East
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Yeah, thanks. I'm here to talk about Shiny. My name is Joe Cheng. I am the CTO of Posit, which was formerly known as RStudio. And if you have heard of Shiny before, probably it's in the context of being an R package, and this is not that.
This is Shiny for Python. So just because I know it's come, like, some people have had this preconception, what it is not is a Python wrapper around the R version of Shiny. Ugh. Yeah, I wouldn't do that to you. It is a pure Python version of the ideas that are in Shiny.
This is an example of, like, a very simple Shiny for Python app. And this is a simulation of a three-body problem. So you have three bodies in space that are interacting, Earth, Moon, and Planet X, and you have all different settings for them, and you can press run simulation, and it will do a simulation using Astropy, and then draw the results on a matplotlib 3D scatter plot here.
So I can play with these settings, change the mass of Earth to be unrealistically low, and run it again and get a different result. So this is a pretty simple... Oh, wow. That's pretty crazy. So it's a good thing that's not what Earth weighs. So that's a super, super simple example, and we'll kind of ramp it up as we go.
What Shiny is designed for
Now, about Shiny, Shiny is designed primarily with data scientists in mind, and that means that no HTML, CSS, or JavaScript skills are required. We also use a very easy-to-use reactive programming model, and I'll talk a little bit more about what that means. So the goal is to be really approachable for people who have a data science background in Python.
However, I'm not a data scientist. I'm a software engineer. I've been doing web development since late 90s, and it would be really frustrating for me to work full-time on a web framework that felt very low as common denominator, or very sort of like a lot of guardrails everywhere. So if you do know HTML, CSS, and JavaScript, you can fully leverage those skills to customize, to extend Shiny, to make your apps more powerful.
And this same reactive programming model that's easy to get started with is also powerful, flexible, and deep. I mean, I've been programming with this model for 11 years, and just I am more and more appreciative of what it can do, and I'll show you some of those things.
Anatomy of a Shiny app
But before we do that, I'm going to show you sort of the high-level anatomy of a Shiny app. Now, I don't expect you to be able to ingest all that code. So for pretty much this whole talk, the goal is not like you don't have to squint and read every line of code. This is going to be a very kind of lean back kind of a talk when it comes to the code. So I want to point out here that there are just two main pieces to any Shiny app. There's the user interfaces, this first chunk of code, and what it does is it generates the look and feel of the app and sends that to the browser.
In this case, there's an input slider and an output plot, and that's what you see in this little thumbnail. So all this is is Python calls that generate HTML. That's all it is. So you don't need to know the HTML, but that's what it's doing.
And the second part you need is this server logic. This is Python code that runs while the user is interacting with the application. So if they move this slider, then this input.n value is going to change because that input slider, that first argument n. So whenever that input.n value changes, because this function called plot has the decorator on it called render plot, that's a signal to Shiny that you should go ahead and rerun this plot whenever something changes and automatically keep it up to date.
So you don't have to tell Shiny when to rerender the plot. You just use the values that you want, and this is reactive programming. Shiny will automatically take care of knowing when to run it. And then you just have one line to bring these two objects together and make an application. So this is probably the simplest Shiny app you'll ever see.
And this is how your workflow will be. You're running VS Code, and we have a Shiny for Python VS Code extension. So you just press this button here, and you can either run or debug the app, and it will run the app in this preview panel on the right, right within VS Code. You just hit save on the app on your Python file, and the application will automatically reload. So it's very quick to iterate.
A more complex app example
Okay, just a couple more things before we launch into some fun demos. So here's an example of kind of a bigger app. So that last one was kind of unrealistically small, so I could fit it all on the screen. Here's an example of one that I frankly made it a little more complicated than it needs to be. So this one has some markup, some markdown at the top that you can use to format text. This one uses matplotlib again. It has an embedded iPy widget that has an interactive map that you can use to show your location.
And what this application does is it uses Astropy to tell you if you're an astronomer and you want to look at some object in the night sky, you tell it what objects you're interested in looking at, where you are on Earth, and what the date is, and it'll tell you these are the best times based on your location and time.
And the way this UI code looks, you have all these different capabilities coming into play. You can use raw HTML tags like a heading here. You can use markdown. You can have a row column layout. You can have outputs. Again, the details are not very important because the thing that I want to point out is that if we zoom out and kind of like not even being able to read the code, but seeing the indentation, there's a shape to it, right?
Like there's two sort of chunks that are part of this UI layout. And that actually is mirrored or reflected by what our layout looks like when it renders in the browser. So there's two pieces of code and two pieces. And for that second one, that body, there's two columns within it. And the code also has this shape, has this indentation. So this is one of the things that I think, it's like a really beautiful property of Shiny that the hierarchy in your UI is reflected in the hierarchy of the indentation of your code. And as your application gets more and more complicated, this is a really nice way to be able to figure out the right place in the code where you want to make a change or check something out.
Live demos
Here's just an example of a pretty typical metrics dashboard. This one is pulling from a live updating database that's simulating some model scores. But you could imagine if you work in a corporation and you have BI dashboards, that's one thing that people really like Shiny for.
But honestly, that is a very important use case of Shiny, but I find it not very interesting for talk. So that's the last BI dashboard I'm going to show you.
Here is a bit more of a fun demo where the visualization you're looking at is Brownian motion. And this is an example that came from Plotly. And Brownian motion is like the motion of a particle on a surface. And I can press new data here, and it'll generate new random traces. And some of them are pretty interesting looking.
Some of them, it's a little hard to tell what's going on, right? Just because of the particular angle that we're looking at. Now, because this is Plotly, you can click and drag and move it around, and you can look at it like that. That's just a feature of Plotly. That's nothing Shiny is doing.
But to make this a little more interesting, I am leveraging the Google Media Pipe library to track the features of my hand. So using Shiny, I can calculate a normal vector that's coming out of my palm. And then I use that to inform the camera angle for this widget.
Now, the reason I think this is interesting and relevant is because this smooth tracking button. So when I came up with this demo, I was at home in my living room like, oh my gosh, this is so cool. And my son, who is my oldest, who is a freshman in college, computer science major, his reaction was, why is it so jittery?
And I didn't even notice, but he's right, right? It's jittery because my hand's a little jittery, sensors are jittery. But I thought it was a good question. After I got over the initial indignation, yeah, OK, what can we do about that? How can we take this very noisy signal and smooth it out? So we tried a couple of things. And what ended up working really nicely was this approach of just smoothing the data over the last five samples. Instead of showing the latest data, we show the mean of the last five.
And I'm not exaggerating. This took 10 minutes to write. And what resulted, that code is not a smoother for this application. It's a smoother for any reactive signal in Shiny. So any data that's coming in fast and noisy, the same 10 lines of code that are behind this app can be used to have the same smoothing effect.
It's a smoother for any reactive signal in Shiny. So any data that's coming in fast and noisy, the same 10 lines of code that are behind this app can be used to have the same smoothing effect.
So that's what I mean when I say that reactivity, it's this easy to get started with, but it also has these really deep and interesting properties about it. And to me, that never gets old.
Shiny vs. Streamlit
So when we came out with Shiny for Python, a lot of people out there are already using Shiny for R. So some people had this kind of a reaction. And other people said something like this. But it's just so easy to throw something up with Streamlit. How many people here have heard of Streamlit? Okay, plenty of you.
And we've gotten this question a lot since the introduction of Shiny for Python, because Streamlit is, as smilodon138 says, just so easy, right? So I wanted to just take a moment to unpack this claim a little bit. And the promise of Streamlit is that you can start with a normal script.py that's just a straight linear set of data analysis function calls. And with Streamlit, you can take exactly that script, sprinkle in a few Streamlit statements, and boom, you have an interactive application.
And even if you've never seen Shiny before this talk, you already know that Shiny doesn't do that, right? That Shiny has this form of a UI and a server and these little functions with decorators. It is asking you to pour your code into its form.
So the way Streamlit is able to do this, because that's super cool, right? Very nice to be able to just take your script and sprinkle in some Streamlit. And the way they're able to do this is this. Streamlit's architecture allows you to write apps the same way you write plain Python scripts. To unlock this, Streamlit apps have a unique data flow. Anytime something must be updated on the screen, Streamlit reruns your entire Python script from top to bottom. Nobody else does this. This is only Streamlit.
And it really is that easy. It is true. I'm not here to debunk Streamlit's ease of use. It is like you might learn Streamlit by accident. You could write a Streamlit app falling out of bed. It's that easy.
Now, granted, it's running the entire script every time you move a slider, every time you click a button. So as a result, by default, it does have slow performance. And there's only so much control you can have when it's running the entire script every time anything changes. You can't opt out of that behavior. So that means as your Streamlit apps get just a little bit more complicated, you are actually left in quite a bad place. So they have introduced some features like caching callbacks and session state that are designed to let you work around the top-to-bottom execution model.
So I can very much endorse Streamlit for simple apps. And I think they made some really fantastic, interesting choices. But for the complicated apps, you're putting yourself and your users in danger because it is very tricky to get that kind of application correct.
So in contrast, Shiny is easy, though not as easy as Streamlit. But it is scalable. And the history of Shiny is a history of people getting so much value out of this framework without being web developers, but building sometimes quite complex apps and getting huge results from those apps. And the kinds of stories I could tell you about the kinds of accomplishments people have made on their Shiny apps would, you know, gives me goosebumps just thinking about it.
So I would sum it up by saying the meantime from getting started to having something of value, it's Streamlit, it's incredible. Like within 10 seconds, you have an interactive plot. And with Shiny, maybe it takes you 60 seconds, okay?
But the meantime before you need to throw that app away, in Streamlit, even people who love Streamlit will generally agree that when you write a Streamlit app, you know you're writing a proof of concept. And that there is going to be, like some other team is going to have to rewrite it in something else for production. And Shiny's not like that. Like you don't go into Shiny expecting to throw it away.
Okay, I can't back this up. Like these are my feelings. I am trying to be unbiased here, but I don't have data. But this is how I genuinely feel about this.
Closing the gap with Streamlit: Shiny in Jupyter
Now, until this talk, I would just leave it at that and say like Streamlit's easier. But this talk is actually a little different. So a couple of weeks ago, my team and I, we sort of challenged ourselves to think like, could we close this gap? Like, are we okay letting Streamlit be so much easier than Shiny? And initially, like I was like, yeah, I'm fine with that. Like they have made some choices that are, they're quite severe trade-offs. And I'm happy with the trade-offs we've made. Until I heard several people tell me that they use Streamlit not just to build applications, but to do exploratory data analysis. And that was surprising to me. Like this is, that's how easy it is that they'd rather do these Streamlit interactions than stay in their notebook.
And I was like, oh, that's actually pretty cool. Like 10 seconds versus 60 seconds. That's a big difference if you're doing EDA, right? So the question we posed to ourselves is can we close this gap without sacrificing this? Like, can we make it as easy as Streamlit, easy enough to do EDA without forcing you to throw away your Shiny app?
So I have been working my butt off for the last two weeks, getting this demo ready for you all. But it is highly experimental. So I really hate doing demos and then not having a pip install at the end. But this is one of those times where I was too excited not to show it to you, but there's also no call to action because this is, I just barely got this working like two days ago.
So what I'm going to show you is a new way of using Shiny. And it is inside the Jupyter notebook. So Shiny has never been compatible with Jupyter. And well, you'll see in a second. So let's start out with just loading some data. I'm loading the Palmer Penguins dataset here.
And then I'll do like a simple seaborne scatterplot.
So you know what? Since I have a little extra time, I'm just going to see if I can live code this.
So the first thing I'm going to do is load Shiny notebook. And then I'm going to load some from Shiny import, some classes I'll need. This one, I really got a kick out of calling a package Shiny notebook magic.
So the first thing that we'll do is let's add some inputs. Okay. So I have the scatterplot here. I'd really like to be able to experiment with different columns here. Like, you know, let's see like what these look like. And I want to do this interactively instead of having to change the code and rerun the cells. So what do I need? I'll need an X, Y, and Q argument that are all columns from this dataset.
So in an effort to make this as simple as possible, I created a new function called magic inputs. And we want X. And that's going to be one of these call names that I saved up here. And Y is going to be the same. And actually color is going to be the same. These are all going to be the same. So I have three widgets now. Species, species, species. And I can choose any of these columns.
So the way I use this is I can say input.X and it's species. And if I change this, it doesn't automatically update. I have to hit control enter again, but it's reflecting the newest value. Okay. So let's now change this code. Input.X. Input.Y. Input.Color. And go ahead and run that. Okay. So that is the least interesting plot we could come up with. But if I change this and then rerun this, that looks good.
Let's actually make this react. So we're going to do... We don't just want to execute this once, right? When I run this cell, I want to tell Shiny, execute this however many times you need to to keep it up to date. So number one, we'll put it into a function. Because in Python, that's how you package up code. And then I say render.plot. This is Shiny's render plot function. So when I run this, this is now live. So I can change this to sex and it instantly updates.
Now, this is really annoying, right? Like, this scrolling. Because the notebook is linear. And especially if I add in, like... If I add in, like, df input.x, input.y. And then actually, let's render this also as a... So I'm going to make this live as well. So this says build length and build depth. But if I change this, this updates.
So this bothered me enough on Friday. All this scrolling around that I added kind of a... Okay. If you triple click, then you can detach it. And then now I can have these things side by side. And then when I'm done, I can click this and it goes back away.
So that, I think, is sort of the EDA side of the equation. All right? Like, you're screwing around in a notebook. You see something and it's starting to get a little bit tedious. Flipping back and forth between different values. Load up shiny.notebook and add a couple of decorations and you're all set.
And I'm not going to attempt to live code this one. But this is that same set of things that I created. Except I can also have relationships between the different outputs.
Again, I'll select some interesting values here. Oh, no!
I regret my decision to do this live.
Okay. Let's try this again. All right. So in this case... There's some interesting values out here. Right? Like, what are those? So I can select them. And this data frame will show whatever I have selected.
And an interesting consequence of running this in a notebook and not just in, like, a .py file is that I'm also interactively writing code inside of this same Python process. This is not something that you could ever do before with Shiny for Python. But I can do it now. So not only can I have these two selected things reflected here, but I could even... I can grab them with code. So now if I execute this, I have this outliers variable that has the thing that I brushed. And I can go do whatever else I want in this notebook with it.
And an interesting consequence of running this in a notebook and not just in, like, a .py file is that I'm also interactively writing code inside of this same Python process. This is not something that you could ever do before with Shiny for Python.
So this is stuff that has existed for, like, 48 hours. So I don't even know what else this is going to be useful for or when it's going to ship. But I will tell you that I think opening that door for ourselves and being willing to revisit sort of the assumptions of how and where Shiny should be useful has been really exciting. I'm actually only showing one of three different things that have come out of this effort from the last couple of weeks. But it will be for other people to share those other things.
So that's Shiny for Python. If you want to take a look, you can use that QR code. And there's a full tutorial. There are articles. There's lots of examples and documentation. And thank you.
