Christophe Dervieux - Exploring Quarto Dashboard for impactful and visual communication
Embark on a journey to explore how Quarto Dashboard can enhance the narrative of your analysis from your Jupyter Notebook. This talk will show how to create cool interactive charts and graphs that bring your data to life, by using Quarto - an open-source scientific and technical publishing system. Learn how to make your data communications more engaging and dynamic using Quarto Dashboard. Practical examples and simple explanations will guide you through the process, making it easy to understand and apply to your projects. https://cderv.github.io/pydata-paris-2024-quarto-dashboard/#/title-slide www.pydata.org PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R. PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome! 00:10 Help us add time stamps or captions to this video! See the description for details. Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Thank you for coming to the talk. I'm Christophe, already presented, I work at Posit and I mainly work on the Quarto project. I want to talk about this project with the line today, the main usage of creating communication, visual communication, and so impactful communication from the data you can do in your analysis and in your notebooks. And so we'll see first what is Quarto, for those of you who don't know that, and then focus on the dashboard output.
What is Quarto?
So what is Quarto? Quarto is an open source scientific and technical publishing system. It builds on standard Markdown, but with more feature targeted to scientific communication and technical writing. And so it extends the penned up Markdown syntax for textual content, or anything like you want to explain and say in your report document, but it will add support for engine and computation. And so it has multi-engine support. One of them is a Jupyter engine, and it will allow to produce various output types, output formats using this layer. And so with Quarto, you can mix together some narrative and some code that are meant to be executed to produce some formatted output.
Depending on your audience, it could be like visual formatting, single documents, PDF, Word document, HTML document, single web page, but it could be also websites, a website with blog posts that you want to publish regularly, some books, HTML books, or books to be published, and other types of formats. And this includes dashboards, and dashboards are the specific format really meant for being visually efficient to get the information. And so you can work with your data, with your computation, get your metrics, and then have a dashboard ready in a single text, which is a rendering.
So more concretely, Quarto is a command line interface. It's something that you install, you get a binary. And this command line, you will be able to call it an input file, which could be Jupyter Notebook, Hyper-WAN NB format, or also plain text document with a specific syntax which is QMD document, or MD document, standard markdown. And we are now at version 1.5. The latest table is 1.5.57. And the two main commands that you need to use are render and preview. The command line interface also have a bunch of other commands, like create, use, add. This is for all the features that Quarto can do. There is extension mechanism, like conversion between the Hyper-WAN NB format and the QMD format. We won't see that today in this talk. I will focus on the dashboard output.
So to render a notebook with Quarto, you just call Quarto on the notebook. You see on the second line, I added execute flag. Because by default, notebook can be executed in like JupyterLab or Jupyter Notebook interface. And so it stores some results computation. So by default, Quarto will just take care of rendering to the output, so to the HTML, to the PDF, to the output you want. But you could also tell Quarto to re-execute all the cells before doing the rendering with the execute flag. For the QMD format, which is a text file format, the execution will always happen of the cells. And then you can pass argument to this function to like, most of this example is a target format output, but you can also pass some configuration option and different things.
And the preview command is the equivalent, except that after rendering, it will start a live server where you can get updates while you modify your document. And each time you re-render, it will update the live server. And for project, it will update only the relevant part of the website. Like when you have a website and you work only on a single page, you don't want the whole website to re-render each time you work on it. So that's the two main commands, how you would use this tool.
But it's also integrated in other tools. It's not mandatory to use it in terminal. You will there is a Quarto extension in VS Code where you could have like keyboard shortcut to render or like a common palette to render and you get the visualization. So the live preview is on the right directly in the tool. We have the same JupyterLab Quarto extension, which also like Quarto, I said it's extended Markdown syntax. So there is like some specificity that JupyterLab won't recognize without the extension. So it will like format correctly in the IPowerNB report, but also allow to render the preview directly. And it's also available in Positron, which is a new IDE developed by Posit for data science, which is based on VS Code. And so the Quarto extension is also available there.
How Quarto works internally
So before understanding the format, let's dive into the workflow and how it works. I said it takes Jupyter notebook as entry. And what Quarto will do internally is that it will make a Markdown representation of this notebook. Then it will pass it to Pandoc, which is a universal converter to be able to write in the different output, targeted output. And when the QMD is passed as input, it will pass by the Jupyter stage. So the QMD will be like transformed to a Jupyter representation so that it can be, code cells can be executed. And then the same workflow.
And so the three main components where Quarto will intervene is the computation. So I give example here with Jupyter, but it's also support observable.js, Nitar engine for our users, and also more recently a native Julia engine directly with the same logic executing the cells and then mixing with the Markdown content. The second step is a Markdown. So we extend Pandoc Markdown. It means that Quarto will add a lot of layers of specific syntax and also Pandoc Markdown syntax with more feature targeted for scientific communication and technical writing. So we have like, for example, cross-referencing figures, having the ability to organize the layout of the figures, things like that. This will be done directly. And for dashboard, this is where we will see all the transformation happen.
So what does it look like? You have your notebook. This is a simple notebook with no Quarto-specific addition. If I'm calling Quarto render on this notebook, I will get an HTML representation of that with the code cells that are shown. And then the graphics of the cell has been put below the cell. Now I can control some part of the document by adding specific syntax at the top, which is the YAML header. So it's a row cell in a IPoWNNB notebook. And this will control here the styling and the seam. And so I get some differences in the notebook. And I can control also, for example, I don't want to see my source code directly. I want the user to be able to see it, but only when unfolding everything. And so just by controlling some configuration of Quarto, I can get that in my HTML. I didn't write any HTML code. I didn't write any JavaScript code. Everything is control to option.
And if I want to change the format, I told about different format, I want slides for my notebook. Then I'm just using format reveal.js. And I will get slides for the graphics I have. By default, the code source is hidden.
Dashboard output
So this is a quick introduction to give you what Quarto can do. And so now let's see the dashboard-specific output, where we want specific communication, visual communication for this.
So let's pick out some example. I will go through different examples just to show you how it looks before explaining how to make a dashboard. So from your IPoWNNB notebook, you can do have like this representation, this kind of representation, where we would have like, we want to the user when opening a dashboard to see directly some metrics of colors, to see the same graphics table. We want a user to directly have some information on the main page, possibly also having some ability on the sidebar. We can show some interactivity with JavaScript library that are available. So this is with IPyLeaflet, for example. This one is with Plotly Graphics. We get like interactivity in the browser with that.
The layout can be also tweaked. We will see how. So here, I want my user to read my information a bit differently. So my numbers are in the row. You see it's like less impactful than the previous dashboard. In this one, the metrics come first with the colors. I get some also smaller metric on the sidebar, different pages. You can also add text. Usually, a dashboard, you don't have like a lot of text. It's not something that you want to read the same as a report or a document, but it's still interesting to have in a page, for example, some explanation about the model or the context of the analysis. You can have like different size and tab sets here on the side.
And the last one is about interactivity. Like the dashboard can also be interactive, either with a backend or also with no backend, which was presented with the talk before. And so here, you can guess like it's not interactive in the presentation, but you can guess like moving the sliders, the graph will update. Here, changing the parameters on the sidebar will update also the data and the plot. And this one, like we see Stock Explorer, we can guess that this is something that we get up to date through like a workflow on daily basis or hourly basis.
And the last example I want to show, the Olympic Games example, which is a recent one where we like have some analysis, some data about medals for summer and winter Olympics. And I will use this example as a demo on how to build this kind of dashboards in the follow, in the rest of the presentation.
Building a Quarto dashboard
So how to build Quarto dashboard? Let's focus on that now. So we've seen before that it's just a matter of changing formats. So I have my dashboard. Maybe I've done my analysing with the Olympic Games on a usual report, so before thinking about making a dashboard. And I want to make a dashboard. So the only thing I need to change will be the format. And so now I will use format dashboard. And this will make my report become a dashboard.
The main component of the dashboards are cards. So the smallest pieces. I want to organise my metrics, my tables, my plots in cards. And these cards will be organised by rows and colons. And I want to define how many rows, eight, the widths, things like that. And I won't possibly have too much things to show. And I need to organise that differently with interactivity, adding some things in the sidebar or adding some pages in the nav bar. And so these are the main thing we want to tweak. These are the dashboard components.
So navigation bar and pages are the highest level. You can set icon titles. Subpages with links to subpages. Then the second level will be sidebars, then rows, column, tab set, and then how I will fill all those pages, rows, column with cards. That can be plots, table, value boxes, and any content. Like usually cards are containers for the results of your cell, and so are free from markdown text. But usually the content of a card will be a map to a cell in the notebook or in the source document.
So all these components can be auto and customised within the UI, and we'll see how. So let's start with the navigation bar and pages. So this is the default for a dashboard. When you will say format dashboard, you will get this specific HTML page with a nav bar. The title will be taken from configuration you pass to the document. So I'm using the QMD form here for the text because it's easier for the presentation, but it would be the same in a notebook just with cells.
Here I want to add a GitHub link on the right side that points to my GitHub repo. So I'm adding that in the configuration, and I want to add a logo, which are configuration that you can guess that are available for also HTML, like other report type. But this one for dashboard, it will go on the far left of the nav bar. And then you see two pages, Summer Olympics and Winter Olympics, and they are created by the structure of the document, and the structure is based on Markdown. So any other one in the Markdown document will become pages. And so this is how I can organise my content.
Then I need to fill that. So I'm just taking a detour through sidebar because I talked about that. You can define sidebar at the page level, and it's really easy to define that. If you want a page level, you set the header 2 because it's inside the page, and then you just set an attribute, which is a sidebar attribute here, and this will add the sidebar. No HTML, nothing else, just adding that. Any content below this header 2 will be in the sidebar. If you want a sidebar for all your pages and it's a global level, you use header 1, you use the same attributes class, and then you will get the sidebar.
Regarding the layout, so the sidebar gives us a peek into that. It's header 2. So by default, dashboard will be organized by row. It means any header 2 in my structured document will become a row, and I can control the height of the row. So here I have two rows, one with one card, you see, one Python cell results, and another one with two cards, and I change the size.
Now you see the olympic games example were organized a bit differently, and so I need to change the orientation. And so it's not in the structure, it's in the configuration. I'm saying like the orientation will be column, and in that case, any header 2 will become a column or orientation. And so the name inside column or row is not important, it just really is the structure, the header 2 or header 1 that define these kind of things.
Now how do I fill those columns? So I took the example directly from the olympic dashboards, and so the cards will be automatically created based on the cell content, the cell output that I have into my structure. So I have a column, you see, I get a name, it's not used in the dashboards, just for me to know what I'm like calculating. So I want the first row in my column being medal byte port, and then I'm calculating something. So I will set a label because it's easier to reference later, and also a title which will become the title on the cards if I want one. And then I calculate different things, and I'm outputting a plot, and directly I will get that when I Quarto render this notebook, directly my plot.
I can add specific content like the value boxes, and those value boxes are creating the same. I want rows, I want value boxes, you can get them using Python data, so you can use a dict to do that. The key here is a component value box, we will say it's not a regular card, it's a value box which has a specific tiling, and I can set icon, color, and value. And I can do the same with some data I get from, or like, I can define the color, the value, and the icon from that I calculate in Python, or I retrieve from databases or something like that.
But you can also create those value boxes with Markdown syntax, and this is where like Quarto extension comes into play. So it's specific syntax which is called div, I'll put a class value box, which means it will become a value box, and I set some attributes, icon, color, and inside I will use Markdown text. But I still want to get some Python calculated value into that, and so Quarto knows how to handle inline code. It means like what you see here in those two two boxes, so the gold silver and bronze boxes, this syntax is inline code syntax, and it's a way to mix Markdown text with results, and when you Quarto render, it will know how to insert the output into there, and so you can also create some like complex content with specific value directly from your Python cell.
And the last component we add in the dashboard were the tabsets, and it's the same as SATBAR, it's just an attribute, a class you set on the row to say the cards inside this row, I want them to be in tabsets, so one behind each other, and so by doing that you can put a title on each of the title options, so this is a syntax for putting option into cell, a title option inside the cell, and this will give a name to the dashboard, and this is how we get those tabsets for the metals table, like so, this is how it is created.
Styling the dashboard
So these are the components, and the last part of the dashboard construct that usually you want to spend time on will be the styling of the dashboard, because this is what you want to control, so we saw we can control some of the style using the value boxes colors, that can get either fixed or I can get depending on some values, but for the rest it's the style come from some themes, and so HTML format, all HTML format in Quarto are based on bootstrap, so it can be opt-out, but the default is to be based on bootstrap, and dashboard is based on bootstrap, it's based on the bootstrap themes, and so Quarto includes dashboard, includes more than 20 themes available, so already built-in, which like predefined default that you can choose.
And so here in the example of the Olympic dashboard, the theme used was sketchy, so just by saying theme sketchy, I get this dashboard which will be different from the default, and if I use like another theme, like flatly, I will get a different dashboard with different fronts, things like that.
Now I want to customize that, and to customize you can use Quarto theme files, so it's a second line here, we have an Olympic dash dot SCSS theme file, and those theme files are based on SAS with SCSS syntax, and it's an easy way to declare new themes, new variation on new themes, to be layered into what bootstrap and bootstrap will do to build the the CSS, and so this is an example of a theme file, it's a singular file, like if you know a bit about SAS or SCSS, usually you have different like layers you need to do, here we have a single file that Quarto can understand with the specific comments, like SCSS colon default, but basically the idea is that you can define some colors that will change the default from the bootstrap theme you are using, or from bootstrap, but you can also add rules, specific rules that don't exist in the default dashboard that can like fit your different style.
And we are even spending time now for next version on making that even easier for brand, what we call brand branding, and so a way to pass information like this more easily through YAML configuration, so that you can apply the same style to your dashboard, but if you want to do also HTML report you can have the same style, or a PDF report you can also have the same style. Currently this SCSS file works across all HTML formats, but if you want to do a PDF document, like it's not CSS, so it's not working, so we are taking a step like back and making something that will be more cross-format, which is the aim of Quarto, being able to have like single source to produce different type of output.
the aim of Quarto, being able to have like single source to produce different type of output.
So if you want to learn more about Quarto dashboard, like you can look at our doc, there is advanced feature I didn't talk about, interactivity with channel, I mentioned it can be interactivity with Shiny for Python or any other tooling, either with back-end or just inside now inside the browser with Wasm, and anything that exists in this, maybe with Perspective, which was a talk before, I didn't try, but you can have variation of dashboard with parameters, so it's at execution level, if you want to have different variant depending on who you are targeting, like C level or your colleagues, like maybe the information should not be the same, so you could parametrize your dashboard, and then you can deploy easily through the Quarto publish command. So about Quarto, like here's the link of the presentation, it was in the footer all along, so I hope you got it, so you have user guide and gallery of examples, so thank you.
Q&A
Thank you Christophe for your talk, does anyone have any questions?
Firstly thank you for the wonderful presentation, the question which I had is like since we are talking about Quarto as a dashboard, is it like a competitor to the other tools like Tableau, QuickSight, MicroStrategy BI tools, or I mean it is a different use case, or like when should I think of using Quarto? It will like on the output it can look like similar, because you are building dashboard, but there is no exploration feature here, like it's all like the source is like the IPAW NMB file or QMD text file, and you will build your data, and then you want to format. I don't use Tableau and those tools, but usually in those tools you can like play with the data, organize things interactively in the dashboard, before like making a snapshot of it or publishing it. Here you can't play with that unless you build it, so if you want to play with the data in your dashboard you need to like use the Python library or the thing that will make your data interactive, so that you need to play in the dashboard, so it could be using a web framework or Python library that do that, or maybe something advanced like Perspective which was the talk before, maybe it can be used if it can be used like with anything with JavaScript you can use, and anything like with a server IPywidget you can use also with Quarto dashboard, the only thing is that there won't be any Python server running at the end unless you specifically use something like that, so it's different in this way, everything is inside the product, so it's more like a HTML report, but instead of getting linear you're making it more targeted to specific audience or to show specific number quickly.
Yes, it's an output format for communication, as I said Quarto is mainly used for like a technical communication, authentic communication, so it's you can use it for like if you want to read your analysis in a HTML format or a PDF format, but if you are already working in notebook or you get something that is quite like nice with the report, if you are working in a text file you may want to have the live preview while you're working, but otherwise usually you're working with Quarto to produce something to send to someone or to share on your website, to share to your colleague or to share at the production report or something like that, and you use Quarto to build that, you can build like for your own interactivity and so it's easy to make a dashboard from your report if you want, but when you start thinking about the seeming the organization and things like that, you are already at the stage where you want to communicate something and you want to organize that communication.
If you want to do like, you can use any library, graphic library you want, like for plots, for tables, for anything, like Quarto is just the framework that allows you to organize, but then any Python cell you use can use anything you want from Python, and if something doesn't work, it's a problem on our side and you can report an issue because we don't underwhelm the output, but like if it's a plot we'll make a graphic, if it's a table we will put a table, we will like put things nicely and so everything that Python can do in Jupyter notebook, it can be shown in the output.
Thank you very much for a great presentation, I've been using Quarto for about a year to produce websites and I have not used dashboard yet, are there any limitations in terms of size of amount of data you would recommend to so that the dashboard still, you know, being nice and usable from the user perspective? It will depend on the tool you use inside the dashboard, if you wanted to make it interactive or static, like if you compute all the data before and you are just like putting summary and you get a static dashboard, there's no constraint except of the size of your plot, like any HTML document, like if you are like making big plots with a lot of points with Plotly, maybe it will be very big and a bit slow to download, but otherwise there's no data limitation. Now if you want to do interactive dashboard and you are using Python in the browsers for Wasm, then it's not a Quarto dashboard limitation, it will be what the browser allow and what the Wasm like Python running can do, these kind of things, and if you're doing like interactive data, it also depends on the library you use to like stream the data between the server and the dashboard. You need to think of Quarto dashboard as a way to easily produce like this type of layout and output without writing any HTML or loading bootstrap or caring about like JavaScript and things like that.
I apologize if I missed it, but what's the, from the user perspective, what's the major difference between JupyterBook and Quarto? I guess you mentioned that, yeah, that Quarto produces dashboards. I think JupyterBook doesn't, please correct me. It's a similar project, it's a similar project. Quarto is multi-engine by default, so we can use like Jupyter like, and obviously your audience like, you care about Jupyter, so it's really similar, but for our user, like they can use Quarto, it won't use Jupyter. For Julia users, they can use Quarto and do the same, it won't use, it won't use Jupyter necessarily. So if you are working in a multi-language team, for example, a multi-language company, it can make sense to have everyone using the same tool.
And, and the different thing is that Quarto aims to have single source for all outputs. So you can do books, books format, so JupyterBook can do that, but you can also do like PDFs through types, you can do PDFs through LaTeX, you can do HTML documents, you can do websites, you can do RubyJS presentation, Word document, PowerPoint document, anything like that. So if you need this kind of things, then it can be interesting. If you just need like JupyterBooks, then it's just a matter of syntax, like it's heavily built upon Markdown syntax, so the source file at Markdown, the construct, you see the structure of the dashboard is based on Markdown syntax. I think JupyterBook use a different syntax for configuration and also for some content, so after that it's a matter of feature, but it's similar project, yeah. Thank you, Christophe. Can we have another round of applause?
