Expanding Quarto's Capabilities with Lua (Christophe Dervieux, Posit) | posit::conf(2025)
Beyond the Basics: Expanding Quarto's Capabilities with Lua Speaker(s): Christophe Dervieux Abstract: Are you familiar with Quarto and eager to push its boundaries? This session is for those ready to explore the power of Lua for customization. Whether you're a novice implementing simple Lua filters or a seasoned developer seeking inspiration for Quarto Extensions, this talk offers valuable insights into Pandoc's and Quarto's Lua features. We’ll explore Quarto's unique Lua support, including custom AST nodes and helper functions, showcasing how both straightforward and advanced techniques can transform your documents. Through practical examples, you'll gain the confidence to extend Quarto's functionality and unlock new possibilities. Join us to elevate your Quarto projects and contribute to its growing ecosystem! Materials - https://cderv.quarto.pub/posit-conf-2025-quarto-lua/ posit::conf(2025) Subscribe to posit::conf updates: https://posit.co/about/subscription-management/
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
So, hi everyone. As a core Quarto developer, I know firsthand all the amazing features that we built into Quarto and that we document in quarto.org. But I also know all the advanced features that Quarto offers and can help you, users, go beyond the limits of Quarto.
And we do document this feature also on the websites, but I personally know that it's hard to explain, although there is like, we need to be so thorough, there are a lot to explain, and so I can definitely relate if it's hard to feel confident trying to use those features or even knowing what they could use for, like they could seem so advanced. And so, today, in this talk, I want to bridge that gap.
I want to give you the foundational knowledge that you need so that you can consider Lua filters, one of these advanced features, for your own problems and solutions. And hopefully, after, you will be more at ease to navigate the advanced documentation that we offer.
Why you might need a Lua filter
So why would you need Lua filter in the first place? When we use a tool, we want this tool to solve our problems but sometimes this tool doesn't offer the solution we need and this is what I call the I'm stuck problem. So maybe you ask yourself, like, can I Quarto do X that I need for my organization report? Or is this possible to do Y? Documentation says I can do Z but I really need Y.
And so, usually, as a user, those problems are a huge pain point for you but they may be so specific that you may ask on forums but nobody has a solution or maybe it's like not enough generic to be built in ever into the tools. But that's not a problem. You should be able to solve your own problem with some solution and this is where Lua filter with Quarto come in.
And when you think about it, as a R or Python user or doing data analysis code, when you have a problem that some of the package you use or what you are trying to use, you can't do it, then the thing you will do is write probably some function. So maybe some of you will write a full-blown package at first for every specific problem but usually the first step is to write a single function, put them in script, a second one, and then solve your own problem as this goes.
And in the Quarto ecosystem, it can be very similar. So you have a problem to solve, this time inside your Quarto document and you may have heard about Quarto extensions as a way to extend the capabilities of Quarto. And really, you no need to go like further than doing some scripts and in the Quarto ecosystem, those scripts will be the Lua filters which will contain some Lua functions. And you don't need to be a full like Quarto extension developer. As a user, you should be able to use those tools.
A simple example: the trademark filter
And this is what we'll see today, what Lua filters can be used for, how they work, and how you can start writing your own functions. So let's take a very simple example for the purpose of explaining all that. This user works in an organization that is called Acme Inc. And that requires him to use the trademark symbol of the brand each time the company's name in a report. And so this can be really tedious, if you are a Windows user like me, like adding a trademark symbol is like a weird combination of keys.
So it's a normal question to ask, can this be automated? And if you were doing that in our analysis, for example, you may be writing a function that look like this. You would find a way to split the text and then look for the text. And if this is like the name, Acme Inc., then you would append the trademark symbol to it and return the modified or unmodified value.
If you want to do that in Quarto, to process the Quarto documents, then this would look like this. It may be the first time that you are seeing Lua, so the logo you see is the Lua logo. So this is a Lua function, and this is like a screen of the full Lua filters that will aim to do that. So it's a single function. And maybe it's new to you, this language, maybe there is some syntax you don't understand, and we will go deeper into it, but the important part is that this is a function. Lua offers functional programming. Indentation matters. And you can find some construct, like the if then text and the return function. So this function look quite similar to the R version.
How Quarto processes documents
So to understand how this function can apply on the document to modify it, we need to take a broader view of the Quarto processing. So when you write a QMD document, the first step of the processing will be computation engine. And so Quarto will run your code, usually so your R, Python, Julia code, and using the correct engine, and produce an intermediate markdown file.
And this is the first step where you, as a user, you can interfere with the processing, using some custom function with your language of choice in this part. The second step of the processing is writing to the desired output. So you may want a PDF report, an HTML report. And so this is where Pandoc comes into play. So Quarto will call Pandoc. And this part of the processing can also be customized. And this time, it can be customized using Lua function inside Lua filters.
And Quarto extensively uses Lua filter inside to bring the features that you see. So all the features that you see in Quarto that doesn't exist in Pandoc, for those who know Pandoc, they probably are made using some Lua filters. We also use Lua filters to modify the default behavior of Pandoc. And as a user, you can insert your own processing into there in form of functions.
The Pandoc abstract syntax tree
And so how does this function can apply on the text from your document? And the specific thing is that Quarto does not really work on the text itself. As you can see, like maybe you think it's using regex or something like that. Not at all. So under the hood, we need to look into there. And this is an abstract representation.
And so this is what we call the Pandoc abstract syntax tree. So it's not something easy to explain, but I hope like this simple schematic will give you like the right amount of understanding about that. So there is a full document that you have and that you see on the screen. And the first step of the reading part will be to split this document into blocks elements. So the bigger elements. So it could be headers, paragraphs, some code blocks. And those elements are called blocks and they are like the branch of the tree. And the old document is a trunk.
And each of these block will then be split into more granular elements, which are called the inlines. And so this will be the strings, some bold text, some links, some emphasized word, like some more granular element that are inside those blocks. And so those are called the inlines. And this is on that representation that our function will apply. So we come back to this function, a Lua function inside the Lua filters will apply on any node of this AST. So this abstract representation. So our function here will apply on all the strings in the document.
And for each of these strings, it will like look at the, it will take the string element as input and it will look at the text information to modify it in the output. So either return the unmodified value or return the modified value. So it's quite similar to the R function that we've seen.
And how this filter is used inside the document, this is using a configuration key in your YAML editor. So this is the filters key. And so this script can be added this way and then document will process. And so here you see on the left, the source document, where you have like the insertion of the filter, and then you see the name of the company from our user. And on the right, you see the rendered HTML output. And this output as the trademark symbol. And so this is a simple Lua filter that did this processing. So it modified the document. So we have a source and the render output.
A second example: the card header filter
So this was a simple first example. And we'll see another one, which is maybe as simple, but will help us understand a bit more how Lua function are written on the inside. So this time, I will do it reverse. I show you the results right now. So we have a Lua filter that's called card header.lua. So the name, like maybe give hint on what it does. And we see we have some headers on the source. And those headers inside the output document will have some spades or arts around them. And so this Lua filter will do this modification.
So maybe those of you already can know or guess what the Lua filter will be. But the first step is to look at the abstract representation. And for that, Quarto offers a format that it's called a native format that you can use either in the command line interface, or you can use as any other format as a YAML key. And this will produce a .native file. And this file is the abstract syntax tree representation at the end of the processing, right before the writing step to the desired output, if you take the schematic. So this is the representation of the document. And so it looks like that. So this is a folded version.
But we can see that the main node is the document node. This is all document. And this document node is split into blocks. Here we have a mix of header and paragraph. So header and paragraph. We have also a meta block, which is a special block for metadata. And each of these blocks is then split into inlines. And we can observe that in this format. So the first header has one inline, which is a string introduction. And the last yellow box shows a paragraph, which is a mix of strings and space, a granular element.
So each of the nodes in the AST will have some contents in them. But they will also have some information in the form of attributes. So they can be ID. They can be class. They can be key values. And so most of the nodes will have those information. And some of them will have specific information. And here our header have information of their level. So we have level two headers on the first arrow and level three headers on the second arrow.
And with those information, we can now understand what the Lua function can be for this processing. It's still a single function in Status Script. This function will target all header nodes. But the processing will differ based on the level information that we have access to. And so if this is a level two header, then we will add the space. And if this is a level three header, then we will add the odds. And how those elements are added, the content is modified using some API from Lua for Pandoc. And the temple.insert function is a way to add some elements inside the contents of one of the nodes. And the pandoc.str function is coming from the Pandoc API. And it's a way to create new elements. So you can modify. You can create elements.
Targeting nodes and helper functions
So with those two simple examples, I hope you have now a better understanding of how Lua filter works and how you can write some for your own problems. So the first thing you will need is targeting nodes in the abstract syntax tree. Those nodes can be blocks. They can be, so there is a set of blocks. Here's a list. I won't go into detail in all of them. There can be inlines. So the blocks are the containers. And the inlines are usually the contents. And your Lua function can target any of these nodes.
So most of the AST types here that we see are coming from the Pandoc API. And so you can find information on them on the Pandoc documentation. But some of the blocks come also from custom AST nodes that we add into Quarto. So the callouts, conditional blocks, and tapset, they are nodes that you can find in this abstract representation that are in relation to some of the features you may know in Quarto, which are the callouts, the conditional blocks, and the tapset, which are not basic Pandoc features. And the documentation for those customized nodes are inside the Quarto documentation.
So there is a bit of navigation. It's what, like, it's advanced documentation. But this is where you can find information on them. Then you need some helpers function. You need to write the processing of your filter. And for that, you have, like, three layers of API. You have the Lua base API, which are the base function that Lua provides for string and link, pattern matching, some table manipulation. This is the basic from the language.
And then the second layer will be the Pandoc Lua API. So this is the core API that Pandoc provides to work with the AST type that are coming from Pandoc directly. But there is also some helpers function to use with some of the specific features that Pandoc offers. And the third layers, you probably guessed right, is Quarto layers. So there is a Quarto Lua API, where we add additional function. Obviously, we add the necessary function to work with a custom AST type that Quarto is adding. But there are also useful function to work with the Quarto processing that has custom formats, things like that.
And we also, as we own this layer of the Lua API, we use that also to add some useful function that is the Lua filter development. So we keep adding some function into there. There are some function, for example, to add HTML dependencies, as you would do in R package, for those who know that. And you can do that to add things into there.
And the last part is inserting, once you have your filter, is inserting that filter into the processing. And the nice part of modifying or working with the abstract representation is that you don't need to care that much about the raw output. So you can make a different processing based on the format that you target. But as you've seen in the two examples, we didn't consider the format at all. And so for any format, the modification will happen.
Quarto extensions for reuse and distribution
And it's a bit natural to ask yourself, like you have this one-shot filter that you've made. And maybe you want to reuse it across your project. Or maybe you are working in an organization where different users have the same problem. So the question of reusing these Lua filters makes sense. And this is, at that point, that you can consider Quarto extensions. They are a way to reuse and distribute some feature of Quarto. So filter is one of them. And you see that Quarto extension can be of several forms. There are also shortcodes. So with the knowledge of Lua function, you can also write shortcodes. And there is other types. You can do extension for custom formats or brand. And they don't require knowing Lua at all. But they are a way to ship that to several projects or several users.
So with your filter, how would you do that? You would use the Quarto create function to have the skeleton of the folder with all the file you need. And you would copy your Lua filter into these organizations and configure in the YAML configuration in the right way. And then you need to share it somewhere. And Quarto add a function to ease the distribution. This is the Quarto add function. And this function can work with extension that are shared on a shared drive, for example, in your organization. It can work with extension that are shared online as a zip in a URL. But the most common way to share extension will be through a remote GitHub repo. And so Quarto add can also install from there.
So that way, you have one place where you can put your extension with your filters that solve your problem. And then it can be reused across project by just using Quarto add and then calling the filter in the same way as we did before, just naming the filter you want to use for either your document or the full project.
And once you've done that, you have made a filter and a Quarto extension. And you're probably not alone doing that. There is a full list of Quarto extension. So this nice listing made by the community of all the Quarto extension that are hosted on GitHub. And some of them are for filters, as we discussed today. And so this is a great place if you have a problem and maybe you think someone else have it, you can look for a solution in there instead of doing your own. But if you start doing your own solution, then you can also go there to find examples of Lua filters out there, what function exists, how people do that. And you see that some of those extension like Quarto Live are in this list and they are using Lua filters to bring you those nice features.
Wrapping up
So I hope this talk give you the minimum foundational knowledge to do consider really Lua filter next time you feel stuck with something and you want to solve a very specific problem, really go try it. They are not that complicated. After all, they are just functions. And they can be useful to modify, replace, even remove, or maybe just for side effect because you have an insertion into the Quarto processing.
After all, they are just functions. And they can be useful to modify, replace, even remove, or maybe just for side effect because you have an insertion into the Quarto processing.
So thank you. Hopefully you will now be able to read those advanced documentation and understand a bit better. But feel free to reach out for any question now or after, or even later on social or GitHub. We have a discussion forum on the Quarto repo.
Q&A: resources for learning Lua
I think we have time for one question. And the question is, do you have any recommended resources to learn Lua?
It's not, it was the aim of why I wanted to make this talk is that you don't need to start learning Lua before trying to do that because of the example and things like that. But we do in the Quarto documentation, we do have a page dedicated. So it's hidden in the extension sections, but we have a page dedicated to Lua development where we try to guide you in what you need, the basics you need to know to do your Lua filter. So you don't need to be a Lua developers. You just need to need to write a function that does things and we provide API for that, useful functions and that.
Yeah, and I would also add, if you want, there is a GitHub organization called posit-conf-2025. And you can see all of the workshop materials that went on a couple of days ago or yesterday. And one of them is a extending Quarto workshop. So you can probably also use that as a reference. There's also related to what Becca was talking about. There's a branding related one as well. So they're all Quarto related, but yes, thank you.
