Transcript#
This transcript was generated automatically and may contain errors.
Hello, and welcome to the Python Exchange. My name is Cameron Riddell, and I will serve as your host this afternoon. Today, we are joined by a special guest, Rich Ianan, whom you all may know through his various open source projects, Great Docs and Great Tables. If there's a good chance that his project is great, then Rich has had a hand in making it. But before I share more about our guests, let's first introduce our panelists who will help us drive discussion. First up, today, we are joined by, as per usual, James Powell. James, for anyone who has not yet had the opportunity to meet you, can you answer this question for our audience? James, what is one of your favorite Python topics as of today?
Hi, everybody. As Cameron said, my name is James Powell. It's a pleasure to be here. One of my topics that I've been thinking about a lot recently related to Python is, what might come after Python? I know that might be a very controversial thing, but I've been thinking a little bit of what the next language, the next hype, the next thing might be.
Fantastic. We have some other hosts who are happy and chomping at the bits to share their favorite Python topics as well. Next up, let's hear from Tom Caswell. Tom, can you answer the same question just so our audience can get to know you a little bit better? And that is, what is one of your favorite Python topics as of today?
I think one of the things I'm most excited about is the new debugging, the new profiling, the sampling profile that is coming in Python 3.15. That is a thing I'm looking forward to being able to use for good.
Let's also hear from our host, Tani. Tani, same question to you. What is one of your favorite Python topics as of today?
I think it's my background in AI and ML. I think it's definitely related to how can we build Python-based pipelines that can receive the fast-paced advancements of AI and ML and new things coming up. So how to make sure that we can do this transfer learning appropriately across different projects.
Last up for our host panelists, we are also joined today by Dan Allen. Dan, can you also share with us what your current favorite Python topic is?
Sure. I'm excited to see how the free threading work forces me to rethink the assumptions that have been true about Python for as long as I've been doing it. I guess that pairs well with the sampling profile that's going to check my intuition.
Well, these are our host panelists who, as I mentioned, are going to help us drive some discussion. And we are going to be discussing a topic that our guest today, Rich Iannan, is very passionate about.
Rich is a software engineer who spent a long time thinking about how we actually present information in documentation, especially when it comes to tables, summaries, and making complex things easier to scan and understand. He's worked across the open source ecosystem for years, starting primarily in R and recently contributing heavily in Python as well. Today, Rich will be sharing with us about Great Docs. So Rich, take it away.
Introduction to Great Docs
I will do that. And what I got is actually a bunch of slides and some browser tabs because I figured it's good to show Great Docs. So we're going to have a mix of both.
So Great Docs. It's a documentation framework in Python for Python packages. And we're going to begin with some background. Challenges with Python package documentation. It's often the first thing people look at before they commit to actually using the software. So it might be a deciding factor as to whether they use your software.
Documentation has to be good to keep existing users engaged. Otherwise, people may turn to alternatives with better docs. More lately, an agent might read the docs first. So we probably should keep in mind that a doc site might be better suited for agents.
So I got some responses for people that do docs. Three key points. One is, design for a good initial impression. Organization, presence of reference and guides should be there. And then to keep the users engaged, we need good authoring tools and use those good authoring tools for the reference and guides. And finally, we need things suited for agents, like pages in markdown form, llms.txt, and maybe even documenting skills and having skills available.
Examples of documentation problems
So many tools that make documentation for you or help you make documentation, they get you to a pretty decent look and feel, and then they may not go beyond. So let's take a look at a few docs pages. These are not all Python related. I just kind of surveyed some of the internet and looked for some examples of docs.
Here's one. It's actually a package I work on. And these docs were made by me. They were made by Cran. And we can learn a lot just by looking at what not to do with docs. If you look on the left-hand side, the contents list is not arranged in any sensible manner. It's alphabetical order, which is good if you know the name of the thing, but otherwise kind of bad. The logo is missing, if you look in the top left. That's not a great good initial impression. The topography is not that great. And the tables, they're hardly recognizable tables. They're not styled. There's no lines.
This is from a Python package. And if you look from the left side, a lot of those objects are encroaching on the main content area, which is obviously not good. And also the code examples are oddly indented, almost centered. And it's got this problem of many, many methods on a single long page. And maybe that's the fault of the documentation generator. It doesn't give you the option to split those out in some sort of gripped manner.
Here's another one. It's actually for a LaTeX package called long table. And it's a PDF. So you don't really get the benefits of documentation on web pages. You can't get there with a Google search, typically. The command reference, there actually is one on some page, is pretty terse. And it's all stuffed into a summary table.
Here's another one for a project called juice. And I was confused about this one, because I looked for a doc site. But essentially the docs are on the readme, the GitHub readme. So that's a problem. There's no generated API reference. So the issue there is that this could drift from the actual code. It's basically handwritten. And it only ever reflects main. So anyone on an older version can't read docs for that version.
So I showed a few egregious examples of maybe not-so-great documentation. But I want to say, a lot of the documentation now is really, really good. FastAPI, Polars, FastMTP. These are great. Like Seaborn, these are fantastic. Authoring features that they use to support teaching. There's a lot of polish there.
But still, there's still challenges that remain. Like building docs for AI agent consumption. There's a little bit there. You can sort of see that in FastMTP. But it's kind of like only the beginnings of that. So a while back, when I was doing this survey, I thought, maybe there's an opportunity here to do something. So enter Great Docs.
Design goals and inspiration
I figured I would throw my hat in the ring with a new Python documentation library. So some of the design goals I had early on is that with it, you can create a beautiful, complete site with almost no configuration. It'll have depth features and control when you need that stuff. And a big thing was, it's got to be ready for human readers. Like, be really, really nice if you're just reading it. And also be useful for AI agents. Because a lot of times, they're just thrown onto some docs.
So when I was looking at stuff, I looked for inspiration everywhere. I looked at things like pkgdown, MATLAB. A lot of good ideas, wherever you look. So I'll show you what I liked from each of these. pkgdown, that's an R thing. MATLAB, that's just the MATLAB docs. Hexdocs, that's for Elixir. And I think Erlang as well. And Vue.js, which is its own thing.
So pkgdown is kind of where it started for me. It made good docs by default for R. And I did a lot of R stuff. So basically, I use this a lot. A really cool idea that they had a long, long time ago was this right sidebar. And it just made a lot of sense to me. It just has metadata for the package. It has the authors, some useful pages like the guides, the contributing guide, citation, license. You know, good stuff. So I basically, I stole that. It's in Great Docs now.
But you know, pkgdown. Like I said, it's my home turf. I built tons of R packages like these ones here. And they all have a pkgdown site. And I'm pretty familiar with it. So a little bit biased, you might say. What else did I steal? Well, the homepage sidebar, like I showed you. The idea of a zero-config build. Almost zero-config. Conventional reconfiguration. Like if you have certain files in certain spots, the framework will just look for them and then use that in making the docs and the build. Beautiful publishable defaults. And pkgdown's site looks wonderful the instant you make it.
And this wonderful package API index page, which is kind of cool. It kind of gently puts you into the reference section. It gives you an index and some descriptions. And lo and behold, that's in Great Docs now.
Okay. MATLAB. I gotta admit, I don't really like MATLAB. I used it before in the past. Not really a big user. But I love their docs because they really go so deep in their docs. It's all super accentuated at the top. They tend to overexplain, which I think is kind of a good thing. They have tons of examples. Like, this page is really, really long. Like, they have so much in there. And I kind of appreciate that. And the parameters. Nobody really documents them like MATLAB does. They give lots of room for examples. They have them in disclosures so you can just fold them up because there's so many of them. And again, I love the idea of a short description for each parameter.
Okay. And Elixir. They have this very consistent set of documentation pages. And it just looks good because of the structure, I think. So especially the sidebar. I mean, what I took from this is navigation is very, very important. Like, if we can just simplify it and make it really, really good, I think that's one big sort of piece of documentation that's solved. And they also, like many other frameworks, they put narrative guides as top level with a reference section.
Okay. Another one is the Vue.js docs. They're so easy to navigate, and they're really great at teaching new users especially. They also get navigation right, which is really nice. I like the fact that their sidebar is very flat and easily scannable. They have good use of space. They don't make it too complex. There's no nesting. The header itself is not clickable. It's basically just a group, essentially. The CTAs they put at the end of different sections, they're wonderful. They're basically like learning paths you can take. And they have these in-page callouts that are used for tips that you can divert yourself to a different learning path.
I think that's great. I looked at all these tools. I was really inspired, and some big concepts I used were pkgdown's ease of use, MATLAB's insane depth, Elixir's structure, and Vue.js' clarity. Great Docs tried to incorporate all these things along with some well-considered defaults.
Great Docs tried to incorporate all these things along with some well-considered defaults.
And what I think we do a bit differently with this project is beautiful out of the box, tries to look good on mobile as it does on desktop widths, and document as many things as possible, especially for LLMs. Like, the API, of course, CLI, if you have it, MCP servers, and also document agent skills, if you got those as well.
Tour of the Great Docs site
Okay, so I'm going to take you out of these slides, like I said I would, and show you the Great Docs site. Okay, first I'll just show you the landing page, like essentially the GitHub repo. This is the Great Docs site. I just want to show you they can have light mode or dark mode. Kind of cool. And here, again, is the right sidebar with different links to go to different places, like you can view the package on PyPI, you have links inside GitHub to report a bug.
Another cool thing is, if you have certain files kicking around, Great Docs will look for that in the directory, or other little places that these files may be. For instance, right here, contributing guide, it looks for contributing.md, it basically publishes it in a separate page. Code of conduct, project roadmap, if you got it, roadmap.md, security.md, your license as well. I just want to show you one of these. I'll show you license. It's kind of cool. It'll take you to this page right here, and it'll actually tell you little details about licensing things, because it knows about the MIT license, and it'll just tell you a little bit more about it. In case you come up against a license you don't really know, can you use it or not, it basically dispels some uncertainty there.
I also want to show you the user guide right here on this. You can go to this page in the top-level nav, and you get sent to the user guide, and it has a sidebar right here which floats, and you can go to any of these pages, and we have user guide stuff there. And the cool thing is we have things like page tags right here, and lots of authoring tools for things like bits of code. This entire site uses Quarto. It basically builds on top of Quarto, so we get lots of Quarto affordances and authoring tools, and we basically build on top of that and enhance a few things as well.
Another thing is the function reference section, and here we have a reference item table preview inside of Great Docs. When you start on this page, you start with the index, and you can dig deeper, and you can specify how to organize your left sidebar with different groupings. You can specify the different methods inside one page, or you can split them out to different pages like this. And the cool thing about this is it gives your readers a lay of the land on this index page. You can have rich descriptions. You have these short summaries for individual methods or classes or what have you. It's kind of like a good sort of easing in instead of just being hit with the very first thing.
One more thing I want to show you is that we document all sorts of stuff. We document, of course, a Python API. We will document that as well, and it's pretty good. And more lately, we've been doing MCP as well. If you have an MCP server in this project, we will look at that and write those API pages out.
One more thing I want to show you is the changelog. I find this pretty important. It keeps users up to date. Essentially, what we do here is we look at your releases on GitHub and essentially create a page, and every time you fire off a build, it just looks for any updates to changes, like releases.
Documentation for the AI era
I'm thinking, just because of the age we're in, we now have to think a little bit more about documentation for the AI era because now your documentation is going to be read by agents, whether you like it or not. What we need to have are skills that agents can discover and load. We don't have to have that, but what we do is we offer a basic skill. If you want it, you can opt out of it. It's essentially just a very basic skill that we can write essentially just as a guide to your project.
But if you have custom author skills, like you've made the skills yourself, Great Docs will find that and actually publish that, and it's quite nice. If you have multiple things, you'll see there's actually some information here, and it'll show you all the components of each skill.
Okay, and I touched upon MCP reference. If you do have that, like an MCP server, it'll actually just go ahead, unless you opt out, and make MCP reference pages for you.
And this is a thing that you can also opt out of, but it's nice to have. Essentially, it is llms.txt and llms full doc text full API. You don't have to write this yourself. Great Docs will just do this. llms.txt, if you don't know it, is not a very big file. It's essentially like a sitemap. It's just like a very simple text file that agents sometimes look for. And llms full is essentially the whole API inline, and it's quite a bit bigger, obviously, depending on the size of the package, but LLMs do sometimes look for that as well.
So, I mentioned agents as the first readers. They're increasingly reading your docs before any human does, which changes a lot. If an agent can't parse it, your users will feel it in some small way, maybe a big way. There'll be more turns, longer conversations, maybe some bad decisions. So, in a small way, maybe in a big way, docs are sort of like an API service now, because decisions are made on your docs sometimes. And you're writing for humans and machines alike in this scenario.
Docs are sort of like an API service now, because decisions are made on your docs sometimes.
So, I came across this really interesting project, whatdoagency.runtype.app. It's a tool for getting an agent or simulating LLM looking at a site. You just pop in a URL, and I did so here with a Great Docs page. And it shows you whether it can find the things that would help an agent. So, it does find llms.txt, it does find llms full text, and it finds markdown pages, essentially a version of the page which is .md, so a markdown version of the page, which is useful.
Feature spotlight: Term Show
Moving aside from AI and agents, I have to confess I love features, and I stuffed a lot of them into Great Docs and tried to make each one good and useful and something that I would use. A big one is Term Show, and I made that because I was really frustrated with showing or demonstrating CLIs. The tools that were available were kind of good, but had major shortcomings, I thought. So I built a thing to help teach users your CLI or your TUI application.
CLI and TUI demos usually mean a GIF or a video. They can be huge, sometimes blurry because the text is not really well recorded or you're recording at a low bitrate. You often can't select the text and you often can't even pause the playback. So you want to rewind a bit, but you have to wait until the next go around if it's a GIF. Very frustrating.
So I built a thing where you just record your terminal activity, you get a file, and then there's also a paired YAML file. This is for configuration and presentation, essentially, like the way you want to show it. And as you imagine, it can get pretty crazy editing a YAML file to make it just right. So what I did was also I created an editor for this. It's a browser-based editor, and it allows you... It's almost like a little final cut for terminal presentations, and I'll show you that in the demo pretty soon.
What users, like viewers, see is sharp SVG animation playback. It's got its own full-featured player. It's a lot like a video player. And chapters, annotations, you can have those in there, and you can teach with that. You have all the tools to demonstrate what your CLI does and show workflows.
So, I've got a tab open right here called Term Show Editor, and here's a video I made a while back. And essentially, you can scrub around and see almost like the live video of your recording. And you can just add chapter marks. I can just double-click here, add a chapter mark, and I can label it new chapter, for instance.
You can add things like cuts. Say you don't want to see a bunch of the video, just double-click here, change the width of this, and you can cut out some of the extra dead space if you want. So, if I start playing back, it'll just jump over that, which is cool, because sometimes you just have a lot of, you just stutter over certain things, and your demonstration is a bit overlong. You might just want to cut bits out.
We also have something called snippets. What that does is it shows you in the top right corner a little button, and you can have text that's copyable. Say, for instance, the command. You can just have users click that, and they can grab that command and put it in their terminal if they need it. And it's shown at the right time and in the right place up here.
The thing I want to show you also is, this is nice, but what if you wanted to just see how it would look on a page? There's a preview button here, and what it does is it just goes back, of course, and does a render. And this is how it actually looks inside the player. You can pause it, scrub around, go to the chapter marker here, click here, and you see new chapter I just made. And this is essentially an unsaved state, so before you even commit it to save, you can just look at it right here and go to the end. You can change the player speed in case you like to go really fast. But this is kind of nice. Before you even commit to rendering it, you can make sure it's just right. So that, in a nutshell, is Term Show and the Term Show Editor.
Table Preview, Table Explorer, and Lightbox
Okay, so it doesn't end there in terms of features. There's lots of features. A big one I wanted — I'm really into tables. I have this project called Great Tables, so definitely tables is kind of my bag. But I think it's useful to have that in documentation sometimes, without importing Great Tables or some other table library. So I did something called Table Preview and Table Explorer. Two different things, but they're kind of cut from the same cloth.
Table Preview is kind of like a simple static table. It gives you a small preview of the data, usually like ten rows, or it could be the first five rows and the last five rows. And it shows you things like column types, and it's pretty smart about being wide, but not too wide, in terms of column width controls. And the really good thing about it also is that you can import from many different formats, like DataFrames, CSV files, JSONL, Feather files, Parquet. It's really great in terms of importing.
Okay. If you need a table that's a bit more interactive, full-fledged, Table Explorer might be what you need. So in this case, you have something that's JavaScript-enabled. It has all the things you might expect, like sortable columns. You can sort by multiple columns. It has buttons for downloading the filtered and sorted data, and a means to copy as CSV text. And the filtering interface — you just click the Add Filter button, and it presents to you a small interface which, depending on the column type, will present different options for filtering. And then they just appear as these tokens on the field right there, which you can remove, or you can add more. So it's really quite nice.
The lightbox thing is basically born from frustration of showing a few images, and I figure it's a nice thing to have in documentation. You never know when you'll need it until you need it. So what it does, basically, you click on an image, it brings you in this view, and it just makes it large, essentially. And if you have a gallery, you can just use the arrow keys to page through the gallery. And you can have dark mode, image swap, so you might have images for light and for dark mode, like a pair. You can have before and after comparisons, which basically presents to you this slider. And another cool thing is annotations. You can have certain regions of an image annotated, having dots and callouts.
And we offer things like, of course, zoom, the ability to copy image. You can download the image, and the really cool thing is getting a link to the image. So basically the link will just mean, if you provide that link to someone else and they follow it, it'll bring that person to exactly this lightbox view.
Closing thoughts
Okay, wrapping up. So I want to conclude with a few points. Great Docs. It tries to be nice by default, but powerful when you want more. I kind of believe there should be a tradeoff between an easy setup and having depth control. So Great Docs has sort of been designed to give you both.
Just a thought, I think we should treat documentation as a product, put a lot of weight on it. It's really what people and agents judge your project on before they really commit to using it. So it deserves some design depth and care.
Okay, this is a bit of a spicy statement, but I'm going to say it anyway. So your documentation is now kind of like an API for agents as much as people. So when an agent is sometimes their first reader, often, you might say, things like skills, mcp docs, llms.txt, they're kind of like table stakes now. They stop being extras and become required, you might say.
Great docs, they teach you. They don't just describe things. So if you have things like runnable examples, callouts, and guides, which are top level with your reference, you might even have terminal presentations, they're all there to actually teach your users.
And being good on mobile, it shouldn't be an afterthought. Docs should be just as good on a phone as they are on a desktop for human readers. This is why things like parameter tables, sometimes they're put in tables and you get a very long parameter on the left side and description that wraps over multiple lines on the right. Probably shouldn't do that on mobile. There's some thought there to make it read better and have better architecture of the information constrained with devices.
So I want to say thanks. Check out Great Docs. It's available in GitHub under positivegreatdocs. And if you do try it out, let me know how it goes. I'm really interested in people that do try it out, and I really want to help people if there's issues at all. Thank you.
Q&A discussion
Thank you so much, Rich, and thank you for the tour of Great Docs. Let's jump into our discussion. I'd love to open things up to our audience as well. So if you're in the audience listening in, then please go ahead. You can submit your questions in the Q&A widget, and we'll have them read aloud for Rich and our panel to discuss.
And James, I see that you have your hand up, so if you want to kick things off, go ahead.
Let me first premise this with two pieces of, I think, context. The first piece of context is I have recently been teaching myself the Zig programming language, and their docs are basically the worst you have ever seen. They're out of date, they're all machine-generated, they're impossible to read through, yet I still keep using the language, and the docs don't really slow me down that much, and I just kind of get used to writing code. The second piece of context is one thing that has always impressed me about the work of two of our colleagues, Tom and Dan, is that even for very new projects, I was always impressed by the volume of documentation that they would be able to put out. My question is, there's some space in between this really good docs and absolutely terrible docs, where the project is actually successful. How does one avoid what appears to me to be the vanity of writing docs, that's something you do for yourself, you make them look really beautiful, but avoid actually getting users, developing features? How do you find yourself not falling into that trap?
People talk about docs. Usually what I see are disparaging posts like, these docs are just the worst, but occasionally you see some validation, especially if you're the author. You feel good at that, you know that's extra press per, you know, like if it's hard enough to get traction in Python projects or any project. So I'll take a win wherever they can get it.
Well, actually it supports three of them, like NumPy doc, um, Google doc, um, is that right? Google form and, um, RST. So you choose one actually during introspection, it actually determines through a simple sort of a heuristic, what you're actually using and it'll just set that and then otherwise it kind of gets into like a quirks mode where it tries to correct, it does a pretty good job if you have mixed documentation styles. So I say, just choose one that's good. And another thing that Great Docs does if this is Great Docs specific, obviously, it'll actually run like examples. Like if you use sort of like this Quarto markdown sort of notation, basic code fences and such, which is like an extra thing on top. I think the key thing, just write docs, like complete, pick one style and like just go to town on the docs, I would say.
Fantastic. So it sounds like as long as you're using a common style, NumPy doc and Google docs are pretty standard.
Dan, did you have a question on top of that?
I have a comment and a question. I really like Term Show. That's super interesting. We do a lot of internal tutorials at NSLS two for staff, and we've talked about generating videos out of those so people could revisit them. But the problem with video content is it gets stale. I like the idea of a terminal script that is version controlled. So an API or best practice changes and I can just tweak it. I've seen versions of this, I think called ASCII cinema or whatever. And I really liked the full featuredness of Term Show. It seems like it gets it to a level of like actual usefulness. That's very cool.
Did you think about synchronizing it with audio ever?
Yes, I did think about that. So I left some space at the bottom of the Term Show editor for extra sort of like timelines for new things. Audio definitely want the extra thing. Basically it'll just be like a little snippets that you can like pull in and arrange. And of course, you know, it's huge, but I think it's important. It's a teachable thing, to have audio besides like just text callouts. And I think it might even be in the roadmap, which is sort of listed in the front page of Great Docs. So yes, I have thought of it and I think it's a great idea.
And I think we looked at the same sort of things like VHS and ASCII cinema. I've actually used VHS on a different site. It's okay, but it takes longer than you expect. It was just little GIFs or videos. It just runs your terminal with a script and then creates the recording and it takes like a good long while. But it's got the problem of like playback. It's basically just like running these GIFs, which are on repeat. I know you've solved a lot of the things that made it like almost good. Yeah. I mean, I'm glad those other things exist, but like, I thought, oh, we're not quite there yet. Maybe this I'll have a stab at it.
So then second part of my question, which is a bit blasphemous, suppose that we're heavily bought into Sphinx or heavily bought into Myst MD, but I wanted to embed Term Show in one of those two sites. Is that doable?
Not yet, but that's a plan as well. Right now it's only for Great Docs, which is a little unfortunate. I don't know why I did that. I maybe it's just like convenience and such. But I'm thinking I'm just going to have that be a separate thing, cause essentially it almost kind of is right. It just missing the exporting part essentially. And like the embedding part. Cause really you're just running Great Docs my Term Show and then like create essentially. And then the other little tools there. So the last missing pieces like is that, and yes, that'll be a thing pretty soon.
Rich, I had a question as I was looking through Great Docs. I have not yet used it myself for a project. We recently moved our open source project narwhals to use Zensicle, but I did see a little mention about the Great Docs gauntlet, and I was wondering if you could tell us just a little bit more.
Yes. So I take testing pretty seriously and to like maniacal level. So the gauntlet is essentially a lot of small Great Docs sites, basically rendered in serial and then creating their own page with a hub. And I have a separate repo for that and it builds daily. And it's a way of sort of seeing what fails, catching regressions with all sorts of possibilities, cause it's really hard to test like a large framework like this really well. You have to have something like basically a test harness or framework and that's basically what it is. And I try to build lots of dimensions of coverage into that little tool. So anybody can just see this. I have now a badge on the Great Docs repo itself that'll take you there. And it's a great tool for other people to see what can this thing do. And also it's a good demonstration that this thing is kind of tested. I think there's a little bit over 300 now. That covers a lot, but every time there's new features, a few of these have to be added.
Tanya, did you have a question as well?
Yeah, I think that this one did go through slightly during the talk, but I was wondering if you could elaborate a bit more about kind of like having now the documentation moving towards not only being human readable, but also machine readable. And you know, what are different strategies to evaluate both? Especially I think something that we will be super interested in will be kind of like that skill generation feature. In like how agents can make use of it. It's like, how can we evaluate how effective they are beyond them maybe just being machine readable?
Yeah. So the whole deal with skills being published there is kind of like, so if you have an agent look at the site, they can discover it on their own. You may not even know that just there's a skill there or a set of skills. And there's actually some, it could be versioned, but it might be just an extra thing that you have or your library itself. And yeah, it's quite nice. So you don't have to like, it's basically self-advertising. It goes into a special directory or a hidden directory called well-known and then skills. And I've done some like basic evals of this and different agents do tend to find it if you suggest to look for them. Sometimes you have to sort of lead it, but it's not quite great for discovery yet, but I'm trying to be a little bit ahead of the curve in terms of like having these out there. It's also actually kind of useful to have as well, just to have transparency on what the package has. So I'm of the mind, just publish everything on the doc site that could possibly be published and it's either going to be useful for an agent or for somebody else or for yourself down the line.
Piggybacking off of that, Rich, I was curious. Is that done just by serving both the markdown and HTML file just separately, and then the AI will get pointed towards the markdown file?
Yeah. You're talking about just like the standard pages, right? Having like a markdown version. Yeah, essentially we have buttons there to switch to the mode, but essentially each HTML page is paired with an md page. And so an agent might look for an md version of the page. It's starting to become more common. But if it doesn't do that, you can just point it towards it yourself in terms of just going to that, hitting the button where you can copy the text and just feed the agent. So you have a few avenues to get like a page's worth of content into an agent. It might be self-discovery or it might just be a little bit of feeding. But it's kind of like all there for all pages and you don't have to really worry about it being there, because it's automatically there unless you opt out.
Now I noticed that in your examples of good documentation, you put Matplotlib's documentation. We have Tom Caswell from Matplotlib. And so I was curious, what were some of those things that really make that good documentation stand out that you were striving for to make the defaults in Great Docs?
Yeah, I think a big thing, and this doesn't always get done, is examples, like rich examples with prose, you know, explain the example like before and after, like a good setup for an example, the code, the results, and then some additional commentary on that. I think that's overlooked and has been for so long. But it's really useful, really very useful, especially when it's as close as possible to the thing. And not only that, like having a few examples, I tend to go a little bit maximal on examples myself. You don't have to, but I think it makes, cause a user may just land on that API reference page, it shows them the thing, but then it shows a lot of examples and you can get your imagination going on what the thing that's above can actually do.
If I can jump in playing devil's advocate a bit, the problem we have in Matplotlib is that we say our docs are both too big and too small, and that it does not cover everything, but what is there is sufficiently large that it essentially becomes unapproachable. So how do you go about organizing, just considering something with a scale of Matplotlib gallery, how would you organize that to make it actually discoverable for a user that's not getting lucky from a search engine?
I think just sectioning is, you know, essentially I know it can go and it's huge and there's many topics, but there has to be some sort of way to section it such that there's some beginner level things and then you go a bit deeper in the middle and you end off going with more use cases type things. There's gotta be some ordering and sectioning, just make it palatable, those chunks.
You can have some things like what I found is good, it's having lots of cross-linking, like essentially interlinks between different topics and have that be prominent, like if related. Or you can have page tags as well, which I incorporate in Great Docs. So basically you can go to a separate page tags index, and then so long as the page tags are sufficiently divided up and specific enough and has enough pages underneath them, you can probably isolate the pages pretty well. The thing of last resort might be search, which is this keyword search that you actually end up in a pretty good state there, but I think tags like page tags might be the way forward, aside from interlinks and just good organization.
And James, I see you had a question. But Rich, can you explain what you mean by the difference between cross-linking and page tags there?
Cross-linking would just be essentially interlinks, like little links embedded in the text, the prose itself. Typically it might be in the middle of the page, it might be close to like a see also or see more on page. Or you might have tags, which are typically at the top of a page, and they're just like little things you can click and they take you to an index with similar pages, essentially like a sitemap or an index. And that might be good because you might want to go back and say, oh yeah, I want to see all the pages that have this tag. So that's another useful way of organizing things.
Yeah, I have a follow-up to Tom's question, because I know Tom, I have joked with you more than once that I simply don't ever read documentation, but that's not strictly true. In fact, I end up reading an enormous amount of the units of documentation. I never, unlike I suspect you and your colleagues, sit down at the beach with a manual or a man page in hand and read it linearly. I consume these things in an extraordinarily nonlinear fashion, which is why for me as a user, my entry point into the Matplotlib documentation is either one, I'm going from Google, or these days I'm asking LLM to find the articles for me and then going through that.
From that perspective, the organization of things like the gallery is honestly completely irrelevant to me because I'm never going to use that organization. I'm going to reconstitute it myself. What also gives me a little bit of a pause is when we're talking about the LLM versus human use cases here, it seems that we're drawing a very unimaginative contrast between two very strict modalities. A human being is reading these documentation that a human being wrote in the way that we assume that people read these, like printing it out and reading it on the beach, or the LLM is doing it automatically. And I wonder if there's not some middle ground by which the LLM can serve as an automation tool to create a customized nonlinear pathway through the documentation so that the organizational structure is no longer built into the way in which the documentation has been written, but it's constructed by the LLM as part of some kind of learning process.
I think one thing about that — that's a fair point about the search engine or the robot giving you links deep into the docs, but what you lose there is the, I don't know how to do anything with the tools. So that means for search to have worked, you typed words at a text prompt, you've actually described that plot with the same words we use to describe that plot. Whereas if you like go to our gallery and just hit your scroll wheel and let it run, you can very quickly see like, oh, that's the one I want. And that serendipity of, I don't know the words for it, but like I know it when I see it, is something that I think is important not to lose, particularly for plotting tools.
I can probably speak on both of these. So on earlier packages I made, I knew people would land on different pages, like of the reference API. So my solution there was to, for each of those pages, no matter how small the little thing was, I would have examples there too, because they show you the end product and you can work your way backwards. If you like the end product and this page is the page for you and you land on the right page and you have some working code there as well. So I resolved to do that. It was very time consuming to write all these examples. But it was worth it because you never know where people land and they may just bounce and give up if they don't find anything right away. So you have to anticipate people will land in the weirdest spots. You should show something useful, I think, in every spot they land.
You have to anticipate people will land in the weirdest spots. You should show something useful, I think, in every spot they land.
And in terms of an LLM helping it through, I mean, these days it does a pretty good job finding where it has to go. At least when I use agents, it does find things pretty fast and it does bounce around a little bit, but it does eventually land on the right things. And depending on the documentation it may have to go into the source code, but usually the documentation does enough and it's truthful and current. It's even better. It doesn't have to stumble over the documentation.
On Tom's point that there is some necessary organizational structure to these documentation, I actually agree quite a bit. Cause if you think about the non-linearity, it's not a random order in which I consume the units of documentation. What it is, is it's an order that is subordinated under desiring to build like a world model for how this works. So for Matplotlib, I'll try and figure out what are the boundaries of the core concepts, and I'll look for pieces that then allow me to reconstruct that. Cause usually for tools like pandas and Matplotlib, the ideas of the library are very rarely explicated. Like the pandas documentation doesn't really highlight index alignment. The Matplotlib documentation probably talks about artists and things like that, but not in a way that surfaces it in any, you know, the assumption is nobody actually cares, but in fact, this is what I care about.
As a consequence, when we look at the structure of the docs, the docs are usually structured in a very kind of reference based authoritative fashion. But in truth, the non-linearity arises because what I want to do is find disparate examples that then allow me to formulate hypotheses about the underlying model, test those hypotheses, and then build that such that I no longer am reliant on the documentation. So that if you have methods A, B and C, I learned method A, I learned method C and I never read the documentation for method B because I can guess how method B works from the model I've created.
