maidr: Empowering Accessible, Multimodal Data Visualizations (JooYoung Seo) | posit::conf(2025)
maidr: Empowering Accessible, Multimodal Data Visualizations Speaker(s): JooYoung Seo Abstract: maidr is a Python package that transforms visualizations into accessible multimodal representations. Designed for both blind/low-vision and sighted users, it integrates with libraries like matplotlib and seaborn and supports interactive environments such as Quarto, Jupyter Notebooks, Google Colab, Streamlit, and Shiny. With a simple function call, maidr synchronizes visual, tactile (Braille), textual, audible (sonification), and conversational outputs, enabling reproducible, inclusive workflows. By creating, modifying and sharing accessible visualizations, maidr fosters collaborative insights by bridging the gap between blind/low-vision and sighted users, empowering inclusive, data-driven decision-making that leverages diverse perspectives. GitHub Repo - https://github.com/xability/py-maidr posit::conf(2025) Subscribe to posit::conf updates: https://posit.co/about/subscription-management/
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Good afternoon. I'm JooYoung Seo. I'm an assistant professor in the School of Information Sciences at the University of Illinois Urbana-Champaign, and I'm also the director of Accessibility Design Lab, where I design and develop data science tools, inclusive data science tools and practices with my amazing students and collaborators. And there's one more thing about me.
I'm a blind professor who teaches data science and data visualization to my sighted students.
So if that makes you pause, good, because it reveals an assumption that vision is the only reliable path to insight. Over the next 20 minutes, I want to challenge that premise.
So if that makes you pause, good, because it reveals an assumption that vision is the only reliable path to insight. Over the next 20 minutes, I want to challenge that premise.
So let's think about it. What good is a chart that you can't see?
How do you read a chart that you can't see? So now when you hear these two questions, you might feel a bit puzzled. That's because the data visualization inherently assumes that you assumes that you have the ability to see. We often can't separate the idea of visual ability from how we think about data visualization.
For sighted users, data visualization is an incredible, powerful tool.
So there's that saying, a picture is worth a thousand words. And through data visualization, you can get instant insights and a much clearer understanding of complex data. It's a fantastic communication tool. But what about users with visual impairments just like me? For them, data visualization that assumes visual ability is like a wall of inaccessible information. It's a locked door behind which valuable insights are trapped.
In that sense, data visualization becomes a barrier to understanding for users with visual impairments. So that's why I like to call this as data visualization divide. In our data science community, we've long tackled this issue with what I call a design for them approach. So basically, a sighted person creates a visual chart. And then as an afterthought, they might add a separate text description, also known as alternative text, alt text, for blind and screen reader users. But the problem with this design for them approach is that it creates just one way street problem. So one group produces and sets the boundaries while the other group just consumes. Just consume whatever is handed to them.
So this kind of power imbalance means that blind or screen reader users are just stuck within the boundaries defined by sighted creators.
Shifting to a design for us approach
So now, I think it's time for us to shift from the conventional design for them approach to a, what I call design for us approach. So designing for everyone and making a two-way street for everyone is a collaborator. So by the time I say this, you may think, well, that sounds like a great idea, but it also sounds a bit, a little bit idealistic. How are you going to actually do this? So that's a big question. So well, today, I'm going to introduce you the solution.
The solution is called MATER. It's a design framework that I've been working on for the past few years with my amazing colleagues as students at the University of Illinois. MATER is a multi-modal access and interactive data representation framework. It decouples a vision-first assumption from data interpretation, and it encodes multi-modalities into the data. So it doesn't create a separate or watered-down version for only for screen reader or blind users. Instead, everyone uses the same tool, unified tool, and experience the same insight.
So it doesn't give up on visualization. Instead, it builds on top of the visualization by encoding multi-modalities like touchable braille, readable text, audible sonification, and conversational AI. So let's take a look at each of the features.
So first, you can fill the data using touchable braille, touchable representation. So what that means is that we can represent data patterns in Unicode braille. So for users with a braille display, refreshable braille display, they can actually fill the peaks and valleys of the data with their fingertips. As presented on the table on the screen, the idea is that as the data values increase, we use higher braille patterns. And as the values decrease, we use lower braille patterns.
So this screen I'm showing you, it demonstrates how it works. And in the bottom right corner, you can see the hardware that people with visual impairments use. It's called a refreshable braille display that can represent the braille Unicode pattern physically so that they can feel through a fingertip.
So next, we can listen to the data with audible representation. So this technique is called sonification, data sonification. What that means is that we can map data pattern into musical representation. So for example, we can map the x-axis to left to right stereo panning. And we can represent increasing or decreasing y values pattern with different pitches. So here's an example. On the left side of the screen, you may see a conventional, a strong positive correlation between x and y variable.
So you may hear the gradually increasing tone from left to right that can represent the positive correlation. Next, we can understand the data with readable text. This one is simple and straightforward. So what that means is that we can provide summaries or values in text format.
And finally, we can chat with data using conversational AI and everyday large language model like GPT or Cloud and Jebedi.
MATER in action
Okay, so I've explained all about MATER theoretically.
Beyond theory, let's take a look and let's see how MATER works. So MATER in action.
So today, I'm so proud to be able to introduce a Python implementation of MATER. I'm sorry for our lovers. But here are three very simple steps to create a multimodal representation. Step one, import MATER package, which is available in PyPI. And step two, create your plot using Matplotlib or Seaborn. And step three, show using MATER.show, just like how you do using PLT.show.
And just like this Python code. So part one and part three, these are the two simple changes you need to introduce to your existing code. So you may have a question, oh, are you kidding? No kidding. Actually, this is really all it takes. And once you use these two simple lines of code, your existing Matplotlib and Seaborn chart will be automatically translated into a multimodal chart.
So this is just a keyboard shortcut key. Once it's rendered into your web browser, you can toggle on and off braille using single letter key before braille. And T for text and S for sonification. And question mark for AI chat.
So let's take a look at some examples.
So we have a conventional multiline example written in Matplotlib. And once it's rendered, it creates interactive SVG chart like this. So I'm going to switch my sound card so that you can hear my screen reader sound as well. So I'm going to use left to right arrow key.
So it has three lines. Population index. Life expectancy index. GDP per capita. So each line has distinct audio.
So that blind people can see here with a musical note. And then it has braille representation. I just press B. So when I move around, the braille representation is synced with a visible highlight. And I can even have a conversation.
So when I press question mark, describe this to a person and hit enter.
So we send the same prompt to two different models. AI, open AI, GPT, and Gemini. And even users can configure the model. The reason is that blind people cannot easily verify, so that by giving them multiple responses, we wanted them to have more informed decision.
So what about box plot? So box plot is also rendered in the multimodal format.
And this is a braille. And you can compare GDP per capita between Africa and Europe using different tones. So this one has a higher tone.
And what about multi-layer? So this is a histogram having two layers. The second layer has KD, kernel distribution estimation. And we can move between layers by pressing page up or page down. So this is the first layer.
This is second layer. So second layer, first layer. And all layer has braille representation as well. What about more complicated example like multi-panel subplot? Yes, it's completely accessible. So I can move around between subplots. Shiny? Yes. It's supported.
So we have added custom render, made a render. So once it's rendered, Shiny is made fully accessible for the matplotlib. So you may wonder how this is possible, what's happening beyond this message. We use three different layers. So we capture, we intercept the matplotlib and Seaborn, and we extract their data, raw data, and we bake the raw data into SVG. And with audible sonification, braille pattern, and we have developed our core JavaScript engine to make it accessible.
Performance and compatibility
So you may wonder whether there is any performance issue when we use this monkey patching approach. Actually, the performance overhead is very manageable. So you can ignore any overhead. Because for matplotlib, it's only 1.48 millisecond on average with mater. So this is literally nothing. So millisecond. So for matplotlib, it's 1.48. For Seaborn, it's only 3.26 millisecond overhead. So practically, this is nothing.
So as I demonstrate today, it's not only working for matplotlib and Seaborn vanilla code, but it supports reactive dashboard like Shiny and Streamlit. And what about reproducible document? Absolutely, it supports Quarto. Because I made this slide with Quarto. And what about interactive computing? Interactive computing like Jupyter Notebook and Google Colab? Yes, that's supported.
So it's time for us to build the future together. So mater is not just about tool. It's more than tool. So this is about different perspective. So it's about fundamental shift in how we think about communication and visualization.
So mater is not just about tool. It's more than tool. So this is about different perspective. So it's about fundamental shift in how we think about communication and visualization.
So please build the future together. You can start today. You can just pip install mater in your terminal. And its source code, open source contribution is available on GitHub. And its user manual is also available at py.mater.ai.
