Danielle Navarro | Toward a grammar of psychological experiments | RStudio (2020)
Why does a psychological scientist learn a programming language? While motivations are many and varied the two most prominent are data analysis and data collection. The R programming language is well placed to address the first need, but there are fewer options for programming behavioural experiments within the R ecosystem. The simplest experimental designs can be recast as surveys, for which there are many options, but studies in cognitive psychology, psychophysics or developmental psychology typically require more flexibility. In this talk I outline the design principles behind xprmntr, an R package that provides wrappers to the a javascript library (jsPsych) for constructing web based psychology experiments and uses the plumber package to call server side R code as needed. In doing so, I discuss limitations to the current implementation and what a "grammar of experiments" might look like
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hi, so yes, I am Daniel Navarro. I'm a psychologist at the University of New South Wales and I'm going to talk today about a project that I've been doing on building an R package that will let you write behavioural experiments in R. And it's not the sort of thing that maybe huge chunks of the R user community need, but I'm going to explain why it's really important to a subset of users and how it really matters a lot in my work.
So I should start by saying something about who I am. So this is me. I have like a website thing, like everybody does, and really the reason that most people pay any attention to me is side projects, very much like what I'm talking about. Most of mine involve art and I usually do a whole bunch of generative artwork with R, but in my spare time I try to act as an academic and so I do extremely boring things like write academic papers, but in between doing that I sneak in some R packages, which gives me something useful to do with my time. But all of that, and kind of reinforcing in some ways Ryan's message in his talk, is that playing around with these things taught me a whole lot of useful stuff and it got me to the point where I was able to start writing a textbook.
So I wrote an introductory statistics textbook, pitched it, undergraduate psychology students, not typically the people most interested in this kind of stuff. And it was reasonably successful to the point where I released it under a creative commons license and it got translated into other programming languages and apparently now it's being translated into other actual languages, like French and Japanese and somebody's doing a Spanish one and it's very exciting and I don't know why everyone likes it this much, but I'm very happy.
The case for writing experiments in R
Okay. That wasn't just bragging. It also brings me to sort of the point, right? The thing that I want to talk about writing is, there it is, okay, is a package that will allow my students and my collaborators to be able to write flexible behavioral experiments in the R programming language. And to put it in some context, this is how behavioral experiments were typically done in the early 20th century. So you have a machine like the one shown on the right, the Skinner box, and we would have our subject, who is typically a rat, and there would be shown stimuli, which might be lights or sound or something, and so they're delivered by some mechanism and then they're rewarded with food, punished with, you know, by electric shocks and things like that. I'm not claiming psychologists are good people.
In fact, I might go so far as to say we're probably really bad people. But this is the nature of how things used to be done. And in many places still is. But it's much more common these days in the early 21st century that the machinery that we're using is either a laptop, if you're doing it, if you're running something locally, or you might be running experiments over the web, which is now extremely common in psychology to the point that it's causing problems. But that then means that your experimental design looks kind of like the code that I've got written there on the left. And in this case, this is something from one of my own experiments, and it's written in JavaScript, and I hated it. The experiment, actually, the experiment itself worked fine. It was really nice psychology. It's terrible JavaScript, because I'm not a great JavaScript programmer, and I hate JavaScript. I hate it. I mean, I understand why people like it, what it's useful for, but I hate it. And more to the point, my students and a lot of my colleagues hate it, because it forces them to learn another programming language at a time that they don't want to.
So think about this from the perspective of the kinds of people I work with most of the time. And to them, this is how R looks. They can walk into my office and say, okay, Danielle, I've heard really good things about R. Explain to me why I should invest the time. And it's really, really easy to make the case that R is brilliant for statistical data analysis, it's great for data visualization, data wrangling, general programming, and they get excited. They are really, really keen on putting the effort in. And every single time, the next question is, can I write my experiments in this? Because psychology, being an experimental science, is deeply reliant on the ability to do this. And this is where I start crying, and my students tell me, I'm out of here. They are not pleased to discover that they're going to have to learn JavaScript to do that job.
They are really, really keen on putting the effort in. And every single time, the next question is, can I write my experiments in this? Because psychology, being an experimental science, is deeply reliant on the ability to do this. And this is where I start crying, and my students tell me, I'm out of here.
So what do other languages do to deal with this problem? Well, Python has a framework for dealing with it, PsychoPy. Matlab has a framework, a site toolbox that a lot of people use. R doesn't have any clear analogs to this at the moment, and a lot of people don't want to switch from Matlab to R because of it. Or when they're thinking of making a switch, they will move to Python instead. And I genuinely do love Python, so I don't feel sad about that. But I also think that R is a great thing for a whole lot of purposes, so can we do something this way? So that's my goal.
Exploring options: native R and Shiny
My first idea is I'm going to do it natively in R, right? I mean, why not? If I've got a system that allows me to open up an X11 graphics device, this is interactive. It will respond to keyboard presses and mouse clicks and things like that. So this is a really simple psychology experiment. You show people these visual arrays, and your task in this case would be judge whether there are more triangles pointing up or pointing down. And there are models of this, like quite complicated models that go back decades, and people care about low-level features in this data. They care about it a lot.
So, okay, the plus side, before I complain, is that it's actually really easy to write these experiments, right? You just have to learn one or two new commands, and they're sitting right there. Get graphics event, once you know that, you can do quite a bit. So it's surprisingly easy to code. There are a few dependencies, but it does have some problems. You can't do it online easily. The interactivity is limited. I can't get my participants to type in a paragraph of text. Occasionally, psychologists have this wacky idea of asking people, why did you do that? And this is really common when you go, I don't understand my results. I'm going to replicate the experiment, and then I'm going to ask people to explain what they did wrong. So I need that. And that's missing. And ultimately, this is not really what R is built for, and as a general philosophy, I feel like working against your programming language is always a bad idea.
Shiny. Everyone loves Shiny. I love Shiny. This is the very first attempt I had to make a psychology experiment within Shiny, and so this would be a belief elicitation task. Sometimes you ask people to give interval estimates. So I've got to say, give a range of plausible values, you know, an analog of a confidence interval here, for people to judge various, in this case, nonsense propositions, right? And so I click my submit button. It then refreshes, gives me a new trial, and so on. And okay. I can do this. This is a reasonable thing for Shiny. Though notice, you know, this is a real app that's running in the background here, and it's running on shinyapps.io, and you'll notice it took quite a while to set up, right? Because I'm not doing anything fancy to make this thing scale.
In order to build it, here's the brief tutorial that I wrote for my students. Hold on. Now we're getting to fluid page. Oh, my God. It's having data. Oh, and then at the end, you've got something where you ask people to judge how strong a correlation is. And my students are gone. I can't even scroll down to the bottom of the page in time for the students to be willing to do it. Not a single one of my students has done this, because Shiny, as magnificent as it is, has this learning curve.
So, okay. Shiny, it works online. It's got very flexible interactivity, which we need for behavioral experiments, but it has these shortcomings. It's hard for novices, and if you've got a novice, and you have to hit them, and they want to run their experiment online on a tool like, say, Amazon Mechanical Turk, they need their app to scale to not very large scales, but still moderately large. Like, we're talking hundreds of concurrent users. And I do not want my undergraduate psychology students trying to do that.
Introducing xprmntr: design goals
So, here's my approach to dealing with that. I'm going to give you the gist of how I do it. Here's what I want. It has to be written in R code, because I am not trying to teach two programming languages to psychology undergraduates simultaneously. It needs to be relatively easy for novices to learn. The experiments need to be browser-based, because that's where most of the stuff is these days in psychology. So, because we're going to care about high-precision timing, a lot of the computation needs to be put client-side. So, it has to be done with client-side JavaScript. You need proper randomization, flow control, programming constructs, the sort of things that we would hope to have in any programming language, of course. And it needs to support rich experimental design. So, richer than surveys. We're not just doing surveys here.
Okay. So, I found, like, a friend of mine has actually written a JavaScript library that does almost everything I want. It has only one problem. It's JavaScript. Okay. So, I'm crying again because of JavaScript. So, what I've ended up doing is starting... This is a work in progress, but I'm basically writing a package called jscir for j, sci, r, psychology. Yeah, I'm funny. So, you code in R. The experiment then builds to jsci. So, it will compile it. It will write all of your JavaScript files for you. And there's a very, very thin R server at the back end that doesn't do very much, but it's constructed using plumber that will handle back-end stuff when you need it.
And then when you want to run the experiment, you can either deploy it locally, running it on your own laptop, or you can push it to Google App Engine or whatever your favorite cloud service is. I use App Engine, so that's where I've started. So far, this is the sort of functionality we've got. So, here is a super brief illustration of the sorts of things you can do with the package. It's not going to be a tutorial in doing it. I'm not going to explain code in a ton of detail other than to sort of illustrate that it's not spectacularly complicated.
Demo: what the package can do
So, load the package. Okay. Experiments in psychology will typically begin with an instruction set. So, you've got text here that can be navigated. You will then specify some trials. So, the trial here might display, in this case, some HTML, and it's just going to print out some text. I'm doing the dumbest thing I possibly can. Participants have to respond with a button, and they're just going to evaluate whether an equation is true or false. And then you build the experiment using a function called build experiment. And that's the entire code right there for a functioning experiment. And here it is. It's currently running on my GitHub's pages site. Notice it loads immediately, which is nice. So, you can sort of go back and forth. Like, instruction sets need to be navigable and all that kind of stuff. And now we begin the experiment. I don't know. Is 13 plus 23 36? Sure. Why not? I'm a participant, so I'm not paying any attention anymore, so I'm just pushing buttons at random. And all of this stuff is up on the GitHub page, so you can actually go and browse all of this stuff over there.
Generally, of course, we want richer things, so there's ways of including different kinds of media files. So, we'll support resource files of various different kinds, images, audio files, or video, that kind of thing. So, you might show people a picture of some generative art and get them to rate whether or not they like it. And we'll say, no, I don't like that one. Sorry. Or you could get them, you know, this is me teaching myself GG animate and playing around with it. Do you like that one? Okay. That's pleasant. And then this, you know, will also do you probably can't hear it, but this is playing sound. So, yep. And I would say that's pleasant and then ask it to shut up.
So, you can do that. You can change the response method. And the API is fairly consistent for that. So, hey, if you want people to respond with a keyboard, the command says keyboard. If you want them to respond with a slider bar, it says slider. This is supposed to be fairly straightforward and easy for novices to use. Right? So, again, there's a demo of it, but I'm just going to skip it because it's exactly as interesting as you'd expect.
You can write survey pages. So, if you need something richer where people actually respond to surveys, then you've got some mechanics for that. And so, you can do things like, oh, okay, so I'll say that. Do I consider myself to be LGBTQ? Sure. I'll click on that. You can have things where you'll choose as many as apply. So, if I want dplyr and per, whatever. And you can fill out scale surveys as well, which are extremely common in psychology. So, if you need people to rate things like that. And that's all being done within, again, the same framework.
At the back end, then, when everything gets written, it's just a plain CSV file, nothing fancy. So, that means that my students, when I teach them dplyr, I can do that. They will be able to read this in. And it will fit in with the existing teaching framework and the learning framework for them. Or, you know, if you need it, it will deploy to Google App Engine and write stuff to the data store there.
But I'm not going to talk about that. I will very briefly point out that it does allow proper flow control and programming constructs. So, there are ways of basically saying that, well, suppose if my first question had been, do you identify as LGBTIQ? And I click no. I really shouldn't have a follow-up question that asks you to specify exactly which category you belong to. So, there's some logic in there for modifying. I'll just highlight the one bit that actually matters. There's a timeline display if. So, as long as a particular condition is met, then that trial will be displayed or that block of trials will be displayed. If it's not, then it doesn't show. So, you've got your if statements. There's a similar logic that will let you do looped execution with a command called, oddly enough, TL display while. So, it's got the equivalent of a while loop.
Limitations and the road ahead
And so, the things, firstly, they've got all the usual complaints. The whole thing's a work in progress. It's janky. Okay? Like, that's part of the reason it's a work in progress. But there's a deeper issue here, which has to do with the flexibility of behavioral experiments.
Here is a subset of the different trial types supported. It's fairly long. And if you look at it on some of the pages, you start getting worried because this is a very, very small subset of the things a psychologist might want to use. And the problem ultimately is trials, this is whole events where you show something, make a person do something, respond to them, are not primitive objects. Right? And so, when you look at what's happening in the JS site library itself, you get this massive proliferation of experiment-specific plug-ins that are all written in JavaScript. And they do have a plug-in framework, which means you can do this. But honestly, if you're not a linguist, you don't want closed trials. If you're not a social psychologist, you don't want implicit association tests. If you're not interested in perception of monkeys, then you're not really that interested in random dot motion tasks. And honestly, unless you are Richard Aslan specifically, you do not need a visual statistical learning animate occlusion trial or a visual statistical learning grid scene trial. I am not going to write plug-ins for every single one of those things. No. These are not going to happen. So, this is a problem that needs to be overcome.
So, the direction I'm planning on heading with all of this is actually to look at decomposing trials into smaller parts. Right? And so, that you can build at build time, have it construct the task-specific plug-in that you need. And so, the logic is something similar to things that we talk about in the art community a lot. Right? We know this stuff from things like the grammar of graphics. You don't think of plots as primitive objects. Instead, you think of them as composed of parts. And those parts are reusable. So, you've got your data, your aesthetics, layers, scales, coordinates, themes. And each of those things can be specified individually. So, that the range of plots that ggplot2 can support is much broader than it's much wider than the number of actual functions in the package.
So, this is this open question that I'm working on at the moment is thinking about what does the logic of that look like for psychological experiments. And that's where things start becoming messy. Right? I mean, I'm not going to go into details. And this is probably not the sort of room where everybody cares about it. But these sorts of images are grossly typical in the psychological literature. And this is simply displaying a stimulus. Like, this is one thing that is shown to a person over time. So, your core element is some kind of spatiotemporal object at its base. And that's only one part of what a trial might be. So, you've got something like primitive elements, which might be images, audio, or snippets of HTML. Those things can be composed together into stimuli in some fashion. And so, you need a set of rules for combining those. But you also need a similar kind of logic that will handle the responses. And so, exactly what method will the human use to respond? So, in our traditional Skinner boxes, that was things like the rat presses a bar. But humans are slightly more complicated than rats. And so, we tend to have things like text boxes and HTML button presses. And we can move the mouse around. And you could support all sorts of other devices if you really wanted to. And so on and so forth.
So, at the moment, where I am at is actually thinking about what that decomposition looks like and what kind of future package I'm going to build over the top of it in order to support the full range of experiments that my users require.
But right now, though, despite these limitations and grumbles, the package that I have solves a problem that a lot of my colleagues, my students, and my friends have. And so, that makes me pretty happy. And so, I will leave it at that.
Q&A: getting started with generative art in R
Okay. Looks like we have one question about your artwork, Danielle. Do you have any advice for how to get started with a generative art in R?
I do, actually. So, I mean, the first one is have fun. Play. Like a huge part of it is a little bit like everything that Ryan said. Go do fun stuff. But the other, you know, besides that sort of generic going and playing around, I actually so I should be able to find it on my website if I just scroll right back to the beginning. Okay. So, if I go over here under the blog part, I actually wrote a little is it going to let me do this? Oh, I have to bring this back up again just a moment. And I actually because I write random things from time to time that people will find interesting, I actually wrote a little blog post that covers exactly this. So, the point of the post here is actually just to talk about the this was me on the fly building a new generative art system. And so, it was kind of tracking some successes and failures and how to actually build something that eventually generated me some interesting images like that. And that. So, almost everything that I would say as my advice for getting started is have a like I already wrote it down. So, you can go and read that. And it also has sort of hints about what you do to get started and start playing with these things. But playing is pretty much the key thing. Playing around and making a lot of screw ups.
