Hacking Productivity with LLMs (Nic Crane, NC Data Labs) | posit::conf(2025)

Transcript#

This transcript was generated automatically and may contain errors.

So I'm going to be talking about using LLMs to hack productivity, and some experiments I've been doing over the past 18 months. I was thinking recently about a job I did during my career. I was working on a project, and as part of the project, I had the tedious task of taking a database export as a CSV, converting it to Excel, applying various sorts and filters, getting them just right, and sending it to the stakeholders.

The work itself was tedious, and it was really boring. It was genuinely repetitive, and if things went wrong, I had to start from the beginning again. It was putting me at serious risk of ILD, Imminent Laptop Defenestration. But rather than throwing my laptop out the window, I started thinking about something more genuinely useful. I created a Shiny app, and the Shiny app connected to the database. The users could apply the filters and things they needed themselves, and I didn't have to do that tedious work every day.

So I was happy, I could focus on more interesting things, and they were happy. They didn't have to come and bother me at that point. And actually, the thing with creating apps and things to automate stuff, there's always the risk that the solution you come up with, and the time you spend putting it together, can take a lot more time than the underlying task would have in the first place. And I think, actually, that's something a lot of us are thinking about right now in terms of AI.

And so, as Cat Hicks wrote in her keynote the other day, it's not just what these tools can do, but actually our experience of interacting with them. And that's something that has really resonated with me. I've spent a lot of time, over the past 18 months, experimenting with LLMs, and trying to increase productivity in different ways. Some of that has been really helpful, and some of it I've committed whole libraries to something that hasn't worked.

So I'm going to talk to you about my experience of that, and there are three questions I always ask myself now when I'm deciding whether to use an LLM in a task or not.

How difficult will the task be?

So the first question is, how difficult will the task be? I'm going to illustrate this by talking about an R package I've been working on recently. So the package itself is called Daisy UI Widget. It's an HTML widgets package, and that means it uses HTML, JavaScript, CSS and R, and the users interact with the package in R. To work with the package, the user creates a data frame with dates and comments. They call the function timeline, and they get a timeline, like the one on the screen here.

So I decided, when I was starting this project, to see if I could build it with LLM help. So I used GitHub Copilot, but I ended up having to pump or work to get the code. So how did the process go? So if the task is fairly easy or straightforward, you can get started and generate some code, but when we get to the point of getting to the quick question, things will work fine there.

So at this point, I had some of the package, but a friend was helping me to develop it, and I wanted to see if LLMs could help. So I started moving forward on task two. So the AI slot I'd gotten out of GitHub Copilot had come up with this way of working with the package, where the function Daisy was taking a list of lists as input. But actually, for things like this, it's more conventional to have a data frame as input. So what I was doing here was changing the function from list of lists to a data frame. So it felt like a task. I could understand how to do it myself. It wasn't going to take a lot of time, but I wanted to see what would happen if I got an LLM to do it.

So, how did it go? It was pretty good. I ended up with this incredible code that was full of paths I'd never seen before. The code came with a lot of tests. For that small amount of code, I'd never seen so many tests associated with it. The code felt really important to get to fit together. The code was also interesting. So the code worked and it did what it said. But when I went through the code, there was a lot to take in. I feel like code review is more of a social exercise. It can be enjoyable to connect with people. But I kept coming back to asking, was the code going to be used in the right way? Was the other model going to use it? Was the data it was trained on right? Was that code really good?

So if the task is fairly easy, it feels like it takes more time, if the task is fairly hard, I can't really engage with the code, but actually if it's more difficult, that's an interesting place. And actually, when I framed it this way, I realized a model to develop and grow. So I was going to see if I could use an LLM to teach me how to do it myself, which brings me to the first question.

What role should the LLM play?

What role would the LLM play? So at this point, I went and looked up some of the literature on using LLMs in education. And I found a paper by Dr. C. Lynn about how ChatGPT can help as a self-directed learning reviewer. So in this paper, Dr. Lynn goes through this framework for learning and things specifically that ChatGPT can help with. So the first two in the framework are establishing the learning goals, locating and accessing resources, and then adopting and executing learning activities. So I decided to try this out.

So to go back to that task, the stretch zone task. It felt like the perfect stretch task to use here, because I had experience of the DT task. I work with the DT package. So the DT package is a hard HTML widgets task. It renders tables and it has functionality that you can use in the framework and build things from that. So I've worked on this from the outside. And I know a little about Javascript, but at this point, I didn't know very much about HTML widgets at all. And I didn't have experience of using reactive events in Javascript. So it seemed like a stretch task.

So task 1. Establishing learning goals. I like to think about this in terms of how I would have done it if I didn't have LLM help. So I think I would have said, okay, it's obvious to use the DT package, okay, and just gone with that. But actually, when I asked ChatGPT, it asked me a number of questions. It asked me exactly what I was trying to achieve, how long I wanted to spend on it, and what past experience I had. And from my answers to the questions, it ended up coming with this really clear and concise goal. And it broke it down into three practical steps. So, capturing the event in Javascript, then passing that to Shiny, and then testing it in an app. Now, could I have done this myself? Sure. But was I likely to? Yeah, probably not. So it just left me feeling like I really knew exactly what I was trying to achieve here.

So the second step, then, locating and accessing resources. So, again, like, doing this on my own, I'd probably end up Googling things and have, like, 50 browser tabs open and end up down a rabbit hole of kind of reading about a slightly different component. But with ChatGPT help, I had a couple of questions. So how do I prefer learning? I said I prefer learning by doing. I like having kind of snippets of code that I can run and see what happens and play with. And it found me really quickly just two resources that were perfect for the task at hand. So, firstly, the book Javascript for R, which has a free copy available online. And secondly, a blog post from 2016 by Dina Tali that's all about more or less exactly what I was trying to do. So, great. And I guess this had stopped me kind of having to switch between kind of planning and organizing and coding, and I could just focus on reading up and writing some code.

So then, finally, the third step for this was this idea of adopting and executing learning activities. So there wasn't much back and forth between me and ChatGPT at this point, because we kind of talked a lot about kind of my past experience and things, but it came up with this three-step process I was going to follow. Firstly, I was going to have a look in the DT package and kind of try to reverse-engineer the code in there. So seeing if I could find the code that actually implemented that row selection to learn more about exactly how that worked. Secondly, build a minimal working demo, so in a tiny little Shiny app, just showing capturing a user event. And then thirdly, add it to the actual package.

But it wasn't just like this three-step process that was useful. Around this, it did things like it gave me time boxes, so I didn't end up spending too long on each step. It also had useful little bits and pieces, so with the bit around reverse-engineering DT, it actually gave me a link to the exact place in the package that I needed to find the code to read, which was really helpful, not having to track that down myself. And it just feels like a lot of these things are all about reducing the level of frustration that we can end up having when we're trying to figure out a new thing.

But even better than that, it also suggested ways that I could use ChatGPT to help me with these. So for example, when I was reverse-engineering DT, it mentioned, okay, any code you don't understand, paste it in, and we'll go through step by step. And again, that was super useful. When I was building my minimal demo, the first time I ran it, it wasn't quite right, and just being able to paste it straight in and figure out what wasn't working, again, just reduced that frustration. And then when I added the event tracking to the app, we ended up in a situation where it wasn't working, and I was asking why, and the LLM actually asked me, well, what do you think you should need to click on in this situation? And it really felt like a really great just learning activity and being tutored on something.

And so I deem this definitely a success. There's a write-up on my blog as well with more detail if you're interested. And I feel like, for me, it was just a really enjoyable learning experience because I feel like I'd done the work myself. It wasn't the LLM doing the work for me. It was scaffolding my learning and pointing me in the right direction and figuring out what questions I needed to be asking and what the right answers were, but just not taking over. And I genuinely felt quite empowered as a learner.

It was scaffolding my learning and pointing me in the right direction and figuring out what questions I needed to be asking and what the right answers were, but just not taking over. And I genuinely felt quite empowered as a learner.

When is a human the better choice?

So equipped with these ideas of the kind of stretch zone projects and using the LLM as a tutor, I decided to take these ideas and apply them to a completely different project. And this brings me to my third and final question. When is a human the better choice? So I'm one of the maintainers of the arrr package, and I decided I wanted to have a go at contributing to PyArrow, the Python arrow implementation. I decided to work on a task related to magic methods. Now, if you don't know what they are, it doesn't matter too much for this, but it felt like the perfect stretch zone task in that I've got experience of the arrr package, I can write some Python, but I didn't know much about PyArrow and very little about the internals of Python libraries in general. So it kind of felt in the right place.

So next, I needed to figure out what role I wanted the LLM to play. So I decided that I was going to write most of the code. In this case, you know, this task for me was about learning. So I needed to do that myself, really. I also wanted to write the tests, just so I could be really clear about what the intentions of the code that I was writing. And I was going to get the LLM to help me with any code tweaks. Maybe there's bits of how things are done in Python that I don't really kind of get with package internals. So I was going to use it to help me understand errors and to explain any new concepts, like magic methods, which I had no idea what they were at the time. But also, I had some other help from my friend, Alenka, who is a PyArrow maintainer, and she was just going to jump in anywhere I got stuck.

Okay, so how did it go? Well, actually, I feel like this is almost like a kind of fight between LLMs and human knowledge, almost like a kind of retro video game with different levels. So the first level, then, rubber duck. So I think probably a lot of people in this room have experience of that kind of rubber duck debugging thing, where, you know, you've got a problem with some code, and in explaining it to an inanimate object, you actually find yourself getting closer to the solution than you were in the first place, without even getting an answer back. And I found here, actually, when I was trying to create the prompt to kind of put into the LLM, half the time, trying to get something specific enough that it wasn't going to come back with nonsense actually helped me figure out a lot of the problem myself, which was great.

The second level, the LLM as an assistant. So there were points at which I was trying to build PyArrow, and it's got C++ under it, and I'm not super experienced in C++, and I ended up with various errors in the build logs, and instead of having to troll through and figure out what was the problem and what was happening, having an LLM just help me pinpoint it really quickly was fantastic.

So the third level, the LLM as a tutor. So as I mentioned, I had no idea what magic methods were before this, so I got it to come up with a very quick sort of mini-tutorial-type thing, give me a really quick, simple example of exactly what it was, and it just made me feel way more prepared for the task I was trying to complete. But actually, there was a point at which LLM help didn't really take me any further.

So there was a bit where there were two different conventions of how to approach magic methods in Python, and there was just conflicting information there. I could have asked an LLM, but it felt like actually, with that context and the context of PyArrow, this needed human intervention. So I spoke to Alenka, who helped me by just explaining things from her past experience and her knowledge of this, and yeah, I feel like having somebody acting as a mentor on a project is just so important in so many cases.

But actually, kind of like beyond this, I got the pull request done, I got it finished, but I wanted to figure out where would be another place I could contribute, what were other tickets I could take on that would be kind of fun, interesting, and a good learning experience. And I mentioned it to a few other people collaborating on the project, and they started actually tagging me on issues. And again, I think this thing of having somebody acting as a sponsor, helping kind of figure out opportunities to kind of push you further, again, so, so important to have that human kind of side of things there.

So actually, it's kind of not about sort of LLM versus human knowledge, it's more about just where to kind of rely on each. And so to kind of go back to the overall kind of question of how can I be more productive using LLMs, I think it's important to keep in mind that I feel like LLMs shine in the stretch zone, so like projects that can just take you a bit further than where you are at the moment. Not all the time, but sometimes, actually using LLMs to scaffold the work and not doing it for you can be really valuable. And finally, like not forgetting the importance of human connection, like really thinking about when to rely on people instead. And I think by doing all of these things, we can kind of shift from passive use of tools to like really active learning, and that can help to build long-term skills and productivity. Thank you.

I feel like LLMs shine in the stretch zone, so like projects that can just take you a bit further than where you are at the moment. Not all the time, but sometimes, actually using LLMs to scaffold the work and not doing it for you can be really valuable.

Thank you so much, Nick. We did have one question. Did you make the art for your slides because they're so awesome? Yeah, I got obsessed with Canva the other day. Oh, no, the other ones. Do you know what? I saw Hadley's talk at USAR where he'd done a lot of that and was like, well, that's a good idea to pay homage to. So, yeah.