Harnessing LLMs for Data Analysis | Led by Joe Cheng, CTO at Posit
When we think of LLMs (large language models), usually what comes to mind are general purpose chatbots like ChatGPT or code assistants like GitHub Copilot. But as useful as ChatGPT and Copilot are, LLMs have so much more to offer—if you know how to code. In this demo Joe Cheng will explain LLM APIs from zero, and have you building and deploying custom LLM-empowered data workflows and apps in no time. Posit PBC hosts these Workflow Demos the last Wednesday of every month. To join us for future events, you can register here: https://posit.co/events/ Slides: https://jcheng5.github.io/workflow-demo/ GitHub repo: https://github.com/jcheng5/workflow-demo Resources shared during the demo: Ellmer https://ellmer.tidyverse.org/ Chatlas https://posit-dev.github.io/chatlas/ Environment variable management: For R: https://docs.posit.co/ide/user/ide/guide/environments/r/managing-r.html#renviron For Python https://pypi.org/project/python-dotenv/ Shiny chatbot UI: For R, Shinychat https://posit-dev.github.io/shinychat/ For Python, ui.Chat https://shiny.posit.co/py/docs/genai-inspiration.html Deployment Cloud hosting https://connect.posit.cloud On-premises (Enterprise) https://posit.co/products/enterprise/connect/ On-premises (Open source) https://posit.co/products/open-source/shiny-server/ Querychat Demo: https://jcheng.shinyapps.io/sidebot/ Package: https://github.com/posit-dev/querychat/ If you have specific follow-up questions about our professional products, you can schedule time to chat with our team: pos.it/llm-demo
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hi, my name is Joe Cheng I'm the CTO of Posit and today I'm going to talk to you about harnessing LLMs from code in the service of data analysis
This talk is intended for people who know how to code in either R or Python I'm assuming that you have used LLMs via something like chat-gpt or copilot, cursor or windsurf But I've also assumed that you have not written code that calls LLMs That you have not called LLM APIs If you have a lot of experience using LLMs from code Maybe you're also just interested in Posit's take on these tools in which case welcome to you as well
Getting started with LLMs
So as we're getting started with LLMs, there's a couple of things you need to know right up front One is that LLMs are generally accessible these days through HTTP APIs There are other ways to call LLMs than through HTTP APIs, but I don't think there are any other ways that matter This is the way most models are accessed today Therefore Posit has created packages that are designed to make it really easy to call LLMs through these HTTP APIs And we're going to talk a lot about these packages today. For R, there's ellmer and for Python, there's chatlas
So as we are getting started using these LLMs, let's just set our expectations correctly First of all, you can expect that this is going to be super easy as we get started ellmer and chatlas are both extremely easy to use These models, the best models out there at least, the best LLMs are highly capable and they're getting better all the time So if that's not your expectation Maybe because you haven't used these models in a while or you haven't sort of persevered through seeing them make mistakes Just know that they're quite good at a lot of things these days and they are extensible We're going to talk today about some ways that we can add our own behaviors and even add code Well code in a way to these models
However, these models do make mistakes and they make them a lot and not only do they make mistakes. They do them unpredictably Nondeterministically, which makes it even more frustrating. I think in summary today, you know May 2025 these LLMs have a jagged frontier I think that's a term you're gonna hear more and more There are things that they are surprisingly good at Even shockingly good at and then things that they are really surprisingly bad at so as we Explore them. Let's just let's just expect to be surprised in both directions
I think in summary today, you know May 2025 these LLMs have a jagged frontier I think that's a term you're gonna hear more and more There are things that they are surprisingly good at Even shockingly good at and then things that they are really surprisingly bad at so as we Explore them. Let's just let's just expect to be surprised in both directions
So I have a couple of tips as well for how we should not approach getting started with playing with LLMs First of all, you may have heard a lot about local models or open weights models These are models that you can download to your own machine or your own server and run them I highly recommend that you do not start with them and instead start with the very best models that the world has right now Those are clod 3 7 sonnet or actually as of today clod 4 sonnet Open AI has a number of state-of-the-art models that are sort of have different trade-offs and Google Gemini 2.5 is I've heard very good the the open weights ones like llama and its derivatives quen Deep seek. They're just not as good as you have heard. I'm sorry. Sorry, not sorry.
I also suggest that as you're getting started Don't start with non public information or data. Don't start with your customer data. Don't start especially with patient data Instead as you're getting to know these APIs Do it with low stakes publicly available or just non proprietary data And only after you get a good sense of how to use these APIs and have a good sense for how they work Then you can think about security and data privacy and whether it's safe for you to use Non-public data
So to get started. We're going to need to do a couple things the by far most annoying part of this is We need to sign up for a developer account with anthropic or open AI or Google if you want to go the Gemini route And you need to grab an API key It's pretty simple to do this if you just Go to their site and sign up, but it does require a credit card These sites generally will not work will not allow you to make API calls without putting down a credit card and paying a few dollars All I have to say is just do it already. I mean, I know this stops a lot of people from proceeding It's just enough of a pain enough friction But it is really worth pushing through this Step and it really is not going to be a lot of money. You can do a lot of experimentation with just you know $20 that might last you a couple months of Quite intense playing with these models
Once you have that API key Then you need to add it to your environment variables and I have links that I'll drop in the description for doing that For both are for Python and then it's time to install an LLM client package for our there's ellmer and for Python There is chatlas. I just say no when you go to use chatlas. It may ask you as you Try to connect it may ask you to install other packages depending on what provider you're using. So just be aware of that
Hello world with ellmer
So this is sort of the hello world of using ellmer. This is the our version You can see here. It's extremely short. There's just basically two lines of code here the first Creates a chat object Calling this chat anthropic function and passes in the model that we want to speak with This is club 3.7 sonnet and we are asking for the latest version You could change this from chat anthropic to chat open AI and put in an open AI model identifier And now you're chatting with with open AI neither case. You have this client object that you get back We often refer to it as a chat object and you call the chat method on it and pass in your question and you get A response. So in this case summarize the plot of Romeo and Juliet and it does so
Another interesting thing about this client variable is it is not just a way to ask questions It is a way to carry on a conversation So you can call chat multiple times and each subsequent call is like the next question in this ongoing conversation So here I say now Hamlet, this is just this is not just me asking context-free this model Hey model now Hamlet. It is a follow-on to the previous question about Romeo and Juliet So it knows how to answer with a summary of 20 words or less
This is the example in R and then This is the example in Python. It is almost identical. You can see here that the Capitalization has changed because I'm calling a Python constructor instead of an R function But other than that, it's it's extremely similar
So a few things to know about ellmer and chatlas number one They support many different LLM providers Pretty much all the ones that we can get our hands on and this is really important because you want the ability To experiment with different model providers and just kind of see Whether there are certain things that one provider does better than another Secondly you have the ability to have a chat and then take that chat object and just print it at the console or in a Jupyter notebook and you'll be able to View the conversation history Just by printing the object
Third both ellmer and chatlas have built-in chat UIs And in fact, they each have two Two built-in chat UIs one for a web browser and one for the console and I'll show you real quick how that works
So for example, let's take that Romeo and Juliet example and instead of doing this at The console let's do it in an app So we can say So we can ask this question and Yeah, very convenient to be able to do this in in a web interface or maybe instead of a web interface you prefer a console interface and I'm continuing the conversation here. I can say like how about Hamlet And there you go streams the answer back
Customizing with system prompts
Okay, so super easy to use and designed to work really well interactively But that's not actually that interesting right it's not that interesting for us to take a bone stock LLM that you already have a web interface for that. You could just go to claw.ai Instead we want to be able to customize these models and the first way we often want to customize them is by adding knowledge So we want to tell these models something that they don't know from the factory
Just a quick couple of Discouragements here you may have heard a lot about rag. I recommend that you do not start there It is a very useful technology But I don't think it's the one you should be starting with same thing even more so with fine-tuning You may have heard this term I recommend you do not start there either when you want to add knowledge the first place you should start is with system prompt customization It is easy. It is effective. It's super quick whereas these other techniques take a little bit of fiddling
so here is one example of Adding some information via The system prompts What the system prompt is is a way of us telling the model that for this interaction or for this conversation? These are the ground rules that I want you to obey In this case, we're saying you're a bot that answers questions based on this expense policy doc and I have an expense policy doc here that is actually for a fictional company called example Inc and Really boring right? It's saying here's what is eligible for expensing when you're traveling when you're visiting Clients when you're buying office supplies super boring But important to get right So here I'm creating a string that says you're a bot that answers questions based on this expense policy doc And then I put in some like XML tag looking things This is just to organize all the lines. I'm about to paste in and then I put in this entire doc I literally copy and paste this doc into this system prompt string and then I have a couple more follow-up bullet points site policy section numbers and use examples when helpful and Now I can take that system prompt and pass it as an argument when I create a chat object and now When I do that the answer Incorporates the knowledge from that document. So in this case, can I expense a programming book? It says based on the expense policy. It might be Ask ask your manager section 3.4.
So let's do one that is a little bit more in the vein of data analysis. So for this one Maybe you can relate to this experience that I've had multiple times where? There's a data set that I'm very excited to get my hands on in this case It's the college scorecard data that the the federal government puts out and it has a lot of information about different characteristics of colleges and universities as well as a lot of I think data from the IRS about The outcomes that students experience six years and ten years after they graduate so really rich data set There's just one problem with this data set
Got this CSV here And if we look in positron there are thousands of columns and These are quite impenetrable column names and The data dictionary is 800 kilobytes of Excel spreadsheets. I mean, it's just a lot So what I've done here is I've taken that those That data dictionary that Excel data dictionary and I've used Claude code to come up with sort of a condensed summary as a Markdown file and now I have created a system prompt that says you're a bot that helps data analysis for college scorecard data Do not attempt to analyze the data, but you can write our code I'm telling it to use the read our package to read CSVs And that assume that the CSV files are in the working directory and then I read in the entire The Entire file I have that summarizes this data and you can see it talks about hundreds of available variables
And this actually works awesome So I'm able to Run this example and I can ask a question like what are some demographic variables?
All right, pretty cool And by the way note what we did not do here we did not actually load the College scorecard data Into the LLM and ask it to start answering data questions. We did not do that We asked it to write code and there's a really a good reason for that Because on their own the LLM themselves are terrible at analyzing data like they simply are Not suited for this task So if you load up data in into them in the format that they understand best, which is maybe JSON They are actually Not able to reliably do even simple arithmetic In fact, they cannot even reliably count rows in a data frame In fact just today. I tried it with a small CSV 299 rows loaded it into Claude v4 The the latest that just came out today and it said that there were 216 rows so we cannot trust them on their own
Instead what we can almost trust them to do is have them write code that analyzes data and sometimes even Let the LLM have enough control that it can execute the code for you And that that is a much much better way to have LLMs help us with data tasks We want them to write code that then gets executed
Because on their own the LLM themselves are terrible at analyzing data like they simply are Not suited for this task Instead what we can almost trust them to do is have them write code that analyzes data and sometimes even Let the LLM have enough control that it can execute the code for you And that that is a much much better way to have LLMs help us with data tasks
Customizing behavior and building Shiny apps
So those last two examples only really involved adding Knowledge what they did not really involve very much of was customizing behavior So let's use our system prompt again and just by putting some different instructions there we can have The these LLMs act in quite a different way than just a general assistant in this case I'm gonna say like you're a dungeon master for Dungeons and Dragons style game whose theme is this expense policy and I'm reusing this expense policy document So if I go ahead and run that And this time I'm doing it in Python Now I'm running it in Python and I can say begin And It starts a ridiculous game of dungeons and deductions. Sure. Okay so you can see here, I mean, this is really Pretty bizarre that we can you know, just write two sentences and just get this completely different behavior out of this model
So far all of these examples have just created ellmer and chatlas apps and we've called them from either their built-in chat UIs or or called the chat method from the console But it's more common I think to take these Take these customized ellmer and and chatlas instances and build them into a Shiny front-end In R we have a separate package called shinychat that you can use to to put a chat bot into your Shiny app and In Python, there's not a separate package. There's a built-in component for shiny for Python called UI dot chat And I will drop the links for the documentation for both of those in the description as well
It's very easy to do with just a few lines of code you can have ellmer and chatlas Chat bots working very well Because these are Shiny apps they are super easy for you to deploy you can deploy your apps with one click or I guess You know one command depending on how you do it on Prem we have a commercial product called posit connect I Imagine that many many of you on this call already have access to a posit connect at your company's Sometimes you may not even know it, but you do And we also have a cloud hosted service called posit connect cloud This is sort of the the next generation of shinyapps dot io if you're familiar with that Oh, I forgot to put on the slide, but we also have open source on-prem hosting with shiny-server open source
And Just like Shiny apps are so easy to write and deploy that you could Often if you're if you're pretty skilled at Shiny, or you use shiny assistant, which is our AI assistant for writing Shiny apps It's often worth creating Deploying using and throwing away a Shiny app. You know all within a week or a day It's so easy and so fast to do and even more so with ellmer and chatlas because it's so easy to create these LLMs We've definitely found that within posit We've had a proliferation of internal chat bots that just serve one Specific purpose and it it would not have been worth the effort to create and especially to deploy You know a custom chat bot for that one specific purpose. You know like this extent expense report bot But it's just so easy to do with this technology stack That that we do it
Vision, structured output, and tool calling
Couple other ellmer and chatlas features that I want to show you You have the ability to do image as input and the ability to do structured output. I'll come back to that in a moment you have the ability to not just take a response as one big string all at once, but you have the ability to kind of take the response as streams of little chunks of strings and If you know what async is just know that both ellmer and chatlas work in synchronous or async modes
Just to show you how easy vision is This is a an image called photo dot JPG when I'm chatting with in this case Claude It's as simple as instead of passing a string or in addition in addition to passing a string you can pass the result of content image file and you give it a file name and You know Claude is able to see this image and tell you what the mood of the image is in a single phrase This is what it looks like in R and this is what it looks like in Python super easy to do I will warn you this is one of those jagged frontier areas where it's Incredibly sensitive and nuanced to certain aspects of photos and then other things like counting how many objects are in a photo it Just can't can't do sometimes it's really surprising
So definitely worth playing with though. I won't dwell on this too much. Just know that These models also often have the ability to not just give you unstructured text back but to give you JSON objects that you can then use You know for to put into a database or to you know code against for other reasons So this is what it looks like in R You kind of specify what kind of data you want back what the shape of the data that you want back and then you call Chat structured instead of chat and this is what it looks like in Python looks a little bit different here we're using a package called Pydantic, which is a de facto standard in the Python world And we call extract data instead of chat and tell it this is the shape of data that we want And yeah, instead of getting a string you get back some JSON that is very easy to use from code
The last topic I really I don't have time for this but I'm gonna do it anyway, it's just such an important topic Tool calling allows LLMs to interact with other systems. So instead of just taking text in and Giving text back out. It gives the LLMs the ability to call out to APIs to databases the ability to execute code to do things to your file system Just all sorts of things that can be possible and that sounds a little bit scary and it sounds complicated But it is not as bad as you think So I'm gonna try to explain how this works by first telling you how it doesn't work This is how when I first heard about this functionality, this is how I imagined it working, but it is not how it works
So one example of something that a pure LLM cannot do an LLM is is just a model It can't get the current weather, right? You need a weather service for that So in this case, let's say we have a user as the assistant. What's the weather right now at Fenway Park? When I heard that there was something called tool calling that lets LLMs access other systems what I imagined is somehow we would be teaching the assistant to reach out to open weather org it's a fictional service, but just pretend and Call Give me the weather for 0 2 2 1 5 That's the zip code where Fenway Park is Open weather would return the data as JSON and then the assistant would Incorporate that JSON into its response and say 65 degrees and sunny. Okay. This is how I imagine it working But this is not how it works
Instead it works like this The user asks the question. What's the weather right now at Fenway Park and it includes along with that request a Description of a tool called get current weather. So it's not passing code. It is just passing metadata about a function It's saying the name of this function is get current weather It takes a single argument that argument is a zip code and it's a string So the assistant instead of answering with what the weather is it? Instead responds saying I would like to call this tool that you said was available Called get current weather 0 2 2 1 5 and it is actually the responsibility of the client not the server To go off and perform this request against openweather.org or however else it wants to Fulfill the task of getting the current weather
When a response is received then the user continues the conversation By taking this response that the assistant asked for and sending it to the assistant and the assistant now it has the original question it sees the request that it sent and it sees a response that came back from the user and Therefore now it's able to actually give a proper response back to the client Incorporating this this data. So it seems a lot more complicated, but is actually much simpler
basically Just just to recap the user asks the assistant a question and includes metadata for tools or functions that are available The assistant asks the user to invoke a tool on its behalf and passes the arguments that should be used the user invokes the tool and Returns the output to the assistant which the assistant can then use as context for its final response It turns out this is an incredibly general and powerful way to extend the capabilities of these Of these assistants and anytime you hear the word agent used which is a lot these days Just know that all of that is made possible by creating these tools and giving LLMs access to them
So another way to think of it is that the client can do things that the assistant can't do like not only call out to things like a weather service, but To read and write to databases to put files on disk to turn the lights on and off in a smart home Anything you can think of doing in code You can empower the assistant to do and we put the control into the hands of the assistant so that the assistant can decide when it's appropriate to use what tools and what arguments to pass in and What to do with the results and it turns out that that's just an incredibly general and powerful Capability having that intelligence in using and coordinating these tools
Putting it all together: the querychat demo
So Let's take that You know everything that we've talked about so far. We've talked about ellmer and chatlas. We've talked about Shiny apps. We've talked about tool calling Customizing system prompts. We're going to put all of that together Into An example and this is an application that I have demoed before It's a restaurant tipping app What you're looking at right now does not involve any LLMs or AI This is a simple shiny dashboard that lets you you know, move some controls here and it and it updates on the right.
What we have on the next tab is a new version of that desktop of that dashboard that does incorporate AI and here we have We've replaced those controls on the side With with a chat bot in the sidebar and this chat bot you can ask it filtering questions Like show only male smokers who had dinner on Saturday and instead of having to move a bunch of sliders you just you just say what you want in plain English and What this What this application does is it doesn't ask the LLM to Populate all these outputs on the right. I would never trust an LLM to do that All we are asking the LLM to do is to fulfill the users request here by writing sequel So this sequel is passed as a tool call to the rest of the shiny app the tool is called update the dashboard with this filter and you can see here that that we've shown the sequel in two places and The thing that's key about this is that if the sequel is right Then everything below it is guaranteed to be right because the only thing That this chat bot has the ability to influence is What is the sequel that drives the rest of the app? The rest of the app is just normal shiny reactivity All of this code is just normal shiny deterministic code
But the point is This is all using the pieces that we've already talked about here with with ellmer and chatlas and and tool Calling and system prompt customization. We have the ability to embed really intelligent Agents into our shiny apps and deploy them. I Demoed this application Last year at posit conf and at the time. This was just an example That you could fork if you wanted to and and replace everything to have your own data Since then we've made things a lot easier by creating this package called querychat this is for both our and for Python and is designed to make it dead easy to take these kinds of Sidebar chat bots that that return reactive data frames using sequel So it's extremely easy to build your own version of this app with your own data your own visualizations and your own downstream reactives Again that's called querychat, and there's documentation for both our and Python here
Closing thoughts
That's all the time that I have today I hope that over the last 30 minutes. You've seen how easy it is to get started with this technology How easy it is to play with LLMs and ask them to do different things and and just see how it works out And now is really the time for that now is the time to play with these technologies? they are coming I mean they are here and There is a tidal wave of change that's ahead of us and the best thing I can recommend is that you Be on the forefront of that change by understanding how these things work what they're good for what they're really terrible for and Help guide the future of how we should be using these things especially when it comes to data analysis
There are so many more things happening inside of Posit Packages and and tools and services That we are doing that we have already done and released So I encourage you to stay up to date by following the Posit blog, Heidi Verse blog, Simon Couch's blog There's just so much Interesting stuff happening around Posit around AI so we have a lot more interesting announcements coming soon So stay tuned and with that let's go over to Q&A
