Putting an {ellmer} AI in production with the blessing of IT (Andrie de Vries) | posit::conf(2025)
Putting an {ellmer} AI in production with the blessing of IT Speaker(s): Andrie de Vries Abstract: I had an idea to automate a process by letting an AI extract information from PDF attachments. Writing the 20 lines of {ellmer} code was very simple. My users loved the idea. But IT had different feelings, especially initially. It took multiple design changes to get IT comfortable with running experiments in our financial system. They need a solution that is secure, with guardrails, wouldn’t leak sensitive information and could be easily maintained. We wrote a simple browser extension to reduce the friction of copy/paste between different systems. If you want to put an AI in production, make sure you consider the impact on other teams, in particular IT and Information security. posit::conf(2025) Subscribe to posit::conf updates: https://posit.co/about/subscription-management/ Keywords: production, AI, LLM
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Welcome Andrie de Vries Earlier this year, in about February and March, we sat down with much of the company and said, what kind of use cases do you have for AI? What would you like to do?
And many examples came up, and I discovered, much to my delight, that in several use cases I could write a really compelling proof-of-concept in about 20 lines of code using ellmer or Ragnar or whatever I used at the time.
My audience loved it, typically I would then take that proof-of-concept and build it into a shiny app, I would use ellmer to speak to the LLM and to write maybe some DuckDB code and that code will query information from a database and present it back to the user.
Everybody loved this and I then carried on and made more complicated apps using multiple data sources and this unlocked an immense amount of value. Imagine one of my apps that combines information from our Salesforce database and Zendesk tickets and multiple other data sources and now a sales rep or anybody else can query information in one place that previously would have been very, very difficult to go and do and they just didn't do that.
This was great until InfoSec heard about this. You've had the same problem.
My name is Andri, I'm director of product strategy at Posit and I'm going to talk a little bit about how I worked with InfoSec to get to a happy place that everybody was okay with.
Understanding InfoSec concerns
So InfoSec, information security, they care about three things, apparently this is the oath they swear to be inducted in the hall of fame of InfoSec agents, I guess. Confidentiality, integrity and availability. Availability means does this app run? Is it available six sigma, 99.99%?
I let them deal with that but I need to be much more aware of confidentiality so if I surface new information, is the person that views that information, are they authorised to see that? And integrity, in other words, is this information accurate and can it be trusted?
So if I have an LNM app, InfoSec flags a number of risks and I'm going to highlight three of them. You need to work with your own team to see what's relevant for you.
The first one is the risk of data leakage, in other words, if I combine information from multiple sources, all of which has potentially sensitive information like PII, personally identifiable information or trade secrets, you run the risk of grabbing information from multiple places, combining that together and then accidentally surfacing that to somebody who should not have had access to that.
The second risk was, well, what if I use an LNM to generate some data and it makes up some nonsense and I write that back into a production database? Nobody wants that. But you also want to have this balance about keeping data in a sandbox, not creating data silos. So there's a risk that InfoSec is going to want to speak to you about that.
And the third one is a little bit more subtle but everybody is concerned about this. If I'm using an LNM agent, and you can use your agent of choice, there is a small risk that if you don't control for this carefully, that the data, that your own data will be used for training and can accidentally leak to other users, not just on the same server but also if it's retrained in the future.
Solutions and recommendations
I'm not going to pretend that the solutions I have here is going to work for you. But here are a few things to go and think about. First of all, make sure that your authorization is correct, that you give minimal permissions, so you limit your app to trusted users.
Make sure that you have data security of the LNM itself. In our case, we host our models on AWS Bedrock. Most of the cloud providers have a service like that where they ensure that your data will not leak out and will not be used for retraining. And scope to mission. So if you write information somewhere else, make sure that where you write it to has the same permissions and authorization you had before.
In short, don't ask for forgiveness. Ask for permission and ask it early, as soon as you know that you are going to be touching sensitive information.
In short, don't ask for forgiveness. Ask for permission and ask it early, as soon as you know that you are going to be touching sensitive information.
