Integrating AI, Data Science Platforms, and Psychological Safety (Chris Engelhardt, Gen Re)

Transcript#

This transcript was generated automatically and may contain errors.

Hello, everybody. I'm Chris Engelhardt, and I'm a Senior Data and AI Operations Manager at Gen Re. At Gen Re, I work on a lot of cool and interesting challenges. I oversee building and deploying custom AI applications and infrastructure, as well as overseeing our global analytics and data science platform, which runs on Posit.

Today, I'm excited to talk to you about bringing my background in social and personality psychology to how I lead data science and AI teams.

The Nokia cautionary tale

Before diving into this topic, though, I want to take you back to 2005, when Nokia was at the peak of innovation, as evidenced here by the N95 model.

And as reported in a case study, Nokia soon had some challenges because there was a new competitor in the landscape. Apple had just introduced in 2007 their first iPhone OS with a sleek touchscreen, which Nokia did not have.

And obviously, there was some amount of concern going on at this point, especially among senior leadership. They were putting a lot of pressure on lower-level managers to deliver in a timely fashion.

And what senior leadership was not interested in was hearing negative news about meeting deadlines, or the current status of products.

And ultimately, this stifled innovation and led to Nokia's downfall, at least over the 2005 to 2010 period.

What I want each of you in the audience to do at this point is to put yourself in the shoes of a Nokia manager. And imagine if you worked in a team where you were afraid to speak up.

Imagine if you were on a team where no matter what you thought, no matter what you did, would be met with belittlement or beratement. Imagine if your voice didn't matter. Imagine if your lived experiences didn't matter, your mistakes and all.

What would the outcome of that look like? Well, I think it can lead to some pretty negative consequences as a direct result of not being able to speak up.

One, of course, and this was evidenced in the Nokia case study, is that there's information asymmetry. So managers who were on a lower level had access to different information, i.e. negative news, that senior leadership did not have access to.

I think also it led to a certain amount of learned helplessness in the Nokia organization, where the idea here is that no matter what you do, it doesn't matter, you feel powerless to affect some change and change the current state of your situation.

Taken together with information asymmetry combined with learned helplessness, I think it ultimately leads to poor team performance. And we saw this in the Nokia example as well. And it has some pretty dramatic consequences as a result as well, both on individuals' mental health, but also on organizational consequences as well. And this is a challenge. And we need a way to address this challenge as a leader.

Psychological safety as a solution

One strategy that I would submit to you all today is to think about addressing this challenge through the lens of psychological safety. Where the idea here is that the teammates believe, so this is a team-level construct, it's where the teammates believe that the team is safe for interpersonal risk-taking.

By interpersonal risk-taking, I mean that the team feels free to speak up, to offer their unique ideas and contributions, to admit their mistakes and faults, and ask for help. And these are challenging things to do, because each of their own in isolation can lead to perhaps other people perceiving you as incompetent or not an effective teammate. So there's an inherent risk in each of these behaviors.

What I want to do is talk about our journey on our team and psychological safety. And the start of our journey began about a year ago, where I presented to our team on this concept, as well as the empirical evidence supporting it. So I drew on Amy Edmondson's work back in the 90s. And in her work, the earlier study, she focused on furniture manufacturing companies, not among technical teams.

And so I thought if I wanted a great chance of getting the architects and engineers and data scientists on board with my messaging, I also had to present something on evidence among technical teams. And who better to turn to for that than Google?

As some of you may know, Google spends millions of dollars on investigating teamwork, what makes some teams underperform versus overperform. And they actually ran a study back in 2012 to 2014 examining this exact phenomenon. So they included nearly 200 teams, many of which were technical in nature, and they asked them a variety of questions.

A lot focused on individual differences. Does it matter how introverted versus extroverted you are? Does it matter if you socialize on the weekend with your teammates? Does it matter your academic pedigree? Does it not matter? What matters is how people work together as a team.

I want to draw your attention to the bottom of this pyramid here, where of course they identified a few factors that matter, but chief among them in importance is psychological safety, where again the idea is that teammates feel safe to take interpersonal risks and be vulnerable in front of one another.

but chief among them in importance is psychological safety, where again the idea is that teammates feel safe to take interpersonal risks and be vulnerable in front of one another.

Our team's journey

I wanted to model the types of behaviors that I want our team to espouse as well. And I immediately asked for help. And the help that I asked for is how do we sustain psychological safety on our team?

One unique idea that was brought forward during this time, and this is admittedly two words you don't see together very often. We hosted a team anxiety party. Where we started off in this meeting with the fears that we all have, just to get the ball rolling.

So I shared that I have an irrational fear of getting stuck at the top of a roller coaster. And I love roller coasters, but I have an irrational fear of getting stuck at the top. And I also have a really strong fear of bees. So if you see me one day stuck at the top of a roller coaster surrounded by bees, rest assured, I'm not living my best life in that moment.

But there was a surprising amount of overlap in the fears that we all had. And it was a very, very powerful and unique team experience. And I would recommend it. It's a qualitatively interesting experience, really powerful that I think genuinely brought us together as a team.

What I want to do now is fast forward to where we're at today, which is in the middle of our journey. And I think we'll be here in perpetuity because we are an imperfect team striving for perfection.

So what can we do to ask for help? And two ways that I recently asked for help is in each one-on-one that I have with direct reports or others on the team, I ask them a very pointed question, which is, what is one thing we can do today to improve as a team? And I know that our teammates are really thoughtful about what they bring forward during that time.

There's a significant amount of reflection in what is one incremental thing we can do, what's one action or behavior we can take to support this talk, believe it or not. And I ask for their help in helping me prep for this very presentation that you're seeing here. So what you'll hear in the next few slides are our teammates' voices, which is exactly what I think you all should hear in this talk.

When I first joined Gen Re back in 2022, we were a team of three, including me. And today, a mere three years later, we're approaching 25. So we've had some significant growth and with that, some challenges with that growth.

And what we do every two weeks is we have a lot of meetings that are focused around developer productivity. We talk about the development efforts. In these Agile ceremonies, we talk about the current state and what's going on and what's next.

And two changes to these ceremonies. One change that was brought forth by one of our teammates was to split those ceremonies or developer meetings by AI product. Another suggestion was to align those ceremonies with the company objectives so that way people could see visually how the work that they're doing aligns with overall company objectives.

My takeaway from here is to enable people to do what they do best. And if you're a developer, what you're doing best is probably not sitting idly in meetings. You want to be hands-on, developing some artifacts. And to have focus on how and why their work matters so they can see the meaning and the value of their overall organizational goals.

I'm sure none of you have been in this position. But we sometimes will work with vendors that from time to time will guarantee off-the-shelf solutions to solve all of the challenges that we have in a particular use case. While you might not have experienced that, we certainly have.

And what we had to do in that scenario was to empathize with the users. Validate their emotions, right? Or promise something where the solution could not deliver on it. So we created more of a collaborative environment. We sat side-by-side with individuals across business and worked through their specific use case. And the end result of this, I think, is that it improved trust among our stakeholders and accelerated the development process that we had for them.

And as some of you are aware, of course, empathy matters. Putting yourself in the shoes of your stakeholders and building and establishing their trust.

I also want to highlight here how we've thought about asking for help. Because it's not necessarily when you hit a wall you just immediately go and ask somebody to solve it for you. And that's because I also want to empower our team to experiment, to fail, perhaps even multiple times. But importantly, to learn through that process. And asking for help is never guilted. It's never shamed. It's never belittled. It's never berated, importantly.

But, you know, I was in this position once, too. I loved having the aha moment. Where I've been perseverating on a challenge for days or weeks and then all of a sudden, out of nowhere, perhaps in a dream, the solution comes to you. And you have that very special, euphoric aha moment. And there are few things like that in this world. So while I don't want to take that opportunity away from people, I want to make sure that people know that they have help when needed.

Key results: AI framework and platform

I next want to turn to psychological safety and some of the key results that we've achieved as a result of it. Starting first with what we call an AI framework. And an AI framework is an opinionated approach to how we orchestrate back-end services to make AI applications realized for our business. And this foundational layer is reused across multiple applications because it's modular in nature. And it supports today three applications that we have running in production.

We've also developed some internal Gen Re packages that wrap AI services. So that way folks who are developing with R or Python can use these in a Gen Re-approved way. We also do some cool things within here as well, like mock testing and prompt caching.

Additionally, we also have implemented work at the intersection of Azure Databricks, which is a big data compute environment, remote compute environment. Unity Catalog, which can be used for the management and governance of data sources, as well as the Azure Data Lake for storage.

If you've ever thought that it's always easy to write to the Data Lake, you would be wrong. Especially when you're at the intersection of modern tools and technologies.

And I do want to give a shout-out to Edgar Ruiz for his own interesting capabilities with Posit Connect, which is the content hosting platform developed and deployed by Posit, where we offer capabilities to our external clients in the service of facilitating initial underwriting decisions. And this process alone has helped our clients save about 600 human hours per year.

In closing, the takeaway for me here is that psychological safety is not a luxury. It's the foundational piece upon which high-performing teams are built. To empower people to speak up, to empower people to admit their mistakes, to admit their faults, to be a genuine person and to be themselves, bring who you are to work.

And I think as a leader, what you would want to know about is what can I do today to help create a high-performing team? And what I would submit to each of you in here is that psychological safety can be a very efficient vehicle to increase team performance. Thank you all for your time.

psychological safety is not a luxury. It's the foundational piece upon which high-performing teams are built.

Q&A

All right. Thank you very much, Chris. So we have a few questions. All right. So the first one is how do you measure psychological safety?

So psychological safety, if anyone's interested in measuring this on their team, I would encourage you to go visit any of Amy Edmondson's work. She has a seven-item Likert scale that is a reliable and valid measure of psychological safety. We ourselves on our team have not directly measured that more than once where we had in some of our Agile ceremonies, you know, how safe were people feeling and people were indeed feeling safe, which is encouraging.

Because, well, people already will know that that's what I'm intending to measure, so there could be some response bias in there. But you can measure this in reliable and valid ways. And again, I would encourage you to check out Amy Edmondson's scale on psychological safety. It's a great, wonderful measure to use. It's used nearly ubiquitously in psychological safety research today.

Do you plan to implement more measurements? So again, I think there's some tricks with people's expectations to respond in certain ways, but there definitely are ways that we can capture this, and we do often have check-ins with the team on taking a litmus test and how people are feeling and such. But yeah, definitely could have more of a quantitative focus on that in the future.

Okay, thank you. We have another question. So you were in the enviable situation of building a team from scratch, three people to 25. How do you start to turn the ship to make the workplace more psychologically safe?

Right, that's a great question. And I will admit that taking a bottom-up approach will be much more challenging than this coming from top-down. That said, and I've thought about this a little bit, what I would recommend is talking to your leader or talking to senior leadership and asking them this question. Are you interested in team performance? Who would disagree with that, right? So when they say yes, say, well, guess what? I have exactly the right way to approach this. So that way you're aligning what they've already just agreed to to the actual behavior that you can implement.

All right, thank you very much. A round of applause, please, for Chris. Thank you all.