People Analytics at AT&T with Liz Esarove | Data Science Hangout
We were recently joined by Liz Esarove, Lead Advanced Analytics in People Analytics at AT&T, to chat about how her team uses data science to improve business decisions regarding employees, work, and business objectives at AT&T. Liz walked us through several use cases for people analytics, including reducing employee attrition and forecasting future attrition rates and workforce changes. She also discussed how her team uses text analytics to analyze employee comments from surveys. Liz emphasized the importance of ethical considerations when analyzing employee data, such as avoiding the use of race and gender in models to prevent potential discrimination. She also highlighted the value of continuous learning in the rapidly evolving field of data science. Resources mentioned in the chat: [People Analytics Regression Book](https://peopleanalytics-regression-book.org/index.html) [Fundamentals of People Analytics with applications in R](https://www.routledge.com/The-Fundamentals-of-People-Analytics-with-Applications-in-R/Starbuck/p/book/9781032312327) [Predictive HR Analytics: Mastering the HR Metric](https://www.koganpage.com/product/predictive-hr-analytics-9780749482484) [Posit Conf 2024 Announcement](https://posit.co/blog/posit-conf-2024-announcement/) [Data Science Hangout Recordings and Upcoming Events](https://posit.co/data-science-hangout/) â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co LinkedIn: https://www.linkedin.com/company/posit-software To join future data science hangouts, add to your calendar here: https://pos.it/dsh (All are welcome! We'd love to see you!) Thanks for hanging out with us!
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hi everybody, welcome back to the Data Science Hangout. I'm Rachel, I lead Customer Marketing at Posit. I'm excited to have you joining us here today. I actually wanted to remind everybody that registration is now open for Posit conference, which is going to be in Seattle in August. And so Curtis will share a link here in the chat, but just wanted to remind everybody because I think the early bird pricing ends this week. I also put together a like post and a PDF on how to convince your boss for budget approval to go to the conference.
The Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. So we get together here every Thursday at the same time, same place. Is this anybody's first Data Science Hangout? I'd love to have you say hello in the chat just so we can all welcome you in and say hi too. But we are all dedicated to keeping this a friendly and welcoming space for everyone. So we love hearing from you no matter your years of experience, titles, industry, or languages that you work in.
With all that, I am so happy to be joined by my co-host today, Elizabeth Esero, Lead Advanced Analytics and People Analytics at AT&T. And Liz, I'd love to have you kick us off here and share with us a little bit about your role, introduce yourself, but also share something you like to do outside of work too.
Okay. Well, thank you, Rachel. And thanks for inviting me to be here this week. My role is I'm a data scientist on the People Analytics team at AT&T. And our focus in People Analytics is that we use data and insights gained from that data to help make business decisions regarding employees, work, and business objectives to try and improve the environment, improve business outcomes for the company. And something that I enjoy doing outside of work, I am an avid photographer and mostly nature photography and love visiting the local gardens or various areas around the Southeastern US and taking pictures, landscapes, wildlife, whatever I can find that's interesting. It's a lot of fun and very relaxing for me.
Journey into people analytics
I originally started working in the banking industry where I built a lot of my data analysis skills and joined an HR team in Bell South, which eventually became merged with AT&T a little over 20 years ago and was primarily hired to do data analytics. They were building their staffing organization. Back then, data analysis in HR was fairly new, at least at that company. Since then, I've moved into various roles and spent some time working as a project manager serving as a liaison between HR and IT as they were building tools for us so that our HR professionals could develop reports.
From there, I eventually decided, or as I was learning, more and more companies were getting involved in doing predictive analytics in HR and how it could be applied. I decided to go back to school and get my master's degree in data science and have graduated about six or seven years ago and have been applying those skills at work ever since then.
I had reached a point in my career where I was ready for a new challenge and was looking for what did I want to do next. At the same time, I like to stay informed of what's going on in the industry in general. I read a lot of news articles and attended a few conferences where I saw more and more companies were discussing the need to apply predictive analytics. I read a book by Eric Siegel where he presented a case study of a company that applied it to help reduce employee attrition in a business unit that had a very high attrition rate for employees, those leaving the company, that is. I was fascinated with the concept, did a little more research, started checking into programs, and decided on a master's degree program through Northwestern University, which fit my schedule perfectly, and worked on the degree in the evenings while I was working full-time during the day.
I was fascinated with the concept, did a little more research, started checking into programs, and decided on a master's degree program through Northwestern University, which fit my schedule perfectly, and worked on the degree in the evenings while I was working full-time during the day.
At the time, there were not a lot of programs available and I found a few universities, I think only four maybe, that were offering an online program that I could do remotely. And the local Atlanta universities did not have programs like that at the time. I believe Georgia Tech now does have one. So, one, I needed something with the flexibility so that I could do the classes in the evening while working full-time, and that was, narrowed it down tremendously in terms of which universities offered those types of programs. Now, I think there's a lot more courses available through organizations like Coursera and Udemy, and those were not available back then, but there's a much wider range of opportunities for people who want to learn data science. And there was, at the time, I think that was in 2013, 2014 is when I started my degree.
People analytics use cases
Many cases, we work closely with business unit leaders, and we also have HR business partners who serve as the liaison between HR and each of our business units. So they'll typically come to us with a business question that needs to be solved, or we may identify on our own a business problem and do the research and analysis ourselves. But some of the cases where we apply analytics would be reducing employee attrition, trying to find ways that for teams that have a high attrition rate, find out what are the aspects of the job or the work environment, or even the external labor market that could be affecting or making our employees more likely to leave the company, and figure out from there, what can we change to help improve things and reduce that attrition rate? Because it is very expensive for companies to train employees and replace them on a regular basis.
And another area, I've done a lot of forecasting. So if you're thinking about getting into People Analytics, time series forecasting is a very good skill to learn. And I've done forecasting to try and determine what our future attrition rates would be, estimating how our workforce will change in the future in terms of headcount, looking at skills and demand, and trying to figure out what changes do we need to make to the workforce as the company is making changes to our product portfolio and making future plans. And we do a little bit of forecasting two or three years into the future to try and give them some numbers to help estimate what's going on there. And another for employee surveys, there's a lot of need to analyze comments that employees enter in those surveys. So we've done a lot of text analytics there as well.
ROI and measuring impact
We can get into a lot of detail in terms of determining how much it costs to hire and train a new employee. And it varies from job to job. So we need to look at each type of job and how long they spend in training. How long does it take them once they finish training? How long does it take them to get up to speed to a more experienced employee? And in terms of filling the job, we also look at the cost of advertising that job, interviewing candidates, time spent interviewing those candidates and so forth. And all of that can be added together to determine what is the cost to hire and replace a typical employee for a specific type of job. From there, once we determine how much we've been able to reduce our attrition rate, we can go back to that cost of hire number and estimate a cost savings by reducing the attrition. And that's how we figure out the ROI for some of our projects.
Sometimes if the executives come to us asking us to help them with data analysis to solve a problem, it's much easier to get their support and buy in on those projects. In other cases, we've gone to them and said, here, we've done this analysis and we've determined, and I'm not familiar with the SCR template that McKinsey has specifically, but I think it's kind of similar to what we do. We describe the situation, what business problem were we trying to solve, what were some of the key insights that we found that could help solve that problem and make some recommendations. And then it's up to the business leaders to decide which of those solutions they want to implement.
Staying up to date
Challenging staying up to date with all the R packages. Tidyverse did not exist back when I was working on my master's program. So I've been trying to learn that and incorporate those packages into my work. And that makes it much easier. And I'm writing far less lines of code on my projects now. But as far as how I stay up to date, YouTube videos are often very helpful. I keep up with a lot of, I have a large collection of books now on R programming to help me stay up to date there. These days with people analytics, I've gotten to the point where I have LinkedIn connections that share news on what's happening in people analytics and the latest trends there and try to stay up to date that way with what's going on in our industry in general as well. Attending webinars, attending events like the data science hangout where somebody will mention something new that they're trying and I may do a little research on the web on that as well.
Employee surveys and attrition analysis
We do have exit interviews that are answered separately from some of our corporate surveys. We have a regular survey that we do to just measure what's the general attitude of our employees. And it consists of a combination of questions that have a agree to disagree Likert scale. And there's usually maybe one or two questions that just have an open-ended, please tell us about whatever the executives in the company decide is key point. It just allows the employees to type in whatever comments they want and from there we try and extract some of the common themes that show up in those comments.
And as far as analyzing other items related to attrition, we may look at unemployment rates, anything that we can find in the labor market from US census data that we think might be helpful for that specific project. As well as looking, talking with our business unit, developing a good relationship with the business leaders for the area of concern helps tremendously because then we can get insights from them on things that they are observing or that they feel might be a factor in causing attrition. And we work on finding what data can we get that backs that up and then do the analysis and try and determine does the data really show that what you believe is happening is truly happening. And that often helps. Sometimes we may dispel some myths and sometimes the data fully supports what they suspected.
Networking across a large organization
A lot of it is through knowing the right contacts. And our AVP does a really good job at marketing the team, I guess you could say, talking to other leaders within HR and even within the business units to let them know our team does exist and we're available to help them. As far as finding data for our HR professionals, we have a centralized system that we encourage everybody to use to get employee data to answer most of their common questions that they have. And that way everybody has one version of the truth. Everybody's numbers are matching from one report to the next and so forth. But then, of course, there's other systems such as our finance department has their own data sets that are more focused on the financial metrics of the company and budget metrics. And each business unit has different performance metrics that they measure for employees. And it's oftentimes trying to find the right contact person in order to figure out who is the right person to talk to to get access to that data and learn more about what's in that database. But I guess you could say a lot of networking is involved is probably the best way to go about it.
HR analytics community of practice
We just started the group in January and we have not had our first meeting yet. But we've created a virtual environment in Viva Engage. And each week I've been sharing different news tips with the members of the community. The idea was we wanted it to be a community where people who knew nothing about analytics and had no experience could just start from the very beginning and understand what is it that we do in analytics and how do we apply this to HR problems and solving those types of problems. So I started from the very beginning about sharing articles, sharing YouTube videos. I just do a little research to try and find something that's pertinent and relevant that HR people can relate to and share something new each week in short videos. Everybody typically has very busy schedules, so I'm trying to also keep in mind that they may not have time for a learning course that takes several hours, but spending 30 minutes watching a video to learn something new each week, they seem to appreciate that and it helps keep them up to date.
So we've been gradually increasing and getting into more complex topics including things such as how do you present your results of your analysis project to an executive, doing exploratory data analysis, typical what are the key analytic skills that people might want to learn from the very beginner to more experienced and even to more advanced and where could they learn those skills. We are planning in the second quarter of this year to start doing some meetings and I will be contacting some of our community members to get some feedback from them on whether they would prefer an open question answer format similar to this or would they want formal presentations or maybe we'll alternate between those. But it's been, I guess we're in the very early stages and in talking to other people that have started communities, I've learned most of the participants are kind of shy, so we haven't gotten to that point where there's a lot of back and forth conversation going on yet.
Hiring, promotions, and bias
Our team does not get into the details of making recommendations about individual employees, but we have done some diversity analysis in general. I personally have not worked on a project related specifically to promotions or hiring, but we do have some other members of our team that have been doing analysis related to those. We use data to try and help support those decisions, but at the same time, there are the individual hiring manager and the folks in our talent acquisition team get more involved in making those decisions when it comes to the individual employees in each individual job opening.
Large language models and text analysis
Yes, it has. We have a large language model in AT&T that's protected behind our firewall so that our employees do not have a risk of sharing any proprietary data with any external LLM tools. And I've found recently there was a survey that was done after an event. They wanted to get some feedback from the participants and had some open-ended questions in there, and I chose to use our internal LLM model and gave it all of the comments from the employees and asked it to give me some summaries. And by interacting with the LLM and asking it different questions, I was able to summarize those comments much faster than I would have using some of the traditional methods that I was using in R, so now that's become my go-to tool in analyzing comments.
Confidentiality and survey data
There, we have a team of IO psychologists, industrial organizational psychologists, that manage a lot of our surveys, and some of those surveys are done through vendors, and it depends on the situation. Sometimes they are completely anonymous where we have no ability to get that data, and sometimes we might be able to get that data in order to match things up at an aggregate level. We never report at an individual employee's information of what they responded to in a survey, but we can report it at an aggregate level for an entire department once it meets a minimum threshold in terms of responses.
Sentiment analysis and theme extraction
What we found in our questions is it's very hard due to the way our questions are worded. Oftentimes we're asking people what can we do to improve, and therefore we're not getting that some of the traditional methods of examining sentiment does not allow us to get really good sentiment from that. In other words, when employees are making suggestions for improvement, they could consist of a combination of positive and negative sentiment, and so we've kind of have not been using sentiment analysis specifically in the traditional sense of analyzing all comments and saying, is this one positive or is this one negative? We focus more on looking at what are some of the common themes or suggestions that we're getting out of that. What are some of the common concerns that we are seeing among our employees instead of looking at are they positive or negative?
Forecasting models
I have found many times exponential smoothing works very well for many of the cases and in cases where I've tested different types of algorithms. That one is often turned out to be the most accurate, but I will choose a few different methods to test and see which one works out well. We don't have a lot of seasonality in our time series. At least I haven't run into a case yet with that, but it's something that I do check for when I first start any new project for a specific business unit and see if that's something I need to consider in doing my model. So I haven't had much occasion to go into the more complex time series models yet.
The future of people analytics
I am seeing more and more companies are interested in using analytics and are hiring analytics roles. Richard Rose now maintains a website where he posts open people analytics jobs, and that has grown tremendously over the past few months. So I'm seeing more and more companies are realizing the benefit of analyzing their employee data and using that data to help support business decisions. Whereas traditionally, many, many years ago, HR was doing a lot of decisions based on gut instinct or specific scenarios from different business leaders. I think we're moving more and more away from those gut instinct decisions, and more and more companies are incorporating data analysis.
And in our case, we eventually grew to the point, I started doing data analysis 20 years ago, but eventually grew to the point where now we have a central team within AT&T that is focused on our people analytics projects. And we have various team members that focus on reports, and some that focus on data science and machine learning. I think we're going to get more and more efficient and faster at being able to produce those insights to different business leaders. And more, I think, just more advanced analytics and more sophisticated analytics, let me put it that way. I can see that growing and becoming more commonplace among companies.
Advice for organizations starting people analytics
I think if you're just starting a team, developing good relationships with your other leaders in HR to find out what are their big concerns, what are their pain points where we could apply analytics to help solve problems would be a big part of that. And that will help you decide what direction you want to take in building your team if you're just getting started. And from there, if you can find the HR leaders, if they are in touch with the business units, they can also tell you what are the pain points of the different business units.
Since our company is quite large and complex, we found it's very helpful to have people that specialize in serving as a liaison with the different teams and different organizations within our company. And that helps direct them to the appropriate team. But at the same time, they can help get familiar with what are the common problems that this particular business unit is known for experiencing? And are we seeing the same things coming up over and over again in their requests? And from there, can we be proactive in trying to develop other tools to help them answer their questions or solve their problems?
If you are on LinkedIn or any other social media where you can network with other people analytics leaders, just within the past two or three years, there have been a lot of people analytics meetup groups developing in different cities. And from there, that's been a tremendous help to be able to go to those meetings and talk to people in other organizations, other companies, to discuss common pain points that we're all experiencing, common solutions that different people have developed. And you get a lot of ideas from those meetings as well.
AI and the future of hiring
Going back to what we said earlier in relation to large language models, when those large language models were introduced within AT&T, there were a lot of discussions of what would be the benefit of that and how that would affect people's jobs. I think what we've ended up concluding so far is it does help those who use those tools, it's helping them become more efficient. It's serving as a great resource to get answers to our questions much faster than we could on our own when we were searching Google or whatever other tools. And there's the common phrase that a lot of people are saying is your job won't be replaced by an AI tool by itself. It has trouble making decisions, especially the human aspect of those decisions and ethics and all of the things that factor into the types of decisions that are made in human resources. But people using AI will become more efficient, more effective, and better able to respond to those questions.
There's the common phrase that a lot of people are saying is your job won't be replaced by an AI tool by itself. It has trouble making decisions, especially the human aspect of those decisions and ethics and all of the things that factor into the types of decisions that are made in human resources. But people using AI will become more efficient, more effective, and better able to respond to those questions.
Tools and packages
Primarily I use R right now. When I first completed my master's program, I alternated between SAS and Python and R and eventually just got to the point where I found it was easier to focus on one language instead of trying to translate from one to another. And so I'm exclusively an R programmer these days. We have a few other team members that work with Python. As far as R packages, I love Tidyverse. Some of the packages related specifically to time series forecasting like Feast and Fable. And there is Text Analytics. Oh, I can't remember the package names specifically. I'd have to pull them all up to remember them. But at this point I've built my own personal library of R code. And so for different types of projects, and depending on what the project is, I can usually pull up the code that I've used from a previous one, copy and paste, and modify it as needed.
Career advice
In general, I think prepare yourself for that conversation with your supervisor. Do some research. These days there's a lot of information available on different websites to find out what is the average salary for different types of jobs. So you can see whether or not you fall within the expected range in the market. And you can present that data to your supervisor as well. And show your accomplishments. Show the benefit to the company in terms of how that helped the company in your accomplishments. So instead of just saying, I build a report that does this and provides these insights, think more in terms of how did that help solve a business problem? Or did you save time, become more efficient in order to help our company progress in terms of being able to get more work done, be more productive, and things like that. So do your research, be prepared for that conversation, and bring all of that in and schedule some uninterrupted time with your supervisor to discuss it.
Ethics in people analytics
The biggest difference is when we are analyzing employee data. So we have to constantly keep in mind the ethics of what we do with that data. And in our team, we work closely with our legal department. We also have a privacy team that can get involved in our projects and has given us guidelines as to when are we doing things that, even though there may not be a law against it, we feel it could be considered unethical or we just feel this could be an invasion of privacy. So it's important to think about those things when analyzing employee data. For example, when we're looking at attrition, you wouldn't want to put race and gender in a model because that could be leading to a potential situation where you're discriminating against those employees. So we're constantly thinking about that. We're also constantly thinking about testing our models for bias to see, is it harming any specific group? And I think that's one area that's very different from other types of predictive analytics or data science projects where you're not analyzing employee data. That's one of the key differences there is we constantly think in terms of, is this appropriate to put into our model? And we feel very comfortable consulting with others in order to help us make those decisions when we need to.
Managing a code library
It's a very simplistic system right now. I have a series of folder. I have one folder on my drive, and I've tried to put things into different categories according to the type of project it is. And from there, that allows me to easily go back and get that code. At the same time, I still have files from each analytics projects that I completed, which has all of the input files and everything else. So, in a separate folder, I just copy the R code only and none of the other files. And that allows me to have a quick reference where I can say, okay, I know I did a model that was using survival analysis in the past for this type of project. I can go to my folder and copy my survival analysis code for the next project and just replace it.
Continuous learning
Best career advice I've received is continuous learning. Always be willing to keep learning your skills, keep your skills up to date, and as I've seen various trends, I've even volunteered to go take classes and learn those skills on my own before being asked or told that this is the direction we're going to take. Maybe that's the best advice I can give. Necessarily, when you work in data science, you've constantly got to keep your skills up to date because things are changing, and they're changing fast in this field.
Yes, I did have a conversation with my supervisor and leadership above my supervisor as well because AT&T has a tuition reimbursement program, and so I gave them the justifications of how my degree would benefit the company and applied and told them I would like to also have it included in the tuition reimbursement program as well. Even once I got to the point where I narrowed it down and selected Northwestern, I then gave them information about, here's all of the information of what I will learn in this specific master's program, and that had to go up several levels in the company in order to get approval for it.
LLM approach for survey text analysis
More of a prompt interface. I had all of the comments without any identifying data. It was just a list of all the comments that were received from the surveying responses, and in our internal LLM tool, I was able to upload a document that had all of those comments in there, and then I just started interacting with the tool and told them, this is what's in my attachment, and please examine all of these comments and give me a summary of the common themes that were appearing in these comments, and we went from there. It became a two-way conversation with the LLM. As I learned better how to interact with it and tell it what I wanted, then I was getting better responses from it, but it never gave me cosine similarities or any math values or anything like that.
Recommended resources
There are a few books that have been published that focus specifically on coding, and I think some of them also cover Python coding, and have specific examples related to people analytics projects, and one is from Keith McNulty, Handbook of Regression Modeling and People Analytics. He has made it available online. Craig Starbuck recently published a book on the fundamentals of people analytics with applications in R, and there's also a book on predictive HR analytics from Dr. Martin Edwards and Kirsten Edwards.
Bias in MLOps pipelines
As far as the employee-based models are, one thing is our privacy team does require that we do testing for bias, especially with regard to race and gender in the results of the models, but we also consider what information are we putting into those models in the beginning and could any of that really relate to possible bias in the outcome. The other requirement is the privacy team or the ethics team, whichever way you want to refer to them, they often require us to give exact details of here's all of the data that was put into the model. Basically any machine learning models that are using employee data is reviewed by their team and analyzed from a legal perspective as well as ethics and privacy in terms of how we want to work with that. Right now, a lot of our models have been focused on just gaining insights and helping the business make decisions and we're working towards putting those models into production so that we're getting predictions on a regular basis and so that's the direction we're heading to in the future.
Thank you all so much for all the great questions and huge thank you to you Liz for joining us and sharing your experience today. We really appreciate it.