Kevin Snyder & Raju Rayavarapu @ U.S. FDA | Data Science Hangout
We were recently joined by Kevin and Raju @ the FDA to chat about democratizing data science by teaching people the basics of data wrangling, data science and the power of open source and utilizing principles of systems biology, pharmacology, toxicology, bioinformatics, and statistics to collaboratively develop open-source software solutions that support decision making and policy development. Links mentioned: FDA Digital Transformation Symposium: https://www.fda.gov/news-events/fda-meetings-conferences-and-workshops/2023-fda-digital-transformation-symposium-12042023 precisionFDA: https://precision.fda.gov/ R4DS Online Learning Community: r4ds.io/join R Validation Hub Contact: https://www.pharmar.org/contact/ Hangout LinkedIn Group: https://www.linkedin.com/groups/12610075/ Speaker bios: Kevin Snyder - Associate Director of Nonclinical Informatics at U.S. FDA Kevin received his Bachelors in biochemistry from the University of Maryland in 2008 and his PhD in neuroscience from the University of Pennsylvania School of Medicine in 2013. He currently serves as the Associate Director of Nonclinical Informatics in the Office of New Drugs in the Center for Drug Evaluation and Research at the US FDA where he manages data science and informatics initiatives to support the pharmacology/toxicology review program. These initiatives include research efforts to develop methods to optimize the regulatory use of standardized electronic CDISC-SEND-formatted toxicology study data as well as internal informatics projects to promote the development of scientifically sound, data-driven regulatory policies. Dr. Snyder also leads an agency-wide Data Science and Software Development working group that is focused on building out the organizational infrastructure necessary to support the work of data scientists across the agency and is an active collaborator with several consortia efforts, e.g. CDISC, PHUSE, and BioCelerate, to improve the implementation and use of the SEND data standard. Raju (Rama) Rayavarapu - Data Scientist at U.S. FDA Raju comes to the FDA from the great state of Pennsylvania via South Bend, IN and Memphis, TN where he did his Ph.D. and post-doctoral work. He is the lead of ODAR’s DataForward Initiative which is focused on upskilling FDA staff in data related skills and harnessing the power of the incredible existing FDA data science community to drive data literacy and the joy that comes from working with and understanding data.He loves to spend his time talking FOSS (Python/R/whatever), Natural Language Processing, artificial Intelligence (AI), and all things data science. He also spends any night he can staring into the universe with his two dogs and his telescope. ________________________ ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co LinkedIn: https://www.linkedin.com/company/posit-software To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!) Thanks for hanging out with us!
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hi, everybody. Welcome back to the Data Science Hangout. I'm Rachel. I lead Customer Marketing at Posit, and I'm so excited to have you joining us today. I was just saying today I'm calling in from Montreal, so a bit different. But the Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. We get together here every Thursday at the same time, same place.
If this is your first time joining us, it is so nice to meet you. If it is anybody's first Data Science Hangout, we'd love to have you say hi in the chat so we can welcome you in as well. We're all dedicated to keeping this a friendly and welcoming space for everyone, and love hearing from you no matter your years of experience, your titles, industry, or languages that you work in. It's totally okay for you to just listen in here too. Maybe you're out walking your dogs or in a coffee shop, but also awesome to be part of the party that happens in the Zoom chat.
There's also three ways that you can jump in and ask questions or provide your own perspective on certain topics. So first off, you can just raise your hand on Zoom, and I'll keep an eye out and call on you to jump in and ask your question. You could put questions in the Zoom chat and just put a little star next to it if it's something you want me to read instead. Otherwise, I'll just call on you to ask it. And then third, we have a Slido link where you can ask questions anonymously.
I do like to just add, we do have a LinkedIn group for the Hangouts. If you'd like to join that, it can often make it a bit easier to find some people that you might meet here in the chat too. And a quick note that if you are watching the recording at some point in the future and want to join us live, the link to add it to your calendar will be in the details below.
But with that, I am so excited to be joined by my two co-hosts today, Kevin and Raju. Kevin is Associate Director of Nonclinical Informatics, and Raju is Data Scientist at the FDA. And I'd love to have you both maybe just kick things off with introducing yourselves, sharing a little bit about your role, but also something you like to do outside of work too.
Introductions
I'll go first. So I'll start with what I like to do outside of work. I really like to play music. I play bass guitar, mandolin, guitar, learning banjo. All my kids, everyone has a ukulele in my family. The kids don't really play as much as I wish, but yeah, I think that's a big part of my life is playing music. And I'm into like jazz and bluegrass and Grateful Dead and all that kind of stuff.
So I got my doctorate in neuroscience and I was studying how stress affects cognition in rats. And I did a little bit of MATLAB programming and I was playing around with like this machine learning model to model their behavior, which I thought was a lot of fun, but my project really wasn't funded for that. And then I did a short postdoc after grad school learning bioinformatics stuff, doing RNA-seq data analysis, where I learned how to program in R and Python and shell scripting.
And then I got a job soon after at FDA, originally in CDRH and devices in vitro diagnostics for microbe, for infectious diseases, basically. But then I moved to CDER as a toxicology reviewer, so that reviewing animal toxicology studies that we conduct primarily before testing humans, although we continue to conduct them as we test longer durations in humans.
Right around the time that I joined as a toxicologist, the FDA was gearing up to start receiving structured data sets with these toxicology studies. Before that, it was just the study reports on PDF and actually paper not so long ago. But with getting data sets for these studies, that opens up a lot of possibilities of interesting things we can do with cross-study analysis to understand the background rates of findings or really look across large volumes of data and make better policy decisions and understand our policies better.
And so I started volunteering on projects to do whatever I could with these data sets because I had this background that most of the other toxicologists really had no need to know how to program because it was never really a thing in regulatory toxicology. And then eventually, I was lucky enough to get a position created for... She came to me and was like, I have an FTE position for you to do that because she was kind of under a lot of pressure. And so for the past four or five years, I've been in that position where I just do data science and informatics projects to support data-driven policy development for farm tax.
And I'm trying to grow not just the FDA, but with the industry as well, this open source consortium effort to utilize this data effectively and build standard tools because we have standard data.
Raju, do you want to go next? Yeah, thanks. So I'm Raju Rayavarapu, and I guess my main arc is going to be the data science work and stuff that I do is just sort of something that I discovered as like a passion and a love sort of thing. So what I do for fun is in the last few years, especially since like, you know, the COVID stuff, I started to make a mission of like learning one new sort of life skill, as I've been calling them. A couple years ago, I learned how to like ride motorcycles, got my motorcycle license. But this year, I'm doing horseback riding, and I'm getting better at ice skating because as my partner, who's a fabulous ice skater, likes to point out, you know, using inertia to stop, like hitting the hitting like a wall to stop isn't like the best way to stop.
So my background is a little similar to Kevin, but a little different from Kevin. So I'm a PhD molecular biologist and biochemist. And so, you know, I did a lot of wet lab work, I spent most of my time, you know, looking at cell death mechanisms and looking at, you know, signaling pathways that govern cell death, particularly in like metastatic biology. And then from there, I went to St. Jude, where I worked on autophagic regulation and metabolic regulation.
While I was there, I got kind of interested in trying to figure out what the government and, you know, what industry and government and academia can do together, which led to a role at the FDA looking at public private partnerships. And when I'd come into the agency, I'd done a little bit of scripting work. My dad's a computer programmer. So like, I was just kind of exposed to it at all times. I used a little bit of R when I was working on like high throughput screening data to make better plots and all that fun stuff that you can't necessarily do with Excel.
But when I come to the FDA, I'd been just serendipitously sitting next to a friend who's a fabulous Python programmer, he's a semantic web expert. And I was explaining my project to him. It was like, hey, instead of doing it via like, you know, basic methodologies and using just like static, generating a static output. He's like, why don't you learn Python? It's not that you know, you'll pick it up. It's not that hard. He's like, it's kind of like a gateway towards a lot of other stuff. So I started dabbling with Python and R. And I started working on building dashboards and doing things like, you know, doing like web scraping and doing, you know, auto matching of concepts and terms and modeling and all this fun stuff for text.
From there, I ended up being picked up by the Center for Tobacco Products that had a lot of data that they were working with that was unstructured. And I spent a lot of time working on automation methodologies to structure their unstructured data. And while doing that, I was elected as co-chair of the Scientific Computing Board.
And so Jamie and I get to essentially advocate on behalf of the scientific and research computing staff at the agency and their needs and, you know, what kinds of interesting technologies and tools that they want or have access to and ensuring that, you know, that is represented in a meaningful way. I also then was picked up by the Office of Data Analytics and Research, which is the FDA's chief data officer shop. And I'm a lead data scientist in there, helping establish some of their data science pipelines, some of their analytics pipelines.
The data science and software development working group
So, I'm talking a little bit about the DSSD, data science and software development working group, which really started when Raju and I, we saw this talk, I forget the name of the guy. He was from the University of Buffalo Medical System. And he gave this awesome talk on how they were doing all this great informatics work, you know, and they seem to really have not just their data organized, but also like the infrastructure to support analysis. And me and Raju were talking to him and kind of just complaining about how difficult it was at FDA to do good data science work.
We caught the air of one of the more senior staff, this guy, Mark Walderhug, who's awesome in CBER. And he was kind enough to set up, kind of pull some strings behind the scenes and set up this cross-agency working group that he told me and Raju to then go ahead and lead. And so, we, you know, went through our emails and found all the people that we knew across, you know, we tried to get at least a couple people from every center that was doing data science work. And we told them all, like, join this meeting and let's get together and start, you know, figuring out how to support each other.
We decided to create three different workstreams so we could give leaders to each of those workstreams, a couple leaders to each of them. And then we had this kind of group of about nine people who were, like, core people that were invested. And so, we set up a community workstream, an infrastructure workstream, and a process workstream with the idea that, you know, community workstream would be an opportunity to, would focus on bringing people bringing people together.
We did a hackathon at one point, which was really fun to, we formed some teams of people to try to solve problems together. The infrastructure workstream was something, actually, Raju spearheaded that, where we just tried to take a survey of what, what are all the platforms about, because the different centers at FDA kind of have siloed systems sometimes, and not everybody knows what's available to each other.
We started this group, like, the month that the chief data officer started at the agency, and so that was convenient, too, because he kind of was interested in what we were doing, and it was a good window for him into what the data science community was doing, and we've been able to pass off a lot of stuff from, a lot of the kind of grassroots stuff that we collect, or build, we're able to kind of pass off to his office to try to formalize, or, you know, make enterprise level.
We did this SWAT teams pilot, where we'd take a problem, like a real problem that somebody had in the DSSD, or someone they knew had, and we asked them to put it out there to the group, and ask for volunteers to come together and try to solve it. During the COVID-19 pandemic, there was a group that was trying to organize data related to that, and we were able to provide some volunteers to support that.
There was a group in CFSAN, so the foods part of FDA, where they're tracking economic adulteration cases, and they had like this Excel, you know, they're using Excel to track all these cases, and they're like, this is probably not the best way to do it, and we were able to build a pretty simple, but effective, you know, R Shiny app, where you just throw it into a DT table, and put some UI stuff, so that you can use control terminology to, you know, select inputs, and things like that.
I think that template of taking, somebody has some information in Excel, moving it to a DT table, and putting some UI around it, that's like a really common use case, where it ends up feeling a lot more powerful for the end user than what they had before, you know. Amazing how much time you can save somebody, or how much better you can make their workday by just giving them that tool.
I think that template of taking, somebody has some information in Excel, moving it to a DT table, and putting some UI around it, that's like a really common use case, where it ends up feeling a lot more powerful for the end user than what they had before, you know. Amazing how much time you can save somebody, or how much better you can make their workday by just giving them that tool.
Who are the customers?
So, our customer varies depending on, you know, the pipeline that we're either implementing or utilizing. Sometimes, you know, the customer is internal folk that have some sort of data-related issue. We also collaborate with and work with across agencies, right, like we'll work with other agency partners, and so if they have a data need and we're the data repository for that, we'll, you know, work with them and generate either reports for them or produce data tables or data sheets that we can turn out.
In general, the FDA is obviously serving the American public, and a lot of our interactions are kind of customers are like the sponsors who are applying for approval of their, you know, clinical trials or their products, but a lot of the reviewers are kind of the end users who are interacting directly with sponsors and making regulatory decisions. Those aren't typically data science people, and so, yeah, a lot of the data scientists at FDA are kind of serving those review regulatory people by feeding them data and tools to utilize data better so that they can, you know, support their customers outside better.
Open source at the FDA
From my perspective, I think there's been a lot of infrastructure built around SAS in the industry and FDA, and changing that infrastructure over is, causes friction, you know, and there's not a, so there's, it's a process to, you know, things are built around expecting things to be they were, the way they were when they were set up, even the submission process, and so it takes some work to change that. I think a lot of people in FDA see the value of open source, but I think modernizing the review and submission processes, it just, it takes a lot of work to make sure that it, you can't break it while you're fixing it, you know, you have to keep it running, you have to keep the plane flying while you're building it, you know.
The interesting thing is that is kind of a misconception. And, you know, and it could be that there was either, and here's how I'll tackle this one, right. I'm a big proponent of using open source technologies and tools, just much like Kevin, as well as many of my FDA colleagues that are on the call here, because, you know, it gives us flexibility, and it's honestly can be viewed as an industry standard technology platform.
We were able to spin up a program in the office of the chief data officer called Data Forward. And the Data Forward initiative is essentially trying to increase data literacy and data skill use, data skill tool usage at the agency. And what we actually focus on is because there is such a strong community of R users at the FDA, we actually give people the opportunity to learn how to use R and RStudio and Shiny to actually do some work. And that's not from any perspective other than we have people that use it, and it's something that we can get people spun up into fairly quickly.
We give people exposure to things like Tidy, if that's something that they want to use, Tidy R and Tidyverse. And they become a little bit more capable at writing their own scripts to do their own data analytics and data analysis and can go back to their home offices and function as the data science advisors, mentors, or the actual scripters. But also it makes it easier for them to look out into the universe and see what the best tools that are being utilized by, whether it be academia, whether it be industry, are and become a little bit better at sort of bringing those technologies in and actually working with them.
So would you say it's been through this community building that you're starting to, I guess, remove some of the open source panic that may have normally or may have existed before?
Yeah, I think that there's both a cultural shift as well as like there is this idea of this open source community, right? I can't understate how much, rather I can't overstate and I don't want to make sure, I want to make sure it's not understated, that when I came into the agency, I got lucky and was sitting next to a very kind individual that was willing to teach me and take time to teach. And then I met people like Kevin, Alexis, Dave, like all these people that were willing to take the time to sort of like, you know, walk you through something. And so we started building that community and from that community we were able to show that, hey, actually, you know, there is a lot of this literacy that exists and there's a lot of interest to learn this stuff.
And as more people start to use tools like R and Python and other open source technologies, it's sort of this like ground up sort of acceptance. And from a top-down perspective, there has been some, you know, cultural shifts and changes that have kind of helped push us forward. Bringing on a chief data officer was like, you know, a really big and helpful thing because, you know, he's very data and data is a data forward individual, right?
Infrastructure challenges for R submissions
I think I want to preface by saying, kind of reiterating that, again, me and Raju are not really, like our day jobs aren't really in the biostats community of, you know, receiving these coded submissions, you know, the SAS code and potentially R code, hopefully more and more R code. The stuff that we're doing, me and Raju, it's more enabling scientists who aren't statisticians, for the most part, to utilize open source to do their jobs more efficiently, to do their science better, or even admins, you know, even to do administrative things better.
From my perspective, there's less inertia to fight against when, you know, when there's, when you're working against, when you're putting a solution where nothing was in place than if you're changing the solution from SAS to R, you know, if you're talking to people who've never done any data science, there's so much low-hanging fruit, and there's, there's fairly little resistance to move in that direction, and so I think that's where our efforts end up flowing.
Most of the stuff that we've been focusing on is attempting to enable those that have either little experience, or an experience in another piece of technology, and providing them with, you know, an additional tool set, and again, connectivity to this FDA data science network, so they can do things like, whether it be, you know, automating a daily task to actually being able to finally tackle a data project that they previously couldn't.
The example that I use a lot when we're working through with the upskilling teams, is I was once posed a question, and it was on a small team with others, we, there were three of us, we were posed a question where a team thought that x was the outcome, and it was purely because they couldn't handle the mass amount of data that they had to essentially call, like, mine through to get to what they needed, but using a tool that, and again, I was using Python, so using Python, we were able to say, hey, actually, the answer is something else, purely because we were able to bring all that data together, and then show them that actually, like, we were able to correct their hypothesis.
Low-hanging fruit use cases
I think someone, I saw something in the chat where, you know, anyone who's using Excel in any organization, if they're, you know, if they're doing repetitive tasks in Excel, they, those are always low-hanging fruit examples where if, you know, if they have the time to get over the learning curve of using R or Python to do whatever they're trying to do, I'm sure they can do it better, you know, using better tools.
A recent one was we got an external request for some data that was on the FDA.gov page, and the person that sent us the request was, I think, an economics grad student somewhere, and they had made this request, and they were, like, the way that the data is stored, which was stored as flat PDF files, they're, like, it's going to take us forever to get this data out. And so I checked with my supervisor, my, the CDO, I checked with my boss, and I was, like, hey, can I, like, I'm just going to do this for them real quick. And so I was able to convert those things from flat PDFs and send them back to them as CSVs, which is what they were looking for. They were, like, we want to do some, like, you know, complex analytics around this thing. They're, like, this will take us hundreds of hours to do, and I was, like, I can do this in, like, an hour.
The Data Forward upskilling program
We presented about this at the ODT symposium, which happened in mid-December of 2023. So, we started off with a pilot, and we actually used our existing data science community to reach out to individuals that they thought would be good members of the pilot to provide us with feedback on our process. We actually take an application pool, and we use the existing FDA data science community to review these applicants, and I think at the last, last application set, we had something like 200 or 168, or some, some, like, large number of people that had interest.
One of the big things that we look for is, we look to see what their, what their perceived value of these skills is for them to the specific organization or area that they are within the FDA, and anyone that can articulate that fairly clearly, we usually are, like, they move up the pool, because that's exactly what we're looking for, right? Not people that say, oh, well, I already do all this stuff, and I want to hop in with you all, which we're, at that point, we're like, you kind of don't need us.
So, we bring these folk in, and we have them kind of, we organize them into teams around a project with a mentor, and then before they even hop onto the project itself, we give them four weeks ahead of time, and we're experimenting with expanding that four-week piece out to do a standalone program, but we give them four weeks where I walk them through a basic dummy data set that I've created, and we farm it out to the data science communities and the mentors that we select for the program to then walk them through the basics of R, and then we hand them the project and say, okay, here's the specific project that we want you all to, you know, accomplish.
What I like to think about is the basic applied science model of 70-20-10, right? 70% is experiential, where you actually have to put your fingers to the keys and have your code implode, because we want them to understand that it's not going to be something where they can just write a piece of code, and it's going to be magic every single time. Most of the time, they're going to spend their time wondering what this weird error is or how, why my data exploded. 20% of it is with relationships with their mentor, so we pick somebody that we find to be a champion for data science and somebody that we think could really guide people through their learning process, and their job is to essentially help them find answers, not necessarily give them answers, right? So, we do a 20% piece there, and 10% we give them formal, like, lessons and lectures.
We found that that's pretty successful, and it's, you know, some people will go through it and realize they hate this, and they never want to do it again, and that's okay. We want them to at least attempt it, experience it, and realize that it's not for them. Some people will find that they enjoy it, but not enough to continue to script, but they become advocates for data and data-related activities at the agency, which is huge, right? And then we find some people that become the next generation, or the next, like, you know, cadre of advocates, who we then draw back in as mentors to teach the next team of people, and we're trying to build it as a self-feeding mechanism.
And then we find some people that become the next generation, or the next, like, you know, cadre of advocates, who we then draw back in as mentors to teach the next team of people, and we're trying to build it as a self-feeding mechanism.
And they get a little portfolio piece out of it at the end, right? It's sort of something they can point to and be like, hey, look, I made that, and it's, like, it's kind of a very, like, fuzzy feeling. Sometimes the answer is posted by somebody, or someone will be like, hey, let me help you with that. Let's hop off onto a side call before I even see the question, before some of the mentors even see the questions, which is, like, it's because this community wants to advocate and educate and build each other up, and it's kind of delightful.
Validation and open source
I think what we have to deal with is that the validation space is not rational, and very often we, if we call open source, if we didn't call it open source, if we called it something else, maybe there would be less issues with the validation. Because right now, we kind of delegate the validation to whoever the developer is, and, you know, I don't think that the SaaS developers are more reliable than, you know, the SaaS developers don't have emotional investment in the application, right? You very often can't get to them and all that. So, I think some of it is just behavioral, sort of behavioral science. We have to kind of change the narrative and make people understand where the validation truly is necessary.
And then with Excel, that's, I think that's a great example. Like, you can, you know, Excel may be validated, but then a lot of the effort that you do in Excel is coming from the people's brain. So, what are you going to validate there, you know?
There are cases where validation is more important or less important, right, because the consequences are bigger or smaller, right? The efficacy, statistical analysis of efficacy of drugs, like that's like very critical. And that can come, it's a pre-specified plan, you know, and it can come down to like millions and billions of dollars on the line about how a p-value turns out, right? And those cases, like validation is obviously super important, right?
The questions about validation aren't as, you know, it's not, it's not like you were saying, it's not rational to apply high standards of validation to every single step we take, you know?
And so it's actually a pretty fun debate that I get to have as part of the hat I wear from the scientific computing board for the agency where we get to talk to the, like the FDA IT infrastructure folks and the FDA IT, like your enterprise management folk about, you know, how we do one thing or another. And so like, you know, we always talk about, you know, community-based validation and management versus like, you know, security and the security folk doing their like review and management and all that fun stuff. And honestly, the best solution that we have right now is that the security team do validate those packages. And if they have any interesting questions, they reach out to the scientific computing board, myself or Jamie Pettingill, and then we're able to either route the question or route the answer and find the solution.
Tools and recommendations
Like I said, we use our Shiny for some of our projects. We also are looking into like, you know, Dash to do, you know, Python-based development and, you know, just Dash in general. One of the other tools that we use is KNIME, for those of you that have ever heard of or used KNIME, which has been around for a really long time. KNIME is another good one. Honestly, it comes down to like a day-to-day usage of whatever the best piece of technology is. If somebody wants to do OCR work, like, you know, they can either chase down a commercial product that exists within their specific organization or, you know, if they reach out to me, I'm like, hey, have you looked at Tesseract?
R is easier to install at FDA because statisticians use it. There's, you know, so there's already support for it, whereas Python is a little bit more neat, you know. Whatever is already available to all your, you know, your organization, if you can do it with that, do it with that. And then, you know, try to advocate for, to fill the gaps you need. But it's not, so it's not saying R is better than Python, but it's easier to get installed at FDA. It's easier to use here. So we tend to lean towards it, especially for like new data scientists.
Career advice
I mean, I, for me, my career has been built on the old adage of don't ask for permission, beg for forgiveness, where, you know, I just, I do things that, you know, like if you see the opportunity to do something that's going to add value to your organization or your coworkers or the problem you're trying to solve, like don't ask for permission first, because it's going to, it takes longer to get permission than it does to just do it. And once you do it and you show people, Hey, I did this thing and it's really valuable. They usually just say, thank you. And we're, you know, it's easier to explain it after you did it than to try to explain and get permission.
My career has been built on the old adage of don't ask for permission, beg for forgiveness, where, you know, I just, I do things that, you know, like if you see the opportunity to do something that's going to add value to your organization or your coworkers or the problem you're trying to solve, like don't ask for permission first, because it's going to, it takes longer to get permission than it does to just do it.
And, you know, I'm going to shout out to a grad student that I worked with Dr. Callie Versagli, if she ever watches this video. But I mean, this is a piece of advice I've gotten a lot in my life, right? But this one time really stuck with me where if you see something that you find is a need, and it's something that causes you grief or frustration, it's okay to roll up your sleeves and use, you know, Kevin's model of like, you know, ask for forgiveness or whatever you want to do, but roll up your sleeves and actually attempt to make a change, right. Rather than just complaining about it.
If you see a need, and you can find either someone that will help you advocate for it, or you can find some way to solve that problem, or at least help somebody solve that problem, you should just lean in and work with it within the bounds of, you know, what a human being can accomplish. And don't burn yourself out and all that fun stuff. But, you know, get your, just attempt to make a change, I guess.
My other two cents motto is all science is data science. All science is data science. All data science, right? Even grocery lists.
Thank you all so much for being here and for spending time with us in this community. And if this was anybody's first time and you want to join us back next Thursday, we are here every Thursday, same time, same place. But Raju and Kevin, really appreciate you joining us here and sharing your experience. This has been great. Yeah, thanks for inviting us. This was a lot of fun.