Name: Beyond Grades: What a Landmark Meta-Analysis Reveals About Assessing Future Healthcare Professionals
Uploaded: 2026-08-19T18:31:57.127Z
Duration: 1 h 29 s
Description: Beyond Grades: What a Landmark Meta-Analysis Reveals About Assessing Future Healthcare Professionals

Transcript for "Beyond Grades: What a Landmark Meta-Analysis Reveals About Assessing Future Healthcare Professionals": Welcome, everyone. Thank you so much, for joining us today to talk about Beyond Grades, a landmark meta analysis talking about how we can assess future health care professionals. My name is Kelly Dore. I am founder here at Acuity Insights and VP of Science and Innovation. I'm also the co creator of the Casper Assessments, which I developed at McMaster University, where I am also still an adjunct professor in the Department of Medicine, for the Division of Education and Innovation. I'm really thrilled to have all of you here today and excited to jump into talking about the research, the data, and the implications for our programs of all different shapes and sizes. Today, I have three speakers joining me today, who I'm really excited to introduce. I have assistant professor Ivy Erickson, who is a coauthor on the meta analysis that we're gonna be talking about today. She's also the clinical assistant professor at the University of Wisconsin La Crosse, and she's gonna be talking really about the evidence today, and then we're gonna be talking about implications as well. Joining her is doctor Brian Steele, an associate dean for admissions at the University of Georgia School of Medicine. He's gonna be helping us figure out implications, and we're gonna be talking about that. And joining him to talk about implications is doctor Michelle Smude, who is vice provost for enrollment management at Geisinger College of Health Sciences. And we're gonna, across the three of us, also talk a little bit about, or the three of them, talk about how to put this evidence into practice. We welcome each of you, and we're really excited to, have you join us today. Before we jump in and get started, though, I wanna talk a little bit about the background and the context for today's session, what we're gonna do, and and setting some expectations. It is not a secret to any of you that admissions leaders are under a lot of pressure. We have to build strong cohorts, and that means identifying applicants who can handle both the academic demands of the program, but also who have judgment, communication, and the professionalism that we need in training our successful health professionals. Those decisions are incredibly complex, and we all know that the stakes are very high. Programs need admissions practices that are structured, consistent, really grounded in the evidence, while still preserving the essential role of human judgment and making decisions that are right for our schools and our programs. That really brings us to the research that we're gonna be discussing today. So we're going to be overviewing a publication that was published in medical education. It's a meta analysis bringing together 27 studies, making it one of the largest synthesis of validity evidence, especially for constructed response SJTs that we have to date. It found a lot of interesting findings that we're gonna talk about and unpack as we go through today, but also talk about what that means and the implications for some of these conversations for our programs. But before we jump in, I just wanna do a little bit more stage setting in talking about why these qualities are important. This is not new to any of you on this call. All of us know that being a great clinician is more than just knowledge and technical skills. We know that groups like the double AMC, CanMEDS, and the ACGME milestones all set out expectations around these other attributes that are critical and important, not just to becoming a strong health professional, but also to the success and well-being of our patients. These attributes shape patient outcomes, and they help us make sure that we're really thinking about the whole student as they're going through and all of the attributes that they need to develop. These six qualities on the screen are just examples of some of the things that regulators and accrediting bodies agree that are important to go alongside the academic knowledge and technical skills. You can find out some of these things from personal statements or interviews, but we know that there are limitations around the reliability and the predictive validity of some of those methodologies. And so that's what really brought us to trying to look at and understand situational judgment tests. That's the approach that we took back in 2004 when we developed Casper or started to develop Casper at McMaster University. The goal was how can we defensively measure some of these attributes that we know are important so they can be used alongside other metrics? And so situational judgment tests, is our starting place where we started in 2004, and it was where we wanted to understand how we could measure these attributes in a way that provided admissions programs or admissions committees the information that they need. So how do you know that you're measuring something like judgment or like professionalism in a very standardized way? Well, that is where situational judgment tests for us came in. And it's worth pausing on an important distinction here is that not all SJTs are built the same. If you take the very simplistic scenario that we have on the slide, you're working on a team and you notice a colleague has made an error that could affect a patient, what do you do? The traditional ways where SJTs that have been used in the past, they're more about recognition SJTs. So the candidate is given a list of prewritten options, and they can report it to their supervisor, speak to a colleague privately, document it, and say nothing, or do something else. Oftentimes, there's more than three, obviously, options. And they pick from that list, you know, the most effective, rank them. There's different ways that they can approach it, but they are given that set list. In a constructed response, SJT, which is what we're gonna be speaking about today, which are also sometimes called generation SJTs, which is the format of Casper, there's no options on the page. Instead, the candidate has to generate their own written or video response in real time explaining what they do and why they would take that course of action. The response is then scored by trained evaluators against a structured rating guide. It's a fundamentally different task, And it's important to recognize that because a lot of the evidence, and meta analyses that have been done to date have been done on more traditional SJTs, these recognition SJTs or fixed response SJTs. But this really is an opportunity for us to say, we need more data and understanding of how these constructed response SJTs, provide information and do it in a defensible and standardized way. And that's what we'll be talking about a bit today. So the meta analysis that we looked at today covered over 27 studies, over a 100 effect sizes with eight authors across six institutions in three countries, and they looked at over 7,000 records. Why that's really important is that until now, constructive response SJTs have been reliant on single institution studies or institution studies who have been looking at very specific outcomes for them rather than collective or as a whole. And this information and this evidence tells us something a little bit different and helps us understand what the implications are for our program in a slightly different way. So with that, I have the great honor to pass this over to Ivy Erickson. Ivy is one of the coauthors on this study, that really will help us guide an understanding of what is the evidence in this paper and how do we understand it. So, Ivy, over to you. Alright. So starting with the meta analysis, the first question, we wanted to you probably want to know is why does this study exist? And Doctor. Kelly has already alluded a little bit to this, but ultimately, situational judgment tests, specifically Casper, has been used by more than 500 programs internationally with over a million applicants in the last decade. Despite that scale, the evidence was, as doctor Kelly had alluded to, really scattered across individual studies, conference abstracts, internal program reports, what we sometimes term as gray literature. So we wanted to put together, the first systematic review and meta analysis that's ever been conducted specifically on open response SJTs in health professions education. There have been prior meta analyses on fixed response SJTs as doctor Kelly just alluded to. That's where you're picking multiple choice from a list. But as she also said, Casper's different. Applicants write or speak their own responses, and this is the first time that format has been studied at this level. So then the second question that you might have is how was this conducted? So our team, as doctor Kelly alluded to, eight authors, we searched six academic databases plus that gray literature as I just discussed that includes conference presentations, theses, internal program reports. They also directly contacted admission offices for unpublished data, which is how I became involved in the in this meta analysis as well. And before, any of the data was collected, our protocol was preregistered. This is important just from the standpoint it helps lock in the research plan ahead of the results, and that's important from a credibility standpoint. So we have looked at over 7,000 records, 27 studies met the inclusion criteria, yielding a 100 effect sizes from 26,652 participants, and then that spans across sixteen years. The lead author is affiliated with Acuity Insights, but that was disclosed openly throughout the meta analysis and systematic review. And this was also a deliberate partnership with independent researchers from Imperial College in London, University of Illinois Chicago, and other institutes such as UWL where I am from. We also had, external coinvestigators as well that had a built in role to challenge assumptions throughout the process as well. So the results. Across all 27 studies and a 100 outcomes, Casper shows a consistent statistically significant positive relationship with professional outcomes with a correlation of point two two. So if you're coming from clinical or academic background, that point two correlation might sound modest, but you have to remember that Casper is not a academic test. It is in the psychological domain, if you will, where human variability plays a role. So clinical sorry. Situational judgment test that's measuring interpersonal professional attributes, which that's a fundamentally different category than academic behavior. It's a category of human behavior. So when you're looking at applied psychology and behavioral research, correlations in this range are considered meaningful. So a correlation of point two two in a psychological domain, that sits around the sixtieth percentile of all correlations in comparable research. And if it's higher overlapping outcomes, that correlates closer to seventy fifth percentile. So practically from an academic Institute standpoint, what does that mean for you? Practically, using this tool in admissions can translate to a 10 to 15% greater, performing students on a given outcome compared to if you had randomly selected them. Now this is just a illustrative estimate. It's not a guarantee, but ultimately that is something you can anticipate is if you are as opposed to random selection, if you're utilizing this tool, 10 to 15% more students will perform above average on a given outcome. If you have an outcome that aligns higher to that situational judgment test, that strength of the relationship increases twofold compared to when they don't. And I'll talk a little bit more about that later on, as we get going, in this webinar. Super. Thanks for the summary, Ivy. I think I am passing it now to Michelle to talk a little bit more about why these findings matter. Great. Thank you so much, Ivy and Kelly. And so I'm really excited to be here today to talk about why these findings do matter. So these findings reveal that there's a correlation between situational judgment test scores and measures of interpersonal and professional skills. This is significant because it supports mission aligned review of applicants. And that means mission aligned means reviewing a candid or an applicant's academic metrics, their attributes, their experiences, and their competencies. So, again, mission aligned review of applicants, why you would look at SJT, for this is is really to better understand the personal and professional domains outside of the academic metrics. And, again, this is important for admission professionals because we are admitting the future health care workers, who can positively or unfortunately, negatively affect patient outcomes. In addition, there are great pressures in health care in selecting those who are resilient, reliable, ethical, and demonstrate excellent interpersonal skills can only help us to continue to train outstanding clinicians to care for patients in our communities. These findings support the use of a situational judgment test as an element within the admission application review so that we are can effectively evaluate candidates and the potential for the future health a health professional they may be. We can use this information and track students longitudinally from application through to practice based on predefined milestones. We should advance this work. And so with that, we should advance this work, and I call on other individuals within admission offices for health professions education to, think about implementing or adding a situational judgment test as, one piece of information in your application review. So with that being said, now that we've talked a little bit about the findings and why they matter practically, I'm gonna turn it over to my friend and colleague, Brian, for him to, take over. Hey. Thanks, Michelle. So I'm gonna try and not repeat everything we just heard, but I think just pointing out that, I am at the University of Georgia School of Medicine. We're a brand new medical school. And so things like this really help us to figure out what do we wanna do in our process. We do currently not use an SJT in our process. So this is really helpful to kind of evaluate what future requirements we require, what are we looking for out of applicants. And so just know that's where I'm coming from. This is really one of the the first large scale studies, to pull all this information together, really showing different outcomes, curriculum performance, workplace evaluations. And so it really does show us that the test does align with outcomes that are important if we actually care about these outcomes. So it's a huge link on there. So I think the more that your outcomes and your admissions process or your school align with what the SJT is trying to evaluate, the better off your process is gonna be. So this gives us real data to lean on. It's not just instigating how many times have been in admissions committee meeting or at a meeting and just think, oh, I feel like our students would do better if maybe we looked at this or did this. This gives you a real link between those things. And so for us that are leading committees or on committees or making decisions, it really is a shift to what are we evaluating most and how are we actually evaluating that in our process. So we're not asking for people to have a gut feeling of what's the right fit for our program. We're all looking for mission aligned applicants, but this really helps to put a little bit more clarity in some of the gray areas that we might have on our applicants. That was great. Thank you, to all three of you for sort of summarizing this evidence. One of the things I'm hearing sort of as a consistent theme from all of you is, you know, the SJT, and and this is certainly the way it was developed, is not meant to be the only metric or, you know, the primary metric in all of these. It's all meant to be complementary to build an understanding. I use the analogy all the time of adding pixels to the picture to get a clear image of who the applicants are, to understand how they're gonna be successful and successful within our own programs, which are all different, all varied, as you talked about the mission aligned admissions processes. We need to think about these. And so, you know, one of the things that I'm sort of hearing consistently that I really love is, you know, how do we add more evidence informed data points to get that picture? Just like a a clinical picture, we wanna add more information about our patients to get more and more understanding of who they are. In an admissions process? How do we get more information on our applicants to get a better understanding of who they are? And with that, I wanna pass us over to think a little bit more about the implications of this. I think, Michelle, you're gonna join us first to talk about, you know, now that we've dug into the research a little bit and why it's important, what are the implications for our health professions admissions, committees, our peers in the space, and, overall, how we evolve the landscape, you know, your call to action, what else is there, taking us through some of these implications for the real world, and then I'll open it up for our other speakers to join in as well. So, Michelle, let me pass it to you first. Great. Thank you so much, Kelly. And I'm excited to talk about the importance of these inner, personal competencies as it relates to, the admissions field. And so, you know, as admission professionals, we really need to be selecting students who can succeed academically in health profession schools. But we need to recognize the value of emotional intelligence and how assessing personal and professional competencies can enhance this decision making process. I'd like to ask you, how do you accomplish this at your institution if you are, an admissions professional at, on this call today. And so think about that. And if you'd like to put it in the chat, we'd love to hear from you. That would be great. Some other items for you to consider again for those admissions individuals, but, think about this if you're an applicant on on the call and and you are, thinking about taking an SJT. This is what we think about as admission professionals and how we're going to evaluate a candidate. And so some bigger picture items for the higher ed community and admission professionals, if you are gonna implement a situational judgment test, how do you get buy in to implement this? Because it is it is a herculean effort, and you need to be thoughtful and deliberate and really have individuals who will be your champions at the table so that you can, really think about thoughtfully what type of SJT you want to utilize and how that relates to your mission and, more importantly, how you're gonna operationalize that on a day to day basis in the evaluation in in the in a fair evaluation of all candidates. But, again, I'm gonna go back to that mission aligned, thought process, and it really is looking at the gestalt of the application materials. Again, academic metrics, attributes, experiences, and competencies. And so the SJT really does help us to understand emotional intelligence. Right? We want individuals who are smart, so your IQ, your academic metrics. But we want individuals who are emotionally intelligent, who can really, you know, meet the needs of individuals who can speak clearly, effectively, and who are kind, caring, and compassionate. So some ways to assess the professional and interpersonal, competencies are really through this SJT through an SJT, through multiple mini interviews that are mapped to competencies, and then using, which this is new to the game, using artificial intelligence to develop a competency indicator from experiences. These are just a few ways to really better understand personal and professional competencies. I have used all of these tools in my toolbox except artificial intelligence and this competency indicator. Although I would be remiss if I told you I was not investigating and researching, some opportunities to do that. I'd love to partner with anyone on the call who's interested in this research. And for my friends in higher education, ultimately, your mission should drive the review of the review of applicants and your selection process. I hope that you'll consider how to use noncognitive skills in mission aligned review. For this student, the use of a situational judgment test in the admission review process does vary by school. It's important to what I say and what I call it, you need to know the rules of engagement. When you are applying to schools, whichever health profession schools they might be, it's really important for you to understand which situational judgment test they might be using, if any, and how does that relate to who you are as an a candidate? This can help you better prepare and be prepared to enter that application cycle. Thank you so much. Super. Those are great insights, Michelle. Brian, I'm gonna turn it to you to talk about some implications as well. Yeah. So building off of that, I think once again, I'm at a new medical school. A lot of my committee, they're brand new. I think my favorite question to hear and to be asked is why. Why are we doing this? Why do we require this? Why are we looking at this? Why are we asking this question? Whatever it might be, the committee loves to ask why. And so, really, you need to have evidence and, once again, not that instinct of this is why we're doing things. You have to understand what is an exam or a test really trying to assess, and how does that align with your program or what you're trying to look for in your students. So I think that's the most important thing to to answer is why. Why are we doing this? And to tell that to students, this is why we require this. This is why we're doing this. And I think it's easy sometimes to whether it's application numbers or whether it's looking at the the research from this paper to think in just terms of numbers and realize that, you know, you we have to realize that these are people, not just numbers. And so when IU was saying a a 10 to 20% change, if that's 10 to 20 students that are doing better in your program, who wouldn't wanna take a risk on that? And so I think that's really important to remember that all these numbers that we talk about, these are not just numbers. There's real people behind these numbers, and we're always looking how do we get those that, you know, are the above average performers that are gonna do well in our programs and align with the mission that we're trying to accomplish. I know Michelle just briefly touched on this, but AI is really reshaping how we view things in emissions. I think it's safe to say that, it's just you can assume that someone's probably using it to reshape their essays. They might even be using them in their interviews. There's ways that we just have no idea if students might be using AI. So would an SJT maybe be a better response to how AI is being used? Could AI be used in the SJT? Could it help game an SJT? I think those are all the things that we're thinking about right now, but it's important to think about what metrics are we looking at or what are we trying to evaluate. Is an essay going to always be the thing that we're looking for, or could there might be another test or something that might take the place of that because of the the AI and use of admissions right now? And these are real stakes. I mean, taking an exam is expensive, and so I think we have to acknowledge that. It takes time. It takes study depending on what exam they're taking. Health professions, I think all education right now is real expensive. Students are paying for outside test resources. They are, you know, at times struggling making ends meet, and we also just, you know, in most health professions, there's a shortage, but definitely for physicians, we're seeing that in most states. And so there is a real cost and a real stake behind what we're doing, and I think it goes back to the the real people behind these numbers. If you know that you could graduate 10 to 20% better physicians, who wouldn't take a risk on that? Great points. I think I'm hearing too. You know, we really have a responsibility as admissions committees to our programs, to the applicants and our learners, and to the community that we're serving to ensure we're creating, you know, just an evidence based process that we understand and is aligned to our admissions, aligned to our goals. And that that means we need to reevaluate it as things change, as AI changes the world. I mean, I'll tell you, you know, our personal statements back in the day, back back in 2004, you know, the when we started to develop Casper, part of the reason was that we weren't getting the information from our personal statements because at that time, they did what every person should and would do if if well, I was applying to a high stakes program. I showed it to my friends, family members, you know, anybody who was around to say, how should I change? How should I adapt this? What else am I missing? And it really becomes this village statement rather than a personal statement about themselves. And I think we're in that time now too where it's, how do we get authentic information on the applicants that add to our understanding? And I I think it's a great opportunity for research amongst the admissions committee. You know, Michelle, I know you did a call out for other people to collaborate with. I truly believe this is a time where we're really shaping the future of what admissions is gonna look like over the next decade, and it's gonna continue to change. And and we have an incredible community, I believe, to help us evolve that. So now that we have, talked about the numbers, because the numbers do tell a story as Brian said, we it's important to start to understand how those puzzle pieces fit together. Just try and think about, you know, where do we go from here? What are our next steps? And and how do we take this, meta analysis and the evidence and actually apply it practically because there is a big step between, you know, taking a meta analysis and giving it to your admissions committee or doing something and then versus actually thinking about what does your process look like as a whole, how does this fit in, etcetera, and what are what does this all mean? So, Brian, I'm gonna pass it to you to, kick us off here again to talk about, you know, what what do we do for the application of this? Yeah. So, I think for at least in the medical schools, it is a very long admission cycle. And so these are decisions you can't just make right away and just say, hey. We're doing this tomorrow. It takes a lot of time and a lot of planning, and you really have to think about, you know, not just in the mid cycle, but before, like, what are we going to do next year? Especially if you are going to require an exam like a SJT, it's not like a student can rush out and take it the next day. And so you really need to make sure applicants know this is what we're gonna require. This is what we're doing. You really have to be thinking about, you know, why are we doing this and when are we gonna do this? And so I think you have to really think about that pretty early on in the cycle. You can't be doing these decisions right before you open up your application cycle. And so we typically do that, like, over winter break, kinda in the middle of our cycle to talk about, hey. Things for next year. What do we wanna implement? What research is out there? What would it do? I also look at what are schools around us doing. And so if we require this, would this be an extra hurdle for a student to take, or is this maybe an exam or something they've already done before? So I think it also is helpful to do that and maybe even ask some of your current students. Hey. Did you take this exam? Did you do this when you were preparing to apply to to our program? What was helpful? What did you think didn't really show who you were or showcase you were? All those data points matter, especially when you're trying to make decisions to get buy in. And so I think, you know, obviously, there's papers out there, but I think some of that anecdotal data as well is helpful. So talking to your students or talking to other faculty, who's performing well in your courses, That'll really help. The more that you can get that information out to others, that's gonna help because, our programs are there's usually a lot of faculty. Some might know what you're doing in admissions. Some might have zero idea what you're doing. Your administration might have, you know, they might be aware of what you're doing. They might have zero idea what you're doing. And so I I think it's important to keep them informed. So let them know, like, what are the types of SATs that are out there? What are students taking? What are students saying, as far as, like, what's preparing them for a career in your field? So what are pre med students doing? What's out there? What are they doing? I think that really helps them to understand the value of what are students doing, what's out there, what research is out there, depending on what pocket of, you know, pre health or advising that you might be a part of. There's a lot of different resources out there, and you can't assume that what we know, someone else also knows. And so I think I know, at least for me, I'll speak by myself, I get into my own little bubble and think, oh, everyone knows what an MMI is or everyone knows what an SJT is. And a lot of times, they have no idea. And so I think you really have to make sure you're explaining this is what we're doing and what we're doing and why we're doing it. And the same with your committee. You can't just frame it as we're gonna add one more thing to our process, one more data point. In admissions, I think no matter what program you're at, you probably get a lot more information that you need to make decisions. And so what is the value adding of one more thing? So it's really not a, you know, it's not a adding one more thing. It's almost a trade off. Like, hey. Maybe we're not getting what we thought we were getting out of the the personal statement or our short answer responses. Wouldn't SJT be better off suited for answering some of these questions? You really have to frame it that way. It's not a, hey. Let's add one more thing. Let's add one more hurdle for a student. But what are we getting that's gonna really help us to find those mission aligned applicants? And you really kind of back to the research. You have to match the tool to problem. And so if what you are trying to accomplish does not align with whatever exam or test that you are trying to use, it doesn't align. It's it's not gonna help out your process at all. You really are just adding one more thing to think about and consider. And so you really have to make sure that you have all these data points that you are making everyone aware of what is assessing and how to even interpret it. So what does this score look like when you get down to applicant? What does it actually mean? Does it mean they're good? Does it mean they're bad? Or, you know, what was the exam really trying to assess? So you really have to have those data points when you're making these decisions, and really make sure that your committee is comfortable moving forward with whatever you choose to do. So sometimes that means, like so for me, like, we're in a brand new school system. That means just enabling these conversations longer so they can get even more information and have longer discussions and not feel like a rush, like, we're forcing students to do one more thing, or that we have no idea what we're trying to assess and just adding on one more thing. And so have these conversations early and often because it is a long process, and it's tough to add something last minute, especially if a student has to prepare and go out and take it. And so make sure that you're making, you know, making your committee aware of these timelines, your faculty aware of these timelines, administration aware of these timelines, and, really, what is the value add at the end of the day? Brian, that's such a good summary. I wanna add to what you said because I know you're a new school, but you're setting great practices by sort of having this concept of reviewing your process because, I mean, I can speak from experience. When a process has been in place for a while, it is hard to get people even to the table to to discuss changes or evaluation. So this sort of, like, quality assurance review of your admissions process is something we can assurance review of your admissions process is something we can all embrace to a degree, very much like we do across our our training programs. You know, there's an opportunity for us to sort of think about this as a a discussion point that we can have annually to say, are are we getting what we want here? I think that's such a great point. I'm gonna now pass it over to Ivy to join into this conversation to talk about how do we apply some of these findings. Yeah. So I'm just gonna go into a little more detail about some of the results from the study and then break it down into how this can help you from a contextual standpoint and academic standpoint as you are, thinking about how can I use this in my process? So we tested a number of factors to see what changed how predictive CASPER was, the outcome type, the study type, but only one made a difference, and that's what we call construct congruence. So if this is a new term to you, it's a technical term, but the idea is pretty straightforward. Essentially, it's how similar is the outcome you're measuring to what Casper actually assesses. So, example you might be more familiar with is the MCAT and GPA. Those work really well for predicting first year academic performance because we test in the outcome. They're measuring the same kind of things. Casper assesses interpersonal and professional skills, communication, empathy, ethical reasoning, professionalism. So if your outcome measure is also focused on those, you have high congruence. If your outcome is a broad overall clerkship, composite score that mixes academic knowledge with technical skills, with communication. You will have a lower congruence, but it is still statistically significant. So what you can see here is the data showing has higher congruent outcomes, AKA more alignment with what your situational judgment test is, looking at. Your con sorry. Your correlation score is a point three two. So that is at the seventy fifth percentile for applied psychology correlations. If, they are less aligned or lower congruence outcomes, the correlation's point one seven. This is still statistically significant, however, but that's closer in the fiftieth percentile for applied psychology. Ultimately, higher congruence, lower congruence, both are positive. Both show Casper predicts, but the strength nearly doubles when that alignment is strong. So examples of higher congruent alignments could be MMI scores, interpersonal skill ratings in admissions, field work evaluations that focus on communication. Those would be examples of higher congruence outcomes. So in essence, the important thing is making sure you are evaluating what outcome you are trying to align the Casper or situational judgment test to. Perfect. Thanks, Ivy. Michelle, I'm gonna pass it to you for your thoughts on applying this information. Great. I I I love the theory behind it, but I also love the practical application because that means what are we doing boots on the ground and how we're working with our students. So what we've learned is that using a situational judgment test as part of our mission line review can really help us achieve our goals of recruiting, admitting, retaining, and graduating health care professionals who can excel academically at our school while being outstanding patient centered clinicians. That's our goal, and I believe we are accomplishing that. It's also really important to provide your stakeholders with information that supports your decision for using a situational judgment test. We provide our admission committee members with biweekly reports and host an annual retreat where we conduct a deep dive into our data and present research that informs future decision making, all in an effort to support continuous quality improvement and to ensure that we are subscribing to mission aligned review, but that the students we are ultimately selecting and who come to our institution really can do the work, but also that, these professional and interpersonal competencies, they're embedded within the curriculum for our programs. And so it really is synergistic at the end of the day, with our decision of that to use a situational judgment test at the beginning, right, at the admission review stage. So, you know, my recommendation is institutions evolving their admission practices for the programs they offer offer. And it could vary based on, the programs. Right? Something that might be, more relevant for medicine, being a doctor, might be very, very important for another health care discipline and it might be, very important, but it might not be as important for a physician. So really understanding the programs and their accreditation standards really can also help in that process. There are so many opportunities for us as leaders in health professions admissions when we really think about how we can utilize situational judgment test tests. We can help shape how health care is delivered by selecting applicants who will truly be person centered clinicians. We can build workforce pipelines so that communities we serve have excellent health care providers. That's truly remarkable and an honor. I'd be remiss if I did not mention that there are challenges in health professions admissions, such as rising cost and access issues. By hosting this webinar, we're addressing some of those, and I look forward to continuing our conversation together. Thank you so much, Michelle. I'm gonna try and bring us together, as a bit of a summary before we jump into some of the questions, and thanks for everybody who's already submitting questions in the Q and A. That's great to see. You know, we hear a lot about predicting better outcomes, helping us understand how the relationship between an admissions process and somebody's future performance in training can be predicted and can be understood. And Ivy spoke a little bit more about this construct congruence that we're talking about. And to me, that's really interesting because it aligns very similarly with what the WMC is doing when they think about their constructs that they're trying to measure, their competencies that they wanna see in graduating and in applying, medical students. It allows us to think about something that I frame as, like, competency aligned admissions. How do we take the competencies that we wanna see in our trainees, and think about them in our admissions process? And it only makes sense that as we align those competencies, the competencies that we see in admissions should predict those same competencies out into training, and that's the pattern that we see. And that's really, really important because it actually allows us to map or blueprint our admissions processes a little bit to sort of say, how do we make sure that we're covering enough? Even though our processes can all be different, we can actually be very intentional when we discuss the data with our committees and when we go back to do the the CQI on things to sort of say, are we actually covering these competencies that we think we want to cover? Is there something we're missing or something we should be putting more weight onto based on what's important for our program? And we talked a lot about needing that to be evidence driven, data driven so that committees are understanding the why behind things. They understand. And if we are asking applicants to do another step or to do another piece before they apply to our programs, that we believe in it. We believe the data is going to help inform us about how they're gonna do and how they're gonna perform in our programs. So, I'm really excited to jump into the q and a now to think a little bit more about what this means and how programs are thinking about it in a little bit more detail. So I'll invite all of our panelists to join me, in this q and a, as we jump in. So we've got about just shy of fifteen minutes to jump in to talk about some of these questions. So I'm gonna go through some of them in the chat that I think are are really interesting. There's some big questions about AI, in there and some big smaller questions about, practical questions, which I think is really good. One that came in that I think is really interesting is how are other stakeholders being represented or incorporated into the admissions process? So do we have representation through alumni or external partners on our admissions committees, and and how do admissions committees think about that? So I'll maybe turn it to Brian and Michelle to reflect on that question a little bit. Yeah. I can kick us off, and I'll send it over to Michelle. So I will say on our admissions committee, it is all faculty based on our admissions committee, how it's written in our bylaws. So every school is gonna be a little bit different on who can be on the committee. Some schools will have committee members on their committee, and so they really look to have that piece on there. But workforce needs is something that we're always talking about in our committee. And so even though our faculty is all either a physician or basic science, faculty members that are on our committee, I'm looking at their background. So what would you bring to the committee? Where are you from? What do you know about what issues are going on in our state, what we're trying to address. Mhmm. And then our interviewers our interviewers, we bring in a lot of those other people. So we have people from the technical colleges that help us interview. We have community members help us interview. We have people from our larger university help us interview. That bringing all they bring in all these various viewpoints because I think these are all critical things we have to talk about is how do we get this representation and these voices into our committee and also even things like current medical students helping you out in your process. So, we just started our first class, so we'll have two medical students on our committee, and they're usually very aligned with what's going on and from the student perspective as far as what's offered to undergrad students, what are they taking advantage of. But I think workforce needs, it needs to be front and center of what your committee work is doing, of what issues are facing your state or your area. What does your mission say? So I know Michelle talked a lot about mission aligned, but what does your mission actually say, and then how are you trying to accomplish that mission? And how does that align with the workforce needs? And so I think the more voices you can have, the different background you can have speaking to these decisions, the better off it's gonna be. Michelle, I'll send her to you. Great. Thanks so much, Brian. I would echo what Brian said on the med medical school front. So maybe I could share a little bit about my background. I oversee admission, financial aid, and registrar functions for all students applying to Geisinger College of Health Sciences. So our nursing students, that's an undergraduate major. Our. students in our graduate school, we have a genetic counseling program, a master of biomedical sciences program. We're launching additional undergraduate programs in the future. And then, of course, we do have our school of medicine. And so each school does, there are definitely, similarities in terms of the admission process throughout. But committee, membership for, you know, undergraduate, graduate, and medical school is a little bit different. Our model is definitely for the medical students, like what what Brian said, On the undergraduate, level, that is different, and it's structured differently for our graduate programs as well. And that is traditional in higher education on how that, is structured in a larger school that's more than just a medical school. With that being said, our students are part and parcel. I'll talk a little bit more about our medical students because they have an active voice, and they are interviewers for us. And they represent probably about 35 to 40 of, our interviewing pool. So our students have active an active voice. Right? They help us to assess students, by utilizing a multiple mini interview that are pegged to competencies and help the admission committee members really better understand how those students, are prepared in terms of the competencies we want to see them have as entering medical students. We also look at, various competencies for our undergraduate as well as our graduate programs because, again, our programs are health science based. And so, really, these, professional and personal competencies lend themselves lend themselves to our mission aligned review for all of our programs. Super. Thank you to to both of you. I think it's so important to figure out how those those perspectives are brought in. And and I really like too that you're thinking about not just the admissions process. Obviously, all through the, curriculum, you're thinking about how we think about workforce, how we're involving, other stakeholders, and it's just nice to make sure that that admission side is connected. Sometimes it feels a little bit disconnected, from the process of the rest of the curriculum. So I think that's really good. I am going to do a question that has some votes to it. So, again, audience members, if you wanna vote on a question, it'll it'll push it up in the the queue for me. So there was a question about how programs using Casper typically factored into their admissions process. Is it applied as a CAT score as part of an overall scorecard? I will take purview to answer this, but welcome anybody to jump in here as well. So it really does vary by program. Programs need to use the data points that they have in a way that makes sense. We know that people use MCAT in different ways. They might use GPA in different ways. They might include science GPA or other factors, and and that makes sense based on what their goals are of their programs. And very similarly, Casper is used in in the same way. So we do have some programs that use it as a cut score. They often do that, especially early days of using Casper. But we do try and encourage programs to use it in a formula or a rubric, and let me tell you why. Casper is a really interesting data point because it is open ended. When it's open ended, what that means is there's not really a ceiling effect like you might get with a multiple choice question or something where there's sort of a fixed number of ideal responses, you can get a wider range. And so when you're using something as a cut score, you are using the Casper information to identify those who are at the lower level of performance relative to their peers, so those who are not as strong. But what you're missing out on is identifying those who are exceptional in those areas. Why I love the open ended is that all of us on the webinar could watch a Casper scenario. We could come up with responses and options and sort of come up with that, but we are never going to come up with all of the responses that an applicant could. They are, resourceful and thoughtful and creative. And I think sometimes by having that that fix, we we don't get to see, you know, those who are really exceptional at at these types of of competencies. And so the reason why using it in a in a waiting or a rubric can be beneficial is that we do get to identify those who are in those top performers, those exceptional performers, and and really give credit to those skills. You know, professional skills, are sometimes seen as as a negative thing. Like, we're just trying to find those who are struggling, But really, we also want to identify those who are exceptional. So that's sort of my broad response that there is a lot of variability by program that's maybe not unexpected, but there is an opportunity to sort of think about what makes sense for your program. I'm gonna now turn it over back to our panelists to talk a little bit about, you know, looking ahead, what do you think the next major opportunity or challenge is for health professions admissions, and how can institutions prepare? I think that, you know, very simple question, not much to it. You know, so maybe I'll I'll open it up to you to think about what what else is coming. Maybe I'll start that off, and then I'll invite Brian and Ivy to come. I I mean, AI is here. Let's and that students are but we should also acknowledge that admissions offices are using artificial intelligence to look at candidates. And I I think that this is important. Right? You can AI can be used to, help distill information in an application. It could be used to find, AI written generation generated personal statements, narratives. Something that is is really, prevalent is or that the conversation is prevalent right now is to if you're still, hosting virtual interviews, how can we use AI to detect if a candidate during those interview sessions, if they're on their phone using AI or if they're using it on their computer. So what I would say about this is that, I think there's good and bad of AI. It's how do you use it ethically and responsibility, and how do you, let people know that you're doing that whether you're a candidate or whether you're using it in the process. And so I think that that's part of the ethical piece of AI and admissions on both sides. It's here. And so how how can we all work together to better understand what's happening so that, again, we're preserving the mission aligned review process in the most ethical, and mission aligned way. Great. Brian or Ivy? Yeah. Brynn, I can go ahead next. And I'm I'm admissions director two at the UWL PA program. So all this is very, near and dear to me as well. But, what I've seen a trend more towards in the last few years is more of a holistic approach. So for instance, for us, we removed another quantitative factors such as the GRE from our admissions processes, and we substituted it with Casper for a more holistic approach. And another thing too is accreditation is being a little more rigorous with regards to having your domains aligned. So having something like a situational based judgment test, being able to map that to assessment for interpersonal morality that is able, to better define your competencies. So that's something that I've been seeing, just from the standpoint, in the last few years is having a more holistic approach for viewing applicants and having an SGT can help navigate that. Brian, anything? Yeah. I definitely agree with both of them. I think AI, is definitely big thing. I I truly believe in transparency for applicants. And so I think if you're using AI, you should say that. Or if you expect an applicant not to use AI, you should say that and make sure you, you know, you'll hold them to that if you if you can. The other thing I think, like, what Ivy was just saying, I think the the issue that we're facing right now is just workforce issues. And I think when someone has a question about your students, the first thing they don't usually ask is how are they doing professionally. Or the ones that you accept, they ask, what were their metrics? What were their academics? That we want the best of the best academically, but we also need to have that transparency of that's not everything that we're looking for. So like Aida was just saying, making sure that you have a process that is aligned to your mission, the needs of your school, and being able to clearly articulate these are why the things that we're assessing are valuable to our program. Because I think that's one of the biggest challenges right now is, one, how are we getting students, from places that maybe don't have as much access to resources, to get them into our programs and then hopefully send them back to communities that need these health professionals? And it's just a tough it's a tough thing to deal with right now because I think there's a hyperfocus on the academics right now, which are very important, but then how do you also show the importance of all the other factors that we're looking at in our admissions process? Yeah. Could I just add one additional thing on there? So, I I teach in, most of our programs at the school. I shouldn't say most. Many of them. And I'm in I help to develop our professionalism thread. And so, I'm gonna come at this from a faculty member standpoint who actually, looks at the, the the the non, clinical piece, the non I'm looking at the noncognitive. Right? And so we assess within our curriculum their professionalism points that we look at. We talk about how do you recover from an error. Right? That's called ethics. And we also look at clinical skills. Right? We're evaluating your communication skills. So programs are evaluating these noncognitive attributes within their curriculum. So by having a as an SJT at the beginning of the process, you can really identify those individuals who who are presenting with those competencies at the time of application. And, again, yes, they're going to have to take tests, but there are, and they will have to take exams, and they will have to be credentialed. But there are these other pieces throughout the curriculum where you're assessed, where the SJT can really, help individuals along that continuum. And so for us, it was a I I don't wanna say it was an easy process to have buy in, but knowing what is contained within the curriculum and how, SJTs can assess those competencies definitely helped. Yeah. And I think you're you're all raising really important points too that I'll just add to briefly to say, I think, you know, we do wanna assess these things in program. I'm also such a big believer that admissions data should be the first formative information we have on our students. So how do we use their admissions information to help them succeed? Where do they have strengths? Where do they have areas of growth? Where might they struggle? And that is the one big opportunity too with AI as well is we can start to put that data in to understand where people will struggle in our programs. As a wrap, I'm going to ask our panelists to very briefly give me, you know, their single most important takeaway from the meta analysis that, we talked about today. So, Brian, maybe I'll start with you, and then we can go Ivy and Michelle. Yeah. So I keep going back to, think of them as people, not numbers. And so I think, you know, if you can you know, if you have a student that doesn't do well in your program, you're taking away a health professional, from the future work workforce. And so for a medical school, that's a future physician that might not be in their workforce. And so I think if you can align with 10 to 20% better of your students as long as that is a factor that is aligned with your process, that's the thing I keep coming back to. Super. Ivy? Yeah. I would echo what Brian says as well. Utilizing a tool like this to align with your outcome, all of us, you know, want to have a a future provider who is morally sound, ethical, socially socially inept, great social intelligence. And so just remembering that this is one tool that can help in with regards to high congruence outcome does show that. Perfect. And Michelle. And so I'm just gonna echo what Brian and Ivy said, and I'll stop there because it would just be redundant. Absolutely. Thank you all so much. I wanna thank everyone who attended today's webinar, and to those who are watching the recording. Thank you again to our panelists for sharing not just your insights, but also how to take action on this and the practical application of data, because I think that's always just so important. On a note, again, you're going to be receiving a copy of this recording, and the slides in your inbox soon. So feel free to email us if you have any other questions. And, again, thank you all so much for attending.