Name: Lessons Learned While Implementing Competency-Based Education
Uploaded: 2026-05-30T17:16:13.826Z
Duration: 58 min 54 s
Description: Lessons Learned While Implementing Competency-Based Education

Transcript for "Lessons Learned While Implementing Competency-Based Education": Welcome, everyone. Thank you so much for attending today's webinar, lessons learned while, implementing competency based education. My name is Sylvia, and I'm joining today from Acuity Insights. Just a note before we start, our panelists will be taking questions from the audience at the end of the webinar. You're welcome to enter your questions in the q and a box, throughout the webinar, and we'll get to them after the panelists have presented. And we're also going to share some resources for this webinar as well, such as our competency based education best practices guide and the course to go along with it if you're interested in exploring this topic further. So without further ado, let's get started. So competency based education is something that we hear about a lot and, it is shifting the focus from time and grades to whether learners can demonstrate readiness for the real world. It defines clear outcomes, tracks development over time, and uses multiple data points to support a fair and defensible, decisions about profession. In practice, the implementation introduces real tensions, balancing ideal frameworks with existing systems, developmental intent with high stakes decisions, and clarity with feasibility. And so day today, we'll be exploring, everything that CV promises and what it actually takes to make that work. And we're thrilled our guests could join us today. You'll hear from two programs approaching competency based education from two very different starting points. One is an established medical school expanding an existing competency based framework with entrustable professional activities, and the other is a new osteopathic school just about to open its doors, building its preclinical competencies from scratch. Different starting points, different constraints, but the same question is how do you make competency based education actually work in practice? I'd love to introduce you to our guests. We have with us today doctor Holly Coretta Wire, associate dean for admissions and assessment at Stanford University School of Medicine. We have Kieran Brear, director of evaluation and assessment at Stanford. And then we also have doctor Montha Kindy, who's the senior associate dean of preclinical affairs at the Illinois College of Osteopathic Medicine at the Chicago School. So I'm gonna pass things off to our colleagues at Stanford to tell us about CVME at their institution. Sorry, Holly. I think we can't hear you. Let me double check what's happening with audio. Can you hear me now? Yes. There we go. Excellent. Weird. Okay. So I'll start again. Thank you again for having us today. I'm gonna start kind of with a little bit of the background for where this came from for us. And so we previously had a criterion reference assessment system at Stanford many years ago. And like many schools during the pandemic, we went to a pass field grading system. And over the last couple of years, we have realized that, we we needed to bring back some form of, assessment and to allow our students to really stand out for residency selection. But that brings with it a whole host of tensions around norm referencing, equity and fairness. We didn't wanna create a competitive environment for our students. And so, you know, I have a strong background in competency based education from the GME side. And so we decided really to kind of go back to this idea of can we make criterion referenced assessment and competency based assessment coexist with, some element of distinction, not necessarily grades per se, but allowing our students to stand out in some way, shape, or form, particularly around, the clinical domains. And so, fortuitously, the Macy Foundation came out with recommendations in 2023 that basically said, competency based assessment is primarily the focus for equitable assessment. And using program programmatic approach with multiple data points and multiple contexts over a prolonged period of time would allow you, if you use a criterion based system, really to focus on, optimizing equity and fairness while also, you know, allowing students to potentially stand out, for, you know, residency selection, which isn't going away. And so we took a big step back and said, alright. If we're gonna do competency based assessment and we're going to use a programmatic approach, we need to centralize, a couple of things. And it is that opportunity for this for residency selection, for standing out and also equity and fairness, but also really optimizing preparedness for day one of residency. Because I think a lot of the time, you know, what we've assessed in the past is what's easy to assess, not necessarily what actually prepares our students for what comes next. And so being able to, take that big step back and develop this framework and optimize, our to set criterion for, you know, confidence that ensure our res our students are gonna be ready for day one of residency, but also being able to stand out and optimizing equity within our system was really kind of how we thought about, embarking on this process. Alright. Thank you so much for that introduction, Holly. I'm gonna pass this over to doctor Kinney. I'm sorry, Monica. Nice to see you. Hi. Thank you. Thank you. It it's wonderful to be in this discussion, with Holly and Karen, because I do believe that our journeys, represent, very, two very distinct pathways into CVE at the Illinois College of Osteopathic Medicine at the Chicago School. As was mentioned, we are new osteopathic program and we, in fact, matriculate our first class this July. And what makes our this position uncommon is exactly that. So being built from the ground up, we're not trying to retrofit CBME into an existing structure. And instead, our curriculum design process came about right as the foundational competencies for you and me from double AMC, double a common, ACMG, a ACGME were released. So the the the concept of competency base was part part of our blueprint from day one. And while that represents an opportunity, building from inception doesn't necessarily mean building without constraints. It means that, every, structural choice that we've made thus far has been primarily conceptual. And so while we're designing systems and identifying frameworks and making philosophical commitments, we're doing all of that before we have meaningful student interaction from which we as a program can learn. And so the real pressure test actually occurs as we begin to operationalize this summer. And so we're we're kind of pressure testing, in real time as we're operational live live in this theoretical approach. And I'm I'm so glad, that my colleagues previously mentioned, equity and fair fairness because that was a critical design requirement for our program at the Chicago Chicago School. The majority of our students come from historically marginalized population. So student success and equity centeredness was a lens from which we designed our program, and it really shaped how we considered assessment and feedback and what it means for our students to demonstrate readiness. And and when you look at our mission and when you see our four value pillars, those aren't just marketing, schemes to us. Those are the lenses through which we made our choices related to competency based education. Thank you for the introduction, Monica. And thank you, Kieran and Holly as well. It's really exciting to see where you're both, approaching things from and and the decisions that you made along the way. And I love the focus that you both made on, speaking to equity and fairness as part of, one of the pillars of competency based education and how that might, help your programs and your learners going forward. Let's chat a little bit about, where the rubber meets the road, so to speak. When we think about, you know, these high level competency in the frames that you've all mentioned, there's a lot of tension that comes in here. And this is really where the theory starts to meet reality is choosing, and interpreting these these national or crediting frameworks, translating your competencies and sub competencies into something or actually their competencies and sub competencies and something that's meaningful at the local level. And I believe this is all of the things that Monica you actually alluded to well, explicitly said, which is all the things that you before you get started, all the conceptual and structural issues are are really fascinating. So I'd love to hear more about what you all think about that. So I'm going to pass that off to the Stanford team. Alright. So I'm gonna take a little bit of a whirlwind tour through kind of how we how we set this up. And so we took a big step back and said, what are our core principles and then designed from there. And so, we started with kind of these aims of saying, okay. It needs to be equitable because our prior system had not been it needs to be completely relevant to patient care. So it needs to be responsive to what it is that our learners need to be able to do to take great care of patients, particularly, again, on day one of residency because that's what's next. It needs to be things we can actually see them do, and it needs to be meaningful for the students and their learning because they need to see that it's relevant to them. Something that has not necessarily happened for our students is them getting regular feedback. It's often been they may get feedback in the moment, but their written feedback is delivered, you know, weeks later and it doesn't really reflect what they heard. We wanted to facilitate regular feedback. We wanted to incentivize growth over time, meaning if you, you know, start off in one place, you get feedback, you take that feedback, incorporate it, you get better, you know, continue to grow and develop. That's actually rewarded because I think a lot of assessment systems don't necessarily reward that. They do try to facilitate that, but we're trying to actually reward that as well. And it needed to be completely transparent to, our students. So in the development process, we said these are our core principles. This is where we're starting from. How do we then leverage what's been done before but make it unique to us in order to create something that will ensure that our students are prepared for that transition to residency, and for day one of what's expected of them. So we convene a huge working group. In the end, it was probably close to 30 individuals representing GME. We had a number of directors, our UME colleagues, clerkship directors, you know, Dean's office leadership staff. We had a number of students involved, and we took these core principles and said, alright. Stakeholders in the room, how do we do this? How do we ensure that we're doing all the things that we hope that we're doing, and how do we get to where we need to go? And so we leverage prior frameworks, including the core EPAs, which is the pilot I worked on with the AMC. We looked at specialty level EPAs. We looked at the Vanderbilt milestones, for you and me, and said, alright. How do we take these things and design our process? And then how do we incorporate the stakeholder perspectives in the room? And, really, in thinking about, you know, there's core APAs, there's core things that every student needs to be able to do to be prepared for day one of residency. That should inform our pass fail decisions. And then how do we, again, allow them to stand out for residency selection? And that's how we came up with these different domains of distinction, each of which has a number of EPAs mapped to them. So clinical reasoning, interprofessional teaming, systems based advocacy, which is really kind of leveraging, you know, different resources for patients, thinking about social determinants of health, technical skills, which actually came from the students, wanting to be able to show off their technical skills, particularly those who are, more surgically inclined, adaptive expertise, which so many program directors came back to. You know, we really want students who are gonna be able to take feedback and incorporate it. We want students who are gonna be able to facilitate their own learning because that becomes more of an independent process in residency. And then physician ship, which is our rebrand of, professionalism really because professionalism has kind of been this fraught construct where a lot of students who are underrepresented in medicine, feel like they are not reflected in that construct. And so we really boil this down to what the key points are or, which is really how do you seek help, when you need it, how do you follow through on the things that you're supposed to do, really the core tenants of entrustment. And we want to, again, reward those things. And then we've said, how are we gonna do this? We're gonna have programmatic assessment. So EPAs that really boil up to, again, those core things that we require, but and each of these domains has two to four different EPAs underneath it that students are going to get assessed on, iteratively over time. And we've also learned from, you know, a couple of our colleagues. Again, I have talked about the Macy recommendations, but also our colleagues at Washington University, in Saint Louis who've been doing something similar to this, perhaps on kind of a more broad domain scale, and the lessons that they've learned. We've taken a lot of what they've done and said, how do we incorporate the lessons they've learned and try to, make this unique and worked within our context in order to, again, be able to have this programmatic assessment and optimize the equity and stay true to our principles while also ensuring that we're able to move forward these different domains that are so imperative to, both our students and meeting the needs of patients, but also in being prepared for residency. Holly. And I I really appreciate I think one thing I'll call out and why we have you all here is really building on the work that's come before you and learning from everybody else's lessons. So I think this, this webinar is really paying that forward for our, for all of our attendees right now, thinking about how they can then learn from you while they're moving forward as well. So I really appreciate that connection here. I'm gonna pass this over to, Monica now. Yeah. And, just kind of navigating the framework of operationalizing this concept in real time. We've definitely faced some tensions, while centering a competency based education in our program frame framework even from inception. Now some of these tensions are ubiquitous to all program types, and some are more unique to our particular program type. I would say as a new program, especially during the pre accreditation process, you're always accountable to mission alignment, but evaluators, stakeholders, and other outside voices, want to ensure, space for other types of distinction, particularly osteopathic dis distinction. And then you also wanna create some programmatic and mission distinction as well. And and I can assure you that, all of those elements don't always point towards the same outcome. Bandwidth is another, tension that we faced. And I think that's definitely a ubiquitous, tension, that I I think is is is very acutely felt by many osteopathic pro programs. In general, our faculties tend to be much smaller, and those previously mentioned mixed interests, manifest, as previously mentioned by my colleague in different frameworks of use, in order to satisfy a lot of those competing interests and needs. And so, when you're looking at it, comprehensively, this is no longer, a simple project. It it is a sustained, interdisciplinary, multi person effort, and and translating competency language into those behavioral markers, those actual observable behaviors that an assessor can, recognize in a student requires, people, champions who understand both the frameworks, the philosophy, and the clinical reality. And and all of that, it it at scale isn't something that you can easily do. It it it it must be incremental and kind of built piece by piece. But but in order to do any of this work successfully, you you you must create buy in. And I would say, from our perspective, that's what's really worked for us is identifying champions. And we have the benefit of recruiting our staff and faculty from the founding dean. And so even from the recruitment phase, we've identified early adopters amongst our faculty and staff. So people who understood, the intent, who had real clinical and educational experience, and who were willing to do the sustained translational work of helping us define what these competencies actually resemble, what they actually look like for a student at different developmental stages and finding those champions, who who right now are are leading up our own task force, is was 100% the most critical piece for us that have that have moved this sustained effort. And on a system design side, we've made deliberate investments in resources and tools that were tailorable, easy to use, and made visual use of student outcome data. Because we knew that if the tool was burdensome, if the process was cumbersome, then it wouldn't get get used. And while these tools are important operationally, what can really affect the success of a feedback rich competency based program is culture. And culture has to be actively built. And and what I mean when I say culture is is building a program, a system, in which the expectation to both give and receive feedback is baked into the fabric of the program and not treated as a byproduct of assessment. And so without the psychological safety, feedback from a student perspective is always gonna feel icky. And so we've made deliberate effort to carve out time, dedicated time to specifically teach and train our students, staff, and faculty of how to do feedback well. And we're taking the time to generate a shared understanding about CVE, about feedback culture, and related assessments so that as our students matriculate, they will learn how to analyze their developmental data data and translate that with their faculty coaches into an action plan. We'd love to create a supportive learning environment where students value asking for feedback rather than seeing assessments as discrete judgments. And staff and faculty, in turn, are learning how to facilitate feedback discussions rooted in a growth centered coaching model. So that we're taking the time and and making the effort to move feedback from this four letter word into a necessary piece of learner and and program development, that has been, exceptionally useful for us. Hey, Monica. I was taking frantic notes during both of these, sessions because I I love what you're all speaking to. And we'll come back to, I think, the idea of, creating a pro creating a feedback culture in in your program. So I think we'll come back to that. Hopefully, at the end, I'm sure it will come from the questions, but I definitely have, lots of questions there. But let's, move on here. But operationalizing CPE in real programs, this is where, reality becomes real. Right? So designing, implementing, assessments and data practices that are actually usable is something that we you're all thinking about. You've touched upon faculty bandwidth culture and buy in, and so those are definitely things that we have to consider in terms of operationalizing things like your organizational and timeline constraints. Monica, you mentioned, the team effort involved in doing all this work. And then while you're not there yet in terms of actually collecting the data and and doing CQI and and thinking about where you might have to make adjustments, I'd love to, we'll surely hear from you all in terms of what happens when you're starting to actually do the thing. And so I'm gonna hand this off to, I believe, Kieran, next up. Thank you, Sylvia. So, yeah, at Stanford, we recognize that this was gonna be, a huge culture change for everyone involved, especially around the cadence and process of giving and receiving feedback as doctor Kindale also noted. Students are worried about asking for feedback more often. Educators are worried about the student reception to the documentation of this ongoing feedback. And so ahead of the rollout and continuing throughout, we've been meeting with our clerkship teams both in groups and individually to discuss the details of rolling this out in the quotation and just really focusing on how can we help this be successful around the nuances or of their their specific, clinical rotation. So there's been a huge investment in educator development beginning at the top by meeting with the vice chairs to get their support in rolling this out within their departments. We've shared all this information around the updates to the assessment process at different department meetings, ground rounds, town halls. We have dedicated webinars for our community preceptors and residents. And in in parallel, we're sharing all of the same information at student focused town halls and web webinars. And then our transition to course, covers the process and details to really prepare the students as they enter the clinical years. And the other thing that we've been able to do is work really closely with our education technology team to develop some asynchronous materials like short videos and writing good narratives, that are all available to all educators and students at any time. And as part of our goal of transparency around all of this, everything that we're giving to the educators, educators as well. So that way, they are receiving, the same information across the board. Thank you for that, Karen. I I do appreciate the transparency aspect of what you're doing as well. I think having the students know what is expected and what's being shared out really helps create that, that, trust in the program as well. So I'm sure we'll come back to that. I'm gonna pass this off to, I think, Wanda again. Yeah. And what what you're seeing here on this slide kind of represents one of our answers to many of the elements we've discussed today. So we've designed an assessment tool called STLR, and it's it's how we've we've translated these design principles, in the context of the system constraints into something our students and faculty will will use. And the the the matrix that you're seeing on on the right, each of the rows is one of our nine competency domains, and that's mapped across different encounter types in the columns. And so depending on the encounter type, we can activate subcompetentencies, within a given domain for each assessment or encounter type. So very deliberately, not every encounter assesses every domain and that and that's again intentional. Furthermore, encounter types early in the pro pro program, let's say a a standardized pay patient might, assess and and and garner feedback on very different, competency domains compared to a a later program as SP. And one of the foundational insights that we built around this is that, authentic assessment doesn't require evaluating everything in every context. And what you assess is is actually, observable and relevant to that particular encounter. And that's what I mean when I talk about tailorable domain. That it keeps that feedback burden on the faculty and student manageable, which is critical when you're asking faculty who are already kind kind of at a maximum bandwidth, to engage with something new consistently. Another important piece for us was the three sixty degree element. We wanted to be able to gather feedback from faculty, peers, and staff, and self because development doesn't show up in the same way in every context. A student might demonstrate very strong, professionalism based competencies in a faculty facilitated case based, and and struggle to develop, to demonstrate professionalism in a different context. And so you need multiple different vantage points, with multiple different sources of feedback to build that comprehensive development picture. And so using a particular tool, you can you can pull out feedback from the SP. You can see feedback from peers, and you can see feedback from faculty and how those the those competencies, develop across the different context and across the different encounters. So that ultimately, this longitudinal dashboard that we're building kind of ties this all together across all four years where the goal is a developmental record that a student and their faculty coach can can review together and say, you know, here here's where I'm at now. Here's how I've grown. Here's where my opportunities of growth lie, and and here's my plan that I've developed in concert with my coach, to get to that next steps. And and I think the other important piece is is that we're we're training our students how to use their data in this platform before they've ever generated any data on their own. So that the the feedback can can drive growth, when someone knows how to translate it into action. But if you don't build that skill, that that data analysis skill and the expectation to use that date data from orientation, then, unfortunately, the the data will just become noise. Thank you, Monica. Those were fantastic, points that you made. Actually, thinking about training the students how to use the data in the platform, I think, is is obviously one of those key lessons that I think programs might miss. Right? We wanna make sure that learners know what to do with the data and how to drive growth. And then thinking back on, you know, how do we make sure faculty are not overburdened and and how they can actually provide the information for that student growth as well. I think those are such big, aspects of the implementation. So thank you for touching on those. But that was it for our presenters' presentations. I think where things are going to get really exciting is in our q and a. I'm so excited to chat with you all. I know there's some questions coming in the chat. So if you haven't had a chance to chime in yet, please feel free to do so. But I do have some questions, I think, that kind of thinking about things from our perspective. Do you want to come back on camera? Please join me. We can have a little bit of a a fireside chat almost. I'm curious though, and thinking I know you haven't fully launched in terms of your students actually, being assessed on this, but is there an early decision that you made that, really had a bigger impact than you thought it might? Is there something that kind of either tripped you up or actually made things work better from your early days? I'll let you all think for a second. I'm just hoping you jump in. I I think from our perspective, it it was it was designing a set of competencies or or identifying a set of competencies that would adequately reflect our program's mission and vision while meeting the educational standards, the accreditation standards, and and also kind of meeting those mixed needs. Just really kind of almost piecing together, the observable behaviors that would reflect our ideal graduate, and then, you know, kind of backwards designing the competency framework from from that. Right. I love the backwards design idea of working because these these frameworks are so large. Right? And so how do you really adapt that? It was actually one of my questions, so thank you for already reading my mind. Holly and Karen, do you have anything that you think early on had a big impact on you all? Yeah. I think, one of the probably most important parts in this was to have a co friction model that included you and me stakeholders, GME stakeholders, students, to be just very broadly inclusive, very big tent, and to have all of that input in the design process because then everyone's voice was heard. You know? Again, starting with those core principles, they're saying, like, this is what we're after. How do we design a competency framework, outcomes, domains of distinction, a process that is responsive to all of these things, and how do we balance all of these needs, particularly given, you know, that there is a tension between, you know, the growth and development piece and the distinction piece a lot of the time because, you know, there's the I wanna stand out for residency, but, also, I need to be able to be, you know, assessed and to grow and develop. And so I think that for us was a big win. I will say one of the challenges we encountered is that very early on, we started socializing this before it was completely settled. And I think that the student anxiety level was probably higher than it needed to be, because we wanted to be, again, very transparent, and to let them know that this was coming. But I think that they wanted all of the details all upfront. And so being very kind of careful about being transparent while also balancing knowing that your students are going to be potentially very anxious because this is a new process for them, and then how to assuage those fears. I think not just having faculty development, but we've been very, very deliberate about student development as part of this, because I think a lot of times when things are happening to them and the locus of control is very external. We need to try and be very deliberate about bringing that to be more of an internal process for them and having control over the processes they can. That is excellent. So that coproduction model hopefully helps them feel a little bit more in control of of what was going to happen and and give them the ability to, sort of in the driver's seat as you're building the car, not just after afterwards. I was thinking of this during your earlier, presentations as well because I was thinking about this in terms of buy in, in general. So, Holly, you mentioned your students kind of struggled because there was idea that this was new, this was different, it was gonna be harder, or it just even just different. How do you get buy in from your students and your faculty when you started using the word competency based education? I know that can feel very big and overwhelming to a lot of people. How have you, how have you gotten those champions on board? How have you, kind of, associate those fears about what competency based education will look like and the burden of it potentially or just the complexity? I'm not sure if, Monica, you have an answer to that one, but I'm kinda curious, how you got those champions. I can start briefly. I I love that Holly mentioned, their principles, because we have the the very first starting point for us was establishing collectively a shared understanding of what our guiding principles were. Everything from, like, the the our assessment model to our educational program, everything that we've developed has been centered around guiding principles. So that when tensions arise, which they will, or when somebody is asking why are we doing this or to what end or, you know, this is a lot of work for what. Because that's the other thing about this is that the the return on investment is not immediately apparent. You you don't you don't see, the growth and trajectory of your program in your learner immediately. And so when the tensions arise and the questions are asked, you you have those guiding principles to kind of ground, the discussions of that. We're doing we're taking this approach because our guiding principles dictate that path. And and that that's that's that's a real kinda comfortable place to kinda fall back to, because if you have the the the shared understanding about what what kind of that North Star is, then then when the tensions come about, you have an almost easy resolution because of those guiding principles. Right. Thank you, Monica. Actually, and Kieran, just looks like for your question is that how are you, getting your administrators on board? Because we've talked a lot about, you know, faculty and learners, but what about the administrators that will be involved in all touching all the aspects of this from, distribution of the assessments to collecting of the data and then reviewing it. How are you how are you getting on board with this shift? Right. We we we just dedicate a lot of time to meeting with them and onboarding them. And, actually, we have a really close relationship with these administrators. We did just, transition to January, a couple academic years ago. So since then, we we've maintained that relationship knowing that this is, coming down the pipeline as well. So we really provide them with a lot of support. We make sure that they know how to do all of the things that are gonna change for them. We have dedicated user guides actually that we create for each of our departments and their admin. So if there is anything that is up for question, it's actually covered in that guide. We leverage a lot of the one forty five documentation for those as well because it's great documentation, and then we personalize it for within our department. We actually do that across the board. We love we we've noticed that folks love having that type of documentation available so where they don't have to watch a video or listen to someone. They just wanna skim control f and find what they need, and so we we make sure that we have those available for all of our stakeholders. And it goes a long way, I think, with our administrators and that we we do these one on one meetings with them also where it's just we talk about just their process and just the impact on them, just the things that they're worried about, what kind of communications we can support them in and how we can leverage the systems that we have to support that process as well. Thank you. So it sounds like there's really targeted support coming in there. And I love that you actually mentioned the videos and the fact that you still require, you know, text based resources. Because one of the questions that I had was how are the videos actually working? Are they are you are people using them? Are they valuable? So that we always think about in terms of our resources as well is how do you kind of get to the most people so they can and meet them where they're at. And so have you found there been a a definite shift towards text based in the control of idea, or is it still are videos actually still, well received? We we continue to do both. I think there are there's a subset of folks who prefer onboarding materials. And so one of the things that we're working on is, like, for resident onboarding is to have these videos as part of that. Right? And so then they are The then the text resources become follow-up resources for them to reference down the down the road. But they've done this they've utilized these as part of that onboarding. So we see them as as a still a primary resource, because we do think that there's still enough use of them. Whether one outweighs the other, I can't say. I don't have I don't have metrics, unfortunately. I'd love to survey people or or get that data. Then it'd be so rich to figure out is there a difference between administrators versus learners versus faculty, and, how long are they sorry. I'm going down the path, but this is where I get really interested in in trying to figure out the best way to to get to get to users. But, okay, we will lock it down the operational, rabbit hole right now entirely. We had a question in the chat as well. And, Monica, I think this was maybe for you to start that. What platform do you use to provide students with the dashboard tool to visualize their progress and attainment of required competencies? As a new program, we spend a lot of time, weighing options and demoing many different platforms. We we did, invest in acuity one four 45, and the implementation team, including Sylvia, has been phenomenal in helping us design, a a platform and, tools, that would reflect our guiding principles and our our approach. So even the, the longitudinal student assessment tool, STLR, that I mentioned, we are actively building with the acuity team. I saw Wayne was on the attend Tandy list. Shout out to Wayne. He's very patient with us. But it it's been the the support that we've received from the acuity team has really helped us move this all along. And and as we become burdened with the many competing interests that a new program, has, the acuity team has been a a great little, person on our shoulder to be like, hey. Let's let's meet again to move this along. And so it it's been almost like an external motivator for us to to help move this external motivator for us to to help move this project along, because there are always competing in interest. Thank you for that. I appreciate that. And shout out to Hailey. I always hear great things about, programs. We get to work with him. Let me just take a moment. There was a question that came across in in our early registration questions. I think this one might touch upon every program, and that is, how are you all including professionalism in competency based education? How are you, you know, how are you assessing those skills in in your learners? How is it related to them the work that you're doing right now? Maybe let our colleagues stand for maybe chime in for a moment, and we'll come back to to when I come. Sure. So, we actually have EPAs that cover under physicianship, And those include things like demonstrate awareness of limits of one's knowledge and skills, demonstrate thoroughness and dependability and follow through, demonstrate accountability for all professional responsibilities, including call components of patient care. And then we actually revised our, professionalism principles, as we were designing these EPAs so that they would be reflected, not just, you know, the clinical things that we're assessing, but reflected all throughout our entire curriculum, and assessment process, and then also implemented a professionalism form, that can be submitted by anyone really to flag up any issues with professionalism to our student support process so students could get early coaching if there were issues that related to the professionals and principles, so that we set them up for success when they're being assessed on that, with the EPAs in the clinical environment. I can hear that. And can someone submit, like, positive reinforcement of professionalism or is this strictly just a plug? Yeah. So our professionalism form actually has, both, you can flag up issues, but also you can put in accolades, and those accolades will also get aggregated and actually added to the MSPE later. I love that. That's a really, wonderful way to encourage that kind of positive feedback. It's fantastic. Right. We had a question in the chat. And sorry. Actually, no. I'm gonna ask you about Monica, do you wanna talk maybe briefly to that? Because there's a question specifically for you as well that I'm gonna jump over to in a sec about the mind and wellness domains. So maybe we can talk about the three of those together, to support students, to make feedback and just sustainable habits during training. Yeah. So I I think the professional wisdom and the integrated health, competency domains is is are two areas where CBME has a real advantage over traditional grading. Because it's it's notoriously difficult to capture, those behaviors, especially the development of those behaviors, traditionally. So we actually one of our frameworks, that we enveloped in our design process is the integrated behavioral health, for primary care. And that's actually one of our competency domains that that really speak to a lot of that integrated health, the mind and medicine that was mentioned in the the the the chat. And so using kind of a constructivist approach, we've identified, encounters, aligned with our curriculum that will allow us to, create observable opportunities in our students along the trajectory of our academic program. So that students will be receiving as kinda Holly mentioned earlier, we'll be receiving early feedback, early and actionable feedback, across those encounters so that when those developmental opportunities are identified, the the learner can engage with their their faculty coach and and, you know, receive the feedback and develop that action plan to to help kind of, meet those needs as they arise. I think that answered the question. That was very comprehensive. Thank you so much. And it is wonderful how everyone's thinking about these all these aspects of training are really more than just those technical skills because those are the things that in the end really make, a great clinician. And we've got a question in the chat. I wanna I'll hop over this one for a second, but I'm going to come back because I got one that's kind of popping up for me as well and it might be for others. We have a question for our colleagues from Stanford. How have you translated individualized CV data into actual feedback for each student. Now I know you're still working through the the implementation, but I'm curious to see kind of how you anticipate that coming through. Yeah. So the way we've set this up is we've rebranded EPAs as everyday physician activities, and set up our forms such that we basically require, faculty to be doing kind of sixty to ninety second micro assessments, every day, whether the student sends them or the faculty or the resident or, perhaps an APP, initiates them. And so the student is going to get feedback on, you know, what went well today that they should continue to do, so reinforcing things that they're doing well, and then what is one thing they can do to make progress on this EPA for their next patient shift case, day in clinic, day on the wards. And so we're trying to basically say, you know, we're watching very clear tasks happen. Here's what I saw. Here's what I think you should keep doing, and here's what I want you to do differently for the next time. And we're trying to turn that into this kind of, you know, there's always the big c coaching of the aggregate data on the back end, but we're trying to turn this into kind of little c coaching in the moment where they're getting, like, those kind of micro assessments and then being coached for, you know, hey. Here's how you would change for the next, you know, change this little thing for the next patient. And then they can actually demonstrate that they're growing across, you know, individual clerkships, but longitudinally across multiple clerkships. So when the competency committee in the back end basically looks at all that data, they're able to see that growth trajectory over time. They're able to see all of that feedback. And then the students will also kind of have the ability to go back and look at them, but also kind of synthesize that. They have, we have educators for care who kind of serve as coaches. And so if they're kind of or and or advising deans, So we have kind of multiple layers of student, support. So if they're looking at, you know, this data and they have questions where they're like, oh, maybe I should be working on this, they have someone they can go and talk to about that as well in a more aggregate fashion. And thank you for that, and thank you for the question. I actually maybe just mentioned too for folks that are listening and thinking about the really the difference between the assessment of the competencies and the EPAs is really just the level of granularity. So when we're talking about the EPAs that Stanford's implementing, they really are discrete tasks and they can be and they're observable, and we'll come to that in a moment as well. And so when you're thinking about this, you know, looking at CBME, it can or CB in general, it could be both very kind of wide and also much more, granular. And those granular tasks that Holly was speaking to are really are ripe for good feedback because you can say that one small thing that will make a difference, rather than thinking about the whole picture. So I think getting that granular feedback, I suspect, is going to be a lot simpler with EPAs because they're focusing on one thing. So thank you for that. But thinking about also observable behaviors, and, and you've all touched on this one. And I wanna come back to this because when when I think about observable behaviors, I think about, obviously, the something that you can see, right, you can see with your eyes, which makes it easier, hopefully, to assess. But I'm also curious, how do you balance, the observation and the behavior? How do you intend to balance these things with, faculty and rate or burden? Like, what do you think of in terms of, making sure that these assessments are something they can easily complete that is, they can easily understand, which then also the downstream provides better feedback to the students? Is there a tension maybe between those things? I I would say keeping the discreet ask manageable. And and so for for us being intentional about intentional about, volume of encounters and, you know, what we're asking the faculty to observe and provide feedback on for each encounter. So that while a given encounter or given experience, might reflect multiple different competencies. That particular competency that particular encounter relevant to the Stellar tool is only asking feedback on on on particular sub competencies or EPAs, as it relates to the encounter. And I I I also wanna be very clear. My colleagues at Stanford, they're they're looking at program wide implementation or capital I implementation. We're we're lowercase I. Like, we we get to do this, on the preclinical side only first, which will allow us to really pressure test in a a more internal system, before we try to big eye implement this across the distributed clerkship years, which we use in the osteopathic world. So we we will have that CQI, that operational the the pressure test kind of in house in a in a preclinical setting, before and kinda work out some of those those kinks before we send our students out into their their clerkships where a lot of these other constraints and then and other, challenges arise. So just being transparent. No. That is fantastic. And thank you for also just really laying that out really clearly. I think there's it's definitely kind of a a difference between this has been a competencies more broadly in terms of effort when before you move into EPAs and more of that, the frequency that that comes with. So I appreciate you calling that out. I'm not sure if our folks from Stanford, is there anything that you've been able to consider when you're thinking about making things that are observable and clear, without being, really challenging for your, for your faculty? So I think this is a, like, distribution of your assessors and remembering that context matters, because I think a lot of times we think, oh, there's a very clear way to do this that's gonna work for every that everyone. That's not true. So, for example, our medicine colleagues who, you know, spend many days with the same students on the wards, having the same person do the assessment every day, that would feel burdensome. However, the clerkship team from medicine has said, well, we're gonna have the intern do the assessment on the long call day because they're admitting patients with the student. We'll have the attendee do it on the post call day, because the student is presenting to them for the entire all the patients that they admitted. And then, you know, we'll have, on the short call day, you know, the resident do it because they're the one who is basically running rounds with the student. So it's the person who's best primed to to provide that assessment. Our surgical colleagues, basically, like, well, the attending might see them on rare occasion for a case here and there, maybe on rounds, maybe not. The residents are probably in the best position actually to provide the assessment. And interestingly, the surgery residents have been, you know, being assessed in this way for a couple of years now, so they're very useful. You know? And, you know, emergency medicine, they spend one shift with the same attending resident, and they switch for the very next shift. So that's a totally different context. So every clerkship kind of has their own EPA map of here are the things we're gonna require, here are the things that you're able to be assessed on, and here's the way you should probably go about thinking about how you wanna be assessed and who you're gonna ask and who's responsible, on any given day. And so we've just had to be very adaptable, to the different context, again, to make sure that we're able to get the assessments, the students are able to get the feedback, but we're not putting the burden on the exact same person or we're asking people who don't really have the perspective necessary to do the assessments. So being very creative in, you know, who's gonna be the best person to assess x? You know, again, the GME example for emergency medicine, I'm not the best person to assess my resident's pharmacologic management plans. That's the pharmacist who's correcting their bad orders. And so we've had to leverage, you know, different perspectives. And that's gonna be the same, you know, in any context that you're thinking about implementing CVV is thinking about who's the best person to assess this specific thing, how do I make that as feasible as possible for them as the end user, and then how do they provide that feedback directly to the to the learner. I think you're right. I think from both of you, I think it really thinks I think about context, really mattering. And I I had do have we have one more question to chat, and then hopefully, we can squeeze one more question that we got previously. I hope Kieran can help us with. But, Monica, there was a a question about the nine base points for assessment and whether those nine base points are, applied for students in level one through four or some of the nine that can be used at level one, some at level two, and so on. All nine at all levels. But, however, I mentioned our task force developing our observable criteria associated with each subcompetency. They're doing a phenomenal job of developing those criteria based on the learner development. And so even if you have a competency domain, one of nine with 13 subcompetencies. Each of those subcompetencies have learner developmental based criteria developed for all all of those. So, it's been it's been a tremendous effort, and I'm I'm again, cannot overseeing the, praise for my early adopters enough who have worked so diligently to take their clinical expertise and and translate that into those meaningful observable behaviors that will benefit our learners and ultimately our program, particularly in our stage of development. Alright. Thank you so much, and thank you for that wonderful question. I'm gonna try to squeeze in one last question that we got, earlier on. And, because in our limited time, I really do think we want to address this, which is what is the time commitment required, with CBNE and and corporation to implement this. Kieran, I was hoping you could take this home. Yeah. I can speak to it from the administrative side of rolling it out and making sure we have all of our setups and things in place. It's quite a time commitment. Right? So as I mentioned, we are meeting with all of our administrators individually understanding where they're coming from. We're making sure they have all the materials to go out there. So I would say it's a great time commitment, but we are a we're a small but mighty team as well. We're a team of three to five people, on average for these tasks that we're doing. Right? And so, it is possible, but it is very, very mindful around, like, the other things that we need to be doing at the same time. So, while all this is going on, everything else is going on around us, we are still motivated to scheduling those meetings. We preemptively schedule them to make sure that, like, we're addressing anxiety. Even if we don't have the answers yet, they know time is coming to meet with us down the road, and we will we will we will help them feel better. Thank you for that. Thank you for actually just, you know, seeing that there. You have a small but mighty team. You do have a wonderful team, and Monica's team also is fantastic. And so I think, you know, the takeaway is that you can do it. You just, hopefully, have good support along the way. But I think I think despite my wish to squeeze in one more because this is how I I live, it's getting one more thing in. I think we're, that's all the time we have for questions. Thank you all for attending the webinar today, and thank you to our presenters for sharing your insights and actionable strategies. I'm gonna plant the seed here. I will be having you all back in in six months or a little later to see how things are actually going because this has been absolutely fantastic. And so everybody who's attending, please note that you're gonna get a copy of the recording and the slides in your inbox in the next day or so as well as a copy of our CBE best practices guide. And thank you all again so much for your time, for your insights, for just attending also on in your busy days. And, we look forward to connecting with you again in the future. Thank you so much.