The Inquiry Oasis: A UArizona College of Education Podcast

Inquiry Oasis Season 2: CEARE: Dr. Adriana Cimetta, Dr. Rebecca Friesen, & Students

• UArizona College of Education

Share episode

Join us in this engaging episode in which Dr. Adriana Cimetta and Dr. Rebecca Friesen of the Center for Evaluation Assessment Research & Evaluation (CEARE) highlight the journey of three doctoral students from the University of Arizona, who recently clinched the top prize at the 2024 American Evaluation Association's United States Student Evaluation Case Competition. Hannah Douglas, Emilia Mbone Roland-Umasabor, and Yamini Bukhaya share their experiences and insights into the world of evaluation, shedding light on how the discipline serves as a formidable catalyst for change.

Jeffrey Anthony:

Welcome to the Inquiry Oasis, the University of Arizona College of Education's podcast. Here, in the heart of the Sonoran Desert, we bring you conversations with our esteemed faculty members and staff whose research impacts lives from southern Arizona to the far reaches of the globe. We explore the transformative power of education in this border town where diverse cultures and ideas converge, weaving a tapestry of innovation with compassion and a sense of wonder. So join us as we journey through the sands of curiosity, unearthing insights that enrich and inspire. Sit back and relax as we invite you to dive into the inquiry oasis.

Dr. Adriana Cimetta:

Hello, we are excited to talk to you today. I'm Adriana Cimetta. I'm an associate research professor in the Department of Educational Psychology and the director for the Center for Evaluation, Assessment, Research and Evaluation, which is a very long name that we have shortened to CEARE. The centers and my research focus on evaluating the impact of educational practices, supports and experiences in creating equitable educational opportunities for diverse populations of students, including inclusive and culturally responsive evaluation practices. Today, we would like to discuss and recognize the wonderful achievements of a team of three doctoral students from the Center of Educational Assessment, research and Evaluation Hannah Douglas, Emilia Roland-Umasabor and Yamini Bhukya. They are all PhD students in educational psychology and they won the 2024 American Evaluation Association's United States Student Evaluation Case Competition. Great job, congratulations.

Dr. Adriana Cimetta:

These students, known as the evaluators ensemble for the competition, had the top-rated proposal out of 12 teams that participated in this national competition. I, along with Dr Rebecca Friesen, had the honor of serving as the team coaches to support Emilia, Hannah and Yamini in preparing for the competition. To give a brief overview of the competition, pre-read materials were distributed to all the teams with guidelines for the competition. Then the teams received the case materials, which contained details about the organization and the program that needed to be evaluated. The student teams then had seven hours to read through the organization's case materials and develop a comprehensive evaluation plan. The students were not allowed to have any assistance from the coaches on the day of the competition, and I just have to say how proud I am of Hannah, Emilia and Yamini's application of their evaluation knowledge and skills to thoughtfully develop an inclusive and culturally winning responsive evaluation plan. Before we talk to the stars of today's podcast, I wanted to give Dr Rebecca Friesen an opportunity to reflect on her experience supporting and coaching the students' development for this competition, Dr. Friesen.

Dr. Rebecca Friesen:

It has been an absolute pleasure working with these people. They have been so. They are so thoughtful and hardworking and have worked on a variety of different evaluation projects and I am just so, very proud of them. I want to say I did not do very much at all with coaching them before the competition. This is all them. This is their hard work, their ideas, their knowledge that they applied and clearly worked really well together. It's a very fun, very fun, hardworking, thoughtful group. So I'm just very, very proud of them.

Dr. Adriana Cimetta:

Thanks, Rebecca. Okay, so now we're going to switch to the students, and I have a couple of questions. We're going to talk about their experience and what evaluation means to them. So, first I'm wondering if each of you could tell us what inspired you to compete in the American Evaluations US Student Evaluation Case Competition, and can you discuss what it means for you to win, to have won the competition? We'll start with you, Hannah.

Hannah Douglas:

Absolutely. I had previously competed the year before for the inaugural competition and for me that was one of the first experiences working collaboratively with a team and it was one of my first experiences writing collaboratively and I had such a great experience I couldn't wait to do it again the following year. And this year's AEA the conference theme was all about amplifying voices and for me that was something that I was really drawn to and I know that this particular theme would provide an opportunity to really glean into some of the skills and resources that I've worked from other projects for. So it was really exciting to be able to kind of pair that with this year's conference theme.

Dr. Adriana Cimetta:

Great and Yamini.

Yamini Bhukya:

I think I was really inspired to compete in this competition because it's an opportunity for me to apply my academic learning to the real world settings and I feel like evaluation is not just a theory but it's all about making impact in the society, and I think this year's competition theme really, really resonated with me and it was really inspiring to work with a program which was focusing on youth, and I think it felt like an opportunity for us to grow as evaluators and also to collaborate with the team and work on such a wonderful themed competition.

Dr. Adriana Cimetta:

Thank you, Emilia.

Emilia Roland-Umasabor:

Yeah, for me it was just the right time because I was ready at that time to get involved in evaluation. And that project came in just at the right time when that made up my mind to say, okay, now you're done with classwork, it's time to get out there and put all what you've learned, all the theory you've learned, into practice. And when I got that email to be part of the team, I jumped on it and that was the best decision I made because it gave me an opportunity to work collaboratively. For someone who's not tech savvy to be able to do that with a team of other my teammates it was really amazing for me.

Dr. Adriana Cimetta:

Well, it sounds like a great experience for all of you. Could you tell us a little bit about your case competition project, how you prepared for the competition, what it was like? I know you've all talked about the collaboration, but what it was like developing your evaluation plan for this community organization in a day, receiving the materials and producing the plan.

Hannah Douglas:

And I think part of what made it really exciting to do it was because it was done in a day and that it was a real life case that we were working on. The day of the competition we received that case summary with the full details of the program itself, and so this was Caldera Youth, arts, education nonprofit, based in Portland, Oregon, where our conference will take place. And I think that was what was exciting too is that we were working on a real life case and the evaluation proposal, the pre-reading, everything as far as the preparation for it, they really outlined what requirements needed to be in there and so as a team we kind of looked at what our strengths were and kind of decided kind of how to divvy up those tasks and what we could individually work on, what we needed to work on together and then how to proofread it all. So I think preparing prior to the day was looking at just prior examples from previous years competitions, looking at other logic models, focused evaluation plans and different examples that we could pull from that.

Yamini Bhukya:

I think, adding to what Hannah said, our project, our competition about the Caldera program, youth, which focused mostly on the environmental, education, arts and mentorship with the middle school and high school students, so it just gave us about seven hours on the day of the competition. So it was mostly about immersing ourselves, trying to understand about the project that we were working on and it was mostly about brainstorming, research and teamwork and we tried to understand the project's goals, objectives and its stances on the equity and social justice and everything. So we try to understand how to work our evaluation plan around the subject and for us. I think we tried our best to balance the creativity and the practicality to make it interesting and innovative but also, at the same time, manageable, to create a good evaluation plan for the youth. Excellent.

Emilia Roland-Umasabor:

Hannah and Yamini have kind of said it all. But what we did, the previous cases that we looked at and all the pre-reads that were sent to us were really helpful. But for me, not knowing exactly what you were going to get was actually kind of overwhelming in a way, because you're like what do I read, what do I focus on? I have no idea what I'm going to get. What do I focus on? I have no idea what I'm going to get.

Emilia Roland-Umasabor:

So what I did at that point, when we did the pre-reading and determined what we wanted to do, I went back to previous class 5-8, and I looked at my course, the text we had, and I looked at some of the materials, some chapters that we were given, and I went through some of them to kind of refresh my memory for some of the things that we're going to encounter when we actually open the case, because once you've opened that case you cannot get help from anyone. You have to start working on it. Our time starts then. So I needed to be prepared. So when we come together, we all bring in our skills and our resources to make sure we get everything done in time.

Dr. Adriana Cimetta:

Thank you. Well, it clearly makes me happy to hear that your EDP 582 educational evaluation course helped prepare you for this competition. That is the goal of the course to be able to go out and do successful evaluations so I'm glad that was helpful. Could you also speak to what elements of the evaluation were important to the project and why you thought they were important to include in your team's evaluation plan? So you all have had this evaluation course. Some of you have worked on evaluation projects, so what components of evaluation and what parts of evaluation were really important to you to incorporate into your plan for the Caldera Arts Program?

Hannah Douglas:

Especially knowing the theme of the conference was on amplifying voices in evaluation. I think it was really important that we incorporated partner input and prioritized inclusivity throughout the evaluation plan. So one of the things that we really emphasized we were trying to figure out how to create the logic model and our evaluation matrix was how do we include the partner voices and how do we make sure that everyone had a equitable representation in the evaluation plan, and so our partner analysis really prioritized making sure that all the relevant partners were not only considered but really engaged in the process throughout and so that we had more of a holistic and actionable evaluation plan. I think we also looked at taking more of a data justice approach to integrate that into our evaluation plan to make sure that we were transparent, that we were very community driven, because this was a youth arts program within the community in Portland, so we wanted to make sure that we were representative of the community then that we treated the evaluation findings in a way that would like promote that well.

Yamini Bhukya:

I think, in developing our evaluation plan, inclusivity and cultural responsiveness were our top priorities, because Caldera works with the youth from diverse underserved backgrounds, including many BIPOC students and those from low-income rural areas, so we felt that it was crucial to reflect the diversity in our evaluation plan. So we tried to mainly focus on the participatory evaluation and, just like Hannah said, we focus most on data justice and we try to integrate both qualitative and quantitative methods into it, such as surveys, focus groups and creative mechanism creative feedback mechanisms in order to create a holistic evaluation plan.

Dr. Adriana Cimetta:

Thanks, just in case people on the podcast or who are listening don't know what BIPOC means Black Indigenous People of Color. Thanks, just in case people on the podcast or who are listening don't know what BIPOC means Black Indigenous People of Color. Thanks, I just wanted to make sure everybody was aware of that term, Emilia.

Emilia Roland-Umasabor:

Yeah, like my teammates have said from the pre-read, the major information that they wanted us to focus on was equity and data justice, and that was the main thing that we looked into. We made sure that we covered every aspect of that and not to leave out any relevant party. So we looked at all the data, made sure every aspect of it, all the parties involved in it, the youths, the BIPOC, all of them, we made sure. We made sure they were all included in the evaluation.

Dr. Adriana Cimetta:

So thank you. Well, it sounds like a great experience and a successful experience, shifting away from the competition and exploring what interests you about evaluation. Can each of you discuss how you envision evaluation or how you hope to use evaluation to impact communities, to improve people's lives?

Hannah Douglas:

I see evaluation as a tool really for empowerment. I think evaluation really helps communities and individuals gain insight into the program's effectiveness and I think evaluation has the impact to drive meaningful change. And the thing that I love being part of evaluation is it's not discipline specific. I like being able to see all the different programs and what the missions and the values that are driven by the program or the service that's being evaluated, and so I think evaluation really does impact communities. It does improve people's lives by the results. People are doing meaningful, creative things and evaluation just brings that to light and brings to light the effectiveness of that program. I think it also illuminates those inequities at times as well and really supports change if it needs to be brought about. I think it is a problem-solving driven approach to be able to say like what you're doing is very well, this is small ways you can improve, and so I think evaluation is kind of ties into both positive and negative benefits for programs, but ultimately it does drive about meaningful change.

Dr. Adriana Cimetta:

Thanks, Yamini.

Yamini Bhukya:

I think evaluation, when done right, has a real, real power to create a meaningful and a strong change in a societal impact. And we see it not only as a tool of success but also empowering the individuals. And especially when we have meaningful data, we understand what's working, what is not working, how to do like make improvements or suggestions or any kind of changes. And I think evaluation also talks about accountability of any particular organization, helping the organization to improve but also holding on to its mission. And I think in the case that we were working as the Caldera Youth Program, I think that evaluation plan, which focuses mostly on the arts and the mentorship and the environmental education, I think that would really help foster success and the personal growth and build confidence of the youth. So I think it's really important.

Emilia Roland-Umasabor:

I see evaluation as a catalyst for empowerment, just like Hannah, because you're empowering the organization or the people who are intended to receive this benefit from whatever the results of your evaluation are, you're empowering them. I'm going to borrow from a theme. It's a way of amplifying their voices through you including them in the evaluation process. You're amplifying their voices through you including them in the evaluation process. You're amplifying their voices. You're giving them that opportunity to be part of something. I also see evaluation as a tool for data-driven decisions, because when an evaluation is done, it generates insights and that insight is used to generate policies and improve those programs. So that's how I can see evaluation.

Dr. Adriana Cimetta:

Excellent. I think you all touched upon the empowerment, the inclusivity, the social justice and those are all really important things and that really can impact a community and having the information to make these data-driven decisions. Can you talk a little bit more or tell me why or how evaluation work is meaningful to you on your educational journey and your future academic and career plans? What role does evaluation potentially play in that?

Hannah Douglas:

I know for me I love evaluation, mainly for me because it is not isolated to a specific discipline For me as a graduate student and I still don't fully know what I want to be when I grow up, and part of what draws me to evaluation is that I'm not forced to decide, I'm not pigeonholed into one specific discipline, and evaluation is exciting because you get to see different programs and services, from policy to education to research to service. You get to get insight into science courses to community projects in schools and out of schools, and so for me, I love evaluation because I get exposure to so many different people, different disciplines. I find that really enriching and exciting and meaningful. You get to just see what other people are doing and get to play a small part in that by providing evaluation efforts to showcase the meaningful work that they're already doing. Thank you.

Dr. Adriana Cimetta:

Yamini, how about you?

Yamini Bhukya:

I think for me, evaluation is really, really meaningful because it helped me understand my academic knowledge and learning and put it into the real world settings, and I think this competition was a pivotal experience for me right now and also for my future academic career as well, because, looking ahead, I really want to work with social policy, public policy, educational policy and the nonprofit organization, so I think evaluation would really play a critical role in those areas and I feel like it's all about working with the data and making informed decisions.

Emilia Roland-Umasabor:

Yeah, for me, coming from a school psychology background, evaluation is really meaningful to me because there's that transparency and you have the opportunity to make a meaningful impact. You're able to make a change because, once that evaluation is done, you're able to see, the world is open. You're able to see a lot of things, you're able to collaborate with people, you're able to network, you're able to make informed decisions with others. So that's the way I see evaluation. I see it as the what is escaping me right now, but it is an amazing thing and it's something that everyone should have an opportunity to experience, because you're able to put your skills into practice. You're able to figure out things and walk through things you know through step by step.

Dr. Adriana Cimetta:

Great. Well, we've spent a lot of time today talking about evaluation, the competition, what it means to you, and I would love to share some resources with our listeners. And I would love to share some resources with our listeners, so I'd love to hear from each of you about a book or a paper or a resource that has significantly influenced your development as an evaluator, and could you share one of these with the audience, our listeners, and explain why it's impactful to you?

Hannah Douglas:

One of the most powerful ones that I've encountered. I had a hard time narrowing it down to just one, but for me, the Urban Institute has a Do No Harm project. They have a lot of resources focused on data reporting, data visualization, and they provided a lot of ways for applying equity, for data visualization, accessibility and how to manage your data in an equitable way. The Urban Institute's Do No Harm Project is probably one of my favorites that I go to.

Yamini Bhukya:

For me. It's a book by Michael Quinn. It's about utilization-focused evaluation. I came across this book during my undergrad in one of my courses, and I never knew I would be a part of an evaluation team during my undergrad in one of my courses and I never knew I would be a part of an evaluation team during my PhD. So this book mainly emphasizes that evaluations should be designed and implemented keeping in mind the end users. It talks about how the findings are only useful when they're actionable and meaningful. Also, again, being a part of the CR evaluation lab is also very critical to me, so yeah, Well, that's good to hear and, yes, michael Kempatton's work is very useful.

Dr. Adriana Cimetta:

Right, the utilization focused evaluation and Emilia, how about you?

Emilia Roland-Umasabor:

Yeah, I mentioned this before. The main book that really is it for me is the evaluation text. It's titled Evaluation: A Systematic Approach. It's by [Peter H.] ossi. We had that book EDP 582, and that book was it for me, for a novice. That's a book you need to, a text you need to read. It simplifies everything and takes you through steps of evaluation, what every step means and how to go about it. And then there were some other materials from that class that I still refer to, like the Kellogg Logic Model Guide. That was another one that is really really, really good. So if you're into that, check it out.

Dr. Adriana Cimetta:

Excellent, that check it out, Excellent. Well, Hannah, Emilia and Yamini, thank you for your very insightful reflection on your experience participating and winning the United States Student Evaluation Case Competition and sharing with us a little bit about your development as evaluators. And I wanted to let our listeners know that Hannah, Emilia and Yamini will be formally recognized at the annual American Evaluation Association's conference and they will be competing as the United States representative in the World Student Evaluation Case Competition. So I'm very excited for them to have that experience and represent the US. So good luck as you prepare to represent the US in the World Student Evaluation Case Competition. Thank you, thank you, thank you.