Inquiry Oasis Season 2: Adriana Cimetta

Join us in learning about Dr. Adriana's work as a researcher and as the Director of the Center for Educational Assessment, Research, and Evaluation (CEARE). Dr. Cimetta discusses her multifaceted roles in teaching, research, and program evaluation, emphasizing her focus on equity in STEM education, fostering a sense of belonging, and diversifying the STEM workforce.

The conversation explores the distinctions between assessment, research, and evaluation, highlighting how each contributes to decision-making and knowledge generation. Dr. Cimetta also shares insights into CEARE's active projects, which span evaluating undergraduate research experiences, STEM workforce development, and teacher training initiatives. The episode underscores the center's commitment to mentoring graduate students through hands-on evaluation and the overall power of evaluation.

Transcript

Jeffrey Anthony: 

Welcome to the Inquiry Oasis, the University of Arizona College of Education's podcast here in the heart of the Sonoran Desert. We bring you conversations with our esteemed faculty members and staff whose research impacts lives from Southern Arizona to the far reaches of the globe. We explore the transformative power of education in this border town where diverse cultures and ideas converge.

Weaving tapestry of innovation with compassion and a sense of wonder. So, join us as we journey through the sands of curiosity, unearthing insights that enrich and inspire. Sit back and relax as we invite you to dive into the inquiry oasis. 

Dean Berry:

Welcome to the Inquiry Oasis. I'm Robert Berry. I'm the Dean of the College of Education here at the University of Arizona.

Today we are joined by Dr. Adriana Cimetta. Dr. Cimetta is an associate professor in the Department of Educational Psychology. She's also the director of the Center for Educational Assessment, Research and Evaluation. So welcome Dr. Cimetta  to the Inquiry Oasis. 

Dr. Adriana Cimetta:

Thank you. I'm happy to be here today, Dean Berry.

Dean Berry:

Alright, so we're just going to jump right into it, if that is okay with you. And, and just talk about your role here at the college. Tell us what you do and how you do it. 

Dr. Adriana Cimetta:

Okay. Um, well, I, I have a few different roles, uh, because I am an associate research professor. I do teach and advise students, and I enjoy that. I teach, um in the college or in the Department of Educational Psychology. I teach masters and PhD level classes around educational evaluation, um, and an advanced, uh, evaluation, more of an applied evaluation course. And then, um, a research methods course for PhD students. I also teach occasionally at the undergraduate level with adolescent development.

All right. And so that's sort of my teaching hat, but then my research hat and serving as the director for the Center for Educational Assessment, Research and Evaluation, which we call CEARE. Um, I do a lot of work around evaluating programs that have to do with diversifying the STEM workforce. And so, a lot of things around equity and social justice in STEM education and looking at sense of belonging of students, identity of students, their self-efficacy, all things that will help them persist in a STEM major and into the workforce.

Dean Berry:

So, Dr. Cimetta, we know that you are the director for the Center for Educational Assessment, Research and Evaluation. Can you talk to us about the difference between assessment, research, and evaluation?

Dr. Adriana Cimetta:

Yes, that is a great question. Um, so when we think about assessment and I'll, I feel like it's a, the way, it's, um, the order of it for the center is actually a great way to think about it. So, with assessment, we are specifically gathering information around something. It could be knowledge, change in knowledge, it could just be overall knowledge. It could be perceptions, attitudes, beliefs. We are assessing, for lack of a better word, but testing, measuring something and what we learn from that we can either use in research or evaluation.

So, assessment alone provides us information for whatever the topic is that we want. When we talk about research, we are asking questions to inform new knowledge with research. And so, we're using assessment to help us understand and to create a body of evidence around, um, a certain topic and then we're contributing that, um, new evidence to the field or to the most applicable mechanism for that.

Now evaluation has a component well evaluation we do assess and a component of providing that knowledge, oftentimes for the research to inform the research. And we consider that sort of our outcome assessment or our outcome evaluation or summative evaluation. And we're looking at those outcomes. And those outcomes tie very closely to the research that we're looking at. 

But evaluation is larger than that, and it's also, we have this formative or process evaluation and just to understand how things are functioning. Are we doing things the way we said we were going to in our project? In our, on our grant, um, is the organization ding what they're supposed to, the mission that they have; are they actually living and doing what they're, the state, the statement of their mission, um, and the purpose of their mission. 

And so, evaluation is a, I don't want to say it's more expansive, but it, it's a different take on understanding a problem and the. The  key really is for evaluation. We want to provide information to make decisions, and we can make decisions about how something's functioning, or we can make decisions about outcomes and like, of interest that people can use or the information that we provide, they can use to make those decisions.

So, I think that's a difference. Um, research isn't necessarily informing decision making direct. I mean, people use their research and policies and things are created, but evaluation really is used for decision making. It provides information for decision making instead of maybe generating new knowledge.

Dean Berry:

Alright, well can you talk to us a little bit about the work that the center is doing or has done? So what kind of work has the center done? 

Dr. Adriana Cimetta:

Yeah, we're really busy right now. I think we have about 14 active projects. Many are in undergraduate research. Um, and so we are evaluating a lot of different projects from different agencies that the National Science Foundation, um, uh, National Institutes of Health. Um, department of Education.

And, um, so we're, we are the evaluators on many projects working across many different colleges here at the university around, um, undergraduate research experiences, diversifying the STEM workplace, getting students into graduate programs. Um, we also are working with, um, teachers and so teacher training in stem, not always in STEM though. Just general like teacher leadership initiatives with, um, TREC (the Tucson Educational Regional Collaborative). 

Dean Berry:

Yeah. 

Dr. Adriana Cimetta:

Um, with TREC and their teacher leader program and their communities of practice and helping them to evaluate that and inform, um, changes to, uh, better that experience.

Um, so that is what we're doing. We're also so that component that we have these active, these 14 or 15 active projects. But I, what I think is really exciting, I mean, I, I love the work that I do and I, I love evaluating these projects and working with the different teams. Um, I feel like it challenges me personally. Um, but, having the center has allowed us also to have a training component for students interested in evaluation, potential evaluation careers, or going on as researchers or educators and how they would use evaluation in their own, um, careers. And so, we have about 15 graduate students working with us on the different projects. And so, they're getting sort of hands-on experience evaluating. So ,I do a lot of work with undergraduate research experiences and now the center is providing educational evaluation and research experiences for graduate students. 

Dean Berry:

Alright. Well, thank you for that. So I'm going to, I'm going to jump in and ask you, can you share with me three words that talk about your passion for the work that you do?

Dr. Adriana Cimetta:

Um, I would say three words about the passion. Can they be more than a single word? 

Dean Berry:

Absolutely. You can hyphenate all the words

Dr. Adriana Cimetta:

Okay. Um, I love to be able to produce information to inform decision making. Um, another component of that is that the information that I gather I do in culturally responsive, using culturally responsive practices so that I'm including all different voices and I am aware of the impact that the information that I'm gathering and presenting will have on different populations. Um, and so, let's see, and the last set of words would be, um, involvement and engagement. 

Dean Berry:

Alright. Alright, so I came up with three words as well, and, and those, these three words come from, uh, I've read some of your work and so I want to jump kind of right into that. But let me tell you my three words. Okay. So, well, the first word is kind of hyphenated, so it's more than three words. So, undergraduate research, power change. 

So those are my three words. So, I want to kind of jump into this. Tell us about your research, talk about your work and, and your research and, uh, and maybe talk about these three words, undergraduate research, power, and change. 

Dr. Adriana Cimetta:

Yeah. So, a lot of my research is around undergraduate research experience and so that experience that students have in research experience in the STEM fields and in the non-STEM fields. And really understanding what components of the programs work best for different populations of students. Um, here at the University of Arizona, we're a Hispanic serving institution, so we have a lot of, um, Latinx students, we also have a lot of first generation students. So, what practices attract and retain students in undergraduate research is really important to me. 

Some of those characteristics that I look at are the belongingness that students feel to their majors and to the university as a whole. Their identity as a researcher, um, and then sort of their confidence in doing it somewhere around the self-efficacy. So, a lot of the work that I do has to do with undergraduate research and in looking at those programs, um, and initiatives that offer that are addressing undergraduate research and making it accessible to people, there are a lot of dynamics that go into it.

And so that's where power comes into my research and what are the different levels of power and. Uh, my colleague, uh, Dr. Friesen and I have, have sort of broken that down into relational issues of power, um, political issues of power, discursive issues of power, and structural power. So, um, when we talk about relational, right, that's the influence one person is having on the other.

And they can, all of these can be both positive and negative. Um, and then when we talk about political, um, what are the needs that are being promoted, the needs and the outcomes, and what groups are influencing those needs and outcomes? And then when we talk about discursive power, um, what are the beliefs, um, and values for the program and the work around that program? And even when we talk about evaluation, who's driving those values as to what we want to, um, measure? And how we want to measure it? And then when we think about structural, especially here at the university, what are sort of the institutional, um, policies and procedures that may, um, strengthen programs or inhibit programs?

And so again, both are positive and negative, and it's really understanding the influence of these different types of power on the programs and how that affects the outcomes, how they're designed, and how it affects the outcomes. 

Dean Berry:

So, when, when I read your work, I have to say honestly, I personalized it in some ways because I have a son who, um, who majored in electrical engineering and, um, he talked about his own undergraduate research experience.

And so, as I read, as I was reading your work, I was seeing my son Donovan. Partly because, um, he talked about his research team, the folks who were on his research team, and how they influence him. He started off as a mechanical engineer, then switched over to electrical engineer, and so he's since graduated.

But you know, the idea of having the undergraduate research experience was significantly powerful for him because it allowed him to explore, it allowed him to be a part of a team. It allowed him, he talked about you know, his, his, his teammates and his relationships and those things really mattered for him.

But he also talked about some of the struggles, some of the struggles he had with people on the team. But it, it prepared him. It prepared him. So, as I was reading your work, I was imagining that, you know, while I didn't have kind of real insight, but uh, you know, just kind of talking to my son about his work, am I on target with, with kind of envisioning that?

Dr. Adriana Cimetta:

Exactly and so, right, that perspective that your son is having, that's something that in our work, we really strive that the student voice is there, right? So, we have the instructors or we have the program, um, or the grant that's funding this type of initiative and everybody comes into it with different views and expectations and ensuring that we're meeting the needs of the students so that the students feel valued and think this is a value and want to persist is really important, and that the faculty or TAs or staff have the resources they need to provide the environment, um, to allow for students to flourish and thrive is really important.

And so, it's not just one thing and they're not separate. They all work together and it's so, so it's, it's sometimes it's hard and in those articles, we, we disaggregate everything and we have them in their nice little boxes. But as you're reading it, or even as you were just talking, it all sort of starts to overlap and it's, it's really, um, a system overall, um, that you have to look at and see.

What are the dynamics of it and where could we improve to meet the needs of all involved and where, um, what's working well that we want to continue, that we don't want to lose when we're working in those situations?

Dean Berry:

Alright, so, so talk, talk a little bit about change, you know, this idea of change and how you talk about it in your work, um, and how it impacts your research.

Dr. Adriana Cimetta:

Yeah. Much of what we're doing, we're trying, there's an established issue or problem that we are trying to address. So, we're trying to systematically make changes to ultimately, um, retain people in STEM majors or ultimately have an outcome. And so, we're, we're make, the program is designed to elicit a change.

And so, what we want to do, especially as an evaluator in, in my work, is work with the programs and the different organizations to ensure what they're setting up has the best possible chance to produce the change that all of the groups together have decided is important. And so, then we as the evaluators want to come in and, and as the researchers, because I wear both hats as the researcher and the evaluator.

How do we, um, help facilitate that structure to improve, continually improve it and keep best practices in it, but then also be able to assess it? And when we talk about assessing it, it's more than just a survey of students, right? What's really the best way to capture change and what change is important?

So it may be, and for certain audiences, we may need to show the numbers data, and for other audiences, we may need to be able to tell the story. And so ,in our world, we sort of differentiate that sometimes as quantitative data around the numbers and qualitative data that really enriches, um, the understanding of what's happening and the experience of the participants and the people implementing the programs.

Dean Berry:

So, I'm going to ask your opinion on something. Okay. And, and what I want to ask you, because, you know, uh, being a math educator and, and having a, some insight in a broader field of STEM, but not probably as much as yourself, but. We know that there have been policy calls and other types of things pushing the fields in STEM areas to diversify.

And one of the concerns I share is, is the STEM fields ready for the diversification that they're seeking and what are they doing to be prepared for that? What are your, what's your thoughts on that? 

Dr. Adriana Cimetta:

That's an interesting question. I hope the STEM field is ready. I don't know if I've thought of it that way, um, because I'm sort of, I think sort of my thoughts on her that I am on the side of, let's diversify it because we have really complex problems. And to be able to address those problems, I think we need to diversify, and we need to have different perspectives. So, if we really want to solve some of these major challenges, whether challenges, whether they're climate change challenges or um, just new techniques for coding and AI that's coming about (Artificial intelligence). We need to have a diverse perspective on how to handle that. 

I'm a woman. I actually was a biology major, so I was a STEM major. Um, right. And I think sometimes women and men address problems differently. I. It's not that one is right and one is wrong or one is better. It's their difference. So, if we can have a lot of different perspectives addressing problems, really important problems to continue moving and advancing these fields and addressing really big problems that we're facing as a, as a nation and as a world, um, I think it's, I think we have to diversify. 

Dean Berry:

Yeah. I for me, I, I've been, I've been given a lot of thought to that and, and thinking about, um, just, you know, if, if the fields are ready for the diversification, it may you know, bring up different lenses on how the issues are viewed and how the problems might be addressed. And, and, and, and to me that's exciting. So, I hope there's some excitement in across the STEM fields for that kind work. 

Um. So, you talk about this notion of power and this notion of power and how these four dimensions, am I using the word dimensions correctly?

Dr. Adriana Cimetta:

Yes. 

Dean Berry:

These four dimensions of power. Uh, and are these at the forefront in the stem fields or they're thinking about this or, you know, so I see, I see the relevance and the overlap, but I'm wondering the field is considering those relevance and overlap. 

Dr. Adriana Cimetta:

I don't. Think the field is considering the relevance explicitly. I think it's, I think it's something that we've picked up on in our work as to these different influences and factors affecting, um, this diversification of trying to diversify the STEM field and the, um, different practices to do that and the different, um, techniques and strategies they're using. So. I don't know that the, if you are a physicist, that you are talking about power.

Um, so I think that's the issue. Um, coming at it from an educational reach researcher perspective and an evaluator. I'm looking at all of that, but I'm not sure if the, um STEM scientists are, that's at their forefront. 

However, I do think, and you know, there are holdouts, um, in certain disciplines and just certain viewpoints on what STEM education or discipline specific, I don't want to say STEM education as a whole, I want to say certain disciplines and, um, people's views as to what that should entail, have been found to not be best practices for retaining a diverse population in the field. And so, I don't know if they're thinking the power I'm exerting to fight this change in the curriculum maybe needs to be revisited. I'm not sure that's spread across everyone, but I, I think there is a group that's gaining momentum in the different disciplines that's really putting the educational component, and I'd say person center it a little bit more.

Is it completely widespread? Probably not. 

Dean Berry:

So what has been the most surprising outcomes from your research? 

Dr. Adriana Cimetta:

I think the student experience has been the most surprising because while we, I think sometimes go into it thinking if we do this, it's going to benefit the student and it's going to benefit all students, but especially female students, or especially Latinx students. 

And it's been interesting at times to see that that's not the case always. What we go into thinking and what we've, and we've built upon that from experience and from other literature. Um, and research that's been done doesn't always hold with the populations and the situations.

And there are a lot of factors affecting, um. Affecting the changes that we see. And so trying to, I think, have a multi-pronged approach to, um, improving. Again, a lot of the work I do is around sense of belonging and identity and that, um, sort of confidence and self-efficacy. Um, it's not a one size fits all, and I, I think that's important.

Well, I feel like we know that intuitively. We are now seeing the data to show that. That different populations and different, um, groups of students respond differently. 

Dean Berry:

Alright, so I am a person who's who, who, uh, let's be hypothetical here. Okay. Right. So I'm a person who wants to kind of, you know, elicit change in the field. What would you advise me to do? I'm, I'm, I'm the team leader, the professor or the person in charge. What advice would you give someone who has that kind of power, who wants to impact change in the field? 

Dr. Adriana Cimetta:

Um, well, I think first I would want to know what that change is, right? So let's, instead of like, we all want to make a difference in the world. I would hope right here that we're at the institution, we're educating the next generation of scholars, of researchers of, um, in the, in the workforce. And so, what change do you want to make? And really understand what you want to do and then talk to you about how you envision doing it and why, why is this important?

What are the reasons that you're seeing the issue that you want to change? And then, and then I like to have that, um, discussion with sort of the project leader, but then how does the entire project feel? And I like to hear all the voices. So, is this something that you've just determined or is this something that you've gone to the students or the population that you're working with? Is this also coming from them? Um, I think that's really important to know. What is the change? Why, and where did you, how are you going to do it? And where did that come from? That notion of it. 

Dean Berry:

Alright. Alright. Well so I'm going to ask you this. So what one book or paper that is meaningful to you that you would recommend for our audience?

Dr. Adriana Cimetta:

Okay. Well, I mean there are a couple of different ones. Um, however, because I'm also a parent, so I have a few. Um but, I'm going to, I'm going to keep it centered around evaluation, um, and research. And one that, um, I spec really like this is not maybe the, you know, vacation read on the beach, but, um, the Handbook for Practical Program Evaluation is a great book.

It has a lot of different chapters that cover different things. One of my favorite is the chapter by Hood Opsen and, and Kirkhart, culturally responsive evaluation. I think that chapter, I believe it's chapter 11, um, is an amazing chapter to just sort of situate, um, the work that I do. 

Dean Berry:

Oh, wow. Oh wow. Oh wow. 

Um, what question would you like to answer that I didn't ask?

Dr. Adriana Cimetta:

I think one question. Because I do a lot of evaluation work and research around evaluation. Um, I think sometimes evaluation, um, is viewed upon as a, um, sort of somewhat negatively, right? 

It's, um, well, the evaluator's here to make sure we're doing what we're supposed to do and they're not really, they're judging us. Like, I feel like there's a lot of judgment around the term evaluation and sometimes, depending on the different group, I don't always even like to use that term. And we're, um, what I would like for people to take away is that evaluation and research we're really trying to provide information to make informed decisions that allows for equity and social justice in whatever area.

And I think that's really important, that we're well informed before we start doing something or before we start changing something. Let's really understand what's happening. 

Dean Berry:

So thank you for being with us, uh, here at the Inquiry Oasis, Dr. Cimetta Um, Dr. Cimetta again is, she's a, an associate professor in education, psychology, and the director of the Center for Educational Assessment, Research and Evaluation.

And today we touched on several things. We talked about the work of the center, we talked about this notion of power. Uh, we talked about undergraduate research. But more importantly, we talk about, you know, kind of this, the work that impacts change, whether it's change in the STEM areas or just change across broader, the broader field of education and within and around the university.

And so, I really appreciate your work and thank you so much for being here on the Inquiry Oasis here. 

Dr. Adriana Cimetta:

Thank you, Dean Barry for inviting me to talk about my research and the Center for Educational Assessment, Research and Evaluation. I hope that everybody has learned something today.