Measuring the Immeasurable

Most of us in the nonprofit world were taught that real evaluation requires a research team, a grant, and a methodology section. So we measure what's easy to count — and quietly avoid the things that actually matter.

This episode is about doing it differently.

Hillary challenges the way program evaluation typically works: heavy on outputs, allergic to nuance, and overly deferential to "capital-R" Research. She makes the case for "small-r" research — using validated frameworks as a starting point, defining outcomes from lived organizational experience, and building measurement systems designed to help you learn, not just report.

You'll hear from Christian Quijano, Director of Data & Analytics at the Downtown Women's Center in Los Angeles, on how his team took academic research on economic mobility and turned it into something their organization could actually use — an internal measure they called "earning power." (Spoiler: their first definition wasn't good enough, and that's exactly the point.)

The episode closes with a case study from All Square, a social enterprise in Minneapolis working to shift public perception about incarceration. How do you measure something that lives in people's minds? Key informants, customer reviews, and existing research — it's more possible than you think.

Mentioned: Acs, G., Conner, A. L., Lyons-Padilla, S., Markus, H. R., Patel, N. G., Tumolillo, M. A., & Eberhardt, J. L. (2018). Measuring mobility from poverty. Stanford SPARQ. https://sparqtools.org/wp-content/uploads/2018/06/measuring_mobility_paper.pdf

Guest: Christian Quijano is a nonprofit data and strategy leader who helps organizations uncover the story their data is telling. He brings a continuous learning and improvement mindset to connect the dots between programs, operations, and outcomes. As Director of Data & Analytics at the Downtown Women’s Center in Los Angeles, he leads organization-wide data infrastructure, dashboards, and quality and compliance strategy to strengthen outcomes for women experiencing homelessness. With deep expertise in theory of change and monitoring and evaluation, he is known for answering big questions through clear, compelling visualizations, blending technical rigor with community-centered design—and is actively exploring how emerging tools like AI can support this work responsibly.

Transcript

My name is Hillary Frances, and one of the things that I'm thinking about is, how many of you tell me that you'd like to measure the impact of your work but it feels too elusive, too abstract, too hard to quantify, thriving, or equity, or healing or recovery. And you're right, it is hard, but that doesn't mean it's impossible. In my experience, about eight out of 10 of the organizations I work with are measuring how busy they are, not how effective they are. They have systems for tracking outputs, the metrics that demonstrate churn. This is number of people served, number of acres preserved, number of workshops offered average hourly wage. But they do not yet have systems for tracking outcomes. The percentage of people served who transform their lives in some way or another, the percentage of the population that changes their mind about something. All questions they'd like to have answered, but questions they haven't touched with a 39 and a half foot pole because they seem to require a research team to answer, but we are finding ways to answer them without a research team. 

You're listening to tailwinds ideas, fueling nonprofit innovators and social entrepreneurs. Tailwinds is a project that brings momentum to the leaders tackling the world's most impossible problems. Today's episode has three parts. Part one, I'm going to show you how to measure a goal that may seem impossible to measure. Two, I'm going to play pieces of my conversation with a data analyst at one of Los Angeles's largest agencies working to end homelessness for women. Christian is going to tell us that we don't have to do Capital R research in our organizations. We just have to start somewhere. And then part three, I'll outline another case study, which tells us how to measure change in perceptions of a community about people who've been incarcerated. So part one, how to measure a goal that may seem impossible to measure. 

A measurement plan for abstract goals works when you do three things, and I'm gonna go through these in depth, but here they are in short one, we identify a validated framework for measuring similar ideas that already exists in the world. Then we give terms definition according to our experience, and then we give our goals context as to why we chose them. 

So let's start with the first step. Identify a validated framework. A validated framework is a decision researchers have made that has been determined to make a change on a given issue. It's usually an idea that includes steps to take to measure that idea. Sometimes it even includes measurement tools like questionnaires. Researchers have been thinking about how to measure complex ideas and often getting funding to create toolkits for others to use. Topics like economic mobility, workforce readiness, states of political will building early childhood education stages of growth for music students, stages of growth for athletes. These already exist in the world and we just have to find them. I'll give you a little preview of our conversation with Christian by telling you about the validated framework his team chose. So the Downtown Women's Center in Los Angeles works with thousands of women per year experiencing homelessness in LA, and they wanted to measure the impact of their workforce development efforts. The challenge is they had several programs for workforce development, all with different funders asking for different reports. In fact, does this sound familiar? They had built so many methods for collecting so much data for different funders that they could no longer see the larger impact of their work. They had dozens of metrics, but no sense of how well they were generating long-term employability for the women they served. So we did this. 

First, we looked at a validated framework. The good news for the Downtown Women's Center is that there is a framework called Measuring Mobility from Poverty, created by the US Center on Mobility from Poverty, Stanford, and the Urban Institute in 2018. The researchers on this team argue that there are three key ways to exit poverty. One, increase your economic success. Nobody's surprised about that one. Two, increase your power and autonomy, and three, increase your belonging in community. So with the Downtown Women's Center, first we read the detailed paper that these researchers wrote about each of these three factors. Then we considered what they each could mean in the context of the Downtown Women's Center. The result was a measure that we're calling earning power. 

So after you've identified your validated framework, the next step in the plan for measuring elusive outcome goals is to define your terms according to your experience. So we need to take the abstract idea, like earning power in our goal and clarify what we mean by it. So here's a different example. You could say something like 80% of the people we serve will shift from experiencing emergencies to thriving. But you'll need to define experiencing emergencies and thriving. Here's another one. You could say, we will increase our community's commitment to ecological preservation, but you would need to tell us what a commitment to ecological preservation would look like. In other words, you need to set the bar for yourself and clarify that bar for your audience. So back to the Downtown Women's Center. Our next step was to define what we mean by earning power. It isn't defined in the measuring mobility paper because it's a combination of ideas from their research, and you could ask 10 economists and social scientists what they think earning power means, and you'll get 15 different answers. So I encourage the program team and the evaluation team. Yes, they have an evaluation team, and you're gonna get to meet one of them soon. Encourage them to define what earning power is in their context. And we came up with the following list. 

Earning power means that someone has done at least three of the following, increased their income earned an industry recognized credential. Obtained employment, completed 100 hours of job training, received a wellness service, or improved their housing situation. That was their initial list. After a month or so went by and they began collecting data. They realize that this definition of earning power or their list of qualifications might be too simple. They were finding that almost all of the women in their job training program were meeting the full list, but not actually proving that earning power was helping them. So they're considering making the qualifications more rigorous, like obtained employment in a job post-graduation instead of just obtained employment at the Downtown Women's Center or achieved at least four of the six instead of three of the six items. 

So our steps are, choose a concept that researchers tell us is important. A validated framework. Define your terms, and you'll start to see a pathway for measuring your obscure goals. 

Your last step is to articulate how your chosen goals are the right goals for your context, we all need a short explanation of your theory of how these goals fit into the bigger picture. One of the organizations I've worked with, friends of Shelby Park and Bottoms in Nashville, set out a bold goal. They said Shelby Park and Bottoms will become a national model for activating inclusive community support for the care of an urban park. Then they wrote, "this goal makes sense to us because we believe," and then there was a list of things they believed that the park belongs to the citizens. People are more likely to care for a place when they feel a sense of ownership. A park is a unique third space in a community. If our first space is where we live and our second space is where we work or study, a third space is where we go to connect with others. We need more safe third spaces in our community and access to natural places is a commodity, often limited to the most privileged in our society. Thus, the nature gap is a form of societal injustice that parks are uniquely positioned to address. If you read their strategic plan in depth, you would also see a paragraph or five explaining why a national model for activating inclusive care of an urban park is particularly needed and why their neighborhood is a good one to set the pace nationally. 

So feel free to take space to describe how your goals are the right goals based on your experience of the problem. This gives yourselves and your audience confidence in your plan. So those are your steps. One, identify a validated framework for measuring similar ideas. Two, give your terms definition according to your experience. Three, give the goals context as to why you chose them. 

Now I'd like you to hear from someone who recently went through these steps with me on what their experience has been with measuring something that was previously elusive, the impact of their workforce programming. Christian Quijano is the Director of Data and Analytics at the Downtown Women's Center in Los Angeles, an agency in LA that is working to end homelessness for women in LA. They serve thousands of women per year across several programs, which means that Christian manages really complex data systems. I met Christian during his first week on the job. His very large team was gathering to work on identifying outcomes that would measure the impact of their workforce development programming. They'd previously been really good at collecting output level data, like the number of people with jobs, the average hourly wage, retention rates, services provided. But as I said, they had very little sense of how effective that effort was. I, one of the things I noticed about Christian right away was that he was ready for this, almost like he was ready for something rigorous and groundbreaking. And one of the reactions you might have when listening to Christian describe his work is, we can't do that because we don't have a Christian on staff or we don't have a director of data and analytics pipe dream, fair. However, many of the program directors or program supervisors I'm working with are able to think like Christian and set up systems on the front end that are then implemented by their staff. If you wanna hear from one of those program directors, listen to the episode on data collection plans you don't dread. And I've decided to bring you this interview with Christian because in my experience, everything Christian talks about measuring is measurable by a small lean staff.

So, Christian, let's start with the question. Can you paint a picture of something that happens on the front lines at DWC on any given weekday so that we get a sense of what the folks on the front lines are doing? 

Yeah. We're in the heart of Skid Row, and we have our building there at San Pedro Street. And not too far away. We have, our LA Street location and then probably about, 20 minutes away, we have our NoHo site where, we have our other permanent supportive housing that happens, so. Across these three different sites we really need to start with relationships. And sometimes that can start with a meal coming into the day center and having a meal, having a shower, using the restroom, having someone to talk to, a place to watch the news, in a safe environment.

Are there field outreach workers walking along skid row, visiting people? And then there's another team that's working in housing and another team that's working in, employment. Maybe could you talk about the different teams and what their jobs are?


Yeah,So we have our, clinical team, which is quite robust. That actually includes our outreach and engagement team, as well as case management, as well as our mental health services, our DV housing, all of those kinds of things. And then our health and wellness teams really are, providing the day center services, our workforce services, as well as our health services. And then our housing, programs and our housing department, which encompasses both, permanent supportive housing teams as well as, our community-based housing teams, there are different entry points into DWC. One of them is, our outreach and engagement team walking the streets of Skid Row. Familiarizing themselves with the residents of Skid Row and letting them know that DWC exists and giving them some insight around what kind of services that they are, really eligible for. And to let folks know that, just down the way a couple blocks away, there's a warm meal and like a friendly face waiting for them.


And the project we worked on together was focused on your employment services. So, I really wanna highlight made. And the beautiful work that you all do in your social enterprise. Can you tell us a little bit about MADE?

Yeah. MADE by DWC. So MADE is our social enterprise that encompasses both our cafe and our different, consumer products. We make candles. We have a boutique, which by the way, if you have not gone to the boutique, highly recommend, open invite. Definitely come the last Fridays. The deals are outta this world. The social enterprise is really a way for the community at large to interact with the spirit, the brand, mission of Downtown Women's Center in a way that I think highlights, the women who we have an opportunity to serve, right? They are at the forefront of making the candles of serving in the cafe.

You just gave me an idea for another outcome we could measure in the future, which you probably already thought of when you. It gives the community a chance to experience. It made me think, perceptions of homelessness shift as a result of your work and measuring the community's perceptions of the story of Skid Row and how negative it was before they interacted with you all. That could be an outcome.

That is, that is really really compelling. Um, and, so much of the work that we also do at Downtown Women's Center is around advocacy. And that advocacy really, helps to shape public perception, around the issue of homelessness, particularly for women and also helps our policy makers to understand the real world stories and experiences. One thing that I think we're most proud of is our Women's Voices Board. And this is our board of, participants and residents who have gone through our services or are currently in our services. So why not,

Yeah. Okay. Tell us a little about the complexity of measuring things at Downtown Women's Center.

Yeah. This is a great question. We are in the people business, we get to steward this beautiful mission of, ending homelessness for women, in Skid Row and in Los Angeles. And to do that, requires a pretty complex understanding of human behavior, social psychology, systemic racism, generational poverty, different factors that relate to, people's wellbeing. And so when we are talking about these things, it's. It's much more qualitative, in nature. And so we're not measuring, number of, things sold, although in the social enterprise we do, we're also tasked with how do we take something that is qualitative in nature and structure in a way that can be easily analyzed, that's the task at hand.

I think the other thing that's complex about your particular job that I wanna point out is the variety of programs that you do have at Downtown Women's Center and how, if someone asked how effective is DWC at ending homelessness? You would want to say something like, well, it depends, but we wanna be able to answer that question without saying, it depends. So when you have housing, when you have outreach, when you have mental health, when you have employment, when you have all the complexity of programs, The easier question would be, how effective is housing? How effective is employment? How effective is this? But to aggregate it all is a hard task.

I think internally, within organizations, we are often, we're thinking about the success of our program individually. And sometimes it's a little bit challenging to see, if we're a workforce team and we work with our mental health team and with our housing case managers, that. That whole team, is coming alongside a participant to improve the overall livelihood of that person. And how do we measure that? Is, that's a complex task that, takes a methodical approach to get there, right? We need to infrastructure, from like a database systems place, we need to make sure that what we are like pulling together is reliable, that we're all agreeing on definitions on where things are coming from. And then once we can do that, to be able to then connect those dots, then becomes a fun statistics project or task, um, and we are working on that. That's definitely like the vision here. Okay.


The hypothesis would be the more programs somebody interfaces with, the better off their outcomes would be. So if they get services from two programs, they're less likely to be homeless three even less likely. And so sometimes I think in our work, we set ourselves up with data projects that are actually just validating ourselves versus really giving us any data that improves our work. I guess that's just a comment. I observe that frequently.

And for you? Like when you interact with orgs? How do you help folks get out of that place?

Well, I think you're in a more advanced status unlike most orgs, I would say collecting data in the first place is a win for them. So even if we're just proving that we're good at our jobs, I still think that's helpful for an org that has had no proof yet in our situation with this. I think you're already asking that question. Is our system setting us up just to generate proof? We're doing good work and how can we be more rigorous in our evaluation? That's level two. I think I'm happy when an org just starts asking a question and then measuring, and then level two is making sure that you're measuring in a way that, gives you data you can learn from.

Yeah, I mean it also starts maybe with even the org culture. Is there, a culture of continuous learning and improvement embedded I think when there is, these kinds of questions are a natural, like next step once we figure out what we're contractually obligated to track.

Mm-hmm. So let's talk about earning power for you. So we're talking about your team's decision to measure earning power as a case study for our conversation. Why did that particular measure land with you, Christian?

Well, fortunately, I got a chance to work with you, Hillary, and, and because of that, you put us onto some really great research, from the Urban Institute that was really looking at how do we measure economic mobility out of poverty? And knowing that so many of the participants and residents that come through our doors are navigating, homelessness, which is. With structural racism, with generational poverty, with, underinvestment in community in general. So all of these factors play a role. we needed to have a framework that was holistic enough, to hold all of those pieces that I just mentioned. And really that framework is looking at how do we increase earning power? How do we, improve the sort of sense of belonging and community for people, and then also improve their sense of power and autonomy. So those three factors is what that framework from the Urban Institute kind of highlights. And thank you again for putting us onto that, helped us to think about things more holistically.

Yeah. And I should tell folks that, we defined earning power in a way that made sense to the Downtown Women's Center that's I think where the crux of our conversation. About is this robust enough comes into play? Because when I encouraged you to measure earning power, I said, don't worry. You're gonna be able to make a list of things that constitute earning power that in your experience, relate to someone being able to earn more and more money after they're done working with you. And that list is usually different for every organization I work with. Um, and so you all came up with a list and then, a few months later you said, you know, maybe we need to evolve this list. Like maybe this list isn't as robust as we want it to be because, we made a pretty slam dunk list. Originally we made a a list of factors that almost everybody is going to achieve.

Yeah. you know, if I can use a basketball reference here it is, NBA season to feel like it's okay too.

Okay. Go for it.

we were essentially like, we, what we found is maybe we're putting people like. Uh, right underneath the basket.

That's right.

Christian Quijano: 24:48
Um, and room to say, let's go to the free throw line here. But what I would say is, what we took a look at is increasing income from when you started the program to any kind of in change in income throughout the program, and then what you ended with. You know, did we, did we increase your income? That's one of those things. And then in addition to that, did you earn some sort of industry recognized credential? Seeing that kind of gives that person the ability to that credential and now be more competitive in the workplace. And then also we wanted to know, did we help you get employment? Did you obtain employment? By way of working with us. then in that employment, were you able to complete a hundred hours of work? But then we also wanted to know to more of the wraparound model of, services for our population we wanted to know if folks received a wellness service. we define that pretty holistically everything from mental health services to, did we, did you need transportation to, a job interview? Um, did you, speak with a case manager to get whatever resources you need. Then did you improve your housing situation? So from when you started in, tracking your housing status and what that was, and then, throughout the course of your time in the program, did your housing situation improve? these six, Different items are what we looked at to see whether or not was a change in, a person's earning power. Uh, and, and how we defined that change was whether or not you achieved three out of the six, different items in that list.

Yeah. Yeah, because the theory is that that list is the most effective way at getting somebody. The power to earn more money. Those are stabilizing factors and those are accelerants for people in the economy. Now, this goes back to our bigger question, is it okay for an evaluation team to just define their own terms and then go ahead and measure against their terms? Remembering that we did borrow this from a validated framework. The validated framework didn't give us a list. The validated framework said earning power or economic growth. They use different terms, is. Essential. And we're like, yes. Um, but it, I, I gave you all the freedom to say, okay, you define it. So what do you think about that notion of the freedom to define your terms in the evaluation world?

It's a great question. I mean, I think that we can self define in all these different areas when it comes to evaluating our program's effectiveness, I think that can and should be, self-defined with, with the layer of, with the level of really be validated, right? It does not serve us to be able to say, we're doing such a great job. Look, look, we're doing such a great job. I think that we wanna actually take a look at what we're doing well and where we can improve. 'Cause ultimately the reason why we're here is to end homelessness for women. And if our, programs are doing that. That's, that's wonderful. And for whom are they doing that And right to kind of get into the, to the equity conversation. Um, and for whomever that is not happening do we have some insights into why that's not happening? I think the role of data teams and organizations maybe can and should be to help organizations learn more about themselves. And so if we are, if we take that on as like our task, then we need to be honest with what we find when we actually go looking.

Yeah. You said something a second ago about equity and when we were talking about how we define earning power, it made me wanna ask you, do you think that we defined earning power in two white of a way Meaning a white lens would assume that people have access to the powers that help people get ahead. If we remove that white lens, we could realize that not all people have access to powerful relationships. So social capital is access to relationships that help us get ahead.

That's interesting.

One of the things I like about helping a team define their terms before they measure them is that only, you know what your participants are up against. And so if we're talking about earning power, there's so many things in embedded in who gets to earn more powerfully. And so I wonder if one of the things we forgot is, access to. Social capital.

That's huge. Yeah. I I think that, we have yet to sort of map out our community and belonging and the power and autonomy metrics with, um, the same kind of like intention and, that we've put into this. And that's 'cause we needed a starting place.

I think that we can certainly come revisit that and see if maybe that's an overlook on our part. When it comes to whether or not we measured this in a too wide of a lens, I think that's an important question to ask.

Are there other complex things you've tried to measure or evaluate in previous jobs that you figured out? Have you had experience with other hard things to measure and maybe how, how did you even think about those research questions?

Yeah, I would say in previous roles, the topic of belonging, was, is near and dear to my heart. In that setting we are working with, young people who are experiencing homelessness and that experience, particularly if you're like a transition age young person, there's already so many mental milestones that are happening around that time. And to not have a supportive base, whether it's like a biological family or a chosen family or a safe place to, to be. Those things are huge factors in one's overall sense of wellbeing and health. So, the place that I was working for did just such an incredible job of providing a sense of community for those young people. And that org is called My Friend's Place and. Essentially what we wanted to know is like with all of these investments in community, of this really creating a deeper sense of belonging for young people? And so that was the question that we started with. And. We looked at some validated scales that kind of been used in educational settings and we adapted it for our environment. And, then surveyed the young people to ask do they feel a greater sense of belonging? Um, and if so, what are the things that might be associated with that? So yeah, what we found essentially was that, higher belonging scores had to do with greater involvement in programming. And that intuitively makes sense. So the healing nature of that, I think, and the power of community is like something that's really important to me.

How did you find the questionnaire?

Yeah, I was working with a solid team, who was doing research around, different scales that have been used, in different settings. And then from there, we took a look at the questions These kind of make sense for our space, or this doesn't necessarily, you know, entirely, but we can adapt it.
And, and you felt comfortable with the adapting? Adapting the questions was fine.

For, from the perspective of it being exploratory in nature, like I think that, um, so much of the barrier doing kind of this evaluation, work within organizations themselves, is this, maybe this concern that there, there needs to be like capital R research and, I wonder if Maybe we need to rethink that a little bit. For a lot of reasons. Mm-hmm.

It keeps organizations from doing much data work at all when they think it needs to be Capital R, winning the Science Fair Research Project. Yeah. I. Yeah. What do you hope DWC measures next?

What we will measure next is engagement across programs and understand what kind of things keep people housed. So housing retention is really important to us. And I would say we have a really good sense of some of the different pieces that really helping people to maintain their housing and retain it. But I'm also interested to know from a data perspective what those things might be.

We're going to end with one more case study of an organization that has a difficult outcome to measure and their plan for measuring it. All Square is a social enterprise in Minneapolis that invests in people impacted by mass incarceration. They're working to change the way criminal records are used and viewed. They do that through a social enterprise cafe as well as a program that gives incarcerated and formerly incarcerated people access to paralegal and JD degrees. One of their outcome goals is this, by 2027, we will increase proximity between individuals impacted by incarceration and those untouched by the legal system, resulting in 60% of stakeholders increasing positive perceptions. So All Square has three options for measuring this. 

One. They could use a research agency that already has an interest in this research question and would foot the bill. I found one that is already doing research on Minnesotan's belief about people impacted by the justice system. 

Two, they could conduct key informant interviews every quarter. Key informants would be asked to answer, how true are the following statements according to most people in your community, things like people in prison are unsophisticated. Most people in prison committed crimes because they have little or no self-control. I would be willing to date a person who had been in prison. They would ask the questions every quarter to see how they change over time. The aim would be for 10 completed surveys every quarter from a variety of perspectives. A church leader from a conservative congregation, a church leader from a liberal congregation, nonprofit, community leaders, local politicians on both sides of the aisle. Employers, these are considered key informants because they're able to inform on the perspectives of a larger group of people who you don't have access to. And by the way, the questions on the survey would come from a validated framework that researchers have already put together. 

Third option, they could use their customer review data, meaning the Google and Yelp reviews written by customers who visited their restaurant. This works at All Square because the employees working in the restaurant are formerly incarcerated individuals. So we'd be interested in how the public reviews their service, knowing that about them. If they went this route, they could take a baseline of Yelp or Google reviews, count how many reviews comment on the workers themselves, count how many are positive and how many are negative. They would conduct this analysis quarterly to track trends toward more positive comments related to the workers themselves. This is called a content analysis, by the way. The downside to this option is that it's skewed toward people who are already likely to have positive attitudes about your population. Because they're coming into your restaurant, but it is data waiting to be analyzed so you can see that it's possible to measure concepts that seem really complicated when you just define them, but nobody on your team will be interested in putting in the work on the measurement if the goal isn't compelling to them. 

Goals have to be questions you're asking yourselves about your work before they are stories you tell. They should be questions you ask. Christian alluded to this being the job of a data team to teach us more about ourselves. Are we changing negative perceptions effectively? Do we have the intervention necessary to change most people's perceptions, or are we helping women survive on the streets, or are we able to make them powerful enough to earn their own money for years to come? Everything else is tally marks on the wall. How many people did we serve? How many workshops did we offer? Those are compelling if we mark your success by your churn rate, by how busy you are. When you're asking questions that explore how effective you are, that's a question worthy of a goal. The best questions result in goals that require some engineering on the backend in order to measure them, and now we know how to do that. 

Tailwinds is a production of Flying Whale Strategies, a consulting firm that is equipping teams to solve impossible problems. I'd like to thank Christian Quijano and the Downtown Women's Center for their willingness to share their experience with us. If you'd like to learn more about DWC and shop at their gorgeous boutique online, please visit downtownwomenscenter.org. I'd also like to thank All Square in Minneapolis for permission to share their evaluation plan, please check out www.allsquarempls.com to learn about their work in changing the way criminal records are used and viewed. And if you'd like to learn more about Flying Whale Strategies, please visit our website at flyingwhalestrategies.com. Thanks for listening.