What If nonprofits had product designers?

The for-profit world has product engineers, design sprints, and rapid iteration cycles. The nonprofit sector has… a debrief meeting. In this episode, Hillary asks what would happen if we closed that gap.

Hillary breaks down what a "product" actually looks like in the social sector and introduces a way of testing it borrowed straight from tech startups. Then she sits down with Matteo Moore, a product designer at one of the world's largest tech companies, for a conversation that is equal parts art studio, tinkerer's lab, and science research center — Matteo's words.

You'll hear them talk about:

• The awkward robot that delivers toothbrushes to hotel rooms — and what it taught a startup about observation
• How experimentation can actually speed things up rather than slow you down
• The Effort Impact Scale — a dead simple quadrant for deciding what to fix next
• What the scientific method you learned in middle school has to do with your next program cycle

Guest: Matteo Moore is a Product Designer with nearly two decades of experience designing products that people actually want to use. His career spans startups, media, education, and enterprise. He has scaled design sprint methodologies across large organizations and spent years mentoring designers at every level.

Matteo's origin story is not a typical one for tech. He holds a Bachelor of Arts in Fine Arts and English Literature from the University of Colorado Boulder, a Master of Architecture from the University of Colorado Denver, and studied architecture abroad at Sapienza Università di Roma. That cross-disciplinary foundation — art, language, space, and systems — shapes the way he thinks about every product he touches.

He describes his workspace as one part artist studio, one part tinkerer's lab, and one part science research center. That combination is exactly what makes him such a compelling voice on design: he brings the curiosity of an artist, the rigor of a researcher, and the humility of someone who has learned not to hold his work too precious.

Mentioned: 

Michael Seibel, How to build product as a small startup.

Knapp, Jake, et al. Sprint. Bantam Press, 2016.

Transcript

My name is Hillary Frances, and one of the things that I'm thinking about is the fact that the org charts of growing companies often include data scientists, product managers, product engineers, and project managers. Yet, I have very rarely seen these job titles in social sector organizations. Growing companies hire these positions to consistently test and improve their product. Their product is the subject of their obsession. Nonprofit teams, on the other hand, are not working on product improvement. They might be working on delivering a product, but very rarely analyzing and iterating on it. Why? Their funding is predicated on the impact of their work. This drives one to be obsessed with looking for proof that the product is working, not looking at the product itself. We know this because when things break or don't have the results we were looking for, we in the nonprofit sector continue to offer that product with slight modifications. We don't have a clear redesign process. Can we start thinking about product design in the social sector without it getting too slow or expensive? That's today's episode. 

You're listening to Tailwinds, Ideas Fueling Nonprofit Innovators and Social Entrepreneurs. Tailwinds is a project that brings momentum to the leaders tackling the world's most impossible problems. Today's episode has two parts. Part one, I'm going to share what I'm learning from the tech sector about product design that could be applied to the social sector. I'll help you identify what your product might be, first of all, and then how you could test it. Then we'll hear from a product designer, Matteo Moore, who works at one of our world's largest tech companies and lets us in behind the scenes on how his team does design sprints on products and fixes them. He also understands our world in the social sector, so he helps us consider how product design might work for us. 

So part one, what is your product? In the social impact sector, our product is one of the following. One, it could be a program we deliver A job training program for people coming out of incarceration, a STEM lab for sixth graders, a program we deliver. It could also be a service we provide, consultation to police departments on de-escalation, in-home tutoring for language learners. It could also be a subscription we offer. Monthly giving circles. Monthly giving circles are a sort of subscription. Charity:Water calls their monthly giving product The Spring, and they market it, test it, evolve it like a product. Lastly, an actual product we sell in our social enterprise: candles, coffee, silk-screened bags. In any of these cases, it doesn't matter if you make money or give the thing away. What matters is that you treat the effort like a product that is bought and sold. When we think of our social impact this way, we can constantly improve it. In order to improve it, though, we need to learn how to test it. That's where our mentors in the for-profit world come in. 

Let's look at how some of the most impactful for-profit companies are testing their products. So we know that they're choosing a metric that represents the value of their product. Their favorite is called product market fit. Is their product fitting into the market in a valuable way? They have to choose a metric that represents this value, and it's often a measure of usage. Airbnb's usage metric is bookings or stays. Lyft looks at the number of riders for usage rather than rides taken. Instagram looks at the number of active users versus the number of posts. Gusto looks at how many people are running payroll in their system rather than the number of annual subscriptions. So they're very careful about this. Don't worry so much about why they chose that specific metric. Just notice that they've chosen one metric that represents the value of their product, and it's often usage. 

What would that look like for us in the social sector? You could look at program applicants, program referrals, the number of graduates, your retention rate, your turnover rate, your donor meeting acceptance rate. Or your ability to meet specific outcome goals. I think ultimately any social impact organization should have outcome goals that measure how effectively they are meeting their mission. But what we're learning from the for-profit world is to choose a more upstream metric that you think represents the value of your product or its fit in the market. And use that for your design sprints. So if we were a job program, that metric might be how many employers sign up to receive candidates from us. If we were consulting with police departments, it might be our referral rate. How many departments refer us to another? If we want to test the value of our monthly giving product, we might measure the retention rate of our donors year over year. If we want to test the value of our candles in the market, we might look at the number of wholesale customers. These are measures of how valuable our product is in the market. Once we know what we're testing, then we start to test it. 

We're going to hear more about this from Matteo, but when tech startups go through what they call a product cycle or a design sprint, they do the following. I haven't seen this in person, but I can only imagine how it looks. A room full of geniuses with ADHD and coffee cups and fidget spinners leaning back in their chairs or throwing bouncy balls at the wall. Here's what they're doing in a design sprint. First, they determine the product development cycle length. Two weeks, a month, three days. Two, they determine who is in charge of product development, getting it out the door. Three, they establish KPIs. What metric are they testing? Like Uber's might be the number of riders. Four, they create a research question or hypothesis for this product test cycle based on the single KPI they're working on. I'm assuming this would be something like, how does the number of riders change when we make a change to the app's interface in a certain way? Five, they host a brainstorming session. This is the only formal meeting they have. Brainstorming focuses on new features, bugs, things that prevent the KPI, tests to run. Six, everyone gets a turn sharing ideas of what would move the KPI. Seven, every single idea is written on the board. No idea is debated, but you are allowed to ask clarifying questions. Eight, the whiteboard is full of things to test. Nine, a technical person ranks each idea in terms of easy, medium, and hard to do. Ten, everyone looks at all the ideas on the board and their ranking. Eleven, the team is asked to pick the hards first and writes up exactly what they will build on this one idea and who will do what. Twelve, a project management system is created. Thirteen, the meeting is over. Fourteen, everyone gets quiet and goes to work. Fifteen, the manager doesn't change anything anyone is working on during the development cycle. They know there will be another cycle in two weeks, so they feel comfortable keeping everyone focused. That's important. Sixteen, everyone on the team tests everything on the test list, everything that made it through to the final list. Seventeen, everyone writes down bugs they find. Eighteen, all the bugs are addressed. 

Imagine if we did something like this for our work. I think the easiest parallel is debriefing a pilot project. Let's say we piloted a new method for training political activists. First, we determine the KPI we're interested in. Remember, it needs to be something that represents value, like product market fit. So for them, it could be the number of activists who signed up for a second round of training. It could be the number of people they reached. It could be the number of arguments they diffused. Let's say they choose repeat sign-ups. So then they have a brainstorming session to determine what they want to tinker with in the next program to see if it increases repeat sign-ups. They all have a hypothesis as to what might increase repeat sign-ups. They write them on a board. They rank them in terms of hard, medium, or easy to do. They pick a hard one, and they work on it. It might be that they need to introduce an element of peer-to-peer mentorship into the training. So they do that in the next round of work. Then they measure the rate of repeat sign-ups after making that single change. They meet again to debrief. Another version of this is to check in on your organization's outcome metrics on a regular basis and ask critical questions about their current rate. Why are we hitting our goals? Why are we not hitting our goals? What are we learning that we wanna bring into the next year's work? This is a more common meeting that you've probably had. The difference between this meeting and the experiment meeting is that in an experiment meeting or a design sprint, you are isolating a particular element of your product that you want to improve, creating a hypothesis about how to improve it, and testing that hypothesis. Design sprints allow you to be more fine-tuned. Debrief meetings are often broad and theoretical. 

Now, I'm going to play pieces of my conversation with my friend Matteo Moore. Matteo is a product designer for a very large and well-known tech company that would require him to go through several layers of approval in order to conduct this interview, so we're going to let you use your imagination about where he works. What matters more is that he has decades of experience as a designer of products that people use. I wanted to ask someone outside of the social sector to tell us about the process they use to design things. It's also a bonus that Matteo has the unique ability to translate tech or product design to those of us working in the social sector. Matteo has degrees in fine arts and architecture and has held titles such as user experience designer, senior product designer, staff product designer, and design lead. Matteo and I have been friends for many years. He helped me design a custom envelope for my wedding invitations that required him to mock them up on his whiteboard, use several types of rulers, software, and ultimately scratch paper. He and I love to discuss things, so just for you, I cut our lengthy conversation down to the bare bones. Our friends often leave the dinner table when the two of us get talking. Any topic from the ingredients we mixed into our garden soil this year to the excellent customer service experience he had calling the DisneyWorld hotline. But today, you're listening in to a moment where Matteo shares the intricate way he thinks about the process of designing products people will use. And as you listen, try not to worry about how you might do things differently because it might feel overwhelming, but rather how you might adopt Matteo's approach to looking at a product in the first place.

So if we were to do this interview in a location that would represent your work. What room would we be in or what would be going on in the background?

Good question. in order to kind of properly present the work. I, I think I'd set the scene as like, uh, a, a third artist studio, a third tinkerers lab, and like maybe a third science research center. And there'd be, uh, sort of some sort of mashup of those sort of three worlds happening behind me with lots of people kind of, you know, studying things and looking at things. And it's a little, uh, Willy Wonka, but it's the sort of right scene set for, what I do.

I can completely see why you would need it to be like Willy Wonka. Okay. Well that leads us well to the question I have about experimentation in your work. And one of the things I'm thinking about lately is how to help organizations build a culture where the people working there look forward to experimenting. They look forward to the idea of experimentation. this is not. Common in the nonprofit sector. People do not look forward to experimentation. When their boss says, okay, we're gonna run an experiment on something, they think I don't have time for that. But I think that people are motivated by the idea of mastering their craft. And I've read, some of Dan Pink who says that we're motivated by autonomy, mastery, and purpose. So I would think that introducing experimentation to a team would be good for motivation, but. What have you experienced in your various workplaces that gets people to be up for the idea of experimentation, design cycles, testing, learning.

Yeah. I think embedded in the word of experimentation is learning and I think we, Many of us have that sort of history of learning in school or learning a new, skill through a hobby or something. It's sort of innately human to sort of experiment or learn. and so I think there is a fundamental foundation there to build off of. Now in tech world, experimentation I think is like part of the culture and so it often doesn't take much to get people motivated to experiment. 'Cause again, it's built into the culture to built into the processes. However, there are still, times when that, tension of speed, moving fast, is a priority. And so, Some of the conversations around getting teams to. Into the process has to do with, the conversations, upstream. So that would be with, either directors or VPs, to buy into this way of thinking or the value that it brings by experimenting and often it has to do with showing them that. If you took a step back, experimenting, and iterating quickly, actually allows you to things up by actually shipping something that,, users, and people are going to like, enjoy, find value in rather than. Delivering something, going through the entire, product lifecycle. And then realizing once it's in market that actually this thing that we took the last three or four months to design and build and launch actually didn't meet the need that we wanted to. So often you can show that sort of, Not shortcut, but that aligning first on what you're building and designing, will meet the needs of users first is quicker than just getting something out there.

So you are an artist at heart. I know this because when we first met, I was introduced to you via your art studio. You weren't there but one of your roommates Let me in. I don't know. I remember walking, it was a garage and I remember walking in and being amazed at your art studio. I think at the time you were working as an architect, but you had all these art projects going on, which were very thoughtful and curious to me. As an artist though, how do you not get attached? To your work with this rapid iteration going on in your job?

I mean, good question. I think that over the many years that I have been a designer, even though like. My origin story is sort of like in the art world with, both, degrees but also personality and approach. think both 'cause they're very distinct artists and designer. Those are two very different, roles. With a designer, iteration is sort of like you live and breathe that every day. And to not hold it, precious, I think has been very comfortable because at the end of the day, the difference between artists and designer is designers filling a role to value for someone else, that they will use this thing that they will find has value. And so that is the ultimate, vision and creation. Whereas, artistry is something that is like. Much more of an individual thing. And so there's been an ease in, in sort of not holding things precious.

That's beautiful. As you know, this episode, we're experimenting with the idea of product design. Nonprofits, and this is a new concept. So could you explain what a product designer does for a tech company?

Yeah. Yeah. Yeah, Well, as always, there's nuances, based on, the team, the role, the company, seniority. But generally speaking, a product designer, in the tech space a cross-functional role that focuses on an alignment of three things, and that's, viability, desirability, and feasibility. that sort of, those three you can imagine as a Venn diagram overlay. That is a, a, a Venn diagram that was, produced by Famous Design Company, IDEO. And it speaks to. The key focus areas of a product designer. One is desirability. Do users actually want this thing? Feasibility? Is this thing, does the technology support this solution? And, and viability, does this align with business goals? Or what, is happening in the market? Those focus areas have to do with, at a product designer level, craft, collaboration, usability and often, even things like, accessibility, privacy, safety, those are kind of considerations through the the product lifecycle for a designer to get something from idea to, to shift. So there were a couple of designers, Jake Knapp and John Zeratsky. from Google who were trying to, I, the design process accessible to a very wide group of people who were interested in using, the design process in various ways outside of technology. They moved to an arm of Google called Google Ventures that was working with a lot of startups. And they refined that process and what they do is they have boiled it down to five days and each day kind of has a theme the goal of the week is to come up with a prototype to test or experiment with actual people, users, customers, to then collect more data to refine the solution. Because a lot of the times when we experiment, we don't necessarily know what the outcome's gonna be.

So, prototyping is a great way, it's a great vehicle for understanding and refining the solution at the end and so the way they've got the the week mapped out, is, I think a, a good way to, I have the book and I do have the...

I can't believe I haven't read this book. I feel embarrassed.

Uh, yeah, you should absolutely read this book.

Can I ask you about the part of the process where a team has done a pilot, they have a prototype, let's say it's a, afterschool program that they invented in May, and then they, did one round between June and July. And so now it's end of July and they want to really learn how it went, and then they wanna do the fall differently. So, what does the conversation need to entail in order to go into August with a new iteration.

I think. The pilot and the next iteration, we, in the tech world would say like maybe that pilot, another word would be like an alpha release or a beta release where it's delivered to sort of a small subgroup with the purposes of learning, in order to refine. So in, in that, specific sense, the design process is so flexible that like, it doesn't have to be technology or screens or, you know, apps that we think of. You know, I've heard stories of the medical industry, redesigning the experience for kids, going through CAT scans, you know, to make the experience for the kids less frightening. And also what that meant is that, the hospital now we're kind of talking about that viability part again. Was able to have more patients go through the CAT scan because they weren't having to restart the scans over and over again because the kids were getting scared. And so they were able to design the experience to be very kid friendly. Which again, sort of met those two needs of both business goals, for the hospital, but for. The people actually going through the process itself. So in that example, you gave, I think the critical piece to all of this is understanding user behavior and, the investment in learning. So without that, without the observation part of, you know, if you think about the scientific methods, sort of that we learned in middle school or whatever, right? It starts with sort of the observation stage at the beginning. And so having run a pilot, uh, is good, but without observation of what happened in that pilot or without measuring it in some way. There would be no second iteration because then you would just be sort of guessing again. And so sort of that critical investment of knowing we, as an organization will be, putting resources towards observing, being in the room, specific examples in this sense. 

Another, another good story is from the Sprint book actually, where they were designing, in hotels, I'm thinking more like outside of technology space. In a hotel they were testing these little robots. Maybe you've even seen them that would deliver like toothbrushes to your room. And this company, the startup had lots of concerns about it being awkward or hotel guests interact with this thing that we're not used to interacting with. 'cause we're used to having a person come up or answer the call. and so they did basically a, a prototype. They mocked up a room, they had a guest, stay there. And they had, of course everyone knew what was going on, but used, ways of, of observing the interactions, like, between the robot and the person to understand, is there like an awkward handoff if someone called downstairs to order a toothbrush and uh, maybe there was a researcher in the room that was taking notes to as the person was interacting with this robot. And so, so knowing how to observe, how to measure it is key, right? Are we gonna be in the room? Watching this pilot take place, are we going to set up some sort of recording system that, you know, the people going through it know about, but like also need to interact with this product or solution? So that the people that are invested in the ultimate solution. Can be there observing, can be there taking, notes, drawing, insight and conclusions from watching this. It, that, that particular piece that's important.

That particular piece seems like the missing X factor for us in the social sector because, observation is terrifying. It is personal. The stakes are really high. So I wonder if, part of the reason we don't do it as much is that the design cycle is the people delivering the service and it just is so sticky and personal and. Whereas in the tech world, the folks that are experimenting and learning might be removed enough where it's easy to let go of something and move on. Or you don't feel embarrassed about your performance that everyone's observing? I guess I'm just trying to think of this the observation I don't see very often because it's about reflecting on my performance as a worker or having my boss observe me do my job.

Yeah. It's interesting that there really. It necessitates a spirit that goes beyond the workplace, that, that everyone is comfortable and invested in sort of the ultimate that meets needs. And if people are aligned on that, not to say that there wouldn't necessarily be awkwardness, in, in the process. There always is. I've seen this having facilitated a lot of design sprints that like when I have, I don't know, uh, a data scientist or an engineer come into the design process for the first time in a inside a design sprint, takes a lot of encouragement as the facilitator to, Get past some of those initial hurdles, but once they sort of see the recipe of how the design process works, I, I think you sort of see the light go off um, especially as a, in a group setting, it can help build a stronger team because it is, something that you go through together. And sometimes, Those processes look like, uh very Impromptu, right? Like, okay, we're just gonna, you know, put a, put a camera here, or, oh, we're going to sit in the room and observe, silently from behind.

You know, it's very ad hoc and sometimes, Ways of getting closer to the user, customer person just means, looking at dashboards of, of analytic, that, that user behavior.

Let's say that we're in the step after the pilot, after the initial experiment, we're sitting down and we're, we have a whiteboard and we're writing down what we observed about our experiment. Let's say there's a long list. We're really honest with ourselves. We're not afraid of being embarrassed or our ego, and so there's a long list of things that we want to improve. How do you prioritize them or how many improvements do you choose per design cycle?

Good, good, good question around, the end of the cycle. So I, I think I would generally call that sort of the analysis and conclusion phases. Again, using the scientific method. That would be sort of the end of that phase. I'll give you a few different examples, in one project that I worked on, we used the design sprint methodology to, come up with a prototype in a week, put it in front of users, monitored those users. Uh, they came in and we did interviews with them, we took notes. As they were using the product itself. And this was like a, again, a very manual thing. Everybody, I think I had probably seven or eight different people that had been with us, during, throughout the week that, helped create the prototype and everybody watched five users use, prototype we had made. They had, each one had a red. Post-it stack and a green posted stack. If they saw something that was negative, they'd put it on the red one. They saw something positive, they put it on the green one. And then after we did all the interviews, everyone puts those sticky notes on the wall. We organize them as a group and what you can kind of see as a pattern that emerges these kind of things didn't go so well. These things went really well. This was a surprise. That's kind of helps organize the observation, sort of the analysis. Then what I will do after, let's say in this example after the design sprint was over, I would create a document that sort of went into, a deeper dive of analysis of looking at those and organizing those, findings that will usually entail, something called an effort impact scale. So if you imagine an XY axis where you have four quadrants, on, on the X axis, you have impact on the y axis, you have effort and you, can take any one of those. Findings insights, or maybe it was a new idea that one of the people that came in to use the product suggested. And you can kind of say, okay, how much impact

And you kind of go on one axis, okay, well how hard is this. On the other axis and what happens is you start to see, okay, these kind of things end up in the top left quadrant. These kind of things in the top right. And then you can say, okay, for the things that are like high impact, low effort, we should just

Do 'em. Do 'em.

you know, high impact and high effort, we should consider it. And you know, you kind of go through the quadrants there and discuss as a team, these are the things that we should invest in refining for the next step.

That is brilliant.

That next iteration. Could be, Hey, we're just gonna take those ideas that we mapped and just implement them. Or often what you can do is go through another design cycle. Maybe it's a shortened design sprint instead of a full week, maybe it's two or three days where you take those ideas, you put those back into a prototype, you retest and kind of see, did we, did we move the needle in terms of, the. Areas where people were getting hung up or were confused, did we allow them to finish X, Y, or z action. And that helps sort of drive that process.

This is, so, this is exactly it. The way to translate. Scientific method to the nonprofit sector. I guess What I'm learning and taking away is that the resistance we've had is our personal, connection to the work itself. And when you can remove that and think more like a scientist, there's endless possibilities.

There's a lot of freedom in that because when you move into sort of that. Type of space as more of a scientist, you almost been granted freedom from what the results are but this sort of iterative nature of creation and, and designing allows for that pathway to move forward, without. Being hung up on what the results necessarily are and then you can kind of track and map improvements over time. It gets much more scientific. Um. As you move into scaled, scaled products, because in my world, often if we're starting a project, we might do something like a design sprint, which helps drive alignment. But when we're launching products. To say millions of customers, um, even at a sort of small, you know, beta release, that analysis is done, using statistical analysis to understand, the impact of that one change. So let's just through out an example. An example for us might be, Hey, we're going to move that particular CTA, or CTA is just a fancy word for a button from, the top of the page to the, to the bottom of the page.

Okay.

And we wanna know by moving it from the top to the bottom, are, are we increasing? I don't know. Are we increasing purchases? So we will launch an experiment. where you have to imagine 50% are getting, the version at the top and 50% of the people are getting the version at the bottom, and then we can kind of compare the two and comparing those two will tell us, With statistical confidence, did moving the button from the top of the page to the bottom of the page, drive more purchases? We'll talk about things like, confidence intervals and statistical analysis and test power, throughout that process. So it does get even more scientific, because you have to imagine at scale of billions of

It matters. Mm-hmm.

1% is a huge amount.

Yeah.

Yeah.

Yeah. Okay. So. The, the pressure on these tests is huge. Like you all have the enormity of getting your experiments right. When, when that's the, the stakes

Yeah. And, kinda going back to sort of the culture of experimentation is sort of embedded in these companies, it demands, a sort of an approach that is rigorous of that, uh, output, 'cause of that outcome, to where we must have a strong hypothesis in order to test it. Accurately. one of the problems, at various companies or startups, you know, early, early stage might be that like, oh yeah, I'm, I'm bought into the experimentation idea and they measure everything. But when without a hypothesis measuring everything just creates a lot of noise. 'cause you don't know what to look at. And so, by just starting with a strong hypothesis and being able to measure it, Can actually start to move the needle. And without those two things, I think, you're not set up for success.

So my charge to you, treat your programs or services like a product. Try it. Promote someone into the role of product engineer. Choose a product that represents your best work, something you want to master. Identify a KPI that measures the value of this product. Identify an assumption you are making about this product. Test a different assumption. Repeat. Because without this shift in mindset, the social impact sector will stay trapped in a funding model built on caution, charitable gifts given in measured doses, rewarding stability over discovery. Meanwhile, the for-profit world will continue to be fueled by investors who understand that rapid iteration is what produces the world's most transformative breakthroughs, and who are willing to place bold bets to make that happen. 

Tailwinds is a production of Flying Whale Strategies, a consulting firm that is equipping teams to solve impossible problems. A special thank you to Matteo for sharing your experience with us. I'm sorry I cut out the part where we talk about playing cards, why people don't tie more things to the roofs of their cars, how you're redecorating your stairs, and my recent experience with a painting class at the museum. We just ran out of time. If you'd like to learn more about Flying Whale Strategies, please visit our website at flyingwhalestrategies.com. Thanks for listening.