Integrity Matters
We work on complex and challenging issues around the globe.
Integrity Matters
Beyond BASIC Episode 4: Behind the evaluation: Relationships, transparency and trust
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
For donors, implementers and researchers alike, the BASIC evaluation experience suggests that credible evidence emerges not only from distance and impartiality, but also from trusted, transparent engagement.
Dive into why the relationship between evaluators and evaluands matters so much in the fourth and final episode of Beyond BASIC, a podcast mini-series sharing learning and innovation from Integrity’s endline evaluation of the UK Foreign, Commonwealth and Development Office’s £23.5m Better Assistance in Crises (BASIC) programme.
Hosted by Niki Wood, Integrity’s Head of Monitoring, Evaluation and Learning, Episode 4: Behind the evaluation: Relationships, transparency and trust explores how evaluations balance independence with collaboration.
Niki is joined by two guests: Integrity MEL expert, Ed Small and Consulting and Research Team Leader Cathy Shaw from DAI, one of the organisations involved in implementing BASIC
Hello and welcome to Beyond Basic, a podcast series showcasing learning and innovation from evaluating the UK Foreign Commonwealth and Development Office's £23.5 million better assistance in crises, also known as BASIC program. My name is Nikki Wood, and I'm the head of monitoring, evaluation and learning, or MEL for short, at Integrity. I will be your host for this podcast miniseries where we will be talking about integrity's endline evaluation of the BASIC program, exploring the methods used, diving into some of the key findings, and reflecting on the relationship between the evaluators and the program team. I'm joined today by two fantastic guests who bring different perspectives on the program and its evaluation. Could I ask you to introduce yourselves?
SPEAKER_02Hi everyone, I'm Kathy Shaw from DAI, one of the organizations that implemented BASIC. I head up DAI's consulting and research team, and I was program director for STAR, the technical assistance component of BASIC.
SPEAKER_00Hi everyone, I'm Ed Small. I'm a MEL specialist at integrity. I've been designing and operationalizing MEL systems for humanitarian and development initiatives for more than 10 years. And I was an evaluator on the BASIC endline.
SPEAKER_01Thank you both for joining. Ed, to set the scene, could we briefly remind listeners what BASIC set out to do?
SPEAKER_00Sure. BASIC was a centrally managed FCDO program running from 2018 to 2026 with the goal of strengthening the use of social protection approaches in crisis settings, particularly in fragile and conflict-affected contexts. It delivered this through three main work streams: technical assistance, research, and knowledge management and learning. Together, these aim to strengthen systems, build evidence, and improve coordination so that people affected by crises can better meet their basic needs and cope with shocks.
SPEAKER_01Particularly for complex programs like BASIC, and how evaluations have to balance independence with collaboration. Ed, could we start with your thoughts?
SPEAKER_00Of course. One of the defining challenges of evaluating BASIC was that much of its impact wasn't immediately visible. Success often looked like better decisions being taken, stronger relationships between actors, or shifts in how governments and donors thought about social protection. These changes don't always show up clearly. So understanding basic contribution relied heavily on reviewing documents and secondary data and structured but open conversations with multiple groups about how influence may have happened in practice. There's sometimes an assumption that independence requires evaluators to keep their distance from program teams. But in complex adaptive programs, too much distance can weaken the evaluation because it limits understanding of context, decision making, and causal pathways.
SPEAKER_01I think this is a really important nuance and detail, because independence isn't just about standing apart. It's about being clear and credible in how judgments are reached.
SPEAKER_00I completely agree. For us, independence was less about separation and more about transparency, being clear about roles, evidence, and where evaluative judgment was being applied while maintaining enough trust to support honest reflection.
SPEAKER_01Thank you so much, Ed. This brings us nicely on to talking about how all of this worked in practice. Kathy, a lot of DAI's work under Basic focused on advisory support and capacity strengthening. What was it like being evaluated on that type of work? And how did the evaluators engage with you?
SPEAKER_02It can be challenging because success in advisory work often means something doesn't happen. Poor policy choices avoided, coordination improves or systems function more effectively over time. And those contributions are not always easy to evidence externally, so there can be concern that important aspects of the work won't be fully captured. To help with this, we tried to be collaborative with the evaluators from the outset. We provided evaluators with a lot of our monitoring data to help provide understanding on the range and detail of the demand-driven work delivered. And we shared additional administrative data in response to evaluation needs, like specific data on the cost of assignments by country. In terms of the qualitative work on the evaluation, we gave guidance on which stakeholders would be most relevant to speak with. In some cases, the evaluators led follow-up interviews with our team on more specific issues like value for money. There was also ongoing dialogue throughout the evaluation too, including sharing updates on each of our timelines. So this meant being open with our data, helping the evaluators speak to the right people, and staying in dialogue throughout to support planning.
SPEAKER_01I see. So you engaged the evaluation across a number of different areas. And did that approach support learning rather than just accountability? Yes, it did.
SPEAKER_02It encouraged us to step back and reflect across the program as a whole. In some cases, it helped distinguish where technical assistance was genuinely influencing systems and continue to develop our understanding of how a flexible technical assistance program is placed to contribute to change against the initial expectations from the business case. That kind of reflection is difficult to do without structured opportunities for dialogue.
SPEAKER_01From the evaluators' side, one thing that really stood out was how much the quality of evidence depended on trust and creating a safe environment for discussion. This is particularly important where findings were nuanced or constructive. Ed, could you outline what helped make that work?
SPEAKER_00Yeah. Giving clarity early on was essential. We were explicit about the evaluation's purpose, methods, and limitations, including what the evaluation could reasonably evidence and where judgment would be required. We also recognized that evaluations sit at the intersection of learning and accountability, which can make openness difficult. So we tried to create spaces where discussion focused on understanding and accuracy. Sharing emerging findings was part of that. It allowed us to test interpretations, not to always seek agreement, but to strengthen the evidence base and ensure conclusions were well grounded. We also held follow-up discussions on the draft report to debate the evidence openly. Those conversations helped sharpen the analysis and ensure that different perspectives were properly reflected without compromising independence.
SPEAKER_01Cathy, building on this, what are the things future evaluators should have in mind when working with implementers like yourselves?
SPEAKER_02Well, we've definitely valued those spaces to engage and reflect on the program jointly with the evaluators. Having space to discuss not just what the program is doing, but the how. And in addition, having the opportunities to come together also with FCDO as client has been valuable.
SPEAKER_01Finally, thinking about what happens after the analysis, Ed, how did that way of working shape learning and uptake?
SPEAKER_00Well, because engagement and reflection were built in throughout the evaluation, the final findings didn't feel disconnected from delivery. By the time the report was finalized, there was already some shared understanding of evaluation findings. That made the findings easier to engage with and was more useful for learning. We also tried a few different things to support uptake. During the baseline, we led a workshop focused on the next steps for taking up recommendations. During the midline, we held a webinar with FCDO post teams to share experiences of drawing on social protection approaches during crises and shared more visual summaries of our case study evidence. We also facilitated a workshop with implementers on their own dissemination and uptake plans. It helped that the evaluation team had strong subject matter expertise in social protection. That shared understanding of the policy and delivery context meant uptake could focus on substance rather than explanation. And recommendations felt realistic and relevant as a consequence.
SPEAKER_01I see. Lots of different approaches to supporting uptake, and each with a different use and focus. Kathy, are there certain approaches that you prefer as an implementer when it comes to applying evaluation evidence?
SPEAKER_02Well, we've made a point of considering regularly through our reporting and planning cycles our reflections and responses to prior findings and comments from the evaluation team. This helps keep the conversation with the evaluators connected with our live decision making and delivery.
SPEAKER_01Thank you so much for your reflections here. Before we close this final episode, I would like to ask for one more reflection from both of you. If you were to give advice to organizations about to undergo an evaluation, what would you suggest? Kathy, can we start with you?
unknownSure.
SPEAKER_02Well, I don't see evaluation purely as accountability. If approached well, it can genuinely strengthen learning and future delivery. It's an important space in which to surface and articulate our assumptions about how programs work to support change and challenge and validate those assumptions. Linked to this, it's important to be able to present program information in a way that is transparent for the evaluation team. And to this end, we constructed our own internal MEL strategy with this in mind and shared this with the evaluation team early in the program. This supported common understanding and helped to confirm the distinction in roles. For instance, our responsibilities as implementers on documenting intermediate changes and the evaluation team's lead on assessing longer-term outcomes.
SPEAKER_00I would echo Cathy's reflection on the importance of transparent program information. Having clear and consistent documentation of what was delivered, when and why makes it easier for evaluators to assess a program's contribution to outcomes. An additional piece of advice would be to create an open and trusting environment for evaluators and evaluans. The most valuable insights often come from honest reflections on what worked, what didn't, and why. When implementers feel comfortable sharing those lessons, evaluations become an opportunity for learning and improvement.
SPEAKER_01Thank you so much, Ed and Kathy. This has been a really fascinating discussion. For me, I think one of the clearest lessons from BASIC is that evaluation works best when it's understood as part of a shared learning ecosystem. Complex programs operating in fragile and crisis settings require adaptation, judgment, and collaboration, and evaluation must reflect that reality. For donors, implementers, and researchers alike, the basic evaluation experience suggests that credible evidence emerges, not only from distance and impartiality, but also from trusted and transparent engagement. That's it for today's episode of our Beyond Basic podcast series. Thank you for listening, and we'd love you to listen to our other three episodes in the series, which you can find on the integrity website as well as on our LinkedIn. Thank you for listening.