Evidence, Ideology and Citizens’ Assemblies: How Do We Decide What Works?

In this episode of The Science Intersection, I’m joined by Professor Michael Sanders and Julia Ellingwood for a conversation about evidence-based policy — what it means, why it matters, and why good ideas do not always translate neatly into better outcomes.

Michael is Professor of Public Policy at King’s College London. His work focuses on evidence, behavioural change, and how research can be used to improve public services and policy.

Julia Ellingwood is a research fellow at the Policy Institute at King’s College London. Her work looks at how evidence and practical interventions can help improve outcomes in the real world, and her PhD focuses on wellbeing in the UK civil service.

In Part 1, we talk about how evidence-based policy can help public money, professional time and people’s lives be taken seriously. We discuss why some ideas that seem obvious — like homework or reducing class sizes — may be more complicated once you look at the evidence. Julia explains why evidence-based policy does not mean stopping every policy until there is a perfect trial, but instead using evidence as a set of tools to work out what we know, what we do not know, and what we most need to test.

We also look at why social policy is often harder to test than medicine, what researchers can do when a randomized trial is not possible, and how governments can act under uncertainty while still building evidence into the process.

Later in the episode, we move into the relationship between evidence, ideology and democracy. Michael discusses examples including Sure Start, austerity, the Rwanda scheme and synthetic phonics, and we ask where evidence can cut through political disagreement — and where values and ideology still matter. The episode ends by looking at citizens’ assemblies, including how they can help people work through contested issues when they are informed by evidence rather than simply being a space for opinion.

Listen to more from Michael on his podcast here:
https://open.spotify.com/show/3Y7RSs7pBOC7hlafpGVR1G?si=85d114fba4514b27

Glossary

Evidence-based policy
Making policy using the best available evidence, rather than just instinct, ideology or tradition.

RCT / randomized controlled trial
A test where people, schools, areas or organisations are randomly put into groups, so you can compare those who got the intervention with those who did not.

Randomise / randomization
Choosing who gets the policy or intervention by chance, so the groups are as fair and comparable as possible.

Intervention
The thing being tested — for example, a new service, policy, programme, subsidy or support scheme.

Causal chain
The assumed path from “we do this thing” to “this outcome improves.” For example: extra support → better attendance → better grades.

Causal inference
Trying to work out whether the policy actually caused the change, rather than the change happening for some other reason.

Quasi-experimental design
A way of studying impact when you cannot do a full randomised trial, but still want a fair comparison.

Econometrics
Statistical methods often used in economics and policy to analyse real-world data.

Regression discontinuity design
Comparing people just either side of a cut-off point — for example, people just above and just below an eligibility threshold.

Matching
Finding a comparison group that looks as similar as possible to the group receiving the intervention.

Difference-in-differences
Comparing how things changed over time in one place that got the intervention with another similar place that did not.

Robust data collection
Collecting data carefully and consistently enough that you can trust the results.

Pre and post comparison
Looking at what things were like before and after a policy was introduced. Useful, but weak on its own because other things may also have changed.

Comparison group
A group or place that did not get the intervention, used to judge what might have happened otherwise.

Units of randomisation
The things being allocated to groups — people, schools, hospitals, cities, boroughs, etc.

Phased rollout
Introducing a policy gradually in different places or at different times, which can sometimes help researchers compare early and later areas.

Observed characteristics
Things you can measure about places or people, such as income, age, location, unemployment rates or population size.

Municipality
A local government area — roughly a city, town, borough or local authority.

Food desert
An area where people have poor access to affordable, healthy food.

Transcript

SPEAKER_02 0:10

So, on this episode of the Science Intersection, I'm joined by Professor Michael Sanders and Julia Ealing Wood. Michael is a professor of public policy at King's College London, whose work focuses on evidence, behavioural change, and how research can be used to improve public services and policy. Julia also works in the space looking at how evidence and practical interventions can help improve outcomes in the real world. In this conversation, we talk about what it means to use evidence well, why good ideas don't always translate neatly into policy, and how research can help us design systems that work better for people. So I wanted to ask both of you a bit about your background. So what kind of led you to go into looking at evidence and policy?

SPEAKER_00 1:09

And there was an opportunity to come into government for a three-month second. I applied for that and I got to come into Cabinet Office and work in the behavioral eds team. What allowed me to see really close up was firstly the range of things that were being done by the team and by the cabinet office to try and improve outcomes, which is something that I care deeply about. And at that time, what was the beginning of a really exciting movement in public policy around evidence-based policymaking, and particularly the what works movement. And I sort of got hooked on that idea pretty quickly, and then so I just then stayed in that team for for seven years, uh, which was a little bit longer than the three-month uh secondment. So that that's sort of the the the origin story of this work from my perspective.

SPEAKER_01 1:56

I came to policy and research kind of late. So prior, I worked as a classroom teacher, and then I worked in education technology for several years in the US, mostly working with school districts to implement new curriculum, new technology in the classroom, kind of leveraging my experience working as a teacher in those spaces. And I found that the things that I enjoyed most were working with teachers, helping to make their lives easier, helping to implement new things in the classroom, and also importantly helping to support them to support student achievement. But I felt that I could probably get a little closer to that work than what I was doing previously. So I decided to make a career break and study policy. Came to Berlin to do a master's in public policy at the Hurdy School. And from there, I kind of got hooked on the you know evidence-based policy and research space, which just wasn't what I intended, but it's it's something that really spoke to me. So after doing that, I came to King's College London where I now work as a research fellow with the policy institute.

SPEAKER_02 3:04

Have a bit of an overview of both of the your PhDs.

SPEAKER_01 3:08

So mine will be very short to talk about because it's still in the early stages of it. So I'm in my first year of my PhD, focusing on civil service well-being in the UK civil service. So mostly using data from the Civil Service People Survey to understand different drivers and different sort of trends around well-being across all the different departments in the civil service and all up and down the civil service hierarchy as well.

SPEAKER_00 3:34

Right, my PhD was in, as I say, the economics of charitable giving. So it was a series of field experiments nudging people, including most prominently investment bankers, to give money to charity via a series of sort of behavioral science tricks. There were two chapters of basically just statistics around the way you designed these experiments, a mix of giving suites to investment bankers and and maths.

SPEAKER_02 3:55

Judo, in terms of your PhD, what was your kind of feelings in terms of sort of well-being?

SPEAKER_01 4:04

I can give a bit more in like my motivation for it. So actually, the reason why I've been pursuing this project in particular, it actually is kind of an extension of why I was interested in studying policy and working in this space to begin with, which is supporting public servants in helping them do their jobs and deliver positive outcomes for the public. Um so previously that was thinking about how we can support teachers in the classroom and making sure that teachers can continue to come to work every day and support the young people that they do, and that they can create that consistent experience and support, particularly for more vulnerable children and young people within the public education system. This project is taking a broader view than that. Obviously, teachers aren't civil servants themselves, but civil servants are still there to deliver policy to improve outcomes for the public primarily. And so understanding their well-being and understanding their feelings of worthwhile national and happiness and anxiety, feelings of anxiety that they might might have in the workplace does translate to both potentially improved outcomes for the public or going in the opposite direction, costs for the public. So if you have people leaving their jobs or getting sick a lot, that obviously generates a lot of public costs and it also decreases the quality of services that people receive. So that's my main motivation is trying to like understand how we can support civil servants better and in turn, you know, they can leverage their expertise, their skills, their energy to support the public better.

SPEAKER_02 5:41

Michael, going back to your PhD on nudging, charitable giving, what actually came out of the research? Were the findings taken up in practice by workplaces or fundraising organizations?

SPEAKER_00 5:56

It's it's not a field that I've worked in for for some time now. What we've seen is a lot of those findings were taken up by particularly large workplaces that were doing charitable fundraising campaigns. So we published a paper in 2012-2013. What does the evidence tell us about particularly fundraising in workplaces? And that those findings were pretty widely taken up across large corporations and by the Institute of Fundraising. So there's been some positive effects there, but it's not a domain that I work a great deal in anymore.

SPEAKER_02 6:24

Okay. For listeners who are not policy specialists, how would you explain what evidence-based policy is trying to do?

SPEAKER_00 6:32

So the the promise of evidence-based policy, what we're trying to achieve, is better outcomes for the people who policy serves, or at the very least, um a better use of public money. So whenever you're spending public money, you're doing anything public policy, you have to be aware of really three things. The first of which is you are spending somebody else's money, right? It's always taxpayers' money that you're spending. And so we have a moral responsibility to ensure that is well spent. The second thing you have to bear in mind is that you are trying to achieve better outcomes for people. In most of my work, that's young people. And if you are not achieving that goal, then you are wasting not just money, which is obviously very important, but also the lives of those young people. So 700,000 young people turn 18 every year, which means their childhood is over. We have a responsibility to make sure that we have set them up as best as possible for the rest of their lives. And the the data suggests so we are not currently serving that group of people as well as we could do. The third thing which we need to consider, which I think is again underconsidered, is that there are people who are working to try and achieve these goals. So that might be teachers, that might be social workers, it might be people who are guards in prisons, and they are giving their life to improve outcomes for people. And they are often working extremely hard. Quite often we are asking them to work extremely hard and do things that don't make any difference, right? So marking people's maths, homework in three different colored pens takes an enormous amount of time, but does not improve outcomes for the people who you're providing that homework for. Which means when we're looking at teachers who are working until eight o'clock, nine o'clock every night doing their marking, if we know that that isn't making a difference, we can give them that time back for themselves, right? So we have to be respectful of the outcomes for young people, the time and the commitment of professionals and the taxpayer whose money we're spending. So the problem is there is we improve all of those three things, or at least one of them at any given moment in time.

SPEAKER_02 8:28

If I remember rightly, homework came up in the panel discussion. Something that doesn't really make a huge amount of difference. Are there other examples in education where something feels obvious but the evidence suggests that the reality is more complicated?

SPEAKER_00 8:48

Yeah, it doesn't make a great deal of difference. Like certainly on average, it doesn't make a great deal of difference. In the panel, Will also said that making class sizes smaller doesn't make a difference, which isn't true. Making some class sizes smaller does make a difference. It's just the most expensive possible way of making a difference. So it's not necessarily not cost-effective.

SPEAKER_01 9:04

Going back to, I guess, the question of what evidence-based policy actually means and what it's not, I think like this is an area that could potentially cause a lot of people anxiety because I think we know that there is a lot of policy that is not evidence-based, and this feels like a very high bar to achieve to just like test everything, even when like maybe we have a strong intuition about what works and what doesn't, because based off of our own professional experiences or you know our own lived experience. And I think what what's sort of comforting to me, what what I'd counter that anxiety with is that evidence-based policy gives us a set of tools to help us achieve some level of certainty over something. And just because there is a policy out that doesn't have a very strong evidence base, we can still build off of that, right? It doesn't mean that we have to be flying in the dark. And it doesn't mean that we have to like test every step in a causal chain to see if it works. It just gives us a starting place to start understanding what things we can be certain of and to what degree we can be certain of something. So it doesn't mean that we have to like stop all policies in their tracks until we've tested everything thoroughly with RCTs. It just gives us a starting place of like, okay, what are the things that we need to test? What are the things that are costing the most money or causing um the most barriers for people? Like focus on those things first, and then we can start developing an evidence-based from there. Um so I I like to think of it as more of just a set of tools for kind of assessing what we know and what we don't know.

SPEAKER_02 10:38

You've described evidence-based policy as a set of tools for working out what we know, what we don't know, and what we most need to test. One of the tools people often hear about is the randomized control trial, but that clearly isn't suitable for every policy question. How do we think about when RCT is useful and when other methods are more appropriate?

SPEAKER_01 11:04

Yeah, I think anybody can think of plenty of examples of of interventions that we can't really randomize, right? Like we can't randomize people smoking. That would be unethical and impractical for us to do. But you know, just because we can't randomize certain interventions, I would argue there there's certain things that we can still find ways to to randomize, but where we absolutely cannot, we can still try to generate some space for causal inference to happen through quasi-experimental designs. So basically that's trying to replicate as close as possible randomization through various like econometrics, wizardry. So maybe people would have heard about some of these things like regression discontinuity designs or matching or difference in differences modeling. So we're not totally at loose ends just because we can't randomize something. I think in order to start evaluating something where we can't randomize, a really important piece is really robust data collection of like before and after interventions are implemented and making sure that you know that data collection is as comprehensive and as clean as we possibly can get it so that we can still run these types of evaluations.

SPEAKER_02 12:22

Or maybe if you could give a sort of slightly simplified example of examples of these designs of trials.

SPEAKER_01 12:29

I could make up one that I've been thinking about a bit recently. That's fine.

SPEAKER_02 12:33

Sure.

SPEAKER_01 12:34

Um so oftentimes it like if if a municipality is implementing a new intervention and it's going to be an intervention that affects everybody in the city, um, that's probably a difficult thing to randomize, right? We don't have very many units of randomization. We only have so many cities that are comparable to one another, and these things cost a lot of money usually, so we can't like recruit a bunch of cities to do this. So I've been thinking about municipal grocery stores lately. So this is a policy intervention that's being uh you know discussed and rolled out in New York City right now, um, where they're going to establish municipal grocery stores across all five boroughs of the city. And the goal of this is to try and tackle food poverty and and food deserts in different parts of the city and also to create sources of afford affordable groceries for people who are particularly underserved. Now, this isn't something that we could randomize really. I mean, we could try, but I think it would be very challenging. And they're going to be rolling this out regardless of, you know, whether or not, you know, we as evaluators come in and say that they should do it a certain way, right? So one approach you could look at with this is you could obviously, you know, intuition, you could just do a pre and post comparison. Like what was it like in the communities before they implemented the grocery store? What was it like after across the different dimensions? But you know, that's not really going to help us identify the actual impact of that grocery store because maybe some other things happened in that area during that time. So maybe instead of that, maybe we could have like a phased-out rollout through the five boroughs, or potentially we could find another municipality elsewhere in the US that are across a few different observed things very similar to the Bronx or Brooklyn or Staten Island, right? And then we can make comparisons that way. So that's kind of like a basic sort of building of intuition of like how do I actually create a comparison that I can judge the success of this municipal grocery store against a situation where they don't have a municipal grocery store.

SPEAKER_02 14:42

So that's helpful because it sounds like evidence-based policy is partly about dealing with uncertainty, not pretending that we know everything in advance. So with medicine you can kind of set up trials and test something before it kind of goes live, as it were. With social policy, governments often have to act before they can be sure of what will work. How should we deal with that gap between needing to act and still needing evidence?

SPEAKER_00 15:11

So think about the use of youth employment support, the Kickstarter the government uh brought in after after COVID to try and help 18-25-year-olds get jobs. We'd never done anything before like before like this. Basically, the government paid half of the costs of employing a young person to their employer. We thought it might work because it was a pretty heavy subsidy for employing young people. There was no way before doing it that we could have known whether it worked or not. So we had no choice but to bring it in. And that is the role of a politician, right? A politician or even a public servant's job is to come up with ideas about things that might work and then to do them. And the hope is that we can build the evidence base so that they continue to do those things that do work and stop doing those things that don't. Now, there are examples of situations, so we've got uh a problem at the moment with unemployment of young people aged 18 to 25. We can look across the sea to the Netherlands, which has really low levels of youth um unemployment, and see if they are doing things there that seem like they might work. So in that case, we can find evidence from somewhere else, but we don't really completely know that it works just because it works in the Netherlands. The Netherlands is a very small country that is quite different to the UK, much less it's much less complicated a country. And so interventions that work well there might not work well here, or they might work better. Similarly, we find that lots of programs that work really well in the United States don't work very well when they're ported over here, for example, the Nurse Family Partnership. So the job of a politician and a public servant is to find an idea of what might work and then to ultimately take a risk in implementing it based on the evidence that exists at the time, but ultimately recognizing that you can't know for sure whether something's going to have a particular impact unless you build evidence generation into that process.

SPEAKER_02 17:02

I suppose there have been things that have been tried and then abandoned because a different government gets in, even though they're actually shown to work partly on the basis of sort of different ideologies.

SPEAKER_00 17:16

Yeah, so I mean the the go-to example about that is ShureStart, which the new Labour government brought in in the early 2000s, and which was was canned by the coalition government uh when they came in. And that's an interesting case because the evidence base for SureStar at the time was not great, and subsequent evaluations that have been published since the end of the coalition government have shown that it was incredibly effective. Why was it gotten rid of? In part because the government had an agenda of austerity, and at the time it seemed as though Shure Start was expensive, which it certainly was, and it seemed as though it probably wasn't cost effective. So at the time when George Osborne made his decision to can it, the evidence base did not support extending it whilst you were also cutting public services. There's certainly an ideological aspect of that. Austerity was not an ideology-free um set of government policies. But if you told George Osborne that this program actually saved money and there was strong evidence for it, then I think he probably wouldn't have gotten rid of it. So partly that that comes down to the sequencing of the evidence itself. However, it is also true to say that sometimes governments do get rid of things because of ideology, right? So if we were to take an example that might be slightly more controversial, the the Rwanda scheme, the program to deport people who came here, came to the UK claiming asylum to Rwanda, that didn't actually get off the ground because of various reasons, including it being unworkable. But let's imagine a world in which it did happen, and as a result of it, there was strong evidence suggesting that it it did prevent people coming to the UK seeking asylum on small boats, which was its objective. A new government, the Labour government that we currently have, could have said, well, we are ideologically opposed to doing this, even if it has its desired effect, because we consider it to be immoral to deport people to Rwanda just because they came here seeking asylum, right? That's an ideological decision, which is just the one government disagrees with another one ideologically. And that's that's got to be absolutely fine, right? Because we live in a in what we live in a constitutional monarchy, but we live approximately in a in a democracy. And so if the people elect a government that has a different ideology, that government must be free to act on that ideology. The space where evidence-based policy should see consistency between governments is in those areas where the outcome is agreed between political parties, but the mechanism is not highly ideologically charged. So an example of that would be reading, right? The Conservatives, the Labour Party, the Liberal Democrats, the Greens, even reform, are broadly in favour of people being able to read. And nobody has a very strong belief about the way in which people should be, very few people have very strong beliefs about the moral way for someone to learn to read. And so the evidence which tells us that synthetic phonics is the best way of helping children to read, was brought in by Michael Gove and persisted through not only a decade and a half of conservative Secretaries of State for Education, but has been retained by Bridget Phillipson because the evidence base is clear and there isn't a lot of ideological heat in that space. Okay, so there are some spaces where evidence is ideology proof and some cases where it isn't. But the important point is if you want to live in a democracy, sometimes the other side wins and they get to enact their ideology. I might personally think the cancelling of the Rwanda scheme is a good thing, but there I respect the right of other people to disagree with.

SPEAKER_02 20:58

different ideologies but have sort of similar goals in mind. They have different ways of thinking about how you should reach those goals.

SPEAKER_00 21:12

Yeah, and sometimes that's around, for example, the extent of marketization of a problem, which it might be a question that is answerable, right? So you might be able to say, actually, markets like water do not work well in the in the private sector, and people who are broadly reasonable can agree on that regardless of their political ideology. But there might be areas where there is scope for disagreement among some people, but not among well, not among others, or rather the scope for agreement. So the uh the like go to example of a citizens' assembly is the Citizens' Assembly on the repealing of the Eighth Amendment to the Irish Constitution, which was the one that um prohibited abortion, prohibited women's right to choose. You could bring together probably 80% of Irish society's views and have people say, you know, okay, well, some people are very in favor. Of people being able to choose to have a termination until quite late in the day. Other people might say there's a space for abortion, but only much sooner, or only in certain moral cases, for example. And you can bring those people together and you can present them with the evidence and then they can reach a conclusion. Among those people, there is scope. Among people of a particularly traditional religious belief, or people who are identify as pro-life, even if I don't like that phrase, for other reasons, they don't feel there is any scope to move their beliefs at all because it's it's going to be completely immoral, regardless of which way you do it. So a Citizens Assembly can help those people for whom there is space to arrive at a consensus, to arrive at a consensus, but obviously it's not going to change the mind of a big, a big selection of people just there for moral reasons. And then that's why it has to be followed up by something else. So in the Irish case, they had a referendum in which they voted for the 8th Amendment to be repealed. It was a very successful campaign to do that. In other forms of democracy, you could abdicate that responsibility could be taken by Parliament to make that decision on behalf of the citizens, and to say, actually, we have decided that that 20% of people who hold that very strong view represent a minority within our democratic system, and so we're gonna we're gonna change the law. But the Citizens Assembly itself is a great way of bringing those people who can reach consensus together, but it doesn't deal with the rest of society. And those citizens' assemblies need to be informed by evidence, right? It's not just vibe-based, we sit around for a week and chat about it. There has to be evidence around, in that case, for example, viability of fetuses, etc.

SPEAKER_01 23:42

Yeah, that was maybe like the one point I was going to add to that is that I don't think citizens assemblies and evidence-based policy are all in conflict with one another. Because one of the first steps of doing a successful citizens assembly is a very long time of like data download of like this is what the evidence base says, this is what we know about this issue, these are the different sides. It's not a town hall where everybody just comes in and just like expresses their opinion and hopefully you come to a consensus. Like it's meant to be based in the cutting-edge knowledge of what we know about that topic. So that's how evidence-based work can kind of inform those citizens' assemblies and make them more productive.

SPEAKER_02 24:20

Could citizens' assemblies help with issues like climate change, where there are difficult trade-offs, and the people most affected are not always the ones with the most power.

SPEAKER_01 24:46

I think citizens' assemblies probably have a role to play in that, and at least uh helping to get issues like that back on the agenda for people, um, which it feels like it has started to slip. Also, you know, citizens' assemblies are pretty well aligned to some of the goals laid out in the sustainable development goals in terms of like healthy institutions and uh democratic principles and whatnot. And it it also it's also an issue that requires a lot of like political courage, I think, to take on. And so citizens' assemblies could potentially bolster that case, put it on the agenda, and also embolden politicians to take these issues more seriously. I'm not certain if citizens' assemblies are necessarily going to be the ones identifying the policy interventions that are going to move us forward on climate change, but I do think they potentially have a role to play.

SPEAKER_00 25:42

Aaron Ross Powell If I were to take the other side of that argument, there's there are foreseeable economic impacts. For example, the city of Aberdeen is very dependent on the oil and gas industry, and so if we stop drilling for oil and gas, there is going to be an economic consequence of that on the people of Aberdeen in the same way as there was in the north when we stopped mining from happening. And we kind of know that's going to happen. And there are engineering-based solutions to a lot of the problems that we're talking about here, or at least they're partially engineering-based, right? So we need to build more solar, we need to build more wind power, and yes, probably we need to build small modular small modular nuclear reactors, a la and Milivan's videos. I feel like there's an abdication of responsibility by policymakers when they say we're going to ask the people what they think about having small modular nuclear reactors. I just think they should just get on with it. Right. And to the extent that like we know that that's going to this is going to have a consequence for Aberdeen, let's not wait until there's a consequence for Aberdeen and we've got 50%, 60% unemployment like we have in Scunthorpe. And then say, Oh, maybe we should do something about it. We we know what the consequence is going to be. Let's plan for it and let's deal with it now. Right? Let's start building solar panel manufacturing plants just outside of Aberdeen. And why not just outside Scunthorpe while we're at it? So that the jobs of the future that we hope we're going to create are concentrated in those places where we know that the jobs of the present are most likely to disappear. Right. This does not require us to go through a sortition process and get 300 people together in a conference center in Bournemouth for four days. Right? We know what's going to happen, we just need to do it.

SPEAKER_02 27:21

I'm a bland technocrat, so maybe I'm wrong about that. Next week we continue the conversation by looking at what trials can show us that good intentions and common sense might miss, why outcomes like well being, safety and opportunity can be difficult to measure, and how researchers can test policy ethically with the people most affected.