Fire Science Show

228 - Quantifying the expected utility of fire tests with Andrea Franchini

• Wojciech Węgrzyński

Share episode

What do you expect from running a fire test? I would hope that it improves my state of knowledge. But do they do this? We often pursue them blindly, but it seems there is a way to do this in an informed way. 

In this episode we explore a rigorous, practical way to select and design experiments by asking a sharper question: which test delivers the most decision-changing information for the least cost, time, and impact. With Dr. Andrea Franchini of Ghent University, we unpack a Bayesian framework that simulates possible outcomes before you touch a sample, updates your state of knowledge, and quantifies the utility of that update as uncertainty reduction, economic value, or environmental benefit.

First, we reframe testing around information gain. Starting from a prior distribution for the parameter you care about, we model candidate experiments and compute how each would shift the posterior. The gap between prior and posterior is the signal; diminishing returns tell you when to stop. In the cone calorimeter case on PMMA ignition time, early trials yield large gains, then the curve flattens, revealing a rational stopping point and a transparent way to plan sample counts and budgets. The same structure scales from simple statistical models to high-fidelity or surrogate models when physics and geometry matter.

Then we tackle a post-fire decision with real financial stakes: repair a reinforced concrete slab, or accept residual risk. We connect Eurocode-based thermal analysis to two test options—rebound hammer temperature proxies and discoloration depth—and compute their value of information. By translating updated probabilities of exceeding 600°C into expected costs of repair versus undetected failure, we show how to choose the test that pays back the most. In the studied scenario, the rebound hammer provides higher value, even after accounting for testing costs, but the framework adapts to different buildings, cost ratios, and risk appetites.

Beyond pass-fail, this approach helps optimize sensor layouts, justify added instrumentation, and balance multiple objectives—uncertainty, money, and environmental impact—without slipping into guesswork. If you’re ready to move from ritual testing to evidence that changes outcomes, this conversation maps the path. 

Papers to read after this:

----
The Fire Science Show is produced by the Fire Science Media in collaboration with OFR Consultants. Thank you to the podcast sponsor for their continuous support towards our mission.

Wojciech Wegrzynski:

Hello everybody, welcome to the Fire Science Show. In today's episode, we will be discussing how to pick the best tests to fulfill your needs, whatever the needs are in your fire safety engineering. And I've invited the nice guest, Dr. Andrea Franchini from University of Ghent to talk about this. And you may think we will be ranking test methods or giving you comparison between standards, and no, we're not gonna do that. Because what Andrea is doing is something quite more fundamental, which is gonna help those considerations about what test to use at pretty much any decision level you would be making. It is a part of a larger project, ATEST, by Dr. Ruben van Coile. Ruben is leading an ERC starting grant. We have discussed this in the podcast, I believe, two years ago. We had uh this uh podcast episode when Ruben was announcing his project and uh giving me an introduction about what he wants to do. Uh some years have passed, some work has been done, great work has been done, a massive paper was published, and here we are today able to discuss about the mathematical approach that is being used to decide upon the test methods. In this approach, instead of just thinking about uh the potential outcomes of a test instead of thinking about the criteria, pass fail criteria or other ways we would normally measure, this uses those, but uses those to inform what kind of information gain the test can give. This is a very specific term and Andrea will explain it later, but this kind of informs how much your expectations or centenities will move as you pursue the test. Because if you would like to do a test and it's not gonna move your knowledge by a bit, it's not gonna inform your decisions anymore, what's the point of running it? And if you have a limited budget, you would like to spend those money on the tests that will move your knowledge the most, so the reduced incentities the most or provide you some other metric. It's quite some complex mathematics and uh it's not an easy episode to be warned. But I think uh looking at the framework of what Ruben was proposing and looking today at the first outcomes and the proposals and the case study paper, I see this as a really, really good way. It's gonna be a challenge to introduce it into our paradigm. It doesn't work with our paradigm of fire testing. We need to change the paradigm to do this, but perhaps this is a good reason to change one. Um in the episode we go through two case studies, one related to concurmatry, one related to post-fire investigations. While you're uh listening to the episode, there's a Fire Safety Journal paper that covers the exact details of those two case studies. So if you have a chance uh to go through Fire Safety Journal paper, I would recommend doing that. And uh for now, please enjoy learning from Andrea about what has been done. Let's spin the intro and jump into the episode. Welcome to the Fire Science Show. My name is Wojciech Wegrzynski, and I will be your host. The Fire science Show is into its third year of continued support from its sponsor OFR consultants, who are an independent multi-award-winning fire engineering consultancy with a reputation for delivering innovative safety-driven solutions. As the UK leading independent fire consultancy, OFR's globally established team have developed a reputation for preeminent fire engineering expertise with colleagues working across the world to help protect people, property, and the planet. Established in the UK in 2016 as a startup business by two highly experienced fire engineering consultants, the business continues to grow at a phenomenal rate with offices across the country in eight locations, from Edinburgh to Bath, and plans for future expansions. If you're keen to find out more or join OFR consultants during this exciting period of growth, visit their website at OFRConsultants.com. And now back to the episode. Hello everybody. I am joined today by Andrea Franchini from Ghent University. Hey Andrea, good to have you in the show.

Andrea Franchini:

Hi, Wojciech. Thanks for having me.

Wojciech Wegrzynski:

Thanks for coming to discuss your interesting research carried out as a postdoc in the Ruben van Coile's uh ERC grant. How's working for ERC grant? We've advertised with Ruben uh the position in the podcast. I hope we have not overpromised.

Andrea Franchini:

Yeah, yeah. I listened to the postcard before uh before applying for the position. And yeah, so far it has been great. It's a very mentally challenging and uh interesting project. And he gave us the opportunity to collaborate with a lot of people, and so it's very, very interesting project on my side.

Wojciech Wegrzynski:

Yeah, super happy. And now seeing the first outcomes, or maybe not the first, but seeing the first big outcomes of the project that we're gonna discuss today, I'm I'm really excited for the next years of your research. So, Ruben had this presentation at ESFS conference in Ljubljana this year. Uh, he had a keynote on quantifying the utility of fire tests. I've talked with him and he told me that you've done all the hard jobs, so he wants you to uh speak on that, which I appreciate. Uh we know that the concept of Rubens ERC, the whole goal of A test was to like figure out how do we test in the future in such a way that it's the best. But now you have a framework. So let's introduce the listeners to the framework and perhaps let's start with the expected utility. I like that keyword. So so maybe let's start there and see where we get.

Andrea Franchini:

Yeah, yeah, sure. So let me start with repeating the key problem we're trying to address. So we we want to understand which experimental protocol is best, meaning which experimental protocol uh experimenter should choose among all the options that he has. Should he go for uh a furnace test? Should you take a test in a concalorimeter? Which one?

Wojciech Wegrzynski:

Assuming you have a freedom to choose.

Andrea Franchini:

Assuming you have the freedom to choose. Yes, that's part of the FRPEST framework we we envision. So, assuming that you have this freedom, the choice among alternative experimental protocols is challenging because of several reasons, including that experimental outcomes are uncertain. Experiments are costly and time consuming. Probably you will know that better than me. And you can also have an environmental impact of your experiments, which you may want to account in your decision making. So the scope of our framework is to answer the question of which experimental protocol is best and we should choose by quantifying the expected utility of that experimental protocol before we actually conduct the experiment. So, in other words, we want to assess the potential benefit of collecting additional information through the experimental protocol we are planning, and we want to do that before doing the experiment, incorporating the available knowledge we have about the parameters we want to study and reflecting the experimental goals, which could be reducing uncertainty, reducing the economic impact of this uncertainty, or reducing, for example, the environmental impact of this uncertainty. So, in this sense, the expected utility you quantify captures the scopes of your experiment.

Wojciech Wegrzynski:

And that utility is directly linked to your design goal. So, in what case you would be using that method, for example, you have to apply some very specific technology, and uh, I don't know, you want to know which product is the best to fit, and then you figure out which tests to do for that.

Andrea Franchini:

Yeah, so this applies to both tests. So you want to, for example, demonstrate compliance or you want to classify a product, and it also applies to experiments. So for uh explorative experiments, you want to, for example, optimize your testing setup, you want to decide where you should put your sensors, and the utility definition aligns with your scope. So, for example, let's say you want to reduce uncertainty about some parameter. In that case, you can define a utility function that quantifies how much, in expectation, the outcomes of your experiment will reduce the uncertainty in those target parameters that you're tackling.

Wojciech Wegrzynski:

Now, now that you said that I had the same feeling when Ruben was presenting in in Ljubljana that this is a very kind of universal framework which you can twist into different settings. You've uh twisted and fine-tuned it into uh fire testing and uh and and fire safety engineering. But indeed, you know what? When I when I heard Ruben's talk, I immediately thought about CFD and zone model uh dilemma. Like, is it better to run one CFD or uh or a thousand zone models in the same time? You know, because uh it's also like a kind of a challenge where you have like different centered, different levels of insight into the problem cost limitations. I I love it, I love it. But let's go back to the tests, to your applications that you've described. So in your paper, you have uh some practical examples how these have been implemented. So we'll probably discuss more of those in the discussion today. Perhaps let's introduce the listeners to a representative problem which we could then solve through the discussion, applying your your your thing. So uh let's let's go to the case study.

Andrea Franchini:

Yeah, so for example, one of the case studies we presented pertaining the con calorimeter testing, and we wanted to understand how many tests we should run with the con calorimeter in order to reduce uncertainty in estimating the ignition time of a PMMA batch. That was the idea. And to answer this question, we apply the framework that we are discussing and uh we define a utility metric that captures the amount of uncertainty you have in different state of knowledge before you do the experiment and after you do the experiment. And then using this calculation that we can discuss more in detail uh later, you basically get an estimate of the number of tests that in expectation will minimize your uncertainty in the prediction of this ignition time.

Wojciech Wegrzynski:

Okay, okay, so but but what are the things that you are playing with, for example, in the con calorimeter? Because, like, I mean, one way you could do it, the the classical way I would do it is I would just run the tests until I get some sort of convergence and say, okay, you know, this seems enough. But of course, uh, I do not know when the convergence will happen uh before I start doing that. So exactly. So uh this allows me to predict that the that that time when the convergence okay. So what do you say with it?

Andrea Franchini:

Yeah, maybe I I can explain more what we mean by experimental protocol. Yes, so by experimental protocol, we mean a combination of experimental procedures. So you may have the concalorimeter, you may have the furnace test, and so on, and experimental design parameters. So, how do you tune your testing setup to run the experiment? For example, in the co-calorimeter, what is the heat flux exposure that you're going to use? Uh in in a different test, you may choose the temperature to which you're going to expose the sample, and so on. And in the case study of the ignition time that we have introduced, we say, okay, we already decided we want to use concalorimeter, so that's a fixed parameter. And the only experimental design parameter that we want to investigate is the number of con-calorimeter tests that we should run. So in this case, we just limit the analysis to one experimental design parameter. And we want to calculate how many tests we should do so that the uncertainty in predicting the ignition time is minimized. That's the that's the idea. But in in principle, you can include many more experimental design parameters. For example, the it flux and other variables that you can play with in with your experimental setup.

Wojciech Wegrzynski:

And uh the procedure will uh look very very similar, no matter how many outcome variables you include. What changes?

Andrea Franchini:

Yeah, so conceptually it works the same. The problem is that it becomes more challenging computationally and also conceptually, I think, if you have many design parameters, because you need to build models that are able to capture all your experimental design parameters. So conceptually, it works exactly the same. You can include as many experimental design parameters as you want, but you need to formulate the problem in a way such that it can account for all these experimental design parameters. You will need models that reproduce your experiment. So, whatever you want to optimize, you need a model that is able to capture that parameter.

Wojciech Wegrzynski:

Yeah, that's what I wanted to ask. So, optimally, if it's a physical model, but do you need to also know the shape of the distribution of the outcome variable?

Andrea Franchini:

Yeah, so maybe I can give you an overview of how the analysis works so that we know all the different uh pieces that we need.

Wojciech Wegrzynski:

Maybe let's let's even step one step before because I feel it's gonna be interesting and and and intellectually engaging and challenging discussion. You use Bayesian framework for this. So maybe let's start with the framework itself and then build up into how this is applied. And let's try to keep it all in the the time to ignition of PMMA stage that we've set up. So that's great.

Andrea Franchini:

Yes, Bayesian analysis is at the core of the methodology, and there are three main ideas in this methodology. One is the concept of state of knowledge, the second one is the concept of Bayesian analysis, and then the concept of utility. So let me link these three concepts for you. So we generally express our uncertain state of knowledge in terms of probability distributions. For example, back to the ignition time of PMMA, you may say, I have uncertainty about the ignition time, and to describe this uncertainty, I assign a distribution, let's say a normal distribution with a mean and a standard deviation.

Wojciech Wegrzynski:

And then people do this all the time when they say like it's uh one minute plus minus 10 seconds. That's already a distribution you've given in this in this information, even if you don't think it's a distribution, you know. Okay.

Andrea Franchini:

Absolutely. Yes. So you basically express your state of knowledge in terms of probability distributions that reflect uncertainty you have about some parameters. And second concept that introduced is Bayesian analysis. So, what's that? It's a method to compute and update probabilities after obtaining new data. Okay, so the key the all the Bayesian analysis is based on Bayes theorem. And the key idea of Bayes theorem is that evidence should not determine your state of knowledge, but it should update it. So, what does it mean? It means that if we, as we said, we assign a distribution to represent our current state of knowledge, we do the experiment, we get some data, and we use base theorem to get a second distribution that reflects your updated state of knowledge based on the observed experimental outcome. And for example, if you go back to the concrete method of testing, you have a distribution that reflects your knowledge of the ignition time before you do the experiment.

Wojciech Wegrzynski:

So it's let's say 1210 seconds plus minus 30.

Andrea Franchini:

Yes, exactly. You you can say it's a normal distribution with mean uh uh 210, as you said, and a standard deviation, as you said. So you do the experiment, you get some data points, and using Bayes' theorem, you can calculate a second probability distribution that somehow accounts for this updated knowledge that you get from your new data. So you get a second distribution that represents your updated state of knowledge after you do the experiment.

Wojciech Wegrzynski:

So for example, now it's 207 plus minus 17. I do another one and I find it's plus minus 16, but eventually every time I do a new one, I just get the result within my standard deviation, it fits the pattern, and eventually I end up with the final standard uh normal distribution curve, which does not really change that much every time I run the experiment, potentially.

Andrea Franchini:

Yeah, it may change or it may not, because if your experiment I don't know that, yes. Yeah, yeah, exactly. So if your experiments confirm your uh state of knowledge, meaning, for example, you get a result very close to your mean, to your estimating mean before you do the experiment, the distribution will remain essentially the same. But if you get, for example, an experimental data point very far from your initial state of knowledge, your distribution will somehow translate towards that value. So you will get an updated state of knowledge that tries to average your prior knowledge and the evidence that you get from the experiment.

Wojciech Wegrzynski:

Yeah, the the that's probably a biggest challenge in here, like the outliers and how do you handle them. Because okay, going back to the world of fire science, it's not that you can afford to run cone colorimeter on PMMA 10,000 times, because there are also like economical and time restraints, and you probably would burn through three cone colorimeters running 10,000 samples anyway. So, yeah, there are physical limits in how much you can do. And if in general, we are usually working with very low sample sizes in fire science. Like if you have three, you're good. If you have five, you're plenty. If you have 30, you're David Morrison burning chairs because no one else does like 30 samples of one thing, right? Does this theorem also have a way to handle outliers, or you just you get one, you you you you face the consequences, they have to figure it out?

Andrea Franchini:

Yeah, so in the theorem you have something called likelihood, which is the probability of observing that specific data point based on your prior state of knowledge. So if the data that you observe is very far from your current knowledge, it will essentially be assigned a very low probability so that it will uh tend to influence less your state of knowledge. So you you account for this.

Wojciech Wegrzynski:

But in this approach, in no point of this kind of consideration, you try to understand why the deviation occurred, right? Because here it's just about the statistics of the outcome. While in reality, for example, uh someone was doing cone calorimeter tests on 25 kilowatts, the next day someone was doing on 50, and the third day a guy came back and didn't notice it's changed, and and maybe run a test in wrong conditions, so it could have been an error. So so but but the the theorem will not like it's a separate analysis to to clean out the data, I guess.

Andrea Franchini:

Yes, so you you need to be critical about your data before you use the theorem. But one uh nice and elegant thing of Bayes theorem is that it's just centered on updating your belief. So even if you get your outlier, for example, once you can update your self-knowledge based on that outlier. So it's up to you to say, should I use the outlier and should I understand it? Yes, you definitely should, but that's not the the point of base theorem. You in other words, you don't need to take random data and throw them inside, although you can, you can do that, and you will still get a posterior distribution that reflects your updated state of knowledge based on the data that you give to the tier.

Wojciech Wegrzynski:

Yeah, and I assume even if you did that and then you have uh hundreds of data points from your normal data, the the the theorem would result in a distribution that that's closer to the original and outlier will not impact it that much. I I that that's what I guess. Lovely. Yeah, yeah. And now how does uh okay, so so we know the the base theorem assumptions uh now. How do you apply this in in the in the technicalities of of the testing?

Andrea Franchini:

Yeah, so the first component that's I mentioned for this framework is the concept of utility, right? So we define state of knowledge, we explain what Bayesian analysis does. So you update your state of knowledge, meaning you get a new distribution representing your state of knowledge after the experiment. And then what you can do is to assign to this state of knowledge a metric that describes how that state of knowledge is desirable to the user. How do we do that? We define utility function that quantifies the desirability of that state of knowledge based on your objectives. So, for example, if you're interested, as we are discussing for the case of the concolorimeter, in reducing uncertainty, you can define a utility metric that tells you how much uncertainty you have in your prior knowledge and how much uncertainty you have in your updated knowledge. And then you take the difference between the two and you assess whether your experimental data reduces the uncertainty with respect to your prior knowledge or increase it. Okay. So all this is if you have done the experiment, right? After you do it, after you have done the experiment, you can do all these calculations that I described. But as I mentioned in the beginning, the framework aims at calculate making this analysis before we do the experiment, right? So how do we do that? That's clear.

Wojciech Wegrzynski:

So you want to budget how many cone calorimeters you want in your research grant. Do I need to go for five or fifty?

Andrea Franchini:

Yeah.

Wojciech Wegrzynski:

Your boss says, let's go for a hundred. We need uh exactly, exactly.

Andrea Franchini:

And you want to do that before you run the experiment, right?

Wojciech Wegrzynski:

Yes, yes.

Andrea Franchini:

So, how do you do this? You need a model that simulates your experimental outcomes. Okay, based on your uncertain parameters. And you simulate multiple times the possible experimental outcomes of your experiment. You use the Bayesian theory that I described before for each of those outcomes, and for each of those analyses, you get an estimate of the utility of the experiment if the outcome was the one you simulate. And then you take the expected value of all these outcomes that you get, and that represents the expected utility of your experiment.

Wojciech Wegrzynski:

So, as I understand you for the case of cone, a model of a cone would be some sort of uh, I don't know, let's say a machine learning of a cone based on a thousand samples or some formal previous statistical distribution, and then you expect the based on literature that the time of ignition of this material is like 300 seconds, and maybe the standard deviation would be 30 seconds, so you have no some expected outcome. Then, based on some model, you run the analysis and see that you potentially can reduce the uncertainties there. Well, in in this case, you probably can just use the model to predict your time to ignition if you have such a good model that predicts it. But I assume the wealth of this method comes when you have multiple tests to choose from and uh multiple uh utilities uh to balance. So that that that's when it it comes into play. The gain of information is this what was described in the papers as the information gain or the gain in utility? Yes, exactly. Let's introduce this concept to the listeners. So it's about how much more information you get per repeat of experiment. Is that I understand it correctly?

Andrea Franchini:

Exactly. Yes. So what we did was calculating we we chose different number of tests in a concalorimeter. So we went from uh one all the way up to 30 tests, and we calculated for each of these possible number of tests, how much in expectation that number of tests will reduce uncertainty in the distribution of the ignition time. And then we, using the calculation that I described, we get that if you start at low trials, you see a very large increase in information gain. So any trials, like you have uh one, three, and so on, will give you a lot of information. But then you start seeing a plateau in the curve. So you see that the expected information gain that you get from running many tests reduces until at some point the marginal gain in this information is basically zero. Which means that if you want to understand an optimal number of tests that you should run, you should stop when this marginal expected information gain basically goes to zero. Because beyond that, based on your models, based on your assumption, the experiment will not give you more information.

Wojciech Wegrzynski:

Brilliant. It's it's such a challenging concept because at some point like you may also be seeking those outliers, like to understand what happens in the outlier case. This is basically how the particle physics works, you know. Uh this is how uh certain experiments run, that you just collide billions and billions of particles, and most of them just follow the standard model like they should. And uh every now and then one of them goes a little different, and that's the ones that you get the Nobel Prize for. So you're basically seeking those uh those outliers at uh but at like five standard deviations from the from the model. So yeah, that that's super interesting. But we're not there yet, we cannot collide billions of uh cone calorimeter samples. Uh now uh we're we're in in the world of optimization because in the end, this is supposed to be useful. This is supposed to create the utility, not in terms of utility of your test, but utility in terms of being able to be confident that you've run enough to get your information and inform. I think the the cone was a simple sample, but there were some more difficult ones when different measures were compared for utility. So maybe let's switch the case study for something more complicated.

Andrea Franchini:

Yeah, sure. So just to build a link, in the previous case, we defined utility as the reduction of uncertainty. We can make a step forward and assign an economic value to the effects of this uncertainty reduction, which is what we call value information. So we demonstrate this in the context of post-fire assessment, which is a practice that generally combines calculations and testing. But up to now, we don't have a structured framework to decide which tests we should perform and whether we should actually perform a test. So that this decision is left to the experience of the assessing engineer. So we want to show how our calculation approach can support this decision-making process. And this is the second example we present in the FAFER. We assume there was a single compartment fire in a building, and the assessing engineer needs to decide whether the reinforced concrete slab in this compartment needs structural repairs. So, how do we know that? We uh because steel exhibits a permanent reduction of yield stress if the temperature exceeds 600 degrees, we take this the steel temperature at 600 degrees as the threshold, a simplified threshold for structural repair. So the question becomes how do we know if the reinforcement reached 600 degrees? We can do that using calculation methods. So, for example, we consider the Eurocode parametric fire curve as a thermal binary condition, and we run thermal analysis to calculate the temperature inside steel. This model takes requires as input the geometry of the compartment, the thermal properties again of the compartment, and both we assume we know both of them. And then it also requires the fuel load and the opening factor. So since we have uncertainty about these two parameters, we assign them a probability distribution that is representing our prior knowledge. Okay. And the idea is that if we do some tests, we can get an updated distribution for the fuel load and the opening factor and better support the decision-making process of whether we should repair or not. So we consider two possible experiments. The first one is a remound hammer test, which gives us the maximum temperature reached at a depth of 15 millimeters from the cover. The second test we consider is a discoloration test, which gives us the maximum depth of the 300 degrees height of air. And now the question becomes if there is any economic benefit from testing with any of these two tests that I mentioned. And if so, which of these tests should we choose?

Wojciech Wegrzynski:

Sorry, in this case, you already made your mind to use the Eurocode calculation method to approximate the rebirth temperature to see whether the 600 degree threshold was met or not. And here uh you are using the hammer or discoloration as a mean to increase uh what it is it's gonna give you some information about the peak temperature on the wall. Are you gonna loop this to the Eurocode model? Or if you find like that at 50 mils there was 200 degrees, you don't need Eurocode calculation to know that there was no not 600 at the river. How does it tie to the Eurocode model?

Andrea Franchini:

Yeah, so the the point is that the none of the tests give you the temperature of the reinforcement, right? Yes, they give you the temperature of at the first one at a given depth, and the second one gives you the depth of the 300 degrees isotherm. So, what we want to do is to link the measurement of the test to our calculations and use the base theorem to update our knowledge of the field load and the opening factor based on the measurement we get from the test. That's how the test can reduce uncertainty in the uncertain distribution of fuel load and opening factor. So that is the starting point. And then How do we convert this in economic terms? We build a cost model that associates a cost to the distributions of the fuel load and of the opening factor. And this is done by taking into account the expected repair cost and the expected cost of undetected failure, which is given by the cost of failure multiplied by the probability that the maximum temperature of steel exceeds 600 degrees. And we also take into account the choice of a rational decision maker, which is to repair the slab if the expected repair cost is lower than the expected cost of undetected failure. For the device, the best choice is to leave the reinforced concrete slab as is. So all this is in the cost model. We use the cost model to calculate the expected cost with prior knowledge, meaning without doing any tests. And then we use an experimental model, meaning a model that simulates possible experimental outcomes to simulate possible outcomes of the two tests we consider. So again, the rebound hammer test and the discoloration test. And what we do is this we simulate a possible experimental outcome, we update the distribution of field load and opening factor using Bayes' theorem. We calculate again the probability of the temperature of steel exceeding 600 degrees, use this probability to calculate the expected cost for that realization of possible experimental outcome. And then we take the expected value of all these costs, and that is the expected cost of our experimental protocol.

Wojciech Wegrzynski:

So this is uh again, we're not yet talking about solving the person's problem about their particular building being on fire or not. It's about how much narrower the uncertainty of the Eurocode method will be if you inform it through this coloration test. How much narrower will it be if you inform it through the hammer test? And given the two possibilities and the known costs of those two methods, which makes more sense to do, or perhaps none of them, right?

Andrea Franchini:

Exactly. Because now we have an estimate of cost with prior knowledge, so without doing any tests, and we have an expected cost doing any of the two tests. So the difference between the expected cost without testing and the expected cost with testing is what we call value of information. And if this value of information is positive, it means that the test will give you an economic benefit. So the result of this analysis tells us first that any of the two tests provides an economic benefit. And second, we found that for this specific case, the value of information of the rebound Hammer test is higher than the value of information of the discoloration test, which means that if you are an assessing engineer, you want to choose the rebound Hammer test because it provides you a higher value of information. And we also demonstrate that this is true even if we consider the cost of testing.

Wojciech Wegrzynski:

Is this specific for the case study or is it true in general? Because you're applying a general model of a compartment to a compartment, more or less, with two tests that are also like quite general in their own. So this consideration could be true for anyone who wants to apply the same approach to a compartment fire test, right? But uh in in I guess in some more complicated cases, you would have to narrow it down to very specific case of a person that seeks the input and the value of information.

Andrea Franchini:

Yeah, definitely. So the approach is general, you can apply this to different compartments, and you can also choose different tests, but the conclusion is specific to the case study you consider and to your assumption. So that's why I say is in this case. Uh but yeah, the the same approach can be applied if you consider different tests, different experimental protocols. For example, now we just assume we do one measurement, but in a similar way to what we did for the concalorimeter, we could also assess how many tests we should actually perform.

Wojciech Wegrzynski:

And the cost of failure, that's the the collapse of the structure. I assume this means that with the Eurocode method, you have predicted that the temperature of the river has not yet reached 600 degrees, while in reality, in that fire it did reach, which means there's a possibility of failure in the future due to that fire itself, and this is a hidden cost. Uh in your paper, uh, there was a number that uh you are uh okay if the probability of failure is 3.8%. So in the end, you look uh how much of the distribution of your outcomes uh exceeds or is below 3.8% of the 600 degree isotherm, I guess. Can can you comment because also like the the fact it it has been above yet doesn't mean failure, it's just a criterion you set, but you have to assign a cost to that failure, and that probably influences the value of information of the tools, which in this case turns your general considerations of a hammer and discoloration to a specific case because you're toying it to a very specific potential cost of failure.

Andrea Franchini:

Exactly. Yeah, you that you need a model for costs, so you need a model for the repair cost, and you need a model for the cost of failure. And this is very specific to the building you're considering. So in our case, we've made a simplified assumption, which is we assume that the cost of failure is 10 times the cost of repair. And that's a choice we made for illustrative purposes of this example. But as soon as you have more details on your building, you can build up in complexity and include other parameters in the cost assessment.

Wojciech Wegrzynski:

Very good. I I think I finally understood the difference between a sensitivity study and a value of information study, because this economic model part is the one that that really turns a general consideration into a case-specific consideration. If you have a historic building of an immense value and you're not allowed to do repairs, and the cost of collapse will be absolutely tremendous, uh then uh probably this is uh way more informative than if you have a single-story uh house or or you know some other building that that perhaps the the cost of failure is is not that huge. Fantastic. We've covered the uncertainty, we've covered a bit of the cost, let's perhaps move to the utility. You said that the utility is about three things uncertainty, cost, and environmental impact. I feel that we've touched a lot about the uncertainty. I guess we touched about the cost. How does I know it's not in the paper, but how how does the environmental part come come into play in here?

Andrea Franchini:

Yeah, so in the same way as we assign an economic value to the uncertainty you have in your prior and posterior knowledge, you can also assign an environmental cost to this uncertainty. So, for example, you you have uncertainty about some parameters, with this uncertainty you predict the performance of a building, and in the case of a fire, you will have an environmental impact due to the collapse of this building or this part of the building, and that's your environmental cost with prior knowledge. Then if you estimate the outcomes of your experiment, maybe you want to test what is the utility of doing a furnace test, or maybe you want to test the utility of a compressive strength test at high temperature, any test you want, you can calculate what is the expected environmental benefit of the information that you would get from this test. And the calculation works pretty similarly to the calculation for the economic cost, but in this case, we want to we need to estimate the environmental consequences of your uncertainty.

Wojciech Wegrzynski:

Through the same logic, you could potentially go even to life and uh health, you know, Fn curves as well. Was it the choice that you're not doing that?

Andrea Franchini:

Um so the the concept is very general. We can define utility in many different ways. In the paper, we wanted to propose three ways that I think make make a case for using this methodology. And we also commented that you can define many other utility metrics, including the one you just mentioned, based on what is the end use of the experiment that you are going to do. And another uh interesting thing is that you don't need to define a single utility metric for your experiment. You can, in principle, define utility in terms of different metrics, for example, reduction of uncertainty, environmental impact, and economic impact. And you can perform the assessment considering these three metrics and use all of them to inform your decision making on the experimental protocol. And if you want to use the framework to optimize your experimental protocol, you can use something called multi-objective optimization to optimize at the same time all these utility metrics that you defined.

Wojciech Wegrzynski:

Okay, yeah, yeah. The case studies uh it's a lot of work and there's a ton of plots in the paper, but they're kind of simple, like the real-world objectives. Like if if world follows uh Rubin's preaching and starts using tests in a different way, in the way that gives information, we we start to talk about complicated design decisions and complicated systems. How badly it complicates the method when you start having those multiple utility functions? Like, does the math get so much more complicated eventually it's like it's ridiculous, or is doesn't like you're the you're the only one I know who does the math of for this. So tell me how much how how much worse it gets when you start to mess with it.

Andrea Franchini:

Yeah, so the computational cost of this analysis is a relevant aspect to account for, and we need models to represent both the experiment and the performance that we're interested in understanding. So the more complex the system and experiment is, the more complex is the model. And in this sense, you need more advanced models. So one way to hide computational efficiency is using surrogate models. And one part of the uh of the FARTES project is building these throughgate models to hide computational efficiency. Then the other part is uh what utility metrics should you actually use, and that's something that we are going to work on more in the future. So for for now, we we say okay, you can choose in principle all the utility metrics that you want, and you can use all of them to support decision making. Then how do you do that? You have different approaches, you have multi-objective optimization if you want to optimize, you have multi-criteria decision making if you want just to compare different metrics. So there are different tools that we can use to support this decision, and that's definitely something that we are going to work on in the future. And as you said, yes, it becomes more complicated as the application is more complicated, because you correctly notice that what we put in the paper are very simplified examples that aim at making the point of the potential of the methodology. So we wanted to show that we are able to estimate before doing an experiment the expected utility of that experiment. So the examples we put there aim at doing this. Then when we go more complex, yes, it requires more elaborated models, but the the concept and idea that we're going to apply remain the same.

Wojciech Wegrzynski:

And again, I I put the model. Can you replace the model with just you know understanding the statistical distribution of typical outcomes of a test? Like, for example, the the conch colorometer, I'll go back to that. Like you, I mean, you can do the model of flame spread over flat surface, you can perhaps do some pyrolysis modeling, uh, GPyro and stuff. I there's there are things you can model to understand what is the scatter of outcomes, perhaps, of material. Though a model like will tend to give you the same uh value based on the same input. So I guess in here you also have to have distributions of the variables that you go into testing, like the density distribution of your PMMA sample, roughness of the surface, I don't know, the amount of impurities in it, whatever. But it's knowing that the the statistical distribution of a test looks more or less like that, a decent first approximation to find those boundaries.

Andrea Franchini:

Yes, yes, in part, yes. Let me rephrase this a bit. So the framework is a computational modeling framework. So we need computational models for at least for our experiment. Now, when you go to look into how these models should look like, you can integrate different levels of complexity. And what you expect is that if you have a very accurate model, you will be able to optimize the experimental protocol more and to understand it more, but nothing blocks you to use a very simple model to inform your decision making on the experiment. For example, as you mentioned for the concalorimeter example, our model is a normal distribution of the possible outcomes of the concalorimeter. So that's a very simple model, and you can build in complexity. As you said, we can use an FDS model and we can go of different complexities. So increasing complexity enables you to get probably a better outcome, but you can do the same methodology also with simple models, and you should actually start like that, and then you're building complexity to optimize or support your decision making more. So the complexity of the model is definitely important, but you can use very simple models and you're still informing your experimental protocol decision making, even if you use these simple models.

Wojciech Wegrzynski:

No, no, I found the missing link. Well, you you showed me the missing link. So when you refer to the model, I was thinking the statistical distribution is like a replacement of the model, but it was actually the model you were using, and there were advanced, more advanced models which you have not been using. Okay, but this makes a lot of sense, yes, yeah. So you consider the statistical distribution as a model. That's good. Um, in terms of the test outcomes, I think also one interesting thing that you can get from this approach is that like if I do a fire resistance test, what in the end my client gets is you know a classification. Your slab is 60 minutes rated. That's it. But in here, you can take the information about the minutes, but you can also seek different informations that the test gives you: the row temperature measurements, the temperature at your rebar, the temperature like uh at the surface, the deflection rate, the rate of the deflection, how fast it did, like how did the failure look like? When did the failure occur after those 60 minutes? And uh, knowing that the test gives you this much more information, perhaps if you just looked, okay, I'm doing fire resistance tests and it just gives me 60 minutes, so and the probability it will not pass is like 20%. So there's a distribution that's gonna give me this much information. But if you look into the raw data, suddenly you have a plethora of information that you can tap into. But to do that, you would have to define a separate utility function for all of all these parameters, I guess.

Andrea Franchini:

Yes, exactly. So I think the one benefit you get out of this is that you can do better use of the experimental setups you already have. So, for example, what you mentioned now in uh with the furnace, if you have a computational model that includes your parameters of interest and reproduces the furnace test, you are able to improve your understanding of these parameters, being them, for example, material parameters or um thermal parameters, any parameter you want. And this better understanding of these parameters enables you a better prediction of the performance of the system in the real world outside the lab. So, in this sense, the framework enables you to link subsystem performance testing in the lab to the utility that this test will have in real world application.

Wojciech Wegrzynski:

Brilliant. Ah, one thing about like uh uh you're breaking my paradigm. I I feel uncomfortable with that, but I I see I see potential, you know, because for example, when I run a fire test on the furnace, I know where I have to put the thermocouples because the standard defines it and it tells me how much I need. And sometimes clients would like to add some thermocouples. Now, if you think about uh and I'm making up numbers, let's say a test costs 20,000 euros and a single thermocouple costs you a hundred of Euros, it's like you can spend 2,000 euros extra and place 20 more thermocouples, and potentially you could increase the information you gain from the test exponentially, or perhaps you don't gain any information. So the question is will this give you information or not, and how valuable that information is to you? And based on that, you can you can decide, okay, you know what? I should add 25 more thermocouples in this location, just three in this location, and this location doesn't make any sense for us to increase the cost of the test. This is this is really brilliant when you when you start like this could literally be a service, you know, this could literally be uh a tool used to guide, really. I I I I like it.

Andrea Franchini:

And you can use that to go to your client and show, or to the interest stakeholder, show what would be the benefit of them investing high amount of money in this additional thermocapus, for example. And you can map this to the performance of that the beam that you're testing in the real building where it will be implemented, for example.

Wojciech Wegrzynski:

Or or you can perhaps instead of doing three full-scale slab experiments, just over instrument one of them and do, I don't know, five compressive strength under heat or something like that as a replacement, because that's gonna well. I mean, it it's the beauty and the problem because again, here you're doing what's the best, and you have a scientific way to prove that it's the best choice. And the problem is that it's so incompatible with the current paradigm, which just tells you, you know what, you have to achieve this rating, and you achieve this rating by testing this many samples in these conditions, you know. It could also like be used to assess how useful the the current paradigm is or to show how bad it is. Are are you at the well we were escaping the the papers and we're going into future studies, I guess. But are you are you trying to apply that to showcase how much information the current paradigm gives?

Andrea Franchini:

Definitely, yes. So one part of the project is dedicated to that. We have a Apple stock that is working on to understanding the economic value of the current fire safety paradigm. And then we what we want to do is to compare that benefit that you get now with what you would get if you use this different approach of testing. And the final goal is showing that hopefully this uh approach is beneficial for society. So this links back to the fire test overall uh project, and in that case, adaptive fire testing is basically testing using the methodology and the framework that we discussed now, that's a part of the framework, which enters the whole fire safety demonstration paradigm. So there is the one that you mentioned now, and then you could think of how that should be changed based on this approach, and that's what will come as part of the of the EFCR project.

Wojciech Wegrzynski:

Fantastic. Wow, really good. Thank you. Thank you so much, Andrea, for for bringing uh this up and uh explaining to me. Sorry for being a little slow, but it is difficult when you when you think about this, you know, from uh a different perspective, and it's so much easier for me to run a furnace test and do base theorems. But I I guess I'm the voice of the audience, and uh I assume that's the case for uh many of the fire engineers, and I am highly appreciative that you and Ruben and your team at Gantis is working hard on that because indeed the potential impact of that is is is really big, as expected of an ERC grant, of course.

Andrea Franchini:

Thanks a lot, Marzik. Thanks a lot for inviting me.

Wojciech Wegrzynski:

And that's it, thank you for listening. It was not an easy one, but I think we have provided you with the information in the most uh how to say it, digestible way. It's some tough mathematics, some tough concepts, but uh their final outcome is quite profound because if you have in front of you some choices to be made, and those choices can cost a lot of money, then using methods like ones that Andrea and Ruben propose can can really guide you. And I perhaps was venturing too much away from the case studies that Andrea was proposing, but I immediately see uses of this approach in in so many aspects of fire science. I'm not sure if my concepts are correct in terms of the methodology that has been developed by uh University of Ghent, but I still see the potential because it's it's quite generalizable, it's uh quite useful in many cases. If you are able to design your utility functions, objectives, etc., you can really twist this method into working uh on many, many levels. So after this podcast episode, I hope you have a general idea about how the approach to optimize the information gained from testing works. But if you would really like to learn about how it works, you need to read the papers. And there will be two papers linked in the show notes. One is a shorter one, which is the keynote of Ruben from the ESFSS conference in Ljubljana earlier this year, where Ruben just introduced the method and just gave an overview of the potential of it. And then there is a second paper in a Fire Safety Journal, which is a very in-depth uh jump into the topic with all the mathematics explained based on both examples that we have been discussing in this podcast episode. So both the concolarimetry and both the post-fire concrete assessment are step by step shown in the fire safety journal paper that you can go through, follow, and and see how the implementation looks like, how the math looks like, and what is the final outcome of the method. After this podcast episode, I think it's it's quite valuable to quickly, before you forget, quickly jump into the fire safety journal paper and just look at the case studies. So I leave you with this interesting homework and I expect you here next Wednesday because there is more fire science coming your way again. Thank you for being here with me. Cheers, bye.