De-mystifying health economic model classifications and structures

What is health economic evaluation and which modelling approaches can be used to support market access activities?

In this episode, our expert health economists explain the key strengths and limitations of the most common health economic modelling classifications and structures. Here, Hannah Gillies (Consultant – Health Economics) and Daniel MacDonald (Associate Consultant – Health Economics) de-mystify the different approaches to health economic modelling.

Our specialists explore six of the most common health economic modelling structures used to support market access activities;
 - Decision trees
 - Markov models
 - Semi-Markov models
 - Partitioned survival models
 - Cox regression models
 - Discrete event simulations

This episode was first broadcast as a live webinar in January 2024. For more information or to request a copy of the slides used, please visit: https://mtechaccess.co.uk/de-mystifying-health-economic-models/

Work with our health economists:  https://mtechaccess.co.uk/health-economics/

Subscribe to our newsletter to hear more  news, insights and events from Petauri Evidence 

Transcript

Copy Transcript Copied

0:00

- [Announcer] Welcome to this Mtech Access webinar. At Mtech Access, we provide health economics and outcomes research and market access services from strategy through to implementation. Get in touch today to discuss your market access goals. First, though, I hope you enjoy the webinar.- Hello everybody and welcome to this webinar. Thank you all very much for joining today. The topic of our webinar today is "Demystifying Health Economic Model Classifications and Structures." This will include discussing cohort versus patient-level approaches and we will explain the key strengths and limitations of six health economic model structures, which can be used to support market access activities. So to start off with, we will introduce ourselves. There are two of us here today from the Health Economics team at Mtech Access. So my name is Hannah Gillies, I'm a Consultant Health Economist and I've been working in market access since 2012, designing and developing economic models across a wide range of disease areas. I have experience in modelling with many different model structures including Markov models, discrete event simulations and partitioned survival models, as well as budget impact and peer communication tools. And I'm joined today by my colleague Danny.- Hi everyone and welcome to the webinar. My name's Danny MacDonald, I'm an Associate Consultant here at Mtech. A bit of background about me. So I've got roughly six and a half years worth of experience as a health economist and that covers a broad range of disease areas. I've worked on several HTA submissions covering oncology and non-oncology. These include global models, submissions to NICE and CADITH, so in Canada. My modelling experience covers a lot of cost effectiveness models, budget impact models, and also statistical analysis so such as discrete choice experiments and surrogacy endpoint analysis. So it's a pleasure to have you all with us today for the webinar. And Hannah, if you'd be able to move on to the next slide please? So a bit of background about what we're going to be talking about today. So we're going to start off by providing an introduction to health economic evaluation. So we're very much going to be setting the scene, so starting sort of building blocks, providing a definition about health economic evaluation and then we're going to move into sort of a real world application. How is it applied across the world? What are the key considerations when applying health economic evaluation? We're then going to look at specific modelling approaches. So there's two sort of key distinct types of modelling approaches here. So you've got cohort models and you've got patient-level models. There's also a little bit of overlap there. So what we're going to be doing is taking six specific modelling approaches and discussing them in a bit more detail. We're also going to be looking at hybrid modelling. So that's when you merge two models together when a specific situation requires it. We'll then look at a checklist for the choice of model structure, which have sort of a list of key considerations that we think are useful when approaching a modelling project or a model sort of decision problem. We'll then summarise our key findings and then there'll be a section at the end where we'll be responding to questions at the end. So next slide please, Hannah. So to start off, why is health economic evaluation needed? Well, the definition of health economic evaluation is that it's the comparison of alternative options in terms of their costs and consequences. So consequences in this instance being health benefits. So what are the main health benefits and the costs and how are we able to compare them? So if we can see on the side here, health economic evaluations, it involves appraising the health benefits that can be gained from an intervention and evaluating whether the additional gains provide value for money when compared the best available alternative for the use of a cost effectiveness model. So health economic evaluation is crucial for decision making and it allows for this, it allows health agencies to decide what is the most efficient allocation of money. Which is crucial when you've got, say, a constrained budget system, so say in the UK you've got a certain amount of healthcare expenditure, how can we spend that money efficiently? And that's where health economic valuation comes in. A willingness-to-pay threshold is often used to compare the cost effectiveness of treatments. So with NICE, we have a 30,000 pound QALY measurement. So a QALY being a quality-adjusted life year, which is a measure of health benefit which takes into account the additional life years that can be experienced by introducing an intervention and applies a utility value to that to analyse the quality of life that we receive. What you do then is you're able to estimate an ICER, so an incremental cost effectiveness ratio, and assess whether that falls below the willingness-to-pay threshold. So an ICER is the difference or it's a formula which looks at the difference in the cost between an intervention and a comparative treatment and divides that by the difference in the health benefits of the QALYs. What you'll then able to sort of generate is a cost per QALY measurement. So say let's say we've got a 25,000 pound QALY measurement, you compare that to the willingness-to-pay threshold and if it falls below the threshold then it's deemed as cost effective. If it falls above the threshold then it is deemed as not being cost effective. If we look at the righthand side here, the side there are three sort of key areas where cost or where health economic evaluation is specifically needed. So as mentioned, there's increased pressure on budgets, so justification of expenditure within the healthcare system is very important. It allows you to assist decision-making bodies. So it provides a quantitative framework for evaluating technologies. It provides a number in addition to the qualitative submission that might be provided by a pharmaceutical company. And this very much gives you sort of a data point from which you can base several discussions and then explore reasons for why the cost effectiveness value's being reached. It's also allows to support interventions. So a robust model that can describe a real world setting can be crucial in terms of A HTA submission. It can support an argument or it can generate an argument which can then be submitted to the health agency and it can very much frame and begin a discussion for why a technology should be approved. The next slide please, Hannah. So around the world, health economic evaluation is a key and crucial component. As you can see here on the slide, there are five sort of areas we've called out of different ways that economic evaluation can be used. So I'll start especially for those that's on audio readout. So we've got a budget-impact focused approach. So often when you are performing economic evaluation, at least with NICE in particular, you look at a cost effectiveness model and you also look at a budget impact model. The cost effectiveness model usually takes more precedence over the budget impacts model, but it's still there as an analysis of importance for health agencies. In some markets the budget impact aspect has more prominence and it's of more importance, say than the cost effectiveness output. So we can see here markets such as Italy and Spain, for example, specifically look at those areas. There's patient led approaches, so that's in particular looking at Indonesia and Vietnam which look at patient sort of level outcomes or patient focused outcomes. And that's incorporated as a sort of crucial point in the economic evaluation. There's then health-economic or cost effectiveness-focused. So the main example being NICE. So in these instances, a cost effectiveness approach is crucial and is sort of set up as the main modelling approach when looking at economic evaluation. So as mentioned before, looking to achieve cost effectiveness with the ICER relative to a willingness-to-pay threshold. There's then clinically focused countries. So the main example here I'd say is probably France. So with a France, or French market, you've got the ASMR scale which ranks drugs compared to existing treatment options from a clinical point of view. So the first level of ASMR is a major improvement clinically and the fifth scale, it's on a five point scale. The fifth point represents no improvement. And those sort of clinical rankings there are of specific importance when considering the economic evaluation component. And then in America, there's the competitive insurance-based model, which I'm sure we're all familiar with when we've heard it around the news. It's very much sort of an insurer-focused model in terms of a price negotiations. There's probably a webinar in itself talking about the insurance-based model, but it's very unique to the United States. And yeah, it's a very interesting one to go into as yeah, I recommend if you have a look. Okay, next slide please, Hannah. So if we bring it back to modelling and we want to provide a top line overview of why modelling is important in economic evaluation, I think I've displayed it here in this slide and I'll provide a note for you. So cost effectiveness modelling, it underpins health economic evaluation providing an analytical framework to evaluate data and produce a quantitative outcome indicating the cost effectiveness of an intervention treatment against a relative comparator. So there's three key areas of consideration when you are conceptualising a model or when you're looking to approach a modelling project. So we've got the model inputs, the model structure, and the model outputs. So I'm just going to go down them here and give a brief overview of key considerations to have. So in terms of model inputs, I've highlighted four here, so the first one being efficacy. So this is basically, how well does the intervention treatment perform compared to comparative treatment? The efficacy data is usually obtained from a clinical trial that the pharmaceutical company may have run. And then comparative data can either be collected via the clinical trial if it's a head-to-head clinical trial or in some cases if you're using a single arm trial, it may have to be obtained through an indirect measurement. So for example, an indirect treatment comparison which takes at the trial of the comparator, it tries to establish similarities in the standard of care arms that were used in the trial or it tries to adjust for differences in the patient populations between the two trials and it allows you to generate various measurements or ratios which can then be applied to the intervention arms efficacy to estimate a comparator arm. The second area here is adverse events to the safety profile. These are usually collected from the clinical trial and in the model there's usually a threshold of inclusion that's used. So usually in models it could be 3%. So any grade three for adverse event that achieves a 3% sort of instance in the clinical trial would be included and that will also apply for comparator arms. Adverse events are also quite important from a qualitative point of view. If you're able to emphasise the safety profile benefits that can be achieved, it can be beneficial to a submission. There's costs and resource use, so costs key sources say from the UK and NICE would be P-S-S-I-U and NHS reference costs. Resource use are usually applied in health states. So states, so say for example a progression state as we're going to talk about a bit later, you might have a certain amount of resource use and those are accrued in each model cycle. So each duration of time that a model cycle occurs, the resources will be incurred by the patient. Utilities were also consideration here, these are often collected in clinical trials but can also can be picked up from a systematic literature review for the indication of interest. So moving now to the model structure, this is very much the framework which establishes how the inputs are synthesised and turned into outputs. So you're able to get your key sort of output data point. There's key considerations for the structure as we're going to go into. As a top line, I'd say focusing on transitions is quite an important one. So how are patients able to move between health states? Also the incorporation of time in the model in terms of model cycle length and also the time horizon. So how long are you evaluating the costs and benefits over? Usually, a lifetime horizon's used, so that's up until patients are roughly a hundred years old. But in some cases there's an argument that shorter horizons such as 10 or 20 years can be acceptable. Cases like this include when you're looking at a disease area which is quite aggressive in terms of progression and the overall survival. So the number of patients alive in a model decreases quite sharply. So there's no real benefit to having a time horizon of longer than 10 or 20 years in particular from a computational point of view, you want Excel basically to be able to run quite quickly and you don't want to have unnecessary data in there if there's no justification for it. Get outputs from the model, so we've got total costs and total QALYs is the key ones here. So total costs being the cost accrued over the defined time horizon. Likewise for total QALYs. So applying the utility in each model cycle to say, let's say we have yearly cycles and be able to estimate the life years accrued. So let's say it's 0.8 of a life year, so patients are 0.8 of a patient population are alive in let's say the second model cycle. You can then apply utilities to that based on the states that they're in and estimate total QALY from there. The incremental cross effectiveness ratio takes a difference between the total costs of an interventional and comparator and divides that by the total QALYs between an intervention and comparator to get a cost of QALY ratio. And disease-specific outcomes are also often used. These vary depending on the disease area but it could be quite important when you're looking at specific discussions for an indication and they can be very helpful to a submission if collected. Next slide please, Hannah. So this slide talks about the importance of accuracy when reflecting a real world setting. So four key areas which highlight the importance of accuracy include being able to capture clinically relevant endpoints. Examples here being that you want to collect endpoints which are most commonly used in a clinical environment. You want those that are most relevant to the clinical setting and decision makers. It's also important to collect endpoints which are common across clinical trials. So, those of competitors to say if you have a single arm trial, so where it's just your treatment in the trial, if you want to compare it to a treatment that's used in a different trial, but which is the current, say, standard of care, so the current sort of established best practise in an indication, having similar endpoints to that trial would be important to make comparisons of efficacy. The second point here is patient transitions. So assumptions about the health states that patients can move to. An example here would be if you've got a patient that's progressed, so in an oncology setting, a common approach is to have progression-free health states. So when patients are receiving a treatment but haven't progressed from the treatment. And when you have a progressed setting, that's when patients have moved from that progression-free setting to a progressed state, a much deteriorated health state. Patients then can't move from progressed to progression-free usually in most small structures and it's about assessing the applicability of that in a real world setting. Structural assumptions, so incorrect assumptions or structures that are included in the type of model selected, and the flow, the sort of framework. Incorrect use of that can add bias into model outputs through incorrect assumptions and also structural bias. Examples of this being if the transitions, so going back to patient transitions, if you've got a structure which doesn't accurately reflect the real world setting and has incorrect patient movements between states for example, then it will bias the outcomes or it will likely bias the outcomes because you're not accurately reflecting what the real life setting looks like. And then accrual of costs and health benefits. So this focuses on the timings of accumulating the costs and the QALYs. An example, going back to the example of a indication where you've got very aggressive disease, if you see a lot of changes in a clinical or a disease pathway early on, let's say within the first couple of months of a model. However, if you are using a year model cycle so you're only assessing the costs and benefits every year, then you can lose quite a bit of the intricacy that can occur early on in that disease stage. And it's usually, or it is recommended that you capture as much of the sort of granularity of the differences in the disease early on. It's also important with model cycles that they are a risk reasonable length for say computational purposes as well. So for example, if you've got a cycle length of a day, then you are capturing costs and QALYs each day and if you've got a lifetime horizon, that will be a very big burden let's say on a model and it could impact for example, memory size. So it's important to weigh your pros and cons when approaching that. Thanks Hannah. Next slide.- Okay, so it's over to me for the next section. So there are many different ways that health economic models can be classified and often different terminologies can be used to describe the same model structure. So for example, some model structures can be built using either a cohort or a patient-level approach, which means that using the term patient-level model doesn't necessarily describe its structure. Also, for example, a state transition model is an umbrella term of which a Markov model is just one type and a partition survival model is also known as an area under the curve model. So these kind of differences and details can be confusing, especially when you're new to the topic. And although we won't be able to cover all of the possible types, we just wanted to describe some of the key model classifications and structures during this webinar to hopefully bring some clarity to this topic. So firstly we'll describe the difference between a cohort and a patient-level approach. As I mentioned, some structures, some model structures do have the possibility to take either approach, so it's worth bearing that in mind as well. So starting with cohort models, this involves estimating outcomes for a cohort, so a whole average population, without considering outcomes for individual patients. And this is very common approach in health economic evaluation. So a cohort model will take the average population values as their inputs. So for example the mean age of the whole of the population and this will be used to estimate the total population QALYs and costs or the outcomes of the model. The main advantage is that they are very transparent and relatively straightforward to develop, analyse and to communicate. But a disadvantage is that they don't reflect the heterogeneity in a patient population, which is the differences between individual patients because the average value for the whole population is estimated. So in contrast, in a patient-level approach, the outcomes are modelled for individual patients and the results are based on the average of a large sample of simulated patients. The advantage of the patient level approach is that it can capture the relationship between the heterogeneity in the patient characteristics and the model outcomes, but you do require individual patient data in order to develop a patient-level model. Historically, access to this kind of data was limited, but the benefits of the patient-level models have really helped to reduce these data access hurdles. Also, it is worth mentioning that patient-level approaches can be computationally heavy, particularly when performing partition probabilistic sensitivity analysis, which is when the parameters in the model are varied simultaneously to explore the uncertainty of the parameters. So this combined with there being one patient-level data run through the model at a time can cause longer runtimes of these analyses, but there are lots of way to also make this process more efficient. So now that we've discussed these two key approaches to modelling, we're now going to move into explaining our six model structures that we've chosen to go through today and I will hand over to Danny to go through the first one which are decision trees.- Thanks Hannah. So as Hannah mentioned, I'm going to be starting off with decision trees. So decision trees are very much a sort of commonly used approach in health economic modelling. I'd say decision trees and Markov models are probably the most commonly discussed when discussing potential modelling approaches. So to give a bit of an overview of decision tree, it's very much set up in terms of different branches modelling discrete events. So we've got a diagram here. So starting off, the start of a decision tree discusses whether a patient's receiving prophylaxis or not. With a decision tree, what then happens is you reach a chance node which allows you to go via a different or compare two different options. So in this instance, you can go up to no event, so whether you've received an event. Or you can go to the other sort of branch which will go to venous thrombus embolism. With decision trees, what you then do is you then move through a series of events which classify into different categories. The idea being that you continue along the different branches relating to what status you are in until you reach the end of the branch known as the terminal node. So these will be denoted by sort of the triangle symbols at the end, at which point you'll accrue a cost and QALY. So QALY represents a health benefit for that specific branch. There are also probabilities assigned to each specific branch. So the probability of a patient for example going to either no event or to VTE, the probabilities of some to one in that instance. Showing the different options that you have, you can then weight the QALY and cost outcome at the terminal nodes by the probability of a patient ending up at that specific node. And then what you do is you take the weighted average across each of the specific terminal nodes, so weighted by the probability of patients getting to that specific node and then you are able to estimate an estimated or sorry, an expected cost and an expected outcome. So an expected QALY. For specific strengths and limitations to decision trees. So if we start with a strengths, they provide a clear and transparent representation of the decision making process. So in instances such as the one that we have here with the diagram, it's clear to map out the specific options that a patient has at each point and you're able to clearly and transparently move along the model to reach the state at which the patient's at. They are highly adaptable and can model a wide range of healthcare scenarios. So these can be commonplace across different as say indications, not even related to sort of healthcare-specific cross factors modelling. They are very commonly used. They're a logical sort of flow diagram approaches and they're very, as I said, adaptable across different scenarios. With decision trees there are some key limitations. So the first one being that continuous variables such as health states with continuous sort of parameters being analysed are difficult to model. So for example, if we have a sort of chronic disease that we're looking at patients that occur or reoccur different sort of clinical outcomes, on a model cycle basis, that'd be quite difficult because the decisions tree only specifically looks at discrete events that we're looking at. So modelling long term is difficult. Another limitation is that when the decision tree becomes quite complex, so there's specific, there's loads of different branches that are required. So let's say if we expanded this three or four times over, it can be quite complicated and complex to look at. In which case models such as a Markov model might be used where you're able to identify a couple of health states and then you are able to define them and cover sort of each of the different potential events that a patient may have. So for the next model structure that we'll be discussing, Hannah's going to move into Markov models. And Markov models, like I said with decision trees, are two of the probably most common approaches discussed. I'd probably say Markov models slightly more than decision trees, but I'll allow Hannah to go through and discuss it.- Yeah, that's great, thank you. So yeah, as Danny said, you're probably familiar with Markov models as they are quite common. And just to mention that they can take either a cohort or a patient-level approach. And when a patient-level approach is taken for a Markov model, this can sometimes be referred to as a microsimulation. Okay, so Markov models use health states to represent all possible mutually exclusive and exhaustive disease health states. Each health state is associated with a distinct cost and health outcome, for example, utility. The time is considered in discrete time periods called cycles and the movements from one health state to another are represented as transition probabilities which occur at the start and the end of each cycle. So if we look at the example on the right hand side, this is an example of a simple three state Markov model. In this example we have the blue circles representing the health states and the three that we have here are progression-free, progressed disease and dead. And this is a very common Markov model used in modelling of oncology and cancer treatments where patients can be categorised into these health states. And the arrows on the diagram represent where the patient movement can happen from one health state to another and the probability of the movement will be based on the transition probabilities. You can also see that some of the arrows go around into themselves and this represents the probability of the patients remaining within that health state in that cycle. So, as I mentioned, the probability of transition to any particular state depends solely on the current state and the time elapsed. It doesn't depend on the sequence of events that preceded it. And this property is often referred to as the memoryless property or the Markovian assumption, which is a very important underlying assumption of Markov models. They do typically use a cohort approach where the costs and health outcomes are aggregated over the model time horizon to provide a summary of the cohort experience and then the outcomes of this are compared to those of another cohort where you are receiving a different intervention to make that comparison. But as I mentioned, it is possible to run one patient's data through the model at a time to achieve a patient-level approach as well. And so the strengths are that it's intuitive and very widely used in health economic evaluations. It's simple to estimate the costs and health benefits on a per cycle basis. So for example, if you have a monthly cycle, you would need to source the costs and utilities and these can be applied on a monthly basis throughout your model. There are some limitations. The memoryless property that I mentioned is very important. This means that the transitions are independent of the previous cycles and the probabilities of moving from one state to another are therefore assumed to remain constant. So there is a way to overcome this limitation and this is to, one of the ways to do this is to add tunnel states into the model. And a lot of the time, model developers will relax these assumptions by making adjust these kinds of adjustments, but it can quickly become cumbersome and this will be discussed further in our next example which Daniel will explain, which are semi-Markov models.- Thanks Hannah. So as Hannah mentioned, semi-Markov models are an extension to Markov models where they specifically look at the benefits or the look at ways which can overcome that memoryless property. So please going to move on to the next slide. So a semi-Markov model is still built with sort of a similar structure to a Markov model. So there's a diagram on the right hand side here. For people who listen on audio, there's three health states or key health states here. So those are well, mild symptoms or severe symptoms. However, as mentioned with a Markov model, it does not acknowledge the previous states or the sort of history of movement or transition between different states that a patient could have. In some cases there are benefits of incorporating that and that's when a semi-Markov model could be used. So an example of how this can be incorporated is through tunnel states. So as we can see here from the mild symptom state, there's the option of a patient to move into one of two tunnel states which is called recovery. So M1 on the recovery is called mild and then there's also a second tunnel state denoted as M2, which is the second sort of stage of mildness I'll say, or slightly improved treatment outcome. In that recovery state what we are able to do is we're able to say, okay, so if we are in the mild symptom state and we want to show the different sort of stages of recovery, rather than just having one way go from mild say to well or what mild to severe, what you're able to do is you're able to assign these tunnel states, assign costs and QALYs to them, which denote the different sort of levels of recovery you could have and you're able to model these in model cycles. So our resource requirements here, so in a Markov model we have transition probabilities. So the probability each cycle of a patient moving between health states. Having multiple tunnel states would require transition probabilities to be estimated for movement into those states. In this diagram we have six tunnel states, for example. So suddenly going from say three states of well, mild symptoms, and severe symptoms to then having nine states, six of which are recovery-based tunnel states. So you're very much going, you are adding sort of data requirements for the transition probabilities at the not expense, but for the benefit of having additional granularity and detail in the model. So there's several strengths here. So there's flexibility in representing complex health states. So in some cases, health states may have a lot of variation within them. So there might be a lot of heterogeneity of disease within the health states that might be difficult to represent. When you only have a specific health state, you're looking to average the values within that. So you'll be taking the average of the different levels of severity that could occur within a state. If we have different tunnel states, then what you're able to do is you're able to increase the granularity, increase the detail, which allows for a more specific representation of the real world setting and a more specific representation of the disease pathway. It allows accurate representation of time-dependent events. So you can incorporate events which are say conditional on a patient achieving a certain level of health. So in these cases, for example in the recovery M2 state, you can incorporate an event in either the recovery M1 or recovery M2 and you're able to reflect how a patient would respond to that specific event. It might be difficult to incorporate these events if you only have say a three state model, because of these sort of, as I said, variation and heterogeneity in the patient population. These also build into longer term predictive power. So there are more, you'd assume, a more accurate representation of the disease pathway in real world setting which adds to the robustness of a model and adds I'd say to the argument, if you are approaching sort of a HTA submission, for example. The limitations here, the amount of health states you have adds a level of complexity as opposed to a three state model, which would be seen as more sort of simple version. The data requirements can be burdensome and requires a sufficient level of data to be collected from a clinical trial. Also, if you're looking at specific tunnel states, you've got to apply a cost, utility and resource use to those states. So it'll be important to identify for example through a literature search data which is most appropriate for that specific disease state. The interpretability and transparency of the states. As we were discussing with the decision trees, when you have quite a lot of say branches in that instance it can be quite difficult to communicate the differences and it can also look quite messy at times. That's also reflective here. If we have multiple health states, it can be quite complex to discuss for different transitions that can occur between the states. The computational demand should also be considered here. So in Excel, if you're building a Markov model for example, you'd usually have three, say so for example, in this instance, you'd have three health states which can be corresponding into the model engine which should have three states, three lots of transitions to and from. For example, in this case if we have nine states, that triples the sort of burden computationally in the model, which can add to the size the Excel model and add to the sort of difficulty. Or not difficulty, I'd say add to the speed. So for example, if you're running simulations later on, it can add to the time because you want to go through triple the amount of data and movement. So for the next model concept, Hannah's going to be discussing the discrete event simulation, which starts to move on from that sort of cohort-level approach and starts specifically looking at patient level. So specific events and simulations that occur.- Yeah, okay great, thank you. So discrete event simulations are one of my personal favourites and they represent one way of overcoming the challenges of Markov or semi-Markov models when the time dependency is really important. A discrete event simulation is inherently a patient-level model because of the way that it's set up and its structure, which we will get into now. So a discrete event simulation is a type of patient-level simulation model. The time to events are sampled in discrete time, which can be more realistic than moving patients through the model in set cycles as you would in a Markov model. So if I talk through the example model schematic on the right hand side, this will help better explain how the discrete event simulation works computationally. So the model starts by sampling one patient from individual patient-level data. So you take that one patient's data points and their characteristics are stored. So this can be things like their age other characteristics related to the disease, their gender. And then based on these characteristics for that one patient, the time to different events are estimated. This is usually based on regression equations. So these are predictive equations that take inputs such as the patient characteristics and predict an output based on those characteristics. And in this example, the model estimated the time to discontinuing treatment, the time to a change in CD4, which is a measure used in HIV, and the time to death. And then whichever of these three times is the shortest, this event is determined to be the next event that happens. Then the relevant costs and QALYs are calculated based on this and the amount of time that has passed. So whereas before in for example a Markov model, the costs and QALYs would always be based on the same, for example, monthly cycle. Here you have to take into account the time that has passed because the time to event can be changing every time it's estimated. So then if the patient dies, they exit the model. If they discontinue, they switch to the next treatment. And this is useful to know if you want to model treatment sequences, then a discrete event simulation is a great way to be able to incorporate that flexibility. And then if a CD4 change occurs, then the times are all re-estimated and the process continues until the patient has exited the model. Then, every time a new event is estimated, it will depend on the time elapsed, the patient's characteristics, which can be updated over time. So their age will increase accordingly. If an event has happened that can potentially be recorded and the information can be used to predict what happens next. And this is then, this whole process is then repeated, for every patient that goes through the model. So using the individual patient data and then is all aggregated at the end. So just to summarise that again, so the event likelihoods can be determined by individual patient characteristics. These are recorded at baseline and they can be updated over time. The costs and QALYs are estimated and stored when an event happens. And this takes into consideration the amount of time that has passed since the last event. The results are then aggregated over time and for all the patients, to produce the summary experience for the whole cohort. And discrete event simulation is therefore likely to be really useful for modelling complex conditions where there are many different types of events and health states. And one example of this could be the complications of diabetes when there can be different events that the patient is at risk of, for example, cardiovascular events such as strokes and heart attacks and situations, also where the patient's history impacts the future events. So the strengths are that it can account for the patient heterogeneity with it being a patient-level model. The flexibility can be applied from modelling multiple events and also for taking into account the time dependency. However, the main limitation is that it does rely heavily on data and is more complex. So whereas the other models we've discussed so far could be typically developed in Excel, a discrete event simulation would need to use a programming language. One way to do this is to use Visual Basic or VBA, which is Excel's inbuilt programming language. And this allows for Excel to still be used as an interface for the inputs and the VBA programming is being used in the background. The discrete event simulation does rely heavily on having access to patient-level data and that being available to use in the model. And because of that, it can sometimes limit this approach. So if the patient-level data isn't available, then another alternative might be needed. And one of these that we wanted to discuss today is the Cox regression approach, which I will hand over to Danny to walk you through.- Thanks Hannah. So as discussed, the Cox regression approach could be a alternative method used to discrete event simulation. So on to the next slide, thank you. So a Cox regression model, I'd say, is not incredibly commonplace in HTA submissions. There isn't usually a fantastic precedent out there, but it's a very intuitive and interesting approach to consider. From my personal experience, for a previous submission I worked on a model which incorporated a Cox regression approach and it was very useful and logical let's say in incorporating events in the specific indication. So for example, if we're talking about say a cardiovascular indication where you've got significant amount of cardiovascular events which should be incorporated, the Cox regression approach would allow you to incorporate the different events and use them to estimate the likelihood for specific events. So for example, death, the likelihood of that occurring. So with those cardiovascular outcomes, an example, if we take heart failure, what the Cox regression approach does is it takes a series of covariates, so in a regression sort of form of equation and uses it to estimate the likelihood of a heart failure event occurring. In the example that I've used, we use this for several events. So we worked out, for example, heart failure and cardiovascular-related mortality. We were able to work out the likelihood of each event occurring and then what we're able to do is estimate the cumulative hazard. The hazard being the likelihood of an event in an intervention group relative to the comparator. And using that we were able to estimate transitions for patients over time. We're able to estimate the proportion of patients dying'cause we're able to work out okay, what are the impacts of the individual covariates or so for example, age or demographic or smoker status for example, what's the impact of those different factors on the likelihood of an event of death taking place? We could then incorporate that into a model sort of matrix format to estimate okay, what the number of patients done at each stage were. We were also able to incorporate the different events, so the likelihood of a patient having heart failure for example, and we're able to identify costs for that. And we're able to model, okay, so how many patients are experiencing a number of events, can we assign a cost to that, a utility to that? And it allowed a reflective modelling process to be undertaken for that specific indication, looking at cardiovascular outcomes. There's various strengths to the modelling approach. So it allows you to look at the association between the survival time of patients and one more predictive variables to assess, okay, what's the likelihood, what happens if we increase the likelihood of one variable having, what impact will that have on survival? It allows the incorporation of patients that are yet to experience an event. We're able to simulate basically what the impact of each varying each covariate would be on the likelihood of survival and use that to incorporate future patients, the expectation of what would happen to them on the likelihood of them receiving an event. There are a couple of limitations. So it doesn't incorporate the changes in risk over time. So with the hazard rate, we assume what hazard ratio, we assume that patients experience an event with a fixed likelihood in an intervention group relative to a comparator. An example being a hazard ratio of 1.5, would assume a 50% increase in the likelihood of an event in the intervention arm relative to the comparator. This won't change over time in the model. The assumption of proportional hazards is often not realistic. So proportional hazards looks at the hazard rates, the likelihood of an event between an intervention and a control group and it assumes that it remains proportional over time. Often the hazard rates at the likelihood of an event occurring can vary in a intervention group, say relative to a control arm. And this assumption isn't particularly realistic always with the Cox regression approach. The final approach we're going to be discussing, it's one that specifically looks at, I'd say specifically looks at oncology modelling. It's very commonplace used there. It's the partitioned survival model which Hannah's going to discuss further.- Yeah, thank you very much. So yeah, the partitioned survival model is used very commonly within oncology modelling. So for that reason you may be familiar with it, some of you listening today, and it is inherently a cohort approach because as the name suggests, the whole population are partitioned into different health states. So it is worth mentioning that it can also be referred to as an area under the curve approach as well. And what happens in a partitioned survival model is that a cohort are followed through time as they move between a set of mutually exclusive and exhaustive health states, which sounds familiar from the Markov model. However, the difference versus a Markov model is that the partitioned survival model estimates the proportions in each state based on a set of parametric survival equations. So one of the reasons that it is so frequently used in oncology is because it can utilise the endpoints that are commonly collected in clinical trials such as overall survival and partitioned survival curves. So if you look to the example on the right hand side, this is exactly what we've shown here. So using the overall survival and progression-free survival curves, the cohort are partitioned into three sections as you can see from the colour coding. So, above the overall survival curve are the patients who have died and underneath are those who are still alive. And then of the patients who are still alive, this is further proportioned into two separate sections. Underneath the PFS curve is the progression-free patients and the difference between the overall survival curve and the progression-free survival curve gives you the patients who are alive and have progressed. And, so this is a very simple way of using these two curves to partition the cohort into these health states. And what we would also typically need to do is to extrapolate the curve. So these data would be available from a clinical trial which may for example only be for one year duration. And then we extrapolate these to be able to estimate the long-term health effects of the intervention. What we would do is use different distributions and have options available. So these would typically be Weibull, Gompertz, gamma, log-logistic and log-normal. Survival modelling, again, is a different topic, but just to give you an overview of that's the approach that's taken here. So the strengths are that it's widely used in oncology modelling and allows the oncology-focused endpoints to be easily incorporated so that the data from the trial, which typically is the overall survival and the progression-free survival curves can directly be used, which is the the main strength. The limitations are, and it's more of just something to be aware of than a limitation, is that it doesn't include the ability to transition to an improved health state from the current state. So this means that in this example, patients can't move backwards from the progressed disease into back to progression-free. So it's important to make sure that assumption holds and of course this approach can be used in other disease areas as well where the cohort can be partitioned into different health states. And so now that we've described these six modelling types, we want to mention that it's also possible to use a combination of approaches. And I'll hand it back to Danny again to show you, and talk you through how that can be possible.- Thank you Hannah. So we've discussed the six sort of key modelling approaches and structures. However, hybrid modelling tries to approach decision problems where there may not be a specific one of those six which we want to look at. In some cases there's an argument that using two structures at the same time or in within a model sort of overall modelling process is applicable. So in economic modelling it is crucial that the model provides an accurate reflection of disease characteristics and the disease pathway over time. Examples where we may use hybrid modelling are when we have say, response-based models. So models where a patient could experience a specific event and then that is expected to have an implication on the health outcomes for the remainder of the time horizon or potentially the remainder of their lifetime. So in cases such as response-based modelling, one of the modelling approaches may not be sufficient. So what we may use instead is say a decision tree process. So we'd be able to look at the decision tree and say, okay, in each of the branches, what's the likelihood of a patient experiencing a specific response status? So for example, a complete response or a partial response, what's the probability of each of those response status happening? What we could then say is, okay, that's defined over say a month or a yearly time period. Can we then assume that the longer term outcomes can be represented in say a Markov model? So that could be in cases where a decision tree is not appropriate to model over the entire time horizon. Could we then assume that patients go into a specific state in the Markov model and then have the model continue with transitions in a simple Markov process? Hybrid modelling approaches, they're not negatively viewed by NICE, although structure will require justification. As long as it's representing the disease pathway in an accurate and optimal way, there will always be an argument and it's just about being able to communicate the argument that the structure is adequate and a good reflection of a real world setting, in which case it'll be most appropriate. So the next slide please, Hannah. So just building upon the example that I described here, if we had say a response-based model, patients, we'd be able to define that patients for the first half a year or year of a time horizon are in a decision tree health state. Patients are allocated to any of the branches based on the probability of experiencing an event. QALYs and costs are then estimated. And then in if it's assumed that patients then move into a Markov model, so we've got a three state Markov model here as an example, patients can then, in the model, join at time equal zero, but they would join with the costs and QALYs payoffs being incurred in that first period. The distribution of patients across the different states would also be reflective of the decision tree outcomes and the response status they've received. So for example, if we have different response statuses, you may have say multiple engines in the model and the likelihood of a patient being in a progressed state from time equals zero in the Markov model, but in the whole model process it'll be time equals one year could be greater than the likelihood of that happening. If you have patients who have responded to treatment, in which case you would have different numbers of patients in the specific states at the beginning of the Markov approach and they'll then be accruing at different costs and QALYs in terms of the total amounts that are accrued at each model cycle compared to other response statuses that have occurred. Okay, I think Hannah's now going to discuss the taxonomies, the likelihood or the key considerations when selecting a model to choose.- Yeah. Thank you. So the final thing we just want to cover here is like the question, how do you choose a suitable model structure for your decision problem? And there will be numerous factors that guide the decision on which structure to use. One example of a reference here by Brennan et al, 2006 is the taxonomy of model structures for economic evaluation of health technologies that you can see here on the right. And there's some useful considerations in this reference if you do want to go in and have a look at that separately. However, it's important to note that it's not exhaustive and there are some commonly used model approaches that aren't mentioned here. For example, partitioned survival modelling. So we just wanted to finish off with a checklist of 10 things to ask when selecting a model structure. This choice is usually made relatively early in projects, so it is often not possible for model developers to switch to a different model structure and therefore it's really important to thoroughly consider the options and choose an appropriate model structure early on. So first of all, which structure really represents the disease pathway? This is something that we've talked about throughout I think and is one of the most or the first things that should be considered. One way to help with this is looking whether there's any precedent for modelling in the disease area. And then number three is what quantity of data will be available and how long is the follow-up duration? And specifically does this data, does the data available particularly lend itself to a certain model structure? So for example, do you have patient-level data available to be able to do a discrete event simulation? If you have overall survival and progression-free survival data for a partitioned survival model or transition probabilities for a Markov model? Will the population experience events in discrete time? In which case you may want to look at a discrete event simulation or a Cox regression or in continuous time periods, in which case you may want to look at something like a Markov model. Whether there's a large amount of heterogeneity in the patient population will help inform whether to use a patient-level approach. The time horizon is also important to consider as a short time horizon may be more, a decision tree would be maybe more suitable in that case. Do the necessary assumptions hold? So if you are looking at a Markov model, the memoryless property needs to hold, for example. Do risks and transitions change over time? If so, you may need to include a semi-Markov approach or look to do more complex modelling such as a discrete event simulation. And the last point is more of a practical consideration whether a simple approach is needed due to external factors such as time and data restraints. So we did receive lots of questions and thank you very much for submitting these. Because we've run out of time, I think we'll probably look to answer those, follow up with those individually afterwards rather than covering any questions at this point. So yeah, thank you very much and if you have any additional ones to add, please go ahead and do that and we will try our best to reply to them all. And finally, and in case you're wondering how you can get in touch with us to be able to support you, we do offer introductory and intermediate level training sessions on key modelling concepts. Here on the left, you can see some examples of training sessions that we have available. And of course, please do reach out to our team if you want support with developing your next health economic model. We have a wide range of models that we can support with and we'd be happy to get in touch with you here at the Mtech Health Economics team. So you can email us below for next steps or if you want to put a message in the chat now, then we can contact you after the webinar. We'd be really pleased to hear how you found the webinar and to get in touch. So thank you to Danny as well for the presentation and I think that's everything. So, thank you again.- Thank you very much, everyone.- [Announcer] Thank you for watching. If you'd like to find out more about our work or how we could support your market access goals, please email info@mtechaccess.co.uk or visit our website at mtechaccess.co.uk.