{"version":"1.0.0","segments":[{"startTime":0.06,"endTime":1.29,"body":"- Well good afternoon everyone."},{"startTime":1.29,"endTime":4.05,"body":"Thank you for joining us"},{"startTime":1.29,"endTime":4.05,"body":"this afternoon to discuss"},{"startTime":4.05,"endTime":7.44,"body":"the use of real world"},{"startTime":4.05,"endTime":7.44,"body":"evidence in HTA submissions."},{"startTime":7.44,"endTime":8.94,"body":"My name's Sam Gillard,"},{"startTime":8.94,"endTime":12.297,"body":"I'm the director of HTA"},{"startTime":8.94,"endTime":12.297,"body":"here at Mtech Access,"},{"startTime":12.297,"endTime":13.8,"body":"and I'm joined this afternoon"},{"startTime":13.8,"endTime":17.116,"body":"by colleagues from both"},{"startTime":13.8,"endTime":17.116,"body":"Mtech Access, Arcturis"},{"startTime":17.116,"endTime":18.666,"body":"and Delta Hat."},{"startTime":18.666,"endTime":20.941,"body":"And we are delighted to sort of talk you"},{"startTime":20.941,"endTime":22.71,"body":"through our webinar this afternoon."},{"startTime":22.71,"endTime":24.81,"body":"I'm going to briefly hand"},{"startTime":22.71,"endTime":24.81,"body":"over to my colleagues"},{"startTime":24.81,"endTime":26.04,"body":"to introduce themselves"},{"startTime":26.04,"endTime":28.23,"body":"before we get started with the webinar."},{"startTime":28.23,"endTime":30.813,"body":"So I'm going to hand over to Dan first."},{"startTime":32.01,"endTime":32.843,"body":"- Thanks, Sam."},{"startTime":32.843,"endTime":33.96,"body":"Hi, I'm Dan Howard,"},{"startTime":33.96,"endTime":36.823,"body":"associate Director in Health"},{"startTime":33.96,"endTime":36.823,"body":"Economics here at Mtech Access."},{"startTime":36.823,"endTime":38.373,"body":"I've got around 10 years of experience of"},{"startTime":38.373,"endTime":39.206,"body":"technical modelling"},{"startTime":39.206,"endTime":42.54,"body":"and consulting and industry"},{"startTime":39.206,"endTime":42.54,"body":"primarily for HTA submissions"},{"startTime":42.54,"endTime":44.94,"body":"including submissions to NICE, the SMC"},{"startTime":44.94,"endTime":47.52,"body":"and the NCP in Ireland."},{"startTime":47.52,"endTime":50.07,"body":"Most of the submissions I've"},{"startTime":47.52,"endTime":50.07,"body":"worked on have been in oncology"},{"startTime":50.07,"endTime":51.9,"body":"and as a result I have"},{"startTime":50.07,"endTime":51.9,"body":"firsthand experience"},{"startTime":51.9,"endTime":53.85,"body":"of everything that we'll"},{"startTime":51.9,"endTime":53.85,"body":"be discussing today."},{"startTime":53.85,"endTime":55.083,"body":"I'll hand over to Joe."},{"startTime":57.12,"endTime":58.08,"body":"- Thanks, Dan."},{"startTime":58.08,"endTime":59.13,"body":"Good afternoon everyone."},{"startTime":59.13,"endTime":61.672,"body":"My name's Joe and I'm a"},{"startTime":59.13,"endTime":61.672,"body":"principal medical statistician"},{"startTime":61.672,"endTime":63.6,"body":"at Acturis and in this role I design"},{"startTime":63.6,"endTime":65.25,"body":"and implement real world evidence studies"},{"startTime":65.25,"endTime":67.59,"body":"using the secondary electronic"},{"startTime":65.25,"endTime":67.59,"body":"healthcare record data."},{"startTime":67.59,"endTime":69.51,"body":"We have access to Acturis."},{"startTime":69.51,"endTime":70.343,"body":"I'm looking forward"},{"startTime":70.343,"endTime":71.247,"body":"to talking about both real world evidence"},{"startTime":71.247,"endTime":73.29,"body":"and real world data today."},{"startTime":73.29,"endTime":75.59,"body":"I'll hand up to Nick how"},{"startTime":73.29,"endTime":75.59,"body":"to introduce himself."},{"startTime":76.83,"endTime":77.663,"body":"- Thanks, Joe."},{"startTime":77.663,"endTime":79.98,"body":"Hi everyone, I'm Nick Latimer."},{"startTime":79.98,"endTime":81.84,"body":"I'm a professor of health economics"},{"startTime":81.84,"endTime":83.49,"body":"at the University of Sheffield"},{"startTime":83.49,"endTime":87.21,"body":"and also an analyst at Delta Hat."},{"startTime":87.21,"endTime":90.24,"body":"So I'll be talking about"},{"startTime":87.21,"endTime":90.24,"body":"decision maker perspectives"},{"startTime":90.24,"endTime":92.52,"body":"because I've sat on a"},{"startTime":90.24,"endTime":92.52,"body":"NICE appraisal committee"},{"startTime":92.52,"endTime":94.797,"body":"for five years."},{"startTime":94.797,"endTime":96.79,"body":"Back to you, Sam."},{"startTime":96.79,"endTime":98.76,"body":"- Thank you very much."},{"startTime":98.76,"endTime":100.47,"body":"I just wanted to say"},{"startTime":98.76,"endTime":100.47,"body":"at this point thank you"},{"startTime":100.47,"endTime":102.87,"body":"to everyone who's already"},{"startTime":100.47,"endTime":102.87,"body":"submitted questions"},{"startTime":102.87,"endTime":104.73,"body":"ahead of the webinar."},{"startTime":104.73,"endTime":107.49,"body":"Please feel free as we're"},{"startTime":104.73,"endTime":107.49,"body":"going through the the webinar"},{"startTime":107.49,"endTime":110.16,"body":"to add any additional"},{"startTime":107.49,"endTime":110.16,"body":"questions to the chat."},{"startTime":110.16,"endTime":111.84,"body":"And as we get to the end of the webinar,"},{"startTime":111.84,"endTime":114.3,"body":"hopefully we'll have some time to to go"},{"startTime":114.3,"endTime":115.713,"body":"through your questions."},{"startTime":117.09,"endTime":121.398,"body":"So what we're going to talk"},{"startTime":117.09,"endTime":121.398,"body":"about today is we're going to,"},{"startTime":121.398,"endTime":122.25,"body":"I'm going to hand over to Dan to talk"},{"startTime":122.25,"endTime":124.647,"body":"about partition survival modelling for HTA"},{"startTime":124.647,"endTime":128.233,"body":"and some of the common"},{"startTime":124.647,"endTime":128.233,"body":"challenges that are faced."},{"startTime":128.233,"endTime":130.483,"body":"Then we're going to talk"},{"startTime":128.233,"endTime":130.483,"body":"about how real world data"},{"startTime":130.483,"endTime":131.316,"body":"can be used"},{"startTime":131.316,"endTime":133.44,"body":"to generate external control arms"},{"startTime":133.44,"endTime":135.03,"body":"and as Nick said, he's going to be talking"},{"startTime":135.03,"endTime":137.2,"body":"about sort of some real world examples"},{"startTime":138.15,"endTime":139.53,"body":"of how real world evidence can sort"},{"startTime":139.53,"endTime":141.027,"body":"of support economic modelling"},{"startTime":141.027,"endTime":144.48,"body":"and some insight on how"},{"startTime":141.027,"endTime":144.48,"body":"committees might see that."},{"startTime":144.48,"endTime":147.18,"body":"We loop back around for some"},{"startTime":144.48,"endTime":147.18,"body":"reflections and next steps"},{"startTime":147.18,"endTime":149.19,"body":"and I said hopefully we'll"},{"startTime":147.18,"endTime":149.19,"body":"have about 10 minutes"},{"startTime":149.19,"endTime":151.533,"body":"at the end for some Q\u0026A."},{"startTime":157.23,"endTime":158.973,"body":"So Dan, over to you."},{"startTime":160.8,"endTime":161.833,"body":"- Thanks, Sam."},{"startTime":161.833,"endTime":163.413,"body":"I should have control now."},{"startTime":165.12,"endTime":165.953,"body":"Great."},{"startTime":167.16,"endTime":170.22,"body":"So in my section I'll be doing a refresher"},{"startTime":170.22,"endTime":172.23,"body":"on partition survival modelling"},{"startTime":172.23,"endTime":174.36,"body":"and more importantly"},{"startTime":172.23,"endTime":174.36,"body":"I'll be doing an overview"},{"startTime":174.36,"endTime":176.37,"body":"of the types of challenges that we face"},{"startTime":176.37,"endTime":178.32,"body":"and the evidence that we use"},{"startTime":178.32,"endTime":180.57,"body":"to inform partition survival models"},{"startTime":180.57,"endTime":182.97,"body":"and in particular how"},{"startTime":180.57,"endTime":182.97,"body":"real world evidence can"},{"startTime":182.97,"endTime":184.443,"body":"contribute to these analyses."},{"startTime":188.19,"endTime":191.88,"body":"So partition survival models,"},{"startTime":191.88,"endTime":194.13,"body":"they're an ever more popular method"},{"startTime":194.13,"endTime":195.96,"body":"of modelling cancer in HTAs."},{"startTime":195.96,"endTime":200.07,"body":"Most oncology appraisals for"},{"startTime":195.96,"endTime":200.07,"body":"NICE and HAS use this approach."},{"startTime":200.07,"endTime":202.56,"body":"So the figure for NICE is 54%"},{"startTime":202.56,"endTime":207.56,"body":"of a hundred oncology STAs"},{"startTime":202.56,"endTime":207.56,"body":"conducted between 2013, 2018."},{"startTime":208.02,"endTime":210.51,"body":"STA is a single technology appraisal"},{"startTime":210.51,"endTime":212.457,"body":"utilised partition survival approach."},{"startTime":212.457,"endTime":217.02,"body":"And this figure was 82% for"},{"startTime":212.457,"endTime":217.02,"body":"in France for submissions"},{"startTime":217.02,"endTime":221.013,"body":"to the HAS and that's"},{"startTime":217.02,"endTime":221.013,"body":"between 2016 and 2022."},{"startTime":222.33,"endTime":224.34,"body":"They're used so often"},{"startTime":222.33,"endTime":224.34,"body":"because they're conceptually"},{"startTime":224.34,"endTime":225.63,"body":"really attractive."},{"startTime":225.63,"endTime":228.09,"body":"You only need two outcomes in principle,"},{"startTime":228.09,"endTime":230.34,"body":"which are captured very"},{"startTime":228.09,"endTime":230.34,"body":"frequently in cancer trials."},{"startTime":230.34,"endTime":232.35,"body":"And these are overall survival"},{"startTime":232.35,"endTime":234.27,"body":"and progression-free survival,"},{"startTime":234.27,"endTime":236.13,"body":"which is a composite endpoint of time"},{"startTime":236.13,"endTime":237.633,"body":"to progression or death."},{"startTime":243.12,"endTime":246.81,"body":"With these two outcomes you"},{"startTime":243.12,"endTime":246.81,"body":"can model three health states."},{"startTime":246.81,"endTime":248.31,"body":"So you can see on the graph on the right"},{"startTime":248.31,"endTime":252.27,"body":"we've got progression free,"},{"startTime":248.31,"endTime":252.27,"body":"progressed disease, and death."},{"startTime":252.27,"endTime":253.103,"body":"However,"},{"startTime":253.103,"endTime":254.836,"body":"you're rarely going to come"},{"startTime":253.103,"endTime":254.836,"body":"across complete survival"},{"startTime":254.836,"endTime":256.92,"body":"data where the Kaplan-Meier plots"},{"startTime":256.92,"endTime":259.773,"body":"for OS and PFS reach"},{"startTime":256.92,"endTime":259.773,"body":"zero with no censoring."},{"startTime":260.85,"endTime":263.88,"body":"So in order to estimate what"},{"startTime":260.85,"endTime":263.88,"body":"the full curves would look like"},{"startTime":263.88,"endTime":266.49,"body":"or to rather to estimate them,"},{"startTime":266.49,"endTime":268.95,"body":"we often fit survival models."},{"startTime":268.95,"endTime":271.6,"body":"There are lots of different"},{"startTime":268.95,"endTime":271.6,"body":"methods for extrapolating"},{"startTime":272.46,"endTime":274.29,"body":"these survival curves that can be"},{"startTime":274.29,"endTime":276.33,"body":"with independently"},{"startTime":274.29,"endTime":276.33,"body":"fitted parametric models,"},{"startTime":276.33,"endTime":277.773,"body":"jointly fitted models,"},{"startTime":278.85,"endTime":280.59,"body":"some more complicated ones like flexible"},{"startTime":280.59,"endTime":282.12,"body":"splines and mixed secure models."},{"startTime":282.12,"endTime":284.55,"body":"But they basically all"},{"startTime":282.12,"endTime":284.55,"body":"attempt to do the same thing,"},{"startTime":284.55,"endTime":286.74,"body":"which is estimate long-term outcomes"},{"startTime":286.74,"endTime":288.243,"body":"from incomplete clinical data."},{"startTime":289.86,"endTime":291.36,"body":"These methods are all described in detail"},{"startTime":291.36,"endTime":293.49,"body":"in NICE technical support documents."},{"startTime":293.49,"endTime":294.81,"body":"If you want to go and have a look at them,"},{"startTime":294.81,"endTime":298.14,"body":"which I recommend, that's 14, 21 and 24,"},{"startTime":298.14,"endTime":299.82,"body":"which were all essential"},{"startTime":298.14,"endTime":299.82,"body":"reading if you're going to"},{"startTime":299.82,"endTime":303.003,"body":"submit to NICE and use a partition"},{"startTime":299.82,"endTime":303.003,"body":"survival model structure."},{"startTime":303.84,"endTime":305.79,"body":"Once the curves have been"},{"startTime":303.84,"endTime":305.79,"body":"extrapolated to the end"},{"startTime":305.79,"endTime":307.59,"body":"of the time horizon that"},{"startTime":305.79,"endTime":307.59,"body":"you're going to need to model,"},{"startTime":307.59,"endTime":310.62,"body":"which is typically the"},{"startTime":307.59,"endTime":310.62,"body":"entire lifetime of patients,"},{"startTime":310.62,"endTime":312.97,"body":"you can categorise patients"},{"startTime":310.62,"endTime":312.97,"body":"into health states."},{"startTime":317.64,"endTime":320.1,"body":"So here I've got a different"},{"startTime":317.64,"endTime":320.1,"body":"graphical representation"},{"startTime":320.1,"endTime":321.9,"body":"of a partition survival model,"},{"startTime":321.9,"endTime":324.15,"body":"which illustrates the health states"},{"startTime":324.15,"endTime":325.53,"body":"that we're going to model."},{"startTime":325.53,"endTime":327.0,"body":"These are the exact same ones"},{"startTime":327.0,"endTime":328.7,"body":"that are in the graph on the right"},{"startTime":331.085,"endTime":332.22,"body":"and we've got the proportion"},{"startTime":332.22,"endTime":333.51,"body":"of progression-free patients,"},{"startTime":333.51,"endTime":334.86,"body":"which is just simply calculated"},{"startTime":334.86,"endTime":336.783,"body":"as the area under the PFS curve."},{"startTime":338.94,"endTime":342.81,"body":"Then to work out the proportion"},{"startTime":338.94,"endTime":342.81,"body":"of patients who've died,"},{"startTime":342.81,"endTime":345.837,"body":"that's calculated as"},{"startTime":342.81,"endTime":345.837,"body":"one minus the OS curve"},{"startTime":345.837,"endTime":348.78,"body":"and that leaves us with"},{"startTime":345.837,"endTime":348.78,"body":"the final health state,"},{"startTime":348.78,"endTime":349.92,"body":"which is the number of patients"},{"startTime":349.92,"endTime":351.927,"body":"who progressed but haven't died."},{"startTime":351.927,"endTime":356.4,"body":"And that's calculated as"},{"startTime":351.927,"endTime":356.4,"body":"OS minus the PFS curve."},{"startTime":356.4,"endTime":358.26,"body":"So that very briefly,"},{"startTime":358.26,"endTime":360.63,"body":"has covered a simple"},{"startTime":358.26,"endTime":360.63,"body":"partition survival model."},{"startTime":360.63,"endTime":363.39,"body":"You can have more complicated"},{"startTime":360.63,"endTime":363.39,"body":"structure than this,"},{"startTime":363.39,"endTime":364.83,"body":"but this is the most common structure"},{"startTime":364.83,"endTime":367.13,"body":"that's typically used"},{"startTime":364.83,"endTime":367.13,"body":"the three health states."},{"startTime":371.01,"endTime":372.99,"body":"Right now I'm going to"},{"startTime":371.01,"endTime":372.99,"body":"cover some universal issues"},{"startTime":372.99,"endTime":374.89,"body":"that you'll face in oncology modelling."},{"startTime":378.27,"endTime":381.03,"body":"So as I mentioned, like"},{"startTime":378.27,"endTime":381.03,"body":"while the structure"},{"startTime":381.03,"endTime":383.7,"body":"of these models can be"},{"startTime":381.03,"endTime":383.7,"body":"quite conceptually simple,"},{"startTime":383.7,"endTime":385.92,"body":"the main areas of complexity"},{"startTime":385.92,"endTime":388.92,"body":"and in my opinion the really"},{"startTime":385.92,"endTime":388.92,"body":"interesting part of this"},{"startTime":388.92,"endTime":392.52,"body":"is to do with the data"},{"startTime":388.92,"endTime":392.52,"body":"that informs these models."},{"startTime":392.52,"endTime":394.59,"body":"So even a well-designed phase three trial"},{"startTime":394.59,"endTime":396.15,"body":"can still have issues"},{"startTime":396.15,"endTime":398.25,"body":"like the wrong comparators being included,"},{"startTime":399.21,"endTime":401.703,"body":"desirable comparators being excluded."},{"startTime":403.26,"endTime":405.72,"body":"For example, if you submit to"},{"startTime":403.26,"endTime":405.72,"body":"NICE using an American trial,"},{"startTime":405.72,"endTime":406.95,"body":"you might have comparators which aren't"},{"startTime":406.95,"endTime":408.333,"body":"yet reimbursed in the UK."},{"startTime":409.35,"endTime":411.72,"body":"You can have heterogeneity"},{"startTime":409.35,"endTime":411.72,"body":"in your trial population"},{"startTime":411.72,"endTime":414.12,"body":"versus the target reimbursed population."},{"startTime":414.12,"endTime":417.42,"body":"And you'll definitely almost"},{"startTime":414.12,"endTime":417.42,"body":"always have uncertainty"},{"startTime":417.42,"endTime":420.24,"body":"around your long-term"},{"startTime":417.42,"endTime":420.24,"body":"outcomes for OS and PFS."},{"startTime":420.24,"endTime":421.83,"body":"And this is an area"},{"startTime":421.83,"endTime":424.73,"body":"where real world evidence can"},{"startTime":421.83,"endTime":424.73,"body":"really help fill these gaps."},{"startTime":429.39,"endTime":434.22,"body":"So what to do if you have"},{"startTime":429.39,"endTime":434.22,"body":"irrelevant comparators"},{"startTime":434.22,"endTime":437.58,"body":"or a comparator that"},{"startTime":434.22,"endTime":437.58,"body":"you need isn't included."},{"startTime":437.58,"endTime":439.65,"body":"So as I mentioned previously,"},{"startTime":437.58,"endTime":439.65,"body":"treatment pathways differ"},{"startTime":439.65,"endTime":442.23,"body":"by market and some"},{"startTime":439.65,"endTime":442.23,"body":"treatments will be reimbursed"},{"startTime":442.23,"endTime":444.39,"body":"in one market and not in another."},{"startTime":444.39,"endTime":446.04,"body":"So in these cases you're going"},{"startTime":446.04,"endTime":448.05,"body":"to lack head-to-head evidence."},{"startTime":448.05,"endTime":451.71,"body":"So you'll need to syn"},{"startTime":448.05,"endTime":451.71,"body":"synthesise indirect evidence."},{"startTime":451.71,"endTime":453.63,"body":"Now when it comes to indirect evidence,"},{"startTime":453.63,"endTime":455.55,"body":"HTA bodies typically accept it,"},{"startTime":455.55,"endTime":457.413,"body":"but historically this is focused on"},{"startTime":457.413,"endTime":458.896,"body":"what I'm going to call"},{"startTime":458.896,"endTime":461.58,"body":"traditional evidence synthesis methods,"},{"startTime":461.58,"endTime":464.58,"body":"things like Bucher, indirect"},{"startTime":461.58,"endTime":464.58,"body":"treatment comparisons"},{"startTime":464.58,"endTime":466.56,"body":"and network meta analyses,"},{"startTime":466.56,"endTime":469.62,"body":"which typically require at"},{"startTime":466.56,"endTime":469.62,"body":"least some direct evidence"},{"startTime":469.62,"endTime":472.38,"body":"in order to build your evidence network"},{"startTime":472.38,"endTime":473.48,"body":"with your comparators."},{"startTime":474.48,"endTime":476.73,"body":"However, this isn't always possible"},{"startTime":476.73,"endTime":480.18,"body":"and recently some other"},{"startTime":476.73,"endTime":480.18,"body":"methods which do not require"},{"startTime":480.18,"endTime":482.19,"body":"direct evidence have gained popularity."},{"startTime":482.19,"endTime":484.02,"body":"And these include things like matching"},{"startTime":484.02,"endTime":486.12,"body":"adjusted indirect treatment comparisons,"},{"startTime":486.12,"endTime":488.31,"body":"simulated treatment comparisons"},{"startTime":488.31,"endTime":491.463,"body":"and external control arms"},{"startTime":488.31,"endTime":491.463,"body":"in particular to name a few."},{"startTime":492.3,"endTime":493.35,"body":"These are all approaches"},{"startTime":493.35,"endTime":495.12,"body":"where real world evidence can be utilised"},{"startTime":495.12,"endTime":496.203,"body":"and is really useful."},{"startTime":501.63,"endTime":503.07,"body":"You're also going to have uncertainty"},{"startTime":503.07,"endTime":505.26,"body":"in your long term survival"},{"startTime":503.07,"endTime":505.26,"body":"curve extrapolations,"},{"startTime":505.26,"endTime":507.69,"body":"which I mentioned earlier, this can be due"},{"startTime":507.69,"endTime":509.82,"body":"to having immature trial data."},{"startTime":509.82,"endTime":511.563,"body":"You might also have strange"},{"startTime":509.82,"endTime":511.563,"body":"behaviour if your PFS"},{"startTime":511.563,"endTime":514.113,"body":"and OS curves when they're"},{"startTime":511.563,"endTime":514.113,"body":"estimated independently."},{"startTime":515.01,"endTime":517.352,"body":"For example, these curves can cross"},{"startTime":517.352,"endTime":519.99,"body":"if they're estimated independently,"},{"startTime":519.99,"endTime":522.513,"body":"so your PFS can exceed"},{"startTime":519.99,"endTime":522.513,"body":"OS, which isn't possible."},{"startTime":524.13,"endTime":526.11,"body":"The subsequent treatments,"},{"startTime":526.11,"endTime":527.28,"body":"the impacts of subsequent treatments,"},{"startTime":527.28,"endTime":531.69,"body":"sorry, on OS, can also have"},{"startTime":527.28,"endTime":531.69,"body":"a factor in uncertainty"},{"startTime":531.69,"endTime":534.0,"body":"in your long term extrapolations"},{"startTime":534.0,"endTime":538.38,"body":"and uncertainty in the specific"},{"startTime":534.0,"endTime":538.38,"body":"method that you're choosing."},{"startTime":538.38,"endTime":542.04,"body":"So the the specific model"},{"startTime":538.38,"endTime":542.04,"body":"that you're fitting to it"},{"startTime":542.04,"endTime":544.32,"body":"is always a sticking point overall"},{"startTime":544.32,"endTime":546.93,"body":"that usually a is sticking"},{"startTime":544.32,"endTime":546.93,"body":"point in oncology appraisals,"},{"startTime":546.93,"endTime":548.97,"body":"just an example, like a Weibull model,"},{"startTime":548.97,"endTime":550.623,"body":"gamma, lognormal, et cetera."},{"startTime":554.76,"endTime":557.58,"body":"However, real world evidence"},{"startTime":554.76,"endTime":557.58,"body":"data sets can address"},{"startTime":557.58,"endTime":561.48,"body":"some of these concerns"},{"startTime":557.58,"endTime":561.48,"body":"often by a longer duration"},{"startTime":561.48,"endTime":564.783,"body":"of follow up and more complete"},{"startTime":561.48,"endTime":564.783,"body":"survival data as a result."},{"startTime":571.71,"endTime":575.13,"body":"So another issue is"},{"startTime":571.71,"endTime":575.13,"body":"treatment effect waning"},{"startTime":575.13,"endTime":576.63,"body":"that we need to account for."},{"startTime":576.63,"endTime":579.9,"body":"This is particularly"},{"startTime":576.63,"endTime":579.9,"body":"relevant in immuno-oncology"},{"startTime":579.9,"endTime":582.78,"body":"where trials often don't"},{"startTime":579.9,"endTime":582.78,"body":"have a long enough follow-up"},{"startTime":582.78,"endTime":584.04,"body":"to observe treatment waning"},{"startTime":584.04,"endTime":585.603,"body":"so we need to use assumptions."},{"startTime":588.48,"endTime":591.09,"body":"But we can inform and"},{"startTime":588.48,"endTime":591.09,"body":"validate these assumptions"},{"startTime":591.09,"endTime":593.22,"body":"by using real world"},{"startTime":591.09,"endTime":593.22,"body":"evidence and shed light on"},{"startTime":593.22,"endTime":594.45,"body":"what treatment effect waning might look"},{"startTime":594.45,"endTime":595.773,"body":"like in our population."},{"startTime":599.67,"endTime":602.07,"body":"And another area that I'm"},{"startTime":599.67,"endTime":602.07,"body":"going to have a look at"},{"startTime":602.07,"endTime":604.92,"body":"as well is, while I've been"},{"startTime":602.07,"endTime":604.92,"body":"focusing on clinical outcomes,"},{"startTime":604.92,"endTime":606.12,"body":"the other major components"},{"startTime":606.12,"endTime":609.57,"body":"to a cost effectiveness"},{"startTime":606.12,"endTime":609.57,"body":"model are obviously the costs"},{"startTime":609.57,"endTime":612.21,"body":"and historically micro costing studies"},{"startTime":612.21,"endTime":615.84,"body":"and key opinion leader views in ad boards,"},{"startTime":615.84,"endTime":617.55,"body":"elicitation sessions and interviews"},{"startTime":617.55,"endTime":619.62,"body":"have been used to inform all costs,"},{"startTime":619.62,"endTime":621.69,"body":"but real world evidence"},{"startTime":619.62,"endTime":621.69,"body":"can potentially offer"},{"startTime":621.69,"endTime":622.893,"body":"a solution here as well."},{"startTime":627.96,"endTime":630.81,"body":"So now I'm going to quickly"},{"startTime":627.96,"endTime":630.81,"body":"go over some different types"},{"startTime":630.81,"endTime":634.23,"body":"of evidence which are used in"},{"startTime":630.81,"endTime":634.23,"body":"oncology modelling for HTA."},{"startTime":634.23,"endTime":636.6,"body":"These are going to be"},{"startTime":634.23,"endTime":636.6,"body":"categorised broadly into direct"},{"startTime":636.6,"endTime":639.21,"body":"and indirect evidence, which"},{"startTime":636.6,"endTime":639.21,"body":"will lead onto the next section"},{"startTime":639.21,"endTime":640.31,"body":"that Joe will present."},{"startTime":645.9,"endTime":649.35,"body":"So first of all, the"},{"startTime":645.9,"endTime":649.35,"body":"evidence which companies"},{"startTime":649.35,"endTime":651.96,"body":"are submitting for HTA is changing."},{"startTime":651.96,"endTime":655.35,"body":"Direct evidence is typically"},{"startTime":651.96,"endTime":655.35,"body":"held as the gold standard"},{"startTime":655.35,"endTime":658.53,"body":"with good reason provided"},{"startTime":655.35,"endTime":658.53,"body":"it contains the population"},{"startTime":658.53,"endTime":660.9,"body":"and comparators that you're interested in"},{"startTime":660.9,"endTime":663.333,"body":"and in great enough"},{"startTime":660.9,"endTime":663.333,"body":"quantities and quality."},{"startTime":664.17,"endTime":665.22,"body":"It's often straightforward"},{"startTime":665.22,"endTime":667.65,"body":"to analyse head to head data as well."},{"startTime":667.65,"endTime":670.86,"body":"But this isn't always"},{"startTime":667.65,"endTime":670.86,"body":"the case and especially"},{"startTime":670.86,"endTime":673.02,"body":"in recent years with legislative,"},{"startTime":673.02,"endTime":674.94,"body":"legislative, sorry, changes"},{"startTime":674.94,"endTime":676.98,"body":"which have impacted the trial data,"},{"startTime":676.98,"endTime":678.993,"body":"which we end up submitting to HTA."},{"startTime":680.52,"endTime":682.98,"body":"So some of these changes"},{"startTime":680.52,"endTime":682.98,"body":"include the the innovative"},{"startTime":682.98,"endTime":685.38,"body":"licencing and access"},{"startTime":682.98,"endTime":685.38,"body":"pathway, which was introduced"},{"startTime":685.38,"endTime":686.91,"body":"by the last government with the goal"},{"startTime":686.91,"endTime":689.82,"body":"of speeding up marketing authorisation."},{"startTime":689.82,"endTime":692.49,"body":"The current government have"},{"startTime":689.82,"endTime":692.49,"body":"also cited early approval"},{"startTime":692.49,"endTime":694.2,"body":"of drugs as a goal in their life sciences"},{"startTime":694.2,"endTime":695.643,"body":"strategy in the manifesto."},{"startTime":696.78,"endTime":698.31,"body":"In addition, we've got Project Orbis,"},{"startTime":698.31,"endTime":700.65,"body":"which means that the FDA in the US"},{"startTime":700.65,"endTime":705.33,"body":"and the MHRA can concurrently"},{"startTime":700.65,"endTime":705.33,"body":"review new cancer drugs,"},{"startTime":705.33,"endTime":708.0,"body":"meaning that rather than"},{"startTime":705.33,"endTime":708.0,"body":"UK marketing authorisation"},{"startTime":708.0,"endTime":710.22,"body":"happening after US"},{"startTime":708.0,"endTime":710.22,"body":"marketing authorisation,"},{"startTime":710.22,"endTime":711.24,"body":"this can potentially,"},{"startTime":711.24,"endTime":713.25,"body":"if you go down the Orbis type A route"},{"startTime":713.25,"endTime":714.55,"body":"happen at the same time."},{"startTime":716.49,"endTime":720.51,"body":"And, but why am I talking"},{"startTime":716.49,"endTime":720.51,"body":"about regulatory pathways here"},{"startTime":720.51,"endTime":723.06,"body":"and why does this matter for HTA?"},{"startTime":723.06,"endTime":725.4,"body":"Well in the normal NICE STA pathway,"},{"startTime":725.4,"endTime":728.52,"body":"the first committee meeting is"},{"startTime":725.4,"endTime":728.52,"body":"normally held within a month"},{"startTime":728.52,"endTime":730.89,"body":"of marketing authorisation being granted."},{"startTime":730.89,"endTime":732.15,"body":"So there's a knock on effect"},{"startTime":732.15,"endTime":734.16,"body":"of having earlier regulatory approval"},{"startTime":734.16,"endTime":736.12,"body":"with having earlier reimbursement"},{"startTime":737.58,"endTime":740.22,"body":"and with earlier data,"},{"startTime":740.22,"endTime":743.02,"body":"maybe quite a bit earlier"},{"startTime":740.22,"endTime":743.02,"body":"than we've been used to using."},{"startTime":748.86,"endTime":749.693,"body":"Moving on."},{"startTime":750.87,"endTime":753.69,"body":"So as I just said, HTA"},{"startTime":750.87,"endTime":753.69,"body":"bodies generally prefer"},{"startTime":753.69,"endTime":755.61,"body":"direct evidence for obvious reasons,"},{"startTime":755.61,"endTime":756.84,"body":"but when you use earlier data,"},{"startTime":756.84,"endTime":759.81,"body":"you're often using phase"},{"startTime":756.84,"endTime":759.81,"body":"two data, for example,"},{"startTime":759.81,"endTime":761.52,"body":"which might not have a comparator arm"},{"startTime":761.52,"endTime":763.41,"body":"for technical or ethical reasons"},{"startTime":763.41,"endTime":765.21,"body":"and will often have lower patient numbers"},{"startTime":765.21,"endTime":766.623,"body":"than phase three trials."},{"startTime":768.24,"endTime":771.57,"body":"So in these circumstances you'll need"},{"startTime":771.57,"endTime":774.63,"body":"to rely on indirect evidence,"},{"startTime":771.57,"endTime":774.63,"body":"which comprises of a variety"},{"startTime":774.63,"endTime":777.117,"body":"of techniques which I've"},{"startTime":774.63,"endTime":777.117,"body":"listed on this slide."},{"startTime":777.117,"endTime":778.32,"body":"It's not exhaustive,"},{"startTime":778.32,"endTime":780.54,"body":"these are just a few that I've picked out."},{"startTime":780.54,"endTime":783.54,"body":"Even though direct evidence is"},{"startTime":780.54,"endTime":783.54,"body":"often seen as more reliable,"},{"startTime":783.54,"endTime":785.67,"body":"we can use statistical techniques."},{"startTime":785.67,"endTime":788.58,"body":"For example, matching and"},{"startTime":785.67,"endTime":788.58,"body":"waiting to decrease the impact"},{"startTime":788.58,"endTime":791.493,"body":"of heterogeneity on our study results."},{"startTime":796.17,"endTime":798.87,"body":"So real world evidence data"},{"startTime":796.17,"endTime":798.87,"body":"sets have the potential"},{"startTime":798.87,"endTime":801.18,"body":"to be a lot larger than"},{"startTime":798.87,"endTime":801.18,"body":"anything you would see"},{"startTime":801.18,"endTime":804.0,"body":"in a normal trial and more complete"},{"startTime":804.0,"endTime":805.62,"body":"because we potentially have a longer"},{"startTime":805.62,"endTime":807.057,"body":"duration of follow up as well."},{"startTime":807.057,"endTime":809.22,"body":"And may be subject to less uncertainty"},{"startTime":809.22,"endTime":811.95,"body":"when you're extrapolating"},{"startTime":809.22,"endTime":811.95,"body":"long-term outcomes."},{"startTime":811.95,"endTime":814.08,"body":"NICE themself have"},{"startTime":811.95,"endTime":814.08,"body":"acknowledged the importance"},{"startTime":814.08,"endTime":815.1,"body":"of real world evidence"},{"startTime":815.1,"endTime":816.24,"body":"and have published a framework"},{"startTime":816.24,"endTime":818.61,"body":"to guide the use of it in appraisals."},{"startTime":818.61,"endTime":821.81,"body":"So where does real world"},{"startTime":818.61,"endTime":821.81,"body":"evidence actually fit in,"},{"startTime":821.81,"endTime":824.19,"body":"in the context of what I've just described"},{"startTime":824.19,"endTime":825.69,"body":"in terms of the techniques we can use"},{"startTime":825.69,"endTime":827.29,"body":"and the analyses that we can do?"},{"startTime":831.3,"endTime":834.18,"body":"So here I've just got"},{"startTime":831.3,"endTime":834.18,"body":"a very simple diagram"},{"startTime":834.18,"endTime":837.24,"body":"of some indirect evidence"},{"startTime":834.18,"endTime":837.24,"body":"synthesis methods."},{"startTime":837.24,"endTime":839.88,"body":"I've tried to categorise"},{"startTime":837.24,"endTime":839.88,"body":"them into ability to adjust"},{"startTime":839.88,"endTime":842.228,"body":"for heterogeneity across the top"},{"startTime":842.228,"endTime":843.81,"body":"and the data requirements for each"},{"startTime":843.81,"endTime":846.33,"body":"of these methods in terms of cohort level"},{"startTime":846.33,"endTime":848.883,"body":"versus patient level data down the side."},{"startTime":852.6,"endTime":855.51,"body":"So the top row here covers"},{"startTime":852.6,"endTime":855.51,"body":"network meta analyses"},{"startTime":855.51,"endTime":860.46,"body":"and meta regression and naive"},{"startTime":855.51,"endTime":860.46,"body":"comparisons and Bucher ITCs."},{"startTime":860.46,"endTime":862.41,"body":"Now these are all extremely useful"},{"startTime":862.41,"endTime":864.09,"body":"and well understood methods which make use"},{"startTime":864.09,"endTime":865.35,"body":"of cohort level data,"},{"startTime":865.35,"endTime":867.42,"body":"especially the the network meta analysis"},{"startTime":867.42,"endTime":869.67,"body":"and meta regression methods."},{"startTime":869.67,"endTime":872.695,"body":"But we can't always use these"},{"startTime":869.67,"endTime":872.695,"body":"techniques due to a lack of"},{"startTime":872.695,"endTime":875.828,"body":"head-to-head data, meaning"},{"startTime":872.695,"endTime":875.828,"body":"that you can't form a network"},{"startTime":875.828,"endTime":878.13,"body":"or they might just be less"},{"startTime":875.828,"endTime":878.13,"body":"desirable for whatever reason."},{"startTime":878.13,"endTime":881.01,"body":"For example, I, I would argue"},{"startTime":878.13,"endTime":881.01,"body":"that an naive comparison is,"},{"startTime":881.01,"endTime":883.29,"body":"is going to be less desirable"},{"startTime":881.01,"endTime":883.29,"body":"than any of the other methods"},{"startTime":883.29,"endTime":886.14,"body":"because you can't adjust"},{"startTime":883.29,"endTime":886.14,"body":"for patient characteristics."},{"startTime":886.14,"endTime":887.7,"body":"These methods all have their place,"},{"startTime":887.7,"endTime":890.4,"body":"however they're just not the"},{"startTime":887.7,"endTime":890.4,"body":"focus of today's webinar."},{"startTime":892.5,"endTime":896.37,"body":"Instead we'll be focusing"},{"startTime":892.5,"endTime":896.37,"body":"on these bottom ones"},{"startTime":896.37,"endTime":898.44,"body":"which we've highlighted here."},{"startTime":898.44,"endTime":900.54,"body":"So matching adjusted indirect comparisons,"},{"startTime":900.54,"endTime":903.51,"body":"simulated treatment"},{"startTime":900.54,"endTime":903.51,"body":"comparisons and in particular"},{"startTime":903.51,"endTime":905.73,"body":"and arguably the strongest method here,"},{"startTime":905.73,"endTime":907.68,"body":"external control alarms."},{"startTime":907.68,"endTime":909.48,"body":"These techniques are all data hungry"},{"startTime":909.48,"endTime":910.71,"body":"but they have huge potential"},{"startTime":910.71,"endTime":912.9,"body":"for generating high quality"},{"startTime":910.71,"endTime":912.9,"body":"evidence which can be used"},{"startTime":912.9,"endTime":914.103,"body":"for HTA submissions."},{"startTime":919.53,"endTime":924.53,"body":"So now I'll hand over to"},{"startTime":919.53,"endTime":924.53,"body":"Joe from Arcturis Data,"},{"startTime":924.6,"endTime":926.94,"body":"who's going to talk about"},{"startTime":924.6,"endTime":926.94,"body":"real world evidence synthesis"},{"startTime":926.94,"endTime":928.983,"body":"techniques in a bit more detail."},{"startTime":934.05,"endTime":935.466,"body":"- Yeah, thanks for that, Dan."},{"startTime":935.466,"endTime":936.44,"body":"That was a really nice overview"},{"startTime":936.44,"endTime":938.43,"body":"of the current state of play."},{"startTime":938.43,"endTime":942.18,"body":"So as we've seen real world"},{"startTime":938.43,"endTime":942.18,"body":"evidence may provide solutions"},{"startTime":942.18,"endTime":943.56,"body":"to many issues that arise in the health"},{"startTime":943.56,"endTime":945.57,"body":"technology appraisal process."},{"startTime":945.57,"endTime":946.8,"body":"And in this section of a webinar,"},{"startTime":946.8,"endTime":949.26,"body":"I'd like to take a detailed"},{"startTime":946.8,"endTime":949.26,"body":"look at how real world data"},{"startTime":949.26,"endTime":951.36,"body":"could be applied to achieve this."},{"startTime":951.36,"endTime":953.145,"body":"But what is real world data"},{"startTime":953.145,"endTime":954.525,"body":"and what does it actually"},{"startTime":953.145,"endTime":954.525,"body":"look like in practise?"},{"startTime":954.525,"endTime":957.03,"body":"Well there are multiple ways"},{"startTime":954.525,"endTime":957.03,"body":"in which real world data"},{"startTime":957.03,"endTime":958.2,"body":"is defined in literature,"},{"startTime":958.2,"endTime":961.02,"body":"but most definitions do agree"},{"startTime":958.2,"endTime":961.02,"body":"on the following aspects."},{"startTime":961.02,"endTime":963.39,"body":"So first, while real"},{"startTime":961.02,"endTime":963.39,"body":"world data can be obtained"},{"startTime":963.39,"endTime":964.62,"body":"from several different sources,"},{"startTime":964.62,"endTime":966.54,"body":"so things such as the"},{"startTime":964.62,"endTime":966.54,"body":"electronic healthcare records,"},{"startTime":966.54,"endTime":969.18,"body":"or insurance claims"},{"startTime":966.54,"endTime":969.18,"body":"data, a unifying feature"},{"startTime":969.18,"endTime":971.22,"body":"is of the data has not"},{"startTime":969.18,"endTime":971.22,"body":"been explicitly collected"},{"startTime":971.22,"endTime":973.17,"body":"for a clinical study, but"},{"startTime":971.22,"endTime":973.17,"body":"it's instead been collected"},{"startTime":973.17,"endTime":976.47,"body":"for purposes of things such"},{"startTime":973.17,"endTime":976.47,"body":"as audits or reimbursement."},{"startTime":976.47,"endTime":978.99,"body":"Second, real world data"},{"startTime":976.47,"endTime":978.99,"body":"can take many forms."},{"startTime":978.99,"endTime":981.66,"body":"It may be structured data such"},{"startTime":978.99,"endTime":981.66,"body":"as quantified lab results,"},{"startTime":981.66,"endTime":983.07,"body":"or demographic information,"},{"startTime":983.07,"endTime":984.66,"body":"or maybe unstructured free texts,"},{"startTime":984.66,"endTime":988.38,"body":"so things such as pathology"},{"startTime":984.66,"endTime":988.38,"body":"reports or clinical notes."},{"startTime":988.38,"endTime":990.18,"body":"It's also the case for"},{"startTime":988.38,"endTime":990.18,"body":"real world data be obtained"},{"startTime":990.18,"endTime":992.55,"body":"from a single electronic"},{"startTime":990.18,"endTime":992.55,"body":"healthcare record system,"},{"startTime":992.55,"endTime":994.32,"body":"but may also be"},{"startTime":992.55,"endTime":994.32,"body":"constructed through linkage"},{"startTime":994.32,"endTime":996.48,"body":"across multiple different systems."},{"startTime":996.48,"endTime":998.49,"body":"Third, real world data will likely consist"},{"startTime":998.49,"endTime":1000.44,"body":"of individual patient level data,"},{"startTime":1000.44,"endTime":1001.7,"body":"but it may occasionally take a form"},{"startTime":1001.7,"endTime":1003.5,"body":"of aggregate or summary data"},{"startTime":1003.5,"endTime":1005.42,"body":"or data in this aggregate"},{"startTime":1003.5,"endTime":1005.42,"body":"formats usually going to"},{"startTime":1005.42,"endTime":1007.733,"body":"have less utility than"},{"startTime":1005.42,"endTime":1007.733,"body":"individual level data."},{"startTime":1008.84,"endTime":1010.88,"body":"Fourth, collection of real"},{"startTime":1008.84,"endTime":1010.88,"body":"world data can be either"},{"startTime":1010.88,"endTime":1012.86,"body":"retrospective or prospective,"},{"startTime":1012.86,"endTime":1014.45,"body":"but it should always capture the realities"},{"startTime":1014.45,"endTime":1016.28,"body":"of routine clinical care."},{"startTime":1016.28,"endTime":1019.76,"body":"And finally, just the way that"},{"startTime":1016.28,"endTime":1019.76,"body":"real world data is collected,"},{"startTime":1019.76,"endTime":1021.86,"body":"any analysis that makes"},{"startTime":1019.76,"endTime":1021.86,"body":"use of this type of data"},{"startTime":1021.86,"endTime":1024.32,"body":"will have to be observational in nature."},{"startTime":1024.32,"endTime":1026.12,"body":"But I think it's fair"},{"startTime":1024.32,"endTime":1026.12,"body":"to say the most common"},{"startTime":1026.12,"endTime":1028.671,"body":"and I could believe the most"},{"startTime":1026.12,"endTime":1028.671,"body":"useful real world data sources"},{"startTime":1028.671,"endTime":1031.55,"body":"will be derived from"},{"startTime":1028.671,"endTime":1031.55,"body":"retrospectively collected electronic"},{"startTime":1031.55,"endTime":1033.2,"body":"healthcare records and will consist"},{"startTime":1033.2,"endTime":1035.243,"body":"of individual patient level data."},{"startTime":1036.86,"endTime":1039.53,"body":"So as the quantity and"},{"startTime":1036.86,"endTime":1039.53,"body":"quality of information stored"},{"startTime":1039.53,"endTime":1042.2,"body":"within electronic healthcare"},{"startTime":1039.53,"endTime":1042.2,"body":"records systems grows,"},{"startTime":1042.2,"endTime":1043.1,"body":"the breadth of insight"},{"startTime":1043.1,"endTime":1046.52,"body":"that could be obtained from"},{"startTime":1043.1,"endTime":1046.52,"body":"this data source will grow too."},{"startTime":1046.52,"endTime":1048.35,"body":"The information content of"},{"startTime":1046.52,"endTime":1048.35,"body":"this data is now so great"},{"startTime":1048.35,"endTime":1050.72,"body":"that it often allows us to"},{"startTime":1048.35,"endTime":1050.72,"body":"perform indirect treatment"},{"startTime":1050.72,"endTime":1052.22,"body":"comparisons with clinical study data."},{"startTime":1052.22,"endTime":1053.96,"body":"And this is particularly useful"},{"startTime":1053.96,"endTime":1055.37,"body":"as you've heard in situations"},{"startTime":1055.37,"endTime":1057.47,"body":"where a randomised clinical"},{"startTime":1055.37,"endTime":1057.47,"body":"study may not be possible"},{"startTime":1057.47,"endTime":1060.11,"body":"for ethical or logistical reasons."},{"startTime":1060.11,"endTime":1062.54,"body":"And an increasingly popular"},{"startTime":1060.11,"endTime":1062.54,"body":"approach to performing this type"},{"startTime":1062.54,"endTime":1064.49,"body":"of indirect treatment"},{"startTime":1062.54,"endTime":1064.49,"body":"comparison is for use"},{"startTime":1064.49,"endTime":1067.79,"body":"of something called an"},{"startTime":1064.49,"endTime":1067.79,"body":"external control arm or ECA."},{"startTime":1067.79,"endTime":1070.07,"body":"So what is an ECA?"},{"startTime":1070.07,"endTime":1071.727,"body":"Well, as with the concept"},{"startTime":1070.07,"endTime":1071.727,"body":"of real world data,"},{"startTime":1071.727,"endTime":1073.91,"body":"there are a few ECA definitions"},{"startTime":1073.91,"endTime":1075.23,"body":"that are present in the literature,"},{"startTime":1075.23,"endTime":1079.01,"body":"but broadly speaking, an ECA"},{"startTime":1075.23,"endTime":1079.01,"body":"is a control group consisting"},{"startTime":1079.01,"endTime":1080.78,"body":"of patients who have"},{"startTime":1079.01,"endTime":1080.78,"body":"been sourced from outside"},{"startTime":1080.78,"endTime":1082.52,"body":"of some clinical study of interest"},{"startTime":1082.52,"endTime":1085.7,"body":"who have individual level data available."},{"startTime":1085.7,"endTime":1087.74,"body":"Now this individual level"},{"startTime":1085.7,"endTime":1087.74,"body":"data will have been used"},{"startTime":1087.74,"endTime":1090.32,"body":"to assess each patient for"},{"startTime":1087.74,"endTime":1090.32,"body":"the eligibility criteria"},{"startTime":1090.32,"endTime":1092.3,"body":"of that clinical study of interest."},{"startTime":1092.3,"endTime":1095.18,"body":"The patients who fulfil these"},{"startTime":1092.3,"endTime":1095.18,"body":"criteria will then constitute"},{"startTime":1095.18,"endTime":1096.56,"body":"the external control arm."},{"startTime":1096.56,"endTime":1097.393,"body":"They can act as a control"},{"startTime":1097.393,"endTime":1099.11,"body":"for the clinical study of interest."},{"startTime":1099.11,"endTime":1101.6,"body":"And this is achieved through"},{"startTime":1099.11,"endTime":1101.6,"body":"comparison of outcomes"},{"startTime":1101.6,"endTime":1102.917,"body":"between the external control"},{"startTime":1102.917,"endTime":1104.27,"body":"and the patients who have received"},{"startTime":1104.27,"endTime":1105.47,"body":"an investigational therapy"},{"startTime":1105.47,"endTime":1107.93,"body":"in the clinical study of interest."},{"startTime":1107.93,"endTime":1110.33,"body":"I think to understand the"},{"startTime":1107.93,"endTime":1110.33,"body":"role played by an ECA,"},{"startTime":1110.33,"endTime":1112.28,"body":"it's important to first"},{"startTime":1110.33,"endTime":1112.28,"body":"consider the gold standard"},{"startTime":1112.28,"endTime":1113.69,"body":"of clinical evidence generation,"},{"startTime":1113.69,"endTime":1116.39,"body":"which is a randomised trial."},{"startTime":1116.39,"endTime":1118.67,"body":"In a randomised trial, a"},{"startTime":1116.39,"endTime":1118.67,"body":"control and intervention"},{"startTime":1118.67,"endTime":1119.96,"body":"arm constructed from patients"},{"startTime":1119.96,"endTime":1122.15,"body":"who fulfil the study eligibility criteria"},{"startTime":1122.15,"endTime":1125.57,"body":"with assignment to each study"},{"startTime":1122.15,"endTime":1125.57,"body":"arm being performed at random."},{"startTime":1125.57,"endTime":1127.88,"body":"This ensures that the received"},{"startTime":1125.57,"endTime":1127.88,"body":"treatment depends purely on"},{"startTime":1127.88,"endTime":1129.11,"body":"that random assignment,"},{"startTime":1129.11,"endTime":1131.18,"body":"allowing us to assign"},{"startTime":1129.11,"endTime":1131.18,"body":"a causal interpretation"},{"startTime":1131.18,"endTime":1133.16,"body":"to any differences in"},{"startTime":1131.18,"endTime":1133.16,"body":"outcomes that we observe"},{"startTime":1133.16,"endTime":1134.96,"body":"between these two study arms."},{"startTime":1134.96,"endTime":1136.94,"body":"Now as you've seen,"},{"startTime":1134.96,"endTime":1136.94,"body":"unfortunately a randomised study"},{"startTime":1136.94,"endTime":1137.9,"body":"is not always possible."},{"startTime":1137.9,"endTime":1140.75,"body":"It's due to ethical or"},{"startTime":1137.9,"endTime":1140.75,"body":"logistical constraints"},{"startTime":1140.75,"endTime":1143.21,"body":"and in such cases a single"},{"startTime":1140.75,"endTime":1143.21,"body":"arm trial may instead be used"},{"startTime":1143.21,"endTime":1145.81,"body":"to demonstrate safety and"},{"startTime":1143.21,"endTime":1145.81,"body":"response to novel therapy."},{"startTime":1146.87,"endTime":1147.92,"body":"In a single arm trial,"},{"startTime":1147.92,"endTime":1149.84,"body":"again, a cohort of patients are selected"},{"startTime":1149.84,"endTime":1152.66,"body":"based on their fulfilment of"},{"startTime":1149.84,"endTime":1152.66,"body":"some eligibility criteria,"},{"startTime":1152.66,"endTime":1154.22,"body":"but all patients in the study will receive"},{"startTime":1154.22,"endTime":1156.57,"body":"the investigational therapy."},{"startTime":1156.57,"endTime":1157.79,"body":"There's been recent interest in safely"},{"startTime":1157.79,"endTime":1160.19,"body":"expediting the health"},{"startTime":1157.79,"endTime":1160.19,"body":"technology appraisal process"},{"startTime":1160.19,"endTime":1162.29,"body":"using these results"},{"startTime":1160.19,"endTime":1162.29,"body":"from single arm trials."},{"startTime":1162.29,"endTime":1164.57,"body":"This is achieved by taking"},{"startTime":1162.29,"endTime":1164.57,"body":"the single arm trial data"},{"startTime":1164.57,"endTime":1166.25,"body":"and performing by indirect"},{"startTime":1164.57,"endTime":1166.25,"body":"treatment comparison"},{"startTime":1166.25,"endTime":1168.003,"body":"with some external control of the ECA,"},{"startTime":1168.003,"endTime":1170.54,"body":"a clear option for this process."},{"startTime":1170.54,"endTime":1172.01,"body":"Now in the real world,"},{"startTime":1172.01,"endTime":1173.33,"body":"patients will receive therapy"},{"startTime":1173.33,"endTime":1174.83,"body":"based on clinical decision making"},{"startTime":1174.83,"endTime":1176.57,"body":"and reimbursement guidelines."},{"startTime":1176.57,"endTime":1178.07,"body":"But if the eligibility criteria"},{"startTime":1178.07,"endTime":1180.35,"body":"of the single arm trial trial are applied"},{"startTime":1180.35,"endTime":1181.76,"body":"to these real world patients,"},{"startTime":1181.76,"endTime":1183.08,"body":"we can effectively filter them down"},{"startTime":1183.08,"endTime":1186.02,"body":"to create an external control"},{"startTime":1183.08,"endTime":1186.02,"body":"lab consisting exclusively"},{"startTime":1186.02,"endTime":1188.39,"body":"of patients who could have"},{"startTime":1186.02,"endTime":1188.39,"body":"hypothetically participated"},{"startTime":1188.39,"endTime":1190.13,"body":"in that same single arm study"},{"startTime":1190.13,"endTime":1191.9,"body":"but who've actually received usual care"},{"startTime":1191.9,"endTime":1193.4,"body":"instead of a novel treatments."},{"startTime":1194.48,"endTime":1196.46,"body":"If data from this ECA has been combined"},{"startTime":1196.46,"endTime":1197.75,"body":"with a single arm trial,"},{"startTime":1197.75,"endTime":1200.27,"body":"use of statistical causal"},{"startTime":1197.75,"endTime":1200.27,"body":"inference methods can allow"},{"startTime":1200.27,"endTime":1203.03,"body":"for a comparison of outcomes"},{"startTime":1200.27,"endTime":1203.03,"body":"in which the bias introduced"},{"startTime":1203.03,"endTime":1204.2,"body":"by baseline differences"},{"startTime":1204.2,"endTime":1206.42,"body":"between the cohorts can be mitigated,"},{"startTime":1206.42,"endTime":1208.67,"body":"allowing us to generate"},{"startTime":1206.42,"endTime":1208.67,"body":"insightful indirect treatment"},{"startTime":1208.67,"endTime":1209.963,"body":"comparison evidence."},{"startTime":1211.16,"endTime":1214.04,"body":"So ECA analysis are a"},{"startTime":1211.16,"endTime":1214.04,"body":"particularly impactful application"},{"startTime":1214.04,"endTime":1216.47,"body":"of real-world data in the HTA process."},{"startTime":1216.47,"endTime":1218.077,"body":"And this is because for"},{"startTime":1216.47,"endTime":1218.077,"body":"indirect treatment comparison"},{"startTime":1218.077,"endTime":1220.82,"body":"that's being conducted with"},{"startTime":1218.077,"endTime":1220.82,"body":"an external control arm"},{"startTime":1220.82,"endTime":1222.5,"body":"can really integrate large quantities"},{"startTime":1222.5,"endTime":1225.14,"body":"of representative data"},{"startTime":1222.5,"endTime":1225.14,"body":"from real world sources"},{"startTime":1225.14,"endTime":1227.3,"body":"providing estimates of the"},{"startTime":1225.14,"endTime":1227.3,"body":"effects of a novel treatment"},{"startTime":1227.3,"endTime":1228.68,"body":"relative to one or more therapies"},{"startTime":1228.68,"endTime":1230.53,"body":"currently being used in routine care."},{"startTime":1232.01,"endTime":1233.99,"body":"But how can we actually"},{"startTime":1232.01,"endTime":1233.99,"body":"construct an ECA analysis"},{"startTime":1233.99,"endTime":1236.3,"body":"in practise and what should"},{"startTime":1233.99,"endTime":1236.3,"body":"guide our decision making"},{"startTime":1236.3,"endTime":1238.46,"body":"when applying concepts we"},{"startTime":1236.3,"endTime":1238.46,"body":"see in randomised studies"},{"startTime":1238.46,"endTime":1243.02,"body":"such as eligibility assessment"},{"startTime":1238.46,"endTime":1243.02,"body":"to real world data sources?"},{"startTime":1243.02,"endTime":1245.78,"body":"Well a document that can"},{"startTime":1243.02,"endTime":1245.78,"body":"help guide us in this process"},{"startTime":1245.78,"endTime":1247.79,"body":"is NICE as real world evidence framework."},{"startTime":1247.79,"endTime":1249.11,"body":"This describes best practices"},{"startTime":1249.11,"endTime":1252.01,"body":"for generating high quality"},{"startTime":1249.11,"endTime":1252.01,"body":"evidence from real world data."},{"startTime":1253.13,"endTime":1254.51,"body":"Obviously it covers a range of topics,"},{"startTime":1254.51,"endTime":1255.59,"body":"but fundamental aspects"},{"startTime":1255.59,"endTime":1257.26,"body":"of a real world evidence"},{"startTime":1255.59,"endTime":1257.26,"body":"framework is the use"},{"startTime":1257.26,"endTime":1260.665,"body":"of something called the target"},{"startTime":1257.26,"endTime":1260.665,"body":"trial emulation framework."},{"startTime":1260.665,"endTime":1262.43,"body":"And this is particularly"},{"startTime":1260.665,"endTime":1262.43,"body":"relevant when conducting"},{"startTime":1262.43,"endTime":1264.5,"body":"an indirect treatment comparison."},{"startTime":1264.5,"endTime":1266.99,"body":"So target trial emulation"},{"startTime":1264.5,"endTime":1266.99,"body":"framework explains"},{"startTime":1266.99,"endTime":1269.33,"body":"that analysis utilising an"},{"startTime":1266.99,"endTime":1269.33,"body":"external control should aim"},{"startTime":1269.33,"endTime":1271.07,"body":"to actually emulate the randomised study"},{"startTime":1271.07,"endTime":1272.93,"body":"that ideally would've been performed"},{"startTime":1272.93,"endTime":1274.13,"body":"instead of a non-randomised study"},{"startTime":1274.13,"endTime":1276.35,"body":"if it was actually performed."},{"startTime":1276.35,"endTime":1279.35,"body":"So to achieve this and"},{"startTime":1276.35,"endTime":1279.35,"body":"must consider all aspects"},{"startTime":1279.35,"endTime":1281.3,"body":"of a construction of"},{"startTime":1279.35,"endTime":1281.3,"body":"an external control arm"},{"startTime":1281.3,"endTime":1282.62,"body":"and do our best to align them with"},{"startTime":1282.62,"endTime":1284.06,"body":"that hypothetical randomised study"},{"startTime":1284.06,"endTime":1285.66,"body":"we would like to have performed."},{"startTime":1286.52,"endTime":1288.83,"body":"Of particular interest here, are the ways"},{"startTime":1288.83,"endTime":1291.47,"body":"in which we should perform"},{"startTime":1288.83,"endTime":1291.47,"body":"eligibility assessment,"},{"startTime":1291.47,"endTime":1293.39,"body":"how we should define the study index date,"},{"startTime":1293.39,"endTime":1295.88,"body":"how study outcomes can be"},{"startTime":1293.39,"endTime":1295.88,"body":"harmonised across real world"},{"startTime":1295.88,"endTime":1297.47,"body":"and trial data sources"},{"startTime":1297.47,"endTime":1299.45,"body":"and what our choice of"},{"startTime":1297.47,"endTime":1299.45,"body":"statistical method will be"},{"startTime":1299.45,"endTime":1302.72,"body":"to perform about comparison"},{"startTime":1299.45,"endTime":1302.72,"body":"of clinical outcomes."},{"startTime":1302.72,"endTime":1305.99,"body":"So to demonstrate best"},{"startTime":1302.72,"endTime":1305.99,"body":"practise for ECA construction,"},{"startTime":1305.99,"endTime":1308.57,"body":"I'd like to focus on a fairly"},{"startTime":1305.99,"endTime":1308.57,"body":"recently published study"},{"startTime":1308.57,"endTime":1310.04,"body":"that incorporated many aspects"},{"startTime":1310.04,"endTime":1313.52,"body":"of a target trial emulation"},{"startTime":1310.04,"endTime":1313.52,"body":"framework into its design."},{"startTime":1313.52,"endTime":1315.95,"body":"So in this study the authors"},{"startTime":1313.52,"endTime":1315.95,"body":"compared participants"},{"startTime":1315.95,"endTime":1319.16,"body":"who participated in ZUMA-5,"},{"startTime":1315.95,"endTime":1319.16,"body":"which was a single arm study"},{"startTime":1319.16,"endTime":1321.89,"body":"of axi-cel for follicular lymphoma"},{"startTime":1321.89,"endTime":1323.57,"body":"with an external control arm consisting"},{"startTime":1323.57,"endTime":1325.73,"body":"of patients source electronic"},{"startTime":1323.57,"endTime":1325.73,"body":"health care records"},{"startTime":1325.73,"endTime":1327.35,"body":"who had received routinely used therapy"},{"startTime":1327.35,"endTime":1328.733,"body":"for follicular lymphoma."},{"startTime":1329.63,"endTime":1331.04,"body":"The authors called the ECA cohort"},{"startTime":1331.04,"endTime":1336.04,"body":"for SCHOLAR-5 cohort they"},{"startTime":1331.04,"endTime":1336.04,"body":"compared multiple categorical"},{"startTime":1336.05,"endTime":1338.36,"body":"end times within outcomes"},{"startTime":1336.05,"endTime":1338.36,"body":"between the ZUMA-5"},{"startTime":1338.36,"endTime":1339.773,"body":"and SCHOLAR-5 cohorts."},{"startTime":1340.61,"endTime":1342.83,"body":"To handle differences in"},{"startTime":1340.61,"endTime":1342.83,"body":"these cohorts at baseline"},{"startTime":1342.83,"endTime":1344.51,"body":"authors use propensity score based methods"},{"startTime":1344.51,"endTime":1346.25,"body":"when comparing outcomes"},{"startTime":1344.51,"endTime":1346.25,"body":"and this allowed them"},{"startTime":1346.25,"endTime":1347.93,"body":"to adjust the differences"},{"startTime":1346.25,"endTime":1347.93,"body":"in the demographic"},{"startTime":1347.93,"endTime":1349.19,"body":"and clinical profile of patients"},{"startTime":1349.19,"endTime":1351.41,"body":"across these two study arms."},{"startTime":1351.41,"endTime":1353.24,"body":"Now as mentioned previously, the design"},{"startTime":1353.24,"endTime":1355.28,"body":"of an external control"},{"startTime":1353.24,"endTime":1355.28,"body":"arm could be deconstructed"},{"startTime":1355.28,"endTime":1357.65,"body":"into several concepts, each of which align"},{"startTime":1357.65,"endTime":1359.12,"body":"with the different"},{"startTime":1357.65,"endTime":1359.12,"body":"aspects of a target trial"},{"startTime":1359.12,"endTime":1360.869,"body":"emulation framework."},{"startTime":1360.869,"endTime":1362.33,"body":"And what I'll do on the"},{"startTime":1360.869,"endTime":1362.33,"body":"following slides is detail"},{"startTime":1362.33,"endTime":1363.68,"body":"how this study was conducted"},{"startTime":1363.68,"endTime":1365.06,"body":"and how the study designed aligned"},{"startTime":1365.06,"endTime":1366.05,"body":"with those key concepts"},{"startTime":1366.05,"endTime":1368.05,"body":"from a target trial emulation framework."},{"startTime":1371.87,"endTime":1374.45,"body":"So for this ECA analysis,"},{"startTime":1371.87,"endTime":1374.45,"body":"the first concept"},{"startTime":1374.45,"endTime":1375.95,"body":"to consider is the assessment"},{"startTime":1375.95,"endTime":1377.6,"body":"of each available real-world patient"},{"startTime":1377.6,"endTime":1380.63,"body":"for the eligibility criteria of ZUMA-5."},{"startTime":1380.63,"endTime":1382.01,"body":"Now due for prevalence"},{"startTime":1382.01,"endTime":1384.74,"body":"of missing this in the electronic"},{"startTime":1382.01,"endTime":1384.74,"body":"health healthcare records,"},{"startTime":1384.74,"endTime":1387.59,"body":"it's not always possible to"},{"startTime":1384.74,"endTime":1387.59,"body":"assess all eligibility criteria"},{"startTime":1387.59,"endTime":1389.45,"body":"when using real world data."},{"startTime":1389.45,"endTime":1391.85,"body":"So it's important to identify"},{"startTime":1389.45,"endTime":1391.85,"body":"which criteria can be assessed"},{"startTime":1391.85,"endTime":1394.16,"body":"directly, which cannot be assessed,"},{"startTime":1394.16,"endTime":1396.83,"body":"and which criteria may be"},{"startTime":1394.16,"endTime":1396.83,"body":"assessed using proxy criteria"},{"startTime":1396.83,"endTime":1399.08,"body":"that are constructed specifically"},{"startTime":1396.83,"endTime":1399.08,"body":"to work with the content"},{"startTime":1399.08,"endTime":1400.523,"body":"of real world data sources."},{"startTime":1401.51,"endTime":1403.19,"body":"When generating a SCHOLAR-5 cohort"},{"startTime":1403.19,"endTime":1404.9,"body":"authors were able to directly assess many"},{"startTime":1404.9,"endTime":1407.45,"body":"of the ZUMA-5 eligibility"},{"startTime":1404.9,"endTime":1407.45,"body":"criteria directly,"},{"startTime":1407.45,"endTime":1408.83,"body":"including things such as the number"},{"startTime":1408.83,"endTime":1411.26,"body":"of prior therapy lines"},{"startTime":1408.83,"endTime":1411.26,"body":"each patient had received."},{"startTime":1411.26,"endTime":1412.67,"body":"They couldn't assess criteria such"},{"startTime":1412.67,"endTime":1414.71,"body":"as whether patients were"},{"startTime":1412.67,"endTime":1414.71,"body":"pregnant or breastfeeding."},{"startTime":1414.71,"endTime":1416.84,"body":"This is because this information"},{"startTime":1414.71,"endTime":1416.84,"body":"may not always be easily"},{"startTime":1416.84,"endTime":1419.75,"body":"determined from electronic"},{"startTime":1416.84,"endTime":1419.75,"body":"healthcare records."},{"startTime":1419.75,"endTime":1422.15,"body":"The authors also built a"},{"startTime":1419.75,"endTime":1422.15,"body":"proxy assessment criteria"},{"startTime":1422.15,"endTime":1424.4,"body":"for ECOG performance"},{"startTime":1422.15,"endTime":1424.4,"body":"status being zero or one"},{"startTime":1424.4,"endTime":1426.71,"body":"and they did this"},{"startTime":1424.4,"endTime":1426.71,"body":"converting Karnofsky scores,"},{"startTime":1426.71,"endTime":1429.71,"body":"which is a similar measure"},{"startTime":1426.71,"endTime":1429.71,"body":"to ECOG into ECOG scores"},{"startTime":1429.71,"endTime":1431.27,"body":"where ECOG scores were not available"},{"startTime":1431.27,"endTime":1432.98,"body":"but Karnofsky scores were."},{"startTime":1432.98,"endTime":1434.39,"body":"And then assessing these new observations"},{"startTime":1434.39,"endTime":1436.25,"body":"for eligibility criteria."},{"startTime":1436.25,"endTime":1439.01,"body":"This approach of using proxy criteria"},{"startTime":1439.01,"endTime":1441.77,"body":"where necessary makes best"},{"startTime":1439.01,"endTime":1441.77,"body":"use of all available data"},{"startTime":1441.77,"endTime":1444.323,"body":"and allows for more"},{"startTime":1441.77,"endTime":1444.323,"body":"assessment of eligibility."},{"startTime":1446.27,"endTime":1448.94,"body":"Following the construction of"},{"startTime":1446.27,"endTime":1448.94,"body":"the ECA eligibility criteria,"},{"startTime":1448.94,"endTime":1451.25,"body":"it's necessary to identify"},{"startTime":1448.94,"endTime":1451.25,"body":"when the follow-up period"},{"startTime":1451.25,"endTime":1453.08,"body":"for each rule a patient should start"},{"startTime":1453.08,"endTime":1455.09,"body":"and usually this index date will coincide"},{"startTime":1455.09,"endTime":1458.54,"body":"with the initiation of some"},{"startTime":1455.09,"endTime":1458.54,"body":"comparative therapy of interest."},{"startTime":1458.54,"endTime":1460.7,"body":"In SCHOLAR-5 the"},{"startTime":1458.54,"endTime":1460.7,"body":"comparator was any therapy"},{"startTime":1460.7,"endTime":1462.29,"body":"that was routinely used for the treatment"},{"startTime":1462.29,"endTime":1465.41,"body":"of follicular lymphoma if it"},{"startTime":1462.29,"endTime":1465.41,"body":"was initiated after eligibility"},{"startTime":1465.41,"endTime":1467.48,"body":"was first achieved by a patient."},{"startTime":1467.48,"endTime":1468.89,"body":"This analysis that meant"},{"startTime":1468.89,"endTime":1470.69,"body":"that multiple index dates"},{"startTime":1468.89,"endTime":1470.69,"body":"were actually possible"},{"startTime":1470.69,"endTime":1471.92,"body":"for any patient who had received more"},{"startTime":1471.92,"endTime":1473.66,"body":"than three prior lines of therapy."},{"startTime":1473.66,"endTime":1474.89,"body":"As any subsequent line could actually"},{"startTime":1474.89,"endTime":1476.96,"body":"be used as a comparator."},{"startTime":1476.96,"endTime":1479.27,"body":"So the presence of multiple"},{"startTime":1476.96,"endTime":1479.27,"body":"potential index dates"},{"startTime":1479.27,"endTime":1481.25,"body":"as a common issue"},{"startTime":1479.27,"endTime":1481.25,"body":"encountered when constructing"},{"startTime":1481.25,"endTime":1483.38,"body":"an external control arm and"},{"startTime":1481.25,"endTime":1483.38,"body":"can lead to the introduction"},{"startTime":1483.38,"endTime":1486.65,"body":"of the multiple time bias if"},{"startTime":1483.38,"endTime":1486.65,"body":"not properly accounted for."},{"startTime":1486.65,"endTime":1488.75,"body":"In this study, the authors"},{"startTime":1486.65,"endTime":1488.75,"body":"randomly selected the start"},{"startTime":1488.75,"endTime":1490.13,"body":"of one eligible therapy line"},{"startTime":1490.13,"endTime":1491.357,"body":"as the index date for each patient."},{"startTime":1491.357,"endTime":1493.46,"body":"And this represents one"},{"startTime":1491.357,"endTime":1493.46,"body":"of a number of approaches"},{"startTime":1493.46,"endTime":1495.74,"body":"that are applicable in the"},{"startTime":1493.46,"endTime":1495.74,"body":"target trial emulation framework"},{"startTime":1495.74,"endTime":1497.39,"body":"that can be used to circumvent the issue"},{"startTime":1497.39,"endTime":1498.803,"body":"from multiple index dates."},{"startTime":1500.48,"endTime":1501.98,"body":"Once the index date can be identified"},{"startTime":1501.98,"endTime":1503.09,"body":"for each real world patient,"},{"startTime":1503.09,"endTime":1505.07,"body":"the next step is to"},{"startTime":1503.09,"endTime":1505.07,"body":"capture key eligibility"},{"startTime":1505.07,"endTime":1507.41,"body":"and measured confounded"},{"startTime":1505.07,"endTime":1507.41,"body":"data around that index date."},{"startTime":1507.41,"endTime":1510.62,"body":"So to do this, a capture window"},{"startTime":1507.41,"endTime":1510.62,"body":"is defined around index date"},{"startTime":1510.62,"endTime":1512.36,"body":"and data points that reflect the status"},{"startTime":1512.36,"endTime":1514.52,"body":"of the patient at baseline and retained."},{"startTime":1514.52,"endTime":1516.26,"body":"This step the diagram we can see how ECOG"},{"startTime":1516.26,"endTime":1519.2,"body":"and Karnofsky score data may"},{"startTime":1516.26,"endTime":1519.2,"body":"be captured around index date"},{"startTime":1519.2,"endTime":1521.0,"body":"enabling assessment of whether"},{"startTime":1519.2,"endTime":1521.0,"body":"each real world patient"},{"startTime":1521.0,"endTime":1523.61,"body":"had an ECOG score of"},{"startTime":1521.0,"endTime":1523.61,"body":"less than two index date"},{"startTime":1523.61,"endTime":1525.38,"body":"and consequently whether"},{"startTime":1523.61,"endTime":1525.38,"body":"that patient fulfilled"},{"startTime":1525.38,"endTime":1526.58,"body":"with eligibility criteria"},{"startTime":1526.58,"endTime":1528.92,"body":"for entry into the external control arm."},{"startTime":1528.92,"endTime":1531.92,"body":"Now importantly, multiple"},{"startTime":1528.92,"endTime":1531.92,"body":"relevant data observations"},{"startTime":1531.92,"endTime":1534.17,"body":"may be made during the capture window"},{"startTime":1534.17,"endTime":1536.51,"body":"and rules must be defined"},{"startTime":1534.17,"endTime":1536.51,"body":"during the study design stage"},{"startTime":1536.51,"endTime":1538.85,"body":"to ensure that eligibility is"},{"startTime":1536.51,"endTime":1538.85,"body":"assessed using only the data"},{"startTime":1538.85,"endTime":1540.41,"body":"that's most reflective of the status"},{"startTime":1540.41,"endTime":1542.768,"body":"of each patient at index date."},{"startTime":1542.768,"endTime":1544.04,"body":"In the example presented on this slide,"},{"startTime":1544.04,"endTime":1547.67,"body":"we can see multiple observations"},{"startTime":1544.04,"endTime":1547.67,"body":"captured around index date,"},{"startTime":1547.67,"endTime":1549.53,"body":"but only those are closest to index date"},{"startTime":1549.53,"endTime":1551.57,"body":"and preferentially goes before index date"},{"startTime":1551.57,"endTime":1553.37,"body":"are used to assess that eligibility."},{"startTime":1555.11,"endTime":1557.51,"body":"Once eligibility and confounding"},{"startTime":1555.11,"endTime":1557.51,"body":"data have been captured,"},{"startTime":1557.51,"endTime":1560.03,"body":"it's possible to apply the"},{"startTime":1557.51,"endTime":1560.03,"body":"eligibility criteria themselves"},{"startTime":1560.03,"endTime":1563.75,"body":"and actually identify those"},{"startTime":1560.03,"endTime":1563.75,"body":"patients who will fall by ECA."},{"startTime":1563.75,"endTime":1566.09,"body":"Once the ECA cohort is identified"},{"startTime":1566.09,"endTime":1567.8,"body":"the next step is to"},{"startTime":1566.09,"endTime":1567.8,"body":"construct outcome measures"},{"startTime":1567.8,"endTime":1569.54,"body":"for these patients."},{"startTime":1569.54,"endTime":1570.617,"body":"In the ZUMA SCHOLAR-5 study"},{"startTime":1570.617,"endTime":1573.05,"body":"the authors considered"},{"startTime":1570.617,"endTime":1573.05,"body":"standard clinical trial outcome"},{"startTime":1573.05,"endTime":1574.64,"body":"measures including"},{"startTime":1573.05,"endTime":1574.64,"body":"progression-free survival"},{"startTime":1574.64,"endTime":1577.67,"body":"and overall survival alongside"},{"startTime":1574.64,"endTime":1577.67,"body":"therapy-based proxies"},{"startTime":1577.67,"endTime":1578.777,"body":"of clinical trial outcomes."},{"startTime":1578.777,"endTime":1580.79,"body":"And in this case the"},{"startTime":1578.777,"endTime":1580.79,"body":"authors considered time"},{"startTime":1580.79,"endTime":1582.98,"body":"to next treatment, which"},{"startTime":1580.79,"endTime":1582.98,"body":"often acts as a proxy"},{"startTime":1582.98,"endTime":1584.48,"body":"for progression-free survival"},{"startTime":1584.48,"endTime":1585.68,"body":"as progression is usually closely"},{"startTime":1585.68,"endTime":1586.97,"body":"followed by a change in therapy."},{"startTime":1586.97,"endTime":1589.37,"body":"So if that change in therapy"},{"startTime":1586.97,"endTime":1589.37,"body":"has been taken to be indicative"},{"startTime":1589.37,"endTime":1591.56,"body":"of a progression event having occurred."},{"startTime":1591.56,"endTime":1592.88,"body":"On this diagram though we can see"},{"startTime":1592.88,"endTime":1595.13,"body":"that in certain scenarios there"},{"startTime":1592.88,"endTime":1595.13,"body":"may be a considerable time"},{"startTime":1595.13,"endTime":1597.44,"body":"difference between observed"},{"startTime":1595.13,"endTime":1597.44,"body":"disease progression"},{"startTime":1597.44,"endTime":1599.09,"body":"and any change in therapy that may be used"},{"startTime":1599.09,"endTime":1602.3,"body":"to construct a proxy outcome"},{"startTime":1599.09,"endTime":1602.3,"body":"such as time to next treatment."},{"startTime":1602.3,"endTime":1604.79,"body":"And because of where"},{"startTime":1602.3,"endTime":1604.79,"body":"possible the outcome measures"},{"startTime":1604.79,"endTime":1605.623,"body":"that are compared between trial"},{"startTime":1605.623,"endTime":1607.61,"body":"and real world sources"},{"startTime":1605.623,"endTime":1607.61,"body":"should be constructed"},{"startTime":1607.61,"endTime":1609.38,"body":"using the same criteria."},{"startTime":1609.38,"endTime":1611.33,"body":"This approach is preferable as it allows"},{"startTime":1611.33,"endTime":1612.74,"body":"for a like for like comparison"},{"startTime":1612.74,"endTime":1615.35,"body":"between trial participants"},{"startTime":1612.74,"endTime":1615.35,"body":"and ECA members."},{"startTime":1615.35,"endTime":1618.17,"body":"This reduces the possibility"},{"startTime":1615.35,"endTime":1618.17,"body":"of systematically"},{"startTime":1618.17,"endTime":1620.06,"body":"inflating event times of a control arm,"},{"startTime":1620.06,"endTime":1622.94,"body":"which could bias any"},{"startTime":1620.06,"endTime":1622.94,"body":"estimates treatment effect."},{"startTime":1622.94,"endTime":1625.19,"body":"And at this stage it's"},{"startTime":1622.94,"endTime":1625.19,"body":"also important to determine"},{"startTime":1625.19,"endTime":1626.3,"body":"how censoring will be handled"},{"startTime":1626.3,"endTime":1628.4,"body":"for the external control arm cohort."},{"startTime":1628.4,"endTime":1630.83,"body":"For most the electronic"},{"startTime":1628.4,"endTime":1630.83,"body":"healthcare record based studies,"},{"startTime":1630.83,"endTime":1632.72,"body":"the censoring date will"},{"startTime":1630.83,"endTime":1632.72,"body":"reflect the data which data"},{"startTime":1632.72,"endTime":1636.143,"body":"has been cut by the data provider"},{"startTime":1632.72,"endTime":1636.143,"body":"and received for analysis."},{"startTime":1636.143,"endTime":1637.939,"body":"But it's also important to"},{"startTime":1636.143,"endTime":1637.939,"body":"consider whether patients have"},{"startTime":1637.939,"endTime":1640.789,"body":"become unobservable for any"},{"startTime":1637.939,"endTime":1640.789,"body":"reason before that data cuts."},{"startTime":1642.2,"endTime":1644.63,"body":"And then the final step in"},{"startTime":1642.2,"endTime":1644.63,"body":"ECA analysis is actually about"},{"startTime":1644.63,"endTime":1646.543,"body":"comparison of outcomes between"},{"startTime":1644.63,"endTime":1646.543,"body":"the external control arm"},{"startTime":1646.543,"endTime":1648.263,"body":"and the trial cohorts."},{"startTime":1649.22,"endTime":1650.66,"body":"Importantly in this case study"},{"startTime":1650.66,"endTime":1653.66,"body":"authors did identify a series"},{"startTime":1650.66,"endTime":1653.66,"body":"of confounding covariates"},{"startTime":1653.66,"endTime":1656.15,"body":"that included prior stem cell"},{"startTime":1653.66,"endTime":1656.15,"body":"transplantation, relapse,"},{"startTime":1656.15,"endTime":1657.83,"body":"or refractory status at baseline"},{"startTime":1657.83,"endTime":1660.41,"body":"and the number of prior"},{"startTime":1657.83,"endTime":1660.41,"body":"therapy lines amongst others."},{"startTime":1660.41,"endTime":1661.79,"body":"These covariates have an association"},{"startTime":1661.79,"endTime":1663.35,"body":"with both the treatment assignment process"},{"startTime":1663.35,"endTime":1665.635,"body":"and the measured outcomes,"},{"startTime":1663.35,"endTime":1665.635,"body":"meaning that they were both,"},{"startTime":1665.635,"endTime":1667.49,"body":"they were confounders and that"},{"startTime":1665.635,"endTime":1667.49,"body":"they needed to be controlled"},{"startTime":1667.49,"endTime":1670.24,"body":"for to minimise bias in any"},{"startTime":1667.49,"endTime":1670.24,"body":"estimated treatment effect."},{"startTime":1671.24,"endTime":1673.19,"body":"By capturing these covariates"},{"startTime":1671.24,"endTime":1673.19,"body":"around index state,"},{"startTime":1673.19,"endTime":1674.96,"body":"applying them in a model"},{"startTime":1673.19,"endTime":1674.96,"body":"which produced estimates"},{"startTime":1674.96,"endTime":1676.73,"body":"of propensity scores and"},{"startTime":1674.96,"endTime":1676.73,"body":"then using these scores"},{"startTime":1676.73,"endTime":1679.578,"body":"to re-weigh the contribution"},{"startTime":1676.73,"endTime":1679.578,"body":"of each patient's the analysis,"},{"startTime":1679.578,"endTime":1682.19,"body":"the influence of measured"},{"startTime":1679.578,"endTime":1682.19,"body":"confounders in comparisons"},{"startTime":1682.19,"endTime":1684.83,"body":"of outcomes was mitigated"},{"startTime":1682.19,"endTime":1684.83,"body":"in this case study."},{"startTime":1684.83,"endTime":1686.69,"body":"Now in any ECA study is imperative"},{"startTime":1686.69,"endTime":1687.95,"body":"of a choice of confounders"},{"startTime":1687.95,"endTime":1690.92,"body":"included in the analysis are"},{"startTime":1687.95,"endTime":1690.92,"body":"based on clinical expertise"},{"startTime":1690.92,"endTime":1693.29,"body":"and careful consideration of"},{"startTime":1690.92,"endTime":1693.29,"body":"how all measured variables"},{"startTime":1693.29,"endTime":1694.94,"body":"are associated with treatment,"},{"startTime":1694.94,"endTime":1697.04,"body":"outcome measure but also with one another."},{"startTime":1698.057,"endTime":1701.06,"body":"And in this case, after"},{"startTime":1698.057,"endTime":1701.06,"body":"following key aspects"},{"startTime":1701.06,"endTime":1702.62,"body":"of the target trial emulation framework,"},{"startTime":1702.62,"endTime":1704.72,"body":"the authors did identify"},{"startTime":1702.62,"endTime":1704.72,"body":"a clear treatment benefit"},{"startTime":1704.72,"endTime":1708.14,"body":"for axi-cel versus usual"},{"startTime":1704.72,"endTime":1708.14,"body":"care for follicular lymphoma."},{"startTime":1708.14,"endTime":1710.48,"body":"We have a progression-free"},{"startTime":1708.14,"endTime":1710.48,"body":"survival hazard ratio estimated"},{"startTime":1710.48,"endTime":1713.15,"body":"between 0.18 and 0.49."},{"startTime":1713.15,"endTime":1714.68,"body":"And as you can see on"},{"startTime":1713.15,"endTime":1714.68,"body":"this Kaplan-Meier plot"},{"startTime":1714.68,"endTime":1716.98,"body":"there's clear separation"},{"startTime":1714.68,"endTime":1716.98,"body":"between these curves."},{"startTime":1718.64,"endTime":1721.28,"body":"So we've seen how an"},{"startTime":1718.64,"endTime":1721.28,"body":"ECA can be constructed"},{"startTime":1721.28,"endTime":1722.36,"body":"from real world data"},{"startTime":1722.36,"endTime":1724.7,"body":"and an overview of what"},{"startTime":1722.36,"endTime":1724.7,"body":"constitutes best practice"},{"startTime":1724.7,"endTime":1725.99,"body":"for the design of an ECA study"},{"startTime":1725.99,"endTime":1727.7,"body":"for an indirect treatment comparison."},{"startTime":1727.7,"endTime":1728.81,"body":"But it's important to recognise"},{"startTime":1728.81,"endTime":1730.7,"body":"that real world data can"},{"startTime":1728.81,"endTime":1730.7,"body":"provide important evidence"},{"startTime":1730.7,"endTime":1732.863,"body":"across the wider HTA process too."},{"startTime":1733.82,"endTime":1736.28,"body":"So for example, construction of a cohort"},{"startTime":1736.28,"endTime":1738.35,"body":"of patients receiving"},{"startTime":1736.28,"endTime":1738.35,"body":"a comparative therapy"},{"startTime":1738.35,"endTime":1740.6,"body":"for a given technology"},{"startTime":1738.35,"endTime":1740.6,"body":"appraisal enables clear"},{"startTime":1740.6,"endTime":1742.457,"body":"characterisation of real"},{"startTime":1740.6,"endTime":1742.457,"body":"world comparative use"},{"startTime":1742.457,"endTime":1745.49,"body":"and this includes assessment"},{"startTime":1742.457,"endTime":1745.49,"body":"of use of support medications,"},{"startTime":1745.49,"endTime":1746.99,"body":"healthcare resource utilisation"},{"startTime":1746.99,"endTime":1750.71,"body":"and capture of a wider cost"},{"startTime":1746.99,"endTime":1750.71,"body":"associated with that comparator."},{"startTime":1750.71,"endTime":1753.41,"body":"Similarly, real world data can"},{"startTime":1750.71,"endTime":1753.41,"body":"be used to generate evidence"},{"startTime":1753.41,"endTime":1755.93,"body":"to support specific economic"},{"startTime":1753.41,"endTime":1755.93,"body":"modelling assumptions"},{"startTime":1755.93,"endTime":1757.04,"body":"regarding comparative use"},{"startTime":1757.04,"endTime":1759.49,"body":"that may have a meaningful"},{"startTime":1757.04,"endTime":1759.49,"body":"influence on the ICER."},{"startTime":1760.85,"endTime":1762.44,"body":"Now so far we've seen some potential uses"},{"startTime":1762.44,"endTime":1764.45,"body":"of real world data in the HTA process"},{"startTime":1764.45,"endTime":1765.95,"body":"and how this data should be applied."},{"startTime":1765.95,"endTime":1768.59,"body":"But fundamental to the use of"},{"startTime":1765.95,"endTime":1768.59,"body":"this evidence is establishment"},{"startTime":1768.59,"endTime":1770.06,"body":"of the suitability of the data"},{"startTime":1770.06,"endTime":1773.3,"body":"and statistical methods that"},{"startTime":1770.06,"endTime":1773.3,"body":"have been used to generate it."},{"startTime":1773.3,"endTime":1776.18,"body":"Ultimately, regulatory and"},{"startTime":1773.3,"endTime":1776.18,"body":"reimbursement decision makers"},{"startTime":1776.18,"endTime":1778.61,"body":"must make this assessment and"},{"startTime":1776.18,"endTime":1778.61,"body":"there are many considerations"},{"startTime":1778.61,"endTime":1780.5,"body":"they will make when determining"},{"startTime":1778.61,"endTime":1780.5,"body":"whether the evidence"},{"startTime":1780.5,"endTime":1783.35,"body":"they've been with is of suitable quality."},{"startTime":1783.35,"endTime":1785.39,"body":"So I'll now hand over"},{"startTime":1783.35,"endTime":1785.39,"body":"to Nick from Delta Hat"},{"startTime":1785.39,"endTime":1786.74,"body":"who will talk us through what's involved"},{"startTime":1786.74,"endTime":1788.093,"body":"in that assessment process."},{"startTime":1791.6,"endTime":1793.553,"body":"- Great, thanks very much, Joe."},{"startTime":1794.42,"endTime":1796.463,"body":"I think I've got control now."},{"startTime":1802.04,"endTime":1803.78,"body":"Yeah, okay great."},{"startTime":1803.78,"endTime":1807.29,"body":"So as I said before, I'm a"},{"startTime":1803.78,"endTime":1807.29,"body":"professor of health economics,"},{"startTime":1807.29,"endTime":1808.967,"body":"oh sorry, I'll turn my camera on."},{"startTime":1808.967,"endTime":1810.59,"body":"I'm a professor of health economics"},{"startTime":1810.59,"endTime":1812.36,"body":"at the University of Sheffield"},{"startTime":1812.36,"endTime":1814.553,"body":"and also an Analyst at Delta Hat."},{"startTime":1815.42,"endTime":1818.6,"body":"And I'm going to be giving a"},{"startTime":1815.42,"endTime":1818.6,"body":"reviewer or decision makers"},{"startTime":1818.6,"endTime":1821.57,"body":"perspective on the use of real world data,"},{"startTime":1821.57,"endTime":1825.443,"body":"real world evidence to support"},{"startTime":1821.57,"endTime":1825.443,"body":"economic modelling for HTA."},{"startTime":1826.43,"endTime":1830.748,"body":"So just before I start,"},{"startTime":1826.43,"endTime":1830.748,"body":"some quick disclosures,"},{"startTime":1830.748,"endTime":1832.67,"body":"I was a member of NICE's,"},{"startTime":1832.67,"endTime":1836.0,"body":"one of NICE's appraisal"},{"startTime":1832.67,"endTime":1836.0,"body":"committees for five years."},{"startTime":1836.0,"endTime":1838.76,"body":"I'm a member of NICE's"},{"startTime":1836.0,"endTime":1838.76,"body":"decision support unit."},{"startTime":1838.76,"endTime":1841.76,"body":"Sometimes I work with"},{"startTime":1838.76,"endTime":1841.76,"body":"Sheffield's Technology Assessment"},{"startTime":1841.76,"endTime":1844.43,"body":"group and in fact Delta Hat acts"},{"startTime":1844.43,"endTime":1847.67,"body":"as a NICE external assessment"},{"startTime":1844.43,"endTime":1847.67,"body":"group in collaboration"},{"startTime":1847.67,"endTime":1849.923,"body":"with PenTAG at the University of Exeter."},{"startTime":1850.85,"endTime":1854.51,"body":"Also, I'm focusing my slides"},{"startTime":1850.85,"endTime":1854.51,"body":"on the NICE real world evidence"},{"startTime":1854.51,"endTime":1857.3,"body":"framework, which has"},{"startTime":1854.51,"endTime":1857.3,"body":"already been mentioned today"},{"startTime":1857.3,"endTime":1860.18,"body":"and my experience on NICE committees."},{"startTime":1860.18,"endTime":1861.32,"body":"But what I present,"},{"startTime":1861.32,"endTime":1863.9,"body":"what I say just reflects my own opinions,"},{"startTime":1863.9,"endTime":1866.09,"body":"not those of NICE or anyone else."},{"startTime":1866.09,"endTime":1867.563,"body":"Just want to make that clear."},{"startTime":1869.526,"endTime":1872.06,"body":"Okay, so when we are"},{"startTime":1869.526,"endTime":1872.06,"body":"considering the acceptance"},{"startTime":1872.06,"endTime":1873.71,"body":"of real world data analysis"},{"startTime":1873.71,"endTime":1876.65,"body":"by health technology assessment agencies,"},{"startTime":1876.65,"endTime":1879.17,"body":"I think the NICE real"},{"startTime":1876.65,"endTime":1879.17,"body":"world evidence framework"},{"startTime":1879.17,"endTime":1881.18,"body":"is a crucial document."},{"startTime":1881.18,"endTime":1883.28,"body":"Obviously it is specific to NICE,"},{"startTime":1883.28,"endTime":1886.49,"body":"but for many of us, NICE is a key agency"},{"startTime":1886.49,"endTime":1888.92,"body":"and I think a lot of"},{"startTime":1886.49,"endTime":1888.92,"body":"what the document says"},{"startTime":1888.92,"endTime":1890.78,"body":"is also representative of the views"},{"startTime":1890.78,"endTime":1894.17,"body":"of several other HTA"},{"startTime":1890.78,"endTime":1894.17,"body":"agencies around the world."},{"startTime":1894.17,"endTime":1896.6,"body":"Although of course not all agencies"},{"startTime":1896.6,"endTime":1897.85,"body":"will agree on everything."},{"startTime":1898.79,"endTime":1901.79,"body":"In my view, the the real world"},{"startTime":1898.79,"endTime":1901.79,"body":"evidence framework document"},{"startTime":1901.79,"endTime":1904.46,"body":"is a bit like the technical"},{"startTime":1901.79,"endTime":1904.46,"body":"support documents produced"},{"startTime":1904.46,"endTime":1906.74,"body":"by NICE's Decision Support Unit."},{"startTime":1906.74,"endTime":1909.065,"body":"And I think it's a"},{"startTime":1906.74,"endTime":1909.065,"body":"document that review groups"},{"startTime":1909.065,"endTime":1912.587,"body":"and committee members"},{"startTime":1909.065,"endTime":1912.587,"body":"will refer to for guidance"},{"startTime":1912.587,"endTime":1914.873,"body":"and so will inform decision making."},{"startTime":1917.03,"endTime":1920.18,"body":"So the real world evidence"},{"startTime":1917.03,"endTime":1920.18,"body":"framework has a key messages"},{"startTime":1920.18,"endTime":1921.5,"body":"section right at the start."},{"startTime":1921.5,"endTime":1923.93,"body":"And that really highlights"},{"startTime":1921.5,"endTime":1923.93,"body":"NICE's commitment"},{"startTime":1923.93,"endTime":1927.41,"body":"to using real world data"},{"startTime":1923.93,"endTime":1927.41,"body":"within technology appraisals."},{"startTime":1927.41,"endTime":1930.08,"body":"So it says, as described"},{"startTime":1927.41,"endTime":1930.08,"body":"in the NICE strategy"},{"startTime":1930.08,"endTime":1934.04,"body":"of 2021 to '26, we want"},{"startTime":1930.08,"endTime":1934.04,"body":"to use real world data"},{"startTime":1934.04,"endTime":1935.81,"body":"to resolve gaps in knowledge"},{"startTime":1935.81,"endTime":1939.38,"body":"and drive forward access to"},{"startTime":1935.81,"endTime":1939.38,"body":"innovations for patients."},{"startTime":1939.38,"endTime":1941.42,"body":"We developed the real"},{"startTime":1939.38,"endTime":1941.42,"body":"world evidence framework"},{"startTime":1941.42,"endTime":1943.613,"body":"to help deliver on this ambition."},{"startTime":1944.69,"endTime":1947.18,"body":"The framework document"},{"startTime":1944.69,"endTime":1947.18,"body":"then goes on to explain"},{"startTime":1947.18,"endTime":1950.12,"body":"how it will deliver upon that ambition"},{"startTime":1950.12,"endTime":1954.26,"body":"by identifying when real"},{"startTime":1950.12,"endTime":1954.26,"body":"world data can be used"},{"startTime":1954.26,"endTime":1957.38,"body":"to reduce uncertainties"},{"startTime":1954.26,"endTime":1957.38,"body":"and improve guidance"},{"startTime":1957.38,"endTime":1960.95,"body":"and by clearly describing"},{"startTime":1957.38,"endTime":1960.95,"body":"best practices for planning,"},{"startTime":1960.95,"endTime":1964.61,"body":"conducting and reporting"},{"startTime":1960.95,"endTime":1964.61,"body":"real world evidence studies"},{"startTime":1964.61,"endTime":1968.423,"body":"to improve the quality and"},{"startTime":1964.61,"endTime":1968.423,"body":"transparency of evidence."},{"startTime":1969.65,"endTime":1972.44,"body":"And it then says that the"},{"startTime":1969.65,"endTime":1972.44,"body":"key aim of the document"},{"startTime":1972.44,"endTime":1974.33,"body":"is to improve the quality"},{"startTime":1974.33,"endTime":1977.783,"body":"of real world evidence in"},{"startTime":1974.33,"endTime":1977.783,"body":"informing our guidance."},{"startTime":1979.76,"endTime":1982.55,"body":"So the framework document then goes on"},{"startTime":1982.55,"endTime":1986.45,"body":"to describe three core"},{"startTime":1982.55,"endTime":1986.45,"body":"principles around the use"},{"startTime":1986.45,"endTime":1988.76,"body":"of real world data to inform appraisals."},{"startTime":1988.76,"endTime":1990.23,"body":"And I think these are really important."},{"startTime":1990.23,"endTime":1994.51,"body":"So firstly, we must ensure data is"},{"startTime":1994.51,"endTime":1998.54,"body":"of good provenance, relevant"},{"startTime":1994.51,"endTime":1998.54,"body":"and of sufficient quality"},{"startTime":1998.54,"endTime":2000.343,"body":"to answer the research question."},{"startTime":2001.33,"endTime":2005.23,"body":"Secondly, we must generate"},{"startTime":2001.33,"endTime":2005.23,"body":"evidence in a transparent way"},{"startTime":2005.23,"endTime":2008.11,"body":"and with integrity right"},{"startTime":2005.23,"endTime":2008.11,"body":"from study planning"},{"startTime":2008.11,"endTime":2011.143,"body":"through to study, conduct and reporting."},{"startTime":2012.19,"endTime":2014.74,"body":"And thirdly, we must"},{"startTime":2012.19,"endTime":2014.74,"body":"use analytical methods"},{"startTime":2014.74,"endTime":2018.583,"body":"that minimise the risk of bias"},{"startTime":2014.74,"endTime":2018.583,"body":"and characterise uncertainty."},{"startTime":2019.9,"endTime":2022.42,"body":"So I think it's obvious"},{"startTime":2019.9,"endTime":2022.42,"body":"that if a company wishes"},{"startTime":2022.42,"endTime":2027.22,"body":"to use real world data in an"},{"startTime":2022.42,"endTime":2027.22,"body":"appraisal, NICE's review groups"},{"startTime":2027.22,"endTime":2030.28,"body":"and appraisal committees are"},{"startTime":2027.22,"endTime":2030.28,"body":"going to want to see evidence"},{"startTime":2030.28,"endTime":2032.08,"body":"that each of these core principles"},{"startTime":2032.08,"endTime":2034.21,"body":"has been adequately covered."},{"startTime":2034.21,"endTime":2038.08,"body":"So to think about what that"},{"startTime":2034.21,"endTime":2038.08,"body":"means in a bit more detail,"},{"startTime":2038.08,"endTime":2040.06,"body":"I'm going to go through"},{"startTime":2038.08,"endTime":2040.06,"body":"each of these principles"},{"startTime":2040.06,"endTime":2040.963,"body":"a little bit more."},{"startTime":2044.17,"endTime":2048.433,"body":"Okay, so firstly, data"},{"startTime":2044.17,"endTime":2048.433,"body":"provenance, relevance and quality."},{"startTime":2049.45,"endTime":2051.52,"body":"To address this core principle,"},{"startTime":2051.52,"endTime":2055.9,"body":"the framework document"},{"startTime":2051.52,"endTime":2055.9,"body":"helpfully includes DataSAT,"},{"startTime":2055.9,"endTime":2058.48,"body":"which is a data suitability"},{"startTime":2055.9,"endTime":2058.48,"body":"assessment tool,"},{"startTime":2058.48,"endTime":2062.17,"body":"which covers data provenance,"},{"startTime":2058.48,"endTime":2062.17,"body":"quality and relevance."},{"startTime":2062.17,"endTime":2065.05,"body":"So for data provenance information must be"},{"startTime":2065.05,"endTime":2067.21,"body":"provided on the data source"},{"startTime":2067.21,"endTime":2070.423,"body":"and on things like data"},{"startTime":2067.21,"endTime":2070.423,"body":"management and governance."},{"startTime":2071.5,"endTime":2073.81,"body":"For data quality information"},{"startTime":2073.81,"endTime":2075.97,"body":"has to be provided on the completeness"},{"startTime":2075.97,"endTime":2079.72,"body":"and accuracy of each key study variable,"},{"startTime":2079.72,"endTime":2080.86,"body":"which are those variables"},{"startTime":2080.86,"endTime":2083.71,"body":"that are particularly"},{"startTime":2080.86,"endTime":2083.71,"body":"important for the analysis."},{"startTime":2083.71,"endTime":2088.54,"body":"So such as variables on"},{"startTime":2083.71,"endTime":2088.54,"body":"interventions received, outcomes"},{"startTime":2088.54,"endTime":2090.523,"body":"and prognostic characteristics."},{"startTime":2092.2,"endTime":2095.65,"body":"For data relevance, the"},{"startTime":2092.2,"endTime":2095.65,"body":"population and setting of the data"},{"startTime":2095.65,"endTime":2097.0,"body":"needs to be described"},{"startTime":2097.0,"endTime":2100.03,"body":"so that a reviewer can see if"},{"startTime":2097.0,"endTime":2100.03,"body":"the data are really relevant"},{"startTime":2100.03,"endTime":2102.1,"body":"for the decision problem being addressed."},{"startTime":2102.1,"endTime":2104.68,"body":"So for example, this could be"},{"startTime":2102.1,"endTime":2104.68,"body":"about whether the real world"},{"startTime":2104.68,"endTime":2108.31,"body":"data source is from the"},{"startTime":2104.68,"endTime":2108.31,"body":"NHS or from a population"},{"startTime":2108.31,"endTime":2110.263,"body":"that is similar to the NHS."},{"startTime":2112.57,"endTime":2113.95,"body":"Now I think it's important to highlight"},{"startTime":2113.95,"endTime":2117.28,"body":"that completing DataSAT"},{"startTime":2113.95,"endTime":2117.28,"body":"isn't a requirement"},{"startTime":2117.28,"endTime":2120.91,"body":"when using real world data"},{"startTime":2117.28,"endTime":2120.91,"body":"to inform a NICE submission,"},{"startTime":2120.91,"endTime":2123.52,"body":"but it certainly makes"},{"startTime":2120.91,"endTime":2123.52,"body":"sense to complete it"},{"startTime":2123.52,"endTime":2125.17,"body":"if you want to increase the chance"},{"startTime":2125.17,"endTime":2128.71,"body":"that NICE will deem the"},{"startTime":2125.17,"endTime":2128.71,"body":"evidence reliable enough"},{"startTime":2128.71,"endTime":2130.72,"body":"to inform decision making."},{"startTime":2130.72,"endTime":2133.06,"body":"If NICE provide the tool,"},{"startTime":2133.06,"endTime":2135.41,"body":"it makes sense that you"},{"startTime":2133.06,"endTime":2135.41,"body":"should probably use it."},{"startTime":2137.86,"endTime":2141.76,"body":"Okay, so a bit about my"},{"startTime":2137.86,"endTime":2141.76,"body":"experience with key issues"},{"startTime":2141.76,"endTime":2145.153,"body":"for reviews and decision makers"},{"startTime":2141.76,"endTime":2145.153,"body":"on on this particular topic."},{"startTime":2146.32,"endTime":2150.04,"body":"Firstly, I think committees"},{"startTime":2146.32,"endTime":2150.04,"body":"almost always think about"},{"startTime":2150.04,"endTime":2151.96,"body":"where the data are from"},{"startTime":2151.96,"endTime":2155.14,"body":"and whether the data are"},{"startTime":2151.96,"endTime":2155.14,"body":"generalisable to the population"},{"startTime":2155.14,"endTime":2158.95,"body":"of interest, which is really"},{"startTime":2155.14,"endTime":2158.95,"body":"about the relevance of data."},{"startTime":2158.95,"endTime":2162.233,"body":"So for example, NICE is making decisions"},{"startTime":2162.233,"endTime":2165.34,"body":"for the NHS population, so wants"},{"startTime":2165.34,"endTime":2168.31,"body":"to see if the real world"},{"startTime":2165.34,"endTime":2168.31,"body":"data set is generalisable"},{"startTime":2168.31,"endTime":2169.873,"body":"to the NHS population."},{"startTime":2170.71,"endTime":2174.01,"body":"If the real world data"},{"startTime":2170.71,"endTime":2174.01,"body":"source is from the UK,"},{"startTime":2174.01,"endTime":2176.44,"body":"then people in it are"},{"startTime":2174.01,"endTime":2176.44,"body":"likely to be selected"},{"startTime":2176.44,"endTime":2179.05,"body":"from that NHS population."},{"startTime":2179.05,"endTime":2181.15,"body":"But then the question is whether all types"},{"startTime":2181.15,"endTime":2184.84,"body":"of patients in the NHS"},{"startTime":2181.15,"endTime":2184.84,"body":"population are represented"},{"startTime":2184.84,"endTime":2186.25,"body":"or does the real world data"},{"startTime":2186.25,"endTime":2188.62,"body":"have specific eligibility criteria,"},{"startTime":2188.62,"endTime":2190.81,"body":"which could mean that"},{"startTime":2188.62,"endTime":2190.81,"body":"they're important prognostic"},{"startTime":2190.81,"endTime":2194.38,"body":"differences between the"},{"startTime":2190.81,"endTime":2194.38,"body":"patients in the NHS population"},{"startTime":2194.38,"endTime":2196.78,"body":"and the patients in the"},{"startTime":2194.38,"endTime":2196.78,"body":"real world data source."},{"startTime":2196.78,"endTime":2198.703,"body":"So NICE always going to think about this."},{"startTime":2201.01,"endTime":2204.19,"body":"Or if the real world data"},{"startTime":2201.01,"endTime":2204.19,"body":"is from another country,"},{"startTime":2204.19,"endTime":2206.05,"body":"committees will think"},{"startTime":2204.19,"endTime":2206.05,"body":"about whether the data"},{"startTime":2206.05,"endTime":2208.63,"body":"are transportable to the NHS population."},{"startTime":2208.63,"endTime":2210.43,"body":"And transportability is another thing"},{"startTime":2210.43,"endTime":2212.2,"body":"that has been added quite recently"},{"startTime":2212.2,"endTime":2214.543,"body":"to the real world evidence"},{"startTime":2212.2,"endTime":2214.543,"body":"framework document."},{"startTime":2215.65,"endTime":2218.95,"body":"So again, they'll think about"},{"startTime":2215.65,"endTime":2218.95,"body":"where is that data source from"},{"startTime":2218.95,"endTime":2221.29,"body":"and are the populations similar"},{"startTime":2221.29,"endTime":2223.18,"body":"to the target NHS population,"},{"startTime":2223.18,"endTime":2225.343,"body":"or do they have imported differences."},{"startTime":2227.92,"endTime":2230.44,"body":"Beyond where the data are from,"},{"startTime":2230.44,"endTime":2232.33,"body":"I think it's very common"},{"startTime":2230.44,"endTime":2232.33,"body":"for NICE committees"},{"startTime":2232.33,"endTime":2236.38,"body":"to consider the variables"},{"startTime":2232.33,"endTime":2236.38,"body":"included in a real world data set."},{"startTime":2236.38,"endTime":2239.11,"body":"So are important variables measured"},{"startTime":2239.11,"endTime":2241.6,"body":"and amongst the variables"},{"startTime":2239.11,"endTime":2241.6,"body":"that are measured,"},{"startTime":2241.6,"endTime":2244.0,"body":"is there any important missing data,"},{"startTime":2244.0,"endTime":2246.043,"body":"which is about the quality of the data?"},{"startTime":2247.57,"endTime":2250.15,"body":"And fundamentally, the"},{"startTime":2247.57,"endTime":2250.15,"body":"reason committees care"},{"startTime":2250.15,"endTime":2251.227,"body":"about missing data"},{"startTime":2251.227,"endTime":2253.33,"body":"and the variables that are measured"},{"startTime":2253.33,"endTime":2254.83,"body":"is because if they think"},{"startTime":2254.83,"endTime":2256.81,"body":"that there could be important differences"},{"startTime":2256.81,"endTime":2259.81,"body":"between the population in"},{"startTime":2256.81,"endTime":2259.81,"body":"the real world data set"},{"startTime":2259.81,"endTime":2262.72,"body":"and the NHS population or in the case"},{"startTime":2262.72,"endTime":2264.94,"body":"of external control arm studies,"},{"startTime":2264.94,"endTime":2266.89,"body":"the population in the uncontrolled study"},{"startTime":2266.89,"endTime":2268.33,"body":"of the experimental drug"},{"startTime":2268.33,"endTime":2271.54,"body":"and the population in"},{"startTime":2268.33,"endTime":2271.54,"body":"the external controller,"},{"startTime":2271.54,"endTime":2273.67,"body":"then the committee needs to"},{"startTime":2271.54,"endTime":2273.67,"body":"know whether the data are"},{"startTime":2273.67,"endTime":2276.85,"body":"sufficient to allow adjustments"},{"startTime":2273.67,"endTime":2276.85,"body":"to be made to account"},{"startTime":2276.85,"endTime":2279.253,"body":"for the differences in study populations."},{"startTime":2281.29,"endTime":2284.32,"body":"So what this fundamentally boils down to"},{"startTime":2284.32,"endTime":2285.58,"body":"is that the committee,"},{"startTime":2285.58,"endTime":2288.1,"body":"and this is my Chair of committee here,"},{"startTime":2288.1,"endTime":2290.62,"body":"are going to want to know"},{"startTime":2288.1,"endTime":2290.62,"body":"whether they can really rely on"},{"startTime":2290.62,"endTime":2292.48,"body":"the real world data sources"},{"startTime":2292.48,"endTime":2294.37,"body":"and whether the data are relevant enough"},{"startTime":2294.37,"endTime":2296.02,"body":"to address the decision problem"},{"startTime":2296.02,"endTime":2298.3,"body":"being faced in the appraisal."},{"startTime":2298.3,"endTime":2301.15,"body":"And it's likely that the"},{"startTime":2298.3,"endTime":2301.15,"body":"committee will probably think"},{"startTime":2301.15,"endTime":2305.26,"body":"that if the data aren't"},{"startTime":2301.15,"endTime":2305.26,"body":"relevant, or of poor quality"},{"startTime":2305.26,"endTime":2308.38,"body":"that they can't rely on"},{"startTime":2305.26,"endTime":2308.38,"body":"it for decision making."},{"startTime":2308.38,"endTime":2311.2,"body":"So I've dwelt on this"},{"startTime":2308.38,"endTime":2311.2,"body":"data provenance relevance"},{"startTime":2311.2,"endTime":2313.54,"body":"and quality aspect here quite a lot"},{"startTime":2313.54,"endTime":2315.94,"body":"because I think it's really important."},{"startTime":2315.94,"endTime":2318.82,"body":"Sometimes it is tempting to"},{"startTime":2315.94,"endTime":2318.82,"body":"spend a lot of time thinking"},{"startTime":2318.82,"endTime":2322.0,"body":"about exciting intricate"},{"startTime":2318.82,"endTime":2322.0,"body":"analytical methods"},{"startTime":2322.0,"endTime":2323.59,"body":"which are important,"},{"startTime":2323.59,"endTime":2325.78,"body":"but if the data aren't right,"},{"startTime":2325.78,"endTime":2328.93,"body":"analysis won't be useful"},{"startTime":2325.78,"endTime":2328.93,"body":"no matter what we do."},{"startTime":2328.93,"endTime":2331.333,"body":"And I think that is"},{"startTime":2328.93,"endTime":2331.333,"body":"important to bear in mind."},{"startTime":2334.21,"endTime":2336.01,"body":"Okay, so the next core principle covered"},{"startTime":2336.01,"endTime":2339.97,"body":"by NICE's real world evidence"},{"startTime":2336.01,"endTime":2339.97,"body":"framework is study planning,"},{"startTime":2339.97,"endTime":2341.713,"body":"conduct and reporting."},{"startTime":2342.76,"endTime":2345.16,"body":"So obviously the framework"},{"startTime":2342.76,"endTime":2345.16,"body":"document says a lot more"},{"startTime":2345.16,"endTime":2346.69,"body":"than what I have on the slide here,"},{"startTime":2346.69,"endTime":2348.94,"body":"but a really key part is that as much"},{"startTime":2348.94,"endTime":2352.0,"body":"as possible should be"},{"startTime":2348.94,"endTime":2352.0,"body":"pre-specified and published."},{"startTime":2352.0,"endTime":2355.06,"body":"And that includes the"},{"startTime":2352.0,"endTime":2355.06,"body":"study design analysis"},{"startTime":2355.06,"endTime":2356.743,"body":"plans and reporting."},{"startTime":2357.73,"endTime":2360.31,"body":"The document recommends the HARPER tool"},{"startTime":2360.31,"endTime":2363.37,"body":"for developing protocols"},{"startTime":2360.31,"endTime":2363.37,"body":"for real world data studies"},{"startTime":2363.37,"endTime":2366.7,"body":"and refers to several reporting"},{"startTime":2363.37,"endTime":2366.7,"body":"checklists as well to ensure"},{"startTime":2366.7,"endTime":2369.25,"body":"that all important aspects are reported"},{"startTime":2369.25,"endTime":2370.753,"body":"clearly and transparently."},{"startTime":2372.25,"endTime":2375.37,"body":"Well, I think most of this"},{"startTime":2372.25,"endTime":2375.37,"body":"will be naturally covered"},{"startTime":2375.37,"endTime":2378.97,"body":"if a target trial approach"},{"startTime":2375.37,"endTime":2378.97,"body":"is used as explained by Joe."},{"startTime":2378.97,"endTime":2382.36,"body":"But still it's important that"},{"startTime":2378.97,"endTime":2382.36,"body":"all of this is planned upfront"},{"startTime":2382.36,"endTime":2384.523,"body":"and that it's documented transparently."},{"startTime":2385.69,"endTime":2389.74,"body":"Now I think the key questions"},{"startTime":2385.69,"endTime":2389.74,"body":"that committees often ask"},{"startTime":2389.74,"endTime":2393.55,"body":"around this is whether an"},{"startTime":2389.74,"endTime":2393.55,"body":"analysis were pre-specified"},{"startTime":2393.55,"endTime":2395.86,"body":"based on the concern that if they weren't,"},{"startTime":2395.86,"endTime":2397.51,"body":"perhaps they've been cherry picked"},{"startTime":2397.51,"endTime":2399.97,"body":"to provide the desired results."},{"startTime":2399.97,"endTime":2401.743,"body":"And related to this, the"},{"startTime":2399.97,"endTime":2401.743,"body":"committee will want to know"},{"startTime":2401.743,"endTime":2403.6,"body":"that the study has been designed"},{"startTime":2403.6,"endTime":2406.773,"body":"and conducted in a way that's"},{"startTime":2403.6,"endTime":2406.773,"body":"going to minimise bias."},{"startTime":2406.773,"endTime":2409.75,"body":"And for the committee to"},{"startTime":2406.773,"endTime":2409.75,"body":"be confident about that"},{"startTime":2409.75,"endTime":2412.18,"body":"the study needs to be planned, conducted"},{"startTime":2412.18,"endTime":2414.133,"body":"and reported really clearly."},{"startTime":2416.92,"endTime":2418.69,"body":"Okay, so the final core principle covered"},{"startTime":2418.69,"endTime":2422.62,"body":"by the NICE RWE framework"},{"startTime":2418.69,"endTime":2422.62,"body":"about analytical methods."},{"startTime":2422.62,"endTime":2424.84,"body":"So Joe has already"},{"startTime":2422.62,"endTime":2424.84,"body":"talked about the methods,"},{"startTime":2424.84,"endTime":2427.0,"body":"so I'm not going to go into detail here,"},{"startTime":2427.0,"endTime":2429.31,"body":"but some key points are that the framework"},{"startTime":2429.31,"endTime":2431.83,"body":"recommends taking a target trial approach"},{"startTime":2431.83,"endTime":2435.19,"body":"using analytical methods"},{"startTime":2431.83,"endTime":2435.19,"body":"that avoid or remove bias"},{"startTime":2435.19,"endTime":2436.66,"body":"and including sensitivity"},{"startTime":2436.66,"endTime":2439.9,"body":"and scenario analysis to"},{"startTime":2436.66,"endTime":2439.9,"body":"give an idea of the stability"},{"startTime":2439.9,"endTime":2443.23,"body":"of the results and the importance"},{"startTime":2439.9,"endTime":2443.23,"body":"of any potential biases"},{"startTime":2443.23,"endTime":2445.93,"body":"that might remain due to"},{"startTime":2443.23,"endTime":2445.93,"body":"things like missing data,"},{"startTime":2445.93,"endTime":2447.313,"body":"or unmeasured confounding."},{"startTime":2448.99,"endTime":2451.84,"body":"I think that depending on"},{"startTime":2448.99,"endTime":2451.84,"body":"the expertise of members"},{"startTime":2451.84,"endTime":2454.15,"body":"of the review groups and"},{"startTime":2451.84,"endTime":2454.15,"body":"appraisal committees,"},{"startTime":2454.15,"endTime":2455.8,"body":"there are a whole range of questions"},{"startTime":2455.8,"endTime":2458.71,"body":"that could be asked about"},{"startTime":2455.8,"endTime":2458.71,"body":"the analytical methods used"},{"startTime":2458.71,"endTime":2462.04,"body":"around things like choices"},{"startTime":2458.71,"endTime":2462.04,"body":"of time, zero time points,"},{"startTime":2462.04,"endTime":2465.25,"body":"variable selection, missing"},{"startTime":2462.04,"endTime":2465.25,"body":"data, treatment pathways,"},{"startTime":2465.25,"endTime":2468.16,"body":"censoring potential residual biases."},{"startTime":2468.16,"endTime":2470.203,"body":"So lots could come up there."},{"startTime":2473.35,"endTime":2475.57,"body":"Okay, so before I finish in Joe's slides,"},{"startTime":2475.57,"endTime":2478.12,"body":"he talked a little bit about"},{"startTime":2475.57,"endTime":2478.12,"body":"a published external control"},{"startTime":2478.12,"endTime":2481.12,"body":"arm study for axicabtagene ciloleucel"},{"startTime":2481.12,"endTime":2483.55,"body":"for follicular lymphoma."},{"startTime":2483.55,"endTime":2484.78,"body":"I thought it might be interesting"},{"startTime":2484.78,"endTime":2486.76,"body":"to look at the NICE appraisal that used"},{"startTime":2486.76,"endTime":2489.58,"body":"that external control"},{"startTime":2486.76,"endTime":2489.58,"body":"study to see what kinds"},{"startTime":2489.58,"endTime":2491.41,"body":"of things the NICE appraisal committee"},{"startTime":2491.41,"endTime":2493.15,"body":"discussed in that case."},{"startTime":2493.15,"endTime":2496.72,"body":"So, so this was technology appraisal 894,"},{"startTime":2496.72,"endTime":2498.16,"body":"you can look it up"},{"startTime":2498.16,"endTime":2499.36,"body":"and it's important to say"},{"startTime":2499.36,"endTime":2502.39,"body":"that my summary here isn't exhaustive."},{"startTime":2502.39,"endTime":2504.49,"body":"These are just some bits"},{"startTime":2502.39,"endTime":2504.49,"body":"that came directly out"},{"startTime":2504.49,"endTime":2506.62,"body":"the final guidance document on the website"},{"startTime":2506.62,"endTime":2508.22,"body":"that I thought were interesting."},{"startTime":2509.44,"endTime":2512.71,"body":"So first of all about relevance"},{"startTime":2509.44,"endTime":2512.71,"body":"of the real world data."},{"startTime":2512.71,"endTime":2515.35,"body":"If you remember, axi-cel"},{"startTime":2512.71,"endTime":2515.35,"body":"had been investigated"},{"startTime":2515.35,"endTime":2518.23,"body":"in an uncontrolled study called ZUMA-5"},{"startTime":2518.23,"endTime":2520.15,"body":"and an external control arm was created"},{"startTime":2520.15,"endTime":2523.81,"body":"using real world data in"},{"startTime":2520.15,"endTime":2523.81,"body":"a study called SCHOLAR-5."},{"startTime":2523.81,"endTime":2525.25,"body":"The NICE committee concluded"},{"startTime":2525.25,"endTime":2528.91,"body":"that using data from"},{"startTime":2525.25,"endTime":2528.91,"body":"SCHOLAR-5 was acceptable,"},{"startTime":2528.91,"endTime":2531.25,"body":"although it did notice a mismatch"},{"startTime":2531.25,"endTime":2534.07,"body":"between the standard treatments"},{"startTime":2531.25,"endTime":2534.07,"body":"received in SCHOLAR-5"},{"startTime":2534.07,"endTime":2535.62,"body":"and in the NHS."},{"startTime":2535.62,"endTime":2537.283,"body":"So there were some differences."},{"startTime":2538.93,"endTime":2542.47,"body":"With reference to the"},{"startTime":2538.93,"endTime":2542.47,"body":"conduct of the ECA study,"},{"startTime":2542.47,"endTime":2545.65,"body":"the committee discussed the"},{"startTime":2542.47,"endTime":2545.65,"body":"eligibility criteria used"},{"startTime":2545.65,"endTime":2548.32,"body":"for people included in the"},{"startTime":2545.65,"endTime":2548.32,"body":"comparative effectiveness"},{"startTime":2548.32,"endTime":2550.0,"body":"analysis, which was important"},{"startTime":2550.0,"endTime":2551.74,"body":"because the number of prior treatments"},{"startTime":2551.74,"endTime":2554.173,"body":"did differ in the two studies."},{"startTime":2555.13,"endTime":2557.35,"body":"The guidance document also noted,"},{"startTime":2557.35,"endTime":2559.63,"body":"and I thought this was"},{"startTime":2557.35,"endTime":2559.63,"body":"interesting, the methods used"},{"startTime":2559.63,"endTime":2562.289,"body":"to adjust for baseline"},{"startTime":2559.63,"endTime":2562.289,"body":"differences between the ZUMA-5"},{"startTime":2562.289,"endTime":2564.85,"body":"and SCHOLAR-5 populations"},{"startTime":2564.85,"endTime":2567.37,"body":"and that these methods had"},{"startTime":2564.85,"endTime":2567.37,"body":"improved the comparability"},{"startTime":2567.37,"endTime":2569.02,"body":"of the data sources."},{"startTime":2569.02,"endTime":2571.33,"body":"But there also, there"},{"startTime":2569.02,"endTime":2571.33,"body":"was also quite a detailed"},{"startTime":2571.33,"endTime":2575.44,"body":"and pragmatic, I think, discussion"},{"startTime":2571.33,"endTime":2575.44,"body":"about covariate selection"},{"startTime":2575.44,"endTime":2577.42,"body":"where it was noted that adjusting for lots"},{"startTime":2577.42,"endTime":2581.08,"body":"of variables is difficult"},{"startTime":2577.42,"endTime":2581.08,"body":"when sample sizes are small"},{"startTime":2581.08,"endTime":2582.97,"body":"as they were in some cases here"},{"startTime":2582.97,"endTime":2585.37,"body":"because this can lead to"},{"startTime":2582.97,"endTime":2585.37,"body":"problems with convergence"},{"startTime":2585.37,"endTime":2587.53,"body":"of statistical models."},{"startTime":2587.53,"endTime":2589.627,"body":"So the committee was clearly"},{"startTime":2587.53,"endTime":2589.627,"body":"thinking quite hard about this"},{"startTime":2589.627,"endTime":2591.433,"body":"and in quite a lot of detail."},{"startTime":2593.56,"endTime":2597.07,"body":"With respect to the reporting"},{"startTime":2593.56,"endTime":2597.07,"body":"and I suppose conduct as well."},{"startTime":2597.07,"endTime":2600.1,"body":"The committee noted that"},{"startTime":2597.07,"endTime":2600.1,"body":"the company had submitted"},{"startTime":2600.1,"endTime":2603.37,"body":"sensitivity analysis"},{"startTime":2600.1,"endTime":2603.37,"body":"using alternative methods"},{"startTime":2603.37,"endTime":2605.86,"body":"and quantitative bias analysis,"},{"startTime":2605.86,"endTime":2608.023,"body":"both of which were seen as good things."},{"startTime":2610.15,"endTime":2613.18,"body":"Overall the committee's"},{"startTime":2610.15,"endTime":2613.18,"body":"conclusion on the analysis,"},{"startTime":2613.18,"endTime":2616.48,"body":"at least the conclusion recorded"},{"startTime":2613.18,"endTime":2616.48,"body":"in the guidance document"},{"startTime":2616.48,"endTime":2618.49,"body":"is a, I suppose a little bit vague."},{"startTime":2618.49,"endTime":2620.92,"body":"It stated that the adjustment method used"},{"startTime":2620.92,"endTime":2624.13,"body":"by the company is highly complex."},{"startTime":2624.13,"endTime":2626.5,"body":"It's not really directly"},{"startTime":2624.13,"endTime":2626.5,"body":"saying whether the analysis"},{"startTime":2626.5,"endTime":2628.333,"body":"were acceptable or not,"},{"startTime":2629.271,"endTime":2631.63,"body":"but it is clear from the document"},{"startTime":2631.63,"endTime":2634.66,"body":"that the analysis was used"},{"startTime":2631.63,"endTime":2634.66,"body":"for the decision making,"},{"startTime":2634.66,"endTime":2638.32,"body":"although the lack of direct"},{"startTime":2634.66,"endTime":2638.32,"body":"comparative efficacy data"},{"startTime":2638.32,"endTime":2641.08,"body":"was highlighted as"},{"startTime":2638.32,"endTime":2641.08,"body":"contributing to a high level"},{"startTime":2641.08,"endTime":2643.18,"body":"of uncertainty in the appraisal,"},{"startTime":2643.18,"endTime":2645.253,"body":"which I guess is what you would expect."},{"startTime":2647.93,"endTime":2651.58,"body":"Okay, so finally some conclusions from me."},{"startTime":2651.58,"endTime":2655.9,"body":"So although this session is"},{"startTime":2651.58,"endTime":2655.9,"body":"about using real world data"},{"startTime":2655.9,"endTime":2658.9,"body":"in HTA, I think it is"},{"startTime":2655.9,"endTime":2658.9,"body":"important to remember"},{"startTime":2658.9,"endTime":2661.78,"body":"that RCTs remain the preferred source"},{"startTime":2661.78,"endTime":2664.18,"body":"of evidence on the"},{"startTime":2661.78,"endTime":2664.18,"body":"effects of new treatments."},{"startTime":2664.18,"endTime":2667.24,"body":"And that is stated in"},{"startTime":2664.18,"endTime":2667.24,"body":"NICE's real world evidence"},{"startTime":2667.24,"endTime":2668.443,"body":"framework document."},{"startTime":2669.58,"endTime":2673.51,"body":"However, it is also stated"},{"startTime":2669.58,"endTime":2673.51,"body":"that non-randomised evidence"},{"startTime":2673.51,"endTime":2676.36,"body":"can be used when RCTs don't exist"},{"startTime":2676.36,"endTime":2679.423,"body":"or aren't relevant, or"},{"startTime":2676.36,"endTime":2679.423,"body":"are of poor quality."},{"startTime":2680.5,"endTime":2685.24,"body":"So certainly we can use these"},{"startTime":2680.5,"endTime":2685.24,"body":"data sources and analysis."},{"startTime":2685.24,"endTime":2688.75,"body":"HTA decision makers are"},{"startTime":2685.24,"endTime":2688.75,"body":"generally open to this,"},{"startTime":2688.75,"endTime":2692.95,"body":"but we can't just use any old RWE study."},{"startTime":2692.95,"endTime":2696.04,"body":"The data needs to be"},{"startTime":2692.95,"endTime":2696.04,"body":"high quality and relevant"},{"startTime":2696.04,"endTime":2699.4,"body":"and the analysis need to be"},{"startTime":2696.04,"endTime":2699.4,"body":"well planned pre-specified"},{"startTime":2699.4,"endTime":2700.813,"body":"and well reported."},{"startTime":2703.12,"endTime":2705.19,"body":"And finally, even though I am a fan"},{"startTime":2705.19,"endTime":2708.01,"body":"of using complex methods"},{"startTime":2705.19,"endTime":2708.01,"body":"is what I've spent a lot"},{"startTime":2708.01,"endTime":2710.424,"body":"of my career trying to do,"},{"startTime":2710.424,"endTime":2712.6,"body":"a lot of my research is on these methods"},{"startTime":2712.6,"endTime":2714.43,"body":"and these data sources."},{"startTime":2714.43,"endTime":2715.72,"body":"We do need to understand"},{"startTime":2715.72,"endTime":2718.33,"body":"that when we are not"},{"startTime":2715.72,"endTime":2718.33,"body":"using randomised data,"},{"startTime":2718.33,"endTime":2722.17,"body":"even high quality analysis"},{"startTime":2718.33,"endTime":2722.17,"body":"using high quality data"},{"startTime":2722.17,"endTime":2725.95,"body":"are likely to be associated"},{"startTime":2722.17,"endTime":2725.95,"body":"with high levels of uncertainty."},{"startTime":2725.95,"endTime":2726.783,"body":"And I think it's fair to say"},{"startTime":2726.783,"endTime":2730.96,"body":"that decision makers may be"},{"startTime":2726.783,"endTime":2730.96,"body":"more aware of uncertainty"},{"startTime":2730.96,"endTime":2734.71,"body":"and perhaps will be more"},{"startTime":2730.96,"endTime":2734.71,"body":"tentative when making decisions"},{"startTime":2734.71,"endTime":2738.613,"body":"based on real world evidence"},{"startTime":2734.71,"endTime":2738.613,"body":"rather than RCT evidence."},{"startTime":2739.81,"endTime":2742.18,"body":"And with that, thanks"},{"startTime":2739.81,"endTime":2742.18,"body":"very much for listening"},{"startTime":2742.18,"endTime":2745.633,"body":"and I think we can move on to questions."},{"startTime":2748.12,"endTime":2750.25,"body":"- Thank you very much, Nick,"},{"startTime":2750.25,"endTime":2753.583,"body":"and all the speakers, that's"},{"startTime":2750.25,"endTime":2753.583,"body":"been absolutely fabulous."},{"startTime":2755.08,"endTime":2757.357,"body":"So yeah, I'm going to open"},{"startTime":2755.08,"endTime":2757.357,"body":"the floor to some questions"},{"startTime":2757.357,"endTime":2759.49,"body":"and I'm going to start with some of the,"},{"startTime":2759.49,"endTime":2761.29,"body":"well actually I'm going to have"},{"startTime":2759.49,"endTime":2761.29,"body":"a little bit of a reflection"},{"startTime":2761.29,"endTime":2762.82,"body":"before we add on questions."},{"startTime":2762.82,"endTime":2765.97,"body":"So I think today thank you"},{"startTime":2762.82,"endTime":2765.97,"body":"again to all our speakers."},{"startTime":2765.97,"endTime":2768.64,"body":"We've learned a bit about"},{"startTime":2765.97,"endTime":2768.64,"body":"using real world evidence"},{"startTime":2768.64,"endTime":2771.34,"body":"in oncology, we've learned about"},{"startTime":2771.34,"endTime":2774.52,"body":"how external control arms can be devised,"},{"startTime":2774.52,"endTime":2776.77,"body":"sort of the best methods for that"},{"startTime":2776.77,"endTime":2779.8,"body":"and then some real life examples of that."},{"startTime":2779.8,"endTime":2780.88,"body":"So thank you very much."},{"startTime":2780.88,"endTime":2784.9,"body":"And then some real critique of how an EAG"},{"startTime":2784.9,"endTime":2787.66,"body":"and a NICE committee have have critiqued"},{"startTime":2787.66,"endTime":2788.83,"body":"some of that real world evidence."},{"startTime":2788.83,"endTime":2790.12,"body":"So that's, that's perfect."},{"startTime":2790.12,"endTime":2791.383,"body":"Thank you very much."},{"startTime":2792.25,"endTime":2794.2,"body":"So I'm going to start"},{"startTime":2792.25,"endTime":2794.2,"body":"with some questions now"},{"startTime":2794.2,"endTime":2798.07,"body":"and I think first question to you, Dan,"},{"startTime":2798.07,"endTime":2800.5,"body":"can you explain how some of the concepts"},{"startTime":2800.5,"endTime":2804.46,"body":"that we've covered can be"},{"startTime":2800.5,"endTime":2804.46,"body":"applied in non-oncology settings?"},{"startTime":2804.46,"endTime":2807.223,"body":"So for example, rare"},{"startTime":2804.46,"endTime":2807.223,"body":"disease or nephrology?"},{"startTime":2810.88,"endTime":2814.66,"body":"- Yeah, sure, I think that"},{"startTime":2810.88,"endTime":2814.66,"body":"it's important to state"},{"startTime":2814.66,"endTime":2815.95,"body":"that we picked oncology here"},{"startTime":2815.95,"endTime":2819.49,"body":"because it's a really good"},{"startTime":2815.95,"endTime":2819.49,"body":"example of a disease area"},{"startTime":2819.49,"endTime":2821.903,"body":"where you're going to have a"},{"startTime":2819.49,"endTime":2821.903,"body":"lot of data sources available"},{"startTime":2821.903,"endTime":2824.32,"body":"and that the outcomes that"},{"startTime":2821.903,"endTime":2824.32,"body":"you need are very well"},{"startTime":2824.32,"endTime":2827.08,"body":"characterised and often easily attainable."},{"startTime":2827.08,"endTime":2829.99,"body":"I say that sometimes PFS"},{"startTime":2827.08,"endTime":2829.99,"body":"isn't the easiest to get"},{"startTime":2829.99,"endTime":2831.76,"body":"and sometimes you have"},{"startTime":2829.99,"endTime":2831.76,"body":"to use proxies for it"},{"startTime":2831.76,"endTime":2833.65,"body":"like time to discontinuation,"},{"startTime":2833.65,"endTime":2835.87,"body":"but there's no reason why"},{"startTime":2833.65,"endTime":2835.87,"body":"any of these principles"},{"startTime":2835.87,"endTime":2837.82,"body":"that we've been describing"},{"startTime":2835.87,"endTime":2837.82,"body":"today can't extend"},{"startTime":2837.82,"endTime":2839.95,"body":"to other disease areas."},{"startTime":2839.95,"endTime":2841.57,"body":"I think the main issues that you're going"},{"startTime":2841.57,"endTime":2845.65,"body":"to run into are the"},{"startTime":2841.57,"endTime":2845.65,"body":"evidence sources themselves."},{"startTime":2845.65,"endTime":2847.993,"body":"So in rare disease for example,"},{"startTime":2848.89,"endTime":2852.22,"body":"you're probably by the virtue"},{"startTime":2848.89,"endTime":2852.22,"body":"of it being a rare disease,"},{"startTime":2852.22,"endTime":2855.07,"body":"you're probably going to"},{"startTime":2852.22,"endTime":2855.07,"body":"find less real world evidence"},{"startTime":2855.07,"endTime":2857.11,"body":"than you would in a more common disease."},{"startTime":2857.11,"endTime":2859.18,"body":"And it really depends on the registries"},{"startTime":2859.18,"endTime":2861.01,"body":"that are available and the outcomes"},{"startTime":2861.01,"endTime":2863.14,"body":"as well that you're measuring."},{"startTime":2863.14,"endTime":2864.46,"body":"So that's why we picked oncology"},{"startTime":2864.46,"endTime":2866.5,"body":"because OS is an objective outcome."},{"startTime":2866.5,"endTime":2868.09,"body":"It's easy to assess, you know,"},{"startTime":2868.09,"endTime":2869.936,"body":"a patient's alive or they're not."},{"startTime":2869.936,"endTime":2873.43,"body":"And a lot of the other outcomes"},{"startTime":2869.936,"endTime":2873.43,"body":"that you find in oncology,"},{"startTime":2873.43,"endTime":2874.51,"body":"real world evidence sources"},{"startTime":2874.51,"endTime":2877.63,"body":"are very well characterised, understood,"},{"startTime":2877.63,"endTime":2879.543,"body":"and have a large amount of data available."},{"startTime":2881.23,"endTime":2882.52,"body":"- Thanks, Dan."},{"startTime":2882.52,"endTime":2884.17,"body":"And then question for you, Joe."},{"startTime":2884.17,"endTime":2887.68,"body":"So sort of building on this,"},{"startTime":2884.17,"endTime":2887.68,"body":"we also asked about the use"},{"startTime":2887.68,"endTime":2889.84,"body":"of real world evidence in the case"},{"startTime":2889.84,"endTime":2893.04,"body":"of missing comparative data"},{"startTime":2889.84,"endTime":2893.04,"body":"in hard to research areas."},{"startTime":2893.04,"endTime":2894.883,"body":"So any sort of thoughts on that?"},{"startTime":2896.56,"endTime":2898.51,"body":"- Yeah, I guess my answer's"},{"startTime":2896.56,"endTime":2898.51,"body":"going to be quite similar"},{"startTime":2898.51,"endTime":2901.87,"body":"to Dan's one because the"},{"startTime":2898.51,"endTime":2901.87,"body":"questions kind of overlap a bit."},{"startTime":2901.87,"endTime":2904.18,"body":"Maybe I can give a"},{"startTime":2901.87,"endTime":2904.18,"body":"perspective from somebody"},{"startTime":2904.18,"endTime":2907.33,"body":"who like has their hands"},{"startTime":2904.18,"endTime":2907.33,"body":"dirty with the data."},{"startTime":2907.33,"endTime":2908.533,"body":"So yeah, obviously the example"},{"startTime":2908.533,"endTime":2910.483,"body":"that I used wasn't in a rare"},{"startTime":2908.533,"endTime":2910.483,"body":"disease or anything like that,"},{"startTime":2910.483,"endTime":2912.58,"body":"but it was in an oncology setting"},{"startTime":2912.58,"endTime":2914.47,"body":"and I think as Dan said, it would be fair"},{"startTime":2914.47,"endTime":2915.97,"body":"to say the majority of examples"},{"startTime":2915.97,"endTime":2919.45,"body":"of real world evidence use"},{"startTime":2915.97,"endTime":2919.45,"body":"are in the oncology setting."},{"startTime":2919.45,"endTime":2921.763,"body":"And this is a consequence of"},{"startTime":2919.45,"endTime":2921.763,"body":"the fact that a lot of the data"},{"startTime":2921.763,"endTime":2925.75,"body":"that is generated in the"},{"startTime":2921.763,"endTime":2925.75,"body":"oncology setting is required"},{"startTime":2925.75,"endTime":2927.1,"body":"to be collected for audit purposes."},{"startTime":2927.1,"endTime":2928.63,"body":"So it's very, it's available,"},{"startTime":2928.63,"endTime":2929.98,"body":"but also a lot of those single arm trials"},{"startTime":2929.98,"endTime":2932.56,"body":"are also in the oncology setting as well."},{"startTime":2932.56,"endTime":2934.03,"body":"But having said that,"},{"startTime":2932.56,"endTime":2934.03,"body":"obviously there's no reason"},{"startTime":2934.03,"endTime":2935.92,"body":"to be approaches that"},{"startTime":2934.03,"endTime":2935.92,"body":"have been discussed today,"},{"startTime":2935.92,"endTime":2937.72,"body":"couldn't be applied to rare diseases."},{"startTime":2937.72,"endTime":2939.607,"body":"And we do see it happening in practice,"},{"startTime":2939.607,"endTime":2941.38,"body":"but I think, you know, we have to accept"},{"startTime":2941.38,"endTime":2942.76,"body":"you're going to see small patient numbers"},{"startTime":2942.76,"endTime":2944.86,"body":"and therefore you're going"},{"startTime":2942.76,"endTime":2944.86,"body":"to have high variance"},{"startTime":2944.86,"endTime":2946.75,"body":"in the evidence that's been generated."},{"startTime":2946.75,"endTime":2950.17,"body":"But insightful evidence"},{"startTime":2946.75,"endTime":2950.17,"body":"regarding comparative use,"},{"startTime":2950.17,"endTime":2953.23,"body":"best supportive care, et cetera,"},{"startTime":2950.17,"endTime":2953.23,"body":"is definitely achievable,"},{"startTime":2953.23,"endTime":2954.54,"body":"particularly with larger data sets."},{"startTime":2954.54,"endTime":2956.65,"body":"We can capture many patients,"},{"startTime":2956.65,"endTime":2958.51,"body":"but they have to have a"},{"startTime":2956.65,"endTime":2958.51,"body":"sufficient granularity"},{"startTime":2958.51,"endTime":2960.73,"body":"that you can assess some of"},{"startTime":2958.51,"endTime":2960.73,"body":"those more complex clinical"},{"startTime":2960.73,"endTime":2962.29,"body":"outcomes are often associated"},{"startTime":2962.29,"endTime":2963.993,"body":"with rare diseases."},{"startTime":2963.993,"endTime":2966.25,"body":"Out here we spend a lot of"},{"startTime":2963.993,"endTime":2966.25,"body":"our time actually collecting"},{"startTime":2966.25,"endTime":2968.081,"body":"and clean that type of data"},{"startTime":2968.081,"endTime":2969.97,"body":"and which required to construct"},{"startTime":2968.081,"endTime":2969.97,"body":"those types of outcomes."},{"startTime":2969.97,"endTime":2971.17,"body":"You know, that data is out there,"},{"startTime":2971.17,"endTime":2974.003,"body":"it just requires a bit of work to get it."},{"startTime":2974.003,"endTime":2976.39,"body":"And then from like the"},{"startTime":2974.003,"endTime":2976.39,"body":"statistical perspective as well,"},{"startTime":2976.39,"endTime":2979.87,"body":"rare diseases offer some"},{"startTime":2976.39,"endTime":2979.87,"body":"slightly different approaches."},{"startTime":2979.87,"endTime":2981.07,"body":"So we might be inclined to reach"},{"startTime":2981.07,"endTime":2982.18,"body":"for some slightly different methods."},{"startTime":2982.18,"endTime":2984.25,"body":"So things such as Bayesian approaches,"},{"startTime":2984.25,"endTime":2986.62,"body":"they're particularly"},{"startTime":2984.25,"endTime":2986.62,"body":"appealing use for ability"},{"startTime":2986.62,"endTime":2988.3,"body":"to incorporate prior knowledge from other"},{"startTime":2988.3,"endTime":2989.92,"body":"studies in a principled way."},{"startTime":2989.92,"endTime":2991.21,"body":"This should allow us to minimise some"},{"startTime":2991.21,"endTime":2994.06,"body":"of the uncertainty"},{"startTime":2991.21,"endTime":2994.06,"body":"associated with small numbers"},{"startTime":2994.06,"endTime":2995.562,"body":"and really squeeze as much information"},{"startTime":2995.562,"endTime":2998.5,"body":"as possible out of the data that's there."},{"startTime":2998.5,"endTime":3001.23,"body":"But yeah, in general, rare diseases,"},{"startTime":3001.23,"endTime":3003.06,"body":"harder to research areas are complicated"},{"startTime":3003.06,"endTime":3005.07,"body":"by small patient numbers requirement"},{"startTime":3005.07,"endTime":3006.63,"body":"of often quite complex data"},{"startTime":3006.63,"endTime":3008.73,"body":"and occasionally complex"},{"startTime":3006.63,"endTime":3008.73,"body":"statistical methods."},{"startTime":3008.73,"endTime":3010.74,"body":"And in my experience earlier,"},{"startTime":3010.74,"endTime":3012.93,"body":"a process for real evidence generation"},{"startTime":3012.93,"endTime":3014.46,"body":"so integrated into the development"},{"startTime":3014.46,"endTime":3017.31,"body":"of the HTA submission in a rarer disease"},{"startTime":3017.31,"endTime":3018.609,"body":"and the higher quality of evidence"},{"startTime":3018.609,"endTime":3019.59,"body":"that it's going to be produced"},{"startTime":3019.59,"endTime":3020.64,"body":"'cause more time can be taken"},{"startTime":3020.64,"endTime":3022.127,"body":"to actually identify the right sources"},{"startTime":3022.127,"endTime":3024.963,"body":"and right statistical methods"},{"startTime":3022.127,"endTime":3024.963,"body":"to generate that evidence."},{"startTime":3026.55,"endTime":3027.393,"body":"- Thanks, Joe."},{"startTime":3028.68,"endTime":3031.23,"body":"So Nick, we've had a"},{"startTime":3028.68,"endTime":3031.23,"body":"lot of questions about"},{"startTime":3031.23,"endTime":3035.64,"body":"how HTA agencies might"},{"startTime":3031.23,"endTime":3035.64,"body":"view real world evidence."},{"startTime":3035.64,"endTime":3038.46,"body":"What do you feel are the"},{"startTime":3035.64,"endTime":3038.46,"body":"sort of unmet opportunities"},{"startTime":3038.46,"endTime":3041.703,"body":"for HTA agencies when it"},{"startTime":3038.46,"endTime":3041.703,"body":"comes to real world evidence?"},{"startTime":3043.11,"endTime":3046.268,"body":"- Yeah, okay, so I mean I think"},{"startTime":3046.268,"endTime":3049.8,"body":"the real world evidence"},{"startTime":3046.268,"endTime":3049.8,"body":"framework document demonstrates"},{"startTime":3049.8,"endTime":3052.977,"body":"that these agencies are"},{"startTime":3049.8,"endTime":3052.977,"body":"thinking a lot about these data"},{"startTime":3052.977,"endTime":3055.89,"body":"and it's not uncommon for"},{"startTime":3052.977,"endTime":3055.89,"body":"these data to be used."},{"startTime":3055.89,"endTime":3060.0,"body":"So use is being made, my personal opinion,"},{"startTime":3060.0,"endTime":3061.11,"body":"I think there's two areas"},{"startTime":3061.11,"endTime":3062.55,"body":"where they could use it a lot more."},{"startTime":3062.55,"endTime":3066.993,"body":"So the first is in terms of"},{"startTime":3062.55,"endTime":3066.993,"body":"reviews of previous decisions."},{"startTime":3068.16,"endTime":3071.91,"body":"At the moment, you know, a"},{"startTime":3068.16,"endTime":3071.91,"body":"lot of NICE decision making,"},{"startTime":3071.91,"endTime":3074.64,"body":"a lot NICE recommendations"},{"startTime":3071.91,"endTime":3074.64,"body":"say they'll be reviewed"},{"startTime":3074.64,"endTime":3078.15,"body":"in X number of years, two or"},{"startTime":3074.64,"endTime":3078.15,"body":"three years, or something,"},{"startTime":3078.15,"endTime":3079.92,"body":"but quite often that's not really done"},{"startTime":3079.92,"endTime":3082.303,"body":"in a particularly meaningful way."},{"startTime":3082.303,"endTime":3086.46,"body":"And I think it could be"},{"startTime":3082.303,"endTime":3086.46,"body":"using real world data once,"},{"startTime":3086.46,"endTime":3089.25,"body":"if a treatment had a"},{"startTime":3086.46,"endTime":3089.25,"body":"positive recommendation,"},{"startTime":3089.25,"endTime":3091.92,"body":"then it's been used for"},{"startTime":3089.25,"endTime":3091.92,"body":"that number of years."},{"startTime":3091.92,"endTime":3094.17,"body":"You can look, look to see"},{"startTime":3091.92,"endTime":3094.17,"body":"what's actually happening"},{"startTime":3094.17,"endTime":3095.25,"body":"and review the decision."},{"startTime":3095.25,"endTime":3097.11,"body":"I think that would be useful."},{"startTime":3097.11,"endTime":3100.32,"body":"And then the more kind of obvious one"},{"startTime":3100.32,"endTime":3103.17,"body":"is the Cancer Drugs Fund appraisals"},{"startTime":3103.17,"endTime":3107.82,"body":"where the drugs in the Cancer"},{"startTime":3103.17,"endTime":3107.82,"body":"Drugs Fund for a while,"},{"startTime":3107.82,"endTime":3110.37,"body":"normally two years, sometimes longer."},{"startTime":3110.37,"endTime":3112.68,"body":"And then when the review is done,"},{"startTime":3112.68,"endTime":3115.833,"body":"one of the key data sources is from SACT."},{"startTime":3116.91,"endTime":3120.57,"body":"But what you find is"},{"startTime":3116.91,"endTime":3120.57,"body":"that the data from SACT"},{"startTime":3120.57,"endTime":3122.88,"body":"is always just for the experimental drug,"},{"startTime":3122.88,"endTime":3124.65,"body":"which leaves you with problems in terms of"},{"startTime":3124.65,"endTime":3125.82,"body":"if you want to use that data"},{"startTime":3125.82,"endTime":3128.52,"body":"to see what's the treatment"},{"startTime":3125.82,"endTime":3128.52,"body":"effect in the real world."},{"startTime":3128.52,"endTime":3131.16,"body":"You can't really just with that data."},{"startTime":3131.16,"endTime":3134.91,"body":"So obviously the data on"},{"startTime":3131.16,"endTime":3134.91,"body":"the comparator is there"},{"startTime":3134.91,"endTime":3137.88,"body":"from prior years before"},{"startTime":3134.91,"endTime":3137.88,"body":"that new drug had been"},{"startTime":3137.88,"endTime":3140.4,"body":"kind of putting the Cancer Drugs Fund,"},{"startTime":3140.4,"endTime":3142.02,"body":"I would like to see much more use"},{"startTime":3142.02,"endTime":3145.38,"body":"of comparative effectiveness"},{"startTime":3142.02,"endTime":3145.38,"body":"analysis using data"},{"startTime":3145.38,"endTime":3148.143,"body":"from both the comparator"},{"startTime":3145.38,"endTime":3148.143,"body":"and the experimental drug."},{"startTime":3149.37,"endTime":3151.32,"body":"- Yeah, and as you say,"},{"startTime":3149.37,"endTime":3151.32,"body":"that data should be,"},{"startTime":3151.32,"endTime":3153.9,"body":"should be there, so why not use it?"},{"startTime":3153.9,"endTime":3155.22,"body":"Yeah."},{"startTime":3155.22,"endTime":3157.65,"body":"Okay, question for you, Dan."},{"startTime":3157.65,"endTime":3159.66,"body":"Have you had a sort of direct experience"},{"startTime":3159.66,"endTime":3163.29,"body":"with NICE accepting real"},{"startTime":3159.66,"endTime":3163.29,"body":"world evidence that changed?"},{"startTime":3163.29,"endTime":3166.8,"body":"Sorry, the committee's decision making?"},{"startTime":3166.8,"endTime":3169.36,"body":"- Yeah, well, I think"},{"startTime":3166.8,"endTime":3169.36,"body":"we've seen an example"},{"startTime":3170.46,"endTime":3173.16,"body":"that Nick just went"},{"startTime":3170.46,"endTime":3173.16,"body":"through of an appraisal"},{"startTime":3173.16,"endTime":3177.87,"body":"where real evidence was used as part"},{"startTime":3177.87,"endTime":3179.43,"body":"of the main evidence submission."},{"startTime":3179.43,"endTime":3181.83,"body":"And speaking from my own experience,"},{"startTime":3181.83,"endTime":3184.14,"body":"I think I've taken part"},{"startTime":3181.83,"endTime":3184.14,"body":"in NICE submissions"},{"startTime":3184.14,"endTime":3186.54,"body":"that I don't think would've"},{"startTime":3184.14,"endTime":3186.54,"body":"been possible at all"},{"startTime":3186.54,"endTime":3188.46,"body":"without the use of real world evidence."},{"startTime":3188.46,"endTime":3191.58,"body":"I simply don't think that"},{"startTime":3188.46,"endTime":3191.58,"body":"it would've gone through."},{"startTime":3191.58,"endTime":3193.32,"body":"You could've made an evidence submission"},{"startTime":3193.32,"endTime":3194.153,"body":"because there would've been no"},{"startTime":3194.153,"endTime":3196.143,"body":"comparative evidence available."},{"startTime":3197.46,"endTime":3199.56,"body":"So I think more than just"},{"startTime":3197.46,"endTime":3199.56,"body":"influencing a decision,"},{"startTime":3199.56,"endTime":3201.39,"body":"I think it also makes it possible"},{"startTime":3201.39,"endTime":3204.36,"body":"to make a decision as well."},{"startTime":3204.36,"endTime":3206.76,"body":"Or, you know, rather than the appraisal"},{"startTime":3206.76,"endTime":3207.78,"body":"not going ahead at all."},{"startTime":3207.78,"endTime":3210.66,"body":"I think it may have happened"},{"startTime":3207.78,"endTime":3210.66,"body":"just much later in the future"},{"startTime":3210.66,"endTime":3212.52,"body":"when you have a phase three trial"},{"startTime":3212.52,"endTime":3215.31,"body":"and all that time you are,"},{"startTime":3212.52,"endTime":3215.31,"body":"you know, there are patients"},{"startTime":3215.31,"endTime":3217.59,"body":"who are missing out potentially on access"},{"startTime":3217.59,"endTime":3220.35,"body":"to drugs when potentially"},{"startTime":3217.59,"endTime":3220.35,"body":"access could have been achieved"},{"startTime":3220.35,"endTime":3223.05,"body":"earlier with the use"},{"startTime":3220.35,"endTime":3223.05,"body":"of real-world evidence."},{"startTime":3223.05,"endTime":3226.68,"body":"So yeah, it's it definitely does influence"},{"startTime":3226.68,"endTime":3229.17,"body":"the decision making and"},{"startTime":3226.68,"endTime":3229.17,"body":"it even makes it possible"},{"startTime":3229.17,"endTime":3230.12,"body":"to make a decision."},{"startTime":3231.42,"endTime":3232.253,"body":"- Thank you."},{"startTime":3233.7,"endTime":3237.87,"body":"Joe, question for you sort"},{"startTime":3233.7,"endTime":3237.87,"body":"of looking beyond HTA,"},{"startTime":3237.87,"endTime":3239.97,"body":"how can real world"},{"startTime":3237.87,"endTime":3239.97,"body":"evidence contribute to sort"},{"startTime":3239.97,"endTime":3241.5,"body":"of long term monitoring"},{"startTime":3241.5,"endTime":3244.05,"body":"of health technologies post-approval?"},{"startTime":3244.05,"endTime":3247.14,"body":"And what role can it play"},{"startTime":3244.05,"endTime":3247.14,"body":"in reassessing technologies"},{"startTime":3247.14,"endTime":3249.153,"body":"as new health data emerges?"},{"startTime":3250.71,"endTime":3253.32,"body":"- Yeah, so I mean naturally"},{"startTime":3250.71,"endTime":3253.32,"body":"real world evidence does play"},{"startTime":3253.32,"endTime":3255.9,"body":"a key role in post-authorisation studies."},{"startTime":3255.9,"endTime":3257.61,"body":"I think it'd be fair to say"},{"startTime":3255.9,"endTime":3257.61,"body":"this is probably the most"},{"startTime":3257.61,"endTime":3260.58,"body":"established use of real world"},{"startTime":3257.61,"endTime":3260.58,"body":"data in a regulatory setting"},{"startTime":3260.58,"endTime":3261.93,"body":"or a reimbursement setting."},{"startTime":3263.16,"endTime":3265.44,"body":"And you know, as information"},{"startTime":3263.16,"endTime":3265.44,"body":"content in the electronic"},{"startTime":3265.44,"endTime":3267.333,"body":"healthcare records is only growing,"},{"startTime":3268.32,"endTime":3269.94,"body":"sort of the effectiveness of these types"},{"startTime":3269.94,"endTime":3272.04,"body":"of analysis will also only grow."},{"startTime":3272.04,"endTime":3273.96,"body":"But I think I in general,"},{"startTime":3272.04,"endTime":3273.96,"body":"I would actually just agree"},{"startTime":3273.96,"endTime":3276.42,"body":"with a lot of what Nick was"},{"startTime":3273.96,"endTime":3276.42,"body":"saying, his answer previously,"},{"startTime":3276.42,"endTime":3278.88,"body":"you know, the Cancer Drugs"},{"startTime":3276.42,"endTime":3278.88,"body":"Fund is a really clear example"},{"startTime":3278.88,"endTime":3283.29,"body":"of post authorisation use,"},{"startTime":3278.88,"endTime":3283.29,"body":"really where the drug has entered"},{"startTime":3283.29,"endTime":3284.49,"body":"through a patient access scheme"},{"startTime":3284.49,"endTime":3286.8,"body":"and data's being collected"},{"startTime":3284.49,"endTime":3286.8,"body":"in a real world setting"},{"startTime":3286.8,"endTime":3288.48,"body":"and then a number of years later"},{"startTime":3288.48,"endTime":3290.43,"body":"that data's being reassessed."},{"startTime":3290.43,"endTime":3292.32,"body":"And I would completely agree"},{"startTime":3290.43,"endTime":3292.32,"body":"with what Nick said about"},{"startTime":3292.32,"endTime":3294.81,"body":"the comparator, you know,"},{"startTime":3292.32,"endTime":3294.81,"body":"that data is sat there,"},{"startTime":3294.81,"endTime":3296.16,"body":"the same data was being collected"},{"startTime":3296.16,"endTime":3300.84,"body":"for, the novel therapy is being collected"},{"startTime":3300.84,"endTime":3304.143,"body":"just through routine"},{"startTime":3300.84,"endTime":3304.143,"body":"care for the comparator."},{"startTime":3305.292,"endTime":3306.81,"body":"And this same principle"},{"startTime":3305.292,"endTime":3306.81,"body":"applies essentially"},{"startTime":3306.81,"endTime":3309.84,"body":"for any situation where"},{"startTime":3306.81,"endTime":3309.84,"body":"there's been a recommendation"},{"startTime":3309.84,"endTime":3313.11,"body":"from NICE and the drug is"},{"startTime":3309.84,"endTime":3313.11,"body":"entered routine commissioning,"},{"startTime":3313.11,"endTime":3315.092,"body":"all the drugs that are"},{"startTime":3313.11,"endTime":3315.092,"body":"involved in that process"},{"startTime":3315.092,"endTime":3317.45,"body":"are being used in the NHS"},{"startTime":3315.092,"endTime":3317.45,"body":"will be generating data"},{"startTime":3317.45,"endTime":3319.25,"body":"in the electronic healthcare records"},{"startTime":3320.275,"endTime":3324.45,"body":"and that data is"},{"startTime":3320.275,"endTime":3324.45,"body":"available for reassessment"},{"startTime":3324.45,"endTime":3326.4,"body":"future date if required."},{"startTime":3326.4,"endTime":3328.35,"body":"So yeah, I was completely,"},{"startTime":3326.4,"endTime":3328.35,"body":"I'd be completely on board"},{"startTime":3328.35,"endTime":3331.2,"body":"with Nick's answer to his question"},{"startTime":3331.2,"endTime":3333.09,"body":"and the similarities"},{"startTime":3333.09,"endTime":3335.55,"body":"to the question there"},{"startTime":3333.09,"endTime":3335.55,"body":"about post authorisation."},{"startTime":3335.55,"endTime":3336.383,"body":"Yeah."},{"startTime":3337.792,"endTime":3338.838,"body":"- Okay, great, thank you."},{"startTime":3338.838,"endTime":3341.388,"body":"I think we've got time for"},{"startTime":3338.838,"endTime":3341.388,"body":"a couple more questions."},{"startTime":3342.33,"endTime":3344.13,"body":"I guess coming back to"},{"startTime":3342.33,"endTime":3344.13,"body":"you again, Joe, sorry,"},{"startTime":3344.13,"endTime":3345.66,"body":"to pick on you again,"},{"startTime":3345.66,"endTime":3349.5,"body":"but what role can machine learning"},{"startTime":3349.5,"endTime":3352.71,"body":"or AI play when we're"},{"startTime":3349.5,"endTime":3352.71,"body":"generating real world evidence"},{"startTime":3352.71,"endTime":3354.613,"body":"to support HTA process?"},{"startTime":3355.47,"endTime":3357.69,"body":"- Yeah, I mean it's"},{"startTime":3355.47,"endTime":3357.69,"body":"obviously like AI and ML"},{"startTime":3357.69,"endTime":3359.04,"body":"are like hot topics at the moment"},{"startTime":3359.04,"endTime":3360.06,"body":"and it seems like these concepts"},{"startTime":3360.06,"endTime":3361.62,"body":"are being applied everywhere."},{"startTime":3361.62,"endTime":3363.33,"body":"But I think when it comes to"},{"startTime":3361.62,"endTime":3363.33,"body":"the assessment of, you know,"},{"startTime":3363.33,"endTime":3364.96,"body":"novel therapies, we do"},{"startTime":3363.33,"endTime":3364.96,"body":"need to be careful about"},{"startTime":3364.96,"endTime":3366.693,"body":"where these methods are applied."},{"startTime":3367.71,"endTime":3368.97,"body":"Many of these methods are essentially"},{"startTime":3368.97,"endTime":3370.08,"body":"like black box methods,"},{"startTime":3370.08,"endTime":3371.58,"body":"meaning you don't really"},{"startTime":3370.08,"endTime":3371.58,"body":"have any understanding"},{"startTime":3371.58,"endTime":3374.601,"body":"of how they've arrived at"},{"startTime":3371.58,"endTime":3374.601,"body":"result that they've generated."},{"startTime":3374.601,"endTime":3375.99,"body":"And you know, when it comes to assessing"},{"startTime":3375.99,"endTime":3378.72,"body":"whether a technology is,"},{"startTime":3375.99,"endTime":3378.72,"body":"you know, effective or safe,"},{"startTime":3378.72,"endTime":3380.76,"body":"it's not particularly reassuring."},{"startTime":3380.76,"endTime":3382.89,"body":"So yeah, for more traditional"},{"startTime":3380.76,"endTime":3382.89,"body":"statistical methods, you know,"},{"startTime":3382.89,"endTime":3386.22,"body":"parametric approaches, survival curves"},{"startTime":3386.22,"endTime":3388.14,"body":"and so on, you know, they're here to stay"},{"startTime":3388.14,"endTime":3390.21,"body":"because they have clear assumptions"},{"startTime":3390.21,"endTime":3391.53,"body":"about how the data is being modelled"},{"startTime":3391.53,"endTime":3394.083,"body":"and they can be, they"},{"startTime":3391.53,"endTime":3394.083,"body":"can be tested directly."},{"startTime":3395.379,"endTime":3396.69,"body":"So you can assess whether these methods"},{"startTime":3396.69,"endTime":3398.7,"body":"are appropriate for their use."},{"startTime":3398.7,"endTime":3400.62,"body":"But having said that, you know,"},{"startTime":3398.7,"endTime":3400.62,"body":"I think there are some areas"},{"startTime":3400.62,"endTime":3405.18,"body":"where AI or ML methods"},{"startTime":3400.62,"endTime":3405.18,"body":"do have some utility."},{"startTime":3405.18,"endTime":3407.76,"body":"And I think a really clear"},{"startTime":3405.18,"endTime":3407.76,"body":"example is data enrichment"},{"startTime":3407.76,"endTime":3411.06,"body":"and content extraction."},{"startTime":3411.06,"endTime":3415.08,"body":"So at Arcturis we have an ML team who work"},{"startTime":3415.08,"endTime":3417.12,"body":"with the free text"},{"startTime":3415.08,"endTime":3417.12,"body":"documents that we access."},{"startTime":3417.12,"endTime":3419.641,"body":"You know, we've got sometimes"},{"startTime":3417.12,"endTime":3419.641,"body":"tens of thousands of these"},{"startTime":3419.641,"endTime":3420.96,"body":"and we need to pass them quickly"},{"startTime":3420.96,"endTime":3423.81,"body":"and they can develop these"},{"startTime":3420.96,"endTime":3423.81,"body":"ML models which can extract"},{"startTime":3423.81,"endTime":3426.81,"body":"the key data from these"},{"startTime":3423.81,"endTime":3426.81,"body":"unstructured sources,"},{"startTime":3426.81,"endTime":3427.983,"body":"turning them into structured sources"},{"startTime":3427.983,"endTime":3430.08,"body":"that can then be leveraged for analysis."},{"startTime":3430.08,"endTime":3432.81,"body":"And obviously there needs"},{"startTime":3430.08,"endTime":3432.81,"body":"to be some assessment"},{"startTime":3432.81,"endTime":3435.777,"body":"of the precision and"},{"startTime":3432.81,"endTime":3435.777,"body":"accuracy of these methods,"},{"startTime":3435.777,"endTime":3437.91,"body":"but you know, that's completely"},{"startTime":3435.777,"endTime":3437.91,"body":"something that can be done."},{"startTime":3437.91,"endTime":3440.46,"body":"So I think our ability"},{"startTime":3440.46,"endTime":3442.8,"body":"to take an electronic"},{"startTime":3440.46,"endTime":3442.8,"body":"healthcare record system"},{"startTime":3442.8,"endTime":3446.55,"body":"and turn it into an analysable"},{"startTime":3442.8,"endTime":3446.55,"body":"data source using AI methods"},{"startTime":3446.55,"endTime":3449.373,"body":"is a potentially really,"},{"startTime":3446.55,"endTime":3449.373,"body":"really useful application."},{"startTime":3451.83,"endTime":3454.2,"body":"- And then following on"},{"startTime":3451.83,"endTime":3454.2,"body":"from that a little bit,"},{"startTime":3454.2,"endTime":3457.8,"body":"I know NICE have just"},{"startTime":3454.2,"endTime":3457.8,"body":"sort of released a bit"},{"startTime":3457.8,"endTime":3459.81,"body":"of a statement on the use of AI,"},{"startTime":3459.81,"endTime":3463.26,"body":"but it's a little bit"},{"startTime":3459.81,"endTime":3463.26,"body":"unclear and a bit woolly,"},{"startTime":3463.26,"endTime":3464.88,"body":"but do you think that will be something"},{"startTime":3464.88,"endTime":3467.79,"body":"that in the future NICE will"},{"startTime":3464.88,"endTime":3467.79,"body":"be more happy to accept?"},{"startTime":3467.79,"endTime":3470.46,"body":"And I guess I'm opening"},{"startTime":3467.79,"endTime":3470.46,"body":"the floor to anyone,"},{"startTime":3470.46,"endTime":3472.023,"body":"any of our experts on that one?"},{"startTime":3477.63,"endTime":3479.61,"body":"Dan or Nick, you got a thought on that?"},{"startTime":3479.61,"endTime":3482.1,"body":"- Yeah, I mean it's,"},{"startTime":3479.61,"endTime":3482.1,"body":"it's not specific to AI,"},{"startTime":3482.1,"endTime":3484.71,"body":"but I think it's also"},{"startTime":3482.1,"endTime":3484.71,"body":"relevant to AI in any methods"},{"startTime":3484.71,"endTime":3487.44,"body":"that use things like that,"},{"startTime":3484.71,"endTime":3487.44,"body":"that are quite novel."},{"startTime":3487.44,"endTime":3490.74,"body":"Like a lot is down to reporting I think."},{"startTime":3490.74,"endTime":3493.53,"body":"Quite often when complex methods,"},{"startTime":3493.53,"endTime":3496.38,"body":"or new methods are used in submissions,"},{"startTime":3496.38,"endTime":3499.08,"body":"they're not actually that well reported."},{"startTime":3499.08,"endTime":3501.54,"body":"And so for a reviewer"},{"startTime":3499.08,"endTime":3501.54,"body":"or a committee member,"},{"startTime":3501.54,"endTime":3503.88,"body":"it's very difficult to know to what extent"},{"startTime":3503.88,"endTime":3505.2,"body":"you can trust these analyses"},{"startTime":3505.2,"endTime":3507.3,"body":"if they haven't been reported well."},{"startTime":3507.3,"endTime":3509.49,"body":"So I think that's a really key thing."},{"startTime":3509.49,"endTime":3512.43,"body":"The other thing is proof of concept"},{"startTime":3512.43,"endTime":3515.25,"body":"that these methods work"},{"startTime":3512.43,"endTime":3515.25,"body":"is an important thing."},{"startTime":3515.25,"endTime":3519.18,"body":"So more broadly than just the AI stuff,"},{"startTime":3519.18,"endTime":3521.13,"body":"but benchmarking studies are things"},{"startTime":3521.13,"endTime":3523.23,"body":"that people have started doing more."},{"startTime":3523.23,"endTime":3525.69,"body":"So in the US there was something"},{"startTime":3525.69,"endTime":3528.57,"body":"called the RCT-DUPLICATE Study"},{"startTime":3528.57,"endTime":3531.63,"body":"where they used US health data"},{"startTime":3531.63,"endTime":3534.69,"body":"to try to emulate existing RCTs"},{"startTime":3534.69,"endTime":3536.64,"body":"and see if you could get similar results"},{"startTime":3536.64,"endTime":3538.38,"body":"using the real world data."},{"startTime":3538.38,"endTime":3541.62,"body":"We are doing similar work to"},{"startTime":3538.38,"endTime":3541.62,"body":"that using English cancer data"},{"startTime":3541.62,"endTime":3543.09,"body":"at Sheffield actually."},{"startTime":3543.09,"endTime":3546.54,"body":"But I think if you want"},{"startTime":3543.09,"endTime":3546.54,"body":"to use a data set proving"},{"startTime":3546.54,"endTime":3548.25,"body":"that it can give reliable results"},{"startTime":3548.25,"endTime":3551.04,"body":"by benchmarking against an existing RCT"},{"startTime":3551.04,"endTime":3555.36,"body":"and then doing the analysis"},{"startTime":3551.04,"endTime":3555.36,"body":"you actually need to contribute"},{"startTime":3555.36,"endTime":3559.47,"body":"to the submission you need it"},{"startTime":3555.36,"endTime":3559.47,"body":"for is also a good process,"},{"startTime":3559.47,"endTime":3561.42,"body":"which I don't think has been done much."},{"startTime":3563.929,"endTime":3567.21,"body":"- Okay, well, we're just up"},{"startTime":3563.929,"endTime":3567.21,"body":"on the hour so I just want"},{"startTime":3567.21,"endTime":3570.06,"body":"to wrap up by saying thank you everybody"},{"startTime":3570.06,"endTime":3572.07,"body":"for attending the webinar."},{"startTime":3572.07,"endTime":3574.74,"body":"The webinar is available I think on the,"},{"startTime":3574.74,"endTime":3577.02,"body":"one of the links at the"},{"startTime":3574.74,"endTime":3577.02,"body":"top and the materials,"},{"startTime":3577.02,"endTime":3578.88,"body":"but we will be following up with everybody"},{"startTime":3578.88,"endTime":3581.85,"body":"and for everyone who has put in questions"},{"startTime":3581.85,"endTime":3584.64,"body":"that we haven't got round to answering,"},{"startTime":3584.64,"endTime":3587.37,"body":"we will endeavour to follow"},{"startTime":3584.64,"endTime":3587.37,"body":"up with as many questions"},{"startTime":3587.37,"endTime":3589.02,"body":"as possible within the next two weeks."},{"startTime":3589.02,"endTime":3593.273,"body":"So thank you very much for"},{"startTime":3589.02,"endTime":3593.273,"body":"your time this afternoon."}]}