Inside the TGA: The Rules Governing Medical Technology

In this episode, Stephen and Dr. Chee sit down with Dr. Lee Walsh, former Digital Health Lead at the TGA and current head of Platypus MedTech Consulting. Together, they pull back the curtain on the complex, fast-moving, and often misunderstood world of health AI regulation.  

From the high-stakes environment of radiology to the quiet explosion of consumer-facing chatbots like ChatGPT, Dr. Walsh breaks down how regulators evaluate safety, why the current system is actually more flexible than you think, and the terrifying reality of what happens when free, unvetted AI undercuts evidence-based medical technology. 

If you are a clinician, technologist, or patient wondering who is actually keeping the safety net intact as medicine digitizes, this conversation is an absolute must-watch.  

What You Will Learn:

The Blueprint of Innovation:  How the medical device framework is built on principles rather than strict prohibitions, forcing inventors to define safety standards from first principles.  

The Radiology Legacy: Why medical imaging holds a unique, highly protected regulatory status due to its history with radiation and definitive diagnostic power.  

Locked Models vs. Generative Chaos: The staggering regulatory difference between purpose-built machine learning models and general-use LLMs that can hallucinate or change day-to-day.  

The Commercial Death Loop: Why rigorous, evidence-based clinical AI tools are struggling to compete against free, unregulated consumer chatbots.  

The Actionable Framework for Practice Owners: How to implement ISO 42001 or use a first-principles failure mindset to aggressively vet AI vendors, protect data privacy, and establish clear escalation pathways

Transcript

SPEAKER_00 0:06

Welcome to MIAI, the podcast about medical imaging and artificial intelligence. Sit down with the most brilliant voices in radiology and AI to discover how technology is reshaping healthcare delivery today.

SPEAKER_03 0:22

Hi, my name is David Khan Hayes. And today we're looking at health AI regulation. It's a hot topic for many reasons. If you slow down to progress too much, not only do you miss out on the benefits, but in an area with so much promise like AI, you get beaten in business and you fall behind as a society and country. As an area and someone who sees such social benefit in technology, I'm naturally worried about regulators getting in the way. On the other hand, with intense doomsday scenarios being forecast, government control is necessary for AI to maintain a social license and also to stop the real negative possibilities. As a father and husband and someone with a big social conscience, I'm also worried about the dangers, both in the short term and for our children. Gee, I'm especially interested in the regulator's view on AI and how that's impacted by social pressure. I'd love to know specifically about radiology technology and how that's viewed and how it's reviewed and how you get through to be approved to be able to work in the med tech space with something in radiology around AI. And generally the larger social license of AI and how that, you know, how what Lee's perspective on that and yours?

SPEAKER_01 1:36

Look, these are big topics. So I'm glad we have today a very big guest to help us through all this. Joined by Dr. Lee Walsh, who's a leader in this exact space. He's a former lead assessor, lead investigator, and technical lead for the digital health at the TGA. He's a senior research officer and former research fellow. Lee is also chartered engineer and has a PhD in neuroscience, proprioception and human movement. And he's currently leading Tadpus MedTech Consulting, focused on advising medical technology companies through government regulation in this new world of AI. Thanks for coming on this show, Lee. That's a lot of experience in this space. Have I missed anything?

SPEAKER_02 2:15

Probably, but we don't need to get into too much of it. I've done a lot of things in my time in MedTech, particularly health and and um medical science. But yeah, look, I think that the relevance to this discussion is a background in the RD itself. I do make products and have made products. I'm on the engineering and the physiology side. I practice both. And obviously my experience in government, nothing I say today is like I can no longer speak for the government. I no longer work for the TGA. I know we're going to be talking a lot about what government thinks, what government does. But yeah, that's probably the most of it.

SPEAKER_01 2:53

Well it gives you license to say what you want and you're not working for the government then, isn't it, Lee?

SPEAKER_02 2:58

Yeah, within reason.

SPEAKER_03 2:59

I I'm I think there's still a lot there, Lee. I think there's still a lot because you would have understood the impetus, and that's really what we're regulations come and come and go, and the fine-tuning of that regulation will change. But the pressures on what drives that regulation, I think, still really interesting. And especially how that connects to the social contract for AI, because just like any organization or or area of society, society needs to decide, hey, this is good and we want it within certain confines. So I'd love to understand that. And I would imagine also that it can be a pretty thankless job because you kind of only find out about regulation when you're upset about it or when you're complaining about the red tape or how it failed. So how did you get into it in the first place?

SPEAKER_02 3:44

Yeah, look, I I got into it basically because I as as Chi said, used to be uh an HMRC research fellow. Career was dependent on funding from the government, basically, and that started to dry up, and uh it was pretty difficult to find work in the university and research sector at the time. But the TGA was hiring medical scientists or engineers um going through a bit of an intake after the hiring freeze that they'd been under for a while was lifted. And for me also, I was a bit sick of living in Sydney. I grew up in the country and spent the first 10 or 15 years of my career in the city, so I was ready to go to a bit of a smaller town. But that's basically how it started as a um a medical device assessor in the engineering team in the TGA.

SPEAKER_03 4:32

And from there you took an interest or they took an interest in you in moving more into the policy direction, more into you know, interpretation of those assessments.

SPEAKER_02 4:43

It's sort of the way government works, especially inside a regulator, you know, the day-to-day business of a regulator, we when we're outside of government, we think of it as their job is to assess applications and make decisions on behalf of the people of Australia. But an important part of the regulator is is also the policy and you know, there's a public engagement aspect, explaining the regulation to people. And also, I suppose intelligence gathering is one way to put about it, like knowing what's going on in the industry you're regulating. Um and I think because of my background and my own sort of experience as a communicator and a public speaker, I naturally moved towards some of those roles as I have in sort of every role I've had. And I was fortunate that I got to move around inside the TGA a little bit. So I started as a an engineering assessor, but then I moved into a role in the labs and actually headed up one of the groups within the engineering lab, so the electrical and software testing. So I got to see the post-market side of that as well. And then through my sort of own technical background in software and electronics, when the TGA started to have to deal with things like cybersecurity, medical devices, new software legislation, and harmonising this with other countries that we like to trade with, I had some of the expertise that was needed. There aren't many engineers in the TGA as it is, and once you divide us up amongst all the different disciplines of engineering, it's even harder. And at the time I was one of two software engineers in building, and I think one of two electrical engineers at the time as well. So it was just a bit of who was available and who had the skills when when the policy stuff needed to be written. Okay. Um, I think engineers naturally fall into that a little bit as well because we're often involved in writing standards. And by the time I was doing policy in the TGA, I'd been working on and drafting Australian International Standards for a couple of years. So again, it's it's a skill set that sort of is similar.

SPEAKER_01 6:47

Yeah. So Lee worked for research and then the government dried up your funding, so you decided join them, became a government employee, and now you've moved on from there. Tell us about that journey.

SPEAKER_02 7:02

Yeah, look, you're reminding me of something one of my um SD supervisors said to me when I left my postdoc early to join the public service. She said, Congratulations, you've now got a job where keeping your job depends on your performance. And that's it. Isn't that the way jobs are supposed to work? I guess. Yeah, but when when you're dependent on government funding, it's basically a lottery about whether you get to keep your job. Oh, okay. It was pretty rough. I don't know what it's like now. It might be the same, it might have improved, but it was pretty rough, particularly as a postdoc trying to start my own lab. So, yeah, look, I joined, and in fact, the same department, the same minister who wasn't signing my grant checks. But yeah, look, it at the time it it was a big change. It was going from year to year having to find funding to keep keep my job going to basically having what's in the academic sense to be the equivalent of a tenured role. So that's ultimately why I decided to go out into the private sector.

SPEAKER_03 8:03

And Lee, I'm really interested in exactly where that took you because now you're, as I understand it, helping people navigate that regulation and align to it effectively to bring forward good medical technology products and get them being used. So can you tell us a little bit about that company, Platypus Medical MedTech Consulting, and how what it does and what you're involved in?

SPEAKER_02 8:26

Yeah, so and that that sort of happened a little bit by accident. So I did do quite a bit of public engagement when I was at the TGA seminars, webinars, podcasts like this. So people knew me and then um I actually left the health industry briefly. I think it was about six months, and the company I left to work for got bought by a larger multinational. Uh I made the decision not to stay because what had happened is all the people who used to hassle me when I was at the TGA had started hearing I was out and actually started approaching me and saying, Look, well, now that you're out, you can actually help us. You know, this is the customer needs aspect, right? There was this this need, and I basically founded a business around that need. So Platypus MedTech is that business, and I think most people see us as a regulatory affairs firm, and we we definitely are. So helping people navigate, understand the regulations and how they apply to their product is you know central to what we do. But there's so much around the regulation of medical devices that I think people overlook. Probably 90% of our clients come to us looking for help to fill out the paperwork with the TGA, but more than 90% of what we do with them is all the preparation before that. So quality management systems, building those from scratch in a small business when you've never done it before is not an easy thing to do. Quality management is an old thing. It comes from you know the 1950s when standardization started. It's an industry thing. They invented it, not government. But when you've never done it before and the government says you have to, it's it's really are. But we're also engineers, medical scientists, so we can also help with the engineering process. We've we've built products, we've done research, we've done clinical trials. So we're actually much broader than regulatory affairs. We talk about ourselves as being an engineering, regulatory and quality firm who can also do a bit of medical science and clinical evaluation within the right areas. And for us, again, just because of who we are as researchers and entrepreneurs, we have a particular specialty in research translation and novel technology. So we work with a large number of startups, we work with a lot of university departments, a lot of small businesses who are trying to bring their first health or medical product to market. And if you're innovating, there's no checklist to follow. If you're making another ventilator, great, good for you. Good ventilators are important, but we've had ventilators on the market for a long time. There is a sort of there are clear standards and pathways for what's needed to get that approved. If you've got a novel product, nobody's done this before, it's the first of its kind or one of the first of its kind, there aren't any standards to follow. Um the TGA may never have seen this before either. So it falls back on you as the designer, inventor, and the manufacturer to not just apply the standards, but almost to work out what the standards are supposed to be from first principles. Yeah. Um and if you're not a researcher or an engineer, that's really challenging. And even if you are, you may not have done it in this context. So a lot of what we do is actually around that piece in in finding a pathway for novel products, particularly in the digital health space, again, because of our background in electronics and and software.

SPEAKER_03 11:44

Yeah. So this part's really interesting to me because it goes back to exactly how I started this episode, which was you want to let in the good and stop the bad. And part of letting in the good is by being open-minded about these new things, but you don't have regulation for these new things. So I'm assuming is you know it's it's pretty hard. You haven't got a standard to follow.

SPEAKER_02 12:09

It's an interesting misunderstanding that's very common that we need to have regulation for every type of product. The the system that has been built in Australia, which is is harmonized globally, facilitates improvement and innovation because it's built on principles. So it's got very few prohibitions in it, very few explicit requirements. But the downside of that is it forces the inventor to work out what the requirements should be, convince the government, and then assess against them. So we we're in this situation where it's not the government's job to work out the standards. Industry does that, and the inventors and the developers making new products need to do that through good science, good engineering, running their own clinical trials, peer-reviewed literature, all of that. Government's job is to make sure they're doing their job properly in this case. But your sort of comment about, you know, we don't have regulations for new products, that's probably the most common perception people have, even though this framework was actually built from the ground up to deal with that problem. And it's it's much, much more flexible than people give it credit for.

SPEAKER_03 13:13

That's great to hear, Lee, because I'm really interested as an entrepreneur and a technologist about opening the door to the things that improve society and making that as easy as possible. And I think in Australia, that's a problem that we probably haven't been that great with in the past. Not around as a as a country, not particularly around MedTech. But as you would know from this podcast and the name of the podcast, MIAI, we are about medical imaging artificial intelligence. So I'd love to pull us back a little bit to specifically talk about radiology, medical imaging, and what that means from a med tech regulation perspective. Is it a focal point? It does it seem to get a lot more visibility. I'm wondering maybe if it's one of the first real areas, and we've spoken about this before on this podcast, real areas of where you're able to get a bit of traction with AI in medical technology. So maybe it is. I just what what's your take on that?

SPEAKER_02 14:10

Yeah, look, radiology itself as a as a discipline is one of the first things that needed regulation from a safety perspective because it deals with radiation. Right. So, you know, the history of X-rays, which you know, early medical imaging, x X-rays were quite a high dose, they were quite hazardous. So because of that, the regulatory frameworks have a legacy around radiology products. So we've got essential principles, so regulations specifically dealing with radiation safety, which will capture any of those products. And I think that history means that anything radiology is always going to have a special place because even if you're talking about processing of that image, which is what we're going to get to with AI, that image still has to be taken, which means somebody gets a dose of radiation, which means there's always a potential for physical harm. Now, arguably that harm can be minimized or it could be adjacent because the image already existed or these sorts of things. But that's an aspect of radiology that is unique almost. You know, there's nuclear medicine and other laser technology is another one where you know there's inherent harm in the underlying technology. But then when we get to medical imaging as a broader thing, medical imaging is also called out explicitly in the regulations a few times. It's given its own classification rules in a couple of places. Some of the regulations that carve out types of products don't apply for imaging. So it's got this special status as something that clearly the government at least sees as something that needs special protection and oversight. I suspect that that's from the history of how imaging has been used. It's one of the most important diagnostic avenues for a lot of things. It's often quite definitive in the information it provides. I mean, Chi is going to be better placed to comment on this stuff as the medical doctor in the room, but that's I think where a lot of the aspect of radiology comes from in terms of the regulations. In terms of disruption by AI, the two disciplines that I saw getting disrupted first were radiology and pathology, specifically histology. You know, back back when I was a researcher in 2010 or thereabouts, when I just was finishing my doctorate, and these questions started to get asked, and these research studies started to get done to show that actually we could save radiologists a lot of time if we could read these images faster, store them faster, transmit them faster. You know, back then we we had computers and we had the internet, but we were still largely carrying our films around to our doctor ourselves. So there's a lot of potential for efficiency in medical imaging, but there's also a lot of risk if reading of these images misrepresents the underlying pathology. And I think that's where this special status of medical imaging often comes from from a safety perspective or a need to keep regulatory oversight for imaging. At least that's my impression as sort of a medical scientist and engineer who's often working on the technology and the safety side adjacent to the doctors who are actually using the technology in their practice.

SPEAKER_03 17:20

Okay. So that's from a overall med tech perspective. I see you've made some good points around med around radiology being different. What about specifically around AI? Do you see it as being different? Uh higher priority, more important, less important, need to be dealt differently with AI improvement in the radiology space?

SPEAKER_02 17:43

So I think actually AI and uh particularly on the machine learning side is actually more mature in radiology than a lot of other places because it was one of the f earliest places where it was being looked at as a real um having real clinical benefit. Not just the the radiologists having to read images, but also the patients because efficiency goes up, costs go down, and radiologists get to practice medicine instead of reading films and writing reports. And that was some of the feedback that I used to see in some of those early, early studies and and papers around using machine learning models in radiology. AI itself as a technology presents a few challenges to regulators, but more importantly, managing safety and performance. Because really, when we look at the regulations, that's what they're intended to do, is make sure these products are safe and make sure there's a clinical benefit. And the challenge is not really in applying regulations to AI, the challenge is in setting standards for what represents safety and performance in AI. Now, again, if we go back a few years to when most of the AI in in health and medicine was machine learning, which meant the model was trained by AI, but it was then fixed and we had control over it. Most of the challenges there were challenges of evaluation and demonstrating safety, demonstrating that the machine was getting the right answer, consistent with what a doctor would get. As long as the machine is at least as good as a doctor, we're all happy. It can be better and that's okay. It can be the same and that's okay, but we don't want it to be worse, okay, because that affects patient safety. But we can't afford now where the excitement around chatbots has taken over from I don't know how to say it politely, but real AI or useful AI models. These are general use models. They've not been like machine learning model to read a chest x-ray has been built ground up to do just that. A machine learning model that reads ultrasound for I don't know, gynecological reviews has been built from the ground up for just that. Now we've got general models, people are trying to tune them and tweak them to give the same performance as a purpose-built model. And that creates new challenges because these models change themselves out of the control of the people who are building these products. So how do you test and make sure that the model keeps getting better and doesn't get worse? Because regression is a real problem. Also, accessibility greatly increases the risk to society. Again, if you've got a targeted machine learning model that only doctors can buy and only qualified radiologists can use to read their images, and it's all been through clinical trials and proper evaluation, the benefit-risk ratio is quite easy to demonstrate. When you've got a general use model that you know was built to answer general questions on the internet, now being sort of squashed into a clinical box to say, no, no, no, now you're going to read an X-ray and and tell me if it's pneumonia, or you're going to read this ultrasound and tell me if the baby's a boy or a girl or whatever it is. The risk surface is much bigger because so many more people have access. We're not now just talking about trained professionals, we're talking about millions of people. And the control over the model is is much less, as we see through, you know, all the all the general discussion around AI and the hallucinations and all of this. All this stuff is much harder to evaluate and control. And so I think, you know, five five years ago. Six or so, six or seven years ago when I was at the TGA dealing with this problem, we didn't have the chat bot explosion. Okay. So we're able to say, look, here are some good practices to evaluate models, fix them. This is how you can demonstrate to us that they're safe so that so that you can put them on the market. Now, model can change itself on the fly. The model will give you a different answer given the same input on a different day.

SPEAKER_05 21:40

Yeah.

SPEAKER_02 21:41

And these models can build the products themselves. So I can go to the chat bot and say, hey, write me a product that will read X rays. So at least back then you still needed an engineer to make the model. You don't even need that anymore.

SPEAKER_05 21:53

Yeah.

SPEAKER_02 21:54

So it makes the risk surface bigger and bigger.

SPEAKER_03 21:58

Yeah, the variability. Because now it's not just a locked model, accessibility, because now so many more people can get to it, and potentially iterative development without somebody even being involved. There are some really good points. So that would make your life pretty complicated, or maybe your former life really complicated from a regulatory perspective. One thing Chi and I have spoken about in the past is exactly what you touched on, which is the difference between those types of AI technology. And when we're talking about improving radiology practice, we can improve it in many different areas. We could read X-rays, great. We could read MRIs, ultrasounds, etc. But there's also workflow tools, there's decision support tools where you're not reading it, you're just pushing the person in the right direction. There's triage, there's diagnostics, there's a whole different areas where AI can start to support radiology. And they're not all about, you know, a scan. They're about other things. And one of the things you brought up at the start, which I thought was great, was around hey, we want these highly trained, expensive resources, these radiologists, to focus on the stuff where they give the most value. Reading and writing a report, that might not be the highest value part of their job. Whereas the interpretation of something that nobody else can do could be. So it could be that that's great for radiology. I just wanted to take get your take on, you know, it's it on on all those different areas around radiology.

SPEAKER_02 23:27

Yeah, look, it it's a great point, Stephen. We do we do get focused on the medical side, the diagnosis or the therapy, but there's a huge amount that AI can do for healthcare and and medicine that doesn't need to get involved with the clinical decision. That said, some of the examples that you included there, so decision support and triage, they are within the clinical decision workflow. So triage is a is a a clinical practice or a clinical workflow that is used to you know help prioritize resources, basically. That's that's you boil it down in in any part of the world, that's that's what it's about. Decision support is an interesting one because I would argue if you're supporting a decision, you're having input into that decision. And therefore, it's it should be assumed that the clinician making that decision is being influenced by the machine. That machine is providing new information or curating that information or doing something to yes, make the job easier or faster or less prone to error, but that then means that machine has a risk if it fails or that machine introduces a risk if it fails. So if it's if it's informing a radiologist's decision about a clinical case that's in front of them, and there's a failure mode in that machine that causes an error that feeds into that clinical decision, and that clinician doesn't detect it themselves, then you've introduced an error that wouldn't have been there if you didn't use the machine.

SPEAKER_03 24:53

How is that different to a training course, a journal article that is influencing how the difference is the accountability, I think.

SPEAKER_02 25:03

And this is I I'm not a medical professional, but I am a professional. Okay, so you know, one of the things we expect professionals to do is apply their judgment and make a decision. And when they make the wrong decision, they we we hold them accountable for that, okay? And so in medicine, professionals have to be registered to practice. They they hold insurance, it's part of that registration, it's part of their practice, right? And these are sort of controls that are put in place to manage the fact that here's a professional doing a different difficult and challenging job. Mistakes will be made because nobody's perfect, but those mistakes can be managed by that professional and by the systems and support around them. Once you introduce a machine, and it could be any machine, like the X-ray machine itself is an example of a machine that can fail with no control by that professional. So professionals are regulated in in the medical case, they're regulated through the um registration system through APRA and the colleges and the professional associations and all the accreditations. That machine that's influencing that same decision is not regulated by APRA. It's not a member of the AMA or any medical college. So, what do we have to make sure that that machine is not bringing undue risk and that it's making good decisions, and that's what the medical device regulation is for, basically. So they're two separate but interacting, interacting systems.

SPEAKER_01 26:26

You've touched on the interesting point uh about how radiology was the first kind of field for AI to to kind of grow in. What I find interesting is actually the clinicians I find are using AI more than us. With MyBird, there's a lot more installed MyBird, which if people aren't aware, uh they use a live language model to try and create a summary of case notes for patient interactions such that the clinician doesn't have to spend uh spend the next, you know, 10, 15, 30 minutes writing in their case notes. Radiology is moving towards that in the sense that they're using live language models to try and help us um create a radiology report such that um it's a lot faster than starting from scratch for a radiologist. How do you regulators feel about generative AI that has instances where it hallucinates and it can make up incorrect information? It embellishes and sometimes goes off script. What's your thoughts on that?

SPEAKER_02 27:23

I think this is a it's a fundamental risk to using these models in safety environments. So healthcare is a safety environment, it's where safety is is important. It's not unique in that case, but safety is fundamental. And when these models were just transcribing, you know, and there are dozens of these things to be clear, right? You you mentioned one, but there are dozens of what people are now calling clinical scribes or clinical AIs. When they're just transcribing, in my experience, those errors are reasonably easy to catch because you review it and you can see you can see where the errors are and you can correct the case notes and then and then file it, right? And that because I've learned through you know the transcribers in Teams and Zoom and all of this. First of all, my Australian accent is stronger than I thought. And second of all, that it doesn't understand anything about engineering and medicine because when you get into those technical languages, it completely falls apart and misreads it. Now, obviously, when you're making a scribe technology for the purpose of clinical practice, you would put some effort and some training around that model to make sure it can understand that language, to make sure it can understand the context it's working in. But I think, I think, personally, until that issue of hallucination and other similar phenomenon is demonstrated to be either eliminated or in the context of the use case and acceptable risk, we're going to have issues. And that concept of acceptable risk is probably an important point that some of your listeners may not appreciate. But medical device regulations aren't there to force you to eliminate risk. They're there to force you to make sure that the benefit has to outweigh the risk. Yeah. Okay. So there may be clinical context, Qi, where the occasional hallucination of a well-trained model is acceptable. It might be low risk of harm, it might be easy to detect, or some context that makes that okay. But I suspect the general use models, so your chat GTPs, for example, which haven't been specifically designed around clinical context, are going to have much, much bigger problems. And this is, I think, a challenge that regulators are struggling with because they do have these purpose-made products that are targeting healthcare and medicine. And while those companies are addressing those risks and doing that research to improve that technology, they're now competing against the general models, which are starting, you know, the um influential CEOs of those companies are telling us those models can do anything and and do a better than a person. Now, hopefully your typical medical professional doesn't buy that. Okay. But your typical health consumer might. So there's not just a risk of the the purposely designed clinical strike having having some hallucinations or issues. Those risks can be managed because they've got oversight by a manufacturer who's in turn being looked at by a regulator to make sure they're doing their job. But if the patient says, well, instead of going to the doctor, I'm going to ask ChatGTP, I'm going to just record my own case notes and ChatGTP will tell me what I need to do. It's probably a bigger health risk to consumer health than the AI scribes, which are in a in a managed environment and in a professional environment as well.

SPEAKER_03 30:45

From an outside perspective, it's very interesting what you've just said because you're right about the no-risk situation. It can't be eliminated. If it was a no-risk environment, you'd never give a vaccination, you'd never do any surgery, you'd never take an X-ray, you'd never prescribe a drug, because all of them are open to some type of risk. And that that's accepted that there will be some. So yeah, that's important to know. I've got a question I wanted to ask around regulatory view of radiology again, and specifically around language. Intended use. The term intended use and the language terms like assistive triage and decision support are pretty important in the whole regulatory framework. Do you want to explain some of them for us?

SPEAKER_02 31:30

Yeah, intended use, everything in the regulatory framework hangs from intended use. Whether your product is a medical device and subject to regulation or not is based on the intended use. And this is how the product is used as intended by its manufacturer, or in the case of software, it would be its developer. This is important because the manufacturer of the product gets to decide what the product is for. An X-ray that's used to see if I've got pneumonia, okay? Or, you know, the intended use will be something like for the purposes of medical imaging to screen for pathology or inform diagnostic decisions or some sort of medical jargon. But the point is it's an X-ray to see inside my body to work out if I'm sick or injured. The manufacturer intends that product to be used medically. So its intended use is medical, it's a medical device. The same X-ray technology from the same company could be installed in an airport to make sure I'm not taking firearms onto the hairplane. That X-ray, even though it's the same technology, is not a medical device because it's not intended to look inside a person. Now, hypothetically, somebody could put somebody in the airport scanner and get some sort of image. Okay? That doesn't make it medical because the manufacturer does not intend it to be used that way. Okay. The term intended is is what's key here. A non-radiology example is a heart rate monitor. Hospitals have heart rate monitors. They are clinical, they are medical devices, they are for use in a hospital to detect heart pathology. Whereas this one This one, this one is not. This is for human performance and sport and fitness. Me, as a researcher or a doctor, putting this on my patient does not make it a medical device because the manufacturer of this product doesn't want it used that way.

SPEAKER_03 33:21

Yeah. Well they don't want to that use that way, but surely if you were to say we don't want to use it. But they knew that it was going to be used some other way. The same with ChatGBT and stuff.

SPEAKER_02 33:38

Now, there is a requirement there if if your product is a medical device, you you have to help manage foreseeable misuse, off-label use. But if your product's not a medical device, it's not subject to that regulation. So if you're a fitness device and your intended purpose is to just measure heart rate in human beings for any purpose. It's a measurement device, it's a general measurement device. You may know that people are using it in a hospital, provided you're not actively marketing and supplying it for that, it's not a medical device. And this is a challenge, it's a real challenge, especially with wearables. Okay, it was a lot easier before the wearables boom because the obvious application for wearables is to take clinical measurement out of the hospital and put it in the home and let people measure themselves in a comfortable place. Um and there are plenty of medical wearables that are doing exactly that. But I imagine if you're a GP and you know that this one that your patient can get from JB Hi-Fi or $100 is good enough, you would reasonably say, look, go and get this one. It's good enough for what we need. You don't need to spend a grand on the medical one. This one's good enough for what I need. And that's that's professional judgment. That is a professional doing their job. But it does create a difficult blurry line for government because are these companies saying they're a fitness device but actually quietly marketing to medical people for medical use? Or are they genuinely just making a general purpose device and selling it to anyone who wants it?

SPEAKER_03 35:11

From an external perspective, if you've got your finger on the pulse to a certain extent, I would imagine that you can work that out pretty quickly. A lot of the stuff. I mean, if you're deliberately hiding it and there's a legal argument behind it, but eventually people are gonna know, and especially in this space, I would have thought that's gonna become and the internet means you can't hide a whole lot anymore anyway.

SPEAKER_02 35:32

Somebody's gonna write on a chat chat room. Yeah, that's right. And again, this is a a place where the regulation is well drafted. The section of the medical device legislation that defines what a medical device is actually clearly says the intended use is assessed from the instructions for use from your advertising, etc. Basically from what you tell your customers, not what you tell the government.

SPEAKER_05 35:58

Yeah, good.

SPEAKER_02 35:59

So the government has the tools there to deal with that, and when they investigate these sorts of things, they go out and look at what what information the customers are getting, not what the company is telling the government. But it it's getting more and more challenging.

SPEAKER_03 36:12

But that that's common sense to me, and that sounds that sounds great that there's the right regulation in place to be able to support what sounds like should be common sense. I've got a little segue here because I read an article a while ago, uh, I think it was in January this year, about a man who turned up in hospital having followed Chat GBT advice and hallucinating and thinking his I think it was in Britain, his neighbour was trying to kill him. And I'm just reading the notes while I've got here ChatGBT suggested he take some particular type of additive. I think some sodium-based um he had gone what it far too much, it'd been taking it for far too long. And they realized this only after a period of him being in hospital. But it drives back to exactly what you were talking about before. It's not regulated regulated as a medical device, a diagnostic tool. So there's no controls and there's no risk reporting, no post-market surveillance, no requirement for publishing data, etc. That was a question I was gonna ask you a little bit later, Lee. So you've jumped ahead and finished it early, but it's obviously really happening. It's not just all baloney. There are people taking this advice, and a lot of the advice is good. I don't want to have to line up for a doctor. And I'm not saying it's good, by the way. Let me all of it's good. Let me go back. I there are some things where you don't want to have to wait two days to be able to get into GP and it's really basic. But there's some stuff where if you really do follow it without some smart, smarts, like you can get yourself in a lot of trouble. And if we don't be realistic about that regulation, people are going to go around it because as even if you say, hey, don't, they're still going to. Because if you have a waiting line or or it costs a lot of money and you don't have that much and you don't have the time, you're just going to take something which is available. So we need to make realistic regulation and control without it going over the top and and pushing people towards taking some advice going around it.

SPEAKER_02 38:04

Look, for me, the chatbot boom is releasing health, but in in other areas as well, it's the next Google, right? So before the chatbot boom, people were doing this with Google. They're telling Google their symptoms and they find an article or a recommendation or a blog post by an influencer or something and do similar things. But I think because it was Google and because there were articles and it was a little bit easier to separate what was real and what was not. Even for people without the sort of technical or scientific knowledge, it was easier than it is. With a chatbot that is laying out exactly what you want to hear, it sounds professional, it's been trained to behave and sound like an expert. It is much, much, much harder. I am skeptical about it, and I know something about how these models work, and it's getting harder for me to know without doing my own research. And because I don't trust the thing, because I've seen so much junk come out of it, it gets to a point where I'm, well, okay, this is a it's a useful search engine, but I'm not gonna use it for anything that requires empirical evidence behind it because I I can't see that. But that's me as a science and engineering professional who knows about empirical evidence and measurement and this sort of stuff. If you don't have that sort of education, it's gonna be a real challenge for people to be able to tell what's real and what isn't. And I think unfortunately, I think regulation is the answer because how else are we gonna draw a line around what what these products can and can't do in terms of making them safe in these at in these in these use cases?

SPEAKER_01 39:44

Well, I I agree with you, Lee. I think uh when it comes out Chat GPT or Google or or Claude, whatever you use, it sounds like a doctor. And the question is, if it sounds like a doctor, do I still need to like Stephen Goes, you know, wait two days, see my GP, maybe there's a gap involved.

SPEAKER_02 40:07

Yeah, I know, I know. And this is the thing though, if we had a purpose-built model and there are people working on these, okay, there are these products that exist in the pipeline, that would be basically your Dr. Google. But again, because those products are health technology and they would be most likely medical devices, they're subject to risk management and safety standards. And the makers of those products will pull guardrails around them that mean, yes, if you're asking about some minor thing, you know, is this a pimple or or a paper cut sort of thing? Very low risk stuff. Yeah, you don't need to see your GP for that. And I suspect some GPs would be happy that you didn't. But those same products would be designed that if there are more serious questions, the product would say, Look, I'm not answering this one, you need to go and see your GP.

SPEAKER_03 40:57

Which is exactly how the medical system works at the moment. Because if you see a GP and he says, actually, you need to see a specialist because I don't know how to handle this, that's the same triage that happens in the current model. So I I mean, I would see a future where through necessity of there being other options, there needs to be something similar to what you've just said, Lee, because if it's not provided, it will be somebody else will provide it. You know, if it's if it's there needs to be some basic level triage of health medical engagement.

SPEAKER_02 41:28

So I'm gonna throw this problem at you, which is this is where the safety regulations might be working against us. Okay, if if I want to make that product a good, proper, evidence-based product that genuinely makes um provides clinical advice and escalates where appropriate, there's a lot of work in that. That's hugely expensive, and it's a regulated product, which means there's even more work, not just to prove that it works, but to prove to the government and my competitor doesn't do any of that because it's Chat GTP, it's freely available. So I'm trying to make the good product that's actually gonna help and benefit society, and I have to charge higher prices because I have to pay expensive people to help me build it. I have to pay government fees to get it approved. And to to be clear, in the context of building a product like this, the government fees are negligible, right? But all of the work to design and build and prove the product is work and make it safe is a real cost. But ChatGTP is already there. So why why would a consumer pay, I don't know, let's imagine it's a Netflix subscription, so like 30 bucks a month or something, to have an AI doctor when they can ask ChatGTP. Now the three of us would probably pay the 30 bucks. The regulations over the clinical products being I think quite effective when they're applied properly, but the wider industry being allowed to undercut the medical products is actually forcing the good products out of the market because they're no longer commercially viable. A $30 chatbot can't compete with chat GDP, which is free.

SPEAKER_01 43:03

And I guess that's the thing, like unless you sample both of them, will you know that one is superior to the other? Someone who only samples the free product will always think that's gold. It's took it's sounding like a doctor, it's giving me advice like a doctor. It's probably just as good as a $30 product. Yeah.

SPEAKER_02 43:20

And the products are designed to to sound that way to uh give the impression of competence. Yeah, yeah. Yeah.

SPEAKER_01 43:27

And I guess I guess like we said, um one of the issues is you're trying to sell to a very narrow market compared to Chat GPT who sells to the whole population and the medical market's only a small subsets. So the return of better.

SPEAKER_02 43:44

Yeah, I I think in that case the medical market I mean the medical market will pay a premium because a product targeting professionals is for professional use, it is for a higher level of of use case. I mean it's it's the difference between Between a $100 tennis racket and a $2,000 tennis racket, right? One of them is professional use. And yeah, some amateurs might buy it, but it's not really for them. But when we come back to consumer products, a consumer product that costs money versus one that doesn't. Again, you know, look at your browser. How many people use Chrome, even though we know it's harvesting all of our data and giving it to Google? When you could pay 10 or 20 bucks to get a browser that doesn't do that. Which one are you using, Chi? Yeah. Are you the edge? I want to know. You're a doctor. I want to know which tennis racket you're using, Lee. I don't play tennis.

unknown 44:42

Okay.

SPEAKER_01 44:43

Look, Lee, as as you mentioned, look, there's lots of products, and as a radiologist, we are presented with lots and lots of options. Look, it's our assumption that the evidence has been received by TGA and it's been deemed safe. The bar of evidence seems to be always changing. But to summarize, can you please reassure us that it is still quite high and it caters to all different combinations of people and equipment?

SPEAKER_02 45:08

Yeah, look, the way the Australian regulation is written and you know the other countries we trade with, it's written to reference what's called the state of the art. And it's important in this case, the state of the art is not the McLaren F1. The state of the art is the Toyota Corolla, okay, which has had way more research and development than the McLaren F1. Okay. It's sold way more units. It is a robust, safe, reliable, effective product. You try to bring a car to market that doesn't meet that standard, you're not going to compete. The medical device regulation uses the same concept. It says this is a state of the art, this is the minimum standard. If you hit that, you're okay. You can exceed it, but you can't come in under it. So that state of the art exists for every medical product. I mean it exists for every product. But in this context, in radiology, there is a state of the art for how radiology is practiced. There is an acceptable risk. There is an acceptable error rate or a acceptable precision and accuracy is probably a better way to put it. So the system itself, the rules, I think, and the standards that support them are generally fit for purpose. And there are some cases where the standards are lagging a little bit, and that's just the reality of standards needing to update and do these sorts of things. But generally speaking, I do think the system is well designed and well built. Where it falls over is the processes and administration of that system. And this is, you know, the easiest example here is an under-resourced regulator. Which government regulator is not under-resourced? Like they all are. There's always a balance of how much money should government spend here. But if those rules are not if the resources don't exist to enforce the rules or they're enforced inconsistently or people don't understand them, which means they're getting them wrong, um that can be just as bad as having the wrong system. Probably worse. And I think that's probably the situation where we more so than the system being not well designed and well built. I think it is potentially a lack of resources or lack of oversight on the government side that's creating some of these challenges. Now that TGA approval is, I think, hugely valuable to clinicians, certainly all the ones I've talked to about this, because they see you know somebody's checked this product. It it meets that minimum standard. So I I do think clinicians value that. But when there's no products with that stamp, they have to make their own decisions. And I think that's a challenge as well. I've spoken to some GPs who struggle with that. There is no medical device product for this, because this product is not a medical device, it doesn't have to be approved, but it is health tech, and I'm using it with my patients. How do I, who are medically trained, but I'm not a software engineer, I'm not a researcher, I don't know how to do a clinical trial, I don't know how to read this stuff and understand it. How do I make a decision which product to use? I suspect radiology has less of that problem just because that special status in the regulation that I talked about before means that products dealing with medical images tend to be tend to have some oversight from the government. Even if it's not the TGA, it might be the radiation safety people or um the privacy um people, which is another important bit of legislation in healthcare, is is the Privacy Act. So yeah, I think it's a challenge, but I I think the system itself is is well designed. It's the execution and administration of that system that tends to let us down at the moment.

SPEAKER_03 48:49

So, Lee, one area I'm really interested in is this under area of accountability. So, radiologists, if you're getting support from some other areas, from machine learning, from decision support tools, from another system which is giving you some advice about potential uh risks that were identified in an X-ray or an MRI or some kind of scan, or even it's just helping you write some reports. Accountability, you've got that person who has to ultimately sign off on it, a radiologist in this instance. How does that change determination of from a regulatory perspective? Because they might remain the ultimate safety net, even if they're supported by AI. Yeah.

SPEAKER_02 49:32

So look, there's a medical legal question there that I can't contribute to, right, which is the legal accountability. But the way the medical device regulation works in that context, basically, when there's a clinician there, it's assumed that the decision is the clinicians. And you will find most manufacturers work to that standard. And in fact, all these startups bringing digital health products into the market even try to get out of the regulation by saying, no, no, no, it's okay because a clinician makes the real decision. So yeah, but like legally speaking, that that's the clinician's job. Like a diagnosis has to be made by somebody who's registered to do that. We don't accept the diagnosis of a machine at the moment. Consumer-facing products are more challenging because there is no clinician in the loop. So if I buy an app off the app store and take a picture of my freckle and my phone says I've got skin cancer, like, is that a diagnosis? I think the customer will take it as one, but it can't be executed in the health system until it's in the health system because they can't access treatment or anything like that and until a doctor's looked at it and said, Yes, I agree.

SPEAKER_03 50:43

That's a really good point. Because at the moment you could have your brother or uncle or auntie or somebody could say, Hey, that is cancer. A police going to turn up at their door and say, Hey, you gave people a bad device? Obviously not. So it's known to be outside of the health system. It's only when you enter and access the health system that it needs to be regulated. Is that effectively the well what I'm saying?

SPEAKER_02 51:05

So a phone app, to be clear, a phone app that screens for skin cancer is definitely a medical device. Okay. What I'm saying is when that medical device is customer facing, so patient facing, the patient is getting the information, not a doctor. So even though that medical device has said you have skin cancer, that person can't access treatment without entering the healthcare system. So they still have to take that result somewhere and say, hey, my phone says I've got cancer. Could you please cut it out or whatever?

SPEAKER_03 51:36

Yeah. And the doctor will say, hey, slow down, champ. We need to do a little bit more. Yeah, yeah.

SPEAKER_02 51:41

Let's say let's confirm it's cancer before I get get out a knife, right? So I think that's a reality here that inherently, again, forces medical verification to happen. And that's because the health system doesn't accept diagnostic decision from anyone who isn't registered as a clinician, a diagnostic clinician. So it's a complex question. And this this is still ignoring the fact of, you know, from a legal perspective, who would be viable. But the medical device regulations, as we said before, force the manufacturer to make sure the product is giving safe and effective information. So that hypothetical example of a skin cancer app, because it's a medical device and it's subject to regulation, the regulations force a minimum level of performance, that state of the art that says, no, no, no, doctors get some percentage of skin cancer diagnoses correct. Your app needs to be at least that good. Whether or not it's a diagnosis or not is probably a question for a lawyer rather than an engineer.

SPEAKER_03 52:39

So this one I've been really interested in talking about, Lee. I've got a big question for you around the social license for radiology AI. What does social license mean in the context of radiology and medical imaging from your perspective?

SPEAKER_02 52:50

From my perspective, I think you need a license from two groups. You need license from patients, which is us. Like society has to agree that this is okay. But we also need license from the clinicians. Like they need to agree that this is a useful tool that's that's helping them, making them be more valuable, helping them do the part of the job that they like doing. One of the things that drives me nuts about AI is I want AI that's going to mow the lawn and empty the dishwasher. I don't want AI that's taking away the things that I like doing. And yet this is what AI companies are trying to sell me. And I suspect, you know, people who practice medicine are probably partly there. You know, this is the part of my job I like doing. I want to do more of this. So if AI takes away this bit I don't want, great. This bit I like doing this. I don't so I think we'll probably actually see in time that different professionals use different AI tools and different parts of their practice because we all enjoy different parts of our job.

SPEAKER_03 53:44

That's a good point. There is a lot of worry in society around going to take my job, it's gonna replace what I'm doing. But I also see a lot of opportunity for supplementation, and for people taking jobs, like you said, mowing the lawn. The day that an AI-powered lawn mower comes out that you can properly trust, that company's making a lot of money. Like they're just running off out of the shops. So I st uh they I think they're still coming. I think there's still lots of stuff happening there. And potentially there's too much worry about oh, it's gonna take other stuff. But the the you know, industrial revolution, I'm glad we're not breaking rocks with hammers anymore. I'd much rather do it with a bulldozer and dynamite. There's gonna be a lot of opportunities that come from it and that society would change. If I go back to your answer, that was a great answer from my perspective. You're not just looking for a general amorphous social license, two particular parts within that, and maybe a third one, which is the general zeitgeist sort of discussion and political influence, but two particular ones. Does it help the patients? Can they see real value? Does it help help the person who is in that core responsible role right now in society, which is the clinicians? And both of those guys need to be on board to be able to push forward a that that advancement, that technology.

SPEAKER_02 54:58

Some patients like going to see their doctor, they like talking to people as well. Yeah. So those patients will naturally probably resist talking to a computer instead. Yeah.

SPEAKER_03 55:08

But internet banking, which I hate the idea of walking into a bank and having to do anything with any that I desperately want to do online. But there's a huge amount of people who that is how that they are used to it, they feel much more comfortable with it, uh horses for courses, right? Okay, another question around social license, over promising and AI failures, how does that affect confidence in radiology practices specifically? I'm interested in, but also maybe they can put a whole whole general lens on it. Do you have an opinion on that?

SPEAKER_02 55:40

Again, if we're talking specifically in radiology where it's likely that these products are going to be regulated, the regulation is there to prevent overpromising, partly. Like that's one of its core rules. But marketers exist to push the limits of regulation, like that is their job to find new ways to say things, to invent um great campaigns and effectively raise awareness. So, you know, marketing is is a vital part of commercial practice of business. In the context of radiology, if we have too many failures and trust is lost, one of those groups that we just talked about, the radiologists or the patients, rescind social license. I think that's the reality. In medicine, I think when you work in health, you sort of learn how much people um rely on or or value certain parts of the health process. So yeah, you can say, look, everybody's worried about their health. Some people just aren't. Okay. But what they do want is that when they're told something's gonna work, it will. You've told me this AI will be just as good as my GP or my radiologist in this case.

SPEAKER_05 56:52

Yeah.

SPEAKER_02 56:52

And it wasn't. It didn't talk to me. I hate it. So I think there's an aspect there that there's trust in the data and the decisions it makes. And obviously, if those failures come up, hopefully the regulator will step in first. But even if they don't, I think clinicians will lose trust quite quickly. Hopefully that means social license is rescinded and the companies making these products um make better products. Yeah.

SPEAKER_03 57:17

When you start looking at this area, it becomes obvious that there are a lot of checks and balances that happen here that aren't necessarily regulations written on a piece of paper. They are actually society has built in their own checks and balances. We all do in terms of how we make our own decisions and we do as a as a larger group. And yeah, that social pressure, that marketing pressure, the the regulations, they all come back to generally give people and society to say, hey, we're not we're not comfortable with this, we're not sure about this, I'm not you know, something needs to be done. And then generally it does get done. Hopefully sooner rather than later in the in the face of bad stuff. And hopefully we don't get too many.

SPEAKER_02 57:55

The market should help with some of that, right? Because if you've got a genuinely better product, eventually people will get sick of the other one and use yours. But there are barriers to overcome there, like cost and price and and these sorts of things, which is a discussion we had before.

SPEAKER_05 58:12

Yeah.

SPEAKER_02 58:12

Um but there's a lot of stuff built in besides just the TGA. Yes. That's that's quite important. Yeah.

SPEAKER_03 58:18

Just like with and not that we want to get too philosophical here, but just like with wider governments, the checks and balances are for governments and for for how you run a society, are multiple, just like they are in this area. One point you just went on to, which I wanted to dig in was actually my next question, was about transparency. So in business, there's an old saying, what you measure is what you focus on. And because the measurement shows care. So you need to measure the stuff that really matters. Because if you're measuring stuff that doesn't really matter, people will think that you think that it matters. They will think that that's the most important stuff. And that's not directly related, but related to this last point. What people see about a particular product, what they see is happening, whether they hear about it in the news or whether they, you know, it where it gets attention, ends up being what you focus on. And visibility in some areas could be bad because if it's visibility of the wrong thing, then you're either promoting something too much or pulling back on it something too much. So when it comes to regulation, what you measure and what you focus on ends up influencing whether it is adopted and whether it ends up being a social good or a social bad. I'm just wondering how is that that lever of transparency pulled within regulatory framework to ensure that the stuff that matters is getting looked at and the stuff that matters that doesn't matter isn't getting looked at.

SPEAKER_02 59:40

Look, that's one of my frustrations with with the execution of the regulatory framework is transparency is quite low. Now, some of that is is for good reason. For example, when the government is your regulator, it has powers to access a lot of commercial inconfidence information that, you know, is your commercial edge, it's your trade secrets, it's your IP, it's it's what makes your product good. The government should never be, you know, leaking or or releasing that information to the public. So the government does have in order to protect the market, there are certain things they can't be transparent about with good reason. Also, um, you know, patients' personal information that's involved in investigations and all of this sort of thing. But some of the outcomes around penalties of how policy is written, who's consulted on policy, how standards are developed and written, who has input there, creates some opacity that I think leads to a lot of the misunderstanding of how the system works. I mean, if we look at standards, when an Australian standard is published these days, there's a list of the organizations who contributed to that standard written on the front of the document that doesn't happen for some of the international standards that we use to regulate our industry. Now, I because I serve on some of those committees or or have done in the past, I know who sits on them, I know who they represent. You know, the committees I sit on now, I know who's there and I know who they represent. And most of the time it's a good even representation from the community, from government, from industry. But some of those committees, just over time and just through natural attrition, people coming and going and leaving, can become dominated by industry or by government. And then they become biased, and the standards shift towards one stakeholder group. So I think more transparency there would help people understand who's making the rules and probably make those people there more accountable.

SPEAKER_03 1:01:37

What about transparency of like I brought up before around failures or overpromising and that type of stuff? Because again, we talked about risk. Nothing's without risk. If you take immunisation, there's some stuff's gone wrong. Some people have been hurt by it. But so many people have been saved by it and have and life. So again, where you focus ends up being where and you know, and that's where we get in the sort of the social circles we've got at the moment, you can become an echo chamber. What gets published and what gets transparent from issues, that'd be hard. Like, because you want to be transparent, but sometimes you could be blowing a horn too hard.

SPEAKER_02 1:02:17

Yeah, and I think so. In the in the private sector, in the manufacturers making the products, you know, often uh trust is won or lost on how much you tell your customers and how transparent you are when things go wrong. You know, and that's an aspect of business that each company has to decide how transparent they are with their customers with broader society. I think in the healthcare industry where trust is so important, I think some businesses strike that balance quite well. Some businesses are, I don't know, too too scared or whatever reason they decide not to admit to some of the mistakes that everybody can see happen. And that erodes trust, probably more so in some cases than admitting mistakes. Now, obviously, if you're if you're admitting a mistake every day of the week, that's gonna erode trust pretty quickly. But when everybody can see there was a failure here and you're trying to pretend everything's okay, I think healthcare people, particularly clinicians, pretty quickly catch up with that. On the government side, there are challenges for them in in, as I said, managing private information and these sorts of things. But government has a broader role. So, you know, the TGA is part of the Australian government. We say m may say the TGA's job is to um make sure we have safe and effective products and to try to clear out the ones that are not. And you know, when they find somebody they they publish that. When they're reviewing parts of the industry or particular types of products, they publish that. But most people don't realize that's happening, first of all. And those are the really big cases where something big went wrong and they've either decided to find somebody or take them to court. But there's very little information about the day-to-day operations of the regulator, what their job is, you know, how how often are products going wrong. Like they don't necessarily need to tell us, you know, in your case, vaccines, um, sorry, in your example, don't need to tell us which vaccines went wrong to tell us that, okay, overall we found this many problems in vaccines over the last five years, and look, you can see the evidence vaccines are safe. Um, sun cream is an interesting one. Sun cream is so important to Australians, and it feels like every two years there's a new exposure that the TGA wasn't managing sun cream properly, and another stakeholder group, last year it was choice, those that actually all these sun creams don't meet the standard. Yeah. The TGA is now on the back foot, and you know, it's happened like two or three times over the last several years. Clearly, the TGA isn't effectively regulating sun cream because these things keep happening. Now, I say clearly because that's my impression. Now, it might be that actually they are, because actually what choice exposed is is a tiny thing, and last speaking, the industry is fine. This was just a little problem patch, but we don't know that because we have we don't have any data except what choice released.

SPEAKER_03 1:05:14

So visibility in that case could could have got ahead of that choice.

SPEAKER_02 1:05:18

I think so, yeah. Yeah. And it is challenging because although that's the TGA's job, the broader Australian government's job is also to manage community safety, manage stop there being a panic. So it's not necessarily the TGA that always gets to decide which information gets released, either it may be the minister or part of cabinet or a different organization that might have jurisdiction over that particular type of information. So it's a complex problem, but I think there are certainly areas where more transparency would help both industry and the community understand how these AI products in radiology and other disciplines of medicine are being managed. I understand it because I work in the industry and it's my job to help these AI companies do it. But I suspect many of your listeners just they know the TJ's there and they know they used to regulate this stuff. But I suspect most of them don't know how the system operates or how it's supposed to operate.

SPEAKER_03 1:06:19

In in this, our listeners probably are a much higher level of interest in this area. But from a general public perspective, they don't want to know. They just would like to know that it's being well regulated. Just like I don't know all the fiduciary information about how banks are I mean, I know a few of them because I'm interested in that area, but banks are being regulated. And I don't understand how physiotherapists are regulated and how they're checked and or the training regime that goes on with them and you know and and chemists and so from a certain perspective we just want to know that it works, but within the circles of people that matter, it sounds like transparency come is important, and and then they we take the leadership from the people who are in in in the know.

SPEAKER_02 1:07:00

And if they can and that that is important. I th I think you you're absolutely correct there. Not everybody needs to or wants to know, but many people will look to the experts within their sort of circles, right? So they'll look to their doctor, and oh, the doctor said this is okay, the doctor said the TGA is all good, great. I I trust my doctor. Which now means does the doctor need a deeper understanding? Do they need a bit more transparency to advise and make a decision? He asked a question related to this before about you know, radiologists have got all these products coming towards them, how do they decide on the value of that TGA stamp to the clinician? But if you're trying to bring a product to market, and this is the group where transparency really matters, if you're a medical device manufacturer and you cannot work out how the regulations work on your own, how can you bring a safe and reliable product to market? Like if it's not transparent to the people making the products, I think that's the biggest, the biggest problem. And I think at the moment that is a genuine issue. I mean, it's why companies like ours exist to help people through that. But I'm also of the view that you know we should be here to deal with the really complex, challenging cases, not the basic stuff. Industry should be able to deal with the basic stuff itself, and I think if if the government was more transparent about how the system worked, more innovators and small businesses would be able to do it themselves, and that would bring costs down and again help healthcare.

SPEAKER_03 1:08:30

So, Lee, I'm really interested, and a lot of our listeners will be interested in the radiology practice. So, specifically around that, what they should be doing better around AI to get the best and avoid the worst. Maybe you have some insights here. I understand this isn't directly your area of expertise, but I'll throw some questions at you. Is relying on regulatory approval enough? As I'm running a radiology practice, I'm only using stuff that's been approved by TGA. How much more do I need to be worried? How much more do I need to look at what I'm doing from an AI perspective?

SPEAKER_02 1:09:05

Yeah, look, that that's a hard question when I'm not a radiologist. I would absolutely, if it's got the TGA stamp on it, I think that is important. Like I think you can rely on the system that we have, especially because radiology tends to get a bit more attention. Products that are not regulated are a bit more of a challenge. And unfortunately, the mechanism at the moment is is for the clinician to do their own research. The bigger risk is probably the products that are regulated but haven't been through the process. So the non-compliant products, which are there are a lot of them around at the moment. And so radiologists making sure they know if this type of product should be registered and if it should be, checking that the one they're using is is probably the biggest thing.

SPEAKER_03 1:09:52

Great, great insight. Do you also have some insight on governance they could consider implementing around AI? There's all these technology out there. Yes, they've looked at TGA approvals, but what other governance controls maybe would you be looking at if you were running one of these practices?

SPEAKER_02 1:10:08

Yeah, um, so there are actually some good standards now at the international, and they're going to be adopted at this drain level for how to how to govern AI within your business, how to manage it. The size of your organization is probably the biggest challenge here. If you're a small medical clinic, there are genuinely some good practices you can put in place to evaluate and manage your AI tools. Um, if you're a larger organization, you should probably be looking at certifying against some of the standards. So management systems for AI are a real thing now. We have standards for them and we have conformity assessment standards. So if you're a large organization, I'd say consider that. If you're a small clinic, you can apply many of those same principles without going through certification. So that there is guidance and and there there are approaches out there if you go look.

SPEAKER_03 1:11:00

What would you push for our listeners? Is there some area you would push them to if they wanted to start down that path to look first? Is there a particular standard?

SPEAKER_02 1:11:08

Yeah, so if you're a large organization, ISO 40201 is the management systems for AI. I mean, if you're a small organization, you can still read the standard and uh work on the principles. If you're a small organization, you can really just come back to your basic principles for evaluating vendors and their products. Know what your user requirements are, know what you need out of this product. Write it down. When a vendor comes in and tries to sell your product or when you go to market looking, check it against your own requirements. That is the simplest form of sort of software assessment. And when you get into more complex systems for AI, it's just an extension of that basic approach.

SPEAKER_05 1:11:48

Yeah.

SPEAKER_02 1:11:49

But you know, any any good list of requirements, you can test a product against yourself and make your own decisions. And writing it down is a good exercise because you've you've got in front of you what you're concerned about and you don't get distracted by the vendors' marketing.

SPEAKER_03 1:12:03

I've got one more question for radiologists' running practices. Escalation pathways. If you see an issue with some product you're working with and you think somebody should be told, what should they do?

SPEAKER_02 1:12:15

Any problem with a medical device or something you think might be a medical device, you can report to the TGA. And in some cases, that is that is the best approach. You should also report to the vendor or the manufacturer. Because if if they're a good and compliant, especially if it's a medical device, they have obligations for dealing with your complaint or your issue. But even if it's not a medical device, a good vendor will still take that complaint seriously, and how they deal with your complaint might help you decide whether you want to keep working with that product or not. True. Um so I I always encourage report back to the manufacturer. And if it's a medical device and you don't have any luck with the manufacturer, if they don't give you a satisfactory answer, then you can escalate and report to the TGA.

SPEAKER_03 1:13:00

Now looking ahead, I'd love to get your thoughts on a few areas. How do you think radiology AI is likely to evolve over the next decade?

SPEAKER_02 1:13:09

I hope that it comes back to proper targeted models designed specifically for radiology and that we don't start using chatbots for everything. I think before we get there, we're going to go through a phase of chatbots for everything. Yeah. Unfortunately. But look, what should evolve, not just in radiology, but in in you know adjacent clinical practice, is that the standards in the state of the art will get better and better. We will get better at evaluating these products, at understanding what's important, and the standards will continue to improve. And that should mean the products continue to improve as well and become more targeted and the risks we develop better methods for managing the risks. Okay.

SPEAKER_03 1:13:51

Next one. If you were a radiology practice owner, what would you be asking AI vendors?

SPEAKER_02 1:13:56

Is this product a medical device? Is it supposed to be on the ARTG? And if the answer to either of those is yes, is it? That is a big problem at the moment. So and and if they say no, and you're looking at it and you're thinking, yeah, it feels like a medical device, maybe ask some questions of some other people. Ask, you know, a specialist in the area or the TJ if you have.

SPEAKER_03 1:14:21

That's good advice because I would imagine I would have just assumed if somebody's knocking on my door, they've only got the bravado to do that if they're actually regulated and approved.

SPEAKER_02 1:14:30

Yeah, unfortunately it's not the case at the moment. There are a lot of companies who are trying to ignore the question or fly under the radar. If you put a bit of pressure on them, you you can probably find out whether they're one of those or not. Um I probably want to see the clinical evaluation report. I want to know that it's been evaluated. Even if it's not a medical device, I want to know that it's been evaluated in my use case. This is me, somebody who can understand that and read that, and I think most radiologists would be in a position to do that as well. Maybe the practice manager can't if if I'm a non-clinical practice manager, but I can ask one of my radiologists to do it for me. That's a big one because that's your clinical safety. That's evaluation of this product in your use case. Has it actually been built for your use case? Has it been tested in your use case? And I think as a clinician, that will help you understand whether this is a product you want to use or not. And then we're getting into some more adjacent areas related to use of software. So things like data privacy, which is very important in healthcare, data security and how that's managed, perhaps integration with your other clinical systems. Does it put anything else in your business at risk? I think those are the broad ones that I would be looking at no matter what my requirements were for the product. And then, as I said in one of your earlier questions, make sure you've got a short list of requirements of what you expect this product to do and test the product yourself against those requirements. Don't let the vendor say, hey, no, no, look, I promise you it's okay. Sit with the product yourself and see if it does what you expect.

SPEAKER_03 1:16:04

Which is a great advice for any product, regardless of what you buy.

SPEAKER_02 1:16:07

Exactly. And software vendors these days are so much into no, no, it's okay. Like sign up for the subscription or whatever, we'll promise you it works. No, no, no. Like do a proper evaluation before you commit. Um, because a lot of these products will try to lock you in. And so you think, oh yeah, we'll we'll buy it and then we'll we'll get rid of it if we don't like it. But if if they've locked you in functionally, you you can't get out. So you you want to evaluate before you commit any software for your business.

SPEAKER_03 1:16:31

I agree. The investment's not just the money you're paying, the investment is the time that you're spending. Still on radiology practices. Do you have any advice about evaluating risk sensibly or how to think about risk from an AI perspective? We've kind of touched on this from a lot of different areas, a lot of different angles. But if we want to summarize some of that, you know, we have talked about there is it's impossible to have no risk. We've also talked about minimizing that risk from using TGA in your own internal practices and approaches and you know evaluating AI vendors. Do you have any any final points you wanted to add about that? Because I think that's probably one of the biggest questions that radiology practice owners and managers and and operators who are listening will be listen thinking about. What is my approach to this? How do I take care of the risk?

SPEAKER_02 1:17:19

Look, clinicians are generally very good at risk management. It is their day-to-day job, but they they tend to do that in the context of the patient. What I encourage any clinician to do, trust that you you know how to manage risk, you do it with your patients all the time, and sort of extend that principle into the aspects that can affect your patient. So if you're assessing an AI product and you're trying to understand the risk, there are specific risk management methodologies for doing that, you know, in the engineering world. You could go and learn those if you want, but I don't think that's necessary. I think clinicians understand risk and how to manage it. You just need to adapt your thinking. So, how would this affect my patient? What harm could come to my patient through this product? Don't let the vendor tell you there's zero risk. You're a clinician, you know there's no such thing as zero risk. Let them tell you, but just ignore them and think yourself. How could this product go wrong? And if that happens, what is the potential harm to my patient, to my business, to my practice? Once you understand what that harm could be, I think any clinician will then be able to think about that risk in a in a professional way, they the way they do with their patients. But often understanding the chain of events of an AI failure leading to patient harm can be challenging because it requires some engineering or or or software knowledge to be able to understand that chain of events. But some questioning of that software vendor or some research related to how these products fail or what experiences people have had will feed into that. And I think with that information, a lot of clinicians then understand the risk and can then apply their judgment on how to manage the risk as they normally would. I I don't spend a lot of time teaching clinicians how to manage risk. Clinicians know how to do that, they're professionals. Usually it's just putting them in a product failure mindset so they can think about that risk. How can this product fail? And if it fails, how can it harm my patient?

SPEAKER_03 1:19:18

Lee, we have covered a lot. Thank you so much for coming along.

SPEAKER_02 1:19:22

Thank you, Stephen. Thank you, G. Um, it's been a great discussion, forcing me to think about some interesting issues.

SPEAKER_03 1:19:28

Thank you, everyone, for listening. If you have some other questions for Chi or me, please put it in the comments section. Or you can find us both on LinkedIn. And please join us again in the future for another episode of MIAI, covering all your AI technology for radiology topics. As always, please subscribe, follow, and join the conversation.