Helping Healthcare with Apache Kafka and KSQL ft. Ramesh Sringeri

In today’s episode of Streaming Audio, Tim Berglund sits down with Senior Applications Developer of Mobile Solutions Ramesh Sringeri to discuss Apache Kafka®—specifically two Kafka use cases that Children’s Healthcare of Atlanta is working on.

First, they discuss achieving near-real-time streams of data to support meaningful intracranial pressure prediction and managing intracranial pressure (ICP) in a timely manner to help the care team achieve better outcomes with traumatic brain injuries.

Children’s Healthcare of Atlanta is in the process of building machine learning models for predicting ICP values 30 and 60 minutes in the future. This will help the care team better prepare for handling potential adverse conditions, where elevated ICP values could lead to undesirable outcomes. The Children’s team is using Kafka, KSQL, and Kafka Streams programs to build a pipeline in which they can test their machine learning models.

Ramesh also shares about the work they’re doing to mitigate alarm fatigue for care providers. According to him, the current generation of monitoring devices are not equipped to set up multiple alarm conditions, and sometimes a combination of measures need to cross thresholds to be of concern. Children’s is able to leverage stream processing and KSQL to set up multiple conditions, reducing the number of meaningless alarms conditions that might condition care providers to ignore them.

One of the best parts of it all—with Kafka and KSQL, the Children’s team has been able to quickly build data processing pipelines and address business use cases without having to write a lot of code.

EPISODE LINKS

• Join the Confluent Community Slack

For more, you can check out ksqlDB, the successor to KSQL.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_01 0:00

It is always worthwhile to talk to another engineer about a real-life event streaming system they've built. I know I learn a lot that way. But how about one that literally saves the lives of children? That sounds like I'm manipulating you, but I'm not. That's actually the conversation we have in store for you on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to Streaming Audio. I am your host, Tim Bergland, and I'm really excited about today's guest. He's a guy whose work I have followed for some time, and I find it to be super cool work. So let me welcome to the show Ramesh Shringery. Ramesh, welcome.

SPEAKER_00 0:47

Hi Tim. This is Ramesh Shringery, and uh thanks again to Confluent and Tim for allowing me to come on this podcast. Uh I've been listening to uh you and Gwen and all the team members on the podcast uh for uh quite a few uh months now, and I really am excited with all the knowledge that you share. And it it definitely has helped me build some of the things that we have built here at Children's.

SPEAKER_01 1:12

Nice, nice, thank you. That's good to hear. So that's uh uh the uh longtime listener, first-time caller uh kind of thing. Right, yeah. And so yeah, you work at Children's Healthcare of Atlanta. Um, I believe if I may be so bold, your title is Senior Application Developer. So uh talk to us about what you do. I mean, that's is that a hospital? Is it is it should I think of it as a hospital?

SPEAKER_00 1:33

Uh yes, yeah. Uh Children's Healthcare of Atlanta, it's a nonprofit hospital system. And our goal is to make kids better today and healthier tomorrow. Uh, we got uh three major hospitals. We have an autism center, and we also have several urgent cares. Uh, it is a nonprofit. We welcome all donations. Uh and my role, I I work um in a group called Pediatric Informatics. And the charter for our group is innovation. And this is part of a department called uh business intelligence. And uh in this uh department and group, what I do is I wear actually several different hats. Uh my basic background is applications development, but uh with this group, what I do is my one of my focuses uh researching new up-and-coming technologies and then figuring out how they might be applicable to our clinical care, maybe sometimes non-clinical care. Uh and we also have a on-prem uh big data and large-scale computing cluster from Cloudera. And our group is also responsible for that uh big data on-prem uh installation.

SPEAKER_01 2:45

Cool. Cool. Um, so it's um if you don't work in healthcare, and specifically if you don't work in you know large hospital healthcare, probably there's a lot of people in our audience who just don't appreciate the scope of the technology development that happens inside, right?

SPEAKER_00 3:04

Right.

SPEAKER_01 3:05

And and so just take me, for example. Uh I'm a person who doesn't work in healthcare and never has. And and so I I would assume that the software that gets written in a hospital looks like corporate IT, right? Like time cards and records and uh, you know, this cart of stuff went here and had things put on it. Um, you know, that that's that's what I would think of. And what you're describing is not that. I mean, the fact that you work in a group called Pediatric Informatics tells me that um there's something uh I well, I want to say a little bit more interesting. I don't want to say that all that, the rest of that IT work isn't interesting, but something that is a little more connected to the science of what's going on in the building.

SPEAKER_00 3:49

Right. Uh in any hospital system, you know, it's like a very complex organism. There are there are several different things, and you you definitely need corporately IT systems. Uh, you need the electric uh electronic medical records, the health records, and that help our providers, nurses deliver their standard of care, right? So when a patient comes in, you gotta collect some information. And all of that needs to go into a system, and that's where your CRM-like systems come into play. And don't get me wrong, uh, a hospital, you know, uh, it's not just the EMR and not just the medical records and uh what medications are given. There are other systems too, right? For example, what bed becomes available, uh, when somebody's going to be discharged. So there are a lot of interconnected and complex systems. And one of the things that happens is in the healthcare setting in the hospital system, uh, there's tons and tons or terabytes of data that comes forth. So one of the goals of our division, the pediatric informatics, is to try and make sense of a lot of this data and then build systems and then uh do some research on those and figure out what we can do with this data that'll help us deliver better care? How can we make sure we're delivering optimal care so the kids can be healthier quickly and then they can get back on their feet, back to the playground as quickly as possible.

SPEAKER_01 5:18

Yeah, I like it. So where do those terabytes of data come from? Now, I again just think of think of me and everybody who's listening as medical lay people, because most of us are. Right. Um, and you know, we know that in a hospital room there are uh like a patient looks very instrumented.

SPEAKER_00 5:38

Right.

SPEAKER_01 5:38

Uh so what what are some of those things? What what are some of the data sources that uh your group would mess with?

SPEAKER_00 5:46

Uh so in a in a typical uh case of a patient who is admitted to the hospital, so this is what we call an inpatient scenario. The patient is admitted into the hospital, they probably stay uh several hours or several days, and hopefully not very long. But then when they come in, the basic systems that they get attached to are what we call the vital science monitoring systems. Uh, these monitoring systems will record your heart rate, your respiration rate, and basically your basic vitals. And then, depending on what ails you, there are other systems that might be attached to you. Uh, for example, in trauma patients, for example, uh you have uh things that measure pressure in certain parts of the body, maybe the human brain or uh maybe uh your extremities or things like that. So there's a host of systems that get attached, and the measurements that come out of these systems sometimes is so huge. Uh it could be minutely, it could be every second. And we we do have a couple of systems that are uh recently been installed that put out 256 measurements per second.

SPEAKER_01 6:57

Okay. Okay. Now, all right, now we're we're uh we're starting to talk about real money because I'm I'm thinking, you know, the the kind of bandwidth of human vitals, like my heart rate and respiration rate, and you know anything else, blood pressure if you have if you have like an intrusive monitor and you're actually measuring that in real time. Um, like the bandwidth of these numbers is small, and you just don't need to sample them that much. So, you know, do you need a sample every second? And like how much metadata is there? And what do you have? A thousand beds? Like, I'm doing the math. Right. Um I'm just wondering how big does that data get? But now you're starting to talk about hundreds of measurements per second. And I imagine you could tell me that uh, you know, this is not an eight-byte long coming out of the device, right?

SPEAKER_00 7:46

Okay, just yeah, so so uh just to give you an idea, uh so right now we have a high-speed uh White Science monitoring system that is hooked up to about 50 beds, and that data collection, and this is only in one unit, the cardiac intensive care unit, and we collect 23 terabytes of data per year, and we are looking to increase that number of beds to 75 or even 100.

SPEAKER_01 8:13

Okay. Yeah. Uh uh to paraphrase that one guy, uh terabyte here, terabyte there, pretty soon you're talking about real data. Um, yeah. So again, that um it's it's just kind of funny because if you had asked me uh, you know, does a high imagine the most fully instrumented human body that you could instrument in the biggest practical hospital in a large American city, how much data is there? And I'd probably say, eh, you know, some, but not that much. And in fact, I'm just wrong. So uh there's actually a fair amount of data. Yeah. Um there's one uh thing in particular that uh you and I have been talking about over the past a year and a half. Right. One really cool project. Yes. Uh and uh you and I both know what that is, but our audience doesn't, so tell them about that.

SPEAKER_00 9:04

Okay, so that this is a real real special case, and and then hopefully we'll be able to uh get it approved and take it to production uh pretty soon. But uh what happens is we have cases of trauma patients coming in, and it's it's uh one of the critical elements that gets measured in uh a trauma patient is called the intracranial pressure. So what happens in trauma is you're you have suffering head injury and there are fluids uh that are building up, and the skull is a contained uh place, right? And then the brain has, and then if something is leaking in there or there there is uh fluids build up, the blood is uh maybe there is a uh clot in there that is putting some pressure, it's the brain starts to swell, and then pretty soon, you know, it could get into a very adverse situation where it's a life or death. And this is uh monitored by monitoring this intracranial pressure, the ICP. And that happens.

SPEAKER_01 10:06

And this is what what gets you in the case of traumatic brain injury. It's not always the you know, if it's not the actual insult to the brain at the time of the trauma, um, then it's the swelling post-trauma, and that that can be could lead to morbidity and mortality, as your people say.

SPEAKER_00 10:24

Right. The uh adverse outcomes. Adverse outcomes. There we go. Yeah. That's right. Yeah. So what happens is as the brain is expanding, it could be because if it clot here or a leak somewhere else, but then the brain itself is now starting to press on other areas, areas that might control your heart, and then suddenly you're going into cardiac arrest, and or your blood is not pumping, and extremities now have no blood supply. So it's it's very critical that this pressure is monitored, and then it's a very protocol-driven care that has to be put in place, right? Uh, as soon as you see an elevation or a trend in the elevation of the ICP pressure, now you've got to uh gather your care team and they need to make stat decisions, instantaneous decisions. Are we gonna take the uh kid into surgery? Are we going to just give some medication? So there's very high pressure situation. And uh our um neurosurgeon uh what uh what they did was uh they had an idea of maybe this is one place where you know what, everybody talks about machine learning, and that's supposed to be the silver bullet of the magic. And uh, okay, maybe there's an application here uh where we could build machine learning models that could help us predict this ICP out into the future. Now, right?

SPEAKER_01 11:49

Yeah, very interesting. Sorry, I'm just I'm just gonna I didn't mean to interrupt, but that's that is super cool.

SPEAKER_00 11:55

Right. So what that does is like our doctor says, right, so it gives you additional time. Now, based on this model, if the model is able to say that 30 minutes hence or 60 minutes hence, we think the pressure is going to be at this elevated level. So if there's a threshold uh in kids, is it's anything about 25 is really dangerous, and then you need to start talking about innovations, right? So if you see that trend, now they have 30 minutes or 60 minutes advanced notice to assemble their team and then try to figure out what is the best care, what is the course of action that they need to take.

SPEAKER_01 12:34

Right. Right.

SPEAKER_00 12:35

Yeah. So now, yes, model building is good, the model can do a prediction, that's fine, but it goes only so far to have a model. The model needs data, right? So this is where streaming comes into play.

SPEAKER_01 12:50

Okay. So talk to me.

SPEAKER_00 12:55

Keep going. Okay, cool. So uh the model that we built, uh, it takes uh several different features. So our neurosurgeon, we had a uh intern come in, and then they helped us build a model, and uh the model takes several features. So our basic uh idea here is you could actually take uh other proxy vitals like respiration rate, the blood pressure, the uh ICP itself, and the heart rate, and then what is their oxygen saturation, and you could take all of these and feed it to the model and train the model to recognize what is the interaction between these different vital signs. And uh the model they come up, came up with was we can take these five basic vital signs and then we compute various statistics of uh over the sub-time periods. Uh for example, we uh compute the mean, the median, and the standard deviation over uh different intervals of time. Uh, we do five minutes, 10 minutes, 10, 15, and 30. So now you're talking about 60 to 70 features that get fed to this model. So once the model is trained on all of these things, now we're able to attach a new patient stream and have it going. So the model needs about 30 minutes to start priming, uh, because until 30 minutes, you don't have all the features, right? If you're if you're trying to predict uh 30 minutes in the future, you need uh 30 minutes of data to do various computations.

SPEAKER_01 14:26

And that's 30 minutes of being connected to a single patient or correct. Okay. So I am I am uh the the scenario, and again, this is all pediatric stuff, right? So this is a child with some kind of head trauma. Right. And everyone suspects everyone knows that intranial pressure could be an issue some hours from now. So you get the patient instrumented, connected to the devices, and now you're acquiring 30 minutes of streaming data on these vitals. So at this point, there is no intranial pressure monitoring happening?

SPEAKER_00 14:59

Uh there is intracranial pressure monitoring happening. Okay, yes, all right. The instrument is recalled.

SPEAKER_01 15:04

It's intrusive, right? You actually have to make a hole and insert a thing, right?

SPEAKER_00 15:07

Yes, yes. There are transducers that go into the brain that actually measure intracranial pressures. Right. So what we wanted to build, however, was to give the doctors and the care team additional time by the prediction that we wanted to bring.

SPEAKER_01 15:22

Okay, because if you just wait, it's trivial, and I'm sure the ICP devices already gave you thresholds and they would beep and they would alert the nurses station, and but that means things are already going wrong and you're in a huge hurry. Exactly. Adverse outcomes are very likely at this point.

SPEAKER_00 15:38

Very highly likely, yes.

SPEAKER_01 15:40

You're trying to beat that clock.

SPEAKER_00 15:42

Right.

SPEAKER_01 15:43

Okay.

SPEAKER_00 15:44

Beat the clock is a great expression, yes. Yeah, yeah.

SPEAKER_01 15:47

Cool. Okay, so go on. I I interrupted you just uh just trying to make sure I understand.

SPEAKER_00 15:51

But that's right. Uh so the two two things that we wanted to uh the in terms of lead times, we wanted to give the doctor uh 30 minutes and uh 60 minutes. And uh we did develop some models. Uh we used uh a Python script-based model, then we use another tool called Data Robot. And uh the model is one thing, right? So what Kafka and KSQL allows us to do is you can establish a pipeline that feeds these models, and now you have a system where you can plug and play different models and see how they're behaving, right? So, this the whole setup is a medical system, and then it before you can put into production, it has to go through various committees and approvals and potentially FDA and all of those things, right? So, how do we test this over a long time? How do we bring in new data sets and immediately test it? That's where streaming is really helpful, and having a pipeline that can easily do things is really helpful.

SPEAKER_01 16:55

Okay, okay. So um talk to me about that pipeline. And I'm gonna, for a moment, uh, we're we're gonna talk about regulatory uh sort of your regulatory life at the end. So I'm sure I'll ask questions that sound like they're completely ignorant of regulation. And frankly, they are. So you you can inform me later. But for now, um that that pipeline, do you have like historical data from some other patient in the past that you trained the model on?

SPEAKER_00 17:26

Right. So we have several uh years worth of data that we use to train the model on.

SPEAKER_01 17:32

Okay, so that's all there, and we'll we'll say in Kafka topics. And the model itself is a series of case equal queries that then you can you can experiment with.

SPEAKER_00 17:44

The pipeline is a series of case equal and streaming programs that we can experiment the model. The model itself is built using a uh set of Python programs or uh a data robot. And right now we have one up in uh Azure.

SPEAKER_01 18:03

Got it. And so I can you actually can you walk me through that development process? I'm kind of interested in um you know, it starts with uh devices are connected to patient and data is acquired. So tell me where does that data land? And what's the process like for developing the model? And then what's the process like for operationalizing that model? I'll just make that assumption that that's what it means to go write the KSQL queries is you're operationalizing the model. So could you kind of take me through a day in the life of a developer doing those things?

SPEAKER_00 18:36

Uh sure. Uh so the model development itself happens offline. So what I mean by that is uh there's no Kafka or KSQL involved in building the model, right? So it's a it's a machine learning prediction model. We extract all of our data from uh our Hadoop cluster and uh shape it in a certain form, clean it, cleanse it. And our data scientists they build the model.

SPEAKER_01 19:00

And once the model is sitting there in files that are in some format that makes them happy, and they write Python code or whatever.

SPEAKER_00 19:08

Yeah.

SPEAKER_01 19:09

Yeah, okay. So they're they're doing statistics on the numbers.

SPEAKER_00 19:13

Correct. Right. They're doing the statistics and math on the numbers, and then they say, oh, here's a model, and it's the RMSE is so and so and all that. Okay, I'm not a data scientist.

SPEAKER_01 19:23

Neither am I, but we we think of them wearing a white lab coat, holding a clipboard, maybe goggles, doing their data science.

SPEAKER_00 19:28

Right. And the outcome of that is is some kind of a program, right? So it at that at that point, it's still not useful to me as a regular developer because what am I going to feed it? I mean, I can extract a set of things into a CSV file and then take it to the model to have it scored, right? But that still only tells me if the model is behaving good or not in this case, right? So the next step after the model is ready is to get an API on top of it. So now I can make REST calls and send it some features and have it predict me the value.

SPEAKER_01 20:05

Got it. Could you better start out? Tell me about the scoring for a minute. So that scoring I'm gonna assume means that after you've got basically these time series that you're feeding the model, and then you know what the outcome was. Uh right.

SPEAKER_00 20:20

So model scoring uh usually means uh I I'm not a data scientist, I might be making some assumptions here, but essentially what I've seen our data scientists do is uh they have the data set divided into trainings and then also validation, right? So they have done all of that, and they say we think this model is good enough. So at some point they have to validate with some set of data. So we extract another set of data and give it to the model in the form of a CSV file, and then what it outputs is a root mean square error of its predictions versus the actuals. Okay. These are all pre-labeled data, if you will. Okay, and then it's gonna say, well, this is what the model is behaving. You said the actual value of ICP was say 25, and the model is saying it is gonna be 25.5, right? And this is just a rough example. So at that point, you say, Oh, okay, my model seems to be be behaving very well, right? So that is a scoring, and then whichever model uh has the lowest RM RMSE, root mean square error, they choose that and say this is the best model for our use case. And that's that's that's a trained model, right? Makes sense.

SPEAKER_01 21:31

And that's that's uh the output of this data science labor laboratory is that model.

SPEAKER_00 21:37

Right. And on top of that, they give a slap and a REST API for uh developers like us to uh make the prediction calls. Okay. So once that is ready, so now we gotta figure out how to operationalize this or how to make it near real time, right? So that's when our actual work, the Kafka work, the KSQ work starts. So if you remember, I said we have devices, these are physical devices that are connected to a patient and they measure certain vital signs, and then they come out as outputs of uh waveforms and some kind of a file or messages and things like that. Okay. So all of that is aggregated and then it flows to a system, a software system that the uh vendor has provided. So in this case, it's Philips. Philips has monitors, and they also have a system that can read those readings and put out uh records in the form of uh HL7 messages.

SPEAKER_01 22:40

Okay. What's that?

SPEAKER_00 22:41

So HL7 is a uh it's very similar to JSON. Okay, it's an encoding format. Okay, but it's a text-based encoding format and uh it's a human readable. It uh has certain ways of how things are encoded. It's gonna it typically has a header which tells you what type of message is in. Okay, and then it's gonna tell you how each element on a row is demarcated and if there are arrays of things, how they are demarcated and things. It's an encoding uh mechanism, but it's all text-based, so it's easily transmittable and then understood and parsed.

SPEAKER_01 23:20

Got it. And it's a healthcare specific, it's like healthcare JSON.

SPEAKER_00 23:23

Yes, it's healthcare JSON. Okay. Yeah, it's it's a very, very rough analogy, but it'll help.

SPEAKER_01 23:31

Sure.

SPEAKER_00 23:32

Okay. Okay. So what happens then is we got an integration engine. Uh its core point is the integration engine, which reads all of these HL7 messages, and then it creates discrete text files that are uh stored on disk drive.

SPEAKER_01 23:49

Okay.

SPEAKER_00 23:49

Okay. So now what we have done so far is uh we had the devices, we got the vital science out of it, and then we got the integration engine to writing out these text files with all these readings for these patients and a bunch of other stuff, like a metadata. Okay. So now what we then do is what what the pipeline that we built, what it does is it takes these measurements and puts them on Kafka topics. So that's when your streaming starts.

SPEAKER_01 24:20

Okay. So this is and the integration engine is a piece of specialized software infrastructure unique to healthcare, and I guess even to hospitals. And it knows how to talk to the sundry devices in the room. And I imagine there's a small number of standards for how that integration works. Yes. And they uh kind of barf out their HL7 and it parses their HL7 and and puts it somewhere, and that somewhere is just a file drop, or does it know how to speak to Kafka?

SPEAKER_00 24:48

Uh it it probably does have does know how to speak to Kafka, but we have not explored that. But right now it is just a file drop.

SPEAKER_01 24:56

Gotcha. Yeah. Maybe it does, but it's best not to let that happen directly. That probably makes sense.

SPEAKER_00 25:01

Right. So the the one of the one of my goals uh was not to upset anything that is already working, uh, because the integration system integrates with a lot of other systems in the hospital. They were very critical for patient care. So it's not easy to try and monkey with that.

SPEAKER_01 25:18

Exactly. So if there is a directory on a file system where files appear, that's great.

SPEAKER_00 25:24

That is great.

SPEAKER_01 25:25

Yes. Yes. Now we can start, we can we can hook up things. To that, to that point, you know, uh that's just enterprise integration, right? You you have something that's high stakes and you can't touch it. And if you touch it, it might break. And you know what? It does this thing, and yeah, this thing is gross and it's embarrassing, and you don't want to tell your friends, but it works. That's kind of uh always the story.

SPEAKER_00 25:46

So exactly. And oftentimes, you know, in healthcare setting, you gotta remember that a lot of these systems in the integration, they go through a certification process. And anytime you change something, it's a whole other set of processes that you have to go back and re-certify your system.

SPEAKER_01 26:04

And the CIO surely has a dollar amount that that he or she knows that that costs, and it's not small.

SPEAKER_00 26:11

It is not small. So you know, as a developer, you might think, hey, I just want to uh write this little bit of a program, I'm gonna attach this function, but you gotta think of all the other implications.

SPEAKER_01 26:21

Yeah, but you, Ramesh, you wake up in the morning and you think there's nothing I love more than file drops.

SPEAKER_00 26:27

So uh okay. So once it sits in these these file drives and we start putting them on the Kafka topics, uh we use this folder connector. And uh what it does is it just reads these files and transforms them into Avro messages and drops it on topics.

SPEAKER_01 26:47

Okay, nice. Okay. I I was I uh was wondering if there was a Kafka connect in there, and it sounds like there is.

SPEAKER_00 26:52

There is, yes. Uh-huh. Okay. So one of the things that we have here is there are vital signs that are coming in, but uh there are some missing elements in the raw data that comes from these devices. So we have a couple of other streams that we need to be including in our analysis to make sense of the data. So the vital signs themselves, we don't know whom it belongs to. So we got another stream of data coming in which tells us which bed is connected to which device. Okay. And then we also know which patient is in which bed.

SPEAKER_01 27:33

Okay. These are separate, these are separate streams.

SPEAKER_00 27:36

These are separate streams of data coming in.

SPEAKER_01 27:38

Okay.

SPEAKER_00 27:38

Okay. And uh the way the bed information, device information is extracted is we have batch processes uh that run every minute or half a minute, and they directly talk to the backend systems that those bed systems and the patient systems are hooked up to. So the device is a different stream, the bed is a different stream, the uh patient records are a different stream. And all of these just land as file, EDBD files on the file system, and we then take those with a connector and drop it on topics. Got it. Okay, so that is the start of this uh pipeline work. Now that we have things on the topics, we can do interesting stuff. Right?

SPEAKER_01 28:24

Yes.

SPEAKER_00 28:24

That's where KSQL comes in.

SPEAKER_01 28:27

Once exactly, once it's in the topic, we can do things with it.

SPEAKER_00 28:30

We can do things with it. Once it's in the topic, you can do lots of things with it. Right. So one of the first things that we do is uh match up all these devices to patients and beds and create another stream that identifies while this is a vital sign for heart rate, it belongs to this patient, it was taken at this instant. Okay, so we have that information. Okay. All right, so now we're still not ready to feed it to the machine learning model, the model, right? So what we have now are discrete rows of data. So for this patient, we have a record for heart rate, and for that same patient, we have a record for uh respiration rate, oxygen. So this are all discrete rows, and I gotta flatten those out. So we so what we do is then from these topics, read that, flatten it out, and create one record that has for this patient, these are the five vital signs at this instant that was taken.

SPEAKER_01 29:34

Okay. Okay.

SPEAKER_00 29:36

Okay.

SPEAKER_01 29:36

Yep.

SPEAKER_00 29:37

So so far we got a flattened record of for a patient, five vital signs that are of interest. So now it is still not ready to be fed to the model. So now we have to run some uh statistics on these vitals. Well, uh, remember we want the mean, the median, and the standard deviation. And all of these are time-based, right? So mean over five minutes, mean over 10 minutes, standard deviation over uh 15 minutes.

SPEAKER_01 30:04

Got it, got it. I was gonna say that would those would have to be over some window, and and sure enough they are.

SPEAKER_00 30:08

Correct. So this is where we did run into some challenges. And uh we did discover uh uh some bugs in the confluent platform.

SPEAKER_01 30:18

Uh and we thought that was impossible, but um but in fact it's not. So absolutely, yeah.

SPEAKER_00 30:26

Right. So one of the first bugs that I ran into was the timestamp issue, uh, where you know uh when I write to a topic, the timestamp of that record is when I write to I couldn't change it to what I wanted it to be.

SPEAKER_01 30:43

Right. And that was yeah, so you guys were very early adopters of KSQL. So that do you remember what what version that was in?

SPEAKER_00 30:50

Uh this was 4.1.

SPEAKER_01 30:51

Okay. All right.

SPEAKER_00 30:52

Yes. 4.1, 4.2, I think. Yeah, yes. So then the good thing is I was able to reach out to your uh team members and have a one-on-one conversation via Slack, and then they were really able to help me. And we went back and forth several times, and they said, yes, this is a bug, and we will fix it in 5-0. And it has been fixed in 5-0. But the workaround was to have a temporary topic where I pump the data onto it, and then I from that temporary topic, I pump it to another topic with the timestamp that I'm interested in.

SPEAKER_01 31:24

Got it. Okay, okay, that makes sense. Because I I um uh just uh in all honesty, not aware I'm not aware of the bug you were referring to that was in 4.1, but I thought that the create a new topic with timestamp from extract, you know, extracted from this field and all that. I thought that had been working back then. So that was the workaround, just an episode.

SPEAKER_00 31:44

Yes, that was a workaround, and then that was uh and and and kudos to your team because you guys were working very hard to this. Was very close to the 5.0 release, and I think I got a beta copy of 5.0, and it was still an issue, but you guys were able to fix it by the time 5.0 was official.

SPEAKER_01 32:02

Awesome.

SPEAKER_00 32:03

Yeah, that was great. Yeah. So so the next thing that we ran into was so now we have all the uh whittle signs and the statistics and things like that. Uh, we wanted to compute the statistics. So there's there was no easy way of computing statistics in KSQL. So KSQL has very rudimentary sum and a couple of other exactly.

SPEAKER_01 32:26

There isn't a built-in function for any of those things that you were talking about.

SPEAKER_00 32:31

That's right. Yeah. So that was like, okay, bummer, what we do now. And then again, I was able to talk to your team members. I got uh in touch with Apurva and Rohan, and we went back and forth and how to how to how to have this built, right? Well, I need these statistics, those are really important to me, but uh UDFs also didn't seem to cut it. So, as a fallback, what I had to do was I had to re switch to using straight streams processing in a standard uh Java program. Got it. Okay, so however, uh Rohan gave me an algorithm how to compute these statistics with because state is involved in computing these statistics, right? So there was a little trick that uh he showed me how to do, and uh I'm not a Java programmer, so what I did was I pulled out Scala and then in two weeks picked up enough Scala to put together a program that can do streaming statistical uh computation.

SPEAKER_01 33:33

Okay, very nice. Very indeed it was Scala. And this is in a streams application now.

SPEAKER_00 33:38

This is a streams application now.

SPEAKER_01 33:40

Did you did you at any time consider a custom UDAF for any of this? Was that a uh API that crossed your your path at all?

SPEAKER_00 33:49

Uh not recently. I did uh look at UDAF, but then it was only Java only, right? Yep.

SPEAKER_01 33:57

I mean you could probably do one in Scala. You could probably make it happen in Scala. Yeah. In fact, you you can make it happen in Scala. But um, and it's as a sensible API, it's relatively new. I'm pretty sure 5.0. I should know that off the top of my head, but I think it was uh in 5.0 that that became um uh I'll say a responsible API before you could write them, but you had to rebuild all of KSQL yourself. So anyway, you wrote streams applications, which is of course is a completely fine thing to do.

SPEAKER_00 34:28

Right. So yeah, the UDAF thing, you know, you had to rebuild the entire KSQL, you have to download the source and then incorporate it. I think, like you said, it was it was completely changed after Fiverr was released, where you could do it kind of not having to recompile the universe, right? Right. So that's that's good. So we'll we'll go back and probably refactor our code. But at this point, we have a streams application that reads data off of these topics and then uh computes the statistics and puts it back. Oh right. So and then after that, uh we have to again join all these statistics and then form the uh 69 or 70 uh feature list. So that was again done via KSQL join stuff.

SPEAKER_01 35:14

Okay.

SPEAKER_00 35:15

Okay. So after all of that is done, now we have a stream, a single stream with all the 69 features that our model takes and with the patient information. And then now we are ready to uh feed that data to the model. So we I wrote another streams application that would just make a REST API call, get the prediction, and put it back on another topic.

SPEAKER_01 35:40

Okay, so the mod that's how the model is operationalized by virtue of a REST interface. So it's correct. That code is still entirely owned by data scientists. You don't go re-implement it in Scala or anything like that.

SPEAKER_00 35:52

No, no, no, no, no, no. So so one of the first steps of making the uh model operational was a need for a risk-based call for its predictions. Got it, got it.

SPEAKER_01 36:03

And that's uh for each kind of thing in uh in Streams API where for each message you go and you know, cause that side effect out in the world.

SPEAKER_00 36:12

Correct. Exactly. Yeah, there's a little bit of uh promise and waiting and awaiting uh for things to work. But it's it's amazing how uh little lag there is from making a web call across the internet and then getting back a prediction value.

SPEAKER_01 36:26

Nice. That that speaks well of the uh your data science team and their uh their models and the tooling they're used for the using for the API.

SPEAKER_00 36:34

Right. And also the cockp streams, uh the whole thing, how beautifully it works.

SPEAKER_01 36:40

Yeah, yeah. No, it is it is designed for precisely what you're doing.

SPEAKER_00 36:44

Right.

unknown 36:45

Yeah.

SPEAKER_00 36:46

So now what we were able to do is look at what the ICP value is going to be, 30 minutes or 60 minutes, and we can compare that with what the actual recorded ICP is, and that's how we can evaluate the model in real time. I know the RMSE values and all of those will tell you statistically if the map the model is good or not, but then nothing like comparing real data with the actual predictions.

SPEAKER_01 37:12

Right, right. Oh man, that's so cool. Um, are you and you said this is still pre-production, you're able to talk about it, but it's not it's not live yet.

SPEAKER_00 37:20

No, it is not. It is not live, it is still pre-production. The models have to be evaluated, and uh, we gotta gain significant confidence that the model is able to uh perform well.

SPEAKER_01 37:32

Right. Do you have any indication early on that you're able to talk about that that uh you know I I assume you would measure efficacy of the model in terms of like uh number of extra minutes a trauma team has or something like that. Or I guess ultimately health outcome is is kind of your measure of success. So do you have any predictions that of that nature?

SPEAKER_00 37:54

Uh not at this time, unfortunately.

SPEAKER_01 37:57

Cool.

SPEAKER_00 37:57

A little bit early.

SPEAKER_01 37:58

Sure. Not not that you can say, right? I'm sure there are there are numbers in spreadsheets, but they will stay in those spreadsheets.

SPEAKER_00 38:05

In those spreadsheets, yes.

SPEAKER_01 38:07

All right, well, that's exciting. That's a that's a legitimate life-saving technology. You've got this cool mix of Kafka streams that you're uh where you're you know you're doing the statistical computation in there and the the stream wrangling. You know, you've got a bunch of stream stream joins, right? Things like that going on, and you're doing all that in KSQL, which is fantastic.

SPEAKER_00 38:30

Right. So yeah, but there was one other use case that we had uh come across earlier, and that's also a healthcare setting. Uh do we have time to quickly go into that? Oh, totally.

SPEAKER_01 38:44

No, no, I'm I'm yeah, I'm great. We are uh we are in good shape. If you're if you've got time, I've got time.

SPEAKER_00 38:49

Okay, I got time. Okay. All right. Uh so I remember I told you about vital signs being monitored, and then these are uh instruments that are hooked to our patients. And all of these instruments come with alerting systems, right? So you can walk up to the instrument and say, hey, if the heart rate goes above uh 100 or whatever, it starts sounding an alarm, make some noise. Right. Right? So it's good. But the gap there and the challenge there is there's only one set of alarms that you can set up, right? And then there's, as far as I know, there's no windowing operation that you can set up. So when we talk to our uh providers, you know, they say one event is not significant, but if an event X happens over Y interval of time, now that is significant and we need to pay attention. Okay. It's not just one, it's a combination, right? If the heart rate goes about X over five minutes in conjunction with the respiration going below something, in conjunction with oxygen going down. So there's a whole set of interactions that happen. So all of that, it's not possible as far as I know to program the device to set up those kind of conditions.

SPEAKER_01 40:02

Yeah, the the devices are like is this number above or below this threshold? I mean, that's that's um that's sort of the you know, uh high school, after school, uh medical data science club level of competency, you know, but that's not you're not gonna move any needles in terms of health that way.

SPEAKER_00 40:21

Right. And what that is gonna lead is uh it's it's a real thing called alarm fatigue. Yep. So that is a real thing in healthcare.

SPEAKER_01 40:31

Tell us what alarm fatigue is.

SPEAKER_00 40:32

Alarm fatigue. So well, alarm fatigue is the boy who cried wolf story. So it goes back to there's so many alarms, and then so many of these things are benign, meaning they are not significant. And you have so many devices attached, I don't know which ones is really valuable. So what I do is I just shut my ears and close down. I I'm uh overwhelmed with all these alarms.

SPEAKER_01 40:56

And that is that you are in a room that sounds like beeping.

SPEAKER_00 41:00

Yes.

unknown 41:01

Right.

SPEAKER_00 41:01

And then you try to filter it out, and then you is as if it's a silent room. So you don't care. Yep. Because it's so overwhelming, you don't know what to attack or how to uh handle the situation. So what we realized was we can then take all of these data coming out, put them on streams, and once they are on streams and topics, that just opens up a whole world of opportunities. So our first attempt uh was to take uh Spark streaming, and then we built a little system around that to take these streams uh from the monitors and then put a set of conditions around each of those streams to only alert based on what a provider has set up. And how does the provider set up those alarms? Uh I built a little mobile app where you can say, okay, these are the five patients that are under my care, uh, which again talks to a back-end system and pulls it up based on your record. And then you say, I want to be notified if the heart rate goes above 100 for the past five readings and the respiration goes below 72 over the last three minutes, right? So you can cut up and set up a combination of these uh conditions, and then the notification is automatically going to be sent to your phone. Okay. Right? So now it is you're alerted based on the condition that you think, based on your medical knowledge and your experience, that is worthwhile investigating. So we have reduced a whole set of alarms and false positives uh from you having to react to it.

SPEAKER_01 42:43

Okay. That sounds great. That gets you out of alarm fatigue. And I mean, it's it's not just getting you out of alarm fatigue, but it's really uh sophisticated alarming that relies on uh a combination of measurements from a combination of devices.

SPEAKER_00 43:00

Correct. Yeah, and also a care team, there might be three or four different doctors, and each one may be interested in a specific set of vital signs. So somebody might be just interested in heart rate, the other person might be interested in heart rate plus the blood pressure.

SPEAKER_01 43:14

Gotcha.

SPEAKER_00 43:14

Gotcha. So our first thing was to write uh uh Apache uh uh streaming, right? Spark streaming.

SPEAKER_01 43:22

Okay.

SPEAKER_00 43:22

So one thing we noticed was uh when we built that system in Spark Streaming, we had to write a lot of code. And I also used um Azure to set up my conditions. So Azure Event Hubs and push notification services. So the event hubs is a neat little API, but then you have to write some.NET code, and I had to construct the SQL looking code uh so that it's dynamically setting up all these conditions. Now, if you are looking to do the same thing with Kafka and especially KSQL, there's a whole lot of less code that you have to write. Yes. So KSQL, you you just probably have to write one SQL looking. And then this SQL looks as if it you can just copy paste it into an IDE for a SQL server and it'll run. Right? So that's how familiar it is. And uh the amount of code that you have to write to set something up, something similar to the uh Spark streaming in Kafka slash KSQL is way, way less. And it's so easy to do.

SPEAKER_01 44:29

That's great to hear.

SPEAKER_00 44:30

Yeah. So that that was something that you know I thought is a very good use case for uh KSQL. And then with the with the REST uh API layer that you all built around KSQL, it's very easy for me to publish those dynamic SQL for changing conditions.

SPEAKER_01 44:46

Exactly. That's the KSQL server. It's uh well, one of its interfaces with the world is a REST interface, and you can basically just post SQL text at it and it says, Oh, okay, well that's a new query, I'll go run that, and it'll start producing results into its output topic and go for it. Correct.

SPEAKER_00 45:03

Yeah, and your provider joins the team and they want to set up their own conditions, they'll push a bunch of buttons, and then boom, I can just post that message to uh the rest interface, and then it's ready.

SPEAKER_01 45:15

Nice. Nice. And that's great to hear that uh the experience as a developer was a lot more pleasant in the platform. It's always nice for me to know.

SPEAKER_00 45:25

Right. Yeah.

SPEAKER_01 45:28

Cool. Um so what's can can you talk about what's next or other ideas that you have? And I I know um you're in a highly regulated, regulated environment and it's it's sensitive to talk about things, but uh where where do you hope to be able to take this thing next?

SPEAKER_00 45:46

Okay, uh in terms of those specific projects, those have to go through uh approvals and we have to still test it out. Uh there are uh we have to present it to various groups. Uh, we have committees that look at solutions and then they uh come at it from a whole different perspective. They are more knowledgeable of the legal landscape and all of that, right? So that is going on in parallel, that is happening. And besides that, uh at uh children's healthcare, we have tons and tons of data that are shuffled from various systems, and all of these ultimately just land in our Hadoop data lake. Uh today, most of these data wrangling in ETL happens via batch processes. These are custom-built batch processes because some of these systems are so proprietary, you have to know, you have to have grown up with the system to know the ins and outs. Right. So, my hope is that I'll be able to free those data, liberate these various data flows, and get them onto Kafka topics on their way to the data lake. Now, what I'm hoping is once things are on streams, now you can start attaching things to the streams and get additional insight. So that is going to be the next thing.

SPEAKER_01 47:11

Right. And that's that's just Ramesh, that's just so great to hear you say that uh uh, you know, of your own accord, like that, because that's the vision, right? Uh and when I talk to developers, uh, this is very much the vision I'm trying to get in front of people. That like, you know, yeah, I know we built data lakes, and uh, they're a good idea because remember the the white lab coated data scientists with clipboards and python scripts. Right. Uh like where are they gonna get their data from? It's gotta be somewhere. So it's fine that that data lake is there, but uh you know, lakes, lakes are stagnant. Um you want the well what we in general call the business, in this case, it's a healthcare organization trying to make people not die and and and have healthier bodies. Um, you know, you you want to be able to act on that right away uh and not put it in a lake and let it languish and figure out what to do with it, but make real-time decisions. And it's just super cool to hear you talking about those very plans uh for your own organization.

SPEAKER_00 48:13

Yeah. So the the the next immediate thing that we really want to get a handle on is uh the high-speed data that I talked about earlier in our uh cardiac care intensive care unit. We have 256 measurements coming in every second. And right now, before it gets to our data lake, 24 hours have elapsed because it's just a lot of staging is happening. And it's it goes to a SQL server, there are custom batch processes that accumulate these stuff, and then until 24 hours worth of data is accumulated, it's not put in the uh data lake. And even that, so if you want data uh for a patient, you gotta wait at least 48 hours before it ends up in a data lake. So that is our immediate focus to help reduce that and hopefully to minutes, if not seconds, uh, by introducing Kafka and KSQL into the mix.

SPEAKER_01 49:08

Yeah, yeah. If you want to use, if you want to make healthcare decisions, that's gonna have to be a system based on Kafka. If you want to do uh post hoc analysis and data science and and you know, a very beneficial and productive kind of tinkering to get understanding into the data, the current system is fine. But if you're trying to support healthcare decision making, that's clearly not a thing that's gonna happen.

SPEAKER_00 49:32

Correct. I mean, if you want to extract data and white papers, yes, the data lake is there, right? And papers are good, right?

SPEAKER_01 49:38

Like it marches forward because of papers. Let us not leave the listener with the idea that we think that's a bad thing.

SPEAKER_00 49:44

That's no, no, no, sorry.

SPEAKER_01 49:46

I didn't mean to No, I'm I'm the one, I'm always afraid of you know sounding like I'm trash talking data lakes because I guess maybe in my heart I am. But you know, the reality is that you know that supports extremely beneficial activity. But yes, it precludes this other extremely beneficial kind of activity, which is real-time medical decision making, a doctor being informed by you know uh this this sophisticated operationalized model that's responding to things that are happening now.

SPEAKER_00 50:17

Right. And and your goal with a data lake in model building is to be able to then take that model and do something, operationalize it, right? And when you talk about operationalizing, it is real time or near real time. Events. It's all based on things happening and then you're reacting to events. So I understand the importance of the data lake, you but what next? I mean, right? That's where a streaming uh tool set like Kafka and KSIC will greatly benefit.

SPEAKER_01 50:48

And that is uh that is what your plan is next, and I love it.

SPEAKER_00 50:52

Yes.

SPEAKER_01 50:53

My guest today has been Ramesh Shringiri. Ramesh, thanks for being a part of Streaming Audio.

SPEAKER_00 50:57

Uh thank you, Tim, and uh thanks again to Confluent. I've really enjoyed uh talking to you, and thanks for the opportunity. I uh have really enjoyed playing with the platform and uh thanks to the team for being responsive. Uh we have worked on some interesting bugs, and that is what makes this entire platform great. Thank you, Tim.

SPEAKER_01 51:17

Hey, you know what you get for listening to the end? A Kafka Summit discount code. Kafka Summit is coming up on September 30th and October 1st in downtown San Francisco, and you can get 30% off if you go to Kafka-summit.org and use the discount code AUDIONINTE during checkout. Just enter Audio19 while registering at Kafka-Summit.org, and that 30% off is all yours. I'd love to see you there. But hey, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at at TLBergland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Uh or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks for your support, and we'll see you next time.