Aiming to better understand the efficacy of public health messaging in a pandemic, a team of Penn State researchers designed and deployed an unusually open-ended survey that has been translated into 23 languages and reached more than 73 countries. This episode recorded via Zoom on May 20, 2020.
Relevant Links:
Guests:
Aiming to better understand the efficacy of public health messaging in a pandemic, a team of Penn State researchers designed and deployed an unusually open-ended survey that has been translated into 23 languages and reached more than 73 countries. This episode recorded via Zoom on May 20, 2020.
Relevant Links:
Guests:
Cole Hons: Greetings, fellow homo sapiens and welcome to the Symbiotic podcast. I'm Cole Hons and my guests today are going to be talking about a fascinating new project that was seed-funded with some money from the Huck Institutes here at Penn State. The title of the project is COVID-19 Health Messaging Efficacy and its Impact on Public Perception, Anxiety, and Behavior. Certainly a hot topic. We're having this conversation on May 20 and with me today to talk about this project we have Dr. Rob Lennon, Associate Professor from Penn State's College of Medicine, who is the PI for this research; Lauren Van Scoy, a critical care physician from Penn State's Qualitative and Mixed Methods core; Cletis Earle, Chief Information Officer for Penn State Health, and we have Penn State's partner from the private sector on this effort, Russ Branzell, CEO and President of the College of Healthcare Information Management Executives, or CHIME. Welcome to the podcast, everybody.
Robert (Rob) Lennon: Thank you.
Cletis Earle: Thank you very much.
Lauren Van Scoy: Yeah.
Russell (Russ) Branzell: Thank you for having us.
Cole: It's exciting to hear what you've been doing. I know a couple of weeks back, I was able to listen in to an online presentation on where you were then, and it seemed like this thing was moving so fast. I'm eager to find out some updates. But for those who have not heard about this project at all, I just want to launch in with the fact that this is largely about a survey. It's largely about trying to find out how people are learning about COVID-19, where they're getting that information, how that information is affecting them and hoping to take that information and see if we can do a better job with our messaging moving forward. I took the survey myself and was surprised at the depth of some of these questions. So, I would like to first ask how many questions are there and how long does it typically take somebody to complete the survey you put together?
Rob: The number of questions you get will change a little bit based on branching logic so depending on your answers, you may get follow on questions. Roughly 65 questions will be asked and the length of time depends mostly on how much free text you'll enter. We have a number of quantitative data points. If you just go through that, maybe 15 minutes at the top. Then if you enter a lot of free text, which really gives depth to our answers, some people spend a half an hour on it because they want to. We have free text responses up to 700 words at a time. The net overall average time for the last eight weeks has been 24 and a half minutes that people have spent on it, which means about 15 minutes of quantitative study and they've taken about 10 minutes just to give us the depth of their experiences.
Cole: Thank you. Could somebody else talk a little bit about the questions that you ask? What kinds of questions are you asking folks to answer?
Lauren: Yeah, sure. I can take that. When we designed the survey we used what's called a convergent mixed methods design. We looked at various constructs and we asked questions about those constructs with both quantitative questions and qualitative questions. In our first construct, we look at knowledge and we're interested to know what is the knowledge that's out there, knowing that obviously knowledge changes on a day to day basis. We asked some true false knowledge questions and how confident you are answering those knowledge questions. We also ask questions about perceptions, about some of the health behaviors and recommendations of the CDC, the WHO, and various places across the world for the social media, the global survey. We asked those questions both quantitatively and we also asked them with open-ended questions about barriers, facilitators to carrying out the recommendations. We also ask about information sources. Where are people getting their information and to what degree do they trust that information? Giving us information quantitative and qualitative around those constructs as well?
Cole: Thank you. My understanding is that the first time this was launched, it was more local. Is that correct? Rob, was it smaller, just a pilot that went out the door to start?
Rob: That is correct. A very small pilot of about a thousand people. When it showed that the answers were sensical, we did a regional study that was actually so well replied it crashed the system in REDCap. We had over 8,000 responses, 5,984 of which were completed. The answers to that gave us, was very concerning. It showed tremendous discrepancy in knowledge and understanding of COVID by various demographics and we realized we really needed to understand this on a global scale. Since we didn't have the platform capability, we asked our Chief Information Officer, the senior VP of it, Cletis Earle who said, "I know exactly who to help." It was actually April 2nd. It was less than 14 hours after we reached out to Cletis that Russ Branzell and his team showed up the next morning and the global survey launched seven days later. That's how long it took to go from a regional survey to international scope, internationally-oriented questions. At that time, 20 languages translated and not just translated in a Google translate way, but Russ brought in European medical experts.
So the translation was not simply a word for word, it was medically nuanced. That also helped us, using his extensive connection with these international partners, it helped us change the way we asked it so that an international audience would respond well to what we were asking. Many people in the world, they never heard of CDC, but they've heard of World Health Organization. They've heard of the European Commission. So that was an incredible thing to have. What CHIME brought to the table in that was just lightspeed in terms of academic medical terms.
Cole: Fantastic. Russ, did you have a previous Penn State connection or how did you get roped into this thing?
Russ: Well, not necessarily a previous Penn State, but a previous Cletis Earle connection. How about that? Or a current Cletis Earle connection. Cletis has been a long time member of our organization and actually was our chair several years ago of our board of trustees and a long time just good friend. He and I have hung around each other for a long time. He did know one thing from his experience of time on our board. We do have a problem at CHIME of the inability to say no. Even if we definitely don't even necessarily know what we're talking about at the time, we're going to figure it out and figure it out really fast. In this case with work with Cletis, he knew this was an area that we probably could bring the right resources to bear with our network.
Again, we built a quite extensive network of really, really highly talented people around the world, not just technicians, but really leaders in the field of health IT. It was really a blessing to get a weird phone call from Cletis at nine something in the evening on that Thursday night saying, "Hey, got an idea. What do you think?"
Cole: Got it. When Cletis came to you with that, the pilot had already been done, correct, and it had crashed the server, I think Rob said a minute ago. What were you able to do taking the initial data that they had pulled in from the pilot? How did you manage to just deploy that so quickly to the globe?
Russ: Yeah, so it was actually a lot of fun, because I didn't actually call the rest of our team that night. I actually stood on it a little bit and I called a quick meeting in the morning, which we do anyway in response to all the COVID issues going on, we just started a little bit early. And I said, "Hey, you're not going to believe who I heard from last night. It was Cletis, our old friend and chair. He has this idea," and I laid out the idea and I let them hem and haw for about five or 10 minutes. And I said, "Oh, by the way, you can hem and haw all you want. I already said yes. We're going to do it and that we knew how."
Then there was a very long period of silence and they said, "Okay. All right, boss. You do this to us all the time. Now what the heck are you going to do with it?" And I said, "Actually, I've already made a few phone calls." We've got a great tech partner who supports us with all of our efforts to be able to be global. Does a lot of behind the scenes technical work, a lot of work with Amazon and Google and Microsoft, in this case particular. We gave them a quick call and said, "We don't have a lot of money. We don't have a whole lot of technical expertise. We know exactly what we want to do, except we're not sure quite how to do it. You want to help us?" They said, "Sure, we'll jump on board." I think our first meeting literally was that afternoon. Then I think we actually had most of the shell and technology built for this on a scaled basis, not translated, by I think that Sunday afternoon. I'll even say for me, this was a little fast for normal technology.
I like going really fast and really bold. But I think even when I called Cletis that Sunday morning and said, "Hey Cletis, you're not going to believe this, but they think they're done with the first phase of this." Even I think he was shocked and I was shocked. By the time we got together on Monday morning, I think the basic tech platform was built in and good Dr. Lennon there, I think you were even a little bit surprised, but I think it gave us the catalyst to move really fast for the right reasons, to try to make an impact on this.
Cole: Terrific. How many nodes in the network are there globally in total that are participating now with this survey?
Russ: Doctor Lennon, I don't know how many different countries and places are working on this at this point. I know there's a lot, something in the neighborhood of post 70, but I do know the ability to connect globally on this platform now is limitless. We can connect anywhere anytime with, I think the number is 98% of the languages of the world represented at this point. And Doc Lennon, you may want to get more specificity. I will just say, I've never heard of anything like this anywhere to be able to scale out of one point. Again, a lot of it has to do with the cloud technology now. You don't have to host things anymore. You use a scaled global platform.
Cole: Rob, do you have current numbers on number of nations and how are you tracking that? How are you keeping track?
Rob: We do. Well, we do it by asking. We ask people if they're from the United States and if they are, we get the first three digits of their zip code. We've had every three digit prefix in the United States so we have representatives from all 50 States, the District of Columbia and so forth. Then if they're not from the United States, we ask what country they're from and they tell us. We have over 73 countries, every continent is represented and it's all by snowball effect. There are people in different countries that see this and push this out. That's a key element of this. People like Bob Mash who's at Stellenbosch University in South Africa. He runs something called Primafamed, which is a primary care network in Africa. He's very interested in this because this really helps them get at how to take care of folks. That's how it's spreading, though. 73 countries to date. Three countries with over a hundred respondents, which is enough to show significant differences in outcomes, and several more approaching that as we go forward.
Cole: Cool. I'm Cole Hons. This is the Symbiotic podcast. We're going to take just a quick little break and then we're going to get back into a deeper conversation about what kinds of things you're learning from all these people from around the world and what you plan to do with the data now that you're collecting it. Thanks a lot. We'll be back in just a moment. Don't go anywhere.
Commercial: The Huck Institutes of the Life Sciences at Penn State offers six inter-college graduate degree programs in bioinformatics and genomics, ecology, integrative and biomedical physiology, molecular cellular and integrative bio-sciences, neuroscience and plant biology. We also offer an accelerated professional science master’s program in biotechnology. At the Huck, we immerse students in a groundbreaking environment built on interdisciplinary collaboration among some of the world's most innovative research scientists and our students receive unparalleled access to bleeding edge technology in world-class core facilities. If you're looking for a deeper, more holistic grad school experience, you owe it to yourself to look at the Huck. Visit us online today at huck.psu.edu.
Cole: Welcome back to the Symbiotic podcast. I'm Cole Hons here with my colleagues from Penn State and a partner from CHIME talking about a global survey that's digging deep into what people think and feel about COVID-19, what they know about it and don't know. I just want to share that when I took this survey myself, I was really surprised at the depth of the questions and I found myself really doing a lot of deep self-reflection. I got to understand my own feelings and thoughts about COVID in a way that I wouldn't have if I hadn't done the survey. I found that very surprising and I wonder if that was intentional. Rob, I don't know specifically what your purpose was in this, but was that all by design to have people do a deeper dive and what kind of responses are you getting by taking that approach?
Rob: It was by design. I have taught at universities for many years, and I feel that the best exams teach you things. When you get done taking a really good exam, it's not that it's examined your knowledge. It's that you've learned something from how questions can be asked. That was the point of going into depth. We got a lot of pushback from that. People that do surveys know that the longer the survey is the less response you're going to get. We got a lot of pushback on going into depth. My response to that was we have to know. We have to know to this depth. We have to understand this. We're so early in the pandemic, we're not even sure what questions to ask so we have to have open-ended questions and we have to ask a lot of them. I was confident that people would talk about it. This will be the defining event of a generation. Certainly, people can take a few minutes to talk about their experience and how it relates.
Lauren: When I first met Rob and we were sitting around Dr. Ruffin's conference table before social distancing came and we were designing this survey, I was skeptical that we would get good qualitative responses from a survey. Because in general when you ask qualitative on a survey and you ask people to free text, whether their opinions or their feelings about something, they'll give you very superficial answers. Just get her done. What do you think about this? I think it's great. Period, the end. Sometimes people just will write no comment or they'll leave it blank. I was skeptical that we would get good qualitative data, but what we learned pretty immediately actually, when we were doing the cognitive interviews for the survey to validate the survey or do what we could to validate it, is that people were wanting to talk about COVID particularly in March, April when we were making the survey. Because it was happening, it was hot, it was on everybody's mind. There was a lot of chaos and change and people were anxious. One of the questions in our survey is, what worries you the most about COVID-19?
Oh, my goodness. The data that we got from that, just on the cognitive phone interviews was just out of this world rich. As a qualitative researcher, I was so excited because I thought if we can get this stuff written down in the survey form and people can actually share with us, then we're going to have these really rich stories and be able to understand things that others doing survey research may not be able to understand. I think COVID-19 is unique in that way because people want to talk about it and they do spend the extra 10 minutes to provide us with good qualitative data. That's something that's been really, really flabbergasting to me.
Cletis: It'd be interesting to see if there's any correlation to the shift of how people respond because of COVID overload and seeing if there's a dynamic of diversity in that manner. So, it'll be interesting to look at it, and that's what makes this so fun and so interesting. That we'll have enough data to look at this holistically and come back with some conclusions.
Russ: We almost joked about it when we first started. The very first time I took the survey, I think it was still a little bit in draft form. A few of us got together and I said, "Okay, I've been reading everything from the CDC. I've been getting all this stuff from my local doctors and health systems, and I'm an expert on this." I immediately came away and said, "I'm not sure I know anything at this point." I realized how much I didn't know, or there was a lot of misinformation out there. I think as this survey has truly rolled out on a global scale, what we're seeing is a common response. I'm talking about people that run health systems. I'm talking about people that are in technical positions, in corporations that are very involved with this from a frontline perspective and their feedback almost always, to a person, I mean without exception, not one exception, I've talked to probably 100 people that have taken the survey though, every one of them has said the same thing. "I thought I knew stuff that I obviously didn't know." Including me, by the way.
Cletis: You see we're still combating this misinformation or a lack thereof of getting information from certain sectors in certain populations. We see there's a huge disconnect between the minority communities, people of color and the impact there. There's some components that we really want to continue to push out the survey and continue to hit the ground in trying to get as many people to participate, because there's a disconnect that's going on. Again, the whole goal is to remove the misinformation that's out there and try to use our survey results to really make a difference.
Russ: Well, what's really amazing is our organization gets to talk almost every day to people around the globe at least, if not around the United States. In any given day, just talking to our own team in CHIME, we got one group in Ann Arbor, Michigan, which is in complete lockdown still. They're not even allowed to take most things like graduations. Then I live in Georgia, which is the home of the wild and the free, and everybody's doing anything they want at this point it seems like. It's just amazing the perspective that individuals have based on the environment in which they live and have to operate. We even spend time within our team debunking some of this misinformation between the two, because the normal and usually correct answer is probably somewhere in the middle between these extreme perspectives being taken around the country and around the globe.
Cole: Yeah. When you take it to the international community then, are you getting any early indicators? I mean, when you have a survey that's so in-depth with so many words, it's not like processing... Maybe LJ could speak to that a little bit, because it's all this stuff to take in. How do you even condense that down and say, "Oh, the United States versus Norway versus Brazil," or whatever it is. How are you going to manage that? Have you been able to see any trends start to emerge yet?
Lauren: Well, it's going to be really difficult to manage, I will say, as a traditional qualitative researcher. This is probably the largest sample size I will ever encounter. Oftentimes we do qualitative research with 20 people's responses, 100 people's responses – 200 may seem large. We have already 8,000 and we've really just started. In our local survey, I say local, we had 6,000 responses. We're just very slowly starting to read the data and find the themes doing traditional qualitative analysis. But it's abundantly clear that that's not going to be sufficient for such a large database. I only speak one language so we have to get linguists to translate all of our qualitative responses. We're interested in natural language processing and other tools that we can use to try to help us organize the data.
Not such that the machinery or the AI would analyze the data for us but help us to organize it in a way so that we can start to make sense of it and know what better questions to ask of our data set. I think that this process is going to be a long one of reading, rereading, organizing, reorganizing, asking the dataset different questions, both qualitatively and quantitatively, and what they call following the thread, taking the research where the data leads us.
Cole: Thank you. Then, what does the timeline look like? The current version of the survey, how long is that going to be live and what are next steps?
Rob: It will be live at least through June 30th. We have grants pending at the National Science Foundation, Facebook, and soon to be at the Russell Sage Foundation to help continue this going on. As the COVID experience continues, it makes a lot more sense to keep it open. As Cletis has mentioned before, looking at changes in response over time is also likely going to be very valuable. We can pinpoint specific things in time that happened and compare responses afterwards. We haven't been able to do, as LJ mentioned, we can't do the linguistic comparisons yet, but by building the dataset, cultural anthropologists will be able to do this. If the data set becomes robust enough, it literally will be a process of years to do that.
Numerically we're waiting until we get responses in the hundreds to do international comparisons. But just recently we did preliminary analysis on quantitative answers. That was actually prompted by Russ, who said, "What have you found?" And we said, "Well, we thought we'd wait a little longer." And he said, "Well, stop waiting. Tell me what you see." I said, "Okay."
Russ: Lightspeed.
Rob: I'm actually very grateful that he prompted us to do that because we found some very disturbing things. We found that there were statistically significant and almost certainly clinically significant differences in the public's intent to comply with CDC recommendations. The reason that's so important is as you lift the lockdowns, the resurgence rate, we think, will correlate to intent to comply with them. So people... The best predictor of human behavior is, what did they do before? Well, no one's ever come out of lockdown before so we don't know. The next best thing is to ask them what they're going to do. Well, we did, and we found that in a lot of places they're not going to comply with CDC recommendations. That's very, very concerning.
In fact, the paper on that, we completed it rapidly because we were so concerned with the data. It's submitted. It's not peer reviewed yet. It's still under submission, but we'll be releasing it because it's, in our opinion, an imminent public health threat. So we're already receiving this kind of data, which might have pretty profound implications on it. That doesn't mean reopening is good or bad. The data is what it is. The data suggests that if people don't get better education and comply with these, they're at much higher risk. So, the natural response is if you're going to let people make these choices, for goodness sake, get them the information they need to make better choices and keep themselves and their communities safe.
Lauren: I can just add to that. One of the things that we're going to learn in the qualitative data is why we're seeing the lack of intent to comply with the CDC recommendations. The quantitative will answer the question, of what are they intending to do, what percentage of people are planning to comply with the recommendations. But for those that aren't going to do so or those that are going to do so, we want to better understand why or why not. That's why the qualitative questions will be so important because we're going to understand the reasons, the thought processes hopefully, behind some of the data. So, in a sense, we're going to be able to explain some of our quantitative data through looking at the way people respond to the qualitative answers as well.
Cole: Got it. And then, what's the end game here? Is it publish a paper and then maybe a white paper that goes to policy makers to influence things in the future? Is that the vision here?
Rob: The first goal we set out to have is to produce actionable data that can impact human health today. That's what we're doing by releasing this information now to say, "There's people out there. We have the analysis to show that they're at risk and we can change that now." So getting that out, that's the real goal. If we ended with that, that's a noble thing to do. The papers that will come out of this will be, a great number of them, some of the most interesting papers are actually simply on informatics. So the professionals that study news consumption in America were very excited when they heard about our survey. They commented that the last time Americans changed their news consumption was after 9/11. They were convinced that COVID would change it, but nobody had any idea how. They don't know where people are going.
I told them, I was like, "Well, we're about to ask every online person in the world. What do you want to ask?" They helped us craft the questions to get meaningful data out of that so it's not just that we'll get medical information. We will get information-information. Just to the news media alone will find that interesting and how health information is shared. The data that we're going to get on how we built this partnership will also change things. A mixed methods study typically takes three to five years to go from concept to completion in a validated survey. We did that in about three and a half weeks, and we are already getting actionable data.
We did that by an academic, private foundation partnership. That has extraordinary implications for how quickly research can move. Now that doesn't mean it's not without its flaws. Anytime you move quickly like that, there are limitations to your data, to how it's collected. Freely admitted, and we are that we are the first to pick our methods apart. But it shows what you can do so that when we're faced with these catastrophic events, we have shown that you can do this. You can go from zero to robust analysis much quicker than we typically do.
It's not just good for COVID, but imagine earthquakes, hurricanes, other natural disasters. These are all things where, by working in partnership in cross disciplinary teams, we can move very quickly and get very actionable data.
Cletis: If I could add, you asked about the end game. I think this is interesting that this has turned to the now game as well. I'll, again, continue to talk about the disparities between how we're seeing it being impacted, people of color being impacted. What happened while we were seeing this and seeing that, whether the type of responses we were getting or not getting, we actually ended up pivoting and we're doing parallel strategies to deal with the now, to try to figure out how do we now get information to and from certain communities. We're tackling that now. It's become an organic thing.
I'm so glad that Dr. Lennon reached out to us and asked for that kind of partnership. Because now we're really seeing that we are, in our side of the shop, we're in the business of making people better, right, and trying to improve healthcare for our patients and our communities. This is allowing us to adjust in almost a real-time basis and being able to try to do something about it. Working with CHIME to figure out, what could we do from a legislative side? How can we go to DC? What can we do to actually be change agents of this change and start to lobby in different ways to say, "How do we make a change for what we're seeing almost in real-time?" I don't think it's been done before at this level of engagement with this public, private conversation. This is amazing.
Russ: If I could, I'll just throw this out there. How do you get a whole bunch of people to do things over a weekend when they're already working 40, 50, 60 hours a week? You give them a compelling vision. It was this simple at the beginning and I will say between Cletis and Dr. Lennon, it was pretty easy to make this pitch and that was, there's really two goals that I tried to push out there. One, we're going to save lives right on the get-go. I don't know if we can directly prove this, but I have direct confidence in this. If we can help improve the education, make people aware of this, there are going to be people alive. They may not know why, but they will be alive years from now because of stuff that happens today. I know we can prove that many other places within health IT.
I think the other is, we need to be ready for the next time. Even though we're right in the middle of this, it does not take you much long in history, and history is years. You've got MERS. You've got SARS. You've got Zika. You've got West Nile. You can just... The number is multiple hands. These were all within the last five to 10 years. So yes, COVID-19 is here. It is horrible. We want to get rid of it, but whatever is going to be the name after this, it will come. There is no doubt. That it's just the way virology works in the world. The next thing will mutate and it will come. We need to be ready for it. There is a process that we need to put in place differently than they have today. I believe the work that Penn State's doing, and specifically credit to Dr. Lennon for these ideas, will shape the way we respond in the future, hopefully proactively enough that we don't even have the problem to begin with.
Cole: Fantastic. Thank you all for those inspiring comments. Yeah. I can see this is a very purpose-driven project. You can feel it. It's inspiring. I wish you all the luck in the world. I do want to share with all of our viewers and listeners, we're going to put a link so that people can take this survey while it is up. What is the best way that people can get online and just sit down for a little bit, have some reflective time and go deep and participate with this survey while it's running?
Cletis: It's easy. We're asking people to go to covidsurvey.psu.edu, and you get right to the link.
Cole: I did it. I recommend it. I think everybody should participate because it is, it's for us right now, as you're saying, actionable data that people can respond to now, you're moving quickly with this, but also into the future as Russ was just talking about so we can get better at this, so we can be better informed, better prepared for future similar challenges and help to transform the industry. I salute you all. You're doing what we believe in here at the Symbiotic podcast. You're cutting across those disciplinary boundaries, bringing together medicine and IT and research and communication to do something meaningful that helps to evolve the way science is done. So thanks for what you're doing. Thanks again for being on the podcast. I wish you all the best. Take good care [crosstalk 00:31:30]-
Rob: Thank you.
Cletis: Thank you everybody.
Lauren: Thank you.
Cole: ...this pandemic. Stay healthy. Thanks a lot.
Russ: You too.
Lauren: Thank you.
Cletis: Thank you.
Rob: Thank you everyone very much.