Consumer Surveys – Separating Pseudo Science from Admissible Evidence

Many complex litigation claims turn on the ultimate question of how a reasonable person perceives an advertisement, marketing message, or business practice. Courts in recent years have been required to decide the admissibility of consumer surveys, separating flawed survey models from those which rise to the level of competent evidence. In this episode, Dorsey Seattle Partner Mike Keyes explains the nuts and bolts of consumer surveys, and how this discipline is often misunderstood with Podcast Host and Dorsey Partner Kent Schmidt.  This episode discusses how Mike’s background as a trademark litigator led him to obtain further training in this science. Listen to learn how litigants can be prepared to either proffer or rebut consumer survey evidence in trial proceedings.

This podcast is not legal advice and does not establish an attorney-client relationship or create any duty of Dorsey & Whitney LLP or those appearing in this podcast to anyone. Although we try to assure that the content of this podcast is accurate, comprehensive, and reflects current legal developments, we do not warrant or guarantee those things. The opinions expressed in this podcast are the opinions of those appearing in the podcast only and not those of Dorsey & Whitney. This podcast is considered attorney advertising under the applicable rules of certain states.

Transcript

Voiceover

Welcome to another episode of the SharkCast on litigation Risks Management, where we explore why businesses are so frequently sued and how to mitigate and navigate the dangers lurking in these risky waters. Join us now as we welcome our host, Kent Schmidt, litigation partner at the law firm of Dorsey and Whitney.

Schmidt

Welcome again to another episode of Shark Cast. I’m very happy and pleased to welcome my partner Mike Keyes to the SharkCast Studios for what I think will be a very interesting episode. Mike is a partner in our firm Seattle office. Until quite recently he was also a member of firm management and incidentally, I reported to him within the firm hierarchy and so for many years I only had very, very nice things to say about Mike as one does to those two may report, but now that Mike is no longer in management at Dorsey, I only have nice things to say about him as well, because he’s all around a terrific person. Very, very talented lawyer and as we will see is a very enterprising and intellectually curious, and I count Mike as one of the most important thought leaders at Dorsey constantly thinking about new and emerging issues and rethinking his practice, reformulating what he’s doing each year, and so I think he’s going to have a lot of interesting things to say. So, Mike, everything I say about you from this point forward is out of complete sincerity. I won’t vouch for things I’ve said in a flattering nature before, but you have complete sincerity from me from this point forward. Welcome to SharkCast.

Keyes

Well, Kent, thank you so much. It’s a pleasure to be here with you today. I’m a big fan of your show, so it’s great to be here.

Schmidt

Well, thank you very much. I want to jump into this and the topic of our conversation today is consumer surveys. Mike’s practice has historically involved quite a bit of intellectual property, and I think there’s a connection between intellectual property and consumer surveys that we would like to hear you explain to us. First of all, what is your current involvement in consumer surveys and their application in a litigation context and how has this area of the law become part of your practice?

Keyes

Sure. So my practice, Kent, is pretty much exclusively trademark and advertising related litigation, and so in those sorts of disputes, one of the central issues is: Are consumers confused? Or are consumers deceived by a particular ad? So one of the staple pieces of evidence that we develop in those regards is consumer surveys. So we go out and we conduct a survey of the relevant consuming public to get a sense for what people are thinking with respect to a given trademark or a given advertisement.

Schmidt

So I take it in your history as a IP litigator, you’ve been on both sides of consumer survey evidence. Both the proponent and the opponent of consumer surveys. How has that informed your approach to learning about consumer surveys?

Keyes

Yeah, you’re exactly right. I mean, I’ve been involved in numerous cases on both sides of the V when it comes to survey evidence, so probably much like if you’re a litigator and you do plaintiffs work and defense work, you’re able to look at the same issue from two different perspectives. So I think it really helps round out your expertise and knowledge to be able to look at a piece of survey evidence from the plaintiff’s perspective versus the defendant’s perspective.

Schmidt

So let’s start with some fundamental points about consumer surveys. First, what are the types of cases that arise that call for consumer surveys? Including perhaps some that maybe we don’t initially think of as requiring consumer surveys.

Keyes

Yeah. So, most consumer survey cases involve matters of perception like I mentioned. So, by far and away the most prevalent type of cases involving survey evidence would be trademark related cases where we got to again look to what the consumer thinks with respect to a given trademark or  given series of trademarks. So trademarks and advertising cases are really where we see most survey evidence developed, but other types of cases as well. We see it in class action, consumer class action cases, survey evidence is used there, we see it in the employment context too. So it’s not just exclusively trademark and advertising cases, but by far and away those are the most typical types of disputes where we see survey evidence.

Schmidt

Now your experience as a lawyer and your education as a lawyer didn’t spend a lot of time with formal instruction and sort of the academic exercise of learning about consumer surveys. What have you done to advance your knowledge in the Consumer Survey area.

Keyes

When I first started doing trademark cases back in the, I guess it would have been in the late 90s. I had the good fortune of working with some really renowned survey experts, Doctor Jerry Ford, Doctor Edmar Simonson are just two of the notable ones that I worked with, and so I received a lot of good training from those survey experts in terms of survey methodology and learned a lot from them, and my own independent study, and just staying up on the law with respect to survey evidence. But then a few years ago I decided I was going to normalize my education in surveys. So I enrolled at the University of Connecticut in a master’s degree program in survey research and data analysis. So I completed those studies last year, so I now have a real formal education with respect to developing surveys.

Schmidt

That must have been an interesting intellectual pursuit, and you’ve, I’m sure, learned a lot having nothing necessarily to do with the law and the application of consumer surveys in litigation. What are some of the historical or non-legal aspects of consumer surveys that you found most interesting and fascinating just from a from an intellectual standpoint.

Keyes

Sure. So you know something that struck me during the course of my studies is how prevalent survey research is in all facets of our lives and how survey research is really influenced a lot of things that we don’t necessarily think it has influenced. For example, kind of going back historically, there was the Pittsburgh survey as it was called and it was developed at the turn of the 20th century and it was essentially documenting how Pittsburgh life was during the industrialized age. That survey led to a lot of key reforms with respect to child labor and other types of laws that came about as a result of the survey that was done in Pittsburgh. Hollywood and survey research is another area. In the 1940s, George Gallup was hired by Hollywood to conduct a bunch of survey research as to what people thought about various aspects of Hollywood, one of the surveys that he conducted led to Gone With The Wind being filmed in Technicolor. He also predicted certain stars would have staying power and actually would become popular, including the likes of Jimmy Stewart, Lana Turner and even Lucille Ball. You know, it’s pretty interesting when you really start peeling things back and looking at survey research, how many facets of our lives it touches. That was one thing that I found super interesting as I really started study it.

Schmidt

And then on top of that and we haven’t even touched on this, but surveys in the political realm are obviously very significant. You know, the polling and all of that. I grew up not far from Peoria, IL down in farming country, central Illinois and there’s that phrase that’s often used, “How is this going to play out in Peoria?” I think there’s this long history of Peoria being this bellwether city, did you come across that in your studies?

Keyes

Yeah, you know, it’s interesting you mentioned that, you know, polling, you’re exactly right. Survey research was really developed at least in earnest, starting in the 1930s by George Gallup and some of the others, George Gallup, was from Iowa, so he had those, those Midwest roots and I think the other it may have been Elmo Roper was also from Iowa, another pollster of the day. So it definitely, survey research in the Midwest, you know, they kind of grew up together.

Schmidt

So let’s turn this conversation back to the question of how this applies in litigation and the first issue that we need to tackle, I think is admissibility. You have phenomenal surveys that have been commissioned and perhaps cost a lot of money to obtain and may be very helpful to your case, but the issue arises as to whether or not a Court’s going to allow that survey evidence to come into the case and be considered by the Trier of Fact, whether it’s the judge or the jury, what are some of the guiding principles for the admissibility of survey evidence in in most courts?

Keyes

Yes, so you’re exactly right. I mean this is an evidentiary issue in the first instance, and it’s going to be governed primarily by if you’re in Federal Court, for example FRE 702, which details what the proponent of expert testimony needs to.satisfy in order for that particular piece of evidence to be admissible. When it comes to survey evidence in particular, you need to establish that the survey was developed according to generally accepted principles, and these principles are articulated in a number of different sources. There is a treatise that’s published by the Federal Judicial Center that really details how a survey should be conducted. There’s also other third party references. The American Association of Public Opinion Research has its own set of criteria, so that’s an important guidepost to consider in developing survey evidence and then of course, you have a very robust and growing body of case law. You know, the federal courts have developed over the last several decades, very significant set of analyses in terms of how a survey needs to be conducted in order for it to be admissible.

Schmidt

Do you find notwithstanding that body of law on admissibility that Judges have varying types of almost instinctive or visceral reactions to the suggestion that survey evidence should be considered. So I’m saying, you know, some be very skeptical of it and others saying, well, of course, this is a case that we want to know what the public thinks and the best way to do that is with the survey

Keyes

Yeah, I think that’s exactly right and I found over the years that it’s really circuit specific. For example in the 9th Circuit where you and I are situated, the 9th Circuit takes a historically pretty liberal view when it comes to admissibility of survey evidence and trademark. Cases other courts take quite a different approach. For example, in the 11th Circuit, you know over the years it has somewhat soured on use of survey evidence in in trademark cases. So definitely depends on where you’re located in the country before which courts in terms of their kind of animating judicial philosophy in terms of whether survey evidence is appropriate in a given case.

Schmidt

So I’m gathering from this and I’ve had some involved in it with consumer surveys and the consumer class action contacts sponsored by or undertaken by an expert, but there are certain flaws that are recurring in consumer surveys that will end up coming back that maybe result in it being sidelined. What are some of the common flaws that will just leave it with Federal District Court Judges sometimes find and consumer surveys that result in the survey not being admitted into evidence?

Keyes

Yeah. So I’d say probably one of the most common issues that we see is that the survey fails to replicate marketplace conditions, and what I mean by that is when we’re conducting survey research to be used in litigation, we’re really trying to use that as a proxy for getting into the consumer’s minds, right? We can’t necessarily bring every single consumer before the court and ask them, are you confused by this particular trademark or deceived by this particular advertisement? So what we’re trying to do with survey evidence is trying to replicate the marketplace conditions in which consumers are exposed to a particular trademark or particular piece of advertising collateral, and so oftentimes where I think lawyers and other experts get hung up is developing a survey methodology. It really doesn’t replicate the marketplace. Now you don’t have to be exacting in your approach in terms of replicating the marketplace, but under the circumstances, it needs to be reasonable and I think we’ve seen in a number of cases and continue to see cases where that replication just isn’t there.

Schmidt

So just to put a finer point on that, there are surveys that perhaps are exist apart from the litigation controversy, but what we’re talking about here more often than not is commissioning a survey with specific questions and a specific pool of participants, and then taking that group through that series of questions, examining a product and so forth, and in those cases It’s not exactly neutral. It’s a sort of going back to politics. It’s sort of like when a candidate runs their own internal polls in order perhaps to goose up their numbers. It’s not exactly a neutral, non interest objective approach. Is it sometimes the case that the temptation to put the thumb on the scale in the survey comes through and leading questions, biased approaches, including perhaps even in the pool of participants?

Keyes

Yeah, I think that’s right. I mean, we’ve certainly seen instances where you, you mentioned leading questions. I think that’s an issue that we see a lot of and sometimes, leading questions you know at one level don’t look as leading as they might be, but upon reflection they really are quite leading and biased and leading and biased questions really do affect the legitimacy and ultimately potentially the admissibility of a survey. So yes, that does happen, but the challenge for trial counsel and their experts is to really play it straight and to make sure that the survey evidence is being developed in a way that is consistent with industry standards and principles.

Schmidt

How often is it that you have competing consumer surveys and with no real explanation, like the plaintiff has come forward with the survey that basically says everyone was consumed by this trademark. Everyone was deceived by this trademark or the market was confused and the defendant says something, the exact opposite and now you have to figure out how to reconcile this. Or do you just put that, put both surveys to the jury and let the jury decide that?

Keyes

Well, that happens probably in just about every single case that I’ve ever been involved in, where there’s survey evidence, you typically have one party that develops survey evidence and then the other party develops a counter set of surveys to offset or otherwise critique the initial parties survey offering so that does happen a lot and much like other battles of the experts in other contexts with damages and other expert related endeavors. It’s the same with surveys oftentimes that you’re going to have one party that conducts a survey certain way the other party conducts it in a different way with different results, and it’s ultimately for the Fact Finder to decide.

Schmidt

I gather from this discussion on surveys and all that goes into it, both the expertise and just the logistics of getting the participants together, this is not inexpensive undertaking in the course of litigation, is that right?

Keyes

No, it’s not. It tends to be one of the more significant costs associated with the litigation, you know, experts can be expensive and you’re right, the data collection and gathering process can be quite expensive depending on what’s at issue and who are the survey respondents that you need to track down and interview. If it’s a case involving, you know, a consumer product, pizza, soft drink, something like that. Well, consumers of those products are very easy to track down. But if it’s a more nuanced or more complicated case, say involving medical research, and you need to survey physicians, well, that can be more time consuming. It can be more difficult to find them and of course, it’s going to be more expensive. So yeah, the cost can be very significant.

Schmidt

But like anything that we do in litigation, there’s a cost benefit analysis, right? We’re going to spend a lot of money, but this is the benefit we’re going to get. Can you speak to the benefit the outcome in the case, how often do you see these consumer surveys significantly impacting the outcome of a case and at what juncture is it at trial or is it earlier in the process?

Keyes

It’s both. I think we’ve seen survey evidence that’s been developed that can be really powerful at the summary judgment stage, so it can sometimes be dispositive for or one of the issues that’s potentially dispositive at that stage and of course at trial, it’s very important powerful piece of evidence that can sway the Fact Finder.

Schmidt

And I could also just, you know, attest to the fact that not a very small subset of even talented commercial litigators have dealt with consumer surveys. That’s not something that comes up in every case. It’s not something that we just deal with like we deal with depositions and document productions that are just sort of variably part of commercial litigation. So your area of expertise and the things you’ve done to advance your knowledge and it is pretty critical. Is it for the right cases is irreplaceable. Let me turn to talking about that a bit more in terms of your career and the future of your legal practice, and just in general litigation and the and the litigation landscape going forward. How do you see your building and expanding expertise and focus on consumer surveys impacting your practice?

Keyes

Yes. So I mean I am, as I mentioned, trademark and advertising trial lawyer and of course work with survey experts to develop surveys in those sorts of cases. But, I’m also now starting to act as a consulting and testifying a survey expert myself, which has been a lot of fun. I think I have a somewhat unique value proposition, Kent, in that I’ve been trying these cases for, you know, the last 20 plus years. and so I have a lot of experience just in terms of how it works from the trial side and having worked with a lot of great survey experts and now having gone through a lot of formal education, with respect to developing survey research, I think that adds kind of a unique aspect to my expertise.

Schmidt

Well, that must be a huge adjustment and you know, last year I sat for a deposition for the very first time in my entire life. It was an expert witness deposition. It was just being a witness in a case after taking and defending well over 100 depositions in my career and it was a very interesting adjustment. I’m sure it’s that in spades all of a sudden becoming an expert witness as opposed to being a litigator. How do you manage that adjustment?

Keyes

Well, you know we spoke a few moments ago about survey evidence that’s developed as both the plaintiff and the defendant. You look at the case through kind of different prisms. I think it’s the exact same concept in developing survey evidence that you’re going to have to stand behind. So it’s really been interesting and I’m having a lot of fun doing it.

Schmidt

I imagine so. Well, kudos for you for this new adventure. One other area that I could see survey evidence being helpful, perhaps not in a way that it would be immediately apparent, but you have a client, for example, that thinks, oh, there’s no consumer deception here or there’s market confusion at all and then you take a consumer survey and before you even produce it, you say, you know, this is not what the survey came back with. Maybe we do want to think about settlement. Does it does a survey evidence ever cause a reevaluation of the case with the client, because we know that the other side is probably going to be able to replicate that as well?

Keyes

Absolutely yes, it can be a very important tool to use at the outset of a case, you know, and we’ve actually used it for that exact purpose, Kent, that we do some internal testing to get a sense for what the market’s telling us. But, we’ve also used survey evidence offensively to or I guess I should say defensively, if we’ve received, for example a draft class action complaint. We’d go out and test to see if there’s actual deception or potential deception in the marketplace, and if shared, those initial pilot survey results with opposing counsel and that’s led to some really favorable outcomes at the outset of the case. So yeah, survey evidence is something that can be developed ultimately for later on, but it can also be developed at the outset of the case that can be really help.

Schmidt

Mike, we’ve talked a lot about consumer surveys in pending litigation, which is of course a focus of this podcast, and we do a lot of discussion on litigation strategy in managing litigation. But as you know, I’m also interested in all of the creative ways that we can work with clients to identify risk and to try to determine the extent to which in this instance, for example a product or an advertisement may result in a legal claim from either a competitor or from a consumer. Is there any application to consumer surveys in that free litigation context that you’ve found useful?

Keyes

Yeah, I can think of a couple of examples. Trademark context. If you’re getting ready to launch a new product using a particular trademark and there’s a concern or a potential concern that a claim of infringement could be made with respect to the new product name. Sometimes companies will want to test that and see is there any legitimacy or validity to such claim and sometimes that survey research can be useful and can calm some nerves, and sometimes it can exacerbate the problem, so either way it provides insights for the company to be able to make some final decisions. Within the context of hopefully avoiding litigation, I think I mentioned a moment ago about another instance where survey research can be developed and used essentially as a defense to a claim at the outset of a case or even before a case is filed. For example, I’ve been involved in a number of instances where a client has been served with a draft complaint and this typically occurs in the context of a class action claim where the client is served with the draft complaint and the class action lawyer would like to negotiate a settlement. Well, we’ll oftentimes go out and conduct some quick research to see if in fact there’s any validity to these claims, and oftentimes it has come back with favorable evidence in that regard and we have shared that with plaintiff’s counsel and resolved the case favorably.

Schmidt

So I want to ask you a follow-up question related to that and that relates to privilege and attorney work product. If that consumer survey was done just out of curiosity by the marketing team, it’s not going to be protected from later discovery, but if it’s done under the auspices of the lawyer in that direction of lawyer, whether in house or outside it has that protection. Is that a consideration also for how and when to do that defensive type survey work? 

Keyes

No, that’s, you’re exactly right, Kent. I mean if this is. If the marketing department in the normal course just goes out and conducts market research well, that generally is going to be discoverable. But if it’s done under the direction of counsel for the purpose of assessing litigation risks or avoiding litigation, then certainly that’s going to be attorney work product that would not be discovered.

Schmidt

Well, this is all fascinating and I’m growing in my understanding, appreciation of survey evidence, and I look forward to more opportunities to work together with you and you can be sure that you’re my first call when I get a case that calls for a survey evidence in one form or another.

Keyes

Well, it’s always a pleasure working with you, Kent.

Schmidt

So that’s about all the time we have to discuss consumer surveys as fascinating as this topic is, and as long as we could go on about it, what I’d like to do at this point in our show is learn more about you as an individual. We sometimes call this the deeper dive segment and I’d like to ask you, Mike, about your background as a musician. So I’ve been at a number of partner retreats and firm events over the years and one thing that is almost as certain to happen at those events… 

Keyes

Uh oh.

Schmidt

…as any, is that sometime after dinner, after we’ve all had a good time and people are sort of making small talk, we’ll hear the piano playing out in the lobby or out in a common area, and you will be at the keyboard playing some Billy Joel or Neil Diamond or something else and a bunch of lawyers, maybe with an adult beverage in hand to fortify their courage will pretend that they know how to sing and sing with reckless abandon, and it’s always a sweet tradition to see you at the keyboard. But I don’t think I’ve ever really learned about your music career, how it started, and what part of your life, other than playing at Dorsey partner retreats music plays in your life, so can you tell us about that?

Keyes

Yeah. Well, we have had a lot of fun at partner retreats. You know, it’s actually the highlight of the year from my perspective, always great to see everybody and it’s such a fun loving group of lawyers that we have here at the firm and so yeah, we have had a lot of fun late nights playing the piano and singing. So, I actually have a degree in piano performance. That was my undergraduate degree and I always knew I wanted to go to law school, but I really liked music too, so I studied it through college and then went and studied for a year in the former Soviet Republic of Georgia at a very renowned Conservatory there, studied for a year and had a great time, but I think I realized when I was there that maybe it’s time for me to go to law school as great a time as I had there, it was time for me to kind of look  elsewhere.

Schmidt

Well, that’s interesting and in terms of your day-to-day life, how much how much piano playing do you do just as a form of relaxation on the weekends or in the evenings now?

Keyes

I mean, it really varies. It used to be when our kids were younger, they all studied music and so I would work with all of them frequently and my daughters, they both play violin. So the three of us would play at various events and in other locales. So I don’t do that as much these days but I still try to sneak it in when I can.

Schmidt

Does the fact that you do a lot of IP litigation have any connection to the music of, you know, copyright or entertainment law? Is that is that a, was that a segue into IP litigation?

Keyes

Yeah, it absolutely was. So I was really interested in still am in copyright as a result of my music background. So when I first graduated from law school, I was really into trademark or copyright cases. That was a natural segue into trademark and advertising litigation. So yeah, that’s exactly how I got into it.

Schmidt

Well, turning back momentarily to survey research and data analysis, what is the one thing you’d like our listening audience to take away from our discussion today about survey evidence in litigation?

Keyes

Survey evidence can be a very important piece of the litigation puzzle, but it’s also complicated. It calls for special expertise, and it really calls for trial Council to work closely with the survey expert to make sure that evidence is being developed according to generally accepted principles that it’s going to be useful for an issue in the case and ultimately admissible.

Schmidt

Thanks again for being on Shark cast, Mike. Very interesting conversation and look forward to maybe having you back as a guest again sometime in the future. With that, I’d like to thank all of our audience for being here today and for listening. As always, I’m very indebted to the extraordinary team at Dorsey for making this podcast and episode possible for more resources on this and other litigation risk go to litigationrisk.com where more information can be found, including a book on managing litigation risk written by yours truly. Until next time my friends, this is yet another reminder that there are a lot of sharks swimming out there in the murky water, so swim safely.

Voiceover

This podcast is not legal advice and does not establish an attorney-client relationship or create any duty of Dorsey and Whitney LLP for those appearing in this podcast to anyone. Although we try to assure that the content of this podcast is accurate, comprehensive and reflects current legal developments, we do not warrant or guarantee those things. The opinions expressed in this podcast are the opinions of those appearing in the podcast only and not those of Dorsey and Whitney. This podcast is considered attorney advertising under the applicable rules of certain states.