WEBVTT

00:00:00.017 --> 00:00:04.637
Suppose you're designing something new that helps people accomplish a goal.

00:00:04.837 --> 00:00:11.877
Maybe it's a website that sells a product or a procedure someone follows to deliver a service.

00:00:12.097 --> 00:00:16.037
Or maybe it's an instructional curriculum or a mobile app.

00:00:16.217 --> 00:00:20.777
And let's say you want to know whether it really works, if it's clear,

00:00:21.057 --> 00:00:24.857
usable, and actually helpful. So you ask people.

00:00:25.137 --> 00:00:30.997
Seems logical, right? But what if I told you that people often cannot accurately

00:00:30.997 --> 00:00:33.717
explain why they do what they do?

00:00:34.097 --> 00:00:39.437
Today we'll be getting into the idea of self-report data, the stuff people tell

00:00:39.437 --> 00:00:43.237
us when we ask them about their choices, actions, and feelings.

00:00:43.637 --> 00:00:49.257
In this episode, we'll unravel one of the biggest challenges in design research.

00:00:49.517 --> 00:00:53.997
Can we trust what users say about their own experiences?

00:00:54.977 --> 00:01:00.377
For well over a century, psychologists have tried to understand what's really

00:01:00.377 --> 00:01:05.497
going on inside people's heads, how they think, decide, and behave.

00:01:05.977 --> 00:01:11.957
Brain scans, a relatively new-ish technology, offer a peek inside the brain,

00:01:12.097 --> 00:01:15.557
but they're expensive, complex, they're hard to access.

00:01:16.077 --> 00:01:23.117
Other methods exist, like lab-based experiments that use reaction times or priming

00:01:23.117 --> 00:01:25.697
to infer what people are thinking,

00:01:25.817 --> 00:01:31.377
but they're not always practical to use in the real world and applied research settings.

00:01:31.697 --> 00:01:34.657
The easiest option is just ask.

00:01:34.917 --> 00:01:41.097
That's the self-report method. But while it's simple, it's also notoriously unreliable.

00:01:41.317 --> 00:01:46.577
People often misremember, they rationalize and they invent things,

00:01:46.597 --> 00:01:49.897
or sometimes they just don't know why they do what they do.

00:01:50.277 --> 00:01:56.217
In today's episode, we're exploring when self-report data can give us useful

00:01:56.217 --> 00:02:01.617
insights versus when it can seriously mislead our design process.

00:02:01.917 --> 00:02:06.117
So I'd like you to ruminate on a few questions throughout this episode.

00:02:06.377 --> 00:02:11.597
Why do people misjudge their own behavior? And what's the difference between

00:02:11.597 --> 00:02:17.477
asking the user of a product what they think versus observing what they do?

00:02:17.477 --> 00:02:22.477
In this episode, I'll break down the research on self-report data and I'll discuss

00:02:22.477 --> 00:02:28.657
techniques that can help us conduct user research that gives real insight into

00:02:28.657 --> 00:02:30.597
what drives people's behavior.

00:02:31.565 --> 00:02:36.325
Let's start our discussion with introspection. I want you to imagine this.

00:02:36.545 --> 00:02:43.005
In the quiet of Wilhelm Wuntz' laboratory, Heinrich sits patiently at a wooden table.

00:02:43.285 --> 00:02:48.805
Beside him, a research assistant stands holding the first of three objects,

00:02:49.005 --> 00:02:52.005
ready to hand them to Heinrich one by one.

00:02:52.465 --> 00:02:58.605
Research assistant carefully places a first object, a rock, into Heinrich's hand.

00:02:58.905 --> 00:03:06.265
Heinrich grips it gently, And in his trained response, he begins a methodical introspective report.

00:03:07.245 --> 00:03:11.965
It's heavier than I expected, he says. The weight presses down immediately into

00:03:11.965 --> 00:03:14.725
my palm, creating the sensation of density.

00:03:14.985 --> 00:03:18.605
I can feel the rock's mass pulling slightly as I lift it.

00:03:19.305 --> 00:03:23.445
Wundt listens intently, and his assistant jots down notes with precision.

00:03:24.285 --> 00:03:27.865
Heinrich continues, his attention now focused on the texture.

00:03:28.625 --> 00:03:32.485
The surface is uneven, rough in some places, and smoother in others.

00:03:33.105 --> 00:03:38.365
There's a prominent groove here along the side. He traces his finger over it,

00:03:38.645 --> 00:03:40.985
describing the sensation as he experiences it.

00:03:41.205 --> 00:03:43.905
The groove feels deeper than the rest of the surface.

00:03:44.285 --> 00:03:49.125
It's like it's been cut away or worn away. The rock was cool in my hand,

00:03:49.225 --> 00:03:50.545
but it's slightly warming now.

00:03:50.905 --> 00:03:56.425
He places the rock on the table, and the assistant hands him a second rock, a larger piece.

00:03:57.245 --> 00:04:01.165
Heinrich takes it and immediately notices the difference. This one is larger,

00:04:01.405 --> 00:04:03.885
but it's lighter than the first, he reports.

00:04:04.465 --> 00:04:08.385
It feels hollow or less dense than I expected for its size.

00:04:08.565 --> 00:04:13.325
There's no strain in holding it. I can feel a distinct lightness compared to the previous rock.

00:04:13.785 --> 00:04:18.185
His fingers begin exploring the surface again. The texture is much rougher.

00:04:18.505 --> 00:04:21.845
There's a sharp ridge along the edge. It's much more jagged.

00:04:22.345 --> 00:04:26.765
He turns the rock around in his hand. It's colder than the first rock And it's

00:04:26.765 --> 00:04:31.785
staying cold longer After finishing with the second rock Heinrich sets it down

00:04:31.785 --> 00:04:33.265
And the assistant hands him a third.

00:04:34.140 --> 00:04:38.500
Now, this story was a dramatized version of what a laboratory scene might have

00:04:38.500 --> 00:04:44.780
been like for Wilhelm Wundt, the father of psychology in Leipzig, Germany, in the 1860s.

00:04:44.940 --> 00:04:49.540
If that session didn't sound terribly scientific to you, then you're right.

00:04:49.840 --> 00:04:51.820
Introspection was riddled with problems.

00:04:52.260 --> 00:04:54.040
But we had to start somewhere.

00:04:54.760 --> 00:04:59.380
Psychology has this challenge of trying to study something that is unseen.

00:04:59.760 --> 00:05:03.980
While technically it is the study of behavior which can be seen,

00:05:04.140 --> 00:05:08.660
and that's why the behaviorist movement of the early 1900s wanted to focus on

00:05:08.660 --> 00:05:10.560
behavior to be more scientific,

00:05:11.220 --> 00:05:16.520
in psychology we're still trying to theorize or conjecture about what's going

00:05:16.520 --> 00:05:20.720
on inside the proverbial black box of the human mind.

00:05:21.120 --> 00:05:25.540
So for example, we can see the inputs going in the black box,

00:05:25.660 --> 00:05:30.180
that's the stimuli, and we can see the outputs that came out of the black box,

00:05:30.360 --> 00:05:31.120
and that's the behavior,

00:05:31.480 --> 00:05:35.660
but we can't see what's inside the black box, which is everything in between

00:05:35.660 --> 00:05:36.940
the input and the output.

00:05:37.720 --> 00:05:41.220
Nowadays, we've come a lot closer to being able to see the black box.

00:05:41.300 --> 00:05:46.080
For example, we have fMRI machines and other brain imaging techniques where

00:05:46.080 --> 00:05:50.620
we can see brain activity going on inside the black box in real time.

00:05:51.080 --> 00:05:55.160
For years, we didn't have that. And even though we do have that,

00:05:55.300 --> 00:05:56.920
it still doesn't tell us everything.

00:05:57.140 --> 00:06:02.440
For example, if you're in an experiment and the experimenter gives you a complicated

00:06:02.440 --> 00:06:08.000
math problem to solve while you're in an fMRI machine, we might see images of

00:06:08.000 --> 00:06:10.140
your prefrontal cortex lighting up.

00:06:10.220 --> 00:06:13.440
That means it's being activated while you're trying to solve that problem,

00:06:13.440 --> 00:06:17.900
but we still can't see what the neurons are doing on the neural level.

00:06:18.200 --> 00:06:22.940
For example, are they computing a mean or are they comparing slopes?

00:06:23.200 --> 00:06:26.040
What are they doing? It's still somewhat of a black box.

00:06:27.001 --> 00:06:31.441
So we're stuck with clever laboratory techniques most of the time in order to

00:06:31.441 --> 00:06:33.381
understand the human mind.

00:06:33.941 --> 00:06:39.501
In fact, the history of the psychology lab could be described as a long series

00:06:39.501 --> 00:06:46.001
of scientists inventing clever ways of drawing conclusions about something that's invisible,

00:06:46.261 --> 00:06:52.221
human cognition or the mind, by observing something that is visible, behavior.

00:06:52.581 --> 00:06:56.101
So it's difficult to understand what's inside the black box.

00:06:56.101 --> 00:07:01.841
It also turns out that there's this attitude versus behavior problem that makes

00:07:01.841 --> 00:07:05.421
it ineffective to ask people what's going on inside their minds.

00:07:05.801 --> 00:07:11.521
For example, sociologists started discovering in the 1930s that giving people

00:07:11.521 --> 00:07:15.881
opinions on surveys didn't necessarily match their behavior.

00:07:16.721 --> 00:07:20.941
Leon Fessinger developed the theory of cognitive dissonance,

00:07:21.081 --> 00:07:26.361
which is a tension between attitude and behavior that a person attempts to reconcile.

00:07:26.981 --> 00:07:31.761
So attitude and behavior is not the same thing, and it's usually attitude that

00:07:31.761 --> 00:07:38.021
you get when you ask someone how they feel, rather than an accurate description of their behavior.

00:07:38.601 --> 00:07:43.341
And then there's the fact that it's just generally difficult for people to be

00:07:43.341 --> 00:07:47.561
able to articulate their true reasons for doing things.

00:07:48.061 --> 00:07:54.121
There was a famous study in the 1970s where social psychologists Richard Nisbet

00:07:54.121 --> 00:08:00.161
and Timothy Wilson demonstrated that people can be very bad at explaining why

00:08:00.161 --> 00:08:01.481
they made certain decisions.

00:08:01.941 --> 00:08:08.061
In this experiment, participants sat down and had several pairs of stockings placed in front of them.

00:08:08.281 --> 00:08:13.681
The task was to choose the best pair. people would pick them up touch them compare

00:08:13.681 --> 00:08:18.881
them and they'd make a choice and say something like oh I picked these these are way better,

00:08:19.673 --> 00:08:24.273
When asked why, people confidently reported all kinds of different reasons.

00:08:24.433 --> 00:08:27.133
They might have talked about the softness or the texture.

00:08:27.393 --> 00:08:30.193
They might have even talked about the quality of the stitching.

00:08:30.813 --> 00:08:34.093
The only problem was that all the stockings were identical.

00:08:35.133 --> 00:08:39.033
People unknowingly made up reasons for picking one of the two pairs.

00:08:39.433 --> 00:08:42.293
And they also tended to pick the pair that was on the right.

00:08:42.473 --> 00:08:47.693
This was an effect that Nisbet and Wilson ended up calling right-hand bias.

00:08:48.373 --> 00:08:53.753
Nisbet and Wilson concluded that there are certain high-level cognitive processes

00:08:53.753 --> 00:08:58.353
that we just don't have access to when trying to describe what we're thinking

00:08:58.353 --> 00:09:00.113
and why we're doing what we're doing.

00:09:00.713 --> 00:09:05.753
So, returning to our question of introspection, we walked through our first

00:09:05.753 --> 00:09:11.413
story about Wundt in his laboratory doing introspection in the late 1860s.

00:09:11.433 --> 00:09:17.233
But let's take a look at what happened 100 years later in a laboratory on the other side of the world.

00:09:17.693 --> 00:09:23.513
In the late 1960s, inside a cognitive psychology laboratory at Carnegie Mellon,

00:09:24.413 --> 00:09:29.053
Alan Newell and Herbert Simon sat at a table with a research participant.

00:09:29.053 --> 00:09:35.333
This experiment was intended to offer new insights into human problem-solving.

00:09:36.473 --> 00:09:40.973
Newell and Simon were pioneers in artificial intelligence, and they were very

00:09:40.973 --> 00:09:48.153
interested in the logical steps and the elements involved in the problem-solving process.

00:09:48.633 --> 00:09:54.833
In front of the participant were three wooden pegs and a stack of circular disks on the first peg.

00:09:55.153 --> 00:09:59.833
This is the Tower of Hanoi problem. It's a classic problem-solving test that

00:09:59.833 --> 00:10:04.833
requires moving the disks from the first peg to the third, and you've got to

00:10:04.833 --> 00:10:06.253
follow a strict set of rules.

00:10:06.613 --> 00:10:11.873
You can only move one disk at a time, and you can't put a larger disk on top of a smaller one.

00:10:12.985 --> 00:10:19.225
A tape recorder was nearby, ready to capture every moment of the participant's verbalizations.

00:10:20.265 --> 00:10:24.045
Newell presses the record button and instructs the participant by saying,

00:10:24.265 --> 00:10:27.465
tell us everything you're thinking as you work through the problem,

00:10:27.685 --> 00:10:29.765
and he signals the start of the session.

00:10:30.225 --> 00:10:34.965
The participant first stares at the Tower of Hanoi puzzle, thinking out loud,

00:10:35.245 --> 00:10:39.245
I need to move the largest disk, but it's blocked by the smaller ones.

00:10:40.185 --> 00:10:44.325
Newell and Simon listened for the first sign of strategy to be verbalized.

00:10:44.565 --> 00:10:46.425
This was the heart of their research,

00:10:46.605 --> 00:10:51.505
using verbal protocols to capture cognitive processes in real time.

00:10:51.685 --> 00:10:55.825
As the participant moved the first small disk and continued to speak,

00:10:56.025 --> 00:10:58.685
his thoughts flowed as an uninterrupted stream.

00:10:58.985 --> 00:11:04.265
He says, I'll start by moving the smallest disk to the second peg so I can free

00:11:04.265 --> 00:11:10.545
up space to move the others, then carefully picking up the small piece to put it on the middle peg.

00:11:10.725 --> 00:11:17.465
He continues, now I can move the second smallest one to the third peg. And he continues.

00:11:18.065 --> 00:11:22.825
Now, as you listen to those two different scenes, the introspection scene from

00:11:22.825 --> 00:11:29.785
the 1860s and the verbal protocol scene from the 1960s, they might seem awfully similar.

00:11:30.125 --> 00:11:34.825
They both involve a person talking as they do something and the research is

00:11:34.825 --> 00:11:37.945
recording their words and using their words as data.

00:11:38.205 --> 00:11:43.685
So why is it that one of them is considered a long-debunked methodology while

00:11:43.685 --> 00:11:46.885
the other one is embraced as a valid research technique?

00:11:47.385 --> 00:11:49.665
There's a few differences to keep in mind.

00:11:50.551 --> 00:11:56.851
The first one is that introspection was making some big claims that the verbal protocol is not.

00:11:57.431 --> 00:12:03.191
Introspection was claiming to identify core elements of the human mind,

00:12:03.371 --> 00:12:08.131
kind of in the same way that physics can identify elements and build them up in pieces.

00:12:08.131 --> 00:12:11.171
That's what the structuralist movement that's the

00:12:11.171 --> 00:12:13.931
movement that was using introspection that's what

00:12:13.931 --> 00:12:17.511
they were trying to do by picking apart

00:12:17.511 --> 00:12:20.551
the human mind into little elements and then

00:12:20.551 --> 00:12:27.131
building it back up into higher level cognitive constructs verbal protocol isn't

00:12:27.131 --> 00:12:31.551
trying to do that in verbal protocol you're making a much more modest claim

00:12:31.551 --> 00:12:36.911
you're simply saying that as somebody is doing this particular behavior that

00:12:36.911 --> 00:12:39.711
I'm observing and that's the main piece of data,

00:12:39.711 --> 00:12:46.271
I also have an insight into maybe what step they're on or whether or not they're confused.

00:12:46.991 --> 00:12:52.211
So in introspection, the words are the main data and they're presumed to have

00:12:52.211 --> 00:12:56.951
a direct and strong attachment to real things that are going on inside the mind.

00:12:57.231 --> 00:12:59.671
And it was a very unreliable method.

00:12:59.971 --> 00:13:01.991
Different labs got very different results.

00:13:02.431 --> 00:13:06.871
When it comes to self-report data, you've got to be careful.

00:13:07.191 --> 00:13:13.571
Popular business activities like focus groups and customer feedback can be very

00:13:13.571 --> 00:13:17.091
misleading if they're used in the wrong way, and they often are.

00:13:17.331 --> 00:13:22.051
There are more reliable methods of obtaining self-report data.

00:13:22.051 --> 00:13:27.751
Consider things like well-constructed questionnaires and in-depth interviews.

00:13:28.391 --> 00:13:33.991
However, these should be conducted by people who are knowledgeable and skillful

00:13:33.991 --> 00:13:36.211
about how to use those methods.

00:13:36.511 --> 00:13:41.831
You want good data that reflects reality because it was gathered with methods

00:13:41.831 --> 00:13:45.831
that have built-in safeguards against human bias.

00:13:46.511 --> 00:13:51.731
Bias comes from all directions. So when a interviewer is asking questions,

00:13:52.071 --> 00:13:54.051
they can have bias in the way they're asking.

00:13:54.751 --> 00:13:58.671
And when someone's responding, they can also respond with bias,

00:13:58.671 --> 00:14:05.611
and it takes an expert to recognize that they're responding with some kind of a bias and tease it out.

00:14:06.091 --> 00:14:12.751
Now, how do we ensure that the user feedback we gather is both accurate and

00:14:12.751 --> 00:14:15.751
useful to our design process?

00:14:16.351 --> 00:14:22.791
One strategy is to pair self-report data with observable behavioral data.

00:14:22.791 --> 00:14:26.191
Imagine watching someone browse a shopping website.

00:14:26.471 --> 00:14:30.191
If they suddenly stop scrolling, that's a behavioral cue.

00:14:30.671 --> 00:14:35.211
Ask them what's going on and you'll get their perspective, but you cross-check

00:14:35.211 --> 00:14:37.091
it with an observable action.

00:14:37.331 --> 00:14:42.671
That's how you bridge the gap between what users say and what they actually do.

00:14:44.097 --> 00:14:50.337
To hone the skill of interpreting self-report data, there's some good readings and ways to practice.

00:14:50.357 --> 00:14:55.697
I recommend investing a little time in learning qualitative research methods.

00:14:55.937 --> 00:15:01.557
For example, a book called Observing the User Experience by Mike Kuniofsky.

00:15:01.917 --> 00:15:07.677
Or check out the book Time to Listen by Indie Young, who I interviewed for this

00:15:07.677 --> 00:15:09.497
season of The Design Psychologist.

00:15:09.497 --> 00:15:16.317
She fascinatingly redefined qualitative research and she proposes a method for

00:15:16.317 --> 00:15:21.717
extracting concepts from people's minds in a way that's more effective and empathetic.

00:15:22.657 --> 00:15:26.877
So remember that self-report data is far from perfect.

00:15:27.077 --> 00:15:32.697
It has its limitations, but self-reporting can be valuable if you have skilled

00:15:32.697 --> 00:15:40.377
researchers guiding the process and if you combine that self-report insight with real behavior.

00:15:41.237 --> 00:15:48.997
The key is to refine your methods, watch for bias, and validate findings with multiple data sources.

00:15:49.877 --> 00:15:55.757
In the end, self-report data is a tool, imperfect but powerful when we use it with care.

00:15:56.117 --> 00:16:00.837
It reminds us that understanding people is never as simple as just asking.

00:16:01.197 --> 00:16:07.937
Our ability to articulate what we need is often limited, but great design shouldn't be.

00:16:08.097 --> 00:16:15.177
It takes nuance, empathy, and skill to uncover what people truly think, feel, and do.

00:16:15.917 --> 00:16:20.617
That's a challenge of design psychology, seeing the full human picture.

00:16:20.957 --> 00:16:26.557
So don't just take answers at face value because often the quality of what we

00:16:26.557 --> 00:16:30.837
design is tied to the depth of what we understand about people.

00:16:31.137 --> 00:16:36.337
The more clearly we see the people we're designing for, the better we can shape

00:16:36.337 --> 00:16:38.397
experiences that truly resonate.