Episode 20

20 - Is Your Sample Representative? Exploring Inclusivity

Published on: 12th July, 2022

In this week's episode, our host Lenny Murphy, is joined by three individuals leading the industry conversation around inclusivity: Tim Cornelius, Director of Operations at QuestionPro and CEO at P3 Technology; Keyona Osborne, Senior Research Manager at Accelerate (a SmithGeiger company); and Lilah Raynor, CEO at Logica Research.

Join us while Lenny explores with these three industry professionals the importance of representation and the challenges we have in the industry with ensuring that our samples are inclusive.

In this episode, we'll explore:

  • The importance of accessible content and accessible surveys in working towards equitable representation
  • How opening up the door to recruitment criteria allows for a more natural diversity in research participation
  • Ways considerations for race, ethnicity, culture, and socioeconomic status might factor into research outcomes
  • The business case for prioritizing inclusivity and accessibility in research design

Many thanks to Lilah, Keyona, and Tim, for being our guests. Thanks also to our producer, Karen Lynch, our editor, James Carlisle, and our sponsor, InnovateMR.

Producer's Note:

There were two events referenced in this episode that are "out of order" as there is a gap between our recording and launch dates:

  1. The IA presentation Keyona mentioned actually took place on June 3, 2022; you can access information about that here.
  2. The recording for the QuestionPro X day on June 6, 2022, that Tim mentioned can be accessed here.

Mentioned in this episode:

Join us at an IIEX Event!

Visit greenbook.org/events to learn more about events in Asia, the Americas, and Europe. Use code PODCAST for 20% off general admission at all upcoming events.

Transcript
Lenny:

Innovate MR is an independent sampling and ResTech company delivering faster answers

Lenny:

from targeted audiences to support agile research.

Lenny:

Innovate MR also develops forward-thinking products, empowering businesses to create

Lenny:

data-driven strategies and identify growth opportunities.

Lenny:

Hello, everybody, it’s Lenny Murphy back with another edition of the GreenBook

Lenny:

Podcast.

Lenny:

Thank you so much for taking time out of your day to share it with us.

Lenny:

And today we’re going to be talking about the topic of inclusivity, and we have three

Lenny:

great folks that are deeply engaged in this conversation.

Lenny:

I will let them introduce themselves as I call them out.

Lenny:

So first, we have Tim Cornelius, a GreenBook list honoree for 2022, Director of Operations

Lenny:

at QuestionPro, and the CEO of P3 Technology.

Lenny:

Tim, you want to tell our listeners a little bit about you, real quick?

Tim:

Hey, Tim Cornelius.

Tim:

As Lenny said, I’m Director of Operations at QuestionPro and the CEO of P3 Technology.

Tim:

I am an advocate for accessibility in market research and believe that no one should be

Tim:

left out.

Tim:

I’m sure I’ll touch on that a little more in the podcast.

Tim:

But really glad to be here.

Tim:

Born New Orleanian and happy to represent QuestionPro and P3 on the podcast.

Lenny:

Thank you, Tim.

Lenny:

Now, I’m surprised; by being a born New Orleanian, I don’t hear that Cajun accent.

Lenny:

Maybe that’ll come out a little bit more as we’re talking.

Lenny:

Next, we have Keyona Osborne, Senior Research Manager at Accelerate, a SmithGeiger company.

Lenny:

Keyona, welcome.

Keyona:

Thank you so much for having me.

Keyona:

I’m excited to be here.

Keyona:

As Lenny said, my name is Keyona Osborn-Pannell, and I have been in the market research industry

Keyona:

for about ten years now.

Keyona:

And it’s not until I got into the industry that I would say I became passionate about

Keyona:

and an advocate for all things diversity, equity, inclusion, accessibility, belonging.

Keyona:

And so, that is something that I am so passionate about, and would love to help the market research

Keyona:

industry continue to change and lead in that space.

Keyona:

I am actually not a native Southerner but living in Charlotte, North Carolina, right

Keyona:

now, originally from New Jersey.

Keyona:

And looking forward to this conversation.

Lenny:

Thank you.

Lenny:

Welcome.

Lenny:

Good to have another two southerners—

Keyona:

[laugh].

Lenny:

—on the podcast.

Lenny:

That’s rare.

Lenny:

[laugh].

Lenny:

And then Lilah Reiner, CEO of Logica Research.

Lenny:

Lilah, great to have you on.

Lenny:

Tell us a little bit about your background.

Lilah:

Thanks so much, Lenny.

Lilah:

I’m super happy to be here.

Lilah:

I have had Logica Research for about 15 years, and prior to that, I managed research at Charles

Lilah:

Schwab.

Lilah:

And starting last year, I became involved in the Insights Association IDEA Council,

Lilah:

and specifically around doing research on inclusivity.

Lilah:

And I came to that just from my own background, and also our focus on financial inclusion

Lilah:

at Logica Research and having financial wellness for all.

Lilah:

So, excited to talk about some of the work the IDEA Council has done around asking more

Lilah:

inclusive questions in marketing research and helping bring all voices to marketing

Lilah:

research.

Lenny:

Thank you.

Lenny:

Now, for our listeners, I want to kind of set the tone here from kind of the macro research

Lenny:

perspective.

Lenny:

When I think about the topic of inclusivity, what I think about is representativity.

Lenny:

And it is foundational to successful research, right?

Lenny:

If we don’t have representative samples, if we’re not reaching the populations that

Lenny:

we need to reach, for whatever reason, then we have a problem.

Lenny:

And I would venture to say that we have a problem within the industry right now that’s

Lenny:

driven by multiple issues.

Lenny:

Some are age, some are economics, some are cultural, some are racial, but I think that

Lenny:

we have a real challenge right now with ensuring that our samples are inclusive and representative

Lenny:

along many, many different factors.

Lenny:

And I think we’ve had it for a long time, but I think it’s getting worse.

Lenny:

And we’re seeing that reflected in the supply problems within sample.

Lenny:

We’re seeing it represented in bias that is now becoming more apparent in some aspects

Lenny:

of research.

Lenny:

And it’s a systemic issue.

Lenny:

And by that I don’t mean that I think it’s purposeful; I don’t think it’s anything

Lenny:

like that.

Lenny:

I think we just haven’t evolved in our thinking as an industry to ensure that we’re engaging

Lenny:

with everybody the way they need to be engaged with versus the way we think they should be

Lenny:

engaged with.

Lenny:

We get so operationally focused.

Lenny:

And, you know, we’re making money, that makes sense; we have to create efficiencies

Lenny:

in the process, but I think we’re leaving a lot on the table, and it’s starting to

Lenny:

be a challenge.

Lenny:

So, that’s my Uber perspective.

Lenny:

First, does anybody disagree with that?

Lenny:

Feel free to chime in.

Lenny:

I see.

Lenny:

Everybody’s nodding, going, “Nope, you got it.

Lenny:

You got it.”

Lenny:

So, that’s good.

Lenny:

Now, each of you have an area of expertise in those—or an area of focus within that

Lenny:

broad piece.

Lenny:

So, you know, there’s lots of acronyms covering these initiatives from, you know, DI, or DEI,

Lenny:

DEIB.

Lenny:

Let’s go around and get your unique perspective on what this topic of inclusivity means and

Lenny:

what’s included in it.

Lenny:

So Keyona, why don’t we start with you?

Lenny:

Give us your take.

Keyona:

Sure thing.

Keyona:

I couldn’t agree more with what you were saying [laugh] Lenny, we definitely have a

Keyona:

problem in the market research industry.

Keyona:

I’ve read some articles and studied different leaders, literature in this space, and there

Keyona:

seems to be a divide of do we really need all the letters and do we need to keep adding

Keyona:

[laugh] letters?

Keyona:

Or can we just stick with what was originally there?

Keyona:

I think for me personally, the two letters that are most important to me are equity and

Keyona:

inclusion.

Keyona:

I think equity means that everybody is starting from a place that gives them an equal chance

Keyona:

to reach the finish line, not necessarily the same place, but a place that gives them

Keyona:

an equal chance.

Keyona:

And inclusion makes sure that they’re actually seen and heard and a part of what’s going

Keyona:

on.

Keyona:

I definitely think that because everyone has a different perception of what each letter

Keyona:

means, we’ve needed to add the letter A for accessibility and make sure that we’re

Keyona:

talking about people with different types of abilities, and belonging because even if

Keyona:

you’re included, you may not feel [laugh] like you actually belong in that space.

Keyona:

And so, making sure that ultimately, we have a space that is enjoyable for everyone there

Keyona:

and where everyone feels like they can succeed at a similar pace, or at a similar rate to

Keyona:

everyone else in the room.

Lenny:

Okay.

Lenny:

And, you know, I would translate that, being an old fart in the research industry, that’s

Lenny:

the basis of random probability sampling, right, [laugh] you know, is everybody equally

Lenny:

gets a chance to participate in the research process.

Lenny:

And that’s the gold standard of market research.

Lenny:

So, there’s nothing in that—and I understand that there’s some, you know, cultural sensitivities

Lenny:

to some of these things, or whatever, and all that controversy and don’t care, right?

Lenny:

Put all that aside.

Lenny:

We are—to echo what you were saying, we are talking about the foundation of research,

Lenny:

of good sampling.

Lenny:

Everybody starts from the same place and gets a chance to participate without excluding

Lenny:

them.

Lenny:

So, I see you nodding and—[laugh].

Keyona:

Absolutely.

Keyona:

It seems that the market research industry, all of us need to just go back to the basics

Keyona:

[laugh].

Keyona:

So, I like that you took us there.

Lenny:

Okay.

Lenny:

Yay, I did something good.

Lenny:

All right, now, Tim, so I know you’ve got a slightly different take on this based on

Lenny:

your own focus as you introduced yourself.

Lenny:

So, what would you add to that definition, or share a little bit about your perspective?

Tim:

Sure.

Tim:

So, as a researcher, for years, I was disqualifying people due to disabilities and I did not know

Tim:

that I was doing that.

Tim:

My aha moment came after I had a respondent that I had profiled with about ten demographic

Tim:

qualifications and that respondent was kicked out.

Tim:

Her name was Ariel, she was a mother of two, 36 years old, head of the household, gig worker,

Tim:

checked all the boxes, but she was born deaf.

Tim:

I had not considered that.

Tim:

We had a pilot sitcom video with audio that had no closed captions and no way to go back,

Tim:

no way to interact with the video, so when I put up a red herring question after the

Tim:

video, see if they were paying attention, she literally could not pass that.

Tim:

There’s a very strong chance that she was not going to hit the right answer.

Tim:

So, I chatted with her and it was like, “Hey, you know, you were the perfect participant.

Tim:

What happened here?”

Tim:

And so, I asked her about her life.

Tim:

We chatted all night.

Tim:

She was 15 minutes late to pick up her kids, just really strung all of my heartstrings,

Tim:

and it got me to thinking about, you know, who else is left out of the space.

Tim:

So, the most important letter to me on that is inclusion, and a subset of that is accessibility.

Tim:

I would say that the people that are making the surveys, there’s a large chance that

Tim:

they are not disabled.

Tim:

And if you do not have disability representation, at the very beginning of your ideation phase,

Tim:

you’re not going to be inclusive.

Tim:

The Bureau of Labor Statistics says that only 19.1% of people with a disability have any

Tim:

employment, so you’re likely not going to get any feedback on your survey design or

Tim:

your sampling methodology from someone who has a disability.

Tim:

So, I would say, the most important thing to me, and what I focus and what my north

Tim:

star is accessible content, accessible surveys.

Tim:

Disabled people make up about one in four people in the United States, so let’s start

Tim:

from an equal ground, let’s all start at the same yard line, instead of 75% being out

Tim:

way ahead to start.

Lenny:

I love that too.

Lenny:

And for those that don’t know, I have a neurodegenerative disease that compromises

Lenny:

my mobility to a great extent.

Lenny:

And the thought of not being able to participate in anything, frankly, pisses me off.

Lenny:

And if we get to the point where that’s a challenge to participate in some levels

Lenny:

of research, not that I would qualify because I am in the research industry, that would

Lenny:

frustrate me immensely as well.

Lenny:

So, for our listeners, and for you, I’ve got a personal connection to that idea as

Lenny:

well, right?

Lenny:

There’s nothing wrong with my brain.

Lenny:

There’s just things wrong with my body, right?

Lenny:

And I spend money like a drunken sailor, so you know, everybody should be—[laugh] we

Lenny:

should be paying attention to those things even if there are challenges with our respondents,

Lenny:

in terms of the ability to engage with—it’s a form-factor issue, I think, and we certainly

Lenny:

should be able to solve for that.

Lenny:

Now.

Lenny:

Lilah, you and I’ve worked in the past, some studies around finance and particularly,

Lenny:

we—gosh, what was it, five or six years ago that we did a project on incentives and

Lenny:

the future of money, and what was driving particularly Millennials and Gen Z to engage?

Lenny:

And I know that you’ve done a lot more research around that, so I suspect that your perspective

Lenny:

on this is around economic inclusion and thinking about the different plays that we have to—the

Lenny:

different ways we have to evolve to make sure that we’re representing folks at different

Lenny:

economic levels.

Lenny:

Is that—sorry, I didn’t mean to steal your thunder.

Lenny:

I realized I just [crosstalk]—

Lilah:

[laugh].

Lenny:

So, you tell us instead of me trying to put words in your mouth, please.

Lilah:

Yeah, well, that certainly is a starting point for me.

Lilah:

Since we focus on financial services and my own personal, you know, interest in purposes

Lilah:

around financial inclusion.

Lilah:

And what’s interesting, you know, I would say I do focus on the I. What I like about

Lilah:

the name of the IDEA Council and gives me a chance to hedge, is it stands for Inclusion,

Lilah:

Diversity, Equity, and Access, and I think is an optimistic perspective as well.

Lilah:

But I do focus on inclusion.

Lilah:

But I think to have [laugh] inclusive voices, you need to have the diversity, the equity,

Lilah:

and the access, too, so you can’t really have inclusion without those things.

Lilah:

And it’s really interesting, I’ve been doing a bunch of qualitative interviews, and

Lilah:

we are talking to such a diverse group of people.

Lilah:

And we weren’t recruiting on race and ethnicity, we were recruiting on a bunch of other variables,

Lilah:

but when we did that, we got a very diverse group of people on gender and race and ethnicity

Lilah:

as well.

Lilah:

And such a diverse set of perspectives on financial services and wealth-building.

Lilah:

So, it’s been really interesting when you open up the door and you can recruit nationally

Lilah:

for virtual and qualitative, it’s been pretty different.

Lilah:

There’s a lot of work still to be done, especially on panel sample and surveys and

Lilah:

how we screen, for sure.

Lenny:

So, are we seeing—and Lilah, I want to play off what you just said and thinking

Lenny:

about the research that we did together years ago, and how we teed things up that we are

Lenny:

not engaging with a representative, sample simply because we don’t have the right tools

Lenny:

or approaches to engage with them.

Lenny:

Are you finding that in the work that you do across the board in Logica that there’s

Lenny:

possibly a fundamental flaw in the research process where this hypothesis would be, you

Lenny:

know, Millennials or Gen Z are just saying, “Why the hell—why would I do this, you

Lenny:

know?

Lenny:

I don’t participate in research.”

Lenny:

Because we’re not giving them the appropriate incentive, not necessarily, like, monetary

Lenny:

incentive, but an appropriate reason to participate?

Lenny:

Is that something that you see?

Lilah:

That’s interesting.

Lilah:

I think there’s probably room to look at monetary incentives, but I actually think

Lilah:

that the incentives that we have are working in a lot of ways to get Gen Zers and Millennials

Lilah:

to participate.

Lilah:

But I think there’s probably research that needs to be done to figure out what the right

Lilah:

incentives are.

Lilah:

And what I will say about the younger generation, which may not have changed too much is that

Lilah:

they’re looking at all kinds of promotions, and that’s in research, and that’s in

Lilah:

financial services.

Lilah:

And with Covid pandemic and the economic situation, incentives and promotions, monetary promotions,

Lilah:

are really appealing to the younger generation, and they are figuring out pretty clever ways

Lilah:

to work it or to make it work for them.

Lilah:

So, I think we do need to look at it as an industry and figure out what’s going to

Lilah:

incentivize different groups of people and attract different people to participate in

Lilah:

research, for sure.

Lenny:

Well, let’s broaden that out then, as well.

Lenny:

So, not just kind of the generational financial piece, but what are your perspectives on what

Lenny:

we need to do as an industry to engage populations that possibly are not currently being represented

Lenny:

in our samples?

Lenny:

And is that… yeah, well, let’s leave it at that.

Lenny:

Is there something else that we need to be doing?

Lenny:

So Keyona, what’s your thoughts on that?

Keyona:

So, I happen to be a Millennial, and—

Lenny:

[laugh].

Keyona:

—by marriage, I have stepkids, who are Gen Zers, so I have thought about this

Keyona:

for my own self, and then just as a market researcher, thinking about all perspectives.

Keyona:

And then had some conversations recently with a 16 and 17-year-old, and it seems like some

Keyona:

of the stereotype in what the two generations are known for is this access to knowledge.

Keyona:

And a lot of what I’m seeing is that there’s just a desire for there to be an exchange

Keyona:

of that knowledge.

Keyona:

So, “Hey, researcher, you want to ask me questions, I’ll give you those answers,

Keyona:

but I want to know how you’re using them.

Keyona:

I want to know how my answers impacted the way you market, the way you created your product,

Keyona:

the way you created packaging, whatever it is.”

Keyona:

And so, I feel like that’s a lot of what the incentivization needs to be.

Keyona:

Companies, if they’re able to be open to [laugh] this exchange of information, can

Keyona:

provide a little bit of insight to these respondents and incentivize them in a way that, you know,

Keyona:

makes them brand ambassadors and makes them people who are willing to talk about this

Keyona:

brand because they know that they almost feel like they’re an employee, [laugh] or feel

Keyona:

like their voice is being heard.

Keyona:

So, that would be one approach that comes to mind for me is figuring out how we can

Keyona:

make the exchange of information a bit more mutual, rather than asking Millennials and

Keyona:

Gen Zers for information, and then, you know, saying thank you, and giving money that they

Keyona:

may or may not need.

Lenny:

Is there any difference in that perspective—so [unintelligible] kind of the generational

Lenny:

age thing—and, Tim, I’ll follow up with you in just a second on this—there any difference

Lenny:

in approach that we need to be factoring in based on race or ethnicity?

Keyona:

I do think so.

Keyona:

I think some of it comes into play with the socioeconomic status that you touched on a

Keyona:

bit with Lilah.

Keyona:

From, you know, going back to the basics, again, we can think about Maslow’s Hierarchy

Keyona:

of Needs.

Keyona:

So, if we’re trying to reach people who don’t know what they’re going to eat for

Keyona:

dinner, then they’re not going to be very interested in responding to our survey, especially

Keyona:

if that survey doesn’t help them get food for dinner.

Keyona:

So, I think there’s definitely a different approach that needs to come for people who

Keyona:

may be of a lower socioeconomic status.

Keyona:

Those people tend to be people of color.

Keyona:

I am black and so I have seen that and have lived experience with that.

Keyona:

And so, I think that’s probably one of the first considerations we need to make as researchers

Keyona:

when thinking about people of different racial or ethnic backgrounds.

Keyona:

Language spoken at the house comes to mind as well.

Keyona:

Different cultural [laugh] identities and cultural practices that may be important to

Keyona:

white American culture, but not to other cultures should be considered as well.

Keyona:

But I think the first thing that comes to mind is what you and Lilah already touched

Keyona:

on with that socioeconomic status and being conscious of what’s important to your respondent.

Lenny:

Innovate MR has recently appointed Market Research Leader Kristin Luck to the

Lenny:

board of directors and has garnered significant investment from civic partners.

Lenny:

With this, the team has entered a new era of exponential growth, expanding their ability

Lenny:

to help brands around the world make data-driven decisions.

Lenny:

The team has created the Vision Suite, a Stevie Award-winning ResTech platform offering researchers

Lenny:

a comprehensive collection of next-generation products enabling survey design, sample procurement,

Lenny:

fraud mitigation reporting, and do-it-together team support.

Lenny:

I have never thought of sampling from the perspective of Maslow’s Hierarchy of

Lenny:

Needs.

Lenny:

That’s very cool.

Lenny:

We could go on about that.

Lenny:

We may have to circle back at some point.

Lenny:

That is a very cool idea.

Keyona:

Let’s do another episode.

Keyona:

[laugh].

Lenny:

We may, absolutely, because that’s a really cool idea.

Lenny:

Because I’ve been harping on this for, gosh, my entire career that we have to think like

Lenny:

marketers, but act like researchers, and think about marketers from the standpoint of engagement,

Lenny:

right?

Lenny:

We have an engagement problem.

Lenny:

We’re very transactional, we’re very commoditized, we’re very structured, we need to talk to

Lenny:

people the way they want to be talked to, but I never framed it up that way, so that’s

Lenny:

very cool.

Lenny:

All right, Tim, so from an accessibility perspective—because I know that’s one of the areas that you

Lenny:

focus on, and I know QuestionPro has led the forefront of [unintelligible] for

Lenny:

years in, kind of, pioneering different approaches to engage research, whether early mobile platforms,

Lenny:

et cetera, et cetera.

Lenny:

What are the form factor aspects that we need to deal with and think through technologically

Lenny:

in order to address accessibility challenges, as well?

Tim:

So, it’s one thing to be legally compliant, and play nice across the board from a technological

Tim:

standpoint, from a GDPR perspective.

Tim:

You know, those are the minimum boxes that you have to check to send out surveys.

Tim:

What you aren’t considering is the different color contrast for someone who is colorblind,

Tim:

what it looks like at 400 times zoom when you’re on mobile versus a tablet versus

Tim:

a desktop device, auto-defaulting to auto-advance after answering a question, that helps with

Tim:

upper limb mobility issues.

Tim:

So, I was auditing an app the other day, was someone who was a disabled veteran, and they

Tim:

had upper limb mobility issues, and you know, all the apps seem to swipe.

Tim:

And that was really tough.

Tim:

So, because that’s so common, you know, this person was commonly left out.

Tim:

So, they should be able to navigate through a system by saying, just left or right movements,

Tim:

interacting, whether that’s with a blow tube, or a joystick, or keyboard only.

Tim:

But I think that there’s just further considerations that, you know, I haven’t even uncovered

Tim:

yet.

Tim:

So, with the deaf community, I find that, you know, the easiest, and one of the best

Tim:

inventions for them, and where they live is on Twitter because it’s all written out,

Tim:

it’s succinct, like, the language is—ASL is at least—and you’re able to get your

Tim:

point out really quickly.

Tim:

And a lot of times, deaf people will get frustrated because they have to put in so much more work

Tim:

to express how they’re feeling and move around and things.

Tim:

But that’s, you know, that’s not the opinion that everyone has.

Tim:

So, I would say, make sure your surveys are compliant by the ADA, but also go a step further

Tim:

and make sure someone’s doing a blind spot analysis.

Tim:

I will say that I audited the ADA website and they broke their own rules 164 times on

Tim:

the homepage.

Tim:

So, it’s really hard for us to look in the mirror and say, “Man, we’re not accessible.”

Tim:

Well, neither is the person who’s gaining the accessibility putting that up.

Tim:

And the EU equivalent was just as bad.

Tim:

So, from that as a starting point, let’s make our own rules, and let’s do better

Tim:

than the ADA.

Lenny:

Okay.

Lenny:

So, I agree with everything you just said.

Lenny:

I’m going to play devil’s advocate for a minute, more from a business standpoint.

Lenny:

So, all of you have spoken at [IAX], I think, all of you have spoken at IAX at

Lenny:

some point.

Lenny:

And the rule of thumb is we use PowerPoint for presentations because it’s the lowest

Lenny:

common denominator, right?

Lenny:

So, in building scalable architecture, you know, scalable businesses, you automatically

Lenny:

gravitate towards the mass, towards the majority, and so that it can just scale; that’s just

Lenny:

the part of scaling a business.

Lenny:

So, if I’m building software, I’m thinking about the majority of people are going to

Lenny:

use this, which would probably fall within the spectrum of normal abilities.

Lenny:

And my thinking would be that if someone is differently-abled, that it is on them to use

Lenny:

some other technology that could plug into their device to help them participate.

Lenny:

Now, I’m not saying that that’s right, so please be clear.

Lenny:

[laugh].

Lenny:

I’m not saying, “Oh, everyone with a disability, you’re on your own.”

Lenny:

That’s not what I mean.

Lenny:

But there’s a tension from a commercial perspective, right, in building businesses

Lenny:

that is probably one of the challenges that we have to get through here.

Lenny:

And is it the—we’ll just use QuestionPro’s example, is it QuestionPro’s responsibility

Lenny:

to ensure that they have text-to-speech embedded into every survey, or is it the respondent’s

Lenny:

responsibility to ensure they have a plugin that converts text to speech?

Lenny:

You know, just as one example.

Lenny:

Is it QuestionPro’s responsibility to change the design of a survey for folks that can’t

Lenny:

swipe or is it the respondent’s responsibility to have some other type of device they can

Lenny:

use to navigate?

Lenny:

I don’t know the answer to that.

Lenny:

I’m just curious on the panel as a whole, let’s take it up a level.

Lenny:

What is the responsibility for the research industry to adapt to challenges that may be

Lenny:

a very small percentage of the population that are barriers to entry?

Lenny:

And let’s play with that.

Lenny:

So, Tim, since I picked on you, why don’t you respond to that, and then we’ll go around

Lenny:

real quick.

Tim:

So, I love this question.

Tim:

And it really gets you thinking.

Tim:

So, you said build a product.

Tim:

I’m thinking, I know there has to be a point at which financial consideration has to be

Tim:

made; you can’t do everything for everyone, but if we’re talking about one in four people

Tim:

that can’t access your website due to a disability, you’re missing out on a fourth

Tim:

of the market.

Tim:

I think as a free market, people should not use those solutions.

Tim:

If you’re not catering to the respondent experience, then you should be left behind.

Tim:

Disabled people have $490 billion per year of discretionary income.

Tim:

They’re able to be advocates of your product, they will tell their friends about your product,

Tim:

and you’ll have at least 25% of the market already available to use your product over

Tim:

another the product.

Tim:

And win where you can.

Tim:

I can tell you… it is a whole lot less expensive to be accessible up front than to have us

Tim:

come in and audit your design midway through because you got sued because you were inaccessible.

Tim:

So, I would recommend to have a successful product, that you are accessible as much as

Tim:

you can be and work with the different groups of people that you want to get influence from.

Tim:

And Keyona was really, really smart with the hierarchy of needs and meeting the people

Tim:

where they are, but these individuals really care about the social impact that they’re

Tim:

making with helping out design.

Tim:

So, user research is incredibly powerful in the beginning stages.

Tim:

So, I would say a low-dollar way to do it at the beginning is to involve those people

Tim:

from the start.

Tim:

If you can’t, get a good lawyer.

Lenny:

[laugh].

Lenny:

So, thank you, I appreciate you reframed what I was trying to get to now is the business

Lenny:

case.

Lenny:

Why does this matter?

Lenny:

We talked about the representativity component of things.

Lenny:

But now the business case.

Lenny:

So Lilah, can you build on that with, kind of, where Tim was of what’s the business

Lenny:

case for why we need to pay attention to this, why this is an important topic?

Lenny:

What would you add?

Lilah:

Well, I mean, I’ll build on it also in the sense of I do think that industry,

Lilah:

the marketing research industry has a role to play as well in setting guidelines and

Lilah:

standards for inclusivity, whether it’s race, ethnicity, gender identity, access,

Lilah:

and ability.

Lilah:

And our clients, the brands may not be able to or aren’t looking forward, maybe, in

Lilah:

that way to do it themselves, and so we need to bring this to them and say it’s important

Lilah:

and also have solutions on how we’re going to have more inclusive sample and participants.

Lilah:

So, I know again, as part of the IDEA Council, we’re going to be looking at gender and

Lilah:

sexual orientation next, and we also are going to be doing research on access and ability.

Lilah:

And, Tim, I’m definitely going to reach out to you on that for sure.

Lilah:

And we need to approach that research in a different way from what we did on race and

Lilah:

ethnicity.

Lilah:

But, you know, Tim, you have the facts in terms of the percentage population that you’re

Lilah:

leaving out by not doing this, and I would say as an industry, we have a responsibility

Lilah:

to our clients and to our participants to help set up the research for success so that

Lilah:

we make sure we have the people giving feedback on these products and services that we need

Lilah:

to have.

Lenny:

Alright.

Lenny:

J—yes, please.

Keyona:

I would just add to that.

Keyona:

You know, Tim said, one in four people have a disability.

Keyona:

So, what that likely means is that all of us know someone who has a disability.

Keyona:

And I think the way at least American culture is headed right now—this is super important

Keyona:

to Gen Z and Millennials—is they want to know that you’re treating their friends

Keyona:

well.

Keyona:

[laugh].

Keyona:

So, even if they don’t identify with having a disability, they want to support brands

Keyona:

who support their friends with a disability.

Keyona:

And so, I think that the rest of the 75% that we’re not covering—or that we’re already

Keyona:

covering, they really care about it and will voice that with their money as well.

Keyona:

So, you talk about the billions of dollars that Tim already addressed from that quarter

Keyona:

of the population, and then there’s a vast majority of the population, especially, you

Keyona:

know, under 40, who say, “If you show that you care about my disabled mother, or my disabled

Keyona:

brother, or my disabled friend, or me, then I want to support that brand and I will sacrifice

Keyona:

the price that, I pay the dollar amount that I’m paying, if you know, say a product is

Keyona:

$10 more or whatever percent more, they’re willing to pay that because of what it stands

Keyona:

for.”

Keyona:

So, I think just culturally, we’re headed in a direction where that business case, that

Keyona:

quarter of the population is important, but it’s also important to remember that there’s

Keyona:

a vast majority of the population that cares about and loves people who are in that 25%

Keyona:

and want to support a brand that shows that they support those people.

Lenny:

Those are all fantastic points.

Lenny:

So, I want to be conscious of time because I think we can keep peeling this onion for

Lenny:

a while.

Lenny:

So, you’ve all mentioned the IDEA Council through Insights Association, and my understanding

Lenny:

is that there’s research that’s been conducted, or has been conducted, about this.

Lenny:

Actually, right before this, I saw an email come through from the IA about the IDEA Council,

Lenny:

so you know, the universe is aligning from a timing standpoint for this conversation.

Lenny:

Lilah, I think it’s your turn, so I’m trying to keep track of who’s talked.

Lenny:

Do you want to give us an overview of that work and then everybody else can chime in.

Lenny:

So, we can do an explanation of what the IDEA Council is, when it’s done, and where it’s

Lenny:

going.

Lilah:

Yeah, absolutely.

Lilah:

And Keyona will have a lot to say, too, has been working on the analysis.

Lilah:

So, the IDEA Council was formed in 2020, with all the social unrest in the country and with

Lilah:

the purpose of having inclusive, diverse, equitable, and providing access in the industry.

d there were really two paths:

one was for the profession to include diversity and access

d there were really two paths:

in the profession and researchers and talent; and the other one was to do research on research,

d there were really two paths:

to have more inclusive research.

d there were really two paths:

And the first phase of that was, first, a position paper to understand what are all

d there were really two paths:

the ways that we’re asking about race and ethnicity and gender and sexual orientation

d there were really two paths:

today, and then we did this really comprehensive study of how we ask race and ethnicity today.

d there were really two paths:

We tested ten different questions based on, you know, all the ways that we saw different

d there were really two paths:

companies were asking race and ethnicity across 5000 people in the United States.

d there were really two paths:

It was US-centric and it also was panel sample.

d there were really two paths:

And we found that people really want some key things.

d there were really two paths:

They want the option to have select multiple races and ethnicity, they want to be able

d there were really two paths:

to see themselves in the answer choices by having descriptive answers, not the short,

d there were really two paths:

single select that many companies are still using.

d there were really two paths:

That’s a byproduct of old census questions, but the more detailed question and even more

d there were really two paths:

detailed than the census question today.

d there were really two paths:

So people, you know, don’t like ‘other,’ for example, it’s alienating, and want to

d there were really two paths:

be able to list their race and ethnicity.

d there were really two paths:

So, lots of great learnings there and we’re working with panel companies and brands to

d there were really two paths:

get those questions integrated into how we ask and then the next phase is on gender and

d there were really two paths:

sexual orientation.

d there were really two paths:

And again, we’re going to test different ways to ask, as there’s just a lot of, again,

d there were really two paths:

a lot of different ways to ask and confusion about which gender and sexual orientation.

d there were really two paths:

And then the third phase will be an access and ability.

d there were really two paths:

So, I’ll stop there.

d there were really two paths:

Keyona, I know you are really integral and have presenting these, been presenting these

d there were really two paths:

results a lot.

Keyona:

Yeah, it’s been a pleasure getting to work with you, Lilah.

Keyona:

I’m reading the board that’s behind you that says questions are the answers.

Keyona:

And I feel like that’s a lot of what we’ve experienced with even this first phase of

Keyona:

research.

Keyona:

It’s given us answers, but now we have new questions, [laugh] and want to continue to

Keyona:

expand upon that research.

Keyona:

So, we’ve actually taken a deep dive into that data and we’ll be presenting next Friday

Keyona:

to talk about the UX of survey questions, and specifically demographic questions and

Keyona:

how we can make the respondent experience a priority again.

Keyona:

And it’s based on the [open ends] and a lot of the findings that we found in

Keyona:

this IDEA Council research.

Keyona:

And so, I think there will be a lot of phase twos and phase threes of this research.

Keyona:

And one of our biggest encouragements as a research team is a market research industry,

Keyona:

“Will you all get involved and be innovative with us and conduct this research too?”

Keyona:

Because, you know, as we’ve talked about during this time together, it’s on all of

Keyona:

us as an industry to make sure that we’re leading in this space.

Keyona:

So, some of the questions that we’ve started to explore as well, what would this look like

Keyona:

in a global survey?

Keyona:

There are some countries that don’t really talk about race or identify people that way?

Keyona:

How could we address that question in our surveys?

Keyona:

What if we stopped asking race and ethnicity?

Keyona:

And Lilah talked about this earlier on in the survey that she said she conducted where

Keyona:

they didn’t ask about race and ethnicity, but then in the end, saw that there was diversity

Keyona:

as a byproduct of the types of questions and qualifiers for people taking the survey.

Keyona:

So, I think there’s a lot of innovation and opportunity here.

Keyona:

I’m really excited for what the IDEA Council is doing.

Keyona:

I hope I’m not wrong in saying this, but please come join.

Keyona:

[laugh].

Keyona:

If you’re excited about this, and this conversation gets you excited, come be a part of this because

Keyona:

I think Lilah and I have both found that this is a second full-time job [laugh] to sift

Keyona:

through the findings and talk about the research.

Keyona:

And that’s a great thing, but let’s all get involved.

Keyona:

As we’re talking about here, diversity of thought is important in this research and

Keyona:

in deciding what the impact is.

Lenny:

Great.

Lenny:

Tim, would you add anything to that?

Tim:

So, I’m not a part of the IDEA Council, yet.

Lenny:

Yet.

Lenny:

[laugh].

Tim:

[laugh].

Tim:

Plug there, hint, hint.

Tim:

But one thing that really bothers me is that we base questions on the census.

Tim:

Think about how wrong and racist the census has been in its history and how it’s only

Tim:

every ten years, and I would love for there to be a different standard that we hold ourselves

Tim:

up to.

Tim:

And it sounds like that’s what you’re doing with race and ethnicity and sexuality

Tim:

and I would love to have a template for best practices, and IDEA Council approval of the

Tim:

surveys that go out that are accessible as well.

Lenny:

Yeah, and I will go on record and volunteer that in the next round of GRIT in the fall,

Lenny:

let’s get together and find—you know, GRITs already this humongous beast speaking

Lenny:

[unintelligible] [laugh] so I don’t want to add too many questions, but I think

Lenny:

there is room to add a couple of questions to get the perspective from the industry,

Lenny:

from the buyer side and the supplier side, on this as well, so we’ll circle back around.

Lenny:

Now, it does bring up another—and I want to be conscious of time, so this maybe opening

Lenny:

up a can of worms we don’t have time to [laugh] fully dig into—in the era of one-to-one

Lenny:

marketing, you know kind of P&G’s stated, you know, goal to have a one-to-one relationship

Lenny:

with every person, in real-time, on the planet, knowing all of those demographic components

Lenny:

and having that deep knowledge of the individual respondent-made sets, but we are moving into

Lenny:

a world now because of the decline of the cookie, because of GDPR, you know, et cetera,

Lenny:

et cetera, where there are significant barriers of building that one-t—if I’m a brand;

Lenny:

let’s use P&G, as an example, right—having that one-to-one relationship, so therefore,

Lenny:

having the data that I can map is becoming significantly more challenging and potentially

Lenny:

even impossible under the current architecture of the web, the current environment, from

Lenny:

a legislative perspective.

Lenny:

Even the massively changing, almost politically motivated of web users, right?

Lenny:

There’s a lot of fragmentation that is occurring around data and around understanding who consumers

Lenny:

and around platforms, et cetera, et cetera.

Lenny:

So, if that is true, and I do believe that it is, it leads me to the question of the

Lenny:

argument—and you mentioned this Keyona—maybe we don’t even need to ask these questions

Lenny:

at all because they may not be actionable, truly, right?

Lenny:

We may not be able to deliver the right message to the right person at the right time, so

Lenny:

we may not be able to tailor the advertising to everybody individually at the same level

Lenny:

that we were even a year or so ago.

Lenny:

Because there are simply technological barriers.

Lenny:

In some ways, we’re going back to targeted advertising in magazines, or, you know, it’s

Lenny:

all these walled gardens.

Lenny:

You know, we’re basically getting back to kind of billboards, [laugh] in these different

Lenny:

properties that we think have the populations we want to target.

Lenny:

So, if that is the case, in this hypothetical question, do we get to a place, potentially,

Lenny:

where some of this just could be, it’s the right thing to do, but pragmatically, it doesn’t

Lenny:

matter because we can’t use the data from a marketing perspective to power our clients

Lenny:

product development.

Lenny:

And then, how do we juggle those two things?

Lenny:

How do we juggle it’s right to make sure that we are engaging with everybody, but some

Lenny:

of these questions just may go away because they don’t matter anymore.

Lenny:

Just a thought.

Lenny:

So for—well, this is your chance to say, “Lenny, that’s stupid.

Lenny:

That’s crazy.

Lenny:

What are you talking about?

Lenny:

That’s just insane.”

Lenny:

Or if you have a thought on what that future may look like.

Lenny:

So Keyona, you were nodding through that.

Lenny:

You looked the most thoughtful as I was [laugh] going off on that, so I’m going to pick

Lenny:

on you first.

Keyona:

Well, I’m intrigued by it, there’s a book called The Four, and I believe maybe

Keyona:

the author is working on another version, adding a fifth brand in there.

Keyona:

But The Four talks about Apple, Amazon, Google, and Facebook.

Keyona:

And it kind of gets back to what I was talking about towards the beginning, is that if there’s

Keyona:

this equitable exchange of information between the company and the consumer, then the consumer

Keyona:

is more willing to let you have information about them because they are then getting something

Keyona:

in return.

Keyona:

And I heard at a conference recently that Facebook, now Meta, I think, based on the

Keyona:

information that they collect about people, the way they post, the types of posts that

Keyona:

they like, the way they interact, just in Facebook or Meta products.

Keyona:

They’ve got, like, these different segments, I think, like, some people are called a dog

Keyona:

and a cat, and like, there’s these different personas that people fall into.

Keyona:

And I think about, you know, Google and Apple, where people are willing to connect all have

Keyona:

their Apple products, [laugh] and give this plethora of information to this company because

Keyona:

of what they’re getting in return.

Keyona:

And so, I think what you’re talking about scares me because there’s this place where,

Keyona:

right now there’s four companies that do that really, really well, and I do not like

Keyona:

the idea of being monopolized [laugh].

Keyona:

However, if other companies can follow that template, perhaps we end up in a really beautiful

Keyona:

place of companies getting to know their consumers deeply, and being able to offer personalization

Keyona:

that it seems and I know we’ve talked a lot about Gen Z and Millennials and age diversity

Keyona:

is a whole other conversation.

Keyona:

So, they’re not the only focus, but we are headed in a direction where that personalization

Keyona:

is really desired.

Keyona:

And so, if we can follow those templates, and make sure that exchange of information

Keyona:

is equitable and mutual, I do think it could be a beautiful thing, but right now, it scares

Keyona:

me because there’s four companies that are far, far ahead [laugh] in doing that, and

Keyona:

the rest are starting to fall behind, as you talked about, and going back [laugh] to the

Keyona:

old way of marketing.

Keyona:

So, yeah, I want to remain hopeful because I’m a researcher and I think we can do it.

Keyona:

But yeah, I’m a little scared, too.

Lenny:

Right there with you.

Lenny:

Again, I wish your audience could see us as we’re—because I was—I think I hurt my

Lenny:

neck nodding so much—

Keyona:

[laugh].

Lenny:

—as you were talking.

Lenny:

All right.

Lenny:

Lilah, any thoughts on this brave new world we may be heading into, and what that may

Lenny:

mean for this overall topic?

Lilah:

Yeah.

Lilah:

I mean, I still think I don’t want to lose sight of part of the purpose of thinking about

Lilah:

this is to make sure that we have inclusive voices on our research, and so we need to

Lilah:

ask some of these questions to make sure that the range of people are represented in the

Lilah:

feedback.

Lilah:

And also, the way that people identify and want to identify is changing.

Lilah:

And so, that sort of breaks open, you know, a bunch of other ways we might think about

Lilah:

asking questions that go beyond sort of our core standard demographics, and especially

Lilah:

with younger generations, again, thinking about Gen Z, they don’t want to be pigeonholed

Lilah:

in certain ways and want to be able to identify, situationally, based on what their needs are.

Lilah:

So, that gets kind of to my third point, which is for product development, for brands, focusing

Lilah:

on the need that you’re solving for and communicating that need and you will attract

Lilah:

the right people to your product if you focus on solving for that need and communicating

Lilah:

what you’re solving for.

Lilah:

So, [laugh] that’s my two cents.

Lenny:

Thank you.

Lenny:

A more pragmatic perspective than maybe where Keyona and I were going of like, “Holy crap.

Lenny:

Wait, what if that happens?”

Lenny:

So, thank you for [laugh] bringing this back a little bit to the fundamentals.

Keyona:

To rein us back in.

Keyona:

[laugh].

Lenny:

[laugh].

Lenny:

Right, we were getting out there weren’t we?

Lenny:

Or at least I was.

Lenny:

Tim, thoughts from you.

Tim:

I’m in the middle of the road here.

Tim:

I think is very important to have these questions to make sure that we are as equitable as possible,

Tim:

but we also need to get the buy-in from the survey respondent who I think should be rewarded

Tim:

by the questions that they answer, not the surveys that they complete, and with the data

Tim:

points that they’re willing to give you, with their consent.

Tim:

So, I see it, if you’re familiar with the Brave Browser, you can get basic attention

Tim:

tokens by opting into advertising, and at any point, you can withdraw your consent,

Tim:

you can completely not see ads if you don’t want to.

Tim:

I know Lenny has an interest in a company called Veriglif, which is pretty, I think,

Tim:

on the right direction there with consent and making sure that it carries across the

Tim:

web to where, you know, on the blockchain, you know, you say what you’re going to give,

Tim:

and at the end when the brand is analyzing, I think that of the ones who said that they

Tim:

would give their consent to age, gender, sexual orientation, and you know, that can change

Tim:

situationally.

Tim:

Right, Lilah?

Tim:

And so, you vote with your dollars.

Tim:

And I think that we put—make the respondents an equal part by paying for every piece of

Tim:

data, not just the outcome of the survey.

Lenny:

Yeah, now my neck really hurts because I’m really nodding.

Lenny:

So yes, thank you.

Lenny:

Appreciate the plug.

Lenny:

[laugh].

Lenny:

Well, the whole [unintelligible].

Lenny:

And that does inform my perspective on this as well, right?

Lenny:

It goes back to the idea that fundamentally, I think we have a problem with how we engage

Lenny:

with consumers and we have to find any way to do that.

Lenny:

And anyway, I won’t get into Veriglif stuff.

Lenny:

There will come a time where we’ll talk more about that.

Lenny:

Stay tuned.

Lenny:

Stay tuned.

Lenny:

All right, guys, we could go on for a lot longer, but I think we’re at the top of

Lenny:

our time.

Lenny:

Any final thoughts?

Lenny:

Where can people reach you?

Lenny:

Any plugs you want to give that don’t involve one of my business interests, Tim?

Lenny:

[laugh].

Lenny:

And we’ll just go back around.

Lenny:

Start with Lilah, final things that you want to just make sure our listeners hear?

Lilah:

Yeah.

Lilah:

Well, thanks, Lenny, for having me and this group on the podcast.

Lilah:

It’s really fun talking to you and Keyona and Tim.

Lilah:

And final plug I’d have is yeah, if you want to get involved, contact me in terms

Lilah:

of, you know, how we’re asking these questions today.

Lilah:

And I totally agree with Keyona; this is a massive group industry effort, and I’ve

Lilah:

been amazed at the level of enthusiasm and energy around this.

Lilah:

And I feel like there’s a lot of momentum and the time is right, so thanks for having

Lilah:

this conversation, Lenny.

Lenny:

Thank you, Tim?

Tim:

Yeah, Thanks, Lenny.

Tim:

I appreciate it.

Tim:

And thank you to all the listeners who are willing to look in the mirror and see a potential

Tim:

better way to conduct business, more equitable and accessible.

Tim:

QuestionPro XDay is June 6th.

Tim:

Make sure you register online.

Tim:

Anything accessible, you can reach me directly.

Tim:

Any questions you have about representation of the disabled community, please reach out

Tim:

directly to me.

Tim:

I will leave you with one quote from Maya Angelou.

Tim:

“Do the best you can until you know better.

Tim:

Then when you know better, do better.”

Tim:

So, let’s all do better.

Tim:

Thanks for having me.

Lenny:

Oh, that’s a good one.

Lenny:

All right.

Lenny:

Keyona, final word is yours.

Keyona:

Man, you should have ended with Tim.

Keyona:

[laugh].

Keyona:

I need to scramble and find a quote really quick.

Keyona:

No, thanks for having me, Lenny.

Keyona:

It’s been great chatting with you, Lilah, and Tim.

Keyona:

I’m excited.

Keyona:

I think, one, if anyone wants to reach me, my name is Keyona K-E-Y-O-N-A.

Keyona:

I’m at Accelerate right now, so acceleratesg.com.

Keyona:

Also through the IDEA Council.

Keyona:

I would just encourage us to be innovative and open-minded.

Keyona:

I think that right now we’re in a space where we’ve done things a certain way for

Keyona:

so long, and so I would love to just see where open-mindedness takes us.

Lenny:

Thank you.

Lenny:

That’s our show for today.

Lenny:

Thank all three of you.

Lenny:

This was really a wonderful conversation.

Lenny:

We will reconvene at some point and continue it and thanks for the work that you’re doing.

Lenny:

Many thanks to our producer Karen Lynch, our editor James Carlisle, and our episode sponsor,

Lenny:

Innovate MR.

Lenny:

Thank you for your time, for our listeners.

Lenny:

I appreciate you spending it with us.

Lenny:

And that’s it until the next edition of the GreenBook Podcast.

Next Episode All Episodes Previous Episode

Listen for free

Show artwork for Greenbook Podcast

About the Podcast

Greenbook Podcast
Exploring the future of market research and consumer insights
Immerse yourself in the evolving world of market research, insights and analytics, as hosts Lenny Murphy and Karen Lynch explore factors impacting our industry with some of its most innovative, influential practitioners. Spend less than an hour weekly exploring the latest technologies, methodologies, strategies, and emerging ideas with Greenbook, your guide to the future of insights.

About your host

Profile picture for Greenbook Podcast

Greenbook Podcast