14 Jan 2007 1224H

Why Focus Groups Fail: People say one thing and do another

In User Experience, the unlikely love-child of three kinds of design — interaction design, communications design, usability/user-centered design — plus information architecture, which itself comes from library and information science, and psychology and the related social sciences, it’s often observed that people say one thing and do another. It’s why we cannot reliably use interviews and focus groups to give us cues on design direction: what should be the criteria is what people are actually observed to do. How is this relevant to my user experience conjecture about user choice and conversion?

Well, people have no compunction telling you what they don’t like, it’s true. But people often don’t know what they really want. They really do have to be told what they want, to paraphrase the Apple VP I mentioned a few days back. You can call that creating markets or whatever, but I’m thinking insights for that can really only be informed by observed behaviors and domain knowledge.

My first hint of this happened years ago. It seems like a lifetime ago that I was working on BobVila.com and someone had commissioned a focus group for the site, which asked if people would or would not use this site, etc. etc. That’s fine, I think we know people will use the site, there is a pervasive interest in home improvement, but the question really is, what will they use it for, and what are the gaps in the space we can fill that will provide value for them?

For instance, there’s the often-quoted, rarely-referenced case of the Philips boom box. Christina Wodtke cites Kevin Doohan’s notes on Guy Kawasaki’s talk:

Phillips electronics gathered a group of teens in a room and asked them “What color boom box would you prefer? Black or yellow?”. Teens to a person said that yellow was it. Black was conservative and old. Not hip at all. Yellow is definitely the color for them. Later in the afternoon, the teens were informed that they could each take a boom box home with them. Boom boxes were stacked at the exit in two piles, a pile of black and a pile of yellow. All teens selected black. What customers do is more important than what they say. We can watch and influence what they do online like never before.

Another instance: people say they wash their hands after they go to the bathroom, but many actually don’t. Perhaps this is how the idea of automated urinals has come about, you know, the ones that have an IR emitter in front of them, and when you leave, it automatically flushes for you. It’s too bad people can’t have their hands automatically washed too.

So basically I think why focus groups fail is because the questions that we need answers for as User Experience people (as business people, really) cannot come solely from focus groups. Focus groups have their place, but the proof of the pudding is in the eating, really, as Nielsen said back in 1997. And they are often also influenced by group dynamics, as minority or divergent opinions might be lost or suppressed.

Finally, I just want to close this out by mentioning something about mass customization that I’ve been trying to chase down. Not too many people will remember this, but in 1998, Mattel launched a feature called “My Design Barbie” on Barbie.com. I often say My Design Barbie was killed by this phenomenon of people saying one thing but doing another, but that’s not the case. I was misled by a CRM seminar paper by Codoni and Martinelli at the University of Freiburg’s Department of Informatics (PDF, 1.8MB). First, they said less than 0.2% of all visitors converted on MyDesign, but for me, this number confuses how many people who wanted to buy a custom Barbie were actually able to complete. They also speculated additional reasons for the failure of the mass customization initiative:

Most of the users were too young to have credit cards;
Mattel already offers hundreds of doll variations in stores, often at half the $40 price of a customized model;
The cost could not be reduced because each customized doll was made by hand.

Clarifying, in the Journal of Product Innovation Management (2004, 21(6): 401-415), Franke and Piller write:

Mattel, another pioneering company in the field, abandoned its customized “MyDesign Barbie” as well, though the company had a rather sophisticated toolkit for children users on the internet. In an interview conducted by the authors, one manager told us that the reason for stopping the program was indeed too much user feedback. The 39-dollar customized doll (a premium of about 100 percent) attracted so many orders that the supply chain and fulfillment system was not able to handle all orders in the promised time, leading to dissatisfied customers due to long delivery times. The company was not prepared to capture the customer value it was creating in its manufacturing system. Given the relatively small volume of sales of the customized dolls compared to overall sales volumes, Mattel decided not to invest in its manufacturing and logistics capabilities but merely to keep the toolkit online with the order button. Today, users can still configure and reconfigure dolls, but just for the fun of doing it.

Note:: MyDesign Barbie fans might now visit FAO Schwarz in NYC for their fix. We wrote about this a few weeks ago. Here’s another story about that.

So, there ya go. Another case of no matter how spiffy your architecture is, if fulfillment is bad, you’re sunk.

I’m still fascinated about this question about choice management and mass customization; I may open up a whole thread on it. By now I’m sure we must have some knowledge about it as a niche market, but it’s still not clear for what items that holds true.

Permanent link to Why Focus Groups Fail: People say one thing and do another

Filed under Design, Technology, User Experience, Web

Be the first to respond to "Why Focus Groups Fail: People say one thing and do another"

RSS feed for comments on this post. TrackBack URL

No responses yet.

And now it's your turn.

Fire your weapon, soldier. Just be careful of friendly fire. NAME & EMAIL required.

Your response