All Market Research is Wrong

[Focus group discussion of Apple's 1984]

Let's lay this oft discussed issue to rest, shall we? 

I have long held and espoused the view that All Market Research is Wrong. 

The idea of researching a market makes sense to anyone who is about to invest money into engaging with one, using products, services or communications.

However, the basic idea underlying all market research is epistemologically specious.

That is to say, it seems to make sense, but, under examination, it does not.

In essence, the foundation of market research is that, by asking lots of people questions, or asking a smaller group of people more in depth questions, we can gather dependable insights into why they buy what they buy, and whether or not they will buy something in the future, perhaps after having seen some advertising. 

I don't believe this is true.

For two, very simple reasons.

1. We don't know why we do what we do.

We make these [and most] decisions at a subconscious level, which means, by definition, the operations of the decision process are inaccessible to our conscious minds.

That doesn't mean people won't answer though.

They will happily answer, erroneously, in a way that seems to make sense, as their minds create explanatory fictions to explain their own behavior to themselves, or the interviewer. 

2. The gulf between claimed attitudes [and intentions] and actual behavior is vast.

Asking people if they intend to buy something is analogous to asking them if they intend to go to the gym - the results may not correspond well with future behavior.

Focus groups are particularly fraught: they create utterly artificial data as a response to utterly artificial situations and social dynamics. 

I wrote a paper about this a while ago - you can download it here if you want 

Download Uncovering Hidden Persuaders - Faris

- and recently a nice person called Hilary sent me a review copy of an excellent book called 'Consumer.ology' by Philip Graves that examines, in much more persuasive detail than my paper, the same issues.

I enjoyed it immensely.

[Although I am aware that this might be due in some aspect to confirmation bias since it re-inforces my existing views and I am possibly also affected by the fact I was sent it for free, which triggers certain kinds of feelings].

As lots of experimentally demonstrated behavioral economics shows us, attitudes are not a good indicator of behavior and framing, context and subliminal associations are hugely important drivers of choice, but we are unaware of such, and indeed that awareness of subliminal effects entirely negates their impact.

[This is ultimately why asking people if certain media or creative will affect their behavior is utterly pointless.

Forcing people to consider something rationally that operates on an unconscious, emotional level, is always going to give the wrong answer.

As Graves points out in the book:

Ironically, given that the consumer research the feeds it fails to take it into account, it could be argued that most marketing leverages the unconscious mind, and indeed it must to do in order to be effective.

In many consumer experiences it is either impractical or impossible to compare the array of products on offer.

To operate efficiently, consumers rely on their unconscious mind to make decisions.

There are lots of great anecdotes and examples in the book:  

• preference for identical Nike running shoes increasing across the board in test due to the smell of flowers being introduced to the room;
• taste tests indicating that people have a clear preference between identical drinks depending on context;
• the fact that people perform the same tests differently after being subliminally exposed to different stimuli, like logos, but have no knowledge of how this influence as taken place

[people exposed to the Apple logo found more uses for an everyday object in creativity tests, people exposed to the Disney logo behaved more honestly than the control in subsequent tests];

• and of course the obvious fact that, despite rigorous testing, 80% of new products fail, and for a product like Red Bull researchers concluded "that no other product had ever performed so poorly in consumer testing; the look, taste, and mouth-feel were regarded as "disgusting" and the idea that it "stimulates mind and body" didn't persuade anyone the taste was worth tolerating."

And yet - because tests are not the same as the real world and, therefore, do not predict what will happen in it - by 2006 the company had sold more than 3 billion cans. 

[As Goldman famously said of Hollywood: Nobody Knows Anything]

Graves' point is, in essence, if you want to understand behavior, you must:

• observe actual behavior - because claimed attitudes and intensions don't correlate
• surreptitiously - for as soon as someone knows they are being researched, their responses are suspect
• in context - because context is such a vital driver of  behavior -
• with loss factored in - that is to say, saying you will buy a new product in a focus group doesn't force you to make a choice that loss aversion will flag as dangerous - there is no opportunity cost to saying you will do something. 

Market research is an $11bn industry in the USA but all the data it generates should be understood as wrong.

Now, that doesn't mean we should get rid of all of it.

It can be useful or interesting to understand what people think they think - a throw away comment from a focus group may inspire a brilliant idea - but the data it gives shouldn't be understood as 'true' - it needs to be interpreted - and we must supplement it with real behavioral data, from direct observation, or from the web - triangulating insights from as many sources as possible.

Ultimately, asking people questions is simply one way in which we attempt to create a model of the infinite complexities of human behavior.

And, ss George Box once said:

All models are wrong, but some are useful.

[UPDATE: There are lots of really well thought out points on both sides of this in the comments - I urge you to read them and thank you all for them. Clearly this is something people feel strongly about and debate is awesome.

The Herdmeister - Mark Earls - sent this link through with some of this thoughts.

Stefano also sent me a link to his excellent post about this, with some thoughts on how to make market research useful. I especially like this point:

Don’t care about what people say. Just like jazz, research is about the silences and the notes not played. A certain facial expression, a hesitation or a word not pronounced are more insightful than what people are willing to share.]

[UPDATE: the lovely and smart Britton has written a lovely and smart response/build to this post over on the W&K blog.]

[UPDATE: Illya has posted an excellent commentary on this post, and hosted another excellent one.]