I Vote for Experimental Research
Updated: Tuesday, Oct. 3, 7:47 p.m.
I admit it: I do not like survey research.
I will take the limitations and biases of an experiment any day.
During my research methods class this morning, we were talking about some of the many problems with surveys. One of those problems is response rate. We know so little about the people who will not fill out surveys because ... they will not fill them out.
Immediately after class, I open my e-mail to find the link to an Advertising Age story that warmed my heart.
"After all, no one really knows whether people who don't answer surveys are similar to those who do, because they don't answer surveys. But the industry does know nonrespondents tend to be disproportionately male, black, Hispanic and young," wrote Jack Neff.
The article continued, "Just 0.25% of the population supplies 32% of responses to online surveys, said Simon Chadwick, former head of NOP Research in the U.K. and now principal of Cambiar, a Phoenix consultancy, citing research by ComScore Networks. More broadly, he said, 50% of all survey responses come from less than 5% of the population."
Yes, I run experiments in a lab. I often use college students (having never been shown any evidence that their cognitive architecture is somehow different). But at least I admit my limitations to generalizability right up front.
In a grand bit of irony, I just discovered that my colleague, mentor, and friend Rob Potter (Indiana Telecom) also blogged regarding this article. It appears Dr. Potter needs to invest in Caller ID.
I admit it: I do not like survey research.
I will take the limitations and biases of an experiment any day.
During my research methods class this morning, we were talking about some of the many problems with surveys. One of those problems is response rate. We know so little about the people who will not fill out surveys because ... they will not fill them out.
Immediately after class, I open my e-mail to find the link to an Advertising Age story that warmed my heart.
"After all, no one really knows whether people who don't answer surveys are similar to those who do, because they don't answer surveys. But the industry does know nonrespondents tend to be disproportionately male, black, Hispanic and young," wrote Jack Neff.
The article continued, "Just 0.25% of the population supplies 32% of responses to online surveys, said Simon Chadwick, former head of NOP Research in the U.K. and now principal of Cambiar, a Phoenix consultancy, citing research by ComScore Networks. More broadly, he said, 50% of all survey responses come from less than 5% of the population."
Yes, I run experiments in a lab. I often use college students (having never been shown any evidence that their cognitive architecture is somehow different). But at least I admit my limitations to generalizability right up front.
In a grand bit of irony, I just discovered that my colleague, mentor, and friend Rob Potter (Indiana Telecom) also blogged regarding this article. It appears Dr. Potter needs to invest in Caller ID.
posted by Samuel D. Bradley at 6:00 PM
del.icio.us
FOLLOW SAM ON TWITTER
SUBSCRIBE TO THIS FEED
2 Comments:
But does admitting limitations to generalizability up front truely improve the research? Now we're talking about the presentation of the research, not the validity of the actual research. Perhaps you're saying that at least you have a accurate idea of the precise limitations than do researchers who use surveys.
I don't know what I'm talking about. I'm just a PhD wannabe workin for an ad agency and reading blogs.
Good points, of course. I guess that my point was that the goal of generalizing to the population is not possible. We admit that it is not possible. So we try to understand, for example, differences between messages. Does a message treatment have an effect in the lab? Then -- like most scientists -- we try to replicate the result.
When we can replicate the result in a different state with different subjects, we can use logical inference to generalize to other populations. Annie Lang wrote a great piece about this in 1996 in the Journal of Broadcasting & Electronic Media.
Post a Comment
<< Home