Rebuttal: Not Known That Study Bias is Worse For Online

(Sidebar to Jakob Nielsen's column on impact of study methodology on real-world relevance of results.)

Marianne Foley, Senior Vice President of IPSOS-ASI Interactive, writes:

In both the television and online databases, respondents are directed to a program or to content that has been selected for them. In each case, the program or content is held constant across all respondents and advertising is embedded within it. The bias of "artificiality" is built into both databases. This controlled environment has been a widely accepted approach in evaluating television, print and movies. The degree to which the artificial experience " bias" impact any specific media more or less than another has yet to be quantified.

Jakob's response: I agree that both parts of the study have a bias, and I also agree that it is not known for sure how large the bias is in either case. My argument, based on observation of Web users' behavior, is that the bias is likely to be particularly large in their case, particularly since "banner blindness" is a very common finding in other studies. But the traditional conclusion remains: "more research needed" since it is true that we don't know the exact magnitude of the bias.