Much has been written in recent weeks about client concerns over the quality of market research work conducted using online “access panels.” (That’s the term popular in Europe, describing an online panel from which survey respondents may be drawn.) Concern seems centered on two key issues: (1) the problem of so-called “professional respondents;” and (2) the question of repeatability: will the same survey conducted against the same panel on different occasions lead to the same business decision?
The Advertising Research Foundation, under the stewardship of President/CEO Bob Barocci and research guru Bill Cook, has formed an Online Research Quality Council to address these concerns. Unlike some other associations whose membership is comprised exclusively of research companies, the ARF constituencies include advertisers, ad agencies, academia, and media companies in addition to research suppliers, providing the broadest base yet for such an initiative. I’m on the ORQC Steering Committee, along with colleagues from Ipsos, TNS, NPD, Harris Interactive and Synnovate.
One of the challenges in assuring high levels of research quality in online work is that the field of providers is crowded. Marketers have an expectation that the research supplier should be accountable for quality, and should refuse to take on projects where the specifications and cost pressures would lead to compromises in research quality. The problem is, though, that when the best-practices companies draw these lines in the sand, there will always be someone else happy to do the work and send an invoice. So an unfortunate outcome of a commitment to research quality on the part of the leaders in the space will inevitably be a shift in volume of work from high-quality to lesser-quality providers. That is a problem, and one which an industry-wide commitment to a standard set of quality metrics can help solve.
For the part of Comscore, this focus on research quality in the online panel space has only upside. We are able to identify the survey-taking behavior of our panelists in other panels (Gian Fulgoni’s work in this area has been well-received and oft-quoted), and we have been able to demonstrate empirically the behavioral differences of heavy survey responders as compared to the general online population. Indeed, we are able to eliminate these heavy responders from our surveys, or to control for their impact. As for repeatability of results, since we can empirically measure the online behavior of our panelists, much of our research is wholly immune from the vagaries associated with self-reported behavior as collected from respondents through questionnaires. Frankly, I think the industry focus on quality will only increase the demand for Comscore’s services.
The first meeting of the ORQC was held at the Time Warner Center on September 10 with almost 200 persons in attendance (and more on the phone). Four committees have been formed: the “Define Quality” committee; the “Quality Matters” committee; the “Identify and Adopt Metrics” committee; and the “Funding and Outreach” committee. If your company is an ARF member, I would urge you to join the council and volunteer for one of more of the committees.
In addition, the ORQC has formed a Client Advisory Committee, which thus far includes Dr. Tom Evans (ESPN); Leroy Leiker (Bank of America); Amy Raihill (Shell); Michele Salazar (McDonald’s); James Mendelsohn (Capital One); Bob Sanders (Allstate); Keith Stevens (Kraft); and John Willard (Bayer).