Remember the Beatles refrain that goes “Boy, you’re gonna carry that weight a long time?”
One of the big risks in market research is having to pile too much weight on too few respondents. When a study requires a certain number of responses from a given demographic, but just can’t get enough valid completes, the data will distort reality instead of illuminating it. Data from too few qualified panelists gets too heavily weighted — sending the study down the long and winding road to unreliability.
A recent article in the New York Times tells a cautionary tale of what can happen when a thin sample becomes too heavily weighted.
A respected national online political poll was unable to reach enough African American Millennials as it tracked the presidential race. The pollsters found themselves forced to put a drastic amount of weight on the single African American respondent they were able to reach in the 18- 21-year-old age group. He happened to be a Donald Trump supporter — and therefore highly unrepresentative of African Americans overall, who overwhelmingly favor Hillary Clinton. The poll fell out of sync with other national election surveys — and, it appears, with the facts on the ground.
Consumer surveys that can’t access the right respondents face the same burden, and inevitably will fail to reflect reality. It happens all the time, and the problem has become especially troublesome for tracking studies. But it doesn’t have to be that way.
Online surveys designed for desktop and laptop computers and relying on email notifications are faltering because the public they need to reach has embraced today’s technology, the smartphone, over yesterday’s — the personal computer. The solution for researchers is simple: they can go where the public has gone and embrace the smartphone, too.
Recognize that the U.S. public has migrated to mobile, and that it responds faster and more reliably to in-app survey alerts than to email notifications. Seek respondents on their ever-present smartphones and you’ll quickly and efficiently find exactly who you need.
Our panelists include fully representative, reliably accessible numbers of all the supposedly “hard to reach” groups — African Americans, Hispanics, Millennials and parents of young children — who are especially engaged with their phones. When they are present and accounted for, there’s no need for distorting weighting.
Here’s some (unweighted) data that tells the story:
- Among Millennials, majorities of African Americans (57.1%) and Hispanics (51.6%) spend five hours or more per day on their smartphones.
- 92.3% of Millennials use their phones at least several times a day, compared to 32.1% for laptops.
- Only one-third of Millennials (33.1%) say they use a desktop computer at least once a day.
(These are findings from MFour’s Millennial Insights Project, a demographically representative recent survey of 1,000 U.S. Millennials.)
A 2015 study by the Pew Research Center found that “smartphone ownership is nearing the saturation point with some groups: 86% of those ages 18-29 have a smartphone, as do 83% of those ages 30-49.”
Is there any good reason for market research, whose job is to understand what people think and feel, not to reach out to the public on the device it has chosen overwhelmingly for expressing thoughts and feelings?
Granted, older demographics have not gravitated quite so thoroughly to mobile. But even among them the numbers are growing.
In tracking studies that need to encompass all age groups, it can still make sense to continue using online surveys for the older demographics that remain more comfortable with them. Researchers can then blend those results with mobile data from the mobile-oriented majority.
Through integration and calibration, the validity of past online tracking can be preserved through a gradual transition to mobile research. Continuity with historic tracking data need not be broken.
Yes, this is weighting. But it’s judicious weighting that increases tracking studies’ reliability while helping them move gradually into the mobile era.
Change is never easy, but the rewards of all-mobile, app-based research are too great, and the consequences of remaining behind the times with online surveys and email notifications are too drastic. Soon, political pollsters and consumer researchers alike will be happily mobile and relieved to work with data that’s not burdened by a lot of unhealthy weight.
To get a personal demo of how app-driven mobile works, and how it will work for you, please click here.