Weighting is designed to remove bias from a survey sample and make the results more closely represent the target population. For example, if the percentage of female respondents in a survey is less than that of the target population, we increase the contribution of the female responses by a factor (a weight) so that females are representatively proportionate to the target population in the weighted results.
For surveys with representative targeting, Google Surveys dynamically targets respondents to match the demographics of the internet population using their inferred demographics (age, gender, and geography). In the U.S., we use estimates for the national internet population from the U.S. Census Bureau’s 2017 Current Population Survey (CPS) Computer and Internet Use Supplement. The dynamic targeting uses the joint distribution of those dimensions, for example the percentage of 65+ year-old females in Wyoming. This dynamic targeting approach means that surveys are fairly close to the target distribution after running, and the amount of weighting needed is minimal.
After the survey has gathered responses, weights are applied to each response to match the breakdowns of age, gender, and region to those demographic breakdowns in the internet population. We apply weights to both representative and convenience targeted surveys, based on the demographics of internet users derived from a combination of Census data, local indicators, and Google-Gallup polls. Google Surveys uses an iterative ranking process to calculate weights for each demographic dimension. The demographic dimensions used for weighting are listed on the question info card. The weights applied after the fact will match the marginal distributions, meaning the percentage of females and the percentage of 65+ year olds. Weights are applied to the first question only. Weighted results are calculated by normalizing the weights available in each question as respondents screen out, drop off in the survey, or are filtered out in a survey report.
If population data is available for the targeted country and the survey respondent demographics don’t vary too far from our ground truth population demographics data, we show weighted data by default. If you prefer to see unweighted data, you can turn on Raw counts on your survey results page. When a survey is still gathering responses, it may not have enough responses for the weights to reliably match the demographics of the internet population. In those cases, we still show the weighted data by default, but there is a warning in the survey report to indicate that the weights may be unreliable. Similarly, when you filter a survey's results down to a subpopulation, like 18-24 year-olds excluding other ages, the warning message may show even if weighting was reliable for the survey prior to filtering. We don't apply weights to surveys when the respondent sample collected varies too greatly from the country's demographics, due to risk of over-skewing results. This may be more prevalent in convenience targeted surveys, which don't apply representative sampling.
Note: We cannot calculate weights for respondents who have unknown demographics for age, gender, or geography. Responses with unknown demographics are excluded when weighting is applied. Turning on Raw counts displays these responses with unknown demographics.
Please note that we don't weight the following types of surveys at this time:
- Surveys with any demographic targeting (age, gender, geography) that completed before August 19, 2017
- Surveys targeted to your website with Website Satisfaction
Learn more about weighting and the Surveys methodology.
Please contact us to request weighting for surveys from before October 15, 2016 or to ask any other questions about weighting.