Alert: This post does not contain empirical evidence. Rant/size ratio: high. Also, I got a bit carried away, sorry for the length.
One of the most aggrevating phrases I see in research papers, and subsequent news reports, is the one that goes something like:
This study consisted of subjects, selcted using a methodology and weighted to be nationally representative
Now there’s nothing wrong with this sentence, and the aim of achieving results which can be generalized to the whole population is an admirable one. The problem, for me, is that it is one of the few parts of any methods section in a paper that doesn’t require justification, explanation or referencing.
Which is to say, I have no idea how you generated the weights you used here. When the sampling methodology is a stratified or non-proportional one, I can assume that you re-weighted it back based on the stratifying and selection criteria. But that’s rarely the major problem with survey data.
Rather I am likley to be concerned about differential non-response, i.e. who didn’t answer the questionnaire. If weights are applied here, they have to be based on some characteristic of the individuals. To give an example: I note that I selected 3,000 people truly at random to survey about their fruit eating preferences, and only 2,000 of them accepted. I know that ~51% of respondents should have been women, but in fact 55% were. I can adjust for this by weighting each woman’s response a bit less, each man a bit more, to figure out how many people in the USA love oranges.
And here’s where people’s internal biases come in. Sex and age are the relatively obvious things to look at and adjust for. And probably location too – maybe more urbanites respond because you can find them easily, or more rural inhabitants because they’re always home at 8pm. After that, it all comes down to your point of view, and crucially which factors you think affect fruit preferences. I mean, if men and women on average have similar preferences, the hassle of weighting your sample isn’t really worth it.
Do you think that social factors matter, if so, which ones? Race/ethnicity might be do-able if you have census data on the proportion of people of each group living in an area. But SES isn’t routinely on the census, so if you think education, income or wealth matter, that’s going to be tough. Or maybe you think that genes are important, in which case, good luck. In the end, it all comes down to the perspective you have on the issue – something pointed out forcefully to me in the class that this book is based on.
And despite the foregoing rant, I don’t have a problem with weighting. But I do have a problem with us not being told what the weights are based on, in a nice simple manner. Preferably not buried in a refernce of a reference somewhere. And if reviewers insisted on it, this could be an easy change in editorial practice. FTW.
(Oh, and don’t even get me started on biases based on inaccurate responses and the fun of social desirability bias.)
Constrained Optimization 