In what may prove a new level of absurdity, Mother Jones reported late last week about a new “study” by avowed anti-gun academic David Hemenway that purportedly “debunks the NRA’s favorite talking points” on gun issues. An analysis of the “study” the article refers to, however, shows it to be nothing more than a clever—and ethically suspect—attempt by Hemenway to generate anti-gun propaganda under the guise of objective academic research. And even in this, he fails miserably.
In this “study,” Hemenway identified firearms “experts” and then surveyed them on their views of various firearm-related issues such as concealed carry, background checks, firearm storage, etc. One would think such “experts” to come from a range of occupations or life experiences with firearms, but for Hemenway’s study, the “experts” were about 100 academics who had authored at least one article appearing in academic journals from 2011 to 2014. That’s right. Author one article in an academic journal and Hemenway considers you an “expert” on firearm issues.
Even more troubling, Hemenway initially recruited 358 such “experts,” but for his surveys only about 100 or so actually participated. This means that all his findings are suspect due to what researchers call “self selection bias.” With less than 1/3 of those invited opting in, it is very likely such participants were personally motivated to do so. Some might say even ideologically motivated to do so. In short, those who opted-in to take the surveys likely have particularly strong views and, knowing the source of the surveys, such views would tend to be those in line with Hemenway himself.
It is remarkable that even after stacking the deck in this way, Hemenway’s “findings” offer little support for the claim that the academics surveyed offer much expertise on gun issues. When asked whether they were familiar with research on various firearm related subjects, one-quarter to one-third of Hemenway’s “experts” routinely admitted not being very knowledgeable about research in these areas.
In short, Hemenway’s surveys amount to no more than a relatively small number of academics who are likely motivated to support Hemenway’s liberal worldview on guns. When one takes these major design flaws into account, it doesn’t really matter what the numbers say, the study is so flawed as to deny base credibility.
We remind Hemenway that the usual first step in the scientific method is for the researcher to develop a central hypothesis about some underlying question they want to study. Then, an ethical researcher will try to find or develop data to test this hypothesis, not to support an ideological position.
But it doesn’t appear that Hemenway feels constrained to follow this cardinal rule of the scientific process. Instead, Hemenway has an agenda—ban guns, ammo, etc.—which motivates his choice of research methodology to provide him the “results” that promote this end. The “study” of “experts” referenced by Mother Jones illustrates this motivated research design quite clearly.
Is liberal bias in the social sciences—as exhibited by Hemenway’s research—a problem? Last October, The New Yorker reported on a study by Jonathan Haidt that demonstrated how political ideology is skewed hard left in the social sciences. Very hard left. Haidt’s paper addressing this topic points out that ideological bias becomes particularly problematic in the data generation process itself, including the choice of research method to use. The dominant leftward slant in academia means that biased results can become the norm, if not outwardly encouraged. Such is the case with Hemenway’s panel of “experts.”
The Mother Jones article points out that Hemenway’s objective when putting together his “expert” survey was to undermine public debate on gun issues because such debate can be a “problem.” Further, nowhere in his surveys does he ask his “experts” whether they have an ideological point of view on guns, whether they own guns, if they identify with a particular party, or who they voted for in the last presidential campaign. These questions are routinely asked in public opinion surveys because such orientations influence personal views on highly political issues such as gun ownership and gun control. But we know nothing of Hemenway’s “experts,” other than they authored at least one article in an academic journal.
We’re left wondering if Mother Jones has completely given up on the idea of investigating newsworthy issues to provide their readership with useful—or at least interesting—factual information. It would have been useful to their readers, for example, to understand why Hemenway’s “study” is nothing of the sort and how bias can limit our understanding when the scientific method is kicked to the curb in favor of ideological grand-standing. We can hope for better in the future, but let’s not hold our breath in the meantime.
