There has been no dearth of commentary in the last two weeks on the now-infamous “Reproducibility Crisis.” The original in Science was about psychological studies, but no one with half a brain doubts that reproducibility (and fraud) problems extend to sociology, criminology, Gender and Ethnic “Studies,” and even nutrition and health.
Among the more intelligent looks, Scott Alexander thoroughly explains yes, it’s a crisis in response to this head fake in the NYT by some psych prof with her fingers in her ears. Steve Sailer, a former market researcher, wonders if psychology is more like astronomy, or marketing research? The latter only wants results now; psychology(in a pretend quest to be physics) seeks Permanent Laws of the Entire Universe.
Sailer does point out that not all areas of psych seem to have a replication crisis (TRIGGER WARNING – CRIMETHINK AHEAD!):
By the way, some fields in psychology, most notably psychometrics, don’t seem to have a replication crisis. Their PR problem is the opposite one: they keep making the same old predictions, which keep coming true, and everybody who is anybody therefore hates them for it, kill-the-messenger style. For example, around the turn of the century, Ian Deary’s team tracked down a large number of elderly individuals who had taken the IQ test given to every 11-year-old in Scotland in 1932 to see how their lives had turned out. They found that their 1932 IQ score was a fairly good predictor. Similarly, much of The Bell Curve was based on the lives of the huge National Longitudinal Study of Youth 1979 sample up through 1990. We now have another quarter a century of data with which to prove that The Bell Curve doesn’t replicate. And we even have data on thousands of the children of women in the original Bell Curve sample. This trove of data is fairly freely available to academic researchers, but you don’t hear much about findings in The Bell Curve failing to replicate.
And there you are friends: It’s not exactly a reproducibility crisis, it’s a crisis of Progressivism. The entire Prog edifice, that’s supposed to be based on a concrete foundation of Science, is tottering quite badly, because social science is full of big cracks that are widening daily.
This reminded me of something I wrote over four years ago, from a different angle, while ruminating on some of Robert Heinlein’s predictions, in both non-fiction articles and in his science fiction books and stories. I’m an unabashed Heinlein fan, but that has little to do with the piece, which is reproduced (slightly edited) below. It’s not about problems with methodology or even fraud in the social sciences. It’s about their abject failure to deliver what they promised–which is still true as of noon today:
(Original written in May 2012)
The Failure (So Far) of Heinlein’s Vision of “Social Science”
I recently read an article at The Weekly Standard by Andrew Ferguson, “The New Phrenology.” Subtitled “How liberal psychopundits understand the conservative brain,” the piece goes into some detail about the numerous news stories most of us have probably seen lately, with titles like that of Chris Mooney’s book The Republican Brain: The Science of Why They Deny Science—and Reality.
With that background, I want to make explicit that this is not a political post, nor is it an analysis of the psychology of any particular group. The meta point of “The New Phrenology” has been made powerfully by a number of others–the current state of the “social sciences” is barely scientific, after 100 years or more of effort. The mere gloss of scientism is now provided by colored pictures of brain images, publication in peer reviewed journals and the use, and misuse, of statistics.
One hundred years ago Robert A. Heinlein was about to turn five and people of a wide variety of political and philosophical views, from Freud to H. G. Wells to Woodrow Wilson, believed that economics, psychology and sociology were taking their first firm steps toward becoming true sciences, where national and world economies would be managed in steady prosperity without booms and busts, criminals and the mentally ill would be reformed or healed through drugs and therapy, and populations would be managed toward happiness through education, advertising and techniques like mass hypnosis and official propaganda. Eventually, all of these efforts would be put on a firm base of physics and neuroscience and mathematical statistics, with formulas fed into computing devices and the right answers for societal management coming out.
These ideas can be seen clearly in many of Heinlein’s early works. Indeed, the Future History takes place against a background where this social management is often simply assumed and only mentioned en passim when necessary. In other instances, it is made explicit as an important part of the story, as with the extensive explanation of economic management and Monroe Alpha Cliff’s work near the beginning of Beyond This Horizon or the debate about using mass hypnosis to recondition the populace toward freedom at the end of “If This Goes On–“. In Methuselah’s Children there is mention of statistically rating the impact of words, and the strategic planting of useful rumors based on mathematical formulae. For a good short explication of this idea under the general heading of “social engineering” see the “Logos” section of this article on “If This Goes On–“ by Bill Patterson.
Here in 2012 I would argue that these fields have made very limited progress toward being “science.” In economics, the worldwide Big Bust of the last four years provides compelling evidence that legions of Ph.D. economists are subject to forces far beyond their control, their manipulations of money and interest too much, too soon or too little, too late. Criminals are still with us, in plenty, and while the soma of a wide variety of “anti-depressants” masks the symptoms of perhaps 20% of the American populace, all the billions and indeed, trillions of dollars expended on “scientific research” into education, reform of prisoners and the proper raising of children seems to have merely, mostly maintained the status quo ante in these areas.
But back to “The New Phrenology.” Mostly believing that the mind is just a useful, or useless fiction, the reductionists have deployed the truly wonderful tools of modern medical imaging in the study of the brain and declared the colored pictures taken therefrom the answer to a broad number of questions. Why do people do what they do? Hook them to a forest of electrodes and ask them question or show them some naughty pictures, see what lights up, gather some stats and you’ve got yourself a peer reviewed journal article that will help further your career path in the Academe!
I do not claim that this kind of study is necessarily useless, biased, wasteful or harmful. It may be that discoveries from these techniques really will result in a better life for us and our children.
So far though, what we’ve got is that drug users’ pleasure centers light up when they use, that brain scan color pictures prove that there is no free will, and that political” conservatives” are a fearful, authoritarian bunch. I don’t claim to know the entire field–I just read the newspapers, and that’s what I’m seeing.
It’s a long, long way from the vision of Heinlein and others, during those early, heady days, that all of this research would eventually give us scientific solutions to social problems.
Another side of Heinlein, the rugged proponent of individualism, liberty and the generally untamable nature of Man, would probably be delighted at these developments. So far, that’s the side that seems to be winning in the real world.