Part One: Sociology’s Quantitative Methods and Studies are Seriously Flawed
Arguments based on and conclusions drawn from most findings in quantitative sociology are highly suspect, rarely useful for gaining an understanding of the social world, and of little practical use. Sociologists make poorly substantiated claims that go far beyond what their available data can tell them. This is primarily due to their reliance on statistical methods that are inappropriate to their data and the problems they study. At the risk of simplification, I will try to support these claims in a manner accessible to a general audience.
Statistical Analysis in Sociology
Analytic methods are the foundation of a scientific discipline. In order for any science to progress satisfactorily, its primary data analysis techniques must provide genuine insight into the phenomena its practitioners set out to investigate. In sociology by far the most frequently used method is statistical regression.
Regression is used to explain how an ‘outcome’ is accounted for by factors (called ‘variables’) that are hypothesized to be the best determinants of that outcome. Sociologists are asking why something happened. Their goal is to find causes and assess their relative importance.
For example, regression can be used to demonstrate how a person’s annual income (the ‘outcome’) can be explained by variables (e.g. their educational attainment, family background, employment history, etc.) thought to influence their income. Sociology journals are filled with these regression-based studies.
Radically Different Picture Depending on Minor and Arbitrary Decisions
In conducting regression-based studies, the investigator starts with a hypothesis - e.g. explaining why imprisonment or marriage rates vary across countries - and concludes with a regression results table in a published research article. To transit from the initial hypothesis to the data, findings and published study, the researcher has to go through a series of steps involving data acquisition, measurement, handling, and analysis:
(1) Start with a theoretical hypothesis about some social phenomena (i.e. the ‘research question’). It should be a question that can be adequately answered with available data.
(2) Proceed to an operational hypothesis aimed at elucidating the causal mechanisms which are thought to be central to the explanation of the phenomena.
(3) Gather data that bears on the phenomena, for use in the study. This step is crucial. Too often, sociologists rely on poorly measured survey and ‘life trajectory’ data, and problematic measures of ‘social processes’.
(4) Conceptualizations and measures of social variables vary markedly from study to study. Therefore, it is critical to weigh and adjust the data in light of what is known about the quality and representativeness of the measurements, and the consistency of different data sources with available research hypotheses.
(5) Clean and transform the data variables selected for the study. Research integrity is vital here. When data is missing, sociologists sometimes choose to guess, and then present the ‘estimates’ as objective measurements.
Each of these steps has its own challenges and moral hazards, and require making a number of arbitrary decisions. Minor changes in these decisions often make a big difference in the results rendering their substantive validity highly suspect.
The Folly of Causal Modeling in Sociology
Sociology as a ‘positivist’ discipline emerged, in the late 19th century, modeled on the hard sciences - specifically from the idea that things called ‘‘social facts’’ might be studied the way a chemist studies compounds or a biologist studies organisms. It is, however, a grave error to study society and the human domains, quantify human actions, and ‘search for causes’ using the same methods that are used in the natural sciences. The broad realm of human relations and social life is messy and complex. There are a huge number of variables coming from everywhere that can influence the outcome of interest making it difficult to acquire insights, ‘isolate causal effects’, and distinguish substantive from statistical significance.
It is incredibly hard to make strong theories and robust predictions about people acting in a broad social context. Therefore casual models in sociology are deeply flawed. It is not uncommon for investigators to capture only a small fraction of the variation in the outcome they are attempting to explain. In predicting the future, e.g. what will happen to a person, the most accurate models are sometimes barely better than naive guessing. Consequently, I would argue that there have been very few real gains - practical or intellectual - from causal modeling in sociology.
Sociologists Should Have More Modest Aspirations
As Professor Berk and other eminent scholars have pointed out, proper use of regression in sociology should not be about causal statements of the empirical world. Rather, it should have more modest aspirations such as identifying patterns, and characterizing noncausal associations and interactions in data. This ‘descriptive regression’ would at best suggest potential causal relationships that might contribute to an explanation which would require further exploration. Professor Gelman adds that researchers should forget about statistical significance (which is often spurious) and interpret regression coefficients (i.e. the degree of dependence of one variable on another) as relative comparisons.
The esteemed statistician David Cox concludes that establishing causality “needs a lot of care, as well as different kinds of investigations, and different sorts of evidence all assembled together”. With few exceptions, applied statistical work in sociology has not followed this advice.
Part Two: Replication Crisis in Sociology
The replication and reproducibility of research findings is a cornerstone of science. Reported results should be reproducible: applying the same analysis to the same data should yield the same outcome. It is critical for establishing a scientific claim, and key to the credibility of scientific knowledge.
Every new scientific discovery opens new problems, and new ways of experimenting. These new findings can be tested. Experiments are conducted to refute and falsify them. To be demonstrated as correct, results must be consistent and tested against reality in a wide range of settings. In this manner, real science is additive and cumulative. Later results modify earlier ones, thereby increasing their authority.
Replication is especially important in sociology (and most of the social sciences) which - unlike mathematics and the experimental sciences - lack a commonly accepted way to separate truth from falsehood.
Replication Crisis in the Social Sciences
The social sciences are experiencing a ‘replication and credibility crisis’. The publication process is broken, and readers cannot trust what they read in even the most prestigious journals. Across disciplines, much if not most statistical research is unreliable, simply wrong or fabricated. Quality control is limited and there is little incentive to look for errors before or after publication. There are numerous cases of hypotheses being revised in light of the data, and documented cases where sociologists did not disseminate warnings that their studies could not be replicated despite knowing that there were major problems with the data.
Professor Gelman and others explain how this state of affairs results from:
Methodological shortcomings.
Low quality and biased/noisy data.
Pervasive misuse of statistics including cooked up data, generating data to support preconceptions, and the testing of hypotheses known beforehand to be false.
Publication bias, and an academic culture where publishing matters more than truth.
Casual attitude toward measurement.
A system of scholarly journal review where vast majority of review effort goes to papers that nobody reads.
Sociology is the Chief Offender and Unbothered by “Bad Science”
The replication crisis is especially severe in sociology where leading sociology journals have publish studies with weak designs, serious statistical flaws and improper software implementations. Researchers were unable to recreate the results in published articles using the raw data and code of the original author.
Studies that are found to be false continue to get cited (often uncritically in the media) and widely circulated, especially when the conclusions support the field’s progressive ideological agenda. Erroneous conclusions are drawn from them, and policy discussions distorted. The bar to getting corrections published is very high, and retractions are rare.
In contrast to the other social science disciplines, sociologists have expressed little concern over their replication crisis. The culture of the discipline is such that once a paper is published it is considered Truth. Some sociologists reject the replication of studies in principle, claiming it is undemocratic and increases the barriers to conducting research.
Professor Cohen points out that, outside of sociology, it is basic social science practice that if you can’t provide data, code, and other research materials, you offer an explanation as to why that is. Top journals in the more serious social science fields (economics, psychology, etc.) require replication code and that the editors verify that the claimed results are replicated.
There are hundreds of studies in psychology and political science journals explicitly attempting to replicate the findings of prior studies, but few in sociology journals. Only 28 percent of sociologists could provide a replication package. None are members of the Institute for Replication.
Conclusion: A Stagnant Field Where Most Findings are‘ Bunk’
As a result of these methodological flaws and lack of replication, few research findings in sociology are relevant to other disciplines, policy decisions or the real world. No one believes their claims. It is no surprise that the Economist Magazine concluded that “74.6 Percent of Sociology is Bunk”. Ultimately I contend that in many respects sociology is largely a stagnant field. Surely, any field that does not advance is not healthy, and probably wasn’t worth creating or studying in the first place.