Unsighted: Why Some Important Findings Remain Uncited

1 Aug

Poring over the first 500 citations of the over 900 citations for Fear and Loathing across Party Lines on Google Scholar (7/31/2020), I could not find a single study citing the paper for racial discrimination. You may think the reason is obvious—the paper is about partisan prejudice, not racial prejudice. But a more accurate description of the paper is that the paper is best known for describing partisan prejudice but has powerful evidence on the lack of racial discrimination among white Americans–in fact, there is reasonable evidence of positive discrimination in one study. (I exclude the IAT results, weaker than Banaji’s results, which show Cohen’s d ~ .22, because they don’t speak directly to discrimination.)

There are the two independent pieces of evidence in the paper about racial discrimination.

Candidate Selection Experiment

“Unlike partisanship where ingroup preferences dominate selection, only African Americans showed a consistent preference for the ingroup candidate. Asked to choose between two equally qualified candidates, the probability of an African American selecting an ingroup winnerwas .78 (95% confidence interval [.66, .87]), which was no different than their support for the more qualified ingroup candidate—.76 (95% confidence interval [.59, .87]). Compared to these conditions, the probability of African Americans selecting an outgroup winner was at its highest—.45—when the European American was most qualified (95% confidence interval [.26, .66]). The probability of a European American selecting an ingroup winner was only .42 (95% confidence interval [.34, .50]), and further decreased to .29 (95% confidence interval [.20, .40]) when the ingroup candidate was less qualified. The only condition in which a majority of European Americans selected their ingroup candidate was when the candidate was more qualified, with a probability of ingroup selection at .64 (95% confidence interval [.53, .74]).”

Evidence from Dictator and Trust Games

“From Figure 8, it is clear that in comparison with party, the effects of racial similarity proved negligible and not significant—coethnics were treated more generously (by eight cents, 95% confidence interval [–.11, .27]) in the dictator game, but incurred a loss (seven cents, 95% confidence interval [–.34, .20]) in the trust game. There was no interaction between partisan and racial similarity; playing with both a copartisan and coethnic did not elicit additional trust over and above the effects of copartisanship.”

There are two plausible explanations for the lack of citations. Both are easily ruled out. The first is that the quality of evidence for racial discrimination is worse than that for partisan discrimination. Given both claims use the same data and research design, that explanation doesn’t work. The second is that it is a difference in base rates of production of research on racial and partisan discrimination. A quick Google search debunks that theory. Between 2015 and 2020, I get 135k results for racial discrimination and 17k for partisan polarization. It isn’t exact but good enough to rule it out as a possibility for the results I see. This likely leaves us with just two explanations: a) researchers hesitate to cite results that run counter to their priors or their results, b) people are simply unaware of these results.

Addendum (9/26/2021): Why may people be unaware of the results? Here are some lay conjectures (which are general and NOT about the paper I use as an example above; I only use the paper as an example because I am familiar with it. See below on the reason):

  1. Papers, but especially paper titles and abstracts, are written around a single point because …
    1. Authors believe that this is a more effective way to write papers.
    2. Editors/reviewers recommend that the paper focus on one key finding or not focus on some findings — via Dean Eckles. (see the p.s. as well) The reason why some of the key results didn’t make the abstract in the paper I use as an example is, as Sean shares, because reviewers thought the results were not strong.)
  2. Authors may be especially reluctant to weave in ‘controversial’ supplementary findings in the abstract because …
    1. Sharing certain controversial results may cause reputational harm.
    2. Say the authors want to instill belief in A > B. Say a vast majority of readers have strong priors about: A > B and C > D. Say a method finds A > B and D > C. There are two ways to frame the paper. Talk about A > B and bury D > C. Or start with D > C and then show A > B. Which paper’s findings would be more widely believed?
  3. Papers are read far less often than paper titles and abstracts. And even when people read a paper, they are often doing a ‘motivated search’—looking for the relevant portion of the paper. (Good widely available within article search should principally help here.)

p.s. All of the above is about cases where papers have important supplementary results. But as Dean Eckles points out, sometimes the supplementary results are dropped at reviewers’ request, and sometimes (and this has happened to me), authors never find the energy to publish them elsewhere.