UW–Madison researchers put NIH grant review process under microscope

The National Institutes of Health’s system for selecting research projects may be considered the gold standard for equitably awarding funding, but that hasn’t kept the agency from dispatching three University of Wisconsin–Madison professors to probe the system for bias.

“The NIH peer review system is viewed by other countries and organizations as the ultimate review system for research,” says Molly Carnes, a women’s health researcher and UW–Madison professor of medicine. “That’s why it is especially important to study that system. Other institutions look to the NIH as setting the benchmark for peer review.”

Carnes, psychology professor Patricia Devine and English and sociology professor Cecilia Ford will use a $4.8 million Transformative Research Award from NIH — which annually invests more than $30 billon of public funds in medical research — to determine whether scientists judging the work of fellow scientists award more money to particular genders, races or researchers at prestigious institutions.

The UW–Madison group’s work follows on the heels of a 2011 study that showed black applicants for NIH funding were 13 percent less likely to secure grants than their white peers.

“There is a difference in the funding rate. That seems clear,” says Devine, whose research typically focuses on prejudice. “The question is whether there is bias, in which case it would be important for the NIH to know so that we could take steps to address it.”

If reviewers do show preference for certain groups over others, it may not be intentional.

“If it’s not, then we have to think about why it is that black researchers or women applicants don’t come to the point where their grants are funded,” Devine says. “It might be training or the schools they go to — things that precede the actual writing of the grant.”

The group will analyze the text of actual reviews written by the researchers who conduct peer review of grant proposals presented to NIH, looking for differences that may be connected to factors such as the applicant’s gender, race or employer.

They will also manipulate those demographic factors on grant proposals volunteered by researchers to see whether results shift based on what little the reviewers know about the applicants.

Ford will study the interplay in the in-person review sessions, the last step before the peer review group sends its rankings on grant proposals on to NIH.

“That interaction, when the reviewers finally meet to discuss the merits of proposals, is kind of a black box,” says Ford, a linguist who studies language’s role in social organization. “We want to study it not necessarily to uncover bias, but to understand what the mechanisms are by which a proposal gets considered more or less, or how people come to agree or disagree on those decisions.”

Carnes has long been curious about the factors that keep some groups from growing (in number, at least) in academic research.

“I’ve been particularly interested in what happens at evaluation junctures in an academic career,” she says. “On the other side of those points there are always fewer women and underrepresented minorities, and one of the important junctures is the grant award process.”

The enthusiastic response to a recent pilot project was a pleasant surprise to the interdisciplinary trio. A deep pool of scientists offered their time and copies of grant proposals to the effort.

“I think people are particularly interested in the idea of institutional bias — ‘If you’re from Harvard, of course you’re going to do well,’ they think,” Carnes says. “That sort of bias has never been experimentally demonstrated, but there is a sense that it exists anyway.”

NIH, which traces its roots to a one-room lab opened in 1887 as part of the Marine Hospital Service, welcomes the research as part of a continuing effort to ensure equitable funding.

“Ever since the discovery of some award disparities, NIH has been working to find the cause,” says Richard Nakamura, director of the agency’s Center for Scientific Review. “We’re pleased researchers from the University of Wisconsin–Madison received a grant to help us assess the possibilities of bias in our peer review groups. Finding a cause is the first step in finding a cure.”

The researchers are confident that NIH is prepared to take a cure, if necessary.

“We know they support changes if necessary,” Ford says.

“They’re not trying to squelch or hide anything,” Carnes says. “If it’s rigorous science, they want to see the results.”

“They want to be confident that the process they use to identify the best research is identifying the best research,” Devine says.