The question of which colleges and universities lag (or lead) in providing access for low-income students has become a frontline issue in national discussions of educational opportunity. Legislative initiatives such as the bipartisan ASPIRE Act proposed in the U.S. Senate in 2017 would rank institutions based on their percentage of low-income students and impose financial penalties on institutions below a certain ranking. The latest version of the U.S. News World Report “Best Colleges” rankings includes measures of “social mobility.” Other news outlets like the New York Times and the Washington Monthly have prominently published rankings of colleges based on representation of low-income students while taking editorial positions excoriating (or applauding) individual institutions based on such measures.

Unfortunately, these initiatives ignore a thorny measurement challenge, one that can turn good intentions into penalties for institutions that are actually succeeding in providing opportunities for low-income students and trigger rewards for institutions that are less successful than they might appear. What makes measurement challenging is that different institutions face students whose family income and preparation differ.

Suppose that the University of Maine and the University of Connecticut enroll all students from their respective states who fit their academic standards, as defined by receiving a score on a college admissions test that is within the range of most students currently enrolled. Based on the different populations of their states, the University of Maine would draw 22 percent of its students from families with incomes below $40,000, while the University of Connecticut would draw only 10 percent of its students from this income range. The University of Maine would be judged much more favorably by popular measures of “opportunity” and rewarded by proposed accountability systems, while the University of Connecticut would be penalized. However, those rewards and penalties could not result from their differential success in enrolling low-income students since, in the example, all relevant students enroll at each university, regardless of their incomes. The universities would be rewarded based on their circumstances, not their behavior or effort.

More generally, popular measures of “opportunity” confound differences in universities’ effort with differences in their circumstances. Specifically, while the measures mean to measure a university’s effort to enroll well-qualified low-income students, what they actually measure can largely reflect differences in the pools of students from whom the universities could plausibly draw. The popular measures include a university’s share of students who receive federal Pell grants (the “Pell Share”), the share whose family income is in the bottom 20 percent of the national family income distribution (the “Bottom Quintile” measure), and the Intergenerational Mobility (“IGM”) measure, which is based on the percentage of enrolled students whose families are in the bottom 20 percent but who themselves end up in the top 20 percent of the national income distribution.