Category: metrics

This review essay by Christopher Newfield and Heather Steffen makes for an entertaining and incisive read. The review is of three books: O’Neil’s Weapons of Math Destruction, Espeland & Sauder’s Engines of Anxiety and Merry’s The Seductions of Quantification, with a focus on how these come to bear on the turbocharged audit culture of academia. The diagnosis, through a review of the three books of the shortcomings of what might be called the latest ‘quantitative revolution’ of ‘data science’ is pushed further into a reflexive diagnosis of a genre of talking about such things as Metrics Noir. A lovely term – I hope it gains traction. I definitely think there’s folk writing ‘Metrics Noir’ in geographyland.

I’ve blockquoted a nicely chewy bit below but I recommend reading the whole thing.

All of these scholars are well aware of the value of numbers. Numbers allow for abstract picturing of groups, societies, and cities. They regularize anomalies and exceptions, and allow us access to invisible worlds, social and physical alike. Numbers support distributed cognition and collective intelligence. Both are desperately needed in a world damaged by human stupidity. But quantification in its many forms now operates within a complex metrics culture — a contradictory and contested battleground, as these three books explain. Together, they offer an understory that we could call metrics noir.

In the first place, numerical measurement can too readily take on an unquestioned objectivity. It’s an easy mistake to make, because scientists and other experts have a longstanding reputation for unbiased handling of facts, insured by methodological procedures not accessible to the layperson. This objectivity bias is hardened by the production of indicators via expert negotiations hidden from public view, which means that metrics aren’t seen as emerging from the intellectual compromises and culturally conditioned choices that go into their making. The public can remain blissfully ignorant of their baked-in assumptions — say, the idea that the poor are more likely than the middle class to commit crimes. Criticism is easily dismissed as resting on shaky subjective grounds.

Second, metrics culture reinforces the perceived inadequacy of qualitative expertise, of the “liberal professions” that rely on interpretive skills grounded in social, philosophical, and historical learning. If a dean can make promotion or funding decisions by looking at a dashboard of indicators that compare her faculty members to those across the campus and the country (grant dollar totals, prizes, publication rates, citation counts), then he or she need not weigh complex quasi-imponderables and judge the strange mixture of ingredients that make up careers and disciplines. Twenty years ago, Michael Power noted a subtle but determinate feature of the “audit society”: audit slowly weakens judgment, and management becomes a matter of applying formulae whose opacity supplies a false objectivity.

With indicators ascendant over judgment itself, and tied to complicated, obscure, or proprietary procedures, metrics can pacify the interpretive powers of the public and professionals alike. The subjects of assessment rarely interact with quantitative procedures and never demand their abolition. This is a third tendency of metrics culture. Merry discusses “data inertia,” and all these authors note the near-impossibility of putting a finished indicator back in the oven. Policymakers have no stomach for revising indicators beyond the routine tweaking of weightings one sees in U.S. News and similar rankings. Very few scholars analyze the politics of such interventions or detail the losses they create for institutions, scholars, or students. Understanding the history of indicator formation is a minority knowledge project whose negative implications can be brushed aside even when their validity is acknowledged. Although reformers demand that metrics be used only in context, in conjunction with other information, and in collaboration with those being evaluated, metrics weaken the validity of exactly the forms of knowledge that are meant to check them. We thus encounter a Foucauldian nightmare, in which critiques of the ranking system only serve to make it stronger.

Fourth, indicators help create the inequality they measure, while assuring their consumers that the inequality is a natural, preexisting fact. They do this by ignoring distinctive qualities that cannot be quantified and compared. For example, not only is a legal clinic that focuses on the problems faced by recovering opioid addicts not likely to be esteemed or even seen in standard rankings, but the training for such work will be devalued if it is not already a regular component of the top law programs — its very uniqueness will make it incomparable across programs. To put this two-stage process somewhat formally: the set of relevant qualities is narrowed to a common denominator associated with the top schools, and the quantified hierarchy that results then overwhelms the underlying particularities of each school. The gap between the indicators and the actual qualities of a given school is ignored in favor of the gaps among the various institutions. The dominant quality of each school becomes its place in the hierarchy.

The wider effect of all this is particularly damaging in education: ranking renders a large share of any sector — community colleges, chemistry doctoral programs, business schools — inferior to the top programs, and therefore implicitly defective. The deficiencies that rankings always create then justify unequal respect and, more importantly, unequal funding. Rankings undermine the general provision across institutions that created the famous quality of the US public university system, encouraging instead more investment at the top. The general effect is that the rich get richer, which is precisely what has happened in American higher education in the three decades since the U.S. News rankings first appeared. The rise of rankings didn’t cause the breakdown in public funding, but it has naturalized the inequality that results.

The good news, as these books show, is that numbers don’t need to be used as we use them now. But for real change to take place, the wider society has to become involved in the conversation. These books do an excellent job of helping make that happen.

Remaking the University: Metrics Noir – Christopher Newfield and Heather Steffen