An interesting project from Mushon Zer-Aviv, with shades of Canguilhem:
I must confess. I did that lazy thing by asking on Twitter for something perhaps I could’ve found out with a little more effort (I have also been relatively lax at actually writing this blogpost about it!). However, it actually resulted in something interesting.
I had just completed a (peer) review for a journal published by Sage and was asked if I wanted to sign up for “Publons” a system that supposedly let you gain ‘recognition’ for your peer reviewing. So, I did sign up but then thought: “hang on…” and this prompted by question on Twitter:
Academic twitter: Has anyone used/signed up for Publons? Can you recommend (or discourage) using it? Thx!— Sam Kinsley (@samkinsley) August 7, 2018
Click through to read the replies. It’s a great sharing of knowledge and expertise around the process of peer review, with plenty of contributions from colleagues with positions as editor in various journals. For example, this from Martin Coward begins to get at some of the issues.
Also, I feel it overstates benefits for authors (i.e. no UK institution gives any significant weight to your reviewing record in the promotion process). And editors can't rate reviews as they can in scholar one – so there is no sense of the quality of the reviews it records.— Martin Coward (@martincoward) August 8, 2018
I won’t try to summarise, or indeed embed, all of the things that were said, please do click through for the whole exchange. I do want to, very briefly, reflect upon this longstanding concern with making peer review ‘work’. The concerns that “Publons” purports to address are real. Peer review is the life-blood of academic publishing but is assumed, rather under-valued (by publishers, some colleagues and institutions) and, it seems, the constant frustration of editorial board members. As my former colleague Prof. Martin Weller has observed this labour represents rather a lot of unrecognised and under-appreciated investment.
An attempted ‘technological fix’ is, of course, not new. There have been various attempts to think about this over the years. When I was working with Martin on ‘digital scholarship’ (see his excellent open access book) the trial of an ‘open’ peer review system for Nature was a relatively recent talking point.
It is not a novel argument but it seems to me that unless and until academics, publishers and institutions stop thinking about lots of forms of labour as a part of a perceived ‘exceptionalism’ and ‘privilege’ of academic life (which I think, if it did ever exist for a few people, is long gone) we are more-or-less doomed to rehearse this debate ad infinitum.
There’s a really nice piece by Tarleton Gillespie in Issue 04 of Logic themed on “scale” that concerns the scale of social media platforms and how we might understand the qualitative as well as quantitative shifts that happen when things change in scale.
But the question of scale is more than just the sheer number of users. Social media platforms are not just big; at this scale, they become fundamentally different than they once were. They are qualitatively more complex. While these platforms may speak of their online “community,” singular, at a billion active users there can be no such thing. Platforms must manage multiple and shifting communities, across multiple nations and cultures and religions, each participating for different reasons, often with incommensurable values and aims. And communities do not independently coexist on a platform. Rather, they overlap and intermingle—by proximity, and by design.
The huge scale of the platforms has robbed anyone who is at all acquainted with the torrent of reports coming in of the illusion that there was any such thing as a unique case… On any sufficiently large social network everything you could possibly imagine happens every week, right? So there are no hypothetical situations, and there are no cases that are different or really edgy. There’s no such thing as a true edge case. There’s just more and less frequent cases, all of which happen all the time.
No matter how they handle content moderation, what their politics and premises are, or what tactics they choose, platforms must work at an impersonal scale: the scale of data. Platforms must treat users as data points, subpopulations, and statistics, and their interventions must be semi-automated so as to keep up with the relentless pace of both violations and complaints. This is not customer service or community management but logistics—where concerns must be addressed not individually, but procedurally.
However, the user experiences moderation very differently. Even if a user knows, intellectually, that moderation is an industrial-sized effort, it feels like it happens on an intimate scale. “This is happening to me; I am under attack; I feel unsafe. Why won’t someone do something about this?” Or, “That’s my post you deleted; my account you suspended. What did I do that was so wrong?”
This looks interesting…
This review essay by Christopher Newfield and Heather Steffen makes for an entertaining and incisive read. The review is of three books: O’Neil’s Weapons of Math Destruction, Espeland & Sauder’s Engines of Anxiety and Merry’s The Seductions of Quantification, with a focus on how these come to bear on the turbocharged audit culture of academia. The diagnosis, through a review of the three books of the shortcomings of what might be called the latest ‘quantitative revolution’ of ‘data science’ is pushed further into a reflexive diagnosis of a genre of talking about such things as Metrics Noir. A lovely term – I hope it gains traction. I definitely think there’s folk writing ‘Metrics Noir’ in geographyland.
I’ve blockquoted a nicely chewy bit below but I recommend reading the whole thing.
All of these scholars are well aware of the value of numbers. Numbers allow for abstract picturing of groups, societies, and cities. They regularize anomalies and exceptions, and allow us access to invisible worlds, social and physical alike. Numbers support distributed cognition and collective intelligence. Both are desperately needed in a world damaged by human stupidity. But quantification in its many forms now operates within a complex metrics culture — a contradictory and contested battleground, as these three books explain. Together, they offer an understory that we could call metrics noir.
In the first place, numerical measurement can too readily take on an unquestioned objectivity. It’s an easy mistake to make, because scientists and other experts have a longstanding reputation for unbiased handling of facts, insured by methodological procedures not accessible to the layperson. This objectivity bias is hardened by the production of indicators via expert negotiations hidden from public view, which means that metrics aren’t seen as emerging from the intellectual compromises and culturally conditioned choices that go into their making. The public can remain blissfully ignorant of their baked-in assumptions — say, the idea that the poor are more likely than the middle class to commit crimes. Criticism is easily dismissed as resting on shaky subjective grounds.
Second, metrics culture reinforces the perceived inadequacy of qualitative expertise, of the “liberal professions” that rely on interpretive skills grounded in social, philosophical, and historical learning. If a dean can make promotion or funding decisions by looking at a dashboard of indicators that compare her faculty members to those across the campus and the country (grant dollar totals, prizes, publication rates, citation counts), then he or she need not weigh complex quasi-imponderables and judge the strange mixture of ingredients that make up careers and disciplines. Twenty years ago, Michael Power noted a subtle but determinate feature of the “audit society”: audit slowly weakens judgment, and management becomes a matter of applying formulae whose opacity supplies a false objectivity.
With indicators ascendant over judgment itself, and tied to complicated, obscure, or proprietary procedures, metrics can pacify the interpretive powers of the public and professionals alike. The subjects of assessment rarely interact with quantitative procedures and never demand their abolition. This is a third tendency of metrics culture. Merry discusses “data inertia,” and all these authors note the near-impossibility of putting a finished indicator back in the oven. Policymakers have no stomach for revising indicators beyond the routine tweaking of weightings one sees in U.S. News and similar rankings. Very few scholars analyze the politics of such interventions or detail the losses they create for institutions, scholars, or students. Understanding the history of indicator formation is a minority knowledge project whose negative implications can be brushed aside even when their validity is acknowledged. Although reformers demand that metrics be used only in context, in conjunction with other information, and in collaboration with those being evaluated, metrics weaken the validity of exactly the forms of knowledge that are meant to check them. We thus encounter a Foucauldian nightmare, in which critiques of the ranking system only serve to make it stronger.
Fourth, indicators help create the inequality they measure, while assuring their consumers that the inequality is a natural, preexisting fact. They do this by ignoring distinctive qualities that cannot be quantified and compared. For example, not only is a legal clinic that focuses on the problems faced by recovering opioid addicts not likely to be esteemed or even seen in standard rankings, but the training for such work will be devalued if it is not already a regular component of the top law programs — its very uniqueness will make it incomparable across programs. To put this two-stage process somewhat formally: the set of relevant qualities is narrowed to a common denominator associated with the top schools, and the quantified hierarchy that results then overwhelms the underlying particularities of each school. The gap between the indicators and the actual qualities of a given school is ignored in favor of the gaps among the various institutions. The dominant quality of each school becomes its place in the hierarchy.
The wider effect of all this is particularly damaging in education: ranking renders a large share of any sector — community colleges, chemistry doctoral programs, business schools — inferior to the top programs, and therefore implicitly defective. The deficiencies that rankings always create then justify unequal respect and, more importantly, unequal funding. Rankings undermine the general provision across institutions that created the famous quality of the US public university system, encouraging instead more investment at the top. The general effect is that the rich get richer, which is precisely what has happened in American higher education in the three decades since the U.S. News rankings first appeared. The rise of rankings didn’t cause the breakdown in public funding, but it has naturalized the inequality that results.
The good news, as these books show, is that numbers don’t need to be used as we use them now. But for real change to take place, the wider society has to become involved in the conversation. These books do an excellent job of helping make that happen.