“There’s no hiding behind algorithms anymore. The problems cannot be minimized. The machines have shown they are not up to the task of dealing with rare, breaking news events, and it is unlikely that they will be in the near future. More humans must be added to the decision-making process, and the sooner the better.”
I wonder whether we have if not an increasing then certainly a more visible problem with addressing the agency of automated processes. In particular automation that functions predominantly through software, i.e. stuff we refer to as ‘algorithms’ and ‘algorithmic’, possibly ‘intelligent’ or ‘smart’ and perhaps even ‘AI’, ‘machine learning’ and so on. I read three things this morning that seemed to come together to concretise this thought: Alexis Madrigal’s article in The Atlantic – “Google and Facebook have failed us“, James Somers’ article in The Atlantic – “The coming software apocalypse” and LM Sacacas’ blogpost “Machines for the evasion of moral responsibility“.
In Madrigal’s article we can see how the apparent autonomy of the ‘algorithm’ becomes the fulcrum around which machinations around ‘fake news’, in this case regarding the 2nd October 2017 mass shooting in Las Vegas. The apparent incapacity of an automated software system to perform the kinds of reasoning attributable to a ‘human’ editor is diagnosed on the one hand, and on the other the speed at which such breaking news events taking place and the volume of data being processed by ‘the algorithm’ led to Google admitting that their software was “briefly surfacing an inaccurate 4chan website in our Search results for a small number of queries”. Madrigal asserts:
It’s no longer good enough to shrug off (“briefly,” “for a small number of queries”) the problems in the system simply because it has computers in the decision loop.
In Somers’ article we can see how decisions made by programmers writing software that processed call sorting and volume for the emergency services in Washington State led to the 911 phone system being inaccessible to callers for six hours one night in 2014. As Somers describes:
The 911 outage… was traced to software running on a server in Englewood, Colorado. Operated by a systems provider named Intrado, the server kept a running counter of how many calls it had routed to 911 dispatchers around the country. Intrado programmers had set a threshold for how high the counter could go. They picked a number in the millions.
Shortly before midnight on April 10, the counter exceeded that number, resulting in chaos. Because the counter was used to generating a unique identifier for each call, new calls were rejected. And because the programmers hadn’t anticipated the problem, they hadn’t created alarms to call attention to it. Nobody knew what was happening. Dispatch centers in Washington, California, Florida, the Carolinas, and Minnesota, serving 11 million Americans, struggled to make sense of reports that callers were getting busy signals. It took until morning to realize that Intrado’s software in Englewood was responsible, and that the fix was to change a single number.
Quoting an MIT Professor of aeronautics (of course) Nancy Leveson, Somers observes: “The problem,” Leveson wrote in a book, “is that we are attempting to build systems that are beyond our ability to intellectually manage.”
Michael Sacasas in his blogpost refers to Madrigal’s article and draws out arguments that the complex processes of software development, maintenance and the large and complicated organisations such as Facebook are open to those working there to work in a ‘thoughtless’ manner:
“following Arendt’s analysis, we can see more clearly how a certain inability to think (not merely calculate or problem solve) and consequently to assume moral responsibility for one’s actions, takes hold and yields a troubling and pernicious species of ethical and moral failures. …It would seem that whatever else we may say about algorithms as technical entities, they also function as the symbolic base of an ideology that abets thoughtlessness and facilitates the evasion of responsibility.”
The simplest version of what I’m getting at is this: on the one hand we attribute significant agency to automated software processes, this usually involves talking about ‘algorithms’ as quasi- or pretty much autonomous, which tends to infer that whatever it is we’re talking about, e.g. “Facebook’s algorithm”, is ‘other’ to us, ‘other’ to what might conventionally be characterised as ‘human’. On the other hand we talk about how automated processes can encode the assumptions and prejudices of the creators of those techniques and technologies, such as the ‘racist soap dispenser‘.
There’s a few things we can perhaps note about these related but potentially contradictory narratives.
First, they perhaps infer that the moment of authoring, creating, making, manufacturing is a one-off event – the things are made, the software is written and it becomes set, a bit like baking a sponge cake – you can’t take the flour, sugar, butter and eggs out again. Or, in a more nuanced version of this point – there is a sense that once set in train these things are really, really hard to change, which may, of course, be true in particular cases but also may not be a general rule. A soap dispenser’s sensor may be ‘hard coded’ to particular tolerances, whereas what gets called ‘Facebook’s algorithm’, while complicated, is probably readily editable (albeit with testing, version control and so on). This kind of narrative freights a form of determinism – there is an implied direction of travel to the technology.
Second, the kinds of automated processes I’m referring to here, ‘algorithms’ and so on, get ‘black boxed’. This is not only on the part of those who create, operate and benefit from those processes—like those frequently referred to Google, Facebook, Amazon and so on—but also in part by those who seek to highlight the black boxing. As Sacasas articulates: “The black box metaphor tries to get at the opacity of algorithmic processes”. He offers a quote from a series of posts by Kevin Hamilton which illustrates something of this:
Let’s imagine a Facebook user who is not yet aware of the algorithm at work in her social media platform. The process by which her content appears in others’ feeds, or by which others’ material appears in her own, is opaque to her. Approaching that process as a black box, might well situate our naive user as akin to the Taylorist laborer of the pre-computer, pre-war era. Prior to awareness, she blindly accepts input and provides output in the manufacture of Facebook’s product. Upon learning of the algorithm, she experiences the platform’s process as newly mediated. Like the post-war user, she now imagines herself outside the system, or strives to be so. She tweaks settings, probes to see what she has missed, alters activity to test effectiveness. She grasps at a newly-found potential to stand outside this system, to command it. We have a tendency to declare this a discovery of agency—a revelation even.
In a similar manner to the imagined participant in Searle’s “Chinese Room” thought experiment, the Facebook user can only guess at the efficacy of their relation to the black boxed process. ‘Tweaking our settings’ and responses might, as Hamilton suggest, “become a new form of labor, one that might then inevitably find description by some as its own black box, and one to escape.” A further step here is that even those of us diagnosing and analysing the ‘black boxes’ are perhaps complicit in keeping them in some way obscure. As Evan Selinger and Woodrow Hartzog argue: things that are obscure can be seen as ‘safe’, which is the principle of cryptography. Obscurity, for Selinger & Hartzog, “is a protective state that can further a number of goals, such as autonomy, self-fulfillment, socialization, and relative freedom from the abuse of power”. Nevertheless, obscurity can also be an excuse – the black box is impenetrable, not open to analysis and so we settle on other analytic strategies or simply focus on other things. A well-worn strategy seems to be to retreat to the ontological, to which I’ll return shortly.
Third, following from above, perhaps the ways in which we identify ‘black boxes‘ or the forms of black boxing we do ourselves over-simplifies or elides complexity. This is a difficult balancing act. A good concept becomes a short-hand that freights meaning in useful ways. However, there is always potential that it can hide as much as it reveals. In the case of the phenomena outlined in the two articles above, we perhaps focus on the ends, what we think ‘the algorithm’ does – the kinds of ‘effects’ we see, such as ‘fake news’ and the breakdown of an emergency telephone system, or even a ‘racist soap dispenser’. It is then very tempting to perform what Sally Wyatt calls a ‘justifactory’ technological determinism – not only is there a ’cause and effect’ but these things were bound to happen because of the kinds of technological processes involved. By fixing ‘algorithms’ as one kind of thing, we perhaps elide the ways in which they can be otherwise and, perhaps more seriously, elide the parts of the process of the development, resources, use and reception of those technologies and their integration into wider sociotechnical systems and society. These things don’t miraculously appear from nowhere – they are the result of lots of actions and decisions, some banal, some ‘strategic’, some with good intentions and some perhaps morally-questionable. By black boxing ‘the algorithm’, attributing ‘it’ with agency and making it ‘other’ to human activities we ignore or obscure the organisational processes that make it possible at all. I argue we cannot see these things as completely one thing or the other: the black boxed entity or the messy sociotechnical system, but rather as both and need to accommodate that sort of duality in our approaches to explanation.
Fourth, normative judgements are attached to the apparent agency of an automated system when it is perceived as core to the purpose of the business. Just like any other complicated organisation whose business becomes seen as a ‘public good’ (energy companies might be another example), competing, perhaps contradictory, narratives take hold. The purpose of the business may be to make money–in the case of Google and Facebook this is of course primarily through advertising, requiring attractive content to which to attach adverts–but the users perhaps consider their experience, which is ‘free’, more important. It seems to have become received wisdom that the very activities that drive the profits of the company, by boosting content that drives traffic and therefore serves more advertising and I assume therefore resulting in more revenue, run counter to accepted social and moral norms. This exemplifies the competing understandings of what companies like Google and Facebook do – in other words, what their ‘algorithms’ are for. This has a bearing on the kinds of stories we then tell about the perceived, or experienced, agency of the automated system.
Finally (for now), there is a tendency for academic social scientific studies of automated software systems to resort to ontological registers of analysis. There may be all sorts of reasons used as justification for this, such as specific detail of a given system is not accessible, or (quite often) only accessible through journalists, or the funding isn’t available to do the research. However, it also pays dividends to do ‘hard’ theory. In the part of academia I knock about in, geography-land and it’s neighbours, technology has been packaged up into the ‘non-human’ whereby the implication is that particular kinds of technology are entirely separate from us, humans, and can be seen to have ‘effects’ upon us and our societies. This is trendy cos one can draw upon philosophy that has long words and hard ideas in it, in particular: ‘object oriented ontology‘ (to a much lesser extent the ‘bromethean‘ accellerationists). The generalisable nature of ‘big’ theory is beguiling, it seems to permit us to make general, perhaps global, claims and often results in a healthy return in the academic currency of citations. Now, I too am guilty of resorting to theory, which is more or less abstract, through the work of Bernard Stiegler in particular, but I’d like to think I haven’t disappeared down the almost theological rabbit hole of trying to think objects in themselves through abstract language such as ‘units‘ or ‘allopoetic objects‘ and ‘perturbations’ of non-human ‘atmospheres’.
It seems to me that while geographers and others have been rightly critical of simplistic binaries of human/technical, there remains a common habit of referring to a technical system that has been written by and is maintained by ‘humans’ as other to whatever that ‘human’ apparently is, and to refer to technologically mediated activities as somehow extra-spatial, as virtual, in contra-distinction to a ‘real’. This is plainly a contradiction. On the one hand this positions the technology in question (‘algorithms’ and so on) as totally distinct from us, imbued with an ability to act without us and so potentially powerful. On the other hand if that technology is ‘virtual’ and not ‘real’ it implies it doesn’t count in some way. While in the late 90s and early 00s the ‘virtual’ technologies we discussed were often seen as somewhat inconsequential, the more contemporary concerns about ‘fake news’, malware and encoded prejudices (such as racism) have made automated software systems part of the news cycle. I don’t think it is a coincidence that we’ve moved from metaphors of liberty and community online to metaphors of ‘killer robots’, like the Terminator (of course there is a real prospect of autonomous weapons systems, as discussed elsewhere).
In the theoretical zeal of ‘decentering the human subject’ and focusing on the apparent alterity of technology, as abstract ‘objects’, we are at risk of failing to address the very concerns which are expressed in the articles by Madrigal and Somers. In a post entitled ‘Resisting the habits of the algorithmic mind‘, Sacasas suggests that automated software systems (‘algorithms’) are something like an outsourcing of problems solving ‘that ordinarily require cognitive labor–thought, decision making, judgement. It is these very activities–thinking, willing, and judging–that structure Arendt’s work in The Life of the Mind.’ The prosthetic capacity of technologies like software to in some way automate some of these processes might be liberating but they are also, as Sacasas suggests, morally and politically consequential. To ‘outsource the life of the mind’ for Sacasas means to risk being ‘habituated into conceiving of the life of the mind on the model of the problem-solving algorithm’. A corollary to this supposition I would argue is that there is a risk in the very diagnosis of this problem that we habituate ourselves to a determinism as well. As argued in the third point, above, we risk obscuring the organisational processes that make such sociotechnical systems possible at all. In the repetition of arguments that autonomous, ‘non-human’, ‘algorithms’ are already apparently doing all of these problematic things we will these circumstances upon ourselves. There is, therefore, an ethics to thinking about and analysing automation too.
Where does this leave us? I think it leaves us with some critical tools and tasks. We perhaps need not to shy away from the complexity of the systems we discuss – the ideas and words we use can do work for us, ‘algorithm’ for example freights some meaning, but we perhaps need to be careful we don’t obscure as much as we reveal. We perhaps need to use more, not fewer, metaphors. We definitely need more studies that get at the specificity of particular forms, processes and work of automation/automated systems. All of us, journalists and academics alike, need to perhaps use our words more carefully, or use more words to get at the issues.
Simply hailing the ‘rise of the robots’ is not enough. I think this reproduces an imagination of automation that is troubling and ought to be questioned (what I’ve called an ‘automative imaginary’ elsewhere, but maybe that’s too prosaic). For people like me in geography-land to retreat into ‘high’ theory and to only discuss abstract ontological/ metaphysical attributes of technology seems to me to be problematic and is a retreat from that part of the ‘life of the mind’ we claim to advance. I’m not arguing we need not retreat from theory we simply need to find a balance. A crucial issue for social science researchers of ‘algorithms’ and so on is that this sort of work is probably not the work of a lone wolf scholar, I increasingly suspect that it needs multi-disciplinary teams. It also needs to, at least in part, produce publicly accessible work (in all senses of ‘accessible’). In this sense work like the report on ‘Media manipulation and disinformation online‘ by Data & Society seems like necessary (but by no means the only) sorts of contribution. Prefixing your discipline with ‘digital’ and reproducing the same old theory but applied to ‘digital’ things won’t, I think, cut it.
3 Replies to “Our vascilating accounts of the agency of automated things”