AI and “Root Cause” Detection
In my last post I wrote about how a person’s determination of the “root cause” of an issue allows us to infer something about their values, which can be valuable info. One of my readers noted that this point was also made by my colleague, Fred, in a post he wrote and this technique was also used in recent paper by Hutchinson et al on audits.
Yay! Someone read my post and worked it into a network of other material. Thank you for reading, and I’m happy to acknowledge and refer folks to these other texts.
In this one, I want to consider the idea that automation, specifically so-called “artificial intelligence,” is somehow immune to this and can provide a disinterested, impartial way of interpreting events and evaluating candidates for a “root cause.” I think this is a timely thing to consider given the hype around AI and the attempts of big market players like Datadog, IBM, and Dynatrace to incorporate it into their products. I also happen to know that the 2024 VOID Report will get into this, as I was asked to make some editorial contributions. I don’t want to steal the Report’s thunder and so will avoid spoiling its analyses here; however, I will offer some of my own thoughts on the subject.
To start, let me state why I think some people think AI may be a suitable tool for this. The first reason is that it’s supposed to produce efficiency gains for the business. A classic justification for this view comes from the multiple tools that folks often have for representing their systems. These are often irreducibly different and require an act of synthesis on the part of a person or people. That’s work! And organizations would like to automate that away with “a single pane of glass” so that the people can do something else which the organization thinks actually creates value. The organization, of course, doesn’t see that synthesis, aka establishing “common ground,” as valuable work despite its necessity because it can’t be packaged as a sale-able commodity (“product”) in its own right. So let the AI do it! It won’t match any given person’s way of ‘seeing’ things, since it’s an amalgamation of data that no single person has processed in quite the same way, but that’s fine.
A second but related reason is that it may produce more accurate diagnoses of what caused a problem than people. There are several assumptions baked into this that are well-known issues. For one, it requires a baseline of the system’s normal state as non-problematic. This would presumably come from collecting and analyzing data from some rolling prior period in order to establish a norm which is then used to construct that baseline. Deviations of a certain magnitude are then treated as anomalies. Any such change that continues in a trend away from the baseline allows for one to isolate a relatively narrow time range for investigation and then some event is selected as the trigger. Since this is all “data-driven,” it is presumably independent of human judgement and therefore more accurate of what really happened without the messiness of humans.
This establishment of that norm already implies some judgements about how to understand the events under consideration and what counts as relevant data. And this brings us to a most pernicious problem: the partiality baked into datasets. Numerous social scientists like Bowker & Star in Sorting Things Out, the contributors to “Raw Data” is an Oxymoron, and Safiya Noble in Algorithms of Oppression have documented that data are not politically neutral precisely because the creation of the schema we use to create the data used in AI models is created through political struggle. Bowker & Star, for example, memorably note that a local Gay and Lesbian Pride Parade was finally included as a regular event in the Santa Cruz phone book due to activist efforts.
To give another example, OpenAI has reportedly relied upon Kenyan workers making a pittance to define offensive content for an AI tool. Once trained on the resulting data, the tool would then evaluate the data used in ChatGPT itself in an effort to stop it from saying offensive things. The fact that Kenyan people are available as a suitable workforce and are able to do that defining is due to politics. The whole history of colonial occupation and neocolonial practices like establishing “free-trade zones” have produced policies to support and a polity ready to perform this “ghost work.” Hence the condition for producing the data which was used to produce the dataset (the product of training the AI) necessarily baked politics into it.
Thus the idea that AI can somehow work with value neutral or impartial data is inherently flawed. It’s not just “garbage in, garbage out.” Data are always biased. Saying that an AI determined something may make it impersonal, in the sense that no individual person holds the ‘view’ that the AI does based on its training, but that doesn’t make it impartial. Thus the “root cause” that an AI finds will always be biased. Which is not to say that it can’t still be useful. The question then becomes: who benefits? And do we want them to be the ones that do?