In a world increasingly governed by algorithms, the fact that most AI decision-making is opaque is a concern. Deep learning doesn’t just follow pre-programmed rules; it learns through trial and error, becoming increasingly sophisticated and often unfathomable. While these systems might produce reliable and even fair results, they often offer no insight into *how* those results were achieved.
This lack of transparency, this “opacity,” is becoming a major concern, and not just for reasons of trust or accountability, but because it threatens our very autonomy.
The call for transparency in AI is widespread, echoed in policy documents, academic research, and popular media, as well as regulators of privacy and data protection. People understandably find it frustrating and scary to be subjected to decisions they can’t understand. Empirical studies show that trust in decisions, actions, and outcomes increases when a plausible explanation is provided. Transparency also allows us to assess the reliability and fairness of algorithms and makes them easier to improve. Opacity, conversely, compounds the negative impacts of bad outcomes, making it harder to challenge decisions and hold those responsible accountable.
However, not everyone agrees that complete transparency is necessary or even achievable. Critics point out that even human decision-making processes can be equally complex and opaque. We don’t expect judges to explain the neurological underpinnings of their rulings, nor do we fully understand how many common medical interventions work. Demanding full transparency from AI might be holding it to a higher standard than we apply to humans.
Vaassen (2022) argues that the central issue with opaque algorithms is not simply about fairness or accountability, but about autonomy—our ability to shape our lives according to our plans and desires. To understand this connection, it’s crucial to define what opacity and transparency really mean. The author proposes a *causal* account, defining opacity as a situation where a person is not in a position to grasp the causal explanations of an AI system’s outcomes. Conversely, transparency exists when a person *is* in a position to understand those causal explanations.
A causal explanation, in this context, means understanding what caused a specific outcome and how changes in those causes would correlate with changes in the outcome itself. It doesn’t necessarily require understanding the technical implementation of the algorithm, but rather knowing which factors are significant drivers of the decision. Opacity, therefore, hides effective strategies for influencing and predicting the outcomes of AI decisions, while transparency empowers us to take action and affect future outcomes.
Knowing that a particular AI values certain skills or qualifications allows us to prepare accordingly. But when that knowledge is hidden, we are denied the information needed to make informed choices about our education, career, and other important life decisions. This undermines our autonomy by preventing us from shaping our lives according to our goals. This autonomy worry is separate from concerns about reliability, fairness, or trust. An algorithm could be perfectly reliable and fair, yet still undermine our autonomy if its decision-making process remains a mystery.
Imagine an AI system called GOV-1 that selects candidates for government jobs. It’s the most reliable system available, trusted by the government and users alike. Applicants even have the right to have their applications reviewed by a human. Yet, no one understands how GOV-1 weighs the information provided in the application questionnaire. This opacity means that aspiring government employees have no idea which skills to acquire, which experiences to seek, or even what seemingly irrelevant factors might be influencing the decision. GOV-1’s opacity hinders their ability to plan their lives and pursue their goals effectively, directly undermining their autonomy.
The importance of autonomy is well-established in moral philosophy, across a wide range of ethical theories. It’s a core principle in discussions about education, bioethics, political theory, and free will. Furthermore, undermining autonomy also undermines responsibility. If we can’t understand how to influence the outcomes of a decision-making process, we can’t be held responsible for those outcomes.
Given the profound implications of opacity, Vaassen (2022) explores how to translate the demand for transparency into practical solutions, focusing on three key areas: (1) determining the desirable and technically attainable degree of transparency, (2) legally entrenching adequate transparency demands, and (3) weighing the potential downsides of transparent decision-making.
The connection between transparency and autonomy provides a framework for deciding how transparent AI decision algorithms should be. The goal is to understand how changes in input correlate with changes in output. This doesn’t require knowledge of the system’s inner workings at a granular, physical level. The key is to identify robust correlations that are relevant and informative for users. Instead of demanding full physical, design, or algorithmic transparency, the focus should be on achieving “input-output difference-making transparency.” This can be achieved through “glass-boxing” methods, which test AI systems’ conformity to norms by checking their inputs and outputs. However, forensic approaches may lead to mistaken conclusions when details are neglected and are also context-dependent based on the user’s needs.
Despite the potential, legal entrenchment is challenging. Proprietary laws protect AI algorithms, potentially conflicting with demands for transparency. Also, an obligation only to provide *an* explanation is not enough. The explanation needs to be tailored to the needs and perspective of the affected party. Current attempts to regulate AI use, such as the EU’s GDPR and the European Commission’s AI Act, fall short of guaranteeing transparency towards those whose lives are affected by automated decisions.
Finally, transparency isn’t without its downsides. Providing information about how to influence decision algorithms might unfairly benefit those with greater resources. Furthermore, public knowledge of specific criteria can make those criteria unreliable over time (“Goodhart’s Law”). Additionally, forcing experts to make their reasoning accessible to non-experts could limit the scope of considerations that are taken into account. In this way it limits experts from referring to what could be true simply because that truth cannot be understood or reasoned with by non-experts.
Vaassen (2022) concludes that there are numerous reasons to demand transparency in AI decision algorithms, and one central reason is personal autonomy. By obscuring the pathways through which we can influence our lives, opaque algorithms undermine our ability to make informed choices and pursue our goals. The call for transparency, therefore, is not simply about fairness or accountability; it’s about empowering individuals to shape their own destinies in an increasingly automated world. While the path to achieving meaningful transparency is filled with challenges, it is a journey worth pursuing for the sake of our individual autonomy and our collective future.
Further reading
Vaassen, B. AI, Opacity, and Personal Autonomy. (2022) Philos. Technol. 35, 88 https://doi.org/10.1007/s13347-022-00577-5