EDPS Publishes Report on Human Oversight of Automated Decision-Making

Assumption: ADM systems will operate within specific and predetermined conditions.

EDPS Clarification: In reality, ADM systems often encounter novel or unpredictable environments. Overreliance on the assumption that systems will only face the predictable scenarios around which they were designed can be dangerous.

EDPS Example: A 2016 incident involving a Tesla vehicle in autopilot mode illustrates this risk: the system failed to detect a white truck crossing a highway against a bright sky, resulting in a fatal crash. This occurred because the system operated beyond its capacity to interpret unusual lighting conditions and atypical obstacles that were not well represented in its training data, highlighting the dangers of assuming ADM systems will always function as intended in real-world environments.

Assumption: ADM systems will transfer control to humans in atypical situations.

EDPS Clarification: Many ADM systems lack explicit mechanisms to defer to human operators when faced with a novel or uncertain scenario. Without built-in guardrails, the expectation that systems will “know” when to hand over control is unfounded.

EDPS Example: The Babylon Health AI-powered symptom checker, used in the UK’s NHS, failed to detect serious conditions such as heart attacks and sometimes provided inappropriate advice, like misattributing a breast lump to “hysteria.” These failures highlight the risks when ADM systems do not have mechanisms to recognise their own limitations and appropriately transfer control to human operators in complex or uncertain situations.

Assumption: Automation does not influence human judgment.

EDPS Clarification: Automation bias can occur when human operators become overly reliant on system recommendations, sometimes accepting them without sufficient critical assessment or use of their own expertise. This can result in operators effectively “going on autopilot”, allowing the ADM system’s outputs to shape or even override their own judgment, especially in complex or specialised contexts where the human may feel less confident intervening.

EDPS Example: A 2023 study found that radiologists reviewing mammograms with AI-generated suggestions were less accurate when the AI’s recommendations were incorrect, regardless of their experience. This illustrates how automation bias can lead professionals to follow ADM outputs without sufficient independent assessment, underscoring the need for safeguards such as confidence scores and targeted training.

Assumption: Systems that combine human judgment with machine outputs are inherently superior and will seamlessly complement each other.

EDPS Clarification: Without deliberate design, clear role allocation, and appropriate training, hybrid systems can compound the weaknesses of both humans and machines, leading to confusion and reduced accountability. In practice, human operators may struggle to interpret or trust machine outputs, thereby reducing their inherent speed and convenience, while machines often lack the contextual awareness needed to appreciate important context or nuance, thereby increasing the risk of error or bias.

EDPS Example: The Boeing 737 MAX crashes in 2018 and 2019 demonstrate the risks of assuming that humans and machines will inevitably work well together. The Manoeuvring Characteristics Augmentation System, which the planes operated on, was intended to automate stabiliser adjustments while pilots handled complex situations, but limited transparency and inadequate training meant pilots hesitated or struggled to override the system during emergencies, contributing to fatal outcomes.

Assumption: Human operators always have the authority and ability to override.

EDPS Clarification: Operators may lack the independence, expertise, or confidence to challenge ADM system recommendations. Factors such as fear of consequences or deference to the system’s perceived expertise can discourage intervention and reinforce reliance on automated outputs.

EDPS Example: Between 2014 and 2019, Poland’s Public Employment Services used a profiling algorithm to categorise job seekers, with client advisors formally given the authority to override the system’s classifications. In practice, however, excessive workloads, insufficient training, unclear guidelines, and organisational pressures meant advisors were often unable to intervene effectively, demonstrating how design and operational constraints can prevent meaningful human oversight even when formal mechanisms exist.

Assumption: Transparency and explainable AI are sufficient for effective oversight.

EDPS Clarification: While explainability can support understanding, it does not necessarily equip operators to identify when a system is ill-suited to a particular scenario. Additionally, explainability does not necessarily equate to good decision-making and may even cause overreliance.

EDPS Example: A 2021 study found that when clinicians were given incorrect machine learning recommendations for antidepressant treatments, along with simple and easily interpretable explanations, their accuracy in selecting the correct treatment significantly decreased. These findings suggest that simplistic explanations can encourage overreliance on ADM outputs, even when those outputs are incorrect, highlighting the limitations of explainable AI for effective oversight.