Coded Inequality: How AI in Healthcare is Failing Patients in Need

Physician Caring for patient (PennState Health).

Artificial intelligence has been seen as the leading way to revolutionize medicine. It has the ability to catch cancers earlier, predict the onset of sepsis, and create more widespread specialists in medicine. On paper, it sounds like it could be an equalizer in medicine, spreading ease of disease diagnosis across the world and bridging gaps that have long been defined by inequalities. 

However, the prominent information about AI in the headlines leaves out important information: the patients who stand to benefit the most from these tools are the very same patients being failed by them. The use of AI in healthcare is only as fair as the data it learns from and is trained on. In its current use, that data is deeply, structurally biased. 

The AI Algorithm That “Forgot” Patients

Consider one of the most well-documented cases of healthcare in AI gone wrong. A widely used algorithm that has been deployed across major health systems to identify patients who would benefit from extra care management, used past healthcare spending as a proxy for patient health needs. The idea behind this seems sound, with sicker patients costing more, so ideally, spending should reflect need. However, it does not do so equally. 

Due to Black patients historically receiving less care than white patients with the same conditions, which is a product of decades of systemic discrimination and unequal access, the algorithm read lower spending as a sign of better health. As a result, Black patients were far less likely to be flagged for care they genuinely needed. The algorithm itself did not intend to discriminate, but followed the patterns of data that already did. 

This type of incident has become a pattern. 

A Built-In Problem

AI-driven dermatology tools trained predominantly on images of lighter skin tones struggled to detect skin cancer in patients with darker skin. This is especially alarming considering the fact that Black patients already face the highest melanoma mortality rates. Sepsis prediction models developed in high-income settings have shown significantly reduced accuracy for Hispanic patients, because the training data simply did not include them in representative numbers. Additionally, even pulse oximeters, which are a basic clinical tool that have become integrated into AI monitoring systems, have been shown to overestimate blood oxygen levels in patients with darker skin. This causes delayed recognition of hypoxia. 

The problem runs deeper than these few basic models. As researchers have noted, most of the AI training datasets are drawn from large academic medical centers located in wealthy, urban centers. Rural patients, elderly patients, immigrants, indigenous communities, and people who lack access to care are systemically absent from the data used. If they do appear, their data is often incomplete, which is shaped by the same inequities that prevented them from getting adequate care in the first place. Bias goes in, and it continues. 

A physician’s bias is able to be challenged or corrected, but in the case of AI, the algorithm presents itself as a representation of facts. The failures it causes are invisible due to its design. 

Moving Beyond the Technical Problem

While it can be tempting to frame this issue as a software engineering challenge, something that can be fixed with better datasets and stronger code, that is not the underlying problem at hand. These technical solutions do matter with the inclusion of more diverse training data and algorithm audits. However, as a whole, it is a public health crisis that requires a response centered around that framework. 

The communities most harmed by biased AI (Black, Latino, Indigenous, low-income, and rural populations) are the same communities that have faced centuries of medical neglect, non-consensual experimentation, and exclusion from the benefits of medical innovation. By setting up AI systems that quietly replicate these patterns at a larger scale, technical errors become encoded injustice. What has been called innovation is injustice built into the infrastructure of modern medicine. 

Trust is also a matter to consider. Healthcare systems have already had consistent skepticism from many underserved communities that have been earned through historical abuse and ongoing neglect. When biased AI misdiagnoses, denies resources, or underestimates needs, it harms not only individual patients but also erodes trust that public health workers have been attempting to rebuild. 

What Can Be Done

This bias in AI is not necessarily inevitable. Biased AI is a policy failure that can be corrected. Some things can be advocated for in order to decrease the inequality gap in the system itself. 

One is to demand algorithm transparency. The healthcare institutions that deploy AI tools should be required to disclose how their systems were developed, which populations were included in the training data, and how performance varies across demographic groups. Patients and providers using these tools should be able to know when they are being assessed by a tool that was not designed in their best interest or for their use. 

Another is to support equity-centered regulation. The FDA has recently significantly expanded its approval of AI-enabled medical devices in recent years, but approval processes do not consistently require evidence of equitable performance across various subgroups examined. It is key to advocate for standards that make a level of demographic performance a prerequisite for clinical AI deployment. 

A final concrete thing to push for, while there are many more, is to integrate health equity into AI education. It is important to push for a curriculum that teaches how AI works and how it can fail, in order to combat future errors in the healthcare systems. Programs like the National Institute of Health's AI/ML Consortium to Advance Health Equity are already building this foundation by partnering across academic institutions, community organizations, and minority-serving institutions to ensure that these algorithms are developed with the focus of equity. 

Stakes

Currently, we are at an inflection point. With the rapid development of AI,  decisions being made on what regulations are put into AI policy and research development will shape healthcare systems for decades. It is important to stop the deployment of tools without equity as a foundational requirement, in order to continue the advancement of healthcare that is not connected to past failures.

The groundwork for a better future is already being laid. Advocates, researchers, and students are pushing back against the assumption that efficiency and equity are at odds. Programs like the NIH’s AIM-AHEAD initiative are paving the way for future institutional equity-focused algorithms. It is crucial to continue this advocacy and research to ensure that these systems do not become too entrenched to challenge and can break the pattern of past failures.