A Proposed Framework on Integrating Health Equity and Racial Justice into the Artificial Intelligence Development Lifecycle

Abstract

The COVID-19 pandemic has created multiple opportunities to deploy artificial intelligence (AI)-driven tools and applied interventions to understand, mitigate, and manage the pandemic and its consequences. The disproportionate impact of COVID-19 on racial/ethnic minority and socially disadvantaged populations underscores the need to anticipate and address social inequalities and health disparities in AI development and application. Before the pandemic, there was growing optimism about AI's role in addressing inequities and enhancing personalized care. Unfortunately, ethical and social issues that are encountered in scaling, developing, and applying advanced technologies in health care settings have intensified during the rapidly evolving public health crisis. Critical voices concerned with the disruptive potentials and risk for engineered inequities have called for reexamining ethical guidelines in the development and application of AI. This paper proposes a framework to incorporate ethical AI principles into the development process in ways that intentionally promote racial health equity and social justice. Without centering on equity, justice, and ethical AI, these tools may exacerbate structural inequities that can lead to disparate health outcomes.

Artificial intelligence (AI) applications have been widely deployed to understand, mitigate, and address pandemics, including the ongoing COVID-19 crisis.^1,2 Examples include case tracking, projecting virus transmission under different mitigation scenarios, forecasting mortality trends, and predicting disease outbreaks or hotspots.² The increases in computing capacity and AI-generative platforms, which can rapidly identify novel peptides, genes, and drug candidates, have accelerated the scientific discovery of COVID-19 vaccine candidates and medical therapies.^2,3 With the ongoing global vaccine roll-out, AI-driven insights and applied interventions continue to play a significant role in adaptive and predictive technology. Some applications include tracking COVID-19 mutations and variants to inform vaccine design and development;^4,5 predictive impact modeling for describing which populations and regions to vaccinate to rapidly flatten the curve and end the pandemic;⁶ monitoring the supply chain management and vaccine delivery;⁷ as well as post-vaccine surveillance to monitor adverse events and track effectiveness. The pandemic has provided opportunities for leveraging the rapidly evolving data and AI technologies to address this public health crisis. However, concerns about ethics, equity, and justice regarding the development and application of AI technologies in health care settings have intensified during the pandemic.^1,2,8 The pandemic has been devastating, especially in Black and Hispanic communities that experience a mortality rate three times higher than White communities.⁹ National level data in the United States collected by the American Public Media (APM) Research Lab demonstrate that age-adjusted mortality rates for Black Americans, Indigenous peoples, Latinxs, and Pacific Islanders are 2.1, 2.2, 2.4, and 2.7 times higher than for Whites, respectively.⁹ While there have been great advances in personalized medicine and AI-based biomedical discovery based on genomic profiles, there is also a lack of diverse clinical research data used to generate those treatment strategies, which can result in worse outcomes for underserved members of the community.^10–13 The rush for biomedical discovery with poorly representative COVID-19 databases may result in further inequities.¹⁴ With heightened visibility around structural racism, the discriminatory stereotypes created and reinforced with particular technologies, and biases reflected in algorithms are an increasing concern.¹⁴

This commentary provides a framework and recommendations to integrate health equity, racial justice, and ethical AI principles into technology development to address health inequities.

Prioritizing Health Equity and Racial Justice in the AI Development Lifecycle

Stakeholders in the design and development of AI technologies have a critical role in ensuring that mission-driven values to promote health equity are prioritized in implementing AI technologies. These technologies can influence payers, health providers, patient behaviors, and their experiences with the health care system in various ways. The application of machine learning to big data can identify patterns for improving health care delivery and decision-support tools can enable evidence-based care.¹⁵ In addition, AI has become a foundational element in many wearable technologies that support health maintenance or disease management.¹⁶

However, there are significant ethical and social concerns involved when designing, developing, and implementing AI tools and applications both domestically and globally.^17–21 Bias can be introduced into AI applications and affect numerous facets of an organized pandemic response (e.g., resource allocation and priority-setting, public health surveillance, contact tracing, patient privacy, frontline caregiving, health care worker privacy). Health equity and racial justice principles in applying AI, especially in the COVID-19 era, can provide a conceptual scaffold to ensure that efforts to track the virus, improve outcome predictions, and implement effective interventions will benefit all groups in a population for the current and future pandemics.

For the proposed framework, we define health equity as the value and principle underlying a commitment to reduce and ultimately eliminate health disparities.²² Addressing health equity, as asserted by Braveman, Marmot, and other scholars, is a social justice issue and an ethical imperative, consonant with human rights principles to give special priority to act on significant public health problems that differentially affects those with fewer resources and/or may have more obstacles to achieving optimal health.^23,24 Broadly speaking, health disparities have been defined as systematic, unfair, plausibly avoidable differences in health (including its determinants and outcomes) negatively affecting socially vulnerable groups. These social groups are at risk of not achieving their full health potential because of historical discrimination, institutionalized racism, or marginalization (i.e., exclusion from social, political, or economic opportunities, including technologies), among other forces. When developing AI-based solutions in health care, anticipating and addressing potential health disparity concerns is imperative. These concerns must be consciously and appropriately accommodated, or health disparities among racial/ethnic minority and other socially vulnerable populations will continue to widen. Equity and justice principles in the continuum of AI design, development, and use are paramount and foundational. Similar to health equity, racial justice is a moral and value principle that promotes fair treatment of people of all races and ethnicities, resulting in equitable opportunities and outcomes.²⁵ Racial justice includes a deliberate effort to support and achieve racial equity through proactive and preventive measures. We will achieve racial equity when a person's racial or ethnic identity no longer predicts their social or economic opportunities and health outcomes. Simply denouncing or eliminating discrimination or stereotyping and bias is not sufficient to achieve racial justice. Instead, organizations and systems must re-imagine and co-create a different culture and society by implementing interventions that affect multiple sectors, processes, and practices.

Though AI ethics is accepted as critically important in harnessing AI's potential, there are disparate views and varying perspectives on critical ethical issues that inform the AI principles established within governments, the scientific research community, and industry.^17,26–28 Several groups have attempted to summarize such ethical issues to inform policy statements.¹⁷ The Turing Institute defines AI ethics as a set of principles, values, and approaches that use widely accepted standards to guide moral conduct in the lifecycle of AI systems.^29,30 The IBM Institute for Business Value defines AI ethics as a multidisciplinary field of study to understand how to optimize its beneficial impact while reducing risks and adverse outcomes for all stakeholders in a way that prioritizes human agency and well-being, as well as environmental flourishing.³¹ Artificial intelligence ethics research largely focuses on designing and building AI systems with an awareness of the values and principles to be followed during development—such as data responsibility and privacy, fairness, inclusion, moral agency, value alignment, accountability, transparency, trust, and technology misuse.^32–42 These frameworks and statements can be aligned with health equity and racial justice principles. As a part of the efforts to embrace racial and social justice, the IBM Academy of Technology and other Justice and Diversity Councils have launched initiatives to replace terminology that promotes racial and cultural bias, to promote design justice for racial equity, and to integrate equity and inclusive principles across the solutions.⁴³

This paper compiles the range of ethical issues that inform guidelines and propose examples of how health equity and racial justice might be aligned with AI ethics (see Box 1). The paper also builds on the AI development lifecycle and provides a framework with recommendations for operationalizing ethical AI with health equity and racial justice principles.

Unintended Consequences of Limited Health Equity or Racial Justice Deliberation in AI Development

Although ethical statements are being issued by governments, academics, policymakers and regulators in response to the growing visibility of advanced technologies, the number of AI and algorithmic systems with limited equity and justice considerations continues to increase. There are several ways in which AI systems, including the data and evidence on which they are trained, can cause harm, each with ethical, social, and equity implications. The accuracy and quality of the databases and the sometimes inconclusive or misguided evidence on which the algorithms are developed and implemented, shape decisions that have detrimental and adverse outcomes. A lack of explainability of data sources and transparency as well as design bias and limited evidence in the algorithms for AI, suggest how these issues are intertwined. The result is an exacerbation of structural inequities and adverse outcomes when disadvantaged populations are not included in trial data.^29,44

Another unfortunate consequence in product development is the mismatch of the intended use and subsequent actual use. This could happen when there is lack of accountability and moral agency for the entire process from design and development to implementation. For instance, consider an AI tool that may have been developed to identify a population to target with an intervention. Instead, the tool's use may result in discrimination against patients based on factors emphasized in the AI tool, thus influencing future treatment and reimbursement decisions and producing adverse downstream patient outcomes.⁴⁵ Documenting how the dataset was created, curated, validated, implemented, and shared will be important to the development of clinical care guidelines and clinical trials.⁴⁶ The AI Now Institute at New York University created the algorithmic impact assessment to provide awareness and improve processes to identify the potential harms of machine learning algorithms.⁴⁷

In another example, AI-supported clinical decision-support systems may be applied beyond the appropriate scope of use in under-resourced provider or patient settings with unintended consequences.⁴⁸ Human oversight and workflow integration are critical to safety, especially in settings where clinical experts are using clinical decision support systems (CDS) and other technologies and can help avoid harm to vulnerable

populations. Users of AI must maintain accountability when adverse effects arise, especially as some AI applications are maturing to full automation, such as the Apple Watch EKG app that received FDA clearance.⁴⁹ Artificial intelligence should generally be considered augmented intelligence to ensure that providers and patients are the final shared decision-makers.

Addressing algorithmic bias and ensuring data diversity have not been consistent in AI technologies' design and development. The AI development lifecycle should employ a strategic approach that considers health equity and ethical principles in managing the data, model-building, training, and deployment from conception to implementation. Gaps in the current data science and machine learning methods include addressing health equity and racial justice as fundamental requirements. Lifecycle processes that overlook health disparities may promulgate and perpetuate bias. Data often incompletely represent a target population.⁵⁰ The data and knowledge sources used to inform AI technologies require rigorous evaluation to ensure clinical performance, analytical performance, and scientific validity, promoting fairness and equitable outcomes. The black-box nature of AI technologies can act as a barrier to adoption. Unintended exacerbation of biases will be perpetuated if the output is not easily understandable or applicable to the user.⁵¹ For example, a tool that predicted a seven-day mortality risk or disease progression in a high-risk subpopulation might become outdated as new science, data, evidence, or methods evolve. Thus, it is essential to put humans in the loop for accountability in decisions that affect patient care.^52,53

A social concern is the impact of AI on patient-provider relationships. The human touch, empathy, understanding, and judgment are critical components of healing and patient care. Since positive health care encounters are built on relationships with patients, caregivers, and families, automated decisions or recommendations from an AI tool or algorithm can introduce new and possibly complicating elements into these interactions. Additionally, algorithms trained on and dependent on measurable data may not always capture relevant environmental information, social data, or patient cultural beliefs, preferences, and values. Social determinants of health (SDoH) such as educational level, economic insecurity, and other social factors contribute up to 40% towards determining health outcomes.^54–57 Another issue is the effect that AI may have on jobs and the potential task-shifting that comes with automation.^10–12 On a broader scale, the foundational evidence for AI tools must include all relevant populations' data to inform appropriate health equity interventions or decision-making.⁴⁶

The Lifecycle of AI Development in Health Care

Widespread implementation and application of AI in health care have lagged behind expectations due to several factors,⁵⁸ including a lack of robust, integrated data, inadequate trust to foster adoption, notable missteps in consideration of biases, disparities in expected targeted outcomes,^59,60 and challenges in integrating AI into complex workflows. In 2020, the National Academy of Medicine published a special publication on AI in Healthcare.⁶¹ One of the focus areas was a synthesis of best practices for developing, implementing, and maintaining AI systems used in delivering health care, summarized into a lifecycle framework (Figure 1, below). The AI development lifecycle is a continuous

Figure 1.

Ethical AI, Health Equity, and Racial Justice integrated across the Lifecycle of AI development.

Note:

Lifecycle phases (outer circle) adopted From the National Academy of Medicine, 2019, AI in Health Care: The Hope, the Hype, the Promise, the Peril. Reprinted with permission from the National Academy of Sciences, Courtesy of the National Academies Press, Washington, D.C.

process that begins by assessing needs, describing existing workflows, identifying and defining target states, acquiring infrastructure to develop the AI system, implementing the system, monitoring and evaluating performance, and maintaining, updating, or replacing the system when gaps or new needs arise. The lifecycle of an AI technology can provide a framework to identify opportunities to ensure that health disparity and social justice concerns are integrated into the genesis and application of AI solutions in public health and health care. Integrating health equity and racial justice principles into AI development requires building a responsible culture in innovation and establishing ethical building blocks for the reliable delivery of equitable AI technology.

Practical Applications of Health Equity and Racial Justice in AI Lifecycle Frameworks

We propose a framework in developing AI by incorporating health equity and racial justice principles into the different components of the AI lifecycle in health care. The proposed framework, shown in Figure 2, provides suggestions for every step of the lifecycle to consider equity and inclusivity and guard against biases.

The lifecycle of an AI technology can provide a framework to identify opportunities to ensure health disparity concerns are integrated into the genesis and application of AI solutions in public health and health care. The AI development lifecycle is a continuous process that begins by assessing needs, describing existing workflows, identifying and defining target states, acquiring infrastructure to develop the AI system, implementing the system, monitoring and evaluating performance, and maintaining, updating, or replacing the system when gaps or new needs arise.

Figure 2.

Framework for Integrating Health Equity and Racial Justice into AI Development.

Note:

Lifecycle of AI is from the National Academy of Medicine. 2019. AI in Health Care: The Hope, the Hype, the Promise, the Peril. Adapted with permission from the National Academy of Sciences, Courtesy of the National Academies Press, Washington, D.C.

In the context of ensuring that equity and fairness are central to the lifecycle, and aligned with what has been dubbed the Quintuple Aim,⁶¹ the first step of this framework includes identifying or reassessing needs that involve stakeholder, patient, and end-user engagement to ensure incorporating values of the target population. In this step, activities include defining objectives for an AI system that is aligned with promoting equity, including identifying data assets, data content, and policies for data stewardship.

The second step in this framework focuses on describing existing workflows and their effects on existing needs in policies, practice, feasibility, and workflow, as well as assessment of barriers and understanding the necessary training and resources to support the AI system.

The third step in this framework deals with the need to define desired target states. This step includes activities to establish the equity-sensitive metrics and key performance metrics related to the target outcomes. This step seeks to promote humility and self-awareness of systemic racism, discrimination, exclusion, and its effects on adverse health outcomes in socially disadvantaged populations.

The fourth step in this framework focuses on the task of acquiring and developing the AI system itself. Central to this step is understanding the relevant tools, techniques, and methods for data preparation, feature engineering, model training, and development. This step aims to promote and correct internal algorithmic bias in a way that advocates for justice in the development of AI and data-driven health systems by ensuring user-centered design justice principles are employed to uncover and address racial bias prejudices and unintended consequences of data and algorithms. Thus, defining and outlining steps are needed to integrate ethical AI fostering accountability, trust, transparency, fairness, and privacy and ensuring user-centered design justice principles to uncover and address bias, prejudices, and unintended consequences of the data and algorithms.

The fifth step in the framework focuses on implementing the AI system in the target setting and engaging with stakeholders, patients, and end-users in the implementation process in a way that fosters accountability, trust, transparency, explainability, fairness, and privacy.

The sixth step in the framework involves monitoring ongoing system performance to assess factors that include health equity measures in the processes, structures, and outcomes. These metrics include assessing how often the tool is accessed and used in the management and delivery of care, monitoring how often recommendations are accepted and implemented or not, and reasons for changes. Central to this is the requirement to monitor system performance against historical data and data generated in similar settings to assess changes in socio-demographics, practice patterns, and updates to scientific evidence and real-world data.

The seventh and final step in the framework involves activities focusing on maintaining and updating the system by conducting routine AI model maintenance and continuous training to ensure system performance reflects evolving clinical care environments, changing patient demographics, and new evidence being generated. Maintaining established trust and transparency with stakeholders and continuously maintaining and updating policies to ensure ethical AI principles, health equity, and racial justice are integrated in the system lifecycle.

Conclusions

During a public health crisis, AI's application holds great promise for augmenting decision-making, allocating scarce resources, and aiding in decision-making and policy formulation. Challenges persist in applying AI systems that can cause involuntary and unintended harm with profound ethical and social consequences. Merging the health equity and racial justice principles with AI Lifecycle provides a framework, approach, and a set of ethical values, principles, and techniques to guide moral conduct in the development of AI systems. Despite increasingly accurate AI tools, limited evidence exists on their applicability in real-world settings. One reason is the gap between proof-of-concept testing and clinical validation. For example, there is a clear process of scientific evaluation in drug development by which regulatory approval is achieved. Although many AI tools in health care are not regulated, a similar framework has been proposed for a systematic and comprehensive AI evaluation in health care to allow safe and effective adoption.⁶² The adoption of this framework and strategy, guided by justice principles, will support algorithm and tool developers, health systems, and researchers in creating user-driven innovations that fit within clinical workflows, facilitate interoperable information exchange, and evaluate AI in real-world health settings and proactively mitigate risks of exacerbating existing health disparities.