Key risks and challenges

Medical AI systems can propagate biases in multifaceted ways beyond just race or gender, with subtle root causes but serious impacts on equitable care.3 9–11 Data representation biases arise when algorithmic models are trained on datasets that under-represent certain populations, systematically disadvantaging them.10 12 For instance, an algorithm to predict 5-year mortality showed nearly 40% lower accuracy for black patients versus white patients due to under-representation of minorities in the underlying electronic health record data.10 Even well-intentioned efforts to mitigate representation bias can paradoxically penalise underserved groups further.9 13 Additionally, algorithm design choices can introduce bias even when training data are balanced across groups.3 9 Failing to account for differences in comorbidities, social determinants and structural disadvantages between populations can skew predictions and recommendations in ways that propagate disparities.9 This illustrates the subtle but impactful pathways through which biases emerge beyond just data representation issues.3 11 These interconnected sources of algorithmic bias demand nuanced governance approaches.9 11 While technical safeguards like high-quality datasets are crucial, oversight must also address root societal causes that infiltrate data and healthcare structures.11 As leading AI ethicists argue, ‘It takes an infrastructure to prevent an infrastructure from discriminating’.11 Beyond fairness, lack of transparency in commercial algorithms severely constrains accountability.6 14 Researchers have found that only 9 of 53 widely used medical AI systems allowed external validation of their performance claims.14 Absent mandates, developers lack incentives to enable auditing that could reveal deficiencies.6 The Food and Drug Administration’s (FDA) attempt to evaluate one ‘black box’ AI diagnostic device was unfruitful given its inscrutable design.6 This opaqueness leaves clinicians and regulators unable to properly assess hidden biases or monitor models over time.6 Evidence-based governance balancing transparency and innovation is essential but complex given competing interests.6 15 Approaches like holding proprietary algorithms’ methods as trade secrets while mandating disclosure of validation studies offer potential paths forward.6 Global harmonisation could also accelerate progress by preventing ‘forum shopping’ for lenient jurisdictions.15 Ultimately, realising AI’s promise requires reconciling public demands for accountability with developers’ incentives to protect competitive advantages.6 15

Global landscape analysis

While the EU mandates extensive transparency provisions for healthcare AI systems, the US FDA’s flexible approach risks deficiencies going undetected according to experts.6 The FDA’s recent AI/ML software as a Medical Device Action Plan acknowledges the need for greater international cooperation and clarity to avoid fragmented policy landscapes.16 Absent rigorous validation and monitoring, defective or biased algorithms could propagate inequities and cause preventable patient harm.2 3 One study found a widely used algorithm that underestimated health risks for black patients was likely not adequately vetted for racial disparities before deployment.3 Lack of transparency also erodes public trust in AI reliability.17 Thought leaders argue opaque development and deployment processes necessitate strong oversight to ensure safety and accountability.17 18 While the EU’s proactive transparency and oversight mechanisms are a step forward, critics argue the regulations lack sufficient binding enforcement mechanisms.18 The focus on premarket review also risks inflexibility in responding to issues arising post-deployment.19 Though pioneering, the EU framework has gaps to address. Comparatively, the FDA’s voluntary AI/ML software as a Medical Device Action Plan taps industry initiative but lacks binding enforcement mechanisms to compel accountability.6 The plan’s stated goals around Good Machine Learning Practice (GMLP) development, bias evaluation methods and real-world performance monitoring offer a framework for nuanced oversight.16 However, absent mandates, developers may still lack sufficient incentives to fully enable reviews revealing potential deficiencies.17 While the flexible approach avoids burdensome bureaucracy, experts note it relies heavily on corporate goodwill despite competitive disincentives.6 In contrast, the EU’s AI Act would implement binding requirements scaled by risk level.18 For ‘high-risk’ AI like healthcare systems, mandatory conformity assessments and transparency obligations aim to address accountability gaps, though some question if these regulations will be responsive to issues arising post-deployment.19 Analysis indicates the EU and FDA frameworks offer examples of nuanced attempts to balance innovation and oversight through tailored, cooperative approaches. However, experts underscore need to monitor their real-world efficacy at achieving safety and accountability goals.17–19 However, international harmonisation faces political and practical barriers.20 Competing national interests, intellectual property concerns and costs of aligning disparate regimes all pose challenges.20 But multilateral collaborations like the WHO’s new global AI health partnership may provide productive forums for finding common ground on core principles.18 International cooperation is essential for translating principles into real-world impact.17 Smaller-scale progress through transnational networks of experts and policymakers could also gain momentum towards responsible healthcare AI adoption worldwide.17 In summary, preventing unsafe or unethical healthcare AI depends on transparency coupled with gradual multilateral alignment on core principles of accountability and equitable access.17 18 21 Thoughtful regulation and global cooperation can enable AI to enhance care universally through gains in quality, safety and efficiency.

Examining governance for algorithmic equity

Effective governance is imperative to address algorithmic bias and ensure healthcare AI promotes equity. A critical perspective reveals important considerations across potential mechanisms including transparency, testing, auditing, collaboration and oversight. Transparency enables external scrutiny to uncover biases but appropriately scoping requirements remains challenging. Excessive transparency risks negative innovation incentives or privacy violations while limited visibility hampers accountability. Policymakers face difficult tradeoffs crafting rules balancing stakeholder interests. Pre-deployment testing methodologies are valuable but may also introduce new biases if datasets and benchmarks contain flaws. While testing surfaces some issues, real-world performance often diverges, underscoring the importance of ongoing monitoring after deployment. Independent auditing provides fresh perspective compared with internal testing. But legal and intellectual property (IP) barriers limit third-party access currently. Incentives and safe harbour protections may facilitate greater voluntary participation, but mandates may still prove necessary if progress lags. Collaboration on best practices offers benefits but views on acceptable bias levels still vary. Customised standards tailored to different use cases may suit the diversity better than one-size-fits-all edicts. Inclusive multistakeholder processes strengthen legitimacy and buy-in. Incentivising voluntary adherence to processes promoting fairness provides a flexible approach aligned with innovation culture. However, good faith alone risks ‘ethics washing’ without accountability mechanisms verifying outcomes. As Floridi argues, such washing concentrates on superficial ethical branding rather than enacting meaningful solutions.22 Hybrid governance blending top-down rules and bottom-up collaboration may prove most effective.

Stakeholder roles and incentives

Aligning healthcare AI with ethical obligations requires reconciling differing stakeholder motivations shaped by political and commercial forces.5 6 Companies prioritise rapid innovation, profits and IP protections.23 But opacity around algorithms and data to protect IP can enable deficiencies and biases to go undetected without transparency mandates.5 23–25 Proactive governance requiring explainability and validation, like the EU’s documentation requirements, is vital though may be initially opposed.5 23–25 However, heavy-handed restrictions may also limit progress without developer buy-in.24

Potential incentives include research and development tax credits, expedited reviews for voluntary accountability programmes and data sharing policies that protect IP.2 The UK’s collaborative development of AI best practices exemplifies multistakeholder cooperation.24 Patients and providers can also champion reforms through advocacy and expert panels.5 25 Groups like the American Medical Association (AMA) promote transparency, bias evaluations and human oversight.25 But enabling inclusive public input remains challenging.5 24

The ethical implementation of healthcare AI fundamentally requires reconciling the inherent tensions between privatised technological development and public health responsibilities. While incentives and collaborations can partially align stakeholder interests, the profit-driven model underlying healthcare AI likely necessitates some degree of external regulation and oversight to prevent capitalistic motives from wholly capturing development. However, finding the right balance is challenging. Overly permissive approaches risk allowing potentially unethical products to proliferate until issues emerge post-market and public trust erodes. Yet, excessive restrictions may limit innovation and availability, depriving many of healthcare AI’s benefits. In my view as a governance researcher, an evidence-based, iterative approach can help strike this balance. For instance, policy pilots allowing constrained rollout of new AI tools under heightened monitoring may help gauge real-world impacts. Data could inform continuously optimised regulation balancing innovation, accountability and access. Additionally, institutions like the WHO have published extensive guidance on the ethics of AI in healthcare, including recently on generative AI models. National health bodies also play vital roles synthesising cross-disciplinary academic research to derive best practices and core principles for healthcare AI. These can anchor binding regulations and voluntary programmes alike. Overall, the path forward likely requires policy humility—pragmatic trial-and-error governance evolution guided by multidisciplinary research illuminating healthcare AI’s complex sociotechnical dynamics. With careful stewardship and research, the profound risks posed by healthcare AI’s profit-based model can be mitigated to realise its full equitable potential. But this will demand nuanced, adaptable and data-driven cooperation between stakeholders. Innovative deliberative formats like online platforms and weighted lotteries for representation may help.23 In summary, cooperation across sectors coupled with incentives aligning social value and business value is key to optimising healthcare AI equitably.5 23–25 Further research on optimal collaborative governance models is warranted.

Realising healthcare AI’s equitable potential: a strategic governance framework

Advancing healthcare AI responsibly poses complex challenges, but incremental progress is achievable through targeted governance efforts centred on collaboration, participatory processes and evidence-based oversight.5 26 27 I propose policymakers pursue five strategic imperatives: first, implement premarket reviews calibrated to risks but expanded to require bias evaluations including analysis of training data representativeness and testing on diverse populations.5 26 27 This prevents overburdening innovation while mandating assessments of system biases and disparate performance. Second, use public engagement mechanisms like citizens’ juries to incorporate diverse perspectives, as research shows they enable inclusive deliberation on algorithm design and oversight.5 26 27 Directly engaging affected groups through such structured formats will productively surface biases and needs that should shape governance. Third, cultivate global capacity through collaborative networks connecting health agencies, particularly in the developing world, with regional centres of excellence in AI safety and ethics.27 Such partnerships can share expertise and co-create tailored frameworks addressing local priorities and constraints.27 This complements essential top-down efforts by supranational organisations.26 However, reticence around sharing sensitive technologies merits consideration. Fourth, augment premarket assessments with post-deployment surveillance infrastructure encompassing outcome monitoring, algorithm registration and adverse event reporting.2 14 26 28 This creates responsive feedback channels enabling agile interventions when issues emerge in complex real-world contexts.2 26 28 But lacking transparency into proprietary systems poses obstacles.14 26 Policy creativity is required to enable oversight absent full disclosure.14 26 Finally, support collaborative networks to develop best practices for bias detection techniques. Studies demonstrate approaches like clinical trial analyses stratified by social groups and independent algorithm reviews can improve bias evaluation.3

Developing and aligning on standardised bias detection methods will require tapping multidisciplinary expertise across stakeholders. Incentives tying preferential policies to voluntary best practices adoption can also help bridge gaps between business and social obligations.2 26 29 However, scepticism persists regarding the efficacy of self-regulation.26 Oversight mechanisms must therefore verify adherence and accountability. Realising equitable healthcare AI poses complex scientific challenges. Merely asserting aspirations would not progress solutions. Instead, we must illuminate core obstacles transparently. For instance, algorithmic biases emerge from myriad sources—insufficiently diverse training data, mismatching population data distributions, poorly constructed feature engineering and judgement criteria, among others. And biases propagate in equally multifaceted ways. Solely solving discrete technical issues often proves inadequate. Truly overcoming algorithmic biases demands a systems perspective encompassing the sociotechnical complexities of how data are generated, algorithms are constructed and outputs are used in real-world contexts. This necessitates cross-disciplinary expertise spanning computer science, healthcare, ethics, social science, regulatory policy and beyond. Collaborative, participatory processes engaging diverse stakeholders can surface insights and constraints that technologists alone may overlook. Community partnerships, advocacy boards and participatory design are invaluable but underemployed resources. Healthcare AI equitably enhancing access and outcomes is possible but will require scientific humility—acknowledging the inherent uncertainties and multidimensional challenges. With transparent analysis and collective determination, the research community can elucidate solutions. But this begins with clearly explaining the complex problems themselves. In closing, advancing healthcare AI equitably is surmountable through evidence-based, cooperative governance.26 27 We can fulfil these technologies’ promise through pragmatic progress grounded in expertise and shared values.