# Systemic AI Safety: Challenges & Capabilities Assessment ### Overview and guidance This framework defines the 4 core challenge areas associated with managing the systemic safety of AI in healthcare. Each of these areas is defined by a key challenge and a set of capabilities required to meet that challenge. These are structured as a set of questions which provide a step-by-step approach to assessing and exploring the challenges and capabilities of managing systemic AI safety across healthcare. All healthcare organisations and system stewards, at all levels, will need to engage with these challenges and develop safety capabilities commensurate with the speed, scale, complexity and risk of AI deployment. Capabilities for managing systemic AI safety can build on existing infrastructure but will also pose new requirements, particularly regarding system-wide collaboration, escalation and integration. Each challenge and capability should be considered from a sociotechnical perspective: each assessment question should be explored with respect to the sociotechnical capabilities needed to manage systemic AI safety. Further explanation of the importance of these questions can be found in this journal article: ["What happens when 'AI kills a patient'?"](https://doi.org/10.1177/01410768261455555) (Macrae, 2026). ### Sociotechnical capabilities **S: Structural** - Capabilities arising from interdependencies and interactions between technical and social structures **O: Organisational** - Capabilities arising from social processes, organising activities and human and contextual factors **T: Technological** - Capabilities arising from affordances and constraints produced in and by material technologies **E: Epistemic** - Capabilities arising from the ways that knowledge and ignorance are constructed **C: Cultural** - Capabilities arising from collective values, beliefs, norms and practices (See Macrae, [2022](https://doi.org/10.1111/risa.13850) and [2025](https://doi.org/10.1111/risa.14273) for further details.) # AI Safety Challenges & Capabilities: Assessment Questions ![[AI capability and challenges assessment.pdf]] ## 01: Identification and detection ### Key challenge How will we know if an AI technology has contributed to patient harm, or is posing an ongoing risk to the safety of patients? ### Capability requirements A. How will continuous monitoring of AI safety and performance be performed and organised, and who should coordinate and supervise these processes? B. How will specific sources, patterns and indicators of AI risk be defined and monitored, and who will determine and detect when thresholds for further action? C. How will healthcare staff and service users be supported to understand, recognise and report concerns related to AI safety and performance, and who will review these signals? ## 02: Collaboration and information ### Key challenge What data will we need to understand AI risks, who will have it and how will we collaborate to share and use it? ### Capability requirements A. How will all the sources and forms of relevant AI safety data be defined, and how and by whom will these data be routinely collected and maintained? B. How will AI safety data and evidence be shared and accessed by all the actors who collect, hold and need it, and what mechanisms and agreements will assure access? C. How will stakeholders collaborate and contribute expertise to collate and integrate AI safety data, and who will critically interrogate and validate this evidence? ## 03: Explanation and investigation ### Key challenge How will we understand the sociotechnical patterns of risk associated with AI in complex care processes, and what analytical strategies and expertise will be needed? ### Capability requirements A. How will the complex sociotechnical sources of risk (human, technical, organisational, clinical) be routinely and rigorously analysed across different levels and arenas of the healthcare AI ecosystem? B. How will multidisciplinary teams expert in AI safety investigation and analysis be created and supported, how will these teams be organised and where will they be based? C. How will systemic AI safety analysis be conducted in ways that integrates complex domains of knowledge, and develop clear and actionable explanations of sources of risk? ## 04: Accountability and improvement ### Key challenge How will we determine responsibility and accountability for AI safety, and how will improvement be organised and coordinated across the healthcare system? ### Capability requirements A. How will responsibility for AI risks and safety improvement be allocated, monitored and governed across all relevant stakeholders, and who will oversee this? B. How will AI safety improvement and risk mitigation activities be targeted and coordinated across the system, and who will support these improvement efforts? C. How will AI risks and safety insights be shared across the system, and who will ensure that improvements and mitigations are effectively implemented? --- Carl Macrae 2026. [[About the AISIM project]] This work is licensed under <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a><img src="https://mirrors.creativecommons.org/presskit/icons/cc.svg" alt="" style="max-width: 1em;max-height:1em;margin-left: .2em;"><img src="https://mirrors.creativecommons.org/presskit/icons/by.svg" alt="" style="max-width: 1em;max-height:1em;margin-left: .2em;">