Australian health services are being reorganised around a new question: not whether a clinical algorithm works, but how a hospital can prove it is still working, for every patient, long after go-live.
Australian hospital boards once asked whether a clinical algorithm worked. The harder question, the one now surfacing in audit and risk committees, is whether anyone can show it is still working. Not at the vendor’s site. Here, on this population, this quarter. The distinction sounds small. It is not. A growing external validation literature indicates that performance claims made at development rarely survive contact with a new health service unchanged, and that the decay is often silent.2,3
Governance frameworks built for static software struggle with this. A model is approved once; a patient population changes continuously. The gap between those two facts is where clinical risk now accumulates, and it is reshaping how health IT departments are organised.
Pressure · 01
The TGA has clarified when AI-enabled software is a medical device in its own right, which places adaptive and generative systems inside the therapeutic goods framework and makes sponsors answerable for evidence across the product’s life.5 Comparable requirements are consolidating overseas. The EU AI Act began applying its core obligations in August 2026, and the United States now requires certified electronic health records to disclose thirty-one “source attributes” for predictive tools, a format sometimes described as a nutrition label.9,10 Australian procurement sits downstream of both markets, so local hospitals inherit these disclosure norms whether or not domestic law compels them.
Whether labels change behaviour is a fair question. Experience with drug and device labelling suggests disclosure alone rarely alters clinical practice. What a label does create is an auditable record against which a hospital can be held to account, and that may be its real function. A transparency label is only as reliable as the provenance data behind it. Few Australian health services could currently answer, with documentation, which version of a model produced a given recommendation six months ago.
Pressure · 02
The canonical demonstrations are American, but the mechanism travels. Obermeyer and colleagues showed that a widely used risk model had learned healthcare cost as a proxy for healthcare need, roughly halving the number of Black patients identified for additional care.1 Wong and colleagues found that a proprietary sepsis model missed two thirds of sepsis admissions in external validation, with an area under the curve of 0.63 against substantially stronger developer claims.2 Neither failure was visible from inside the software.
For Australia the translation is direct. Most models arriving in local hospitals were trained on overseas populations, so performance for Aboriginal and Torres Strait Islander patients, for rural and remote communities, and for the culturally and linguistically diverse catchments of metropolitan services cannot be assumed from published validation. It has to be measured. Work on dataset shift indicates that the problem compounds over time even where initial local validation is sound.3
Govern, validate, deploy, monitor, audit — and back again. Monitoring returns models to validation when performance or equity drifts; audits return findings to governance. Safety work never reaches a terminal state.
Response · 01
The institutional response, in Australia as elsewhere, has been to move oversight out of ad hoc ethics review and into standing structures. The Australian Commission on Safety and Quality in Health Care now folds algorithmic oversight into clinical governance and has published pragmatic guidance for clinical AI use.7 Ahpra has reminded registered practitioners that professional obligations follow them into every AI-assisted decision.8 Internationally, the Joint Commission and the Coalition for Health AI released system-level guidance in 2025 and a voluntary certification in 2026,11 and a nationwide network of assurance laboratories has been proposed on the model of reference pathology.12
There is a live debate about where assurance should sit. Centralised laboratories promise rigour and economies of scale; local validation captures the population a model will actually see. The two are complements rather than substitutes, though budgets tend to force a choice. For most Australian services the defensible minimum is local: an inventory of every algorithm in clinical use, risk-tiering at intake, validation on local data before go-live, and the standing authority to say no.
Response · 02
Embi’s term algorithmovigilance captures the required stance: continuous surveillance of deployed algorithms for effectiveness and equity, on the model of pharmacovigilance.4 The analogy is imperfect. Adverse drug events surface through clinical presentation; a drifting model fails statistically, in aggregate, and may harm no identifiable patient in any given week. Detection has to be designed. That means monitoring dashboards in production, pre-committed performance thresholds tied to named owners, and change control on model updates equal to that applied to any clinical system. Training pipelines are attack surfaces in the ordinary cybersecurity sense. Australia’s Voluntary AI Safety Standard, ten guardrails the Commonwealth has proposed making mandatory in high-risk settings, offers boards a workable vocabulary for this design.6
Retirement is the hardest discipline. A model whose evidence has expired should be treated like an expired therapeutic good, which is to say withdrawn, however useful it still feels at the bedside.
For a chief executive the message reduces to ownership. AI safety belongs where infection control sits: a standing function with a budget, an inventory and a reporting line to the board. On present evidence, the Australian services adopting AI fastest appear to be those that built the brakes first. A model that can be explained, monitored and switched off is a model clinicians will use.
Amalthēa Health is building a Clinical AI solution that prioritises governance and security and can be onboarded onto your existing ecosystem. We are now reviewing requests for early access, join the waitlist.
Is your board starting to ask how it would know its algorithms are still working? That is the right question, and we would welcome the conversation.