Natural-language prompts now build and deploy software for hospitals, aged care and the NDIS in minutes. The evidence says the guardrails did not come with it.
In February 2025 the computer scientist Andrej Karpathy described a new way of building software: “fully give in to the vibes, embrace exponentials, and forget that the code even exists”1. He meant it half in jest. Within nine months Collins English Dictionary had named “vibe coding” its word of the year2, and platforms such as Lovable, Replit, Bolt and v0 were converting natural-language prompts into deployed, internet-facing applications in minutes.
For health and care services the appeal is obvious. Clinical and care teams have long waited months for modest digital tools, and regional hospitals, aged care homes and NDIS providers in particular rarely have software engineers to spare. A nurse unit manager can now describe a rostering tool over a lunch break and have something that works by the afternoon; a support coordinator can do the same for participant scheduling. That capability is real, and this article does not argue against it.
The difficulty lies in what the shortcut removes. Conventional software engineering is slow partly because it is deliberate: requirements are documented, code is reviewed by a second person, access controls are designed before data arrives, dependencies are vetted, and changes pass through testing environments before they touch production. Each of these steps exists because something once went wrong without it. Vibe coding strips this friction away wholesale, replacing deterministic engineering with statistical text generation. In most industries that trade-off produces embarrassment. In health, aged care and disability services, where the data describes patients, residents and participants and the software can shape decisions about their care, the same trade-off produces reportable breaches, regulatory exposure and, plausibly, harm.
Evidence · 01
The headline figures are now difficult to dismiss as anecdote. Veracode’s 2025 GenAI Code Security Report tested more than 100 large language models against 80 curated coding tasks and found that the models introduced a known vulnerability class in 45 per cent of cases3. Java fared worst, with failure rates above 70 per cent. For two vulnerability types with long histories in healthcare breaches, the numbers were starker still: models failed to defend against cross-site scripting in 86 per cent of relevant tasks and against log injection in 88 per cent3.
Two details in that study deserve executive attention. First, newer and larger models wrote more syntactically correct code but not more secure code; security performance was essentially flat across model generations3. The pattern suggests a structural cause rather than a maturity problem. Language models are trained on vast corpora of public code, much of it insecure, and they optimise for plausible output rather than verified correctness. Academic work reached a similar figure years earlier: Pearce and colleagues found in 2022 that around 40 per cent of GitHub Copilot’s outputs in security-relevant scenarios were vulnerable4. Three years of rapid capability gains barely moved that base rate.
Second, the human side of the equation may be worse than the machine side. In a controlled study at Stanford, participants with access to an AI coding assistant wrote measurably less secure code than those without, yet rated their own code as more secure5. This overconfidence effect matters for governance design, because it predicts that the people least equipped to audit generated code will be the most inclined to trust it.
Field evidence points the same way. Symbiotic Security scanned 1,072 applications built with prompt-driven platforms and found at least one vulnerability in 98 per cent of them, with 29 per cent carrying critical or high-severity flaws; 172 sites permitted deletion of records without any authentication at all6. A separate exercise by the security firm RedAccess discovered more than 380,000 publicly accessible vibe-coded applications; of roughly 5,000 built for corporate purposes, about 40 per cent were leaking sensitive material, including medical records and clinical trial data, through open URLs that required no exploitation whatsoever7. These scans come from security vendors with an interest in alarming findings, and their sampling frames are opportunistic, so the precise percentages warrant caution. The direction and consistency of the evidence, however, are hard to explain away.
Evidence · 02
Four failure modes recur across the audits and incidents, and they map poorly onto the threat models most health, aged care and NDIS boards currently review.
Access controls that were never built. Prompt-driven platforms frequently ship applications whose databases accept unauthenticated reads, writes or deletions, or whose authorisation checks can be bypassed by changing an identifier in a URL6. The model produces an interface that looks finished, and nothing in the interface reveals that the back end will hand the same data to anyone who asks.
Credentials in the open. Generated code routinely embeds database keys, passwords and third-party API tokens directly in client-side files that any visitor can read. In the Symbiotic sample, 308 of 1,072 sites exposed their database access key in public JavaScript6. A hardcoded key is not a subtle flaw; it is the digital equivalent of taping the ward keys to the front door.
Hallucinated dependencies and “slopsquatting”. Language models invent software package names that do not exist. A large multi-model study found that 19.7 per cent of packages recommended by code-generating LLMs were fictitious, with open-source models hallucinating at roughly four times the rate of commercial ones, and the same fictitious names recurring predictably across prompts8. That predictability is what makes the behaviour exploitable: attackers register the hallucinated names on public package registries and wait. When researcher Bar Lanyado uploaded a harmless package under one commonly hallucinated name, it was downloaded more than 30,000 times in three months9. The attack class, now called slopsquatting, turns the model’s imagination into a supply-chain entry point.
Shadow AI. The ease of creation has produced a proliferation of undocumented applications sitting outside any asset register. RedAccess titled its report “The Shadow Builders Inside Your Organization” for a reason: most of the exposed corporate applications it found had been built by staff without IT’s knowledge7. In a hospital, aged care home or NDIS provider this means rosters, handover notes, incident registers, participant plans and audit spreadsheets flowing through tools with no data governance, no audit logs and no identifiable owner. The organisation cannot secure what it does not know exists.
Evidence · 03
Abstract failure rates land differently when attached to named incidents. Four are well documented.
Replit agent deletes production database (July 2025). During a public build experiment, Replit’s coding agent deleted a live production database holding records on more than 1,200 executives, despite an explicit instruction freeze, then generated thousands of fictitious records and misreported its own actions10. Staging and production must be technically, not procedurally, separated; an instruction to an LLM is not a control.
Lovable platform exposure (2026). A broken object-level authorisation flaw let any free account holder retrieve other users’ source code, database credentials and chat histories; 48 days passed between private report and public disclosure, and existing projects were not retrofitted11. The platform itself is part of the attack surface; a customer’s own diligence cannot patch the vendor.
EnrichLead SaaS compromise (March 2025). A founder publicly launched a paid service built “with zero hand written code”; within days attackers bypassed subscriptions, exhausted API keys and corrupted the database, and the service shut down12. Speed to market is not an asset if the first users are attackers.
“huggingface-cli” slopsquatting proof (2024). A researcher registered a package name that multiple LLMs persistently hallucinated; the empty package drew more than 30,000 downloads and was referenced in a major company’s own repository9. Dependency vetting must assume the model will invent plausible, wrong names.
None of these incidents involved sophisticated adversaries. Each exploited the absence of ordinary engineering discipline, which suggests that the risk is tractable: the controls that would have prevented them are decades old and well understood. What is new is the volume of software entering service without them.
Speed to market is not an asset if the first users are attackers.
A prompt builds a working app in minutes. Governance makes it safe: the review, access controls and audit trails that vibe coding skips must be put back before patient, resident or participant data arrives.
Analysis · 01
Commentary on AI code risk is dominated by United States framing, with HIPAA cited as the governing regime. For Australian health organisations that framing understates local exposure, because the applicable instruments are closer to home and have recently grown teeth.
Health service providers were again the economy’s leading source of notifiable data breaches in 2025, lodging 225 of 1,205 notifications (19 per cent) to the Office of the Australian Information Commissioner, the highest annual total since the scheme began; the majority of breaches economy-wide arose from malicious or criminal attack13. Privacy Commissioner Carly Kind described the threat as “substantial and rising year on year”13. The sector’s capacity for harm at national scale was demonstrated by the MediSecure incident, in which prescription-related personal and health information concerning approximately 12.9 million Australians was compromised14.
The legal environment has tightened in three ways that bear directly on vibe-coded software. The Privacy and Other Legislation Amendment Act 2024 (Cth) clarified that the “reasonable steps” required to secure personal information under Australian Privacy Principle 11 include technical and organisational measures, a formulation that makes the absence of basic controls difficult to defend15. The same reforms introduced a statutory tort for serious invasions of privacy from June 2025, giving affected individuals a direct cause of action, and will require transparency about automated decision-making in privacy policies from December 202615. These sit atop penalty provisions, enacted in 2022, that expose companies to fines of up to $50 million or 30 per cent of adjusted turnover for serious or repeated interferences with privacy.
Separately, the TGA clarified in guidance published in February 2026 that software, including LLM-based and generative tools, is regulated as a medical device when its intended purpose includes diagnosis, prevention, monitoring, prediction, prognosis or treatment16. The test turns on intended purpose, not on how the tool was built or how it is marketed. A vibe-coded symptom checker or triage aid deployed by a clinic may therefore be an unapproved therapeutic good, with the essential principles, including cybersecurity requirements, applying in full, and the TGA has flagged software-based devices as a compliance priority through 202716. The Australian Commission on Safety and Quality in Health Care’s AI Clinical Use Guide, published in August 2025, places accountability for AI tools squarely within existing clinical governance structures rather than treating them as an IT matter17. Its warnings read as a direct rejoinder to prompt-built software: generative tools create information that may not be factual, outputs must be verified before they enter a patient record, an ambient scribe that adds treatment recommendations becomes a medical device requiring ARTG listing, and the clinician remains responsible for the accuracy and completeness of whatever the tool produces17.
The exposure does not stop at hospitals. Aged care and NDIS providers hold the same classes of sensitive information, often more of it per person, and answer to sector regimes of their own: the new Aged Care Act and strengthened Quality Standards, in force since November 2025, make information management an audited provider obligation21, while registered NDIS providers must meet the NDIS Practice Standards, with information management requirements and a Code of Conduct enforced by the NDIS Quality and Safeguards Commission25. A staff-built participant-roster app with an open database would breach both privacy and sector rules at once, and these providers typically carry less in-house security capability than hospitals, not more.
The combined effect is that an unauthenticated database behind a staff-built scheduling app, whether it sits in a hospital, an aged care home or an NDIS provider, is no longer merely an IT hygiene issue. It is a potential APP 11 contravention, a notifiable breach, a tort claim and, where the tool has a clinical purpose, a therapeutic goods matter, simultaneously.
Response · 01
Clinical AI scholarship anticipated this problem before generative tools arrived. Writing in 2021, Bajwa and colleagues described safe clinical AI as a lifecycle discipline: design with stakeholders, validate statistically and clinically in real settings, scale deliberately, then monitor in service for drift and adverse events, with AI augmenting rather than replacing human judgement22. Vibe coding compresses the first stage into a sentence and deletes the other three. The remedy is therefore less a new invention than the reassertion of an old discipline.
The case for the two headline frameworks is best put as the two questions a buyer cannot otherwise answer. ISO/IEC 27001 answers the first: does this organisation run security as a managed, continuously improving system, checked by an accredited outsider, rather than as a set of good intentions? SOC 2 Type II answers the harder second: did its controls actually operate, week after week, across the audit period, on evidence an independent examiner has reviewed? A vendor can assert both without the frameworks. It cannot prove either, and in a sector where the buyer rarely has the capability to inspect a codebase, proof is the only currency that should count.
If the core problem is removed friction, the remedy is to reintroduce friction deliberately, and external assurance frameworks are the most practical instrument available for doing so. ISO/IEC 27001:2022 is an international standard for managing information security as a continuing organisational system, certified by accredited third-party audit18. SOC 2 is an attestation regime, developed by the American Institute of Certified Public Accountants, in which an independent auditor examines an organisation’s controls against defined trust services criteria; a Type II report tests whether those controls actually operated over a period of months, not merely whether they existed on paper19. Neither was designed with generative AI in mind. Both, however, mandate exactly the practices that vibe coding discards, and Figure 1 maps the correspondence. (Note that the 2022 revision of ISO/IEC 27001 restructured Annex A; the A.12, A.14 and A.15 control numbers still quoted in older guidance have been superseded.)
A third instrument has joined them. ISO/IEC 42001:2023, the first certifiable management system standard for artificial intelligence, extends the same discipline from the pipeline to the AI itself: organisations must maintain an AI policy, run risk and impact assessments for each AI system, control data and models across their lifecycle, keep humans in positions of meaningful oversight and monitor behaviour once in service24. Where ISO/IEC 27001 secures the information around an application, ISO/IEC 42001 governs what the AI system does and how it changes over time, which is precisely the gap a prompt-built tool leaves open. Adoption in Australian health settings remains early, so certification against it is still a differentiator rather than a baseline, though that gap seems unlikely to last.
Two features make these frameworks suited to the problem. They are auditable from outside, which means a hospital procuring from a healthtech vendor can demand evidence rather than assurances. And they regulate the system around the code rather than the code itself, which is the correct target: the evidence reviewed above indicates that generated code will remain unreliable for the foreseeable future, so safety must come from the pipeline that catches defects, not from hoping the generator improves.
The field scans make the contrast concrete, because nothing in those samples appears to have been built under such a regime. The 1,072 vibe-coded applications in which 98 per cent carried flaws, and the thousands found leaking medical records through open URLs, are in effect the uncontrolled arm of a natural experiment6,7. Without the frameworks, the measured risks do not merely exist; they compound. Generated code arrives with a roughly 45 per cent chance of carrying a known vulnerability class3, no reviewer stands between that code and production, no access control limits who reaches the data, no monitoring detects the exploitation, and no logs exist to establish what was taken, which converts a single defect into an open-ended, unreportable breach. A certified pipeline does not make the generator write better code. It intercepts the same defect at four separate points before a patient, resident or participant record is exposed, and it leaves evidence at every one of them. That difference in failure shape, bounded and observable against open-ended and silent, is the practical meaning of the certificates on a vendor's trust page.
Nor is governance the enemy of speed. Digital pathology illustrates the pattern: platforms such as Philips IntelliSite entered laboratories as regulated in vitro diagnostic devices, add AI through vetted integration partners rather than open prompts, and still report material gains, with the vendor citing efficiency improvements of 15 to 20 per cent per case and larger productivity gains where AI triage is integrated23. Those figures are the vendor’s own and warrant the usual scepticism, but the direction is instructive: the clinical AI delivering value at scale in Australian health services today is the governed kind, not the improvised kind.
Response · 02
A candid review should also state what certification cannot do. An ISO 27001 certificate attests to a management system within a defined scope, and scopes can be drawn narrowly; audits sample rather than exhaustively test; and a SOC 2 report evaluates the controls an organisation itself defined. The Lovable disclosure illustrates a further limit: when the flaw sits in the platform, a customer’s own certification offers no protection, which is why supplier controls (Annex A 5.19 to 5.23) and contractual audit rights matter as much as internal discipline11. Certification is scaffolding for trustworthy engineering, not a substitute for it, and in clinical settings it operates alongside, never instead of, clinical governance and TGA obligations. Treating a badge as the end state would simply relocate the illusion the frameworks exist to dispel.
Response · 03
The governance response does not require new theory, but it does require sequencing. Discovery comes first: commission a scan for AI-built applications already operating inside the organisation, because the RedAccess findings suggest most institutions are running more of them than they know7. Policy follows: a clear rule that no identified patient, resident or participant information enters any application that has not passed security review, paired with a sanctioned, governed pathway for staff who want to build, since prohibition without an alternative reliably produces more shadow IT rather than less. Procurement is the third lever: contracts with healthtech vendors should require current ISO/IEC 27001 certification or a SOC 2 Type II report whose scope covers the relevant service, plus disclosure of how AI-generated code is reviewed before release. For any tool whose function approaches diagnosis, triage or treatment support, the TGA question must be asked in writing before deployment, not after16. The ACSQHC guide supplies a ready screening set for any proposed tool: what evidence supports its accuracy and safety, is its training data representative of the local population, is it a TGA-regulated device, and what governance will monitor it once in service17. And boards should expect management to report on this quarterly, in the same rhythm as clinical incident data, because that is what the OAIC statistics say it now is: a principal source of institutional harm13.
We build with these essential guardrails as our highest priority. At Amalthēa Health, frameworks like ISO/IEC 27001:2022 secure development, ISO/IEC 42001 AI management, SOC 2 operating discipline, and the ASD Essential Eight are foundational design principles, never an afterthought18,19,20,24. Every single release is carefully reviewed, security-tested, and staged before it ever touches care data. We treat AI-assisted code with rigorous, evidence-based diligence, because deploying medical systems shouldn’t feel like a roll of the dice, and you should never gamble your data security on a raw, unchecked “vibe-coded” application.
Care providers safeguard some of the most sensitive data in the country, often on thin budgets, while navigating APP 11, the statutory privacy tort, the strengthened Aged Care Quality Standards, and the NDIS Practice Standards15,21,25. Building a secure development function isn’t your job; it’s ours. We carry the compliance burden and provide the evidence your auditor needs, allowing you to move fast and adopt AI safely.
Talk to us today to set up these safeguards early and ensure a smooth, supported path forward.