Artificial intelligence may become one of the most transformative technologies healthcare has ever adopted. I believe it will.
But in our rush to put AI into everything, there is a question we may not be asking often enough: Who audits the AI?
I recently read a Becker’s Hospital Review article, “The meter is running: Health system CIOs tame AI token costs.” Healthcare CIOs described something many organizations are beginning to discover as AI moves from small pilots to enterprise-wide deployment: every AI interaction has a cost.
At Jefferson Health, for example, a task that might have taken an employee two weeks was completed using AI in about four hours for roughly $75 in token costs. That is an impressive return. But the CIO also made an important observation: repeat that kind of consumption every day, across thousands of employees and workflows, and those costs can grow quickly.
That made me think about a much bigger issue. Cost may actually be one of the easier AI problems to solve.
What Does It Cost to Know the AI Was Right?
Most AI business cases are straightforward.
If AI can perform work that currently requires thousands of employee hours, organizations can reduce costs, increase productivity, or allow employees to concentrate on more valuable work.
But there is another line that belongs in that ROI calculation: What does it cost to validate the AI’s work?
Suppose an AI system evaluates 100,000 healthcare transactions. It gets 99,000 right. That is 99% accuracy. Pretty impressive.
Except there are still 1,000 wrong transactions. How do we find them? More importantly, how do we even know those 1,000 errors exist?
Asking the AI whether it made a mistake is not an audit. Having another AI check the first AI may help, but eventually we still need an independent way to establish what the correct answer should have been.
That is not an argument against AI. It is an argument for AI assurance.
Healthcare Already Knows This Lesson
This problem is not entirely new. Healthcare organizations have spent decades implementing complex technology and then discovering an important truth:
Just because a system processed something successfully does not mean it processed it correctly. At Softek, we have seen this firsthand working with hospitals using Oracle Health Millennium. A charge can successfully move from one system to another and still be wrong. A workflow can complete exactly as designed and still produce the wrong business outcome. An interface can be green while the data traveling through it is not. That is why one of the principles behind Softek’s work has essentially been:
Trust, But reconcile
Do not assume the system is wrong. But do not assume it is right simply because it did not throw an error, either. Compare what should have happened with what actually happened. Identify the exceptions. Investigate the differences. Fix the underlying problem. AI does not eliminate that principle. AI makes it more important.
AI Introduces a Different Kind of Problem
Traditional software generally follows predefined logic. If X happens, do Y. We can inspect the logic, recreate the conditions, test the result, and usually determine why something happened. Generative AI is different. Its output can vary. It interprets information. It can make judgments. And perhaps most importantly, an AI-generated answer can sound completely reasonable while being completely wrong. Now imagine AI agents performing increasingly complex healthcare tasks.
- What happens when coding guidance changes?
- What happens when a payer changes a policy?
- What happens when a clinical protocol changes?
- What happens when CMS changes a rule?
- What happens when an organization changes its internal procedures?
- Who makes sure the AI adapted correctly and who proves it?
Recent research from UPMC’s Center for Connected Medicine and KLAS Research suggests healthcare is still working through exactly this problem. Their 2026 research found that only 44% of surveyed organizations had a dedicated data platform or environment for testing AI solutions, while 63% described their AI strategy as developing or ad hoc.
The technology may be moving faster than the systems we need to govern it.
Trust Is Not the Same as Proof
This is becoming important enough that the Joint Commission launched its Responsible Use of AI in Healthcare Certification in 2026. Its major areas of focus include governance, risk reduction, and something particularly important: monitoring, evaluating, and validating AI throughout its lifecycle.
Think about that for a moment. Not just validating AI when we purchase it. Continuously validating it. That distinction matters because healthcare changes constantly—and AI systems change too. An AI solution that worked correctly six months ago cannot simply be assumed to work correctly today.
A recent CIO article described this as the “AI assurance gap”: organizations may have AI controls, policies, dashboards, and logs, but eventually someone will ask a harder question: Can you prove those controls actually worked? That is where healthcare needs to go next.
Ask Questions
Imagine an AI agent managing part of a revenue-cycle workflow. Instead of simply measuring how many transactions it processed, we should also ask:
- What decisions did it make?
- Which decisions fell outside expected patterns?
- What changed after a model, policy, or workflow update?
- Can we independently validate the expected result?
And when the AI gets something wrong, can we identify the mistake before it affects the patient, clinician, claim, or hospital’s financial results? Those questions should be part of the architecture—not questions we ask after something goes wrong.
Healthcare does not simply need artificial intelligence. It needs observable AI. Auditable AI. Measurable AI. And ultimately, accountable AI.
For years, Softek has helped healthcare organizations find discrepancies hidden inside complex EHR workflows by comparing what systems did with what should have happened. As AI becomes another participant in those workflows, that same philosophy becomes even more important.
Is it Working?
Because the most important question about healthcare AI may eventually stop being:
What can it automate?
Instead, we may find ourselves asking three much simpler questions:
- Is it working?
- Is it correct?
- Can we prove it?
Because eventually a patient, clinician, CFO, compliance officer, auditor, regulator, or board member will ask. And “the AI said so” will not be a good enough answer.
Further Reading
Becker’s Hospital Review — “The meter is running: Health system CIOs tame AI token costs” — August 11, 2026
UPMC Center for Connected Medicine / KLAS Research — “Validation and Trust: The Governance of AI Solutions at Health Systems” — August 6, 2026
Joint Commission — Responsible Use of AI in Healthcare Certification — 2026
CIO — “The AI assurance gap: CIOs need proof that agentic AI controls actually work” — August 4, 2026
What does Softek® do?
Softek helps hospital systems get the most value from their Oracle Health EHR investments. We combine innovative software with expert consulting to deliver results that neither can achieve alone.
Our team of experienced developers and industry specialists works closely with hospitals across the country to optimize EHR performance and strengthen revenue integrity. We go beyond standard solutions, bringing deep expertise and practical insight to every engagement.
From system performance assessments to revenue cycle and patient accounting optimization, Softek offers a comprehensive suite of services and tools designed to improve outcomes and efficiency.
Let’s connect and explore how to maximize the performance of your Oracle Health EHR system.
