Last Saturday I flew to San Francisco for my first hackathon: The Future of Agentic AI in Healthcare, hosted by Abridge with Anthropic and Lightspeed at Shack15. Seven hours to build. 113 teams. I competed alone and did not make the final round.
The story of the day is on LinkedIn. This post is about the system, because the system is the part that matters on Monday.
The problem
Prior authorization is the process where a clinic must prove to a payer that a treatment is medically necessary before it gets covered. In the AMA’s latest physician survey, 95 percent of physicians said prior auth delays access to necessary care, and practices average around 40 requests per physician per week. It is paperwork that consumes clinical time and stalls treatment.
On the surface this looks perfectly agent-shaped: read a policy, read a chart, write a justification. But there is a second number that should shape any system you build here. In the same survey cycle, 61 percent of physicians said they are concerned AI will increase denial rates. The people this tool is for are already braced for AI that confidently produces the wrong answer at scale.
So the domain does not need a faster guesser. It needs a system whose confidence is earned, and whose uncertainty is visible.
Start from the output type
Most agent demos have two implicit outcomes: it produced an answer, or it errored. I started greenlight from a different place: the type signature of the final verdict.
Verdict = APPROVE | DENY | INSUFFICIENTThat third variant is the whole design. INSUFFICIENT is not an error state or a timeout. It is a first-class result meaning: the evidence available does not support a decision, and here is exactly which criteria are unresolved. The system is contractually allowed to say I don’t know, and everything else in the architecture exists to make that answer specific, auditable, and rare.
This is what fail-closed means in practice. A wrong APPROVE is a coverage decision made on hallucinated evidence. A wrong DENY delays a patient’s care with machine confidence. INSUFFICIENT hands the case back to a human with the unresolved criteria highlighted. Of the three ways to be wrong, it is the only one that is recoverable.
Where the model is allowed to work
greenlight is a six-stage pipeline. The model works in the middle. Code decides at the edges.
Parse. A deterministic parser turns the patient’s FHIR bundle into a typed PatientContext. No LLM touches this stage. The facts of the case cannot be a hallucination surface. If the bundle is malformed or a required field is absent, that is an INSUFFICIENT at stage one, not something to paper over with inference.
Select policy. The clinician chooses which CMS coverage policy applies. This is a deliberate scoping of authority: the human who owns the clinical judgment picks the rulebook. Automating policy retrieval is a ranking problem I explicitly punted on in seven hours, and I would want it evaluated separately before trusting it.
Decompose. The first model stage. The policy’s prose gets broken into atomic, individually checkable criteria, four or five per policy in my demo set. Atomicity is the point. The unit of judgment has to be small enough that a citation can settle it.
Review. For each criterion, a Claude Agent SDK reviewer judges whether the patient’s context satisfies it, and it must ground that judgment in citations pulled through tools from the parsed record. An uncited judgment is not a judgment. It is a claim, and claims do not count.
Argue. Contested criteria get an adversarial pass before anything is final. Borderline calls are exactly where a single forward pass is least trustworthy, so they are the only place that earns extra compute.
Arbiter. Pure code. It aggregates the per-criterion outcomes under fixed rules and emits the verdict. Unresolved or uncited criteria push toward INSUFFICIENT, never toward a guess. The design principle underneath the whole pipeline: the model never gets the last word.
One more piece matters as much as the verdict. Every stage streams TraceEvents over SSE to the frontend, so a reviewer watches the reasoning accumulate criterion by criterion. In this domain the justification is the product. The verdict is just its summary.
The eval problem is the real problem
The repo ships with an eval harness that runs gold-labeled cases: synthetic Synthea patients against real CMS policy structure. And the harness is where this stops being a hackathon project and becomes an engineering program, because accuracy is the wrong single metric for a fail-closed system.
Three numbers matter, in order:
False approvals. Must be zero. This is the invariant, not a target.
False denials. The cost of being wrong in the safe direction. It should fall over time, and every fall must be visible.
The INSUFFICIENT rate. Here is the uncomfortable truth about fail-closed design: a system that answers INSUFFICIENT on every case is perfectly safe and perfectly useless. The entire engineering problem is driving the INSUFFICIENT rate down while holding false approvals at zero. That is coverage at fixed precision, and it means no future change to prompts, decomposition, or tools gets judged by how the demo feels. It gets judged by the harness.
I care about this framing because it generalizes. Most agent systems in serious domains will end up shaped like this: a safety invariant you refuse to trade, and a usefulness metric you grind on underneath it.
What is wrong with it
Seven hours produces a position, not a product. Three known weaknesses, in order of how much they bother me:
Decomposition is itself a model stage, which means my “checkable criteria” inherit its mistakes. If the decomposer drops a criterion, the downstream machinery is rigorous about the wrong checklist. The next eval target is decomposition fidelity against clinician-written gold criteria, and until that exists, the pipeline’s rigor is partly cosmetic.
The deterministic parser is clean on Synthea and would be brittle on real-world FHIR, which is famously messy. At some point real data forces a choice: keep expanding the deterministic parser, or allow schema-constrained model extraction behind hard validators. I lean toward the second, still fail-closed, and I am aware that weakens the purity of “no LLM in the parser.” Purity was never the goal. Auditability was.
The argument layer is thin. It runs, but seven hours bought me one adversarial pass, not a calibrated debate protocol, and I have no eval yet that proves the arguing improves outcomes rather than just spending tokens dramatically.
What Saturday actually proved
Something narrow and useful: a fail-closed, citation-grounded authorization agent can be stood up in hours on today’s tooling. Making it trustworthy is not an hours problem. It is a months-long evals problem, criterion by criterion, and that is the work I want to be doing.
The code is public: greenlight on GitHub. If you work on prior auth, or on any system where a wrong answer costs more than no answer, read the arbiter and tell me where it breaks. INSUFFICIENT is an acceptable verdict. I would rather hear it than nothing.