Autonomous Coding You Can Verify
An auditable accuracy benchmark for autonomous E/M coding.
on real, paid charts
finalized autonomously
per chart
to a certification artifact
Executive summary
- Two numbers, not one: how accurate the coding is — 98.1% — and how much of it the software finalizes on its own — 92.9% of condition & symptom codes, with accuracy on the automated portion held at or above 95% (definitions in §3).
- Measured against the codes a real practice billed and payers paid — 945 charts, every headline figure certified on a 376-chart held-out test set kept out of all development work. Where the billed record itself is imperfect, the disagreement still counts as our error, pending the pre-registered independent audit.
For citation and print
A PDF of this report, typeset for the page. Free, no email required.
1Every figure in this report is verifiable
Every autonomous-coding vendor advertises a big number. In a July 2026 review of publicly available vendor materials, we found none that publishes the evidence behind it: a held-out test set, confidence intervals, calibration, the accuracy–coverage curve. This report publishes all four, for every figure.
| How the field reports accuracy | Headline number | What is actually disclosed |
|---|---|---|
| Advertised autonomous-coding claims (2024–26)† | “95–99%+ accurate · 85–96% automated” | Self-reported · no held-out test set · no intervals · no calibration · no error direction |
| General-purpose LLMs, peer-reviewed (Soroush et al., NEJM AI 202410) | 33.9% exact ICD-10-CM · 49.8% CPT | Full method |
| Domain-fine-tuned LLM13 | 69.2% exact-match | Peer-reviewed, full method |
| RAG-augmented LLMs vs ED coders14 | 17.6–26.4% exact-match | Preprint, full method |
| Purpose-built neural coders7,8,9 | macro-F1 0.09–0.10 on the full code set (micro-F1 ≈ 0.5–0.6); zero recall on the rare half of ICD-10 codes9 | Public benchmark, full method |
| Human coders, re-abstraction studies11,12 | 82–87% inter-coder agreement | Same charts, two certified coders (ICD-10 variants) |
| CodeSight™ (this report) | The full accuracy–coverage curve | Held-out test set · CIs · calibration · billed ground truth · error direction |
The number that matters is the one shown with its coverage, on a held-out test set, against real billed charts.
† Drawn from publicly available vendor marketing materials reviewed July 2026: self-reported figures, against vendor-defined denominators. No vendor is identified and no head-to-head comparison is made. It describes what advertised claims typically disclose, not the practices of any particular company. Apply it to any vendor you are evaluating, including us.
We are not asking you to take our number on faith. We are asking you to check it — and to ask every other vendor for the same disclosures.
2The system: a calibrated ensemble
CodeSight™ is an ensemble of specialized models — each owning one decision — behind a calibrated confidence gate:
Fig. 1 — The CodeSight™ ensemble. Specialized models author the codes; an independent reviewer flags suspected errors without authority to edit; deterministic checks apply written rules, not model judgment (NCCI/MUE/bundling — scheduled; the practice-policy E/M check in Methods — certified).
The benchmark. 945 billed office-visit (E/M) encounters from a de-identified outpatient specialty practice — every runnable E/M chart the corpus contains, not a curated subset. Ground truth: the codes the practice billed and payers paid — a record outside our control.
| Methods at a glance | |
|---|---|
| Corpus | 945 complete SOAP-structured office notes (one empty note excluded); born-digital text in — no OCR confound |
| Ground truth | The codes billed and paid, scored per encounter — a record outside our control |
| Split | Random split locked before certification: development n = 569 · held-out test set n = 376, kept out of all development work |
| Calibration | Isotonic map fit on the development split only, applied unchanged to the test set |
| Deterministic E/M check | Where a note documents total visit time, the practice’s own written coding policy overrides the model’s proposed level — a written rule, not a model guess. Certified in two passes: 93.4% before the check, 98.1% with it; the models and prompts were identical in both passes |
| Model training | No fine-tuning and no custom training on this practice’s data — the models are off-the-shelf. The only practice-specific element is the written policy above |
| Case mix | Corpus over-weights low acuity vs the practice’s 28,112-encounter billing mix — case-mix-reweighted figures reported alongside raw; the highest-acuity bands are thin, so high acuity is not certified (§5) |
| Privacy | Operated under HIPAA safeguards and Business Associate Agreements; every figure in this report is aggregate and de-identified |
| Scheduled | Redaction-controlled arm masking codes that appear verbatim in a note’s Assessment/Plan — separates reasoning from transcription |
The gate. Every code carries a confidence score. Diagnosis confidence is recalibrated by isotonic regression5 — a standard technique that rescales stated confidence so a stated “90%” means about 90% in reality, because modern models are systematically overconfident.4 E/M confidence is used as-is: at 98% accuracy there is too little error left for recalibration to act on. Above the confidence bar, a code is finalized; below it, a certified human coder decides — shown the draft code and the reason it was flagged.6 The models were not fine-tuned — no custom training on this practice’s data; these are baseline numbers.
Safety by design. Autonomy is gated by guardrails: an independent reviewer model that can flag a code but never change it · a calibrated confidence gate that routes anything uncertain to a certified human coder · zero wrong-family codes across every chart tested · and published error direction — 4 over / 3 under on the held-out test set. Built for healthcare: operated under HIPAA safeguards — accountable, auditable, human-in-the-loop.
3Results: the curve, not a number
n = 945 charts · development 569 / held-out test 376 · every figure traces to a versioned certification artifact.
| Certified results — held-out test set (n = 376) | Value | 95% CI |
|---|---|---|
| E/M level — exact match with the billed code | 98.1% | 96.2–99.1 |
| E/M level — reweighted to the practice's full billing mix | 93.8% | — |
| Diagnosis codes — F1 | 0.885 | 0.870–0.899 |
| Codes finalized autonomously — condition & symptom | 92.9% (F1 0.906) | — |
| Codes finalized autonomously — all codes | 87.9% | — |
| Codes finalized autonomously — all codes, at a 97% accuracy bar | 73.9% | — |
| Codes finalized autonomously — all codes, at a 99% accuracy bar | 26.9% | — |
E/M is one code per encounter, scored as exact match, with Wilson intervals. Diagnosis is multi-label micro-F1 (precision 0.867 / recall 0.904) with bootstrap intervals. The 1,872 test-set codes are 376 E/M levels plus 1,496 diagnosis codes. Across all 945 charts, development split included: E/M 98.7%, diagnosis F1 0.878, all-codes coverage 82.3% / condition-&-symptom 88.3%. Ease the accuracy bar to ≥94% and 90% of codes clear the gate, at 94.2% measured accuracy on that slice.
Fig. 2 — How much we automate, and how accurate it stays. This chart plots the all-codes gate, not the headline condition-&-symptom operating point (92.9% at ≥95%), which is tabulated above. Gray: the measured single-gate sweep across all 945 charts (4,703 codes) — coverage opened from the VAR@95 point (82.3% at ≥95%) to 98% coverage (90.0% slice accuracy). Green diamond: the certified held-out VAR@95 operating point, labelled on the chart. All points are measured values from the certification artifacts; nothing here is extrapolated.
Verified Automation Rate (VAR@95). VAR@95 answers one question: how many codes can be finalized with no human touch while the finalized group stays at least 95% accurate? Both halves are measured on the held-out test set — the coverage is counted, the accuracy is scored.1,2,3 The same trade-off is reported at stricter bars as VAR@97 and VAR@99.
Two ways to count — and we publish both. The headline 92.9% scores condition & symptom codes: the codes that describe what is clinically wrong with the patient. Administrative status descriptors (ICD-10 “Z” codes — “history of,” “long-term medication use,” and similar) are left out of both sides of the comparison — prediction and billed truth alike. These codes carry real clinical and medical-necessity weight; they are excluded because whether and how consistently a practice reports them varies by internal policy, which makes them a measure of house style as much as of coding skill. That leaves 1,748 of the 1,872 test-set codes.
Counting every code, including those status descriptors, the same gate finalizes 87.9%. Neither figure replaces the other, both trace to the same certification artifacts, and we publish them together — because a coverage number without its denominator is exactly the kind of claim this report exists to argue against.
3aWhere the errors live
Across all 945 charts, CodeSight matched the billed E/M level on 933. The level disagreed on just 12 — and in 6 of those, the documentation supports our level over the billed one: under-billed revenue the practice left behind. On the held-out test set there were 7 disagreements: 4 one level above the billed code, 3 below, and 0 outside the code family (all 945: 7 over / 5 under / 0 wrong-family).
Ground truth is still what was billed. Every disagreement is scored as our error until the pre-registered independent CPC audit reports.
What that record means in practice
- Zero hallucinated codes. Not one, in 945 charts. No invented diagnosis. No code from the wrong family. The single failure mode that keeps autonomous coding out of most practices did not happen — not once.
- It walked in cold. No fine-tuning, no custom training, no learning period on this practice’s data — the models are off the shelf, and the record above is what they did on charts they had never seen. The only practice-specific input is the practice’s own written coding policy, applied as a fixed rule.
- We were evaluating software. It found revenue instead. On six charts the documentation supported a higher level than what actually went out the door — revenue already earned, written off, never noticed. Nobody was auditing for it.
3bWhy the gate can be trusted
The system only finalizes a code without review when its confidence clears a threshold. That is only meaningful if the confidence is honest — so we publish whether it is.
Fig. 3 — Reliability diagram, diagnosis confidence (held-out test set: 376 charts, 1,496 scored diagnosis codes in bins). Each dot is a confidence bin (area ∝ bin size); the diagonal is perfect calibration. After isotonic recalibration (fit on the development split, applied unchanged), expected calibration error is 0.037 on the test set (0.026 across all charts) — and the top bin, where the auto-finalize gate operates, runs slightly conservative: codes stated at 0.945 confidence were right 98.1% of the time.
These results predate model fine-tuning and the pre-registered frontier-model upgrade; the next certification run will report whether the curve climbs.
3cAccuracy by E/M level — including the thin ones
| E/M level | Test-set n | Exact accuracy | 95% CI |
|---|---|---|---|
| 99202 | 109 | 100.0% | 96.6–100.0 |
| 99203 | 33 | 100.0% | 89.6–100.0 |
| 99204 | 4 | 100.0% | 51.0–100.0 |
| 99211 | 30 | 96.7% | 83.3–99.4 |
| 99212 | 136 | 99.3% | 96.0–99.9 |
| 99213 | 47 | 95.7% | 85.8–98.8 |
| 99214 | 11 | 90.9% | 62.3–98.4 |
| 99215* | 6 | 66.7% | 30.0–90.3 |
We publish the thin rows too. 99204 (n = 4), 99214 (n = 11) and 99215 (n = 6) carry intervals too wide to certify, and 99205 does not occur in this corpus — high-acuity performance is explicitly not certified by this benchmark (§5). This is why the headline is also reported case-mix-reweighted (93.8%) against the practice’s 28,112-encounter billing distribution.
* Both 99215 misses are conservative under-calls: the model proposed 99214 rather than award the top level without support. At the highest-value level the system errs downward, never upward. At n = 6, two charts decide the row.
3dSpeed
10.9 seconds is one chart on one processing lane — capacity is simply lanes × ~300 charts/hour, and lanes scale horizontally, so enterprise volume is an infrastructure setting, not a staffing plan. (Manual production coding typically runs 20–30 charts per coder-hour.) These are retrospective batch timings, not a production SLA; one chart in 945 hit a cloud-infrastructure stall, disclosed in the run log and since hardened. Marginal compute cost per chart is a small fraction of manual coding.
4How to read these numbers
The human ceiling. Certified coders disagree with each other. In re-abstraction studies of ICD-10 diagnosis coding, two trained coders agree roughly 82–87% of the time at billing-level specificity — and only about 47% at full terminal-code specificity (κ ≈ 0.42).11,12 Those studies measure a different task than E/M leveling, so they are context, not a head-to-head comparison. What they establish is that any figure scored against one biller’s labels — including every number on this page — carries irreducible disagreement, and that some scored “misses” are defensible judgments or documentation the biller under-credited.
Settling which is which requires expert review. That is why we have pre-registered an independent CPC audit against this same corpus — the study that would let anyone, including us, say something rigorous about how autonomous coding compares with a human coder. It has not reported yet, and we will publish it when it does, whichever way it lands.
Our setting is deliberately different from the benchmarks in §1 — focused, specialty-scoped, calibrated, human-in-the-loop. A raw frontier model, however capable, is not a coding system: published exact-match rates for general-purpose LLMs sit near 34%.10 The distance from there to the numbers in this report is the system — specialization, domain calibration, and a calibrated reject option1,2,6 that automates only what it has earned, at a precision you can verify.
The disclosure checklist
If you are evaluating any autonomous coding vendor, these are the questions worth asking. We have filled in our own column; the third records what an advertised claim typically discloses.
| What a verifiable accuracy claim discloses | This report | Typical advertised claim |
|---|---|---|
| Sample size (charts and codes) | 945 charts · 4,703 codes | rarely stated |
| Held-out test set | n = 376, locked before certification | not stated |
| Ground-truth definition | codes billed and paid | vendor-defined or unstated |
| Confidence intervals on headline figures | Wilson (E/M) + bootstrap (diagnosis), shown above | absent |
| Calibration of the gating confidence | reliability diagram + ECE 0.037 — Fig. 3 | absent |
| Error direction (over- vs under-coding) | published: 4 over / 3 under | absent |
| Case-mix adjustment | reweighted to real billing mix | absent |
| The full accuracy–coverage curve | Fig. 2 — measured points only | absent |
5Limitations
- One practice, one specialty. A single outpatient specialty practice, one EMR, one billing team’s coding culture as ground truth. Generalization to other specialties is untested here; a procedure and ultrasound benchmark on roughly 12,000 notes from the same corpus is in progress as a separate track.
- High acuity is not certified. 99205 is absent from the corpus, and 99204 / 99214 / 99215 are too thin for tight intervals. The per-level table above shows the thin rows rather than hiding them.
- Scope is E/M office visits. No procedure codes, modifiers, laterality, or units are covered by this benchmark, and the deterministic NCCI / MUE compliance gate was scheduled but not part of this certification.
- Billed truth is imperfect. The ground truth is what one practice billed, not what a panel of auditors would ratify. That cuts in both directions, and the pre-registered independent audit is how we intend to settle it.
Bottom line
Do not buy an accuracy number. Buy the curve — and the evidence under it.
The certified answer: 98.1% matched the billed E/M level, and 92.9% of condition & symptom codes finalized autonomously at ≥95% accuracy on the automated slice — measured on a held-out test set of real billed-and-paid charts, with confidence intervals, calibration and error direction published, and every figure traceable to a versioned artifact.
No invented codes. No upcoding at the levels where it would pay to. And six charts of revenue the practice had already earned and never billed. Ask any vendor for the disclosure table above.
References
- C. K. Chow. “On Optimum Recognition Error and Reject Tradeoff.” IEEE Trans. Information Theory, 1970. doi:10.1109/TIT.1970.1054406
- Y. Geifman, R. El-Yaniv. “Selective Classification for Deep Neural Networks.” NeurIPS 2017. arXiv:1705.08500
- R. El-Yaniv, Y. Wiener. “On the Foundations of Noise-free Selective Classification.” JMLR 11, 2010. jmlr.org/papers/v11/el-yaniv10a
- C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger. “On Calibration of Modern Neural Networks.” ICML 2017. arXiv:1706.04599
- B. Zadrozny, C. Elkan. “Transforming Classifier Scores into Accurate Multiclass Probability Estimates” (isotonic calibration). KDD 2002. doi:10.1145/775047.775151
- H. Mozannar, D. Sontag. “Consistent Estimators for Learning to Defer to an Expert.” ICML 2020. arXiv:2006.01862
- J. Mullenbach, S. Wiegreffe, J. Duke, J. Sun, J. Eisenstein. “Explainable Prediction of Medical Codes from Clinical Text” (CAML). NAACL 2018. arXiv:1802.05695
- C.-W. Huang, S.-C. Tsai, Y.-N. Chen. “PLM-ICD: Automatic ICD Coding with Pretrained Language Models.” ClinicalNLP 2022. arXiv:2207.05289
- J. Edin et al. “Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study.” SIGIR 2023. doi:10.1145/3539618.3591918
- A. Soroush, B. S. Glicksberg, E. Zimlichman, et al. “Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying.” NEJM AI 1(5), 2024. doi:10.1056/AIdbp2300040
- J. Stausberg, N. Lehmann, D. Kaczmarek, M. Stein. “Reliability of diagnoses coding with ICD-10.” Int. J. Medical Informatics 77(1), 2008. PMID 17185030
- M. Peng et al. “Coding reliability and agreement of ICD-10 codes in emergency department data.” Int. J. Population Data Science, 2018. PMC7299481
- Hou et al. “Enhancing medical coding efficiency through domain-specific fine-tuned large language models.” npj Health Systems, 2025. doi:10.1038/s44401-025-00018-3
- E. Klang, I. Tessler, D. U. Apakama, et al. “Assessing Retrieval-Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to Human Coders.” medRxiv, 2024. doi:10.1101/2024.10.15.24315526
https://www.medmio.com/case-studies/codesight-accuracy-benchmark/
All figures derive from versioned certification artifacts (n = 945, held-out test set n = 376), available for inspection under NDA. This is a living benchmark: each scheduled certification run is published as a new dated release, and earlier releases remain available at their original URLs.
Media & research contact: press@medmio.com
The practice is not named, and identifying details have been generalized. Every figure on this page is aggregate and de-identified.