Structured interpretation of EEG reports with language models
Status: public medRxiv version 2; minor revision invited by the IEEE Journal of Biomedical and Health Informatics. This is a preprint, not an accepted journal publication, and it has not been certified by peer review.
Clinical evidence is often preserved in narrative reports that are useful to specialists but difficult to analyze at scale. This project studies how locally run language models can convert free-text EEG reports into structured research annotations while retaining the uncertainty expressed by the original reader.
Abstract at a glance
- Objective. Extract structured diagnostic labels and confidence levels from unstructured, free-text EEG reports.
- Method. Compare grammar-constrained local language models with expert annotations and lower-resource NLP baselines.
- Result. The studied pipelines produced high agreement on core report classifications, with a remaining challenge in representing nuanced certainty.
- Pathway. The work provides a principled route toward auditable annotation support by preserving reader-expressed uncertainty, benchmarking against expert annotations, and keeping outputs auditable; prospective validation and clinical governance are the next gates.
One label, three views of agreement
The schema encodes five judgments—overall abnormality and four specific finding categories—on one four-level ordinal scale. Each label pairs a presence-or-absence decision with an expressed certainty tier.
That single label supports three nested readings:
- Core Agreement asks whether the compared labels agree on absence (levels 1–2) versus presence (levels 3–4).
- Certainty-Adjusted Agreement requires the same exact level from 1 to 4.
- Conditional Certainty Agreement asks, among decisions that already match on absence or presence, what proportion also match on the certainty tier. It is an agreement proportion, not a calibrated probability.
Exact four-level agreement = Core Agreement x Conditional Certainty Agreement.
Thesis-to-publication pathway
The study originated in Wanying Tian’s Computing Science MSc research and grew through interdisciplinary clinical and computing collaboration, a reproducible analysis pipeline, and a major journal revision.
Collaboration map
Read medRxiv version 2 · Cite the DOI · Steven Bergner’s ORCID
Publication status and versioning should be read from the linked medRxiv record. If the manuscript is accepted and published, this page will be updated to point to the journal version of record.