Status: public medRxiv version 2; minor revision invited by the IEEE Journal of Biomedical and Health Informatics. This is a preprint, not an accepted journal publication, and it has not been certified by peer review.

Clinical evidence is often preserved in narrative reports that are useful to specialists but difficult to analyze at scale. This project studies how locally run language models can convert free-text EEG reports into structured research annotations while retaining the uncertainty expressed by the original reader.

Abstract at a glance

  • Objective. Extract structured diagnostic labels and confidence levels from unstructured, free-text EEG reports.
  • Method. Compare grammar-constrained local language models with expert annotations and lower-resource NLP baselines.
  • Result. The studied pipelines produced high agreement on core report classifications, with a remaining challenge in representing nuanced certainty.
  • Pathway. The work provides a principled route toward auditable annotation support by preserving reader-expressed uncertainty, benchmarking against expert annotations, and keeping outputs auditable; prospective validation and clinical governance are the next gates.

One label, three views of agreement

The schema encodes five judgments—overall abnormality and four specific finding categories—on one four-level ordinal scale. Each label pairs a presence-or-absence decision with an expressed certainty tier.

1confident absence
2low-confidence absence
3low-confidence presence
4confident presence

That single label supports three nested readings:

  • Core Agreement asks whether the compared labels agree on absence (levels 1–2) versus presence (levels 3–4).
  • Certainty-Adjusted Agreement requires the same exact level from 1 to 4.
  • Conditional Certainty Agreement asks, among decisions that already match on absence or presence, what proportion also match on the certainty tier. It is an agreement proportion, not a calibrated probability.

Exact four-level agreement = Core Agreement x Conditional Certainty Agreement.

Diagram separating prompt development, local EEG report processing, and agreement evaluation
The paper separates development, local report processing, and evaluation. Reference labels re-enter only after inference, and grammars constrain machine-readable structure rather than clinical correctness.

Thesis-to-publication pathway

The study originated in Wanying Tian’s Computing Science MSc research and grew through interdisciplinary clinical and computing collaboration, a reproducible analysis pipeline, and a major journal revision.

Collaboration map

Wanying Tian equal contributorSFU Computing Science · Biomedical Physiology and Kinesiology
Steven Bergner equal contributorSFU Computing Science
Alexander MoiseevSFU Biomedical Physiology and Kinesiology
Fred PopowichSFU Computing Science
George MedvedevRoyal Columbian Hospital · Fraser Health
Mark P. RichardsonKing's College London · King's College Hospital
Roman RodionovUCL Queen Square · UCLH
Pengcheng XiNational Research Council Canada
Sam M. DoesburgSFU Biomedical Physiology and Kinesiology
Urs RibaryBC Children's Hospital Research Institute
Joel S. WinstonKing's College London · King's College Hospital
Vasily A. Vakorin corresponding authorSFU Biomedical Physiology and Kinesiology · Royal Columbian Hospital · Fraser Health

Read medRxiv version 2 · Cite the DOI · Steven Bergner’s ORCID

Publication status and versioning should be read from the linked medRxiv record. If the manuscript is accepted and published, this page will be updated to point to the journal version of record.