Strategy Analysis
How each serialisation format affects model performance
Strategy A — Raw FHIR JSON: Raw FHIR R4 JSON bundle passed directly to the model. Maximally verbose, includes all fields.
Mean F1
0.655
valid runs only
Median F1
0.941
50th percentile
F1 p25–p75
0.00–1.00
interquartile range
Parse Failures
242
of 1000 total runs
F1 Distribution
All models, all patients
Per-Model Breakdown
| Model | F1 | Precision | Recall | Avg time | Failures |
|---|---|---|---|---|---|
| Llama 3.3 70B | 0.996 | 1.000 | 0.993 | 81.1s | 0 |
| Llama 3.1 8B | 0.918 | 0.963 | 0.905 | 7.2s | 0 |
| Mistral 7B v0.3 | 0.725 | 0.889 | 0.668 | 13.7s | 11 |
| Phi-3.5-mini (3.8B) | 0.636 | 0.730 | 0.593 | 14.5s | 33 |
| BioMistral 7B | 0.000 | 0.000 | 0.000 | 8.4s | 198 |
Mean Inference Time — All Models × Strategies
Seconds per patient run
Patient-level Results
All patients · click any row to open drill-down
200 patients
| Patient ↕ | Meds ↕ | Mean F1 ↑ | Phi-3.5 | Mistral | BioMistral | Llama 8B | Llama 70B |
|---|---|---|---|---|---|---|---|
| c5f21693…Lou594 Langosh790 | 4 | 0.330 | 0.000 | 0.250 | fail | 0.400 | 1.000 |
| 41b6d605…Sharan181 Kulas532 | 4 | 0.331 | 0.000 | 0.400 | fail | 0.400 | 0.857 |
| f2334b9a…Lauralee67 Murphy561 | 8 | 0.371 | fail | 0.000 | fail | 0.857 | 1.000 |
| bb88b4a2…Christal240 Schneider199 | 2 | 0.380 | 0.400 | 0.000 | fail | 0.500 | 1.000 |
| 6c6acf0d…Jarrod316 Wiza601 | 12 | 0.400 | fail | fail | fail | 1.000 | 1.000 |
| 7aa87a8d…Miguel815 Hegmann834 | 5 | 0.444 | fail | 0.333 | fail | 0.889 | 1.000 |
| 04e8a125…Rolando809 Nicolas769 | 7 | 0.450 | fail | 0.250 | fail | 1.000 | 1.000 |
| 5d93cfec…Wan724 Cormier289 | 4 | 0.451 | fail | 0.400 | fail | 0.857 | 1.000 |
| 0754f573…Hassan290 Parisian75 | 1 | 0.457 | 1.000 | fail | fail | 0.286 | 1.000 |
| 7ed81b56…Jerry654 Jones311 | 7 | 0.458 | fail | 0.727 | fail | 0.727 | 0.833 |
| 1e18e750…Dalila583 Hartmann983 | 8 | 0.465 | 0.400 | 0.222 | fail | 0.933 | 0.769 |
| 7e4a103b…Basil991 Brekke496 | 5 | 0.467 | fail | 0.333 | fail | 1.000 | 1.000 |
| 458050db…Clark193 Parisian75 | 1 | 0.480 | fail | 1.000 | fail | 0.400 | 1.000 |
| 8f51273e…Rocio28 Valdés907 | 4 | 0.480 | fail | 0.400 | fail | 1.000 | 1.000 |
| 9aac1ba6…Lou594 Conroy74 | 8 | 0.480 | 1.000 | 0.400 | fail | 0.000 | 1.000 |
| baff5abc…Evan94 Keeling57 | 7 | 0.499 | 0.769 | 0.000 | fail | 0.727 | 1.000 |
| 24f7ebda…Dante562 Welch179 | 1 | 0.500 | 0.000 | 1.000 | fail | 0.500 | 1.000 |
| 4346ccbc…Sage164 Lubowitz58 | 6 | 0.500 | 0.500 | 0.000 | fail | 1.000 | 1.000 |
| 8231b5d9…Horacio317 Ocampo729 | 3 | 0.500 | 0.500 | 0.500 | fail | 0.500 | 1.000 |
| ade0959f…Marge692 Paucek755 | 3 | 0.500 | fail | 0.500 | fail | 1.000 | 1.000 |
| ab618d09…Germán350 Dávila115 | 6 | 0.501 | fail | 0.800 | fail | 0.706 | 1.000 |
| 463c9aa2…Tasia358 Yundt842 | 5 | 0.514 | 0.571 | fail | fail | 1.000 | 1.000 |
| 8ca841f6…Coralee911 Bayer639 | 5 | 0.514 | 0.571 | fail | fail | 1.000 | 1.000 |
| ef09508c…Cedrick207 Ryan260 | 7 | 0.518 | 0.667 | fail | fail | 0.923 | 1.000 |
| 2e2b7e2f…Courtney281 Nolan344 | 6 | 0.520 | 0.600 | fail | fail | 1.000 | 1.000 |
| 7ed9b2f9…Von197 Strosin214 | 6 | 0.520 | fail | 0.800 | fail | 0.800 | 1.000 |
| f11e7ecb…Tisha655 Schulist381 | 16 | 0.523 | 0.897 | fail | fail | 0.720 | 1.000 |
| 51b90e2e…Ellie521 Nader710 | 1 | 0.533 | fail | 1.000 | fail | 0.667 | 1.000 |
| 522f53ae…Hollie21 Padberg411 | 2 | 0.533 | fail | 0.667 | fail | 1.000 | 1.000 |
| 591a451d…Doug189 Prosacco716 | 2 | 0.533 | 0.667 | fail | fail | 1.000 | 1.000 |
| 608002f4…Tisha655 Hilll811 | 5 | 0.533 | 0.667 | 0.000 | fail | 1.000 | 1.000 |
| 7092d859…Jamika47 Schmidt332 | 2 | 0.533 | fail | 0.667 | fail | 1.000 | 1.000 |
| 9018e566…Kirsten270 Jenkins714 | 2 | 0.533 | 0.000 | 0.667 | fail | 1.000 | 1.000 |
| bbeebbe2…Samuel331 Villanueva621 | 1 | 0.533 | fail | 1.000 | fail | 0.667 | 1.000 |
| e2fcc788…Haydee844 Treutel973 | 2 | 0.533 | 0.000 | 0.667 | fail | 1.000 | 1.000 |
| 1b3c405a…Burma963 Goyette777 | 6 | 0.542 | fail | 0.800 | fail | 0.909 | 1.000 |
| f360fbf6…Trenton748 Nitzsche158 | 6 | 0.542 | fail | 0.800 | fail | 0.909 | 1.000 |
| ed17470c…Curt84 Satterfield305 | 6 | 0.545 | 0.727 | fail | fail | 1.000 | 1.000 |
| 39cadc39…Darwin703 Conn188 | 6 | 0.545 | 0.667 | 0.667 | fail | 0.727 | 0.667 |
| d7837d3b…Xiao631 Schultz619 | 4 | 0.547 | 0.667 | 0.400 | fail | 0.667 | 1.000 |
| e5181fea…Rodrick370 Reichert620 | 3 | 0.551 | 0.400 | 0.500 | fail | 0.857 | 1.000 |
| 80e670cf…Cedric746 Hessel84 | 8 | 0.554 | 0.333 | 0.667 | fail | 0.769 | 1.000 |
| 0f218e56…Garret233 Durgan499 | 7 | 0.554 | 0.727 | 0.444 | fail | 0.600 | 1.000 |
| 88223d62…Katheleen349 Gutkowski940 | 6 | 0.560 | fail | 0.800 | fail | 1.000 | 1.000 |
| dd0a3ee7…Pasquale620 Barrows492 | 3 | 0.560 | 1.000 | 0.000 | fail | 0.800 | 1.000 |
| f2e5eb1e…Ying817 Dickinson688 | 3 | 0.560 | 0.800 | 1.000 | fail | 0.000 | 1.000 |
| 7021b2b7…Daniel959 Mohr916 | 6 | 0.564 | 0.000 | 0.909 | fail | 0.909 | 1.000 |
| adf124e5…Austin578 Hackett68 | 9 | 0.569 | 0.706 | 0.200 | fail | 0.941 | 1.000 |
| e304f79d…Milford485 Gaylord332 | 7 | 0.571 | 0.769 | 0.250 | fail | 0.833 | 1.000 |
| 703fc921…Leopoldo762 Kovacek682 | 8 | 0.572 | 0.545 | 0.545 | fail | 0.769 | 1.000 |
Page 1 of 4