Experiment · Clinical AI · Roman Urdu
Telling a local AI to write the injured side made it write the wrong side
The injured side went missing from almost every Roman Urdu note. I tried three simple fixes. Each one brought the side back, and each one also brought back the wrong side.
By Muzammal Hussain · 1 November 2026 · 5 min read
In my 20-consultation benchmark, two small local models wrote physiotherapy notes from Roman Urdu consultations and almost never said which side was injured: "daayan" and "baayan" were simply dropped, in 13 of 14 llama3.1:8b notes and all 14 qwen2.5:3b notes. An earlier test had already shown these models mixing up left and right in Roman Urdu. So the obvious fix is to tell the model to write the side. I tested whether that works.
Key findings
- Every fix made the side appear more often. llama3.1:8b named the correct side in 10 of 14 notes with a rule and a glossary, up from 1 (exact McNemar p = 0.0039).
- Every fix also made the wrong side appear more often. The same llama3.1:8b setting named the other side in 4 notes, up from 2. One retry gave 7 correct and 6 wrong.
- For the smaller model the glossary made things worse: qwen2.5:3b named the other side in 9 notes and the correct side in 6, and added a side to 3 consultations that had none.
- A check that only asks "is a side written?" would pass these notes. Notes with no side fell from 11–15 to 0–2, while wrong sides rose.
What I did
I used the same 20 fictional consultations, prompt and checks as the benchmark, with only the Roman Urdu transcripts. 14 of them have an injured side. The models were qwen2.5:3b and llama3.1:8b, running locally with Ollama on a MacBook M4 at temperature 0. I compared four conditions:
- No rule: the notes from the benchmark, reused.
- Rule: one sentence added to the prompt, asking the model to write the side next to the body part.
- Rule and glossary: the rule plus one line: daayan, daayen, dayan mean right; baayan, baayen, bayan mean left.
- One retry: if the transcript has a side word and the note has none, the model is asked once to rewrite the note with the side.
For each note I counted whether the correct side was named, whether the opposite side was named, whether a side appeared that the consultation never mentioned, and how many listed facts were missing.
Results
| Model | Condition | Correct side (of 14) | Other side (of 14) | Facts missing (of 168) | Critical missing |
|---|---|---|---|---|---|
| qwen2.5:3b | No rule | 0 | 0 | 50 | 26 |
| qwen2.5:3b | Rule | 8 | 4 | 54 | 28 |
| qwen2.5:3b | Rule and glossary | 6 | 9 | 47 | 24 |
| qwen2.5:3b | One retry | 6 | 8 | 49 | 25 |
| llama3.1:8b | No rule | 1 | 2 | 16 | 8 |
| llama3.1:8b | Rule | 5 | 3 | 18 | 9 |
| llama3.1:8b | Rule and glossary | 10 | 4 | 17 | 7 |
| llama3.1:8b | One retry | 7 | 6 | 15 | 7 |
The fixes did not change how many other facts were lost. Two qwen2.5:3b notes with the plain rule kept repeating until they reached the 1,024-token limit I set.
What the wrong notes said
| Case | Model and condition | Consultation | Note |
|---|---|---|---|
| C3 | llama3.1:8b, rule | left shoulder ("baayan") | "pain in the right shoulder (baayan ka...)" |
| C19 | llama3.1:8b, retry | right leg ("daayen") | "the left leg (daayen pair tak)" |
| C14 | qwen2.5:3b, retry | left knee replacement | "right hip replacement" |
| C6 | llama3.1:8b, retry | left hip osteoarthritis | "osteoarthritis of the right lower back" |
In C3 and C19 the model copied the Roman Urdu side word into the note and still wrote the opposite side next to it.
Why a wrong side is worse than a missing one
When the side is blank, a physiotherapist can see the gap and ask the patient again. When the note says "right shoulder" for a left shoulder, it looks complete and can be trusted, and the wrong side may be examined or treated. A simple completeness check makes this worse: it rewards the model for writing any side.
The glossary helped the larger model and hurt the smaller one. My guess is that qwen2.5:3b does not map the words reliably and the rule pushes it to write some side anyway, but one run cannot show why.
For MeetPhysio, the clinical documentation system I am building, the lesson is that the side should not be left to the model. Code can read the side words in the transcript and compare them with the note, and send any mismatch to the physiotherapist. The check has to compare sides rather than demand one: in one consultation about neck pain the patient says the pain spreads to the right shoulder, but there is no injured side.
Limitations
Twenty consultations I wrote, 14 with a side, two models and one run at temperature 0, so a difference of a few notes may be chance. The checks use keywords, and I read only the notes where the side looked wrong. The new notes had a 1,024-token limit that the reused notes did not. The glossary covers only the spellings in my transcripts; real speakers also say "seedha" and "ulta", and speech-to-text may spell the words differently.
What I will test next
- The code side check: how many wrong-side notes it catches, and how many correct notes it sends for review.
- Sentence-by-sentence labels for every note where the side changed.
- Whisper transcripts in place of clean text.
Follows: Asgari E., Montaña-Brown N., Dubois M., Khalil S., Balloch J., Au Yeung J., Pimenta D. (2025). A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine 8, 274. doi:10.1038/s41746-025-01670-7