The problem
In everyday conversation in East Java, people move between Indonesian and Javanese without marking the switch, often within a single sentence. The two languages share so much vocabulary that many words can't be assigned to either one. Speech models like Whisper assume the opposite: one language per clip, written in a formal register.
The challenge scores word error rate (WER), the share of reference words a transcript gets wrong, and it's strict. A colloquial spelling such as gak instead of nggak, or 2019 instead of dua ribu sembilan belas, counts as a full error. So a good system has to understand the speech and also reproduce the transcribers' conventions.
Reading the errors first
The starter baseline (faster-whisper small, automatic language detection, voice-activity filtering on) scored 79.2% on the development set: 372 clips, about 2.5 hours, 4 speakers and 17,473 reference words. Three distinct failures accounted for most of the error.
The model was guessing the wrong language. 132 of 372 transcripts contained non-Latin scripts, including Cyrillic, Chinese, Korean and Hebrew, alongside stray Portuguese and German words. Whisper detects one language per clip, and on code-switched speech it often picked a language nobody was speaking.
A quarter of the speech was missing. Deletions alone were 26.4% of reference words. 75 transcripts were less than half the length of their references, and 4 were empty. The voice-activity filter was discarding quiet speech, and clips longer than Whisper's 30-second window were losing their endings.
The transcribers and the model wrote differently. The reference transcripts contain no digits at all, but the model's output had 228 digit characters. Every number written as numerals was guaranteed to be wrong.
What I changed
Each change targets one finding, and none of them changes the model itself.
- Forced Indonesian decoding instead of per-clip language detection.
- Turned off the voice-activity filter and stopped the decoder from conditioning on its own earlier output, so one bad window can't derail a long clip. Added fallbacks against repetition loops.
- Gave the model a short prompt written in the transcribers' colloquial style, nudging spellings such as nggak, gitu and nek.
- Post-processed the output: Indonesian number spelling (including ordinals, decimals and percentages), removal of non-Latin tokens, and collapsing runaway repetitions.
Results
| Run | Model | Dev WER | Substitutions | Deletions | Insertions |
|---|---|---|---|---|---|
| Starter baseline | faster-whisper small | 79.2% | 48.5% | 26.4% | 4.3% |
| After diagnosis | faster-whisper small | 35.4% | In progress | In progress | In progress |
Lower is better. The baseline was scored with my reimplementation of the documented normaliser, and the second run with the official scoring script.
The same small model went from 79.2% to 35.4%. That changed my plan. I had expected to need a larger model or fine-tuning first, but most of the baseline's errors came from how the model was being used, not from what it knew.
What I'm careful not to claim
All four changes went in together, so I can't yet say how much each one contributed; isolating them is the next experiment. The development set has only four speakers in two conversations, which makes the number a sanity check rather than a precise estimate. And forcing Indonesian decoding may be quietly pushing Javanese words toward Indonesian spellings. A per-language error breakdown should show whether that's happening.
Where the human decisions are
This project kept showing me how much of "accuracy" is set by people rather than models. Whether a word is Javanese or Indonesian, whether a hesitation is written e... or left out, whether a number is spelled out: these are transcriber conventions, and the metric treats departures from them exactly like mishearing. A model can be penalised for a spelling that native speakers would consider correct.
For underrepresented language varieties, that matters. It suggests evaluation should involve speakers of the variety in deciding what counts as an error, and that convention-aware scoring alongside raw WER would give a fairer picture of whether a system serves the people it transcribes.
Next steps
- Ablate each decoding change and keep only those that help.
- Move to a larger Whisper model (large-v3 or large-v3-turbo) within the challenge's size and two-hour runtime limits.
- Fine-tune on the Jember Javanese Spontaneous Speech Corpus, about 10 hours from the same dialect family, so the model learns the conventions instead of being prompted toward them.
- Validate in the official container through the platform's tests.