A note on framing. This is an unfinished hackathon build, not a completed or validated tool. The problem framing draws on my PhD fieldwork and published literature rather than new interviews for this project, and the study plan below hasn't been run yet.
The problem
People with aphasia often make up for word-finding difficulty with co-speech gesture: miming an action, pointing to something, or using a conventional sign when a word won't come. Standard speech-to-text tools discard all of this. Phonetic paraphasias and silences register as noise, and the gestural half of the message is invisible.
Speech-language therapists annotate this by hand, tier by tier, in software like ELAN. In my fieldwork, and in the aphasia research literature, this is very slow compared with the length of the footage. It limits how much longitudinal patient data clinicians and researchers can realistically review.
Design approach
The core design decision was trust. A tool that silently guesses would be hard for clinicians to rely on, especially when a single gesture could carry the patient's whole intent. So the agent does the first pass but treats ambiguity as a conversation: it flags the moment, asks a specific question, and records the clinician's answer for next time.
Planned validation
| Aspect | Plan |
|---|---|
| Design | Within-subject comparison. 6–8 speech-language therapists and clinical linguists with ELAN experience annotate matched clips manually and with the agent, in counterbalanced order. |
| Recruitment | University speech-language pathology programmes and clinical linguistics research groups. |
| Measures | Time per minute of footage; agreement with two independent expert annotations (Cohen's kappa); System Usability Scale; think-aloud and interviews on trust in the question loop. |
| Analysis | Wilcoxon signed-rank tests on time and SUS scores; thematic coding of interviews for trust, perceived accuracy and workflow fit. |
Where it stands
Built for the hackathon: the multimodal parsing pipeline with McNeill classification, the clarifying-question interaction design, the adaptive lexicon data model, and export scaffolding for ELAN, CSV and SOAP notes.
Not done yet: any testing with clinicians, accuracy benchmarking against expert annotation, validation of the problem framing beyond fieldwork and literature, and a production-quality, accessible interface.