Hackathon Project · Gemini Multimodal Agent · 2026

An Agent That Assists Research Annotation of Aphasic Gesture & Speech

Personal / Hackathon In Progress UX Multimodal AI Inclusive Design React · Gemini API

An autonomous agent concept, prototyped for a Gemini hackathon, that assists Speech-Language Pathologists in annotating co-speech gesture and aphasic speech from video — work directly informed by patterns I observed during my PhD fieldwork on inclusive data visualisation for people with aphasia, and by published literature on communication compensation strategies.

A note on framing: this is an unfinished hackathon build, not a completed or validated tool. The problem framing draws on my own PhD fieldwork and published literature rather than new interviews conducted for this project. The persona and validation plan below are explicitly labelled as such — provisional and not yet tested with real clinicians.

Problem

Manual Annotation Is a Bottleneck for Aphasia Research and Care

People with aphasia often compensate for word-finding difficulty through co-speech gesture — pantomiming an action, pointing to reference something spatially or temporally, or using conventionalised signs when a word won't come. Standard speech-to-text tools discard this entirely: phonetic paraphasias and anomic silences just register as noise, and the gestural half of the message is invisible to automated transcription altogether.

Speech-Language Therapists currently annotate this manually, tier by tier, in software like ELAN. From what I observed during my own PhD fieldwork with aphasia research participants, and consistent with what's documented in aphasia research literature, this manual process is extremely slow relative to the length of footage being coded — a real bottleneck that limits how much longitudinal patient data clinicians and researchers can realistically review.

Artifact 01 — Problem Statement

Problem Statement

This problem occurs in my current research context. I want to build an app that enable gesture and speech annotation from a video recording for people with aphasia. Currently this process is done by hand and normal speech to text transcription could not handle people with aphasia's speech and video needed to be annotated for meaningful gesture that convey intent that are not captured by speech.

Artifact 02 — System Design

Design Approach

Rather than a silent auto-labeller, I designed this agent around a collaborative-partner model: it does the first pass of multimodal annotation, but flags ambiguity and asks the clinician rather than guessing — directly responding to the trust concern above.

Parse & Classify

Tracks hand kinematics and classifies gestures against the formal McNeill taxonomy (Iconic, Deictic, Metaphoric, Emblem, Beat), aligned to phonetic speech tokens.

Flag & Ask

When the agent detects communicative ambiguity, it generates a proactive clarifying question for the clinician instead of guessing silently.

Adapt

Every clinician resolution is committed to a Personalized Adaptive Lexicon, so the agent learns each patient's unique non-verbal vocabulary across sessions.

Export

Outputs standard EUDICO ELAN (.eaf) XML, CSV timelines, and SOAP-format clinical notes — fitting directly into existing clinical workflows.

Artifact 03 — Study Plan (Not Yet Conducted)

Planned Validation

The hackathon build itself hasn't been clinically tested. Here's the study I'd run next to find out whether it actually helps — the deliverable is the plan, and I want to be upfront that none of this has been executed yet.

Design

Within-subject comparative study: 6–8 SLTs and clinical linguists with existing ELAN experience each annotate matched video clips using (a) their standard manual ELAN workflow and (b) an Aphasia Annotate-assisted workflow, order counterbalanced.

Planned

Recruitment

Recruit through university speech-language pathology programs and clinical linguistics research groups — the same channels used for the BBC R&D study, adapted to a clinical population.

Planned

Measures

Time-on-task per minute of footage; classification accuracy against two independent expert-only ELAN annotations (Cohen's kappa); System Usability Scale; think-aloud + semi-structured interview on trust in the clarifying-question loop.

Planned

Analysis

Paired comparison (Wilcoxon signed-rank) on time-on-task and SUS scores between conditions; thematic coding of interview transcripts for trust, perceived accuracy, and workflow fit.

Planned

Status

Where This Stands

Honest snapshot of what's actually built versus what's still ahead.

Built for the hackathon

Multimodal parsing pipeline classifying gesture against the McNeill taxonomy
Clarifying-question interaction loop design
Adaptive Lexicon memory concept and data model
ELAN (.eaf) / CSV / SOAP note export scaffolding

Not yet done

Any testing or validation with real SLPs or clinicians
Accuracy benchmarking against expert gold-standard annotation
Problem framing validated beyond literature + fieldwork
Production-grade UI polish and accessibility pass