Industry Project · BBC R&D · Apr – Sep 2019

Usability Study & White Paper —
BBC R&D Interactive Application

Industry Usability Testing Data Analysis in R White Paper BBC R&D · London Apr – Sep 2019

An end-to-end usability study commissioned by BBC R&D for an interactive research application. I independently recruited participants, designed and facilitated sessions, performed quantitative data analysis in R, and authored a white paper that directly informed the development team's design decisions.

Framing

Research Questions

BBC R&D was prototyping an "enhanced" podcast player that layered images, transcripts, chapters and contributor links on top of standard audio. Before investing further in the concept, the team needed to understand whether and how these visual additions actually changed the listening experience in real, everyday conditions — not just in a lab. The research questions are as follows:

Valued Interactions

Which features of the enhanced player do listeners actually value, and why?

Environment Effects

How does the listening environment change the way people interact with the player?

Visual Layer Impact

Does adding a visual interface help or hinder the core podcast-listening experience?

Study Design

Method & Recruitment

I designed this as a remote field study rather than a lab test, so participants could use the prototype in the settings where podcast listening actually happens — a deliberate trade-off of experimental control for ecological validity. We tested three environments: home, work/study, and travel (public transport or walking). Driving was scoped out early, since it wouldn't let participants safely interact with the interface.

We targeted 18–35 year-olds, a demographic the BBC was specifically trying to grow among podcast users, and recruited via university and internal research-group mailing lists. Screening criteria required participants to already listen to at least one podcast weekly and to regularly commute by public transport or on foot, so everyone could meaningfully engage with all three test environments. To keep the content itself from skewing results, I selected three real 10-minute podcast clips spanning distinct genres — factual, entertainment, and current affairs — matching the motivations prior research identified for why people listen to podcasts at all.

Fig 1. Interactive Podcast Interface, taken from the white paper.

Artifact 01 — Study Plan

Protocol

Each session ran across five stages, moving from unmoderated first impressions through to a structured field deployment and a closing interview.

Pre-Interview & Survey

Briefed participants, took consent, and captured their existing podcast habits as a baseline before they touched the prototype.

Exploration

Five minutes of unguided, think-aloud exploration of the player — no training given — to capture genuine first reactions.

Training

Task-based walkthrough of every feature (images, transcripts, chapters, sharing, contributors), so later feedback reflected informed use, not missed functionality.

Remote Field Study

Participants listened to one podcast in each of the three environments over the following week, completing a short survey after each session.

Post-Study Interview

A semi-structured video interview covering preferred environment, feature usage, and whether they'd choose the enhanced player over their usual app.

Artifact 02 — Analysis

Analysis Approach

I combined qualitative and quantitative analysis so the findings would hold up on both fronts. Interview recordings were transcribed and thematically coded, then affinity-mapped to surface patterns across participants and environments. In parallel, I analysed the participants' feature and environment rankings quantitatively in R: a Friedman test to check for overall statistical significance, followed by Wilcoxon signed-rank tests with Bonferroni correction to pinpoint exactly where the differences occurred.

Artifact 03 — Insights

Key Findings

12 of 13 recruited participants (aged 18–35, split roughly two-thirds/one-third across the age band, urban students and young professionals) completed the full study.

Home ranked highest overall, but for habit — not focus

Participants listened at home most often simply because it's their default listening moment, not because it was the most attentive environment. Several described only glancing at the visual features when something specific caught their interest mid-task.

Environment insight

Travel produced the deepest engagement

Commuters described giving the podcast their full attention specifically because they had nothing competing for it. Transcripts were especially valued here for following content in noisy environments.

Environment insight

Work/study was the weakest fit

Contrary to general market research on podcast listening, our participants felt their attention was already committed while working or studying, and mostly reacted negatively to interacting with the enhanced features in this context.

Design implication

Some users treated it like a webpage, not a podcast

A subset of participants read the transcript before pressing play at all — and a few found the combination of transcript, images, and chapter sidebar overwhelming on first contact. This is a mismatch between the interface's implied mental model and listeners' existing one.

High severity

Strong overall preference for the enhanced experience

75% of participants preferred the interactive prototype over their usual podcast app. Crucially, though, most wanted the enhancements to be optional — something they could dip into or ignore depending on how much attention they had to spare.

Positive finding

Outcome

Outcome & Recommendation

The clearest recommendation to come out of the study was to make the visual layer opt-in rather than always-on — letting listeners choose audio-only when their attention is elsewhere, and surface the enhanced features when they have room for them. This directly informed the department's thinking for future iterations of the prototype, and the full findings were published as a BBC R&D white paper (linked above), which I co-authored.

Usability Study & White Paper —BBC R&D Interactive Application