← Study 2 · Study 3 of 3 · PhD overview →
Development: Building the interaction with PoseNet and React
I built the movement-based interaction to run in a web browser. PoseNet, a pose-estimation model, tracks body keypoints such as wrists, elbows and shoulders from an ordinary webcam in real time, and React handles the interface.
The initial steps for the process are as follows: First, load TensorFlow.js. Next, create a canvas to contain both the expert video and the live user webcam. It’s important to ensure that the exact dimensions of both the camera and the expert video are determined beforehand. After this, the next step is to add the videos to the canvases. Then, the keypoints should be extracted by run the video through PoseNet. The result from PoseNet is in JSON format, which needs to be converted into an array. After that the next step is to vectorise the key points. The next step is to apply normalisation using L2, as recommended by the mirror technique. Finally, calculate the cosine similarity and perform additional calculations to arrive at the Euclidean distance, which can be interpreted as a cosine distance.
The second approach that was initially used for study 1, using PoseNet from TensorFlow.js. TensorFlow.js is a JavaScript library that allows the direct running of TensorFlow models in a web browser. It provides an API for loading pre-trained TensorFlow models and running them in the browser using WebGL for fast GPU acceleration (Friedhoff and Alvarado, 2018). In the more updated version of PoseNet, it includes PoseNet for TensorFlow Lite: PoseNet is also available as a pre-trained TensorFlow Lite model that can be used in a web browser using the TensorFlow Lite Interpreter JavaScript library (Friedhoff and Alvarado, 2018). This library allows running TensorFlow Lite models in the browser without requiring a server.
The last implementation uses Python run in flask or Django. Utilising Python with Flask or Django allows for the construction of a RESTful API, development of a user-friendly web application, creation of microservices for specific tasks, enforcement of authentication and authorisation, integration of third-party APIs, interaction with databases using Django ORM or Flask-SQLAlchemy, and management of background tasks and scheduling. These frameworks provide a robust foundation for building scalable applications and offer a diverse set of tools and features to streamline the development process.
There are advantages and disadvantages to using P5.js, ML5.js, and React. It is easier to run the canvas in P5.js, webcam, and PoseNet using P5 and Ml5; however, there are limitations due to the limited nature of the platform. Creative artists or specialists mainly use it for easy and quick machine-learning implementation. The downside is that there are limits to how much the researcher can customise the parameters. Another limitation is that there are functions in Javascript that overlap or do not work in p5.js, hindering the functionality of the web browser, such as map functions that have different usage in P5.js and React.
How it works
Pose estimation refers to the process of detecting and tracking the key points of a human body in an image or video frame (Friedhoff and Alvarado, 2018; Papandreou et al., 2018). These key points may include joint positions, body parts, or facial features, depending on the specific task (Friedhoff and Alvarado, 2018; Papandreou et al., 2018). The goal of pose estimation is to accurately estimate the pose or configuration of the human body in the image or video frame (Friedhoff and Alvarado, 2018; Papandreou et al., 2018). Pose estimation can be used for a variety of applications, such as motion capture, activity recognition, and human-computer interaction (Friedhoff and Alvarado, 2018; Papandreou et al., 2018).
A movement-based interactive system is a technology that engages users through their physical movements, allowing for interactive and responsive experiences. These systems often leverage various technologies, including sensors, cameras, and machine learning, to detect and interpret users' movements. The goal is to create a dynamic and engaging interface where user actions or gestures trigger specific responses from the system.
To build the movement-based interactive system for mindfulness practices, such as Tai Chi, utilising pose estimation, general approach of movement-based interactive system for Tai Chi using pose estimation model would be as follows: Train a PoseNet model with an annotated dataset for accurate tracking of users' body keypoints. For real-time pose estimation, seamlessly integrate the trained model into the interactive system, capturing users' movements through a camera or video feed. The PoseNet model processes each frame to estimate the positions of their body keypoints. Generate feedback by comparing estimated keypoints with reference Tai Chi poses, overlaying visual cues on the video feed to highlight alignment errors or areas for improvement. Conduct a movement quality assessment by analysing transitions between poses, detecting deviations, and evaluating balance and symmetry. Provide immediate corrections in real-time to help users adjust their alignment and movement flow on the spot. Measure and evaluate user experiences through interviews and observations, gathering feedback on feedback effectiveness, ease of use, and impact on practice outcomes.
Key design decisions
Designing interactive movement systems using machine learning involves several critical considerations:
Firstly, the quality of training data is essential for the system's effectiveness. Accurate data is crucial, as noisy or biased data can lead to inaccurate feedback, impacting the overall performance of the system. Secondly, the system should show model adaptation capabilities to accommodate diverse user skill levels and physical differences. This adaptability ensures that the interactive feedback system remains inclusive and effective for a broad range of users. Lastly, the user experience is a key factor. The feedback provided by the system should enhance the users' experience without being intrusive or overwhelming. Striking the right balance ensures that the technology seamlessly integrates into the users' practice, augmenting their engagement rather than causing distraction.
It is also important in considering the ethical considerations. Prioritising user privacy and data security is essential when collecting and processing movement data. Upholding ethical standards safeguards user trust and ensures responsible use of personal information.
Lastly, incorporating measurement through interviews and observations adds a qualitative dimension to system evaluation. By interacting with users and observing their engagement with the interactive Tai Chi feedback system, valuable insights into their perceptions, experiences, challenges, and suggestions can be understood. Qualitative data obtained through interviews and observations becomes a crucial feedback loop, guiding iterative refinements to enhance the system's effectiveness and user satisfaction over time.
Technical challenges
In online practice, many challenges are related to angle, perspective, and camera. These included being unable to show the whole body to both students and experts and being unable to show and see the subtle movements. In a live class, the experts think that the students are unable to see the whole body of their students, and the students are unable to see the experts due to camera limitations. The limited number of cameras, the fixed position of the camera, and the conference software that streams the live class are the main challenges of live online classes.
Another challenge is related to giving feedback and correction. It is easier to give a physical touch when necessary and when possible, whereas it is more difficult to give this online. Environment and community are also another challenge since there are more distractions when doing online practice or the students do not want to turn on the camera. Different environments, i.e. moving from the house to the training centre, make the change of mindset easier compared to practising at home.
Issues with mirror image
The pilot study validates the need for feedback for beginners; feedback will still be the primary implementation. However, the type of feedback needs further research. The participants seemed confused about the layout of the visual feedback. Mirror image layout means that two videos are shown in the feedback, in this case, a screen. The videos were of an expert practitioner and the participants' live webcams. The visual video layout needs to be more easily understood by the participants. As shown in the initial feedback and referred to challenges for beginners in online live classes in Study 2, mirroring display confuses the participants because they need to think about how to follow the expert and think about their own body. Therefore, perhaps an overlay video of experts and users is a better alternative. This overlay layout in a previous research called as MatchPose system (Gallo et al., 2022) has been used to evaluate dancer beginners' practice during lockdown. Another overlaying example for feedback using pose estimation was conducted, though, with a difference in the visual feedback and position of the expert video relative to the user's position in the feedback video (Tharatipyakul et al., 2020).
Issues with complicated movements
The second issue is that the short form (24 Tai Chi Yang Style short form) needed to be easier for the complete beginner. Hence, simplifying the movements involves breaking down the 24 movements rather than a single movement in the next iteration might be a good option for the following study. Participants will be required to perform only 3 or 4 initial forms, including the start pose, push hand, grasp the bird's tail, and single whip. These forms only require a few broad movements. Do these single forms in repetitions and go to the following form once the participant feels confident with the previous movement.
Issues with space
Another issue that showed up was space and movement. In this case, the participants will be asked to measure a box with 1x1m. This setup might enable them to stay within the camera view.
Further suggestions and feedback
Simplify the sequences further to address the challenge participants face with complex movements. Focus on fundamental forms that are more accessible to beginners and gradually introduce more intricate movements as participants gain confidence. Implement a progressive learning approach where participants can master foundational movements before advancing to more complex ones. Break down the sequence into smaller segments and allow participants to build proficiency step by step.
Incorporate visual aids, such as on-screen markers or overlays, to help participants better grasp the spatial requirements of the movements. Visual aids can help maintain synchronisation and reduce confusion related to space. Integrate real-time feedback mechanisms to inform participants of their posture and alignment during movements. Similar to the resources finding, feedback might help them make corrections on the spot and enhance their overall practice experience.
Evaluation
Aim
The framing of this study is to understand the interaction between users and movement-based mindfulness practice after the system is implemented. This study wanted to understand how users respond to feedback and investigate user responses to feedback. Qualitative methods were employed to gain insight into users' experiences during interactions. This exploratory study design addresses the need for more research on feedback in interactive systems for movement. Therefore, it is essential to explore this issue further.
To address RQ 1, a short-form user experience questionnaire (UEQ), a short interview, and a questionnaire were collected after each condition to understand the user experience in each condition. At the end of the session, the post questionnaire, FFMQ to measure mindfulness, and Nasa TLX to measure cognitive load are collected. Interviews were recorded. The audio recordings were transcribed. Thematic analysis was utilised to get the themes of the users' feedback and experience.
This study also seeks to understand the opportunities and challenges of the system to address Research question 1a of the thesis: What is the interactive Tai Chi feedback system? Specifically, to examine further the movement limitations and opportunities of interactive Tai Chi. This study collected both Qualitative and Quantitative Data. Qualitative attempts to describe what happened in a dataset or some setting. Qualitative is best for answering what and how, while quantitative usually seeks to correlate variables to answer why (Silverman, 2011).
The quantitative result of the cosine similarity measure was recorded and analysed by each participant in each condition. The qualitative data were collected from interviews and different questionnaires.
Participants
The participants are the general population. The within-subjects design is one in which participants are assigned to all conditions. In more detail, the Latin square was used to minimise the number of recruited participants in this study, i.e. 5 participants do all the conditions (C1 to C5) in the order specified in the Latin square design. Each participant did 6 repetitions to increase the degree of freedom. 16 participants were recruited for this study (7 males) aged from 25 to 43 years old (M = 30, SD = 4.7). Three of them previously had Tai Chi experience, but currently do not practise regularly. One of the participants has colour-blindness (red-green)
Method
The participants will use, evaluate and give feedback individually. The result will be evaluated more on the qualitative components to understand beginner participants' and experts' opinions on the developed apps. Visual cues would be keypoints and video feedback layouts: overlay and side-by-side. Within-subject design with one independent variable. The independent variable is the similarity result between experts video and beginners video.
The initial procedure consists of several conditions (C) designed to evaluate different feedback approaches. In the baseline condition (C1), both the expert's and the user's videos are presented without any feedback, or potentially without displaying the user's video at all. In the next condition (C2), the expert and user videos are shown side by side, with the user's keypoints or skeleton overlaid, as used in the previous iteration. Condition (C3) introduces an overlay of the expert and user videos for direct comparison. In condition (C4), this overlay is enhanced by adding keypoints or skeleton tracking to the user video. Finally, condition (C5) presents the expert and user videos in a side-by-side layout without the skeleton overlay. The screenshots of the setup are as in the following pages.
Measurement for RQ1 might focus on the performance. Usually, this is measured by time spent on a task or the error rate. However, these measurement are not quite relevant to mindfulness practice. This will be measured using UEQ and NASA TLX in the quantitative part, as in using questionnaires. This was measured using short interviews related to the main questions (the feedback and movements that work or not with the system, as these are not covered in the questionnaire) and short memory quizzes.
Measurement for RQ2 is difficult because most UX qualities measure efficiency, performance, effectiveness, or usability, which may not apply to this kind of reflective study. Some metrics about learnability, reflection, or general feedback, such as questions on how users are satisfied with themselves and the system, might work in evaluating the system.
Analysis
After the participants give their consent, participants will be provided with an interactive application that gives feedback on the similarity of their movement to the Tai Chi movement shown to them. The study will use the Five Facets Mindfulness Survey, Cognitive load survey, User experience Questionnaire, and Interview as a measurement for the user experience of the mindfulness application and the mindfulness state of the participants.
The task questionnaire assessed participant cognitive load during the study, followed by a semi-structured interview. The analysis involves a comparison between before and after studies. The interview was analysed using thematic analysis, and questionnaires were scored according to their respective guidelines.
Thematic analysis is a method for analysing qualitative data, suitable for various research approaches. It is a suitable method to understand experiences, thoughts, or behaviours by identifying patterns or meanings or known as themes, that directly address the research question, rather than simply summarising codes. Themes can be developed either inductively or deductively. The widely accepted six-step process includes familiarising with the data, generating initial codes, searching for themes, reviewing themes, defining and naming them, and finally producing the report (Kiger and Varpio, 2020). Thematic analysis was used in the project to analyse the themes from interviews.
Findings
Accessibility
Previous research did not mention or discuss accessibility for visual feedback in mindfulness applications. In the United Kingdom, roughly 3 million individuals experience colour blindness, constituting around 4.5% of the total population, with a predominantly male majority. On a global scale, it is estimated that approximately 300 million people are affected by colour vision deficiency
One of the strategies implemented to avoid this issue was avoiding red and green, as these are colours with the most colour-blinded population. Another strategy to address this was asking the participants before they took the study whether they had colour blindness. Accessibility is an ongoing process. It is essential to consider what colour the feedback should be, whether it is on the key points or text or any other type of feedback, so the users can have the best advantage of the system.
In the future, besides choosing an accessible colour palette for all users. It is suggested that people avoid relying solely on colour cues to convey information. This means not using colour as the only way to distinguish keypoints in the human body and using other modalities such as audio or haptic feedback.
User experience for the feedback during the experiment
Based on the initial findings from the interview and post-questionnaire, there was a sense of novelty, innovation, and interest in the newly presented feedback, particularly regarding the keypoint. Most participants indicated that they had a positive experience with the significant points. However, it is worth noting that some participants, specifically P3 and P8, found the feedback distracting.
The participants described their positive experience with the keypoints by highlighting how it aided their understanding of angles, particularly when the expert moved in a direction not directly facing the front (P1, P4, P8, and P14). These participants expressed that the keypoints helped them comprehend, compare, and gauge how much their limbs needed to be extended or folded.
Participants elaborated on the distracting aspect of keypoints by mentioning that they felt they were allocating more attention to keypoints than necessary (P9). For example, one participant (P15) noted that the key points distracted them when they wanted to focus on the expert and themselves to see their movements.
Interviews with some participants (P2, P5, P7, and P13) showed that key points helped, but in the current state, the key points are helpful but optional. In a more general feedback, it is rather pleasing to have feedback.
In order to enhance clarity, it is crucial to address the issue of overlapping keypoints. Several participants (P2, P5, P7, and P8) have expressed concern that the visual impact of overlapping two keypoints was overwhelming. It creates an overwhelmed feeling with the many visuals and tasks they need to manage. Some of the causes are that the participants were learning the movements and were not familiar with the movements; the participants needed clarification about which ones were their keypoints (especially the ones that have C3/C5 conditions that overlap). These participants needed clarification with the visuals because they wanted to see the expert; however, the opacity was reduced and located behind their webcam results. However, their webcam result had keypoints, so they needed help to see details of the expert's movements. Especially when their keypoints overlap, they could not see their keypoints as it was behind the experts.
References
- Friedhoff, J. and Alvarado, I., 2018. Move Mirror: An AI experiment with pose estimation in the browser using TensorFlow.js. TensorFlow Blog. [To check: confirm this is the source your thesis cites as "friedhoff".]
- Gallo, et al., 2022. [To add: full reference for the MatchPose paper, your thesis key "gallo2022motionperfection".]
- Kiger, M.E. and Varpio, L., 2020. Thematic analysis of qualitative data: AMEE Guide No. 131. Medical Teacher, 42(8), pp.846–854. doi:10.1080/0142159X.2020.1755030
- Papandreou, G., Zhu, T., Chen, L.-C., Gidaris, S., Tompson, J. and Murphy, K., 2018. PersonLab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. Proceedings of the European Conference on Computer Vision (ECCV). arXiv:1803.08225
- Tharatipyakul, A., Choo, K.T.W. and Perrault, S.T., 2020. Pose estimation for facilitating movement learning from online videos. Proceedings of the International Conference on Advanced Visual Interfaces (AVI '20). ACM.