Emotion recognition from keystroke and mouse dynamics for bettering design teamwork performance and well-being

Project: Design for emotion recognition

Design of an emotion forecast tool based on keystroke and mouse dynamics for bettering individual well-being and design teamwork performance (PhD research project)

Individual emotion can significantly impact one’s personal well-being and a team’s emotional environment can affect team performance. In general, a team with high Emotion Intelligence (EI) usually exhibits higher teamwork effectiveness and performance. In addition, one’s emotions in a team can affect others’ emotions. Thus, in order to have a positive team emotional state (environment) and avoid one’s negative emotions propagating to others, a team emotion forecast tool is needed for teamwork management, especially, when the hybrid working mode in post-pandemic is widely adopted with some working in a face-to-face environment while others working online, making awareness of one’s own emotional states and others’ emotions harder since people can’t perceive others’ emotions as easily through facial or body expression or speaking when they work remotely. How to design an easy-to-use team emotion forecast tool is our research question.

Emotion recognition is much studied in affective computing field with facial expressions, gaze, voice, and keystroke and mouse dynamics (KMD), etc. However, emotion recognition in offices is not well studied yet. Therefore, this project is based on office teamwork scenario, and aims to design an easy-to-use emotion forecast tool to help users improve teamwork performance and personal well-being in parallel.

This project applies the research-though-design process: Firstly, we propose to use keystroke and mouse dynamics as potentially effective signals to recognize emotions because of their ubiquity in the work environment, especially in remote work. And the emotion recognition process is non-intrusive and unobtrusive, which helps to identify emotions more objectively. Secondly, we apply the user-centered design methods, prototyping our initial design idea to do survey, interview, and focus group study to understand potential stakeholders’ expectations and concerns about a smart emotion forecast tool for next round design improvement and evaluation.  From this process, we have found that (1) privacy issue is the most concerns of people, (2) people’s task type (mental workload) can also affect their behaviour of using keyboard and mouse. Thirdly, we designed and made a digital tool for collecting keystroke and mouse data, task type and emotional states self-report. Finally, we will build a learning model to predict emotion.