# Using Machine Learning and Smartphone and Smartwatch Data to Detect Emotional States and Transitions: An Exploratory Study https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-17818-accepted.pdf - Madeena Sultana;  - Majed Al-Jefri;  - Joon Lee - ----- [[Ecological Momentary Assessments]] [[Digital Interventions]] ==I think this is important. Give us idea on how to proceed== Maybe related to [[Intelligent real-time therapy - harnessing the power of machine learning to optimise the delivery of momentary cognitive-behavioural interventions]] ---- So, without active self-report, we can detect people emotional state? that will be useful. ## The Dataset We obtained data from a publicly available dataset called ExtraSensory [22]. This dataset was collected by the researchers of the University of California, San Diego (UCSD) in 2015-2016 for automated context labeling from signals captured via a wide range of smartphone and smartwatch sensors such as an accelerometer, gyroscope, magnetometer, compass, location services, audio, phone state, light, air pressure, humidity, temperature [23]. It contains data in free-living condition from 60 subjects, who were mainly students (both undergraduate and graduate) and research assistants at UCSD. The sensor data were collected every minute and the contextual data were self-reported at different intervals by the users. This dataset also contains optionally self-reported discrete emotions at different time-intervals. There was a total of 49 different discrete emotions (e.g. active, calm, happy, sleepy, etc.) that were reported by the subjects and the interval varied from 1 minute to several days. Researchers processed and cleaned the self-reported data by combining various sources of information such as location and other labels [23] to make them reliable. Both the raw and cleaned versions of self-reported data are available. We used the cleaned version in this study. ## The Pleasure, Arousal, Dominance (PAD) model - Developed by Mehrabian and Russell [24] in 1974 to assess persons’ psychological responses to the environmental perception and experience. Persons’ emotional states can be perceived in three basic dimensions: pleasure, arousal, and dominance - Pleasure - dimension for positive or negative feelings [24]. - Arousal represents the states of mental responsiveness [25] - Dominance is the perceptual cognitive dimension of the feeling of influenced or controlled [25] How they do it? - Built personalized models - using each person’s data to analyze the impact of variability across individuals. - built generalized models using data from multiple individuals and validated them using data from other individuals who were left out during training. What is "Affective Ratings of Emotions (ANEW)"? - ANEW was developed by the Center for the Study of Emotion and Attention (CSEA) to provide standardized materials to researchers studying emotion and attention. The latest ANEW database [27] contains affective meanings of nearly 14,000 English lemmas rated by a larger cohort of 1827 participants with a wide range of diversities including age, occupation, and educational differences. - ==The rest of the paragraph, I don't understand. But perhaps important. == ## Feature Engineering - Collected from senors - Motion - Accelerometer, Gyroscope, Magnetometer. Smartphone - Accelerometer, Compass - Audio - ? - Location - Every minute of the persons. - Phone state - app states, battery plugged, battery states, ringer mode, on the phone, wifi, screen brightness, battery level. - Environmental - Light, pressure, humidity, temperature.. - Temporal - Time - Contexture - Indoor, Outdoor, Eating, In a car - Self reported ## Machine Learning Models [[Machine Learning]] #machine_learning - Logistic Regression - Random-Forest - XGBoost - CatBoost - Multi-Layer Perception Lots of technical details. ## Discussion - Contextual details, sensed phone states, motion-related signals are most influential features for emotional transition detection - Prevailing emotional states and direction can be detected by machine learning to information captured by smartphones and wearables. - Wide range of inter-personal variation (Ok. So everyone is different), No single ML model performed best. - Personalised models will be the better way. ![[preprint-17818-accepted.pdf]]