No Pause

Photo of knaveen

Made by knaveen

About the project

A non-invasive solution for hot flashes during menopause using machine learning

Project info

Difficulty: Difficult

Platforms: JupyterM5StackNVIDIATensorFlow

Estimated time: 1 hour

License: Apache License 2.0 (Apache-2.0)

Items used in this project

Hardware components

Seeed Grove - Universal 4 Pin Buckled 5cm Cable Seeed Grove - Universal 4 Pin Buckled 5cm Cable x 1
Seeed Grove - Thermal Imaging Camera / IR Array MLX90640 55 degree Seeed Grove - Thermal Imaging Camera / IR Array MLX90640 55 degree x 1
Seeed Wio Terminal: ATSAMD51 Core Dev Board Seeed Wio Terminal: ATSAMD51 Core Dev Board x 1
M5Stack ATOM Matrix ESP32 Development Kit M5Stack ATOM Matrix ESP32 Development Kit x 1

Software apps and online services

Arduino IDE Arduino IDE
Jupyter Notebook Jupyter Notebook
TensorFlow TensorFlow

Story

Overview

A hot flash is the sudden feeling of warmth in the upper body, which is usually most intense over the face, neck and chest. Although other medical conditions can cause them, hot flashes most commonly are due to menopause. In this project, I build a device which can be used to detect hot flashes and trigger some action to relieve the person, in this case turn on Air Conditioner cooling system using IR emitter. As its input, it takes multidimensional infrared thermal sensor data. Its output will be a simple classification that notifies us if person is recognized and there is a sudden change in temperature has recently occurred.

Hardware consideration

I wanted to make a wearable device which can read body temperature data. After doing some research and implementations I found wearing sensors all the time especially during sleeping is uncomfortable and not feasible in many cases. A hot flash can also cause sweating and wearing a battery powered device needs special housing to accurately measure data. Also, for cooling I investigated thermoelectric Peltier cooler modules and found most of them needs bulkier 12V battery to operate properly for optimal cooling and it is also unsafe if it is not controlled carefully and can cause burns. For a non-invasive solution, I found low resolution thermal camera can read temperature reliably from a distance and they can work in dark. Although the reading accuracy may be affected by distance but calculating continuous or sudden change in the thermal image data can be recorded accurately. I chose Atom Matrix as a microcontroller device because it is small, cheap, can run TensorFlow model and has an IR emitter which can be used as a remote control. I chose Wio Terminal to capture thermal camera data for training because it has an LCD screen with many buttons which is used to capture data for different categories.

Data collection for training

The first and the most important step in a machine learning project is to collect the training data in such a way that it should cover most of the representative cases for a given recognition task. The 3 buttons on the Wio Terminal was used to label the 3 classes.

The captured data is saved to the files on a micro SD card attached to the Wio Terminal. Each thermal image data was captured as a separate file. The file contains no header line, only the comma-separated 768 (24x32) temperature readings. An example readings file contents looks like as follows.

26.47,25.97,25.85,25.72,26.90,26.12,26.60,26.86,27.00,26.68,26.90,26.74,27.78,27.21,27.75,29.12,31.29,31.50,32.24,31.95,31.72,30.80,31.29,30.69,31.18,30.86,31.46,31.37,29.21,28.23,28.18,28.03,25.83,26.33,25.55,26.56,26.59,26.90,26.52,27.38,26.94,27.39,26.85,27.21,27.32,27.66,28.81,30.45,30.97,31.74,31.55,32.14,31.37,31.03,30.63,30.69,31.03,31.52,30.85,31.14,28.80,28.57,27.81,28.39,26.43,26.24,26.67,26.71,27.13,26.99,27.63,28.07,28.59,28.39,27.80,28.19,28.11,28.25,30.91,32.15,31.78,31.46,31.82,31.33,31.10,30.43,30.37,30.06,29.77,29.84,30.58,30.45,29.28,28.42,28.34,27.76,26.31,26.62,26.38,27.24,27.27,27.91,28.94,29.11,30.11,29.73,30.25,29.53,29.59,29.22,32.01,32.70,33.17,32.00,31.15,31.52,30.59,30.46,29.87,30.07,29.43,30.09,30.01,30.68,29.10,28.91,27.99,28.34,26.59,26.60,26.99,27.49,28.68,29.64,31.88,33.14,33.41,33.02,32.48,32.83,32.60,32.60,33.58,34.16,34.79,34.58,32.43,32.15,31.07,30.77,29.84,29.89,30.01,29.48,29.99,29.63,29.33,28.47,28.23,27.90,26.26,26.26,27.13,28.21,30.50,31.41,33.23,33.70,33.44,33.40,33.38,33.18,33.31,33.12,33.65,34.33,34.81,34.93,33.96,32.50,31.29,30.82,29.37,29.93,29.13,29.93,29.29,30.07,28.76,29.00,28.38,28.69,26.83,26.55,27.36,28.68,32.50,33.12,33.78,33.40,34.17,34.14,33.80,33.56,33.84,33.35,33.85,33.54,34.63,34.45,34.68,34.49,31.76,30.94,29.87,29.38,29.67,29.20,29.45,29.68,29.02,28.42,28.63,28.49,26.29,27.30,27.51,28.71,31.96,33.70,33.86,33.76,33.87,34.32,33.60,33.94,33.41,33.39,33.74,34.04,34.32,34.95,34.24,34.63,31.81,31.15,29.59,29.78,29.19,29.62,28.77,29.72,28.88,29.12,28.51,28.79,26.45,27.15,28.06,28.72,32.45,33.05,33.89,33.84,33.60,33.39,34.02,33.69,33.64,33.29,34.11,33.81,34.68,34.55,34.75,34.15,32.55,31.42,30.25,29.76,29.56,29.42,29.64,28.59,28.87,28.66,29.10,28.79,26.76,27.06,27.39,28.97,31.63,33.46,33.78,34.22,33.97,34.11,33.58,34.20,34.01,33.97,33.69,34.16,34.54,34.90,34.33,34.63,33.09,31.81,29.81,30.15,28.89,29.81,28.96,29.39,28.70,29.35,28.54,29.07,27.18,26.59,27.31,27.98,31.52,32.56,33.37,33.52,33.72,33.72,33.52,33.22,33.75,33.27,33.93,34.13,34.69,34.65,34.26,34.12,32.70,31.12,30.58,30.45,29.91,29.42,29.72,28.82,29.66,29.10,28.91,29.04,26.26,26.72,26.66,27.70,29.47,31.74,32.43,33.58,33.48,33.45,32.63,32.79,31.94,33.32,33.36,34.14,34.42,34.55,33.99,33.87,31.15,30.99,30.15,30.84,29.81,30.02,29.24,29.61,29.15,29.18,28.82,29.32,27.21,26.60,27.01,26.83,27.83,27.64,29.27,29.08,30.97,29.96,29.96,28.30,29.66,30.77,34.03,33.60,34.23,33.26,32.36,31.42,30.61,30.63,30.44,30.50,30.45,30.46,29.70,29.37,29.70,29.34,29.17,28.83,26.28,26.53,26.36,27.31,26.68,27.55,27.46,28.27,27.83,28.53,27.72,28.01,28.14,30.84,32.27,33.38,32.09,32.70,31.33,31.32,30.03,30.53,29.86,30.86,30.35,30.88,29.78,29.66,29.58,29.62,29.06,29.77,26.63,26.16,26.65,26.85,27.10,26.79,27.12,26.67,27.47,27.14,27.15,27.15,27.93,28.86,31.42,31.62,31.87,31.27,31.11,30.65,30.47,30.46,30.62,30.17,30.45,29.94,29.73,29.38,29.33,29.33,29.18,29.06,25.92,26.74,26.20,26.93,27.03,26.89,26.71,26.92,26.90,27.08,26.74,27.46,27.06,28.62,31.36,32.15,31.96,31.97,31.03,31.01,30.20,30.71,31.08,31.04,30.51,30.21,29.31,29.81,29.18,29.43,29.07,29.56,26.80,26.61,26.85,26.61,26.86,26.81,26.86,27.13,27.21,26.94,26.84,26.77,27.24,26.68,30.82,31.45,32.56,31.93,31.30,31.13,31.15,31.29,31.83,31.84,31.02,30.16,29.38,29.19,29.56,29.13,29.41,28.83,26.54,26.56,26.58,26.95,26.81,27.11,26.44,27.07,26.81,27.04,26.77,27.05,26.85,26.94,28.29,30.73,31.36,31.84,30.73,30.88,30.73,30.90,31.61,31.92,30.55,30.41,28.76,29.27,29.02,29.64,29.32,29.59,26.79,26.32,26.78,26.93,26.85,26.80,26.95,27.14,26.90,26.59,26.60,26.76,27.19,27.05,27.35,27.16,28.37,28.20,29.84,29.51,31.84,32.69,32.54,31.55,31.03,30.36,29.18,28.63,29.54,28.92,29.18,29.26,26.28,26.71,26.30,27.09,26.60,26.93,26.71,27.05,26.46,27.39,26.90,27.22,26.64,27.33,26.61,27.39,27.15,28.14,28.76,29.57,31.26,32.76,31.79,31.63,30.77,31.15,29.09,29.45,28.78,28.99,29.07,29.83,26.96,26.68,26.70,26.47,26.79,26.53,26.86,26.56,27.08,26.60,26.86,26.50,27.11,26.74,27.29,27.29,27.16,27.50,28.66,28.47,30.35,30.02,30.71,30.36,31.65,31.08,30.13,29.60,29.28,29.08,29.37,29.02,26.45,26.86,26.63,27.07,26.53,27.05,26.85,27.21,26.48,27.28,26.80,27.12,26.54,27.20,26.52,27.19,26.77,27.61,27.91,28.58,28.91,29.42,29.19,30.28,30.07,31.03,30.25,29.85,29.17,29.61,29.30,29.79,26.88,26.71,26.97,26.73,27.01,26.79,26.83,26.99,27.51,26.93,26.96,26.79,27.22,26.88,27.34,26.87,27.20,27.23,27.70,27.45,28.61,28.18,29.45,29.06,29.99,29.77,30.61,29.50,29.72,29.00,29.41,30.15,25.94,27.25,26.77,26.97,26.82,27.24,26.67,27.18,26.75,27.16,27.01,27.37,26.91,27.49,26.81,27.24,26.89,27.97,27.36,28.21,27.73,28.77,28.52,28.88,29.20,30.00,29.95,30.37,29.49,29.48,28.84,29.91

The visual representation of the captured data is as follows:

Thermal Images for 3 categories

An Arduino sketch (Thermal_camera_data_collection.ino) to capture training data is available at the Github repository mentioned in the code section.

Data collection session video

Data splits for training and validation

The data were captured for 3 categories: person (in different orientations and postures), hot or cold objects, and background (nearly uniform temperature for example, wall or empty room). A little over 100 samples for each classes were captured. The collected data has been split into training (60%), validation (20%), and testing (20%) datasets. Since the data was collected from the infrared temperature camera (MLX90640) which is already calibrated and they are already within a specified range so we can use the raw data as is for training and inferencing.

Model Architecture

We can think of the input data as an image of 24x36 pixels. A convolutional neural network is one of the best options suited for recognizing patterns in images and time-series sequence data. The first few layers are 2D convolution neural networks with few other regularization layers. The last layer is a fully connected dense layer with softmax activation which outputs a probability of all 3 classes. The summary of the model is given below.

  1. Model: "sequential"
  2. _________________________________________________________________
  3. Layer (type) Output Shape Param #
  4. =================================================================
  5. conv2d (Conv2D) (None, 24, 32, 8) 80
  6. _________________________________________________________________
  7. conv2d_1 (Conv2D) (None, 24, 32, 8) 584
  8. _________________________________________________________________
  9. max_pooling2d (MaxPooling2D) (None, 12, 16, 8) 0
  10. _________________________________________________________________
  11. dropout (Dropout) (None, 12, 16, 8) 0
  12. _________________________________________________________________
  13. conv2d_2 (Conv2D) (None, 12, 16, 8) 584
  14. _________________________________________________________________
  15. max_pooling2d_1 (MaxPooling2 (None, 6, 8, 8) 0
  16. _________________________________________________________________
  17. dropout_1 (Dropout) (None, 6, 8, 8) 0
  18. _________________________________________________________________
  19. conv2d_3 (Conv2D) (None, 6, 8, 16) 1168
  20. _________________________________________________________________
  21. max_pooling2d_2 (MaxPooling2 (None, 3, 4, 16) 0
  22. _________________________________________________________________
  23. dropout_2 (Dropout) (None, 3, 4, 16) 0
  24. _________________________________________________________________
  25. conv2d_4 (Conv2D) (None, 3, 4, 16) 2320
  26. _________________________________________________________________
  27. flatten (Flatten) (None, 192) 0
  28. _________________________________________________________________
  29. dense (Dense) (None, 64) 12352
  30. _________________________________________________________________
  31. dropout_3 (Dropout) (None, 64) 0
  32. _________________________________________________________________
  33. dense_1 (Dense) (None, 32) 2080
  34. _________________________________________________________________
  35. dropout_4 (Dropout) (None, 32) 0
  36. _________________________________________________________________
  37. dense_2 (Dense) (None, 3) 99
  38. =================================================================
  39. Total params: 19,267
  40. Trainable params: 19,267
  41. Non-trainable params: 0

Model Training and Evaluation

The training of the model was done on an Intel NUC with Linux and an eGPU (NVIDIA GTX 1080Ti). Although it just takes couples of minutes to train on a CPU but the development process becomes pretty slow while testing out different architectures and hyper-parameters. The TensorFlow 2.1 with Keras API is used for model creation and training process. I created a Jupyter notebook for data processing, training and the final model conversion. All code are available at Github repository which is mentioned in the code section. The training accuracy is 99% and evaluation accuracy on test data is 92.86% which can be further improved with more training datasets and model hyper-parameters tuning.

Inferencing on the device

The created model is converted to the TensorFlow Lite model and the converted model is transformed into a C array file for deploying with the inferencing code. The TensorFlow Lite Micro SDK is used to run inference on the device. I have created an Arduino sketch (Hot_flash_detector.ino available at Github repository) for inferencing and displaying the result. The microcontroller receives the samples continuously from the thermal camera sensor at the interval of 500ms and detect the sudden change in the temperature of the recognized person based on last 20 average maximum temperature readings.

Inferencing demo

The use cases for common benefit and cost

It is an easy to use low-powered device which can run on battery for weeks. It can be used safely for women facing hot flashes problem during day or night. This device is also low cost. The total cost of the final working product (Thermal camera + Atom matrix) is well below 50 USD and can be further reduced if it is mass produced.

Scope for improvement

The device can be used to detect sleeping problem by analyzing thermal images time-series data and can trigger a music system to play some soothing music or control smart lighting systems. Also, some analytics data can be saved locally at the Atom Matrix's SPI flash memory and can be synced using a mobile phone app over BLE for further analysis.

Schematics, diagrams and documents

Schematics 1

Code

Hot Flash Detector

Credits

Photo of knaveen

knaveen

Bioinformatician, Researcher, Programmer, Maker, Community contributor Machine Learning Tokyo

   

Leave your feedback...