Snoring Detection Using Nordic Thingy:53

Snoring is estimated to affect 57% of men and 40% of women in the United States, and over 40% of adults in the UK snore regularly. It even occurs in up to 27% of children. These statistics demonstrate snoring is widespread, but its severity and health implications can vary. Snoring can be light, occasional, and unconcerned, or it may be the sign of a serious underlying sleep-related breathing disorder. Snoring is caused by the rattling and vibration of tissues near the airway in the back of the throat. During sleep, the muscles loosen, narrowing the airway, and as we inhale and exhale, the moving air causes the tissue to flutter and make noise. Obstructive sleep apnea is a breathing disorder in which the airway gets blocked or collapsed during sleep, causing repeated lapses in the breath. Snoring is one of the most common symptoms of obstructive sleep apnea. Unless someone else tells them, most people who snore are not aware of it, and this is part of why sleep apnea is underdiagnosed. In this project, I have built a proof of concept of a non-invasive low-powered edge device that monitors and buzzes if you are snoring.

Hardware Selection

We are using Nordic Thingy:53 which is built around the nRF5340 SoC. The application core processing power (128 MHz) and memory size (512 KB RAM) of its dual Arm Cortex-M33 processors enables it to run embedded machine learning models directly on the device. The slim-profile case and inbuilt battery make it portable and a good fit for our purpose.

It has many onboard sensors (see the internal layout in the image below) but we will be using the PDM microphone, Buzzer, and RGB LED in this project. 

Setup Development Environment

First, we need to download nRF connect for Desktop from here: 

https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Connect-for-desktop/Download.

The nRF Connect for Desktop is a cross-platform tool that enables testing and development with nRF5340. Please follow the installation guide in the link above. When the installation is completed, open the app and click on the Toolchain Manager and choose nRF Connect SDK v2.0.0.

By default, the SDK is installed at the /opt/nordic/ncs directory in MacOS. After installation, click on the Open Terminal which opens up a command line terminal with all environment variables initialized to get started quickly with the development.

We are using Edge Impulse Studio for the feature generation and TensorFlow Lite model creation and training. We need to sign up for a free account at https://studio.edgeimpulse.com and create a project to get started. For the local development work, MacOS is used.

Data Collection

We have used Audioset, a large-scale dataset of manually annotated audio events, to download Snoring and other nature sounds which may occur during the night. AudioSet consists of an expanding ontology of 632 audio event classes and a collection of human-labeled 10-second sound clips drawn from YouTube videos. The audio is extracted from the YouTube videos of the select events and converted into Waveform Audio file format (wav) with a 16-bit depth mono channel at a 16KHz sample rate. The following categories selected from the Audioset Ontology are downloaded. The first column is the category ID and the second column is the category label.

/m/01d3sd  Snoring 
/m/07yv9   Vehicle
/m/01jt3m  Toilet flush
/m/06mb1   Rain
/m/03m9d0z Wind
/m/07c52   Television
/m/06bz3   Radio
/m/028v0c  Silence
/m/03vt0   Insect
/m/07qjznl Tick-tock
/m/0bt9lr  Dog
/m/01hsr_  Sneeze
/m/01b_21  Cough
/m/07ppn3j Sniff
/m/07pbtc8 Walk, footsteps
/m/02fxyj  Humming
/m/07q6cd_ Squeak
/m/0btp2   Traffic noise, roadway noise
/m/09l8g   Human Voice
/m/07pggtn Chirp, tweet
/t/dd00002 Baby cry, infant cry
/m/04rlf   Music

The datasets are divided into two categories, Snoring and Noise. Two CSV files are created snoring.csv and noise.csv by filtering the balanced train, unbalanced train, and evaluation datasets CSV files that contain YouTube clip URLs and other metadata can be downloaded from here.

The below bash script (download. sh) is used to download the video clip and extract the audio as a wav file. Please install youtube-dl and ffmpeg before running the command below.

#!/bin/bash
 
SAMPLE_RATE=16000
# fetch_youtube_clip(videoID, startTime, endTime)
fetch_youtube_clip() {
  echo "Fetching $1 ($2 to $3)..."
  outname="$1_$2"
  if [ -f "${outname}.wav" ]; then
    echo "File already exists."
  return
fi
  youtube-dl https://youtube.com/watch?v=$1 
  --quiet --extract-audio --audio-format wav 
  --output "$outname.%(ext)s"
  if [ $? -eq 0 ]; then
    yes | ffmpeg -loglevel quiet -i "./$outname.wav" -ar $SAMPLE_RATE 
    -ac 1 -ss "$2" -to "$3" "./${outname}_out.wav"
    mv "./${outname}_out.wav" "./$outname.wav"
  else
    sleep 1
  fi
}
 
grep -E '^[^#]' | while read line
do
  fetch_youtube_clip $(echo "$line" | sed -E 's/, / /g')
done

To execute the script run the command below.

$ cat noise.csv | ./download.sh
$ cat snoring.csv | ./download.sh

The datasets are uploaded to the Edge Impulse Studio using the Edge Impulse Uploader. Please follow the instructions here to install Edge Impulse CLI tools and execute the commands below.

$ edge-impulse-uploader --category split --label snoring  snoring/*.wav
$ edge-impulse-uploader --category split --label noise  noise/*.wav

The commands above also split the datasets into training and testing samples. We can see the uploaded datasets on the Edge Impulse Studio's Data Acquisition page.

The Snoring events audio clips have background noise in between multiple snoring events which are removed from the clips by splitting the segments. The Noise category audio clips are used without any modifications.

We can do splitting by selecting each sample and clicking on a Split sample from the drop-down menu but it is time-consuming and tedious work. Luckily there is an Edge Impulse SDK API that can be used to automate the process.

import json
import requests
import logging
import threading
 
API_KEY = "<Insert Edge Impulse API Key here from the Dashboard > Keys"
projectId = "<Your project ID, can be found at Edge Impulse dashboard"
headers = {
  "Accept": "application/json",
  "x-api-key": API_KEY
}
def segment(tid, ids):
    for sampleId in ids:
    url1 = "https://studio.edgeimpulse.com/v1/api/{}/raw-data/{}/find-segments".format(projectId, sampleId)
    payload1 = {
        "shiftSegments": True,
        "segmentLengthMs": 1500
    }
    response1 = requests.request("POST", url1, json=payload1, headers=headers)
    resp1 = json.loads(response1.text)
    segments = resp1["segments"]
    if len(segments) == 0:
        continue
    payload2 = {"segments": segments}
    url2 = "https://studio.edgeimpulse.com/v1/api/{}/raw-data/{}/segment".format(projectId, sampleId)
    response2 = requests.request("POST", url2, json=payload2, headers=headers)
    logging.info('{} {} {}'.format(tid, sampleId, response2.text))
 
if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
    datefmt="%H:%M:%S")
    querystring = {"category":"testing", "excludeSensors":"true"}
    url = "https://studio.edgeimpulse.com/v1/api/{}/raw-data".format(projectId)
    response = requests.request("GET", url, headers=headers, params=querystring)
    resp = json.loads(response.text)
    id_list = list(map(lambda s: s["id"], resp["samples"]))
    div = 8
    n = int(len(id_list) / div)
    threads = list()
    for i in range(div):
    if i == (div - 1):
        ids = id_list[n*i: ]
    else:
        ids = id_list[n*i: n*(i+1)]
 
    x = threading.Thread(target=segment, args=(i, ids))
    threads.append(x)
    x.start()
    for thread in threads:
        thread.join()
    logging.info("Finished")