Kitchen Activity Detection By Thermal Sensor Array Thingy:91

The Application is based on the Application Event Manager Library as it is used in the Nordic Applications like the Asset Tracker V2. The modules are structures like modules in the Nordic Applications using their own thread and messages to pass incoming events to this thread. Depending on the module's state and the event type an action is invoked.

Startup Sequence

Firmware Update

Sensor Control

• manage multiple connection references,

• advertising should continue as long as not all connections are in use

• MTU size needs to be managed per connection

• services like BT NUS seem to manage only one connection

• and when notifying or sending data you need to select the correct connection.

So think twice if you really need it.

Services are created in a reusable way. The service itself declares all services and attributes. All functionality for reacting to a value written is given as callback to an init function.All functionality for sending notifications or setting values are defined in the interface. These functions take a connection parameter to decide if the notification shall be sent to all or a specific connection. I specified characteristics to control values and send sensor readings in a UART like manner encoding in JSON and base64 as 768 bytes exceeds the MTU size. I also read the maximum supported MTU size (increased buffers to about 250 bytes) and apply it.

The Shell and the bridge to the nRF9160 are also available via dedicated services.

Firmware Updates

The firmware can be updated via the following possibilities

nrf52840

• USB Uart using the MCU Manager SMP Shell

• BLE GATT using the MCU Manager SMP Bluetooth

nrf9160

• USB Uart using the MCU Manager SMP Shell via the UART bridge running on the nrf52840

• BLE Uart using the MCU Manager SMP Shell via the BLE UART bridge running on the nrf52840

Actually the mcumgr tool does not support Serial style updates via BLE, so a workaround to make mcumgr think it is talking to a serial connection is required. I am using the socat tool in combination with a python script that forwards the virtual terminal to the BLE serial service. Increasing the MTU size helps a lot for increasing the transfer rate. See scripts/ble_nrf9160_shell_forMcuboot.py

Memfault integration

I decided to integrate memfault to detect errors early and also when the device is running in production mode.

The Memfault integration for the nrf9160 is done according to the documentation and samples by setting appropriate configurations in overlay-memfault.conf. Logs, coredumps, memory dumps and added memfault trace reasons upon error detections are available.

While sending the data to the memfault server is integrated into the nrf connect sdk fro the nrf9160, you need to do it manually for the nrf52840. The issue with functions intented to upload via https provided by the nrf connect sdk is that they are hardwired with the memfault credential configuration (the project-key). That means I would use the nrf9160 configuration when I try to upload the nrf52840 data. I evaluated to partly re-use functions, but I ended up writing the same functionality again based on the available functions. I use the Memfault packetizer API to get the data, encode it as base64 and pass it to the nRF9160 where it is sent via the https API. (MCU transfer not yet fully implemented)

Thermal Sensor MLX90640

I selected this sensor because it provided a balance between price and functionality. I wanted a range > 150 °C with a certain resolution, in this case 32x24 pixel. Also the FOV should be wide, in this case 110°.

The other candidates were 

• FLIR Lepton which is far too expensive (and unnecessarily high spec for my case)

• AMG8833 which can only measure up to 80 °C (which is less than boiling water)

• One more (I forgot) which also had a range < 100 °C

Unfortunately Zephyr does not come with a MLX90640, so I needed to integrate the vendor provided driver library myself by implementing the HAL (TWI layer) also taking other libraries like the Adafruit library for Arduino as reference . I decided to integrate the driver in user space as I am not aware of a sensor driver that transfers data larger than a single float via Sensors API. The pin and register configuration is done via device tree overlay. The sensor is attached to the spare pins on the thingy91. As the pins are 1.8V compatible, a level shifter is required. Actually I used none nearly all the time (and messed up voltages). Therefore I had troubles that the debugger and/or sensor was sometimes not working well when the USB cable was attached.

A dedicated work queue reads the sensor periodically and passes it via events with dynamic memory.

Modules with dynamic memory events have two event types. One is for the sensor reading that has dynamic data and one is for the control and error feedback without dynamic data. The main reasons for having two types is that the event macros do not work well for dynamic data events as the constructor requires a parameter.

Watchdog

Data forwarding to nRF9160 (CommCtr, Connectivity Bridge and MCU Exchange)

The source code for the Bridge module that forwards the USB UART and BLE UART to the nrf9160 is a 1:1 partial copy of the Nordic Connectivity Bridge application. Please check out the documentation at Connectivity bridge.

I wanted the modules to be reusable and therefore independent from each other. That means they should not directly communicate to each other by events, but define their own events (1) they react on and (2) own event to notify any results. The glue between modules is the CommCtrl module. It is the heart of the application and defines its bussines logic.

In this case it receives the heat map sensor data, reduces its size by conversion of floats into unsigned 16bit integers, optional compression (lz4, commented out to remove complexity) and sends the values both to BLE and MCU Exchange module. The data sent to BLE is a JSON encoded string.

The MCU exchange module is written in a way that the same code runs on the sender and receiver side. Null terminated JSON strings that encode the event type and data are sent via UART between the MCUs. Base64 is used to avoid Null bytes inside the message. On the receiving side strings are re-assembled in an ISR function on the UART RX, parsed and converted to the original event.

I decided for using JSON Strings because JSON is easy to handle as there are libraries available and readable in logs. I am also using JSON when sending data via BLE. The alternative I had was working out an own protocol or using AT commands, but they are more complex and the AT library (this library was used as a reference for the UART transfer code) was merged into the modem library recently. It might be not the most efficient way, but I think simplified code is more important these days. (unless you are out of resources)

CAF

I integrated the LED and Sensors CAF module as it is a convenient way to work with these. They only need to be configured and then you can send/receive events. LED is used to signalize error states on the nRF9160. Unfortunately there is no LED attached to the nRF52840. Temperature and humidity are registered and forwarded to the analysis module.

Analysis and Edge Impulse

Analysis is meant to combine the sensor data and extract meaningful events like described in the Plan. The idea was to integrate the Edge Impulse deep learning module. Due to limited capacity this part is not yet finished. Machine learning takes more effort than estimated.

It seems like in my case I would need to fully prepare the whole machine learning pipeline as there are no samples for my use case which is an array of temperature measurements over time. It is like an image, but not really an image. So I am having a look at the book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron.

Data collection and labeling

Sending raw sensor data into the cloud would be hundreds of MB - too much for a mobile connection. I mounted the sensor on the ceiling and checked readings via companion app. I saw that just looking at the recording, it is hard to say what exactly going on. Therefore I thought I need also images to make sense of the data and to know how to label them.

As there are not enough pins to attach a camera or an SD card for recording, I decided to start a side project (the datacollector) which is to use the available code on another nRF52840 board (the nRF52840 dongle) that can attach a camera (I worked with an SPI based Arducam in a previous project - even more re-use) and SD card. This was a good test for the aimed re-usability and it turned out to be painful because this side project at up nearly a month of time as it was not working at all even failing to start up in the first week and I needed to start building up each feature step by step from zero on.

At this time I also refactored the code over and over again and reached to the current state where I have modules in a separate folder and extracted cmake and KConfig into each module.

Analysis and Feature Extraction

I had a look at the heat map readings and took some statistics in a jupyter notebook (data/Analysis.ipynb). It appears that there are also faulty readings as one image shows a chess pattern. That obviously needs some data cleaning / preprocessing.

I am not yet sure what features can be extracted besides just using the image as input features. I thought about using some history (or a deep network with history states as used for speech processing?) and some statistics. However as I do not have sufficient data yet and data is also not labeled yet, I cannot say more at this point of time.

Algorithm

I plan to start with a dense net and played with all the parameters

TODO parameters.

Model Evaluation

Still to do

Integration of Edge Impulse EdON

Still to do

Send Sensor Data as PNG images

As a possibility to collect sensor data without an SD card, I tried to integrade LodePNG (I knew it from PC development and did not do a lot research to know in advance that it consumes a large amount of memory) to convert the 24*32 pixel heat map into a greyscale png to upload it to adafruit.io and store it there for further use. I found out that  the algorithm underlying pngs (zlib) takes few memory to decode, but megabytes of memory to encode. Most of the memory is used for pre-building a hash map used for the compression. I reduced it and reduced the window size significantly. Second a lot of memory is used building up a ColorTree that take up a lot of memory per pixel. I reduced that by removing all color channels except one. With these changes the library is likely not useful anymore for re-use, but just fits my needs here.

After conversion to png, I encoded it as base64 and sent it via MQTT.

MQTT

MQTT module follows mainly the simple MQTT sample. I configured it with Adafuit.io which I used as cloud MQTT server. Two things to take care about

• Provisioning of the full certificate chain

• nRF9160 does not support parallel handshakes

The Memfault SDK uses the full certification chain, so I did the same for the adafruit.io. I decided to commission them at startup before activating the modem. So you do not need to provision them manually in advance with the cost of some flash memory. The certificates can be retrieved with the following command.

openssl s_client -showcerts -connect io.adafruit.com:8883

There is a comment about the ssl handshakes limitation here. I did not find it in the documentation. When multiple modules react on the same event this easily happens though.

Homebridge

I used a Homebridge installation on my local PC to convert MQTT topics into smart devices that appear into the Homekit app on the iOS/MacOS. It runs on top of node and can be easily setup by (see full instructions)

sudo npm install -g --unsafe-perm homebridge homebridge-config-ui-x
sudo hb-service install

The plugin Homebridge Mqttthing can connect to MQTT topics and the plugin Homebridge Motion Switch can send push messages. Only a few messages can send push messages. Temperature sensors for example cannot. The trick is to create a rule like if temperature is below 23 °C then activate the motion switch. It is a workaround, but easy and works well.

The homebridge installation and configuration is not part of the repository. Setup and configuration is straight forward.

2022-05-10: Unfortunately the Homebridge setup stopped working, likely due to updates of MacOS and/or iOS or some unintended change to the OS. I can see Homebridge is advertising on the network, but HomeKit is not able to connect anymore.

Companion App

I decided to write a flutter app with the library flutter_blue to handle bluetooth and graphic for rendering the heat map as I wanted to run it both on my development Mac and phone.

There were quite a few (tedious) issues to get the build and application running on a mac.

1. Use the latest version of flutter_blue from github
2. I needed to manually raise the platform target to 11.5 inside macos/Podfile, clean and rebuild 
3. Android Studio must be started from command line by "open /Applications/Android Studio.app", otherwise the PATH gets messed up and CocoaPod will fail to install dependencies
4. To start an app from the home screen and not via Android Studio, you need to run "flutter run --release" to install the app onto your phone
5. Manually give Bluetooth permissions to your application via Settings -> Security and Privacy -> Bluetooth
6. For some reason, starting the app on MacOS from Android Studio e.g. for debugging, the Bluetooth rights are not granted. But if you start the app via Finder double click it works. Therefore start the app via Finder, leave it open in the background and start another instance from Android Studio.

It is a one screen app providing buttons for managing the connection, a slider for setting the interval and a plot to display the heat map. The Provider package is used to connect data to the screen.

All logic for handling the Bluetooth connection, service and characteristics discovery and data handling is inside BleStatemachine.dart. It uses a state machine I manually implemented with the following logic.

• There are states that have an action on entry and exit.

• There are events that trigger transitions or actions while residing within a state. An action can be attached to a transition.

The companion app logic works as described by the following state diagram.

Side Project Data Collector

I reused several modules and added a SD card module and Camera module to periodically take a heat map reading with a camera reading and save it to SD card. They are connected via events and the architecture is the same. To track filenames over re-boot a counter is persisted via the settings subsystem. Later I added the possibility to set the time via BLE service. As the main project is about the Thingy:91 (and time is short) I will not go into detail. Source code is available in the repository and schematics are as below.

The next steps

• Collect more data and play around with some deep networks (and it if does not work out implement some dump threshold based algorithms)

• Try out the modules on a new project

• Find out why the bluetooth core is unstable, maybe replace it with open thread? Or change the architecture to one business logic core and two network core.

OK the last point sound like eating up time ... let's do something that brings more value with all this ground work learnings.

Licenses

The following licenses can be found in the source code.

• LicenseRef-Nordic-5-Clause - Copyright (c) 2021 Nordic Semiconductor ASA 

• Samples, Application and Libraries inside the nRF Connect SDK

• Apache License, Version 2.0 - copyright (C) 2017 Melexis N.V.

• Driver Library for the mlx90640 Sensor

• Apache License Version 2.0 - Adafruit 

• Arduino Driver Library for the mlx90640 Sensor (taken as reference)

• Apache License Version 2.0 - Zephyr

• MIT License or LGPL 2.1 (both are stated in the same repo) - Copyright (C)2011-2015 ArduCAM.com 

• Driver Library for the ArduCAM (only for the datacollector side project)
• zlib License - Copyright (c) 2005-2022 Lode Vandevenne
• LodePNG'

Thanks for reading to the end or at least scanning though all chapters ;)