Clisensio - Climate Sensing And Insect Infestation Control

Climate Change, a highly debated high school topic, but do we really care about it? We see politicians, scientists, researchers, philanthropists, conservationists and almost everyone blaming each other for every climate catastrophe occurring today. The real question is what climate change or change are we talking about?

https://www.iberdrola.com/wcorp/gc/prod/en_US/comunicacion/cambio_climatico_economia_2_res/Infographic_Impact_Climate_Change.jpg

https://www.iberdrola.com/wcorp/gc/prod/en_US/comunicacion/cambio_climatico_economia_2_res/Infographic_Impact_Climate_Change.jpg

Change is never easy for any species or life form but it happens in order to survive and evolve and this is the change we need in order to challenge climate change - the change could be in form of laws to reduce carbon emissions, conserve biodiversity, advanced technologies which can outperform existing techs in terms of greener energy or total innovative revamping of traditional ways of all human activities.

ImportantNote: This project was the result of a discussion with my parents on environment conservation especially saving food crops, beneficial insects and supporting our farmers. My father made me understand a very critical thing- "We all know thechanges happening, we allcan see climatecatastrophes and loss of living organismsandtheir habitats but we haven't made much effort in saving those because over the past decades we haveonlymade detection and warning systems when the world needs a self activation systems to control the situation." Those words were like North Star for my project development. Also, this project was initiated on May 20, World Bee Day - aimingto protect the unsung heroes of nature. My innovation and solution here is to 'adapt and mitigate' and not to 'detect and forget'.

CliSensio - Climate Sensing and Insect Infestation Control is a new adaptive device which runs SensiML AutoML audio recognition on the edge to recognize disease spreading vectors, hungry pests and threatful species. CliSensio device firmware can be flashed for mosquito species identification, pest, rodentsidentifications and elimination responses.

1. Infestations of Insects turning Farmland into Deserts:

A peer-reviewed study which stated that the threat of rising temperatures and unpredictable weather patterns has multiplied tremendously. By 2050, droughts and insect infestations could destroy 26.2 percent of the World's largest tea exporter, Kenya's tea growing areas. A research study concluded that climate change is more dangerous than we can imagine, it could empty 'World's Grain Bowls' - Rising temperatures will stimulate insects' appetites -- and make some prone to reproducing more quickly -- spelling danger for key staples like wheat, corn and rice which feed billions of people.

Thousands of acres of crops could be destroyed by locusts and at a time when the global economy is struggling to recover after the COVID-19 crisis in Asian Nations.

Thousands of acres of crops could be destroyed by locusts and at a time when the global economy is struggling to recover after the COVID-19 crisis in Asian Nations.

While the world is still reeling from the coronavirus pandemic, several countries Brazil, Argentina, India, Pakistan and other East African nations are facing locust attacks which threaten to trigger food crisis in them.

• France stands to lose about 9.4 percent of its maize to pests in a world that is 2 degrees C warmer, compared to about 6.6 percent of yield losses today due to pests.
• In Europe, currently, the most productive wheat producing region in the world, annual pest-induced yield losses could reach 16 million tons.
• Eleven European countries are predicted to see 75 percent or higher losses in wheat from pests, compared to current pest damage.
• In the United States, the world's largest maize producer, insect-induced maize losses could rise 40 percent under current climate warming trajectories, meaning 20 fewer tons of maize per year.
• China, home to one-third of the world's rice production, could see losses of 27 million tons annually.

2.Losing Grassland Birds is another Widespread Ecological Crisis:

Growing unnecessary and competitive use of pesticides and insecticides that sweep out all insects affect the entire food chain of birds. Moreover, the insects are becoming resistant to the chemicals day by day but the birds which eat those insects barely survive for reproduction. Using lots of chemicals to boost farm yields but we tend to forget that due to excessive use of those pesticides, insecticides and inorganic manures, cross pollinating agents, the arthropods( 90% of living organisms found on earth are insects) don't get attracted to those plants or otherwise die due to excessive chemical exposure and pollution. More than 2, 000 pollinating insects are extinct by now and only 500 are left to becomeextinct if not protected - "Even if there is no insect disease in farms, farmers uselessly spray chemicals and kill useful insects like honeybees and other birds too."

More than 90 per cent of the losses are from 12 species including sparrows, warblers, blackbirds, and finches.

3.Eliminating threat species to our pollinators(our best friend, bees):

Insects are pollinators and waste management experts, controlling other insects, keeping the soil healthy while also serving the most important function – being food for those higher up the food chain. In the absence of pollinators, many plant species will disappear.

Flowers can hear buzzing bees—and it makes their nectar sweeter

Also, scientists are now on a hunt for the hornets, hoping to eradicate the species before they wipe out US bees. According to the scientists of the Washington State University (WSU), "it's a health hazard, and more importantly, the shockingly large hornets are a significant predator of honeybees." They will begin trapping queen murder hornets this spring, aiming to detect and eradicate the species.

Populations of honeybees and other pollinators in the US were already under pressure.

4. Billions at risk of mosquito-borne diseases:

Climate Change Will Expose Half of World’s Population to Disease-Spreading Mosquitoes By 2050. According to a study from PLOS Neglected Tropical Diseases by the end of this century almost all of the world's population would be exposed to mosquito-borne diseases. Aedes aegypti and Aedes albopictus are two species of mosquitoes known to carry and transmit potentially deadly viruses like dengue,chikungunya,Zika and yellow fever. These bugs can thrive in urban environments, often traveling alongside humans as we ship goods and travel around the world.

And, according to the study, these tropical mosquitoes are poised to expand their ranges, exposing a billion additional people to the diseases they carry in the next 50 years. It’s important to note that this study is on a global scale, and mosquito management efforts will need nation-by-nation or even city-by-city information while trying to combat the viruses. Detecting mosquito breeding grounds is more difficult which is one of the reason for such widespread population of mosquitoes.

1. As far I have researched the only potential solution in long term for insect infestation control is to develop a model to predict breeding areas that may have been missed by ground monitoring- ''The focus will be on stopping hoppers becoming adults, as that leads to another cycle of infestation. We want to avoid that. We want to advise governments early before an upsurge happens." So far it has attracted an investment of £35 million by the United Kingdom as part of its Weather and Climate Information Services for Africa program($60m but, if an upsurge occurs, the cost will soar to $500m) to forecast the movement of locusts using data such as wind speed and direction, temperature, and humidity.

2-3. For saving the grassland birds and beneficial organisms, the only solution is to avoid unnecessary use of insecticides and pesticides. Our CliSensio device will stop the competition among the farmers for using excess fertilizers, pesticides, insecticides to boost production.Using the correct farm audio analysis the device would auto-assist and spray insects/pests repellents with precise localization in the farm without chemical overdosage or wastages and thus save farmers money, efforts and stop degradation of environment.

4.Most mosquito-breeding spots are found in homes and not at construction sites. This makes the job of detecting breeding spots harder because, unlike entering construction sites, the authorities cannot go into homes easily or frequently to check on them. We need such solution which works on ground level autonomously for everywhere including home. Our device CliSensio has a special feature to combat this problem at community and household levels. Source:https://www.straitstimes.com/singapore/why-detecting-mosquito-breeding-spots-is-difficult

The heart of the device is QuickFeather EOS3 https://www.quicklogic.com/products/soc/eos-s3-microcontroller/ All SensiML AI algorithms are going to be performed on this device. See all below images, for device setup.

How is liquid-chemical atomizer utilized in Clisensio?

The liquid atomizer has two components, a piezoelectric atomising head and a driver circuit. A wick is used for supplying the chemical liquid to a piezoelectric atomising head arranged in the spray apparatus. In CliSensio, the devices is used for chemical liquid spraying for targeted species,i.e. insect removal, insect growth retardation, insect repellent, disinfection based upon SensiML audio recognition and feature extraction models trained to detect harmful pests and insects.

1 / 4

Things required. Though I planned to use motor driver and haptic responses but proved to be of no use for this project.

Fixing the solar panel on the lid. Also, made holes so that the insects can enter the box or can enter holes if they are attracted by some lure compounds present in the enclosure.

The device firmware is based on QORC-SDK, https://github.com/QuickLogic-Corp/qorc-sdk all instructions are well documented here.

Clone this QORC SDK repository using the shown command, and check if make and GCC compiler is set correctly or not (if not follow the steps in github link README.md from above link).

$ git clone --recursive https://github.com/QuickLogic-Corp/qorc-sdk
# run gcc -v and make -v to find gcc compiler toolchain and make utility
# versions on your system
$ gcc -v & make -v

You can use any code editor, Eclipse C/C++ IDE is preferred but other IDEs can be used too(I used VS Code as it is easier and swiss knife for my projects).

All the data extraction and training has been done using SensiML Analytics Studio and SensiML data capture lab: https://sensiml.com/products/ Currently data capture lab is only for windows platform, that's one downside for linux and Mac users, but I managed it since I have both systems at my hand(windows and linux separately not WSL). Our CliSensiodevice can run two differentfirmware:

• Data Collection: Since we are using microphone as the only data source for our CliSensio (I had planned to bring in audio as well as sensor data into the project but currently either sensors or audio ML models are supported at a time, not both together.) The audio dataset were already collected, this project is inspiration from the other projects I made in past to solve pest and insects problems, links here: 1. https://www.hackster.io/vilaksh01/tensorcrop-crop-quality-and-farm-control-3831b8 2. https://www.hackster.io/vilaksh01/sigfy-sigfox-farm-yield-with-crop-and-health-monitoring-3d8116All audio data used is uploaded on this github page: https://github.com/vilaksh01/CliSens.io/tree/main/CliSensioMosquito-audio_data However, I recommend you to record it via your own device through SensiML data capture lab to achieve more accuracy, as those audio files are not recorded on same microphone architecture and bit rate.
• Data Segmentation: This was performed through Data capture lab after data collection step. I recommend to use Audacity to check spectrogram for the audio data before proceeding to segmentation, it gives clear analysis about noise-to-valuable data in audio file.

1 / 4 • Aedes Aegypti Audio data segmentation, 55 segments

Aedes Aegypti Audio data segmentation, 55 segments

Anopheles Mosquito Audio

Culex Mosquito Audio

• Training and Saving model: Log into your SensiML cloud or Data Analytics Studio to check our SensiML data and proceed training steps. We trained on two pipelines to make model more efficient and accurate. Pipeline 1 Check below images carefully for all steps.

1 / 3

Segmentation distribution for our audio data

Pipeline 1 achieved an accuracy of 77% after training

Pipelines for our model: I tried testing two pipelines for mosquito breed detection with different Optimization Metric and Feature Extractors. Pipeline 1, is default parameters with f1-score as optimization metric whereas Pipeline 2 has Sensitivity as Optimization Metric and unknown classifier parameter is allowed with custom feature generator set.

1 / 2 • Pipeline 2 achieved accuracy of 75%

Pipeline 2 achieved accuracy of 75%

Feature Extractors config

• Visualizing Model Summary: This is one of the most important and highly rewarding feature for ML training engineers(they get to know their models better). Below are the summary for pipeline 1 and pipeline 2.

1 / 2 • Pipeline 2 Summary

Pipeline 2 Summary

Pipeline 1 Summary

• Download the model: This is the final step to download Knowledge pack for Mosquito Breed Classification and Control for our firmware.

Our generated model json file for Pipeline 2, check the class maps- 1. Aedes Aegypti- is known to transmit dengue virus, yellow fever virus, chikungunya virus, and Zika virus. It is suggested to be a potential vector of Venezuelan Equine Encephalitis virus, and is capable of transmitting West Nile virus., 2. AedesAlbopictus- This mosquito species is a known vector of chikungunya virus, dengue virus and dirofilariasis., 3. AnophelesMinimus- association with malaria, filariasis and arbovirus infections., 4. Background- Noises like wind, white noise, etc., 5. Culex Tarsalis- spread the viruses that cause West Nile fever, St. Louis encephalitis, and Japanese encephalitis, as well as viral diseases of birds and horses., 0. Unknown Classifier;

• Integrating Knowledge Pack with QORC-SDK and our firmware: The app structure is similar to qf-apps/qf_ssi_ai_app in QORC-SDK. First step is to set up our GPIO pin for liquid-atomizer module in src/pincfg_table.c file.

{ // setup Water Atomizer Digital Output
.ucPin = PAD_23,
.ucFunc = PAD23_FUNC_SEL_GPIO_7,
.ucCtrl = PAD_CTRL_SRC_A0,
.ucMode = PAD_MODE_OUTPUT_EN,
.ucPull = PAD_NOPULL,
.ucDrv = PAD_DRV_STRENGTH_4MA,
.ucSpeed = PAD_SLEW_RATE_SLOW,
.ucSmtTrg = PAD_SMT_TRIG_DIS,
},

After setting up GPIO let's modify src/sml_output.c function to direct the action response based upon audio classification or inference performed. Also, from here we can see the serial communication prints out classifications which can be used to send to dashboard for Spatio-Temporal Vector Ecology map over LoRa or cellular, helping governmentto be prepared for unforeseen events(epidemic or pandemic).

static void sml_output_serial(uint16_t model, uint16_t classification)
{
int count;
int wbytes = 0;
int buflen = sizeof(sensor_ssss_ai_result_buf)-1;
int ret;
kb_get_feature_vector(model, sensor_ssss_ai_fv_arr, &sensor_ssss_ai_fv_len);
// you can send other sensors or GPS data in below line for dashboard
count = snprintf(sensor_ssss_ai_result_buf, buflen,
"{"ModelNumber":%d,"Classification":%d", (int)model, (int)classification);
wbytes += count;
buflen -= count;
switch((int)classification)
{
// case if unknown audio found
case 0:
HAL_GPIO_Write(GPIO_4, 1); //blue LED for unknown
HAL_GPIO_Write(GPIO_5, 0); //green LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;

// case if mosquito species 'Aedes' 'Anopheles' 'Culex' detected
case (1 || 2 || 3 || 5):
HAL_GPIO_Write(GPIO_7, 1); // chemical atomizer GPIO 7 on
HAL_DelayUSec(15*1000*1000);  // wait 15 seconds for atomizer
HAL_GPIO_Write(GPIO_7, 0); // chemical atomizer GPIO 7 off
HAL_DelayUSec(15*1000*1000);  // wait 15 before resuming
break;

case 4:
HAL_GPIO_Write(GPIO_5, 1); //green LED for background
HAL_GPIO_Write(GPIO_4, 0); //blue LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;
}

After everything is in place, let's compile and flash the firmware to the device using the below command in project directory.

$ make clean
$ make
$ qfprog --port /dev/ttyACM0 --m4app ./qf_ssi_ai_clisensio_app.bin --reset --mode m4

• Data Collection: We are going to collect audio dataset for different pests, currently for this POC, I am more focused on locust sounds here (as India’s agriculture ministry is hoping to control the invasion before monsoon season hits north India at the end of June, when locusts mature and breed. If the infestation is not controlled, it may threaten summer crops such as rice, maize and sorghum.) However, gathering locust swarm audio data was difficult so I tested with cricket sound files available for free here(Locusts belong to the same order of insects as grasshoppers, katydids and crickets - the Orthoptera). Here, are all the audio data for cricket sound https://mixkit.co/free-sound-effects/crickets/

1 / 5 • Testing the device in rain ;)

• Data Segmentation: We have segmented the audio data for two categories, viz. Cricket and Background.

1 / 2

• Training and Saving model: In SensiML Analytics Studio, we are going to create model and train the data for classification. Our query name is: Q_PESTS_SPECIES. Training for Pipeline_1 is shown in one of the below images. Here, I tested two pipelines as before, the other pipeline is also shown below.

1 / 4

• Integrating Knowledge Pack with QORC-SDK and our firmware: Here's our Pest detection and control json file generated:

Similar to previous firmware, first step is to set up our GPIO pin for liquid-atomizer module in src/pincfg_table.c file.

{ // setup Water Atomizer Digital Output
.ucPin = PAD_23,
.ucFunc = PAD23_FUNC_SEL_GPIO_7,
.ucCtrl = PAD_CTRL_SRC_A0,
.ucMode = PAD_MODE_OUTPUT_EN,
.ucPull = PAD_NOPULL,
.ucDrv = PAD_DRV_STRENGTH_4MA,
.ucSpeed = PAD_SLEW_RATE_SLOW,
.ucSmtTrg = PAD_SMT_TRIG_DIS,
},

After setting up GPIO let's modify src/sml_output.c function to direct the action response based upon audio classification or inference performed.

static void sml_output_serial(uint16_t model, uint16_t classification)
{
int count;
int wbytes = 0;
int buflen = sizeof(sensor_ssss_ai_result_buf)-1;
int ret;
kb_get_feature_vector(model, sensor_ssss_ai_fv_arr, &sensor_ssss_ai_fv_len);
// you can send other sensors or GPS data in below line for dashboard
count = snprintf(sensor_ssss_ai_result_buf, buflen,
"{"ModelNumber":%d,"Classification":%d", (int)model, (int)classification);
wbytes += count;
buflen -= count;

switch((int)classification)
{
// case if unknown detected
case 0:
HAL_GPIO_Write(GPIO_4, 1); //blue LED for unknown
HAL_GPIO_Write(GPIO_5, 0); //green LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;

// case if 'cricket' detected
case 2:
HAL_GPIO_Write(GPIO_7, 1); // chemical atomizer GPIO 7 on
HAL_DelayUSec(15*1000*1000);  // wait 15 seconds for atomizer
HAL_GPIO_Write(GPIO_7, 0); // chemical atomizer GPIO 7 off
HAL_DelayUSec(15*1000*1000);  // wait 15 before resuming
break;

// case if 'background' detected
case 1:
HAL_GPIO_Write(GPIO_5, 1); //green LED for background
HAL_GPIO_Write(GPIO_4, 0); //blue LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;
}

After everything is in place, let's compile and flash the firmware to the device using the below command in project directory.

$ make clean
$ make
$ qfprog --port /dev/ttyACM0 --m4app ./qf_ssi_ai_clisensio_pest_app.bin --reset --mode m4

This pest detection and control use case can be improved for detecting killer Hornets and kill those in their breeding seasons before they reproduce in surplus and pose threat to our friend, bees. If pests are not coming near your device, you can put some luring compounds to attract the pests, attraction will lead them to make wingbeat sound near device and the device in turn would activate chemical-atomizer killing those mature pests before they make breeding grounds everywhere.

Also, I am working to add LoRa and Cellular connectivity for the device to extend the notifications and demographic status features (Just the same IoT stuff, sending the data from RX/TX and visualizing on dashboard) but for a POC this is well-understood......More work ahead :)

We were already troubled and suffering from Covid now "Australia is overrun with its worst plague in decades: millions and millions of mice. The eastern region of New South Wales has been swarmed by an immeasurable number of mice, reports The Associated Press. The mice will cost over half a billion dollars in crop damage, according to the AP." Our CliSensio device can prevent such future outbreaks by detecting mice squeaks and classify their sex, populations in all homes and grain storage houses and eliminate those before the area is swarmed by them using liquid-chemical atomizer module.

If you are interested in seeing this device, or swarm of CliSensio devices to solve urgent problems visit us here: https://clisensio.tech/

1 / 2 • Smart Farm CliSensio Fleet

Smart House CliSensio Fleet

• https://www.theguardian.com/global-development/2020/may/28/make-noise-and-dont-panic-india-tries-to-ward-off-locust-invasion
• https://patents.google.com/patent/KR100511485B1/en
• https://news.mongabay.com/2021/03/as-climate-change-brings-more-floods-mosquito-numbers-could-swell-study/
• https://www.straitstimes.com/singapore/why-detecting-mosquito-breeding-spots-is-difficult
• https://e360.yale.edu/digest/climate-change-will-expose-half-of-worlds-population-to-disease-spreading-mosquitoes-by-2050
• https://www.wionews.com/science/earth-is-losing-2-bugs-every-year-scientists-call-it-insect-apocalypse-356251/amp
• https://www.wionews.com/india-news/watch-india-suffers-worst-locust-swarm-attack-in-decades-alert-in-delhi-301088
• https://apnews.com/article/plague-ravenous-destructive-mice-tormenting-australians-b8b85ecc13e147f151510e8635d2d032

Thank you for reading my project :)