Open Source Flipper Zero, Thread, Matter, and Zigbee on ESP32, Arduino go Zephyr, and more!

0:00:00 / -

Watch now on YouTube Watch now

Plant Identifying, Poison Ivy Killing Gloves

Made by donutsorelse / Artificial intelligence / Environmental Sensing / Plants / Wearables

About the project

The gloves will identify plants, among other things, and let you know if they're edible or dangerous & automatically spray poison ivy.

Project info

Difficulty: Moderate

Platforms: Adafruit, Arduino, Sony

Estimated time: 7 hours

License: GNU General Public License, version 3 or later (GPL3+)

Items used in this project

Hardware components

	Trimmer Potentiometer, 10 kohm	x 1
	Breadboard (generic)	x 1
	SG90 Micro-servo motor	x 1
	5 mm LED: Green	x 1
	5 mm LED: Yellow	x 1
	5 mm LED: Red	x 1
	Adafruit RGB Backlight LCD - 16x2	x 1
	Sony Spresense camera board	x 1
	Sony Spresense boards (main & extension)	x 1

View all

Software apps and online services

	Sony Neural Network Console
	Arduino IDE

Story

Overview

My wife likes to go mushrooming, and I like when she continues living as a result of not eating dangerous mushrooms! She also is highly allergic to poison ivy, so if we're already identifying plants on the go with a wearable we may as well kill the poison ivy.

What We're Aiming For

In a literal sense, we're aiming at poison ivy in an effort to destroy it and all it holds dear. But, in terms of what we're going for with the project, the idea is fundamentally pretty simple. There will be a wearable someone can have on their hand/arm such that they can walk around and it'll tell you what stuff is. If it's a plant, it'll tell what it is and light up green if it's edible, yellow if it might be edible but more info is needed (like mushrooms with poisonous look-alikes), or red if it's inedible/poisonous. And, for brownie points (and because it's somewhat neat), we'll automatically spray poison ivy.

The crazy contraption can be worn on your arm just below a glove because it's a lot easier to control where you're spraying for poison ivy and grab whatever edible mushroom it's pointing at.

Step 1 - The Electronics

To do this, I used the Spresense with the extension board and camera as a starting point. Beyond that, I'd need an LCD screen to display the result of any given identification, 3 different colored LED lights, a potentiometer for the LCD, a servo for spraying the poison ivy killer, and somewhere in the ballpark of a hundred million wires (I didn't count but I think that's right). Needless to say, the breadboard got a little crowded, but there was enough space for everything and, since everything cooperated, this was by far the easy part! I did my best with the schematic to make it readable despite how much is going on.

Step 2 - Neural Networking

Sony developed a Neural Networking console that can be found at https://dl.sony.com/ and can be downloaded to your computer. A lot of resources are in Japanese, but they do have some resources in English as well as many sample projects. Especially considering Sony also built the Spresense I was using, it felt like an ideal fit.

Step 3 - The Onset of Despair and Screaming at The Darkness

Mostly kidding, but for awhile there the Neural Networking results were not where they needed to be at all. They looked a little something like this:

And, honestly, for awhile there this was one of the better ones.

But then something amazing happened. I tried a model that was not only NOT hot garbage, but was about as perfect as I could possibly hope for with 28x28 resolution. Check out this confusion matrix:

See all that confusion? Me neither! It's working PERFECTLY! At first I thought I had screwed up and had the same training and evaluation data or something, but no it was actually just cooperating incredibly well, at long last. In looking at the output results, it was actually determining the likelihood of which image was what and just legitimately getting the highest likelihood for the correct result.

Yes, you're seeing a dog and an airplane in there. I wanted this to be able to recognize more than just plants, and figured it might help to have things that looked different than just a bunch of different leaves and mushrooms.

This is what the setup that cooperated for me looked like:

Step 4 - Prepping Your Model

To use the results of the Neural Networking project, you right click the evaluation and select Export --> NNB.

It'll bring up the exported folder in Explorer. From there you just copy and paste the model.nnb file onto a Micro SD Card. Important side note - the SD Card has to be formatted as FAT 32 to be recognized by the Spresense. Once your SD card is nice and FAT(32) and plugged into the Spresense with your model.nnb file on it, you're ready to move onto the code.

Step 5 - Code

Code, glorious code. Now that it's all functional, it's boiled down to a pretty simple setup.

We initiate the camera and DNNRT (neural networking part), and tell the camera to start streaming. The logic all happens within the stream results code.

Important note: if your model.nnb file isn't super tiny, you may get a memory error. I was experiencing it such that if I initialized the camera first, the DNNRT wouldn't initialize. If I initialized the DNNRT first, the camera wouldn't initialize. The fix is super easy. Just go to Tools --> Memory and increase to 1024KB. I initially tried turning it to 1536KB and it gave a different error, but 1024KB just makes it happy and work correctly.

Back to the code!

With each image from the camera's stream, we loop through the DNNRT and see which of the results has the highest likelihood. If the highest likelihood result is an Oyster Mushroom, we can handle logic accordingly.

For all results, we display what it is. For any plant, we turn on a light to indicate whether we should eat it (green), get more info to determine if it's edible (yellow), or not eat it (red). And, of course, if it's poison ivy we spray it with poison ivy killer.

Step 6 - The Crazy Contraption

While I can't see this ending up in any Walmart's anytime soon, it turned out in a pretty fun way.

To get the poison ivy spraying aspect to work, I had to use a weak servo because the powerful one I bought apparently wants to spin indefinitely, which really defeats the purpose. So, I held the spray button down such that it just barely wasn't spraying, so that all that the servo needed to do was provide just a tiny amount of additional pressure. This worked about as perfectly as I could have ever hoped something engineered together with duct tape and rubber bands could work.

For power, I found an old on-the-go phone charger from the dawn of time. Amazingly, it still worked.

I found some random metal piece in the garage and attached a cardboard base to it for a place for all the electronics to sit. To attach that to my arm without the involvement of duct tape and a horrifying sacrifice of arm hair, I found one of those old ipod holders that attaches to your arm so you can listen to music while you run. This worked great.

Time to Identify and Destroy!

There were admittedly multiple occasions that the contraption identified my lawn as an airplane, which I'm fairly confident is untrue but who am I to make that kind of ruling? Regardless, for 28x28 resolution I was very pleasantly surprised at how well this worked. It was reliably confused about whether morel mushrooms were dogs and vice versa, but the thing is it consistently thought it was one of the two (with a couple exceptions). The area in which it performed best was, thankfully, in identifying poison ivy! This was where it very consistently would see poison ivy leaves and activate and, by the power of God and anime, the servo actually successfully had just the right amount of force to turn the spray on when it saw it and then back off.

So there you have it, the crazy glove that is surprisingly good at finding and killing poison ivy. It's also convinced half the time that dogs are oyster mushrooms and vice versa. Either way, it was a good time and an entertaining way to get started with the Spresense hardware, the Neural Network, and to learn more about the world of Arduino. Hope you enjoyed the ride!

Schematics, diagrams and documents

Schematic

There were a lot of components squeezed onto my little breadboard, but I've tried to make the schematic as pretty as possible for you!

Code

SpresensePlantIdentifierGlove.ino

This runs the show! It uses the nnb file from our Neural Network and the Spresense camera to identify plants and other objects and displays what it is on an LCD screen as well as turns on a green light for edible plants, yellow light for something maybe edible, and a red light for something you shouldn't eat. It also uses a servo when it sees poison ivy, which is used to press the button for poison ivy spray.

/**Code credits to TARO YOSHINO from Youtube, sonydevworld, and Elegoo. I also wrote some I guess.**/ #include <Camera.h> #include <File.h> #include <Flash.h> #include <DNNRT.h> #include <SDHCI.h> #include <Servo.h> // include the library code: #include <LiquidCrystal.h> // initialize the library with the numbers of the interface pins LiquidCrystal lcd(7, 8, 9, 10, 11, 12); #define BAUDRATE 115200 #define OFFSET_X 48 #define OFFSET_Y 8 #define BOX_WIDTH 224 #define BOX_HEIGHT 224 #define DNN_IMG_WIDTH 28 #define DNN_IMG_HEIGHT 28 #define TOTAL_PICTURE_COUNT (9) SDClass theSD; DNNRT dnnrt; DNNVariable input(DNN_IMG_WIDTH*DNN_IMG_HEIGHT); int take_picture_count = 0; int led_color=0; int greenLedPin = 6; int yellowLedPin = 5; int redLedPin = 4; int servoPin = 2; String gStringResult = ""; Servo myservo; // create servo object to control a servo int pos = 0; // variable to store the servo position void printError(enum CamErr err) { Serial.print("Error: "); switch (err) { case CAM_ERR_NO_DEVICE: Serial.println("No Device"); break; case CAM_ERR_ILLEGAL_DEVERR: Serial.println("Illegal device error"); break; case CAM_ERR_ALREADY_INITIALIZED: Serial.println("Already initialized"); break; case CAM_ERR_NOT_INITIALIZED: Serial.println("Not initialized"); break; case CAM_ERR_NOT_STILL_INITIALIZED: Serial.println("Still picture not initialized"); break; case CAM_ERR_CANT_CREATE_THREAD: Serial.println("Failed to create thread"); break; case CAM_ERR_INVALID_PARAM: Serial.println("Invalid parameter"); break; case CAM_ERR_NO_MEMORY: Serial.println("No memory"); break; case CAM_ERR_USR_INUSED: Serial.println("Buffer already in use"); break; case CAM_ERR_NOT_PERMITTED: Serial.println("Operation not permitted"); break; default: break; } } void CamCB(CamImage img){ if(!img.isAvailable()) { Serial.println("No image available"); return; } img.convertPixFormat(CAM_IMAGE_PIX_FMT_RGB565); gStringResult=""; int x; int index=0; static uint8_t label[9]={0,1,2,3,4,5,6,7,8}; uint16_t*buf=(uint16_t*)img.getImgBuff(); float* input_buffer = input.data(); for (int i =0; i <DNN_IMG_WIDTH * DNN_IMG_HEIGHT; ++i, ++buf){ input_buffer[i] = (float)(((*buf & 0x07E0) >> 5) << 2) ; // extract green } Serial.println("DNN forward"); dnnrt.inputVariable(input, 0); dnnrt.forward(); DNNVariable output = dnnrt.outputVariable(0); float max_value =0.0; for(int i=0; output.size() > i; ++i){ if(output[i] > max_value) { max_value = output[i]; index = i; } } Serial.print("Result : "); Serial.println(label[index] + " (" + String(output[index]) + ")"); if(label[index]!=10){ gStringResult+=String(label[index]); } else{ gStringResult+=String(" "); } digitalWrite(greenLedPin, LOW); //This turns off the LED light digitalWrite(yellowLedPin, LOW); //This turns off the LED light digitalWrite(redLedPin, LOW); //This turns off the LED light //Displaying like this because my labels in my NN are bad and I should feel bad switch(index){ case 0: poisonIvy(); break; case 1: displayResult("Turkey Tail", 2); break; //No dangerous look-alikes case 2: displayResult("Grass",0); break; //Maybe fine but let's just not eat grass, ok? case 3: displayResult("Oyster",1); break; //There are dangerous look alikes case 4: displayResult("Dog", -1); break; //Don't eat dogs. Not cool. case 5: displayResult("House", -1); break; //This isn't Hansel and Gretel case 6: displayResult("Airplane", -1); break; //Didn't some guy eat an airplane sometime? Maybe it was a bike. That'd make more sense. case 7: displayResult("Fig Tree", 2); break; case 8: displayResult("Morel",1); break; } //Wait a bit so we're not just spammed with readings delay(5000); } void displayResult(String plant, int ledColor){ //0 - Red (unsafe to eat) //1 - Yellow (maybe safe to eat - need to look into it more) //2 - Green (safe to eat) //display.println(plant); lcd.clear(); lcd.print(plant); if(ledColor==2) digitalWrite(greenLedPin, HIGH); else if(ledColor==1) digitalWrite(yellowLedPin, HIGH); else if(ledColor==0) digitalWrite(redLedPin, HIGH); } void poisonIvy(){ displayResult("POISON IVY!",0); //Spray the poison ivy!! for (pos = 0; pos <= 90; pos += 1) { // goes from 0 degrees to 180 degrees // in steps of 1 degree myservo.write(pos); // tell servo to go to position in variable 'pos' delay(15); // waits 15ms for the servo to reach the position Serial.print("POS1: "); Serial.println(pos); } delay(4000); //Spraying poison ivy for this time for (pos = 90; pos >= 0; pos -= 1) { // goes from 180 degrees to 0 degrees myservo.write(pos); // tell servo to go to position in variable 'pos' delay(15); // waits 15ms for the servo to reach the position Serial.print("POS2: "); Serial.println(pos); } } void setup() { Serial.begin(BAUDRATE); pinMode(greenLedPin, OUTPUT); digitalWrite(greenLedPin, LOW); //This turns off the LED light pinMode(yellowLedPin, OUTPUT); digitalWrite(yellowLedPin, LOW); //This turns off the LED light pinMode(redLedPin, OUTPUT); digitalWrite(redLedPin, LOW); //This turns off the LED light myservo.attach(servoPin); // attaches the servo Serial.begin(BAUDRATE); while (!Serial) { ; /* wait for serial port to connect. Needed for native USB port only */ } // set up the LCD's number of columns and rows: lcd.begin(16, 2); lcd.setCursor(0,0); lcd.print("Identify Plants!"); CamErr err; Serial.println("Prepare camera"); err = theCamera.begin(); if (err != CAM_ERR_SUCCESS) { printError(err); } /* Initialize SD */ while (!theSD.begin()) { /* wait until SD card is mounted. */ Serial.println("Insert SD card."); } Serial.println("Loading network model"); File nnbfile = theSD.open("model.nnb",FILE_READ); if(!nnbfile){ Serial.println("nnb not found"); while(1); } Serial.println("Initialize DNNRT"); int ret = dnnrt.begin(nnbfile); if(ret<0){ Serial.println("DNNRT initialize error"); while(1); } /* Start video stream. * If received video stream data from camera device, * camera library call CamCB. */ Serial.println("Start streaming"); err = theCamera.startStreaming(true, CamCB); if (err != CAM_ERR_SUCCESS) { printError(err); } else{ Serial.println("Camera should be doing something now"); } Serial.println("Set still picture format"); err = theCamera.setStillPictureImageFormat( DNN_IMG_WIDTH, DNN_IMG_HEIGHT, CAM_IMAGE_PIX_FMT_JPG); if (err != CAM_ERR_SUCCESS) { printError(err); } } //Logic is in the camera stream instead void loop(){}

Credits

donutsorelse

I like to make stuff and make something different every week!

Shop

Project Hub

Video

Blog

Shop home

3D printing

Single Board Computers

Development kits

Sensors

Internet of things

Robotics

Accessories

Product of the Week

Electromaker Educator

The Electromaker Show

The Electromaker Podcast

Plant Identifying, Poison Ivy Killing Gloves

About the project

Project info