Open Source Flipper Zero, Thread, Matter, and Zigbee on ESP32, Arduino go Zephyr, and more!

0:00:00 / -

Watch now on YouTube Watch now

Smart Ai Gloves

Made by ashokr / Artificial intelligence / Communication / Displays / Home Automation / IoT

About the project

AI-based hand gloves, gives you a super power to control any devices with motions.

Project info

Difficulty: Moderate

Platforms: Avnet, NXP, Raspberry Pi

Estimated time: 7 days

License: GNU General Public License, version 3 or later (GPL3+)

Items used in this project

Hardware components

	Avnet SmartEdge Agile Brainium	x 1
	Raspberry Pi Zero	x 1
	NXP Rapid IoT Prototyping Kit	x 1
	Fan Kit, 120 mm Fan	x 1
	LED Light Bulb, Frosted GLS	x 1
	NXP Kinetis KW41Z	x 3
	Golf Gloves	x 1
	RC car	x 1

View all

Software apps and online services

	Raspberry Pi Raspbian
	Brainium AI Studio

Hand tools and fabrication machines

Soldering iron (generic)

x 1

Story

AI Gloves

It was my old thought of making some gadgets of super heroes like Iron Man. After analyzing the Avnet SmartEdge device and Brainium cloud, got some idea to make such kind of wearable gloves. Controlling devices in gesture manner.

I received the Avnet kit, the first thing I explored about the Branium cloud and basic features. I have done very basic level project here.

Followed by my basic demo, I started this project activity. I listed out the motions required in my project. Then with AI Studio I reordered many motions and created modules. Started testing with mobile app initially. Then later I started using Brainium app in Raspberry Pi.

Motions Set

Brainium APP on Raspberry pi zero

Also integrated the Rapid IoT kit for thread networking and display. Its talks with all the thread end nodes and controls it.

Rapid IoT Kit

The Main Features of AI Gloves:

1. Unlocking by drawing pattern

2. Light ON/OFF

3. Fan speed control

4. Gesture car control

5. Driver style analyzing in real-time

Blocks:

Python is used to detect the motions/alerts from Brainum cloud using MQTT services. Also it process the motion types then send commands to Rapid IoT kit via UART. Rapid IoT takes the action via Thread. Due to cloud based processing, it will take 3 to 5sec to update the results. But still Brainuim performed well overall.

Avnet SmartEdge device has other inbuilt sensors like magnetic, PIR, accelerator sensors which helps in the driver style analyzing, where the driving information can be used to automate or learning purposes in autonomous car industries.

There are some more demos I have planned in future with this all sensors.

Python MQTT

The working demo is here:

Schematics, diagrams and documents

Schematic

Code

Node

void APP_Init
(
    void
)
{
    /* Initialize pointer to application task message queue */
    mpAppThreadMsgQueue = &appThreadMsgQueue;

    /* Initialize main thread message queue */
    ListInit(&appThreadMsgQueue.msgQueue,APP_MSG_QUEUE_SIZE);

    /* Set default device mode/state */
    APP_SetState(gThrDefaultInstanceId_c, gDeviceState_FactoryDefault_c);
    APP_SetMode(gThrDefaultInstanceId_c, gDeviceMode_Configuration_c);

    /* Initialize keyboard handler */
    pfAppKeyboardHandler = App_HandleKeyboard;

    /* Use one instance ID for application */
    mThrInstanceId = gThrDefaultInstanceId_c;

#if THR_ENABLE_EVENT_MONITORING
    /* Initialize event monitoring */
    APP_InitEventMonitor(mThrInstanceId);
#endif

    if(gThrStatus_Success_c == THR_StartInstance(mThrInstanceId, pStackCfg[0]))
    {
        /* Initialize CoAP demo */
        APP_InitCoapDemo();

#if USE_TEMPERATURE_SENSOR
        /* Initialize Temperature sensor/ADC module*/
        APP_InitADC(ADC_0);
#endif

#if THREAD_USE_THCI && THR_ENABLE_MGMT_DIAGNOSTICS
        (void)MgmtDiagnostic_RegisterAppCb(THCI_MgmtDiagnosticAppCb);
#endif

#if THREAD_USE_SHELL && SOCK_DEMO
        /* Initialize use sockets - used from shell */
        APP_InitUserSockets(mpAppThreadMsgQueue);
#endif

#if APP_AUTOSTART
       if(!THR_GetAttr_IsDevConnected(mThrInstanceId))
       {
          mJoiningIsAppInitiated = TRUE;

          if(THR_NwkJoin(mThrInstanceId, THR_APP_JOIN_DISCOVERY_METHOD) != gThrStatus_Success_c)
          {
              /* User can treat join failure according to their application */
          }
       }
#endif
    }
}

/*!*************************************************************************************************
\fn     void App_Handler(void)
\brief  Application Handler. In this configuration is called on the task with the lowest priority
***************************************************************************************************/
void APP_Handler
(
    void
)
{
    bool_t handleMsg = TRUE;

    while(handleMsg == TRUE)
    {
        handleMsg = NWKU_MsgHandler(&appThreadMsgQueue);
        /* For BareMetal break the while(1) after 1 run */
        if(!gUseRtos_c && MSG_Pending(&appThreadMsgQueue.msgQueue))
        {
            (void)OSA_EventSet(appThreadMsgQueue.taskEventId, NWKU_GENERIC_MSG_EVENT);
            break;
        }
    }
}

static void APP_ProcessLedCmd
(
    uint8_t *pCommand,
    uint8_t dataLen
)
{
    /* Set mode state */
    APP_SetMode(mThrInstanceId, gDeviceMode_Application_c);

    /* Process command */
    if(FLib_MemCmp(pCommand, "on", 2))
    {
        App_UpdateStateLeds(gDeviceState_AppLedOn_c);
    }
    else if(FLib_MemCmp(pCommand, "off", 3))
    {
        App_UpdateStateLeds(gDeviceState_AppLedOff_c);
    }
    else if(FLib_MemCmp(pCommand, "toggle", 6))
    {
        App_UpdateStateLeds(gDeviceState_AppLedToggle_c);
    }
    else if(FLib_MemCmp(pCommand, "flash", 5))
    {
        App_UpdateStateLeds(gDeviceState_AppLedFlash_c);
    }
    else if(FLib_MemCmp(pCommand, "rgb", 3))
    {
        char* p = (char *)pCommand + strlen("rgb");
        uint8_t redValue = 0, greenValue = 0, blueValue = 0;
        appDeviceState_t appState = gDeviceState_AppLedRgb_c;

        dataLen -= strlen("rgb");

        while(dataLen != 0)
        {
            if(*p == 'r')
            {
                p++;
                dataLen--;
                redValue = NWKU_atoi(p);
            }

            if(*p == 'g')
            {
                p++;
                dataLen--;
                greenValue = NWKU_atoi(p);
            }

            if(*p == 'b')
            {
                p++;
                dataLen--;
                blueValue = NWKU_atoi(p);
            }

            dataLen--;
            p++;
        }

        /* Update RGB values */
#if gLedRgbEnabled_d
        Led_UpdateRgbState(redValue, greenValue, blueValue);
#else
        appState = gDeviceState_AppLedOff_c;
        if(redValue || greenValue || blueValue)
        {
            appState = gDeviceState_AppLedOn_c;
        }
#endif
        App_UpdateStateLeds(appState);              
    }
    else if(FLib_MemCmp(pCommand, "color wheel", 11))
    {
#if gLedRgbEnabled_d
        App_UpdateStateLeds(gDeviceState_AppLedColorWheel_c);
#else
        App_UpdateStateLeds(gDeviceState_AppLedFlash_c);
#endif
    }
}


static void APP_CoapDriveCb
(
coapSessionStatus_t sessionStatus,
void *pData,
coapSession_t *pSession,
uint32_t dataLen
)
{

	if (gCoapNonConfirmable_c == pSession->msgType)
	{


	if(FLib_MemCmp(pData, "left", 4))
	{
		shell_write("Turn LEFT");
		Drive_Left();
	}

	if(FLib_MemCmp(pData, "right", 4))
	{
		shell_write("Turn RIGHT");
		Drive_Right();
	}


	if(FLib_MemCmp(pData, "front", 5))
	{
		shell_write("Move FRONT");
		Drive_Front();
	}

	if(FLib_MemCmp(pData, "back", 4))
	{
		shell_write("Move BACK");
		Drive_Back();
	}
	if(FLib_MemCmp(pData, "stop", 4))
	{
		shell_write("Stop");
		Drive_Stop();
	}

	if(FLib_MemCmp(pData, "ledon", 5))
	{
		//App_UpdateStateLeds(gDeviceState_AppLedOn_c);
		shell_write("Turn ON LIGHT");
		Drive_LEDON();
	}

	if(FLib_MemCmp(pData, "ledoff", 6))
	{
		//App_UpdateStateLeds(gDeviceState_AppLedOn_c);
		shell_write("Turn OFF LIGHT");
		Drive_LEDOFF();
	}

	shell_writeN(pData, dataLen);
	shell_write("\r\n");
	}
}

MQTT

import uuid
import paho.mqtt.client as mqtt
import requests
import json
import re

def parsedata(payload):
    global jdata,Motion_Type,Actual_Speed,Actual_Time
    jdata = json.loads(payload)
    Actual_Speed=jdata[0]['speed']
    Actual_Time=(jdata[0]['finishedAt']-jdata[0]['startedAt'])
    print "Time=",Actual_Time 
    print "Speed=",Actual_Speed
    return Motion_Type

def on_connect(client, userdata, flags, rc):
    print('Connected with result code {code}'.format(code=rc))


def on_disconnect(client, userdata, flags, rc):
    print('Disconnected with result code {code}'.format(code=rc))


def on_message(client, userdata, msg):
    #print('Msg received from topic={topic}\n{content}'.format(topic=msg.topic, content=str(msg.payload)))
    rxdata = parsedata(msg.payload)
    print rxdata

def main():
    print "Starting\n"
    client = mqtt.Client(client_id=str(uuid.uuid4()), transport='websockets')
    client.on_connect = on_connect
    client.on_message = on_message
    client.on_disconnect = on_disconnect

    client.tls_set(ca_certs=ca_cert_path)
    client.username_pw_set(mqtt_user_name, mqtt_password)

    client.connect('ns01-wss.brainium.com', 443)
    print "connected"
    client.subscribe(motions_topic)
   

    client.loop_forever()

Main

#include <raspi.h>
#include "board.h"
#include "fsl_uart.h"

#include "pin_mux.h"
#include "clock_config.h"
#include "uart.h"
#include "backlight.h"
#include "emwin_support.h"
#include "GUI.h"
#include "fsl_ili9341.h"
#include "port_interrupts.h"
#include "images.h"
#include "uidraw.h"
#include "button.h"
#include "thread.h"

/*******************************************************************************
 * Definitions
 ******************************************************************************/
#define BOARD_SW_IRQ PORTE_IRQn
#define BOARD_SW_IRQ_HANDLER PORTE_IRQHandler
#define APP_MAX_COUNT 5
#define APP_TIMER 10
#define APP_LOCK_TIMER 2500
/*******************************************************************************
 * Prototypes
 ******************************************************************************/

/*******************************************************************************
 * Variables
 ******************************************************************************/

uint8_t txbuff[] = "version\r\n";
uint8_t ifconfig[] = "ifconfig\r\n";
uint8_t AIData[4] = {0};
/* Whether the SW button is pressed */
volatile bool g_ButtonPress = false;
volatile uint8_t g_ButtonValue = 0;
volatile uint32_t g_cloud_counter=0;
volatile uint32_t g_lock_counter=0;
volatile uint8_t g_AppIndex = 0;
volatile uint8_t g_AppLocked = 0;
volatile uint8_t g_AppLockDisable = 0;
volatile uint8_t g_AppStatus[5] = {0,0,0,0,0};
volatile uint8_t g_AppValue[5] = {0,0,0,0,0};
extern uint8_t RxRpiBuffer[8];
extern volatile uint8_t rxRpiBlockReceived;
extern volatile uint8_t rxRpiDataCounter;
/* Functions required for emwin / Display */
uint32_t DSPI2_GetFreq(void)
{
    return CLOCK_GetBusClkFreq();
}


void Init_Display()
{
    Display_Connect(); /* triggers GUI_Init() */
    Backlight_SetLevel(BLIGHT_LEVEL_HIGH);

    GUI_SetBkColor(GUI_BLACK);
    GUI_SetColor(GUI_WHITE);
}



#define GUI_FONT_TITLE  GUI_Font8x18
#define GUI_FONT_NORMAL GUI_Font8x16

uint8_t Init_Backlight(void)
{
	backlight_fct_t BACKLIGHT_fct;

    BACKLIGHT_fct.connect_hw = Backlight_Connect;
    BACKLIGHT_fct.disconnect_hw = Backlight_Disconnect;
    BACKLIGHT_fct.set_level = Backlight_Set_Level;

    Backlight_Init_Driver(&BACKLIGHT_fct);

    if (Backlight_Init_Hw())
        return 1;

    return 0;
}


/*******************************************************************************
 * Code
 ******************************************************************************/
/*!
 * @brief Main function
 */
int main(void)
{

	BOARD_InitPins();
    BOARD_Boot_Clock_RUN();

    BOARD_INIT_GPIOS();
    BOARD_Init_SPI_2();

    BOARD_Init_RGB_BLIGHT();
    PORT_IRQ_EnablePortAIrq();
    PORT_IRQ_EnablePortEIrq();

    THR_UART_Init();

    RPI_UART_Init();

    Init_Backlight();

    Init_Display();
    GUI_Clear();
    GUI_SetColor(GUI_DARKMAGENTA);
    GUI_SetBkColor(GUI_WHITE);
    //GUI_SetFont(&GUI_Font24B_1);//GUI_Font32_ASCII
    GUI_SetFont(&GUI_Font20B_1);
    GUI_SetPenSize(8);
	GUI_Clear();

	GUI_BMP_Draw(bglogo_176x176, 0, 0);
    App_WaitMsec(3000);
	GUI_BMP_Draw(bg_176x176, 0, 0);
    App_WaitMsec(100);
	draw_apppage(g_AppIndex,1,0);

    while (1)
    {

    		App_WaitMsec(10);

    		g_lock_counter++;
    		if((g_lock_counter>=APP_LOCK_TIMER) && (g_AppLockDisable==0)){
    			g_AppLocked=1;
    			//GUI_DispStringAt(" Locked  ", 40, 140);
    			GUI_BMP_Draw(logo_lock_screen_128x128, 24, 40);
    			Backlight_SetLevel(BLIGHT_LEVEL_OFF);
    			g_lock_counter=0;
    		}

    		if(g_ButtonPress){


				switch(g_ButtonValue){

				case 0:

					break;

				case 1:
					if(!g_AppLocked){
						g_AppIndex++;
					}
					g_AppLocked=0;
					g_lock_counter=0;
	    			Backlight_SetLevel(BLIGHT_LEVEL_HIGH);
					if(g_AppIndex>3){
						g_AppIndex=0;
					}
					if(g_AppIndex==3) g_AppLockDisable=1;
					else g_AppLockDisable=0;

					draw_apppage(g_AppIndex,1,0);
					break;

				case 2:
					if(!g_AppLocked){
						g_lock_counter=0;
						g_AppValue[g_AppIndex]++;
						if(g_AppValue[g_AppIndex]>2)g_AppValue[g_AppIndex]=0;
						draw_textbox(g_AppIndex, g_AppValue[g_AppIndex]);
					}
					break;

				case 3:
					if(!g_AppLocked){
						rxRpiBlockReceived = true;
						rxRpiDataCounter =0;

						RPI_ClearRxBuffer();
						if(g_AppIndex==0)
						Write_Node(0, 1);

						if(g_AppIndex==1)
						Write_Node(1, 2);

						if(g_AppIndex==2){
						Write_CarNode(2, 4);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						}
					}
					break;
				case 4:
					if(!g_AppLocked){
						if(g_AppIndex==0)
						Write_Node(0, 0);

						if(g_AppIndex==1)
						Write_Node(1, 0);

						if(g_AppIndex==2){
							Write_CarNode(2, 5);
							App_WaitMsec(2000);
							Write_CarNode(2, 6);

						}
					}
				break;

				}

				g_ButtonValue = 0;
				g_ButtonPress = false;

    		}

				if(rxRpiBlockReceived){

					RPI_ReadData();
					g_cloud_counter=0;

					if((AIData[0]==1) && (g_AppLocked)){
						g_lock_counter=0;

						GUI_DrawLine(23+112, 38+16, 23+64, 38+16);
						App_WaitMsec(150);
						GUI_DrawLine(23+64, 38+16, 23+16, 38+16);
						App_WaitMsec(150);
						GUI_DrawLine(23+16, 38+16, 23+16, 38+64);
						App_WaitMsec(150);
						GUI_DrawLine(23+16, 38+64, 23+16, 38+112);
						App_WaitMsec(150);
						GUI_DrawLine(23+16, 38+112, 23+64, 38+64);
						App_WaitMsec(150);
						GUI_DrawLine(23+64, 38+64, 23+112, 38+16);
						App_WaitMsec(150);
						Backlight_SetLevel(BLIGHT_LEVEL_HIGH);
						App_WaitMsec(150);
						//GUI_DispStringAt("UnLocked ", 40, 140);

						draw_apppage(g_AppIndex,0,0);
						g_AppLocked=0;
					}




					switch(AIData[2] | (g_AppLocked<<7)){

					case 0:

						break;

					case 1:
						g_lock_counter=0;
						//GUI_DispStringAt(" TurnON  ", 40, 140);
						if(g_AppIndex==0)
							Write_Node(0, 1);
						else if(g_AppIndex==1)
							Write_Node(1, 1);
						break;

					case 2:
						g_lock_counter=0;
						//GUI_DispStringAt(" TurnOFF ", 40, 140);
						if(g_AppIndex==0)
							Write_Node(0, 0);
						else if(g_AppIndex==1)
							Write_Node(1, 0);
						break;

					case 3:
						g_lock_counter=0;
						//GUI_DispStringAt(" MoveUP  ", 40, 140);
						if(g_AppIndex==1)
						Write_Node(1, 2);
						break;
					case 4:
						g_lock_counter=0;
						//GUI_DispStringAt(" MoveDwn ", 40, 140);
						if(g_AppIndex==1)
						Write_Node(1, 1);
						break;
					case 5:
						g_lock_counter=0;
						//GUI_DispStringAt(" Left     ", 40, 140);
						Write_CarNode(2, 2);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					case 6:
						g_lock_counter=0;
						//GUI_DispStringAt(" Right   ", 40, 140);
						Write_Node(2, 3);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					case 7:
						g_lock_counter=0;
						//GUI_DispStringAt(" Front    ", 40, 140);
						Write_CarNode(2, 4);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					case 8:
						g_lock_counter=0;
						//GUI_DispStringAt(" Back    ", 40, 140);
						Write_Node(2, 5);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					}


				}


    }
}

Credits

ashokr

Hello! I’m Ashok. I’m currently working as software engineer. passionate about doing Hobbies Projects and Animations works.

Shop

Project Hub

Video

Blog

Shop home

3D printing

Single Board Computers

Development kits

Sensors

Internet of things

Robotics

Accessories

Product of the Week

Electromaker Educator

The Electromaker Show

The Electromaker Podcast