Visual Gesture Controlled Iot Car

Custom PCB on your Way!

Getting Started

OpenCV and MediaPipe

Hand Tracking and Camera Frame UI

python -m pip install opencv-python mediapipe requests numpy

Code - Software

mp_hands = mp.solutions.hands
hands = mp_hands.Hands()
mp_drawing = mp.solutions.drawing_utils

Initializing Variables

tracking = False
hand_y = 0
hand_x = 0
prev_dir = ""
URL = "http://projectalmanac.local/"

In this step, we initialize several variables that will be used to keep track of hand-related information and the previous direction.

A URL is defined to send HTTP requests to the hardware code of ca

Defining a Function to Send HTTP Requests

def send(link):
    try:
        response = requests.get(link)
        print("Response ->", response)
    except Exception as e:
        print(f"Error sending HTTP request: {e}")

This step defines a function named send that takes a link as an argument and sends an HTTP GET request to the specified URL. It prints the response or an error message if the request fails.

These are the initial setup steps. The following steps are part of the main loop where video frames are processed for hand tracking and gesture recognition. I'll explain these steps one by one:

MediaPipe Hands Processing

ret, frame = cap.read()
frame = cv2.flip(frame, 1)
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = hands.process(rgb_frame)

Inside the loop, it captures a frame from the camera (cap.read()) and flips it horizontally (cv2.flip) to mirror the image.

The code converts the captured frame to RGB format (cv2.cvtColor) and then uses the MediaPipe Hands module to process the frame (hands.process) for hand landmark detection. The results are stored in the results variable.

Hand Landmarks and Tracking

if results.multi_hand_landmarks:
    hand_landmarks = results.multi_hand_landmarks[0]
    index_finger_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
    hand_y = int(index_finger_tip.y * height)
    hand_x = int(index_finger_tip.x * width)
    tracking = True

This section checks if hand landmarks are detected in the frame (results.multi_hand_landmarks). If so, it assumes there's only one hand in the frame and extracts the y-coordinate of the index finger tip. It updates hand_y and hand_x with the calculated coordinates and sets tracking to True.

Direction Calculation

frame_center = (width // 2, height // 2)
if trackin
    direction = find_direction(frame, hand_y, hand_x, frame_center)
if(direction != prev_dir):
                try:
                    link = URL+direction
                    http_thread = threading.Thread(target=send, args=(link,))
                    http_thread.start()
                except Exception as e:
                    print(e)
                prev_dir = direction
                print(direction)

In this step, the code calculates the center of the frame and, if tracking is active, it uses the find_direction function to calculate the direction based on the hand's position. The direction is stored in the direction variable.

We used current direction and previous direction variables. It helps in keeping a semaphore of sending only one HTTP request for every change in command. Then overall store it in a single URL to send the HTTP request.

Visualization

opacity = 0.8
cv2.addWeighted(black_background, opacity, frame, 1 - opacity, 0, frame)
cv2.imshow("Project Almanac", frame)

If tracking is active, this section of the code adds visual elements to the frame, including a filled circle representing the index finger tip's position and text indicating the detected direction.

The code blends a black background with the original frame to create an overlay with adjusted opacity. The resulting frame is displayed in a window named "Project Almanac".

Now that we are done with the software side code, let us look into the software side code -

Importing Libraries:

#include <WiFi.h>
#include <ESPmDNS.h>
#include <WebServer.h>

In this section, the code includes necessary libraries for WiFi communication (WiFi.h), setting up mDNS (ESPmDNS) for local network naming, and creating a web server using the WebServer library.

Defining Pin Constants:

int LeftA = 33;   // IN1
int LeftB = 25;   // IN2
int RightA = 26;  // IN3
int RightB = 27;  // IN4

Here, the code defines constants for pin numbers corresponding to motor control pins (presumably for a robotic project). These pins will control the movement of motors.

Setting Up Wi-Fi Credentials:

const char* ssid = " ";      // Enter SSID here
const char* password = " ";  // Enter Password here

You need to fill in your Wi-Fi network's SSID and password here to connect the ESP8266 device to your local Wi-Fi network.

Configuring Motor Control Pins:

pinMode(LeftA, OUTPUT);
  pinMode(LeftB, OUTPUT);
  pinMode(RightA, OUTPUT);
  pinMode(RightB, OUTPUT);
  pinMode(2, OUTPUT);

In this part, the code sets the motor control pins (LeftA, LeftB, RightA, RightB) as OUTPUT pins, presumably to control motors for a robotic project. It also sets pin 2 as an OUTPUT, possibly for controlling an indicator LED.

Connecting to Wi-Fi:

Serial.begin(115200);
  delay(100);
  Serial.println("Connecting to ");
  Serial.println(ssid);

  // Connect to your local Wi-Fi network
  WiFi.begin(ssid, password);

  // Check if the device is connected to the Wi-Fi network
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.print(".");
  }

  // Display connection status and IP address
  Serial.println("");
  Serial.println("WiFi connected..!");
  Serial.print("Got IP: ");  
  Serial.println(WiFi.localIP());

  digitalWrite(2, HIGH); // Turn on a blue LED to indicate a connected WiFi

This part of the code initiates a connection to the specified Wi-Fi network using the provided SSID and password. It waits until the device successfully connects to the Wi-Fi network and then displays the IP address. Additionally, it turns on an LED on pin 2 to indicate a successful connection.

Setting up mDNS (Multicast DNS):

if (!MDNS.begin("projectalmanac")) {
    Serial.println("Error setting up MDNS responder!");
    while(1) {
      delay(1000);
    }
  }
  Serial.println("mDNS responder started");

Here, the code sets up mDNS with the hostname "projectalmanac." This allows the device to be reachable on the local network using the hostname instead of an IP address.

Defining HTTP Server Endpoints:

server.on("/", handle_OnConnect);
  server.on("/left", left);
  server.on("/right", right);
  server.on("/forward", forward);
  server.on("/backward", backward);
  server.on("/stop", halt);
  server.onNotFound(handle_NotFound);

This part defines different HTTP server endpoints that can be accessed via URLs. For example, "/left" will trigger the left function when accessed.

Starting the Web Server:

server.begin();
  Serial.println("HTTP server started");
  MDNS.addService("http", "tcp", 80);
}

The code starts the web server, making it available for handling HTTP requests on port 80. It also registers the HTTP service with mDNS.

Handling Client Requests:

void loop() {
  server.handleClient();
}

In the loop function, the server continuously handles client requests, responding to various endpoints defined earlier.

HTTP Request Handling Functions:The code defines several functions (forward, backward, left, right, halt, handle_OnConnect, handle_NotFound) that are called when specific endpoints are accessed. These functions are responsible for controlling motors and responding to client requests. The HTML page provides information about available commands and instructions for interacting with the device.

Now that we have understood the code sequence, let us see the work!

We can further add more features if you'd like to. Rest, the UI is simple enough to handle, which comes with not many features, but important one's.