Custom IoT Part 6 - MQTT Overview

The past projects on designing a custom Internet of Things (IoT) platform have looked at what IoT device we will be using as well as a simple demonstration of an HTTP-based platform. However, in this project, we will look at the start of our “MQTT-like” platform so that we can create a powerful IoT platform for any project in the future.

Custom IoT Solutions Part 1 - Intro to the Internet of Things

Custom IoT Solutions Part 2 - ESP32 vs ESP8266

Custom IoT Solutions Part 3 - HTTP vs MQTT

Custom IoT Solutions Part 4 - Create an HTTP Post System

Custom IoT Part 5 - How to Make HTTP GET Requests

MQTT Topology

While we have looked at what MQTT is in a previous project, to refresh our memories we will briefly look at it again. MQTT stands for Message Queue Telemetry Transport and is a protocol that defines how clients and servers connect, send messages, and behave. Unlike HTTP, MQTT clients connect to a server and the connection is kept alive for as long as the client wants. Clients can subscribe to a variable whose value is sent to them by the server whenever it changes on the server and clients can publish values to a variable on the server (any other clients who are subscribed to that variable will be sent the latest value).

However, it should be stated that our platform will implement a custom protocol that will behave somewhat like MQTT but will have added features as well as using text-based messages instead of binary based messages. Text-based messaging makes it easier to error correct as well as distinguish commands from data. One feature that we will be adding, in particular, is device to device messaging whereby one client can send a message to another client using that device's unique ID.

Our Protocol

So, before we can look at the structure of the server and how devices will communicate we first need to establish our protocol and its structure.

1. Connect – A client device initiates a connection request to the server
2. Authorize – The server can choose to either accept or deny this request. This could be done using a unique ID or a physical push button that a server operator has to push
3. Connected – From here, messages can be sent to and from the client/server

All messages between the client and server will be in ASCII plaintext for both commands and data. This means that not only is error correction simpler, but we can also be sure about entire messages being sent and received correctly as well as utilizing ASCII control characters (line feeds for example) as terminators to lines. For our protocol, we will use the following command structure:

START | COMMAND=CMD | END

All messages start with the word START and end with END which useful to ensure that a packet has been received correctly. All parameters are separated by a ‘|’ character (this is not the letter I but a vertical line) and the first parameter is always a command. Depending on the command other parameters will need to be passed such as variable name and data value.

Client to server messages

These are the messages that our client devices can send to the server.

Server to client messages

These are the messages that the server will send to the client device.

Server Code

Our server could be written in a wide range of languages such as C++, C#, and Java, but the language that we will be using will be Python. While Python is not as fast as compiled languages, it has some major advantages including cross-platform support, simplicity, and library with code example support. Socket programming in Python is far easier than that of compiled languages and the ability to write multi-threaded applications means that each client device connected to our server can run in its own thread. This is not to say that our server can’t be written in C++ but doing so would add complexity as well as operating system specifics. The code that we will be creating will be able to run on Windows, Mac, and Linux which incorporates single-board computers such as the Raspberry Pi.

Our server will be constructed using the following layout:

• Main code loop
  • This main loop will open a port and allow devices to connect. Once a device has connected, it will be sent to its own class object that runs its own thread
• Client thread
  • This client thread will handle one specific client. All requests coming in are processed and appropriate responses sent back to the client
  • Values that are published to are stored in a global text file (whose name is the variable name) which is accessible to all other client threads. The date and time which the variable was published is also stored in the text file
  • All client threads store the list of subscribed variables, their latest value, and the time at which these values were obtained. If a new entry is found then it is sent to the connected client so that they can receive the latest value

Conclusion

So now that we have seen how our server will work, the message structure, and how the code will be built the next task is to build the thing! In the next project we will get our main loop code working in Python as well as client threads to accept incoming connection requests as well as some basic connection code on the ESP32.