Gateway ESPNOW to MQTT over WiFi

I am bringing you my ESPNOW to MQTT bidirectional Gateway project with all the necessary explanations so that you can easily use the firmware I have written for ESP32 for your own use.

A simple Gateway that allows you to receive the signals sent by different devices with ESPNOW and forwarding them via WiFi as MQTT messages.

I developed this ESPNOW Gateway because I have several projects with ESP8266 and ESP32 and battery operation, e.g. CO2 Gadgetin which I use the ESP-NOW protocol for reduce battery consumption and increase range and autonomy.

One of the advantages of this project is that it uses a single ESP32 for both ESPNow and WiFi . Many sites say that it is not possible to use ESPNow and WiFi simultaneously and use two ESP32s connected via serial port, one for ESPNow and one for WiFi, but I I have been doing it for several months with a single ESP32 without any problems..

What is ESP-NOW?

ESP-NOW is a radio communications protocol supported by the ESP8266 and ESP32, developed by Espressif, its manufacturer, which allows you to make very fast data transmissions (and when I say very fast, I mean very fast).

Its main objective is to replace Wifi to enable much lower power consumption and thus greater autonomy in battery-powered devices by allowing data to be sent in a few milliseconds, instead of the several seconds (3 to 12 seconds) that would be required with Wifi.

In addition to its speed there is another area where it excels, and that is in the range it provides. Where a wifi communication can have a range of 25 to 100 metres, with ESP-NOW we can easily obtain between 100 and 500 metres rangeand even more, depending on the conditions in both cases.

If you don't know ESPNOW, I recommend that you read the following article in which I tell you what you should know about it:

Radio communication for ESP8266 and ESP32 with ESPNOW

Commissioning of the ESP-NOW gateway to MQTT

Getting the ESP-NOW Gateway up and running is very easy by following these instructions.

First I'll explain the basics of getting the gateway up and running and then I'll discuss various parts of the code so that you can adapt it to your needs easily.

Basically, the implementation of the gateway consists of the following steps:

• Get the source code of the gateway
• Editing a text file with your network configuration
• Write firmware to the ESP32

I am working on a series of improvements that will allow you, in many cases, to install the Gateway on your ESP32 board directly from the browser, without having to compile or install anything on your computer. (as you can already do with CO2 Gadget), and configure it from a web page served by the Gateway itself.

When these functionalities are available, you will not have to do any of the processes listed below if you do not need to configure your own data structures and use the ones that come predefined with the Gateway.

Get the source code of the gateway

The source code for the gateway can be found on GitHub, specifically at:

[IN PREPARATION IF YOU WANT TO BECOME A BETATESTER PLEASE CONTACT ME AT TELEGRAM GROUP]

Just click on the "Code" button and select "Download ZIP".

Once you have downloaded it, you will have to unzip it to extract its contents.

Edit the text file with your network configuration

To make it easier for you and to avoid having to modify the gateway code, you should put the data that is specific to your installation in a text file called "credentials.h".

The data that are specific to your installation and that you must write in this file are:

• SSID of your wifi network
• The password to access the network
• The IP address of your MQTT server
• The HOSTNAME of the gateway (name you want to give to this device in your network).

So that you don't have to start from scratch, among the files you have extracted from the ZIP you will find one called "credentials.tpl". It is a template that you can modify.

You need to copy the file "credentials.tpl" as "credentials.h" and edit it to enter your data (or rename the file "credentials.tpl" to "credentials.h" and edit it).

Compile and save the ESP-NOW Gateway firmware on the ESP32

All that remains is to compile and load the program into the ESP.

If you are using the Arduino IDE, in the "Program/Upload" menu or by pressing Ctrl+U.

If you are using PlatformIO, under "Project tasks -> Upload and Monitor".

In both cases you will first have to select your ESP32 board and the serial port where it is connected. I assume you already know how to do this and I won't go into detail here. If you don't know how to do it, you have plenty of tutorials on the internet or You can ask for help in the eMariete Telegram group..

Please note that the ESP-NOW gateway cannot be installed on a board with ESP8266 because does not allow communication via ESP-NOW and WiFi simultaneously.. You have to install it on an ESP32. Don't confuse it with the clients, who can be ESP8266 without any problem, the limitation is only in the Gateway.

Update the ESP-NOW Gateway firmware in the ESP-32

The gateway has wireless updates OTA (Over The Air). This allows you to update the firmware without having to physically connect to the device.

As this gateway is a "plug and forget" device, it is common to place it hidden or in places that are difficult to access. In these cases, the OTA update is especially useful, as we will be able to update the firmware via your wifi connection directly from the Arduino IDE (or from PlatformIO).

The ESP-NOW Gateway code (how it works)

Sometimes, especially if you are going to use the gateway with your own projects, you will have to modify the gateway code to suit your needs. The eMariete projects are directly supported and you don't need to modify anything.I invite you to do so in order to adapt it even more to your tastes and preferences.

Here are some tips to make it easier for you to modify the code and adapt it to your needs.

The data structure

In order to adapt the gateway to transmissions with different data (the same data is not sent from a thermometer as from a presence sensor or a CO2 meter) it is important that you understand the format of the messages to be sent by ESP-NOW so that you can create new message formats.

When the sensor (let's call sensor The sender that generates the data and that we want to receive at the gateway and forward via MQTT) sends data via ESP-NOW, it does so by sending a message (a set of bytes) that follows a certain structure.

For example, eMariete's ultra-low-power CO2 meter (in preparation) sends a data packet with a structure like this in each message:

  typedef struct struct_message
  {
    int id;
    float temp;
    float hum;
    uint16_t co2;
    float battery;
    int readingId;
    int command = cmdNone;
    uint16_t parameter = 0;
  } struct_message;

It is a structure containing eight facts different.

If you look closely you will see that we have several types of data:

• Transmission support data, such as id (which contains the unit number of the unit sending the data, so that the gateway can differentiate which unit the data is coming from), readingId (which is a sequential number that is incremented with each transmission and is used to let the gateway know if packets have been lost).
• Measurement data, such as temp, hum, co2 and battery (the measurement of temperature, humidity, CO2 concentration and battery voltage).
• Data of the processed commands: which command the sensor has received from the gateway and with which parameter (more on this later).

If we now wanted to add a data structure to be used by a thermometer node, which measures nothing but temperature, we could create a data structure like this:

  typedef struct struct_message
  {
    int id;
    float temp;
    float battery;
  } struct_message;

This structure has only three different pieces of data: an "id" piece of data (so that the gateway knows which sensor has sent it), a "temp" piece of data (containing the temperature) and a "battery" piece of data (containing the battery voltage of the sensor).

Could we have used the structure we used for the CO2 meters for the thermometer so we wouldn't have needed to modify the code? The answer is yes, we could have, simply by leaving the data we didn't need at 0 (hum, co2, readingId, command and parameter).

So why complicate our lives with a different data structure for the thermometer? For optimising transmission and thus reduce energy consumption and improving autonomy.

With this change we have reduced the sending of data from 22 bytes to only 10 bytes. This assumes improved sensor autonomy by about 50% (from 90 days to 180 days on the same battery. [Pending further tests and measurements to provide real data and supporting graphs].

Optimising the data structure

As we have seen in the previous example, from the data structure for the thermometer, every byte we send in the ESP-NOW message counts and will reduce the autonomy of the sensor..

It is important that messages are well optimised. For example, we could optimise the message structure of the CO2 meter in this way:

Instead of using this message format

  typedef struct struct_message
  {
    int id;
    float temp;
    float hum;
    uint16_t co2;
    float battery;
    int readingId;
    int command = cmdNone;
    uint16_t parameter = 0;
  } struct_message;

Use this one:

  typedef struct struct_message
  {
    byte id;
    int temp; // temp * 100;
    byte hum;
    uint16_t co2;
    byte battery;
  } struct_message;

The first message occupies 22 bytes, the second message occupies only 7 bytes. Less than one third of transmission timewhich can probably increase the range by 25 to 50%.

To do this, all we have done is to:

• Eliminate superfluous data that are not essential for a CO2 meter.
• Optimising data size

We have dedicated one byte instead of two to the id. With this we have reduced the number of sensors from 65536 to 256 but do you really have more than 256 sensors?

We have reduced the temperature from 4 to 2 bytes, simply multiplying it by 100 to eliminate the decimals without losing precision. In the receiver we will divide it by 100 and we will have the decimals again.

We have reduced the humidity from 4 to 2 bytes, rounding the data to eliminate the decimals that are not usually necessary when measuring humidity (and if we wanted to keep them we could do it by multiplying by 100, as we have done with the temperature, and it would still be two bytes).

We have reduced the battery from 4 to 1 byte, sending the percentage instead of the voltage (if we wanted to send the voltage in a single byte we could also: for example, we could apply in the transmitter battery = (voltage * 100) - 255 and in the receiver voltage = (battery * 100 + 255); this would allow us to transmit voltages between 2.55V and 5.1V with two decimal places using a single byte).

Another twist on the data structure

We have just seen a good example of optimisation, but we could optimise much more...

Why stick to the official Arduino or C++ data types and make 1-byte (8-bit) groups?

We could structure and optimise information at the individual bit levelThis would allow us to save a lot of space.

  typedef struct struct_message
  {
    byte id:4;
    int temp; // temp * 100;
    byte hum:7;
    uint16_t co2:11;
    byte battery:7;
  } struct_message;

Here, for example, for the id we have dedicated 4 bits, which allows us to have up to 16 sensors, which in most cases will be enough and we save 4 bits (and if we want more we can give it 5 bits and we could have 32 sensors and still save 3 bits).

With 11 bits for CO2 we could send values up to 2047 ppm, saving 5 bits (and if we want more we can give it 12 bits and we could send data up to 4095 ppm and still save 4 bits).

The important thing is that we have reduced the data sent from 56 bits (7 bytes) to only 50 (a 10% saving). From the original structure we have already reduced from 176 bits to only 50. over 70% savings!

And we still have room to optimise... what if instead of using 16 bits to send the temperature we don't send a number that is, for example, the 0.5ºC increments from -30ºC? If we dedicate 8 bits we could send the temperature from -30ºC to 98ºC with an accuracy of 0.5ºC saving 8 more bits out of 50.

What if instead of using 7 bits for the battery we don't send the 100mV increments from 3V? 12 would be 4.2V and we would only need 4 bits, saving another 4.

What if we send only one piece of data at a time? CO2 every minute, temperature every 5 minutes (why would we want it every minute under normal conditions?), humidity every 10 minutes and battery every 6 hours? We would only need to add two bits to the structure to indicate which of the four data is being sent. We would be sending messages of only 20 bits, a saving of almost 88%!

  typedef struct struct_message
  {
    byte id:4;
    int data;
  } struct_message;

As you can see, the possibilities for optimisation are enormous.

I have not optimised the data structure very much on purpose, so that it is easy for anyone to understand the code.

How to forward ESP-NOW messages to MQTT?

In order for the gateway to forward the messages it receives via ESP-NOW through MQTT, it basically has to do the following:

• Identify the received message as a message to be resent
• Extract data from ESP-NOW data structure (struct_message)
• Create the topic and the payload of the message to be sent
• Post the MQTT message

You have to take into account that this gateway has been designed to optimise the sending of non-critical data (some may be lost from time to time without problems) and to ensure that the sensors are awake in the shortest possible time. For this reason, the sending of data from the sensor to the gateway does not have any system to ensure that the data is received.

For more critical applications than I have needed, it would be necessary to establish a protocol for acknowledgement and forwarding of information.

Keep in mind that the more robust and the more options and refinements the protocol has, the more data overhead it will have, so transmissions will be longer and devices will have to stay awake longer (waiting for response messages, for example).

In my case, if I am sending temperature or CO2 data every minute, for example, I don't mind if one message or another is lost, it will be updated with the next message a minute later, it is not critical (experience has told me that very few are lost).

How to forward MQTT messages to ESP-NOW?

In order for the gateway to forward the messages it receives via MQTT through ESP-NOW to a sensor, it has to do the following:

• Subscribe to the MQTT topics that you want to forward
• When receiving an MQTT message extract the data from the payload
• With the extracted data, copy this data to the ESP-NOW data structure (struct_message).
• Send the ESP-Now message at the right moment when the sensor is awake and receiving.

As you might imagine, sending data via ESP-NOW to a sensor that normally asleep, wakes up for a few milliseconds and goes back to sleep is not easy.

The main problem is that we do not know when the sensor is awake.

Normally we do not have an accurate real-time clock on the sensor (which could help us to synchronise shipments).

The most we can do is wait for the sensor to communicate with the gateway and keep the sensor awake for a few milliseconds listening; it is immediately after receiving that the gateway can send the messageassuming that the sensor will still be awake and listening.

The system for sending data from MQTT to ESP-NOW is not perfect as it is implemented, and I hope to improve it over time:

• It is difficult to ensure that the sensor is awake and receiving at the time the gateway sends its message.
• No acknowledgement system has been implemented by the sensor to ensure that messages are not lost.
• We cannot send messages to the sensor at any time, only when it sends us a message.
• If we leave the sensor awake for too long waiting to receive a hypothetical message we will waste a lot of energy. If we leave it awake for too short a time we will have synchronisation problems.

ESP-NOW gateway upgrades and enhancements

I have tried to programme this gateway in a very simple way.The code can be understood and modified relatively easily to implement any other type of gateway, so that any hobbyist with a little bit of programming knowledge can understand the code and modify it with relative ease.

Among the gateways you could include in the gateway, you would have:

• Direct recording of data in databases InfluxDB, MySQL, MariaDB and the like.
• Sending notifications and alerts by email, Telegram, Pushoveretc.
• Integration into any control system, home automation, building automation, PLCetc.
• Any other type of RF implementation, with the necessary RF hardware (LoRa, LoRaWan, BLE, BLE Mesh, Bluetooth Smartetc.).

Improvements to the ESP-NOW data system and structure

An important improvement that I have not yet started with (because I have not needed it) would be a more flexible ESP-NOW data sending system, allowing flexible structures (that only need to be defined in the transmitter or sensor) and somehow including the MQTT configuration.

The main problem is the optimisation of the data size, to avoid too much information travelling in the ESP-NOW messages (e.g. to avoid that the sensor sends with each packet the MQTT topic and format of the MQTT payload where it should be forwarded).

I invite you to discuss the possible implementation in the eMariete Telegram group or, if you develop it yourself, make a pull request to consider including it in the public version of the gateway.