ESPNOW to MQTT Gateway over WiFi

Last modified 12 months

I bring you my ESPNOW to MQTT bidirectional Gateway project with all the necessary explanations so you can use the firmware I have written for ESP32 for your own use easily.

A simple Gateway that allows you to receive the signals sent by different devices with ESPNOW and forward them as MQTT messages via WiFi.

I developed this ESPNOW Gateway because I have several projects with ESP8266 and ESP32 and battery operation, for example CO2 Gadget,in which I use the ESP-NOW protocol for reduce battery consumption and increase autonomy and range.

One of the advantages of this project is that it utilizes a single ESP32 for both ESPNow and WiFi . Many sites say that it is not possible to use ESPNow and WiFi simultaneously and they use two ESP32 connected through serial port, one for ESPNow and one for WiFi, but I I've been doing it for several months with a single ESP32 without any problem..

GATEWAY ESPNOW FIRMWARE IN PREPARATION

The Gateway ESPNOW firmware will be available soon. I'm still working on it, so that anyone can use it with CO2 Gadget, and is still receiving many changes and functionalities.

If you want to participate in the beta testing, please let me know through the Telegram group.

What is ESP-NOW?

ESP-NOW is a radio communications protocol supported by the ESP8266 and ESP32, developed by Espressif, its manufacturer, which allows you to make very fast data transmissions (and when I say very fast, I mean very fast).

Its main objective is to replace Wifi to enable much lower power consumption and thus greater autonomy in battery-powered devices by allowing data to be sent in a few milliseconds, instead of the several seconds (from 3 to 12 seconds) that would be required for Wifi.

In addition to its speed there is another area where it excels, and that is in the range it provides. Where a wifi communication can have a range of 25 to 100 meters, with ESP-NOW we can easily obtain between 100 and 500 meters range,and even more, depending on the conditions in both cases.

If you don't know ESPNOW I recommend you to read the following article in which I tell you what you should know about it:

Setup of the ESP-NOW to MQTT Gateway

Getting the ESP-NOW Gateway up and running is very easy by following these instructions.

First I'll explain the basics of getting the gateway up and running and then I'll discuss various parts of the code so that you can adapt it to your needs easily.

Basically, the gateway setup consists of the following steps:

  • Get the gateway source code
  • Editing a text file with your network configuration
  • Save the firmware in the ESP32

I am working on a series of improvements that will allow you, in many cases, to install the Gateway on your ESP32 board directly from the browser, without having to compile or install anything on your computer (as you can already do with CO2 Gadget,), and configure it from a web page served by the Gateway itself.

When these functionalities are available, you will not have to do any of the processes listed below if you do not need to configure your own data structures and use the ones that come predefined with the Gateway.

Get the gateway source code

The source code of the gateway can be found on GitHub, specifically at:

[IN PREPARATION IF YOU WANT TO BE BETATESTER CONTACT ME AT TELEGRAM GROUP]

Just click on the "Code" button and select "Download ZIP".

Once you have downloaded it, you will have to unzip it to extract its content.

Edit text file with your network configuration

To make it easier for you and not having to modify the gateway code, you should put the data that is specific to your installation in a text file called "credentials.h".

The data that are specific to your installation and that you will have to write in this file are:

  • SSID of your wifi network
  • The password to access the network
  • The IP address of your MQTT server
  • The HOSTNAME of the gateway (name you want to give to this device in your network).

So that you don't have to start from scratch, among the files you have extracted from the ZIP you will find one called "credentials.tpl". It is a template that you can modify.

You have to copy the file "credentials.tpl" as "credentials.h" and edit it to enter your data (or rename the file "credentials.tpl" to "credentials.h" and edit it).

Compile and save ESP-NOW Gateway firmware on the ESP32

All that remains is to compile and load the program into the ESP.

If you are using Arduino IDE, in the "Program/Upload" menu or by pressing Ctrl+U.

If you are using PlatformIO, in "Project tasks -> Upload and Monitor".

In both cases you will first have to select your ESP32 board and the serial port where it is connected. I asume that you already know how to do this, and I won't go into the process in detail. If you don't know how to do it, you have plenty of tutorials on the internet or you can ask for help in eMariete's Telegram group..

Please note that the ESP-NOW gateway cannot be installed on a board with ESP8266 because it does not allow communication via ESP-NOW and WiFi simultaneously. You have to install it on an ESP32. Do not confuse it with the clients, who can be ESP8266 without any problem, the limitation is only for the Gateway.

Update ESP-NOW Gateway firmware in the ESP-32

The gateway supports wireless updates OTA (Over The Air). This allows you to update the firmware without having to physically connect to the device.

As this gateway is a "plug and forget" device, it is common to place it hidden or in places with difficult access. In these cases, the OTA update is especially useful, as we will be able to update firmware via your wifi connection directly from the Arduino IDE (or from PlatformIO).

The ESP-NOW Gateway code (how it works)

Sometimes, especially if you will be using the gateway with your own projects, you will have to modify the gateway code to suit your needs. The eMariete projects are directly supported and you do not need to modify anything.However, I invite you to do so in order to adapt it even more to your tastes and preferences.

Here are some hints to make it easier for you to modify the code and adapt it to your needs.

The data structure

In order to adapt the gateway to transmissions with different data (data is not sent in the same way from a thermometer as from a presence sensor or a CO2 meter) it is important that you understand the format of the messages sent by ESP-NOW so that you can create new message formats.

When the SCD30 (let's call SCD30 to the sender that generates the data and that we want to receive at the gateway and forward via MQTT) sends data via ESP-NOW, it does so by sending a message (a set of bytes) that follows a certain structure.

For example, eMariete's ultra-low-power CO2 meter (in preparation) sends a data packet with a structure like this in each message:

  typedef struct struct_message
  {
    int id;
    float temp;
    float hum;
    uint16_t co2;
    float battery;
    int readingId;
    int command = cmdNone;
    uint16_t parameter = 0;
  } struct_message;

It is a structure containing eight pieces of information different.

If you look closely you will see that we have several types of data:

  • Transmission support data, such as id (which contains the number of the unit sending the data, so that the gateway can differentiate which unit the data is coming from), readingId (which is a sequential number that is incremented with each transmission and is used to let the gateway know if packets have been lost).
  • Measurement data, such as temp, hum, co2 and battery (measurement of temperature, humidity, CO2 concentration and battery voltage).
  • Data on processed commands: which command the sensor has received from the gateway and with which parameter (more on this later).

If we now wanted to add a data structure to be used by a thermometer node, which measures nothing but temperature, we could create a data structure like this:

  typedef struct struct_message
  {
    int id;
    float temp;
    float battery;
  } struct_message;

This structure has only three different pieces of data: an "id" piece of data (so that the gateway knows which sensor has sent it), a "temp" piece of data (containing the temperature) and a "battery" piece of data (containing the sensor's battery voltage).

Could we have used the structure we used for the CO2 meters for the thermometer and thus we would not have needed to modify the code? The answer is yes, we could have, simply leaving the data we did not need at 0 (hum, co2, readingId, command and parameter).

So why complicate our lives with a different data structure for the thermometer? To optimize transmission and thus reduce energy consumption and improve autonomy.

With this change we have reduced the sending of data from 22 bytes to only 10 bytes. This assumes improved sensor autonomy by about 50% (from 90 days to 180 days with the same battery. [Pending new tests and measurements to put real data and graphs to support them].

Optimizing the data structure

As we have seen in the previous example, from the data structure for the thermometer, every byte we send in the ESP-NOW message counts and will reduce the sensor's autonomy.

It is important that the messages are well optimized. For example, we could optimize the message structure of the CO2 meter in this way:

Instead of using this message format

  typedef struct struct_message
  {
    int id;
    float temp;
    float hum;
    uint16_t co2;
    float battery;
    int readingId;
    int command = cmdNone;
    uint16_t parameter = 0;
  } struct_message;

Use this other one:

  typedef struct struct_message
  {
    byte id;
    int temp; // temp * 100;
    byte hum;
    uint16_t co2;
    byte battery;
  } struct_message;

The first message occupies 22 bytes, the second occupies only 7 bytes. Less than one third of the transmission timewhich can probably increase the autonomy by 25 to 50%.

To do this, all we have done is:

  • Eliminate superfluous data that are not essential for a CO2 meter
  • Optimize data size

We have dedicated one byte instead of two to the id. With this we have reduced the number of sensors from 65536 to 256 but do you really have more than 256 sensors?

We have reduced the temperature from 4 to 2 bytes, simply multiplying it by 100 to eliminate the decimals without losing precision. In the receiver we will divide it by 100 and we will have the decimals again.

We have reduced the humidity from 4 to 2 bytes, rounding the data to eliminate the decimals that are not usually necessary when measuring humidity (and if we wanted to keep them we could do it by multiplying by 100, as we have done with the temperature, and they would still be two bytes).

We have reduced the battery from 4 to 1 byte, sending the percentage instead of the voltage (if we wanted to send the voltage in a single byte we could also: for example, we could apply in the transmitter battery = (voltage * 100) - 255 and in the receiver voltage = (battery * 100 + 255); this would allow us to transmit voltages between 2.55V and 5.1V with two decimal places using a single byte).

Another twist on the data structure

We have just seen a good example of optimization, but we could optimize much more...

Why stick to the official Arduino or C++ data types and make 1-byte (8-bit) groups?

We could structure and optimize information at the individual bit levelThis would allow us to save a lot of space.

  typedef struct struct_message
  {
    byte id:4;
    int temp; // temp * 100;
    byte hum:7;
    uint16_t co2:11;
    byte battery:7;
  } struct_message;

Here, for example, for the id we have dedicated 4 bits, which allows us to have up to 16 sensors, which in most cases will be enough and we save 4 bits (and if we want more we can give it 5 bits and we could have 32 sensors and still save 3 bits).

With 11 bits for CO2 we could send values up to 2047 ppm, saving 5 bits (and if we want more we can give it 12 bits and we could send data up to 4095 ppm and still save 4 bits).

The important thing is that we have reduced the data sent from 56 bits (7 bytes) to only 50 (a 10% saving). From the original structure we have already reduced from 176 bits to only 50 bits. over a 70% savings!

And we still have room to optimize... what if instead of using 16 bits to send the temperature we do not send a number that is, for example, the increments of 0.5ºC from -30ºC? If we dedicate 8 bits we could send the temperature from -30ºC to 98ºC with an accuracy of 0.5ºC saving 8 bits more than the 50.

What if instead of using 7 bits for the battery we don't send the 100mV increments from 3V? the 12 would be 4.2V and we would only need 4 bits, saving another 4.

What if we send only one data each time? CO2 every minute, Temperature every 5 minutes (why would we want it every minute under normal conditions?), humidity every 10 minutes and battery every 6 hours? We would only need to add two bits to the structure to indicate which of the four data is being sent. We would be sending messages of only 20 bits, a saving of almost 88%!

  typedef struct struct_message
  {
    byte id:4;
    int data;
  } struct_message;

As you can see, the optimization possibilities are enormous.

I have not optimized the data structure much on purpose, to make it easy for anyone to understand the code.

How to forward ESP-NOW messages to MQTT?

For the gateway to forward through MQTT the messages it receives through ESP-NOW, it basically has to do the following:

  • Identify the received message as a message to be forwarded
  • Extracting data from the ESP-NOW data structure (struct_message)
  • Create the topic and the payload of the message to be sent
  • Post the MQTT message

You have to take into account that this gateway has been designed to optimize the sending of non-critical data (some may be lost from time to time without problems) and to ensure that the sensors are awake in the shortest possible time, for this reason the sending of data from the sensor to the gateway does not have any system to ensure that the data is received.

For more critical applications than I have needed, it would be necessary to establish a protocol for acknowledging and forwarding the information.

Keep in mind that the more robust and the more options and refinements the protocol has, the more data overhead it will have, so transmissions will be longer and devices will have to stay awake longer (waiting for response messages, for example).

In my case, if I am sending temperature or CO2 data every minute, for example, I don't mind if some message gets lost, it will be updated with the next message a minute later, it is not critical (experience has told me that very few are lost).

How to forward MQTT messages to ESP-NOW?

In order for the gateway to forward the messages it receives via MQTT through ESP-NOW to a sensor, it has to do the following:

  • Subscribe to the MQTT topics you want to forward
  • When receiving an MQTT message extract payload data
  • With the extracted data, copy this data to the ESP-NOW data structure (struct_message).
  • Send the ESP-Now message at the exact moment when the sensor is awake and receiving.

As you might imagine, sending data via ESP-NOW to a sensor that normally asleep, wakes up for a few milliseconds and goes back to sleep is not easy.

The main problem is that we do not know when the sensor is awake.

Normally we do not have an accurate real-time clock on the sensor (which could help us to synchronize shipments).

The most we can do is wait for the sensor to communicate with the gateway and keep the sensor awake for a few milliseconds listening; it is immediately after receiving when the gateway can send the messageassuming that the sensor will still be awake and listening.

The system for sending MQTT data to ESP-NOW is not perfect as implemented, and I hope to improve it over time:

  • It is difficult to ensure that the sensor is awake and receiving at the time the gateway sends its message.
  • No acknowledgement system has been implemented by the sensor to ensure that messages are not lost.
  • We cannot send messages to the sensor at any time, only when it sends us a message.
  • If we leave the sensor awake too long waiting to receive a hypothetical message we will waste a lot of energy. If we leave it awake too little time we will have synchronization problems.

ESP-NOW gateway upgrades and enhancements

I have tried to program this gateway in a very simple way.The code can be easily modified to implement any other type of gateway, so that any hobbyist with a little bit of programming knowledge can understand the code and modify it with relative ease.

Among the gateways you could include in the gateway, you would have:

  • Direct recording of data in DB InfluxDB, MySQL, MariaDB and the like.
  • Sending notifications and alerts through email, Telegram, Pushover, etc.
  • Integration in any control system, home automation, building automation, PLC, etc.
  • Any other type of RF implementation, with the necessary RF hardware (LoRa, LoRaWan, BLE, BLE Mesh, Bluetooth Smart, etc.).

Improvements to the ESP-NOW data system and structure

A major improvement that I haven't gotten around to yet (because I haven't needed it) would be a more flexible ESP-NOW data sending system, allowing flexible structures (that only need to be defined at the transmitter or sensor) and somehow including MQTT configuration.

The main problem is the optimization of the data size, to avoid that in the ESP-NOW messages too much information travels (for example, to avoid that the sensor sends with each packet the MQTT topic and format of the MQTT payload where it should be forwarded).

I invite you to discuss the possible implementation in the eMariete Telegram group or, if you develop it on your own, make a pull request to consider including it in the public version of the gateway.

How useful was this post?

Click on a star to rate it!

Average rating 5 / 5. Vote count: 7

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

3 thoughts on “Gateway ESPNOW a MQTT por WiFi”

  1. Hello, I would like to thank you for an excellent article and good explanations of efficient data formats. I have done some similar work and I freqently use 'bitfields' to store system flags for example. Bit fileds could also be used to make very compact data structurs where only 4 or 5 bits of data can represent, for example, a temperature.
    I would be very keen to do some beta testing of your code if it is available. I am only a hobbyist programmer so my interest is only personal. If the code is available could you advise me where it can be downloaded.
    Thanks
    Bob

    Reply

Leave a Comment