Build Multi-Protocol Gateway With Portenta X8 & Max Carrier

This tutorial shows how to setup a multi-protocol gateway environment on Portenta X8 using Max Carrier

Overview

Portenta X8 has the NXP® i.MX 8M Mini MPU (Linux) and STM32H747XI dual Cortex®-M7+M4 32bit low power ARM® MCU (Arduino) stacked together and can be used to design different work loads for these two different microprocessors. We will use the Portenta Max Carriers onboard CMWX1ZZABZ-078 LoRaWAN® module from Murata® and the Wi-Fi connectivity from Portenta X8 to build a Multi-Protocol Gateway.

In this tutorial we will go through the steps on how to setup both the Linux and Arduino side. A device collecting sensor data will transfer the data via Wi-Fi, receive the data and exchange them between the Arduino and Linux layers, for finally using LoRaWAN® to send the information to The Things Network. We will also configure and expose a local communication lane to further expand its capability if a local sensor is desired.

Goals

  • Build an Arduino layer script using RPC to handle sensor readings or data traffic on Portenta X8.
  • Build a multi-protocol script to manage MQTT protocol and LoRa® connectivity to handle data traffic on the Linux layer of Portenta X8.
  • Assemble both layer scripts to build an operational multi-protocol gateway using Portenta X8 and Max Carrier.

Required Hardware and Software

Multi-Protocol Gateway 101

A gateway is a network node and a key-point for data exchange between different networks under certain given specifications. Simply referred to as hardware that communicates between two networks. On the other hand, a multi-protocol gateway goes one step further by implementing variety of protocols in a single gateway.

The idea of Multi-Protocol Gateway is to build a device that will establish a information relay, that handles incoming and outgoing traffic of data using different connectivity protocols.

This means that the gateway can receive the data transmitted in certain protocol type and relay the data in a different protocol for a remote server. Such feature provides the ability to develop distinctive types of protocols and relay the data with less complexity.

The Portenta X8 paired with the Portenta Max Carrier has the potential to create synergy, and you will have following connectivity tools at your disposal:

  • Wi-Fi (MQTT Protocol)
  • Bluetooth® Low Energy
  • LoRaWAN® (The Things Network)
  • NB-IoT & Cat-M1

The following image illustrates how a overall multi-protocol gateway works with Portenta X8 and Max Carrier as the gateway system.

Multi-Protocol Gateway General Architecture Overview
Multi-Protocol Gateway General Architecture Overview

Bear in mind, that this tutorial focuses on developing a multi-protocol gateway using some connectivity modules. Yet, this Portenta combination still has much to offer. To get the most out of it, we will go step by step on how to establish the multi-protocol gateway and add scalability to expand its capability.

Foremost, you will get to know how the multi-protocol gateway will be implemented with Portenta X8 paired with Max Carrier. Some other tutorials will be referenced to guide you through the present tutorial, as it involves mechanisms that are extensive to cover.

Arduino Layer

The Arduino layer is extended within the M4 Core and it is the layer dedicated to the development of real time operations. Thus, you can use the Arduino layer to perform PID tasks and make the RPC calls to exchange data with the Linux layer.

To learn about how to exchange data using RPC between Arduino and Linux layer, please read "Data Exchange Between Python on Linux and an Arduino Sketch"

We will go through how to use RPC to expose data received from the Arduino layer to the Linux layer, if further development requires you to feed data to devices interfaced communicating with the M4 core. We will leave the tasks running and open to be interfaced with for expanding the capability of the Portenta X8 and Max Carrier. It will let you develop a gateway system where:

  1. The Arduino layer will be the terminal to expose the received sensor data to control a local end-device.
  2. A local end-device transferring data to the Linux layer for further networking process.

Hence the multi-protocol architecture will process and manage the data traffic with the desired protocol.

Linux Layer

It is important to understand that all networking processes are made within the Linux layer. All network processes that are Wi-Fi, Bluetooth® Low Energy, LoRa®, NB-IoT and Cat. M1. We will focus on using Wi-Fi with MQTT protocol and LoRa® connectivities to establish a multiple protocol gateway.

The Portenta X8 provides Wi-Fi connectivity and the Portenta Max Carrier provides a LoRaWAN® module that can help us communicate with The Things Network. We will use the MQTT protocol to receive sensor data transmitted by an end device.

We will use a Python script that will configure and handle the connectivity modules and its sensor data. The RPC calls will be used to expose the received sensor data to the Arduino layer, setting up data exchange configuration to further expand the capability of the Portenta X8 and Max Carrier. The process can also be done vice-versa and makes use of the Arduino layer to transmit the data to the Linux layer from the local end-device.

Now that we know the roles of Arduino and Linux layer, we will need a clear picture on how the multi-protocol gateway should look. The next diagram illustrates the in-depth multi-protocol gateway architecture, showing how each layer and module will cooperate.

Multi-Protocol Gateway In-Depth Architecture
Multi-Protocol Gateway In-Depth Architecture

Instructions

To showcase the ability of the Linux layer and Arduino layer extended by M4 Core, we will build a multi-protocol gateway that will receive MQTT protocol packages using EMQX as the broker, retrieve the data from a local sensor attached to Portenta via RPC mechanism, wrap the data together and send to The Things Netowork using LoRa® connectivity within Cayenne Low Power Payload Encoder.

This will help you get a better understanding on how the Portenta X8 and the Portenta Max Carrier can help you develop a multi-protocol gateway.

Hardware Setup

First things first, you will need to configure the hardware to be able to develop and work on the Multi-Protocol gateway. We will attach the Portenta X8 to the Portenta Max Carrier with High-Density Connectors and you will have to make sure to attach an antenna for LoRa® connectvity. The Portenta X8 also needs to have the Wi-Fi antenna attached to it.

Multi-Protocol Gateway Hardware Setup
Multi-Protocol Gateway Hardware Setup

If you have not set up your Portenta X8, please have a look at Portenta X8 Getting Started tutorial.

Setting Up The Portenta X8

Before you begin diving deep into creating Multi-protocol gateway, and having understood that you will frequently communicate between Arduino and Linux layers, you will have to understand how to debug and observe the way these 2 layers interact.

The

m4-proxy
is a service that manages data exchange between these layers. You can use the following command in the terminal to observe if the service is running correctly.

1sudo journalctl -fu m4-proxy

Now you are going to implement RPC (Remote Procedure Call) to establish communication between the Arduino and Linux layers. This is a communication mechanism developed to exchange data between these two layers.

A very important note to take into account: you will not be able to check messages via

Serial.println()
statements to check if the Arduino sketch is running in a desired manner. You will have to use
py-serialrpc
, which is a service that will assist you in listening to those messages, printing them on a console. To have the service active, please download this compressed file to build and run the container on the Linux side of Portenta X8. Please execute following commands in order to have the service running.

1// Copy the decompressed files in ../adb/32.0.0
2adb push py-serialrpc /home/fio
3adb shell
4
5sudo su -
6
7// Head to directory and mount the container
8cd /home/fio/py-serialrpc
9#py-serialrpc sudo docker build . -t py-serialrpc
10#py-serialrpc sudo docker-compose up -d

To access the logs of

py-serialrpc
service, while maintaining the same directory, execute the following command.

1sudo docker-compose logs -f --tail 20

For more details about how data exchange between Arduino and Linux layer works and to understand how to debug, please read Data Exchange Between Python on Linux and an Arduino Sketch

In case you have not configured internal Wi-Fi connectivity within the system, please use following command line.

1nmcli device Wi-Fi connect "SSID" password "PASSWORD"

Setting Up The Things Network

You now have the pre-requisites for the Portenta X8 ready, but, since you are using the LoRa® connectivity, you will need a platform that has the capability to receive data transmitted from the Portenta X8 and Max Carrier. The Things Network will be the platform you are going to use communicate using LoRaWAN®. On the platform, you will need to create an application to add the Portenta Max Carrier as an End-Device.

When adding the End-Device, at the moment you will have to use the Manual option. The Portenta Max Carrier will be added under Arduino SA in the near future to be included in the LoRaWAN® Device Repository. The LoRaWAN® version and parameters compatible with the Portenta Max Carrier are as follows. The frequency plan will depend on the region in which you are going install the device.

General End-Device Configuration
General End-Device Configuration

To learn more about LoRa® and LoRaWAN®, please have a look at our Arduino Guide to LoRa® and LoRaWAN®. Additionally, if you wish to learn on how to properly setup the End-Device in The Things Network, please read this tutorial reference

Let's now dive into developing a multi-protocol gateway using Portenta X8 and Max Carrier!

Building the Multi-Protocol Gateway

It is important to put all the requirements into an operational task that will orchestrate every protocol you are going to use. You will have to create the following files and the required codes for your multi-protocol gateway.

You will need the Docker files that will configure and let you build a working container.

If you are unfamiliar handling with Docker and containers, please read the tutorial on how to Create and Upload a Custom Container to the Portenta X8

You can access the files here. Meanwhile, let's take a look at some of the important details of the included files.

Docker Compose

Beginning with the

docker-compose.yml
file, which is where you define permissions and settings for the involved container.

1...
2extra_hosts:
3 - 'm4-proxy:host-gateway'
4devices:
5 - '/dev/ttymxc3'
6 - '/dev/gpiochip5'
7tty: true
8user: "0"

Requirements

Here you will define which additional components are required to be able to run the script built inside the container. If you decide to further develop with different protocol, you will have to add the package in order to be able to use them for development.

1msgpack-rpc-python
2pyserial==3.4
3python-periphery==2.3.0
4python-dotenv
5pycayennelpp
6paho-mqtt

Multi-Protocol Python Application

This is the main Python script that will handle overall networking process. We will highlight important fragments of the code to help you understand how these codes pieces work together to build a gateway based on multiple protocols. For full Python script please refer to the files here.

First up, the configuration for the M4 Proxy Server, which are the parameters that handles communication with the M4 core that extends the Arduino layer. The

m4_proxy_port
is configured to
5001
, as it is the port used by clients to send the data to the M4.

1#M4 Proxy Server Configuration
2# Fixed configuration parameters
3port = 8884
4publish_interval = 5
5
6# The M4 Proxy address needs to be mapped via Docker's extra hosts
7m4_proxy_address = 'm4-proxy'
8m4_proxy_port = 5001

The next function is dedicated to retrieve data from the M4 (Arduino layer). It will help you set the variables, such as sensor data, to then be pulled and be exposed to the Linux layer. With this, you will have the information available to be used within the Python script.

1def get_data_from_m4():
2
3 rpc_address = RpcAddress(m4_proxy_address, m4_proxy_port)
4
5 data = ()
6
7 try:
8 rpc_client = RpcClient(rpc_address)
9 rpc_data0 = rpc_client.call('Data_0')
10
11 rpc_client = RpcClient(rpc_address)
12 rpc_data1 = rpc_client.call('Data_1')
13
14 data = rpc_data0, rpc_data1
15
16 except RpcError.TimeoutError:
17 print("Unable to retrieve data from the M4.")
18
19 return data

For MQTT configuration, you will need to set the desired parameters. Below you can find the parameters we use for MQTT in this tutorial.

1mqtt_broker = 'broker.emqx.io'
2mqtt_port = 1883
3mqtt_topic = "multiPrGw/mqtt1"
4# generate client ID with pub prefix randomly
5mqtt_client_id = f'python-mqtt-{random.randint(0, 100)}'
6mqtt_username = 'emqx'
7mqtt_password = 'public'

These 2 parameters are required to establish a connection with The Things Network. The

APP_EUI
and
APP_KEY
are required to be configured, as they are provided from The Things Network or from the LoRaWAN® platform that you may try extablishing connection to. Additionally, the
DEV_EUI
will be predefined as the device will request and apply the EUI. However, if it requires to use different
DEV_EUI
, you can make the change in this section.

1# Obtained during first registration of the device
2SECRET_APP_EUI = 'XXXXXXXXXXXXXXXX'
3SECRET_APP_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

With these parameters configured, you have secured the connection between your device and The Things Network. The Things Network would be your end-point where the sensor data is going to be sent over. And to send data, you need to begin by gathering this data, which can be from sensors or modules with status feedback. A sensor can be attached directly communicating via Arduino layer to receive the data, wrap it and send it to The Things Network. Meanwhile, you will also need to have a mechanism that will be able to intercept data sent over Wi-Fi connectivity using a MQTT protocol.

Following tasks are the main processes that will be used to handle MQTT protocol. This will allow to decode incoming packets from the subscribed device and buffer the data if timeout has occurred while waiting on the MQTT packet. In this way you will be able to receive sensor data from any external device, for example using Arduino MKR Wi-Fi 1010 with a sensor attached, using MQTT protocol.

1# MQTT protocol handler
2...
3def connect_mqtt() -> mqtt_client:
4 def on_connect(client, userdata, flags, rc):
5 if rc == 0:
6 logging.debug("Connected to MQTT Broker!")
7 else:
8 logging.debug("Failed to connect, return code %d\n", rc)
9
10 client = mqtt_client.Client(mqtt_client_id)
11 client.username_pw_set(mqtt_username, mqtt_password)
12 client.on_connect = on_connect
13 client.connect(mqtt_broker, mqtt_port)
14
15 # Thread in the background calling loop() in an automatic manner
16 client.loop_start()
17 return client
18
19def on_message(client, userdata, msg):
20 logging.debug(f"MQTT Client: `{msg.payload.decode()}` from `{msg.topic}` topic")
21
22 decoded_mqtt_payload = json.dumps(EncodeFloat(msg.payload.decode("utf-8")))
23 print("MQTT Payload Data=%.15g" % (float(decoded_mqtt_payload)))
24
25 # `mqtt_data` is the payload decoded within receiving MQTT packet from a remote device
26 mqtt_data.append(float(decoded_mqtt_payload))
27 client.loop_stop()
28
29 return mqtt_data
30
31def subscribe(client: mqtt_client):
32 global mqtt_data
33 mqtt_timeout = time.time() + 60
34
35 client.subscribe(mqtt_topic)
36 client.on_message = on_message
37
38 while len(mqtt_data) <= 0:
39 print(f"Awaiting MQTT packet before proceeding")
40 time.sleep(publish_interval)
41 if time.time() > mqtt_timeout:
42 print(f"No MQTT packet is received - Defaulting Data")
43 mqtt_data = [0.0]
44 break
45 else:
46 print(f"MQTT packet received")
47
48 print(f"MQTT packet: ", mqtt_data[0])
49...

For LoRa® connectivity, to establish communication with The Things Network, we are going to use the Cayenne Low Power Payload Encoder as part of the process in this build.

1...
2frame = LppFrame()
3frame.add_temperature(0, rpc_data[0])
4frame.add_humidity(1, rpc_data[1])
5
6# MQTT payload as Generic 4-Byte Unsigned Integer
7frame.add_generic(2, mqtt_data[0])
8
9# Encoding packet for transmission
10payload = bytes(frame)
11lora_module.sendBytes(payload, len(payload), False)
12...

Multi-Protocol Arduino (M4) Application

For the full Arduino scripts please refer to the files found inside this compressed file.

The following sketch is for the Arduino layer that will help us retrieve the data in between Arduino and Linux layer.

1# Arduino side sketch
2#include <RPC.h>
3#include <SerialRPC.h>
4
5const long interval = 1000;
6unsigned long previousMillis = 0;
7
8void setup(){
9 pinMode(PA_12, INPUT);
10 //RPC.begin();
11
12 Serial.begin(115200);
13 while (!Serial) {}
14
15 Serial.println("M4 Layer - Multi Protocol Gateway");
16
17 RPC.bind("Data_0", []{ return analogRead(A0); });
18 RPC.bind("Data_1", []{ return analogRead(A1); });
19
20 Serial.println("Service Begin");
21}
22
23void loop(){
24 unsigned long currentMillis = millis();
25
26 if (currentMillis - previousMillis >= interval) {
27 previousMillis = currentMillis;
28
29 //record random value from A0 and A1
30 Serial.println(analogRead(A0));
31 Serial.println(analogRead(A1));
32
33 }
34}

The sketch above will help to expose and transfer the data that is processed within the Linux side. Transferring data to Linux side can be seen as a direct communication, as the sensor connected and monitored via Arduino side will send this data over LoRa® connectivity. For this build, we are going to use Analog readings to emulate the functionality.

Additionally by exposing, it means you will bring forth the data received within the Linux side to the Arduino side to feed the local-device as a control input. It can be used to display the data if you wish to for instance. This can be implemented in this script if you wish to further develop your desired multi-protocol gateway.

For MQTT publishing, we have also included a sketch that can be used with Arduino MKR Wi-Fi 1010 to test the gateway build. Of course, the code can be extended and be modified according the requirements of the device that will collect data from a certain sensor to send over MQTT protocol.

1#include <ArduinoMqttClient.h>
2#include <WiFiNINA.h>
3#include "arduino_secrets.h"
4
5///////please enter your sensitive data in the Secret tab/arduino_secrets.h
6char ssid[] = SECRET_SSID; // your network SSID (name)
7char pass[] = SECRET_PASS; // your network password (use for WPA, or use as key for WEP)
8
9WiFiClient wifiClient;
10MqttClient mqttClient(wifiClient);
11
12const char broker[] = "broker.emqx.io";
13int port = 1883;
14const char topic[] = "multiPrGw/mqtt1";
15
16//set interval for sending messages (milliseconds)
17const long interval = 50000;
18unsigned long previousMillis = 0;
19
20int count = 0;
21
22void setup() {
23 //Initialize serial and wait for port to open:
24 Serial.begin(9600);
25 while (!Serial) {
26 ; // wait for serial port to connect. Needed for native USB port only
27 }
28
29 // attempt to connect to Wi-Fi network:
30 Serial.print("Attempting to connect to WPA SSID: ");
31 Serial.println(ssid);
32 while (WiFi.begin(ssid, pass) != WL_CONNECTED) {
33 // failed, retry
34 Serial.print(".");
35 delay(5000);
36 }
37
38 Serial.println("You're connected to the network");
39 Serial.println();
40
41 Serial.print("Attempting to connect to the MQTT broker: ");
42 Serial.println(broker);
43
44 if (!mqttClient.connect(broker, port)) {
45 Serial.print("MQTT connection failed! Error code = ");
46 Serial.println(mqttClient.connectError());
47
48 while (1);
49 }
50
51 Serial.println("You're connected to the MQTT broker!");
52 Serial.println();
53}
54
55void loop() {
56 // call poll() regularly to allow the library to send MQTT keep alive which
57 // avoids being disconnected by the broker
58 mqttClient.poll();
59
60 unsigned long currentMillis = millis();
61
62 if (currentMillis - previousMillis >= interval) {
63 // save the last time a message was sent
64 previousMillis = currentMillis;
65
66 //record random value from A0, A1 and A2
67 float Rvalue = analogRead(A0);
68
69 Serial.print("Sending message to topic: ");
70 Serial.println(topic);
71 Serial.println(Rvalue/10);
72
73 // send message, the Print interface can be used to set the message contents
74 mqttClient.beginMessage(topic);
75 mqttClient.print(Rvalue/10);
76 mqttClient.endMessage();
77
78 Serial.println();
79 }
80}

Mounting the Multi-Protocol Gateway

It is now time to make the multi-protocol gateway to run and for this you will need to build the Docker container that will help you operate in the background on the Linux layer. Using the terminal, you can use the following commands to get the multi-protocol gateway container up and running.

You will need to have the files ready in a folder inside the

adb
directory within Arduino root.

1C:\Users\#USERNAME#\AppData\Local\Arduino15\packages\arduino\tools\adb\32.0.0

Having the files ready at that directory, you can use the following commands to push the files to the

fio
directory inside the Portenta X8. The second command will let you navigate inside the Portenta X8.

1adb push multi-protocol-gateway /home/fio
2adb shell

You will now build the container using the following commands. The following command will tag the container with

Multi_Protocol_Gateway_X8
as its name.

1cd ../home/fio/Multi_Protocol_Gateway_X8
2#Multi_Protocol_Gateway_X8 sudo docker build . -t multi_gateway

You will be able to see following results when the image is built successfully.

Multi-Protocol Gateway Docker Build Terminal Log
Multi-Protocol Gateway Docker Build Terminal Log

If you have created the Docker container previously and want to re-create it with new changes made outside the shell, please check that the container and its build directory is stopped and removed. This is for the convenience of having a clean working environment

After a successful container build, you will have to make the image run. To do that, you can use the following command. This command will immediately give an output in your terminal, telling you how the Python script is running. If you wish to have it running in the background, please add

-d
flag at the end of the command.

1#Multi_Protocol_Gateway_X8 sudo docker-compose up

Finally, you will have the multi-protocol gateway running, in which it uses Wi-Fi and LoRa® connectivity. And also RPC for exchanging data between its layers. However, there are cases where you wish to make changes by adding more functionalities, such as including Cat. M1 or NB-IoT to expand its communication spectrum and for this you will need to stop the image. To stop the image from running, you can use following command.

1#Multi_Protocol_Gateway_X8 sudo docker-compose down

Getting to know status of the image is also crucial as it is the indicator of state of operation. The following command brings up active images and shows the status if the image restarted or stopped due to certain reasons. The second command lists built images and it will show you the components that goes with the main image that you're building.

1docker ps -a
2docker images

Multi-Protocol Gateway  Active Docker Image
Multi-Protocol Gateway Active Docker Image

With all this, you have built a running multi-protocol gateway based on Portenta X8 and Max Carrier. You will be able to observe the data sent by the gateway with The Things Network platform End-Device section under Applications.

If you are curious about what to expect from the build you have made in this tutorial, the following image shows you the terminal of the running multi-protocol gateway.

Multi-Protocol Gateway Run Log
Multi-Protocol Gateway Run Log

Conclusion

In this tutorial you have learned how to set up a Multi-Protocol Gateway composed of MQTT protocol, RPC and LoRaWAN®, by using the Portenta X8 and the Portenta Max Carrier. You have built the gateway that will connect to The Things Network to send the desired data. Also, the gateway is capable of exchanging data between Arduino and Linux layer using RPC, in which you have exposed the ports to be able to receive data from the local sensor to be sent directly to The Things Network.

Next Steps

  • Now that you have developed a multi-protocol gateway, using Wi-Fi and LoRaWAN® connectivity, expand the gateway's capability by adding other connectivity types such as Cat. M1 and NB-IoT
  • Expand functionalities for data processing using RPC while using multi-protocol architecture.

Troubleshooting

You might encounter some errors or misbehaviors while working on the code, preventing you from progressing on the development. You can try the following troubleshooting tips to solve the commonly known issues:

  • If the sketch upload process fails, check if your Portenta X8 is in bootloader mode. To put the Portenta X8 into Bootloader mode, double-press its RESET button and verify that the green LED is blinking. After this, you can try re-uploading the sketch for the Arduino layer.
  • Check the position of the BOOT DIP switch of the Portenta Max Carrier. If the Portenta X8 gets into bootloader mode immediately after powering-on, including when connected via USB-C, change the position of the BOOT DIP switch to OFF. This case applies to the Arduino layer.
  • If you encounter an issue regarding terminal input inconvenience, please enter
    export TERM=xterm
    as the command in the terminal to get readable inputs.
  • In case internal Wi-Fi connection cannot be established through the command input due to "unavailable" SSID, although it is in range. Please try using different SSID if available or hotspot from a different device to host network connectivity.
  • If you encounter docker image conflict when running after building, please make sure you have used name tag that matches the one from the
    docker-compose.yml
    file.

Tutorial Toolbox

Contribute to Arduino

Join the community and suggest improvements to this article via GitHub. Make sure to read out contribution policy before making your pull request.

Missing something?

Check out our store and get what you need to follow this tutorial.

Suggest Changes

The content on docs.arduino.cc is facilitated through a public GitHub repository. You can read more on how to contribute in the contribution policy.