Door Intruder Detector Using ML with the Nicla Voice

Introduction

Security has always been a very important factor for our well-being. From the personal to the material, keeping our assets safe is something we struggle with every day, in our cars, lockers, computers, and our houses, we insure everything is locked up, but there are times when that is not enough.

In such a connected world, where we have a lot of information in the palm of our hand, it would also be very useful to have the status of those things that we have insured in a more analogous way, such as the front door of our home. Leveraging the integrated microphone of the Nicla Voice provides the possibility to build this kind of protection system.

Goals

The goal of this application note is to showcase an intruder detection and monitoring system for the front door of the house using a combination of Nicla Voice and Portenta H7 as a host, and the Arduino Cloud. The project's objectives are the following:

• Analyze the surrounding sounds with the onboard microphone of the Nicla Voice and run artificial intelligence algorithms at the edge.
• Correctly identify if a door has been opened in an ordinary manner or suffered a forced attempt.
• Report opening and intruder detection events through Bluetooth® Low Energy to a Host.
• The host must forward the data from the Nicla Voice to the Arduino Cloud using an Internet connection (Wi-Fi® in this case).
• Create an Arduino Cloud dashboard that syncs in real time to inform and alert the user.
• Gather real-world data to train a Machine Learning model.

Hardware and Software Requirements

Hardware Requirements

• Nicla Voice
• 3.7 V LiPo battery
• Portenta H7
• USB-C® cable
• Micro USB cable
• 5 VDC wall power adapter
• DC brushless fan Model: LD3007MS
• Nicla Voice and Portenta H7 3D Printed enclosures

Software Requirements

• Arduino IDE 1.8.10+, Arduino IDE 2, or Arduino Web Editor.
• If you are going to use an offline Arduino IDE, you must install the following libraries:

  ArduinoBLE

  ,

  ArduinoIoTCloud

  , and

  Arduino_ConnectionsHandler

  . You can install them using the Arduino IDE Library Manager.
• The intruder detector example code.
• Arduino Machine Learning Tools
• Edge Impulse® CLI, to easily flash any Machine Learning model created with the Arduino Machine Learning Tools, we can install the Edge Impulse® Cli tool following these instructions.
• Nicla Voice audio firmware to upload recordings to Machine Learning Tools.
• Syntiant® uploader and precompiled ML model.
• Arduino Create Agent to add the Portenta H7 to the IoT Cloud.

The Machine Learning Model

The Machine Learning is a way of programming in which we make our device process raw data from sensors and learn based on extracted meaningful features that will be later used for identifying or predicting the originally trained scenarios.

To be able to identify when the door is being opened or when it is being forced by someone, we trained a machine learning model using the Machine Learning Tools integration that will be running repetitively in the Nicla Voice.

First, we flashed the Nicla Voice with an audio ingestion firmware to record and upload samples to our Machine Learning Tools project of three different scenarios that we want to identify later.

Here is our model design:

As we are using the Nicla Voice, we must follow the model design to be supported by the onboard Syntiant® NDP120 Neural Decision Processor™, taking care of these characteristics.

In the time series data block:

• Window size: 968 ms
• Window increase: 484 ms
• Frequency: 16000 Hz

In the processing block:

• Audio (Syntiant): This DSP block computes log Mel-filterbank energy features from audio signals.

In the learning block:

• Classification: This block includes the Neural Network architecture to learn patterns from audio data.

Here is a graphical representation of how the samples look before and after the digital signal processing:

Our three classes are:

• Opened: here we record the sound of the door being opened.
• Forcing: here we record the sound of the door lock being hit with a metallic tool.
• Z_openset: here we record the environment noise when nothing is happening and unknown sounds like music and conversations.

Here is the neural network classifier configuration for this application:

After the model is trained with a lot of samples of a door being opened, being forced and nothing happening (z_openset), we got considerable good results that we can evidentiate on the "Confusion matrix" with a 97.4% of accuracy being achieved with a validation set of new data.

The model performance can be affected if the application is implemented on a very different environment than the one used for training. It's recommended to feed the datasets with new samples and retrain the model for a new and upgraded deployment.

For a new model deployment, use the Syntiant® uploader and replace the

ei_model.synpkg

Intruder Detector System Setup

In this application, we don't need any particular wiring diagram other than the Nicla Voice and the Portenta H7.

The Nicla Voice will be attached to the guarded door and near the lock, the Portenta H7 host will be somewhere with good Wi-Fi® coverage and not so far from the Nicla Voice due to Bluetooth® Low Energy narrow range of a couple of meters.

The Nicla will communicate with the Portenta H7 through Bluetooth® Low Energy advertising any event to the host that will receive and forward the notification to the Arduino Cloud using Wi-Fi®.

The Nicla Voice is powered by a LiPo battery inside its enclosure. The Portenta H7 will be powered by a 5 VDC USB adapter. Although the Portenta H7 can work for long periods without needing external cooling, it is always recommended to keep the devices that are working 24/7 as cool as possible, having the Portenta H7 enclosure a cooling fan connected.

Intruder Detector System Overview

The Nicla Voice attached to the door is running a machine learning model listening to the surrounding noises and searching for known sounds like the door being opened or the door lock being hit with a metallic tool, for example, to simulate a forcing process.

If the sounds are recognized with a certainty higher than 70% the event will be considered as occurred and a Bluetooth® Low Energy notification will be sent to the Portenta H7 host updating the door status. If the sounds are recognized with a certainty higher than 70% the event will be considered as occurred and a Bluetooth® Low Energy notification will be sent to the Portenta H7 host updating the door status.

The Portenta H7 will be in charge of receiving the door event notification and updating the dashboard variables in the IoT Cloud immediately so the user can be aware in real time of any change. In addition to these features, the application dashboard will show a historic record of events, alongside an "Intruder Alert notification", the Bluetooth® Low Energy connection status between both devices and the Nicla Voice battery level so we can know when to recharge it.

Nicla Voice Code

We will go through some important code sections to make this application fully operative. We will begin with the required libraries:

• Including

  NDP.h

  will enable and run the Neural Decision Processor (NDP120), it's included in the BSP of the Nicla Voice.
• Including

  ArduinoBLE.h

  will enable the Bluetooth® Low Energy communication, install it by searching for it on the Library Manager.

The Bluetooth® Low Energy services and characteristics are standardized for the specific use of this application's features. The service is defined as

Immediate alert

Alert Level

Battery Level

Copy
1#include "NDP.h"
2#include <ArduinoBLE.h>
3
4// Alert Service
5BLEService alertService("1802");  // Immediate alert
6
7// BLE Alert Characteristic
8BLEUnsignedCharCharacteristic alertLevel("2A06",                // standard 16-bit characteristic UUID
9                                         BLERead | BLENotify);  // remote clients will be able to get notifications if this characteristic changes
10
11// Bluetooth® Low Energy Battery Level Characteristic
12BLEUnsignedCharCharacteristic batteryLevelChar("2A19",                // standard 16-bit characteristic UUID
13                                               BLERead | BLENotify);  // remote clients will be able to get notifications if this characteristic changes

The Nicla Voice's integrated NDP needs some files stored in the external flash of the board which are:

• mcu_fw_120_v91.synpkg
• dsp_firmware_v91.synpkg
• ei_model.synpkg

The files must be named exactly as shown above.

Here you can download these files including the Machine Learning model already trained and the uploading tools.

After downloading the required files, to store them in the Nicla Voice memory you must follow the steps detailed on this guide section.

These files won't be overwritten after any sketch update.

Copy
1// Neural Decision Processor firmware and ML model files loading
2  NDP.begin("mcu_fw_120_v91.synpkg");
3  NDP.load("dsp_firmware_v91.synpkg");
4  NDP.load("ei_model.synpkg");
5  Serial.println("packages loaded");
6  NDP.getInfo();
7  Serial.println("Configure mic");
8  NDP.turnOnMicrophone();

The main responsibility of the Nicla Voice code is to listen to and identify the trained sounds. The below code section is in charge of comparing the inferred category and taking a certain action between them.

In the case the Nicla detects the door opening, the label parameter will turn to

NN0:opened

NN0:forcing

Copy
1/**
2  Inference Interruption Callback to be executed with every triggered inference,
3  it controls the built-in LED's and send the alerts through BLE.
4  
5  Possible labels: NN0:opened, NN0:forcing
6
7  Alerts: 1 = mild alert (for door opened), 2 = high alert (for intruder detected)
8
9  @param label The inferred category label
10*/
11void BLEsend(char* label) {
12
13  if (strcmp(label, "NN0:opened") == 0) {
14    alertLevel.writeValue(1);
15    NDP.noInterrupts();
16    nicla::leds.begin();
17    nicla::leds.setColor(green);
18    delay(3000);
19    nicla::leds.end();
20    NDP.interrupts();
21  }
22  if (strcmp(label, "NN0:forcing") == 0) {
23    alertLevel.writeValue(2);
24    NDP.noInterrupts();
25    nicla::leds.begin();
26    nicla::leds.setColor(red);
27    delay(3000);
28    nicla::leds.end();
29    NDP.interrupts();
30  }
31  if (!lowestPower) {
32    Serial.println(label);
33  }
34}

These methods are used to update the characteristics of this application:

Copy
1alertLevel.writeValue(x);  // x = 1 if the door is opened or x = 2 if the door is forced
2batteryLevelChar.writeValue(HigherBatteryLevel);

Portenta H7 Code

The Portenta H7 needs the following libraries:

• ArduinoIoTCloud.h

  This one handles the Arduino Cloud connection and project variables publishing. It can be installed directly from the Arduino Library Manager.

• Arduino_ConnectionHandler.h

  This one manages the Wi-Fi® connection and can be installed directly from the Arduino Library Manager.

• thingProperties.h

  This is automatically generated by the Arduino Cloud, however, if you are using an offline IDE verify it's in the same directory as your sketch.

In the global parameters, we have two variables that will store the local status of the alerts and the Nicla Voice battery level:

Copy
1#include <ArduinoIoTCloud.h>
2#include <Arduino_ConnectionHandler.h>
3#include "thingProperties.h"
4#include <ArduinoBLE.h>
5
6#define DEBUG false
7
8// Global Parameters
9byte BatteryValue = 0;  // last battery level received.
10byte AlertValue = 0;    // last alert value received.

In the

thingProperties.h

• We must update the

  THING_ID

  variable with our Arduino Cloud Thing ID.
• For a successful internet connection, we must update the

  SSID

  and

  PASS

  variables with our Wi-Fi® credentials.

In the

initProperties()

ON_CHANGE

Copy
1const char THING_ID[] = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"; // Arduino Cloud Thing ID
2
3const char SSID[]     = "xxxxxxxx";    // Network SSID (name)
4const char PASS[]     = "xxxxxxxx";    // Network password (use for WPA, or use as key for WEP)
5
6String DoorEvent = "System initiated";  
7int battery;
8bool BLEstatus;
9bool alertStatus;
10
11void initProperties(){
12
13  ArduinoCloud.setThingId(THING_ID);
14  ArduinoCloud.addProperty(DoorEvent, READ, ON_CHANGE, NULL);
15  ArduinoCloud.addProperty(battery, READ, ON_CHANGE, NULL);
16  ArduinoCloud.addProperty(BLEstatus, READ, ON_CHANGE, NULL);
17  ArduinoCloud.addProperty(alertStatus, READ, ON_CHANGE, NULL);
18
19}
20
21WiFiConnectionHandler ArduinoIoTPreferredConnection(SSID, PASS);

The main responsibility of the Portenta H7 code is to connect to the Nicla Voice and be aware of new notifications to forward them to the Cloud. To achieve this, the first thing done by the Portenta is to initialize the Bluetooth® Low Energy communication, then initialize the Arduino Cloud service and start scanning for peripherals.

Copy
1// Initialize BLE
2  if (!BLE.begin()) {
3    Serial.println("Starting BLE failed!");
4    while (1) {
5    }
6  }
7
8  // Defined in thingProperties.h
9  initProperties();
10
11  // Connect to Arduino Cloud
12  if (!ArduinoCloud.begin(ArduinoIoTPreferredConnection)) {
13    Serial.println("ArduinoCloud.begin FAILED!");
14  }
15
16  setDebugMessageLevel(2);
17  ArduinoCloud.printDebugInfo();
18
19    // start scanning for peripheral
20  BLE.scan();
21  Serial.println("Scanning for peripherals.");

The scanning process will automatically stop when it finds a "Nicla Lock" called device, then it will search for its services and characteristics with the

NiclaLockHandler()

Copy
1void loop() {
2
3  // check if a peripheral has been discovered
4  BLEDevice peripheral = BLE.available();
5
6  // Turn on the Green LED if the board is successfully connected to the Cloud. On with Low.
7  if (ArduinoCloud.connected()) {
8    digitalWrite(LEDG, LOW);
9  } else {
10    digitalWrite(LEDG, HIGH);
11  }
12
13  if (peripheral) {
14    // peripheral discovered, print out address, local name, and advertised service
15    Serial.print("Found ");
16    Serial.print(peripheral.address());
17    Serial.print(" '");
18    Serial.print(peripheral.localName());
19    Serial.print("' ");
20    Serial.print(peripheral.advertisedServiceUuid());
21    Serial.println();
22
23    // Check if the peripheral is a Nicla Lock:
24    if (peripheral.localName() == "Nicla Lock") {
25      // stop scanning
26      BLE.stopScan();
27
28      // Nicla Voice node connection handler
29      NiclaLockHandler(peripheral);
30
31      // peripheral disconnected, start scanning again
32      BLE.scan();
33    }
34  }
35  ArduinoCloud.update();
36}

In order to just get connected with the Nicla Voice, the Portenta searches for the specific "Alert Service" using its UUID

1802

2A19

2A06

Copy
1// discover peripheral attributes
2  Serial.println("Searching for service 1802 ...");
3
4  if (peripheral.discoverService("1802")) {
5    Serial.println("Service discovered");
6  } else {
7    Serial.println("Attribute discovery failed.");
8    peripheral.disconnect();
9
10    while (1)
11      ;
12    return;
13  }
14
15  // retrieve the simple key characteristics
16  BLECharacteristic batteryLevelChar = peripheral.characteristic("2A19");
17  BLECharacteristic alertLevel = peripheral.characteristic("2A06");
18
19  // subscribe to the simple key characteristics process
20  Serial.println("Subscribing to simple key characteristic ...");
21  if (!batteryLevelChar || !alertLevel) {
22    Serial.println("no simple key characteristic found!");
23    peripheral.disconnect();
24    return;
25  } else if (!batteryLevelChar.canSubscribe() || !alertLevel.canSubscribe()) {
26    Serial.println("simple key characteristic is not subscribable!");
27    peripheral.disconnect();
28    return;
29  } else if (!batteryLevelChar.subscribe() || !alertLevel.subscribe()) {
30    Serial.println("subscription failed!");
31    peripheral.disconnect();
32    return;
33  } else {
34    Serial.println("Subscribed to Battery Level and Alert Characteristic");
35  }

Finally, the Portenta verifies continuously if a characteristic is updated to upload it to the Cloud:

Copy
1// while the peripheral is connected
2  while (peripheral.connected()) {
3    
4    // check if the value of the characteristic has been updated
5    if (batteryLevelChar.valueUpdated()) {
6
7      batteryLevelChar.readValue(BatteryValue);
8
9      Serial.print("Battery Level: ");
10      Serial.print(BatteryValue);
11      Serial.println(" %");
12
13      battery = BatteryValue;
14    }
15
16    // check if the value of the characteristic has been updated
17    if (alertLevel.valueUpdated()) {
18
19      alertLevel.readValue(AlertValue);
20
21      Serial.print("Alert: ");
22      Serial.println(AlertValue);
23
24      
25      if (AlertValue == 1) {    // if the Alert = 1 means the door was opened
26        control = !control;     // as variables are updated if they change, add a simple dot "." to be able to update it if happens twice consecutive.
27        if (control) {
28          DoorEvent = "Door opened";
29        } else {
30          DoorEvent = "Door opened.";
31        }
32
33        alertStatus = false;
34
35      } else if (AlertValue == 2) {   // if the Alert = 2 means the door was forced
36        control = !control;           // as variables are updated if they change, add a simple dot "." to be able to update it if happens twice consecutive.
37        if (control) {
38          DoorEvent = "Intruder detected!";
39        } else {
40          DoorEvent = "Intruder detected!.";
41        }
42
43        alertStatus = true;
44      }
45    }
46
47    // turn off green LED if disconnected from Cloud.
48    if (ArduinoCloud.connected()) {
49      digitalWrite(LEDG, LOW);
50    } else {
51      digitalWrite(LEDG, HIGH);
52    }
53
54    ArduinoCloud.update();
55  }

The Cloud Dashboard

Taking advantage of the Arduino Cloud, we can seamlessly integrate a simple but powerful dashboard to monitor and visualize the status of the system in real-time:

Within the Arduino Cloud's dashboard, the system variables can be monitored. We have a battery level indicator widget accompanied by a time series graph, a chat-looking widget to store the events historically, a red LED to shine when an intruder is detected by the system, and a green LED to show the Bluetooth® Low Energy connection status between the Nicla Voice and the Portenta H7 host.

We can easily access this dashboard from a PC, mobile phone or tablet from anywhere, receiving an instantaneous update wherever we are. In addition, we can set different integrations to complement our project, for example, setting up an IFTTT automation to receive an email, a cellphone notification, or even triggering some automation like turning on all the lights at home whenever an alert is fired.

Full Intruder Detector Example

All the necessary files to replicate this application notes can be found below:

• The complete code can be downloaded here
• The enclosure 3D printed models can be downloaded here.
• The Machine Learning Tools project is public here so you can clone it and modify it to adapt it to your needs by improving the dataset or model architecture for a custom deployment.

Conclusion

In this application note, we have learned how to implement a Machine Learning Tools project powered by Edge Impulse and the Arduino Cloud at the edge using a Nicla Voice and a Portenta H7. This application could be a simple demonstration of how Arduino's environment simplifies the workflow for developing smart solutions to solve real-life situations by integrating "complex" and mighty algorithms with just a few easy-to-follow steps. We covered sound recognition, Bluetooth® Low Energy communication, and real-time Cloud monitoring.

Next Steps

As you already know how to develop an Intruder Detector with the Nicla Voice and the Portenta H7, it's time for you to continue exploring all the capabilities of the Arduino Pro environment to integrate it with your professional setup and improve it with powerful solutions.

You can create your application by training your own model with different scenario sounds or using the built-in Nicla Voice's IMU to analyze machine vibrations and making smart your existing manufacturing setup.