Gesture Recognition with the Nano 33 BLE Sense

In this tutorial we will use an Arduino Nano 33 BLE Sense for gesture recognition, made possible by the embedded APDS9960 sensor.

We will use the sensor to print out simple hand gesture directions and control the board's RGB LED accordingly. In addition, we will program our board to blink the built-in LED and change colors to the RGB LED. according to the direction of the set gestures. The code will read simple Up-Down-Right-Left hand motions.

Goals

The goals of this project are:

• Learn what a APDS9960 sensor is.
• Use the APDS9960 library.
• Learn how to output raw sensor data from the Arduino Nano 33 BLE Sense.
• Create your own gesture detection monitor.
• Learn how to control the built-in LED and RGB LED, through hand gestures.

Hardware & Software Needed

• This project uses no external sensors or components.
• In this tutorial we will use the Arduino Create Web Editor to program the board.

APDS9960 Sensor

The APDS9960 sensor is a multipurpose device that features advanced gesture detection, proximity detection, Digital Ambient Light Sense (ALS) and Color Sense (RGBC).

The sensor's gesture detection utilizes four directional photodiodes to sense reflected IR energy (sourced by the integrated LED) to convert physical motion information (i.e. velocity, direction and distance) into digital information.

It features:

• Four separate diodes sensitive to different directions.
• Ambient light rejection.
• Offset compensation.
• Programmable driver for IR LED current.
• 32 dataset storage FIFO.
• Interrupt driven I2C-bus communication.

If you want to read more about the APDS9960 sensor module see here.

The Library

The APDS9960 library allows us to use the sensor available on the board, to read gestures, color, light intensity and proximity. The library includes some of the following functions:

Copy
1begin()
2end()
3gestureAvailable()
4readGesture()
5colorAvailable()
6readColor()
7proximityAvailable()
8readProximity()
9setGestureSensitivity()
10setInterruptPin()
11setLEDBoost()

If you want a deeper knowledge on any of the functions of the library, you can check the Arduino reference for this library.

For the purposes of this tutorial we will only focus on the gesture readings, which are based on the detection of the movement of the hand over four photodiodes inside the sensor.

Creating the Program

1. Setting up

Let's start by opening the Arduino Web Editor, click on the Libraries tab and search for the APDS9960 library. Then in > Examples, open the GestureSensor sketch and once it opens, you could rename it as Gesture&LEDs.

2. Connecting the board

Now, connect the Arduino Nano 33 BLE Sense to the computer and make sure that the Web Editor recognizes it, if so, the board and port should appear as shown in the image below. If they don't appear, follow the instructions to install the plugin that will allow the Editor to recognize your board.

3. Blink patterns according to hand gestures

Now we will need to modify the code on the example, in order to change the color of the RGB LED and the built-in LED respectively according to the direction of our hand gesture.

After including the

Arduino_APDS9960.h

setup()

Copy
1//in-built LED
2pinMode(LED_BUILTIN, OUTPUT);
3//Red 
4pinMode(LEDR, OUTPUT);
5//Green 
6pinMode(LEDG, OUTPUT);
7//Blue
8pinMode(LEDB, OUTPUT);

and then at the end, we need to turn all the LEDs OFF by adding the following statements

Copy
1// Turining OFF the RGB LEDs
2  digitalWrite(LEDR, HIGH);
3  digitalWrite(LEDG, HIGH);
4  digitalWrite(LEDB, HIGH);

In the

loop()

if()

Next, in the

switch()

If the sensor detects motions (up, down, left or right) we can add the following code snippets between the Serial.println() and the break; in each switch case, to activate the red, blue and green colors of the RGB LED and the orange on the built-in LED.

In the

GESTURE_UP

Copy
1digitalWrite(LEDR, LOW);  
2delay(1000);
3digitalWrite(LEDR, HIGH);

In the

GESTURE_DOWN

Copy
1digitalWrite(LEDG, LOW);  
2delay(1000);
3digitalWrite(LEDG, HIGH);

In the

GESTURE_LEFT

Copy
1digitalWrite(LEDB, LOW);  
2delay(1000);
3digitalWrite(LEDB, HIGH);

Lastly, in the

GESTURE_RIGHT

Copy
1digitalWrite(LED_BUILTIN, HIGH);  
2delay(1000);
3digitalWrite(LED_BUILTIN, LOW);

Now the code is complete!

4. Complete code

Copy
1/*
2  APDS9960 - Gesture Sensor
3  This example reads gesture data from the on-board APDS9960 sensor of the
4  Nano 33 BLE Sense and prints any detected gestures to the Serial Monitor.
5  Gesture directions are as follows:
6  - UP:    from USB connector towards antenna
7  - DOWN:  from antenna towards USB connector
8  - LEFT:  from analog pins side towards digital pins side
9  - RIGHT: from digital pins side towards analog pins side
10  The circuit:
11  - Arduino Nano 33 BLE Sense
12  This example code is in the public domain.
13*/
14
15#include <Arduino_APDS9960.h>
16
17void setup() {
18  Serial.begin(9600);
19  //in-built LED
20  pinMode(LED_BUILTIN, OUTPUT);
21  //Red
22  pinMode(LEDR, OUTPUT);
23  //Green
24  pinMode(LEDG, OUTPUT);
25  //Blue
26  pinMode(LEDB, OUTPUT);
27  
28  while (!Serial);
29  if (!APDS.begin()) {
30    Serial.println("Error initializing APDS9960 sensor!");
31  }
32  // for setGestureSensitivity(..) a value between 1 and 100 is required.
33  // Higher values makes the gesture recognition more sensible but less accurate
34  // (a wrong gesture may be detected). Lower values makes the gesture recognition
35  // more accurate but less sensible (some gestures may be missed).
36  // Default is 80
37  //APDS.setGestureSensitivity(80);
38  Serial.println("Detecting gestures ...");
39  // Turining OFF the RGB LEDs
40  digitalWrite(LEDR, HIGH);
41  digitalWrite(LEDG, HIGH);
42  digitalWrite(LEDB, HIGH);
43}
44void loop() {
45  if (APDS.gestureAvailable()) {
46    // a gesture was detected, read and print to serial monitor
47    int gesture = APDS.readGesture();
48    switch (gesture) {
49      case GESTURE_UP:
50        Serial.println("Detected UP gesture");
51        digitalWrite(LEDR, LOW);
52        delay(1000);
53        digitalWrite(LEDR, HIGH);
54        break;
55      case GESTURE_DOWN:
56        Serial.println("Detected DOWN gesture");
57        digitalWrite(LEDG, LOW);
58        delay(1000);
59        digitalWrite(LEDG, HIGH);
60        break;
61      case GESTURE_LEFT:
62        Serial.println("Detected LEFT gesture");
63        digitalWrite(LEDB, LOW);
64        delay(1000);
65        digitalWrite(LEDB, HIGH);
66        break;
67      case GESTURE_RIGHT:
68        Serial.println("Detected RIGHT gesture");
69        digitalWrite(LED_BUILTIN, HIGH);
70        delay(1000);
71        digitalWrite(LED_BUILTIN, LOW);
72        break;
73      default:
74        break;
75    }
76  }
77}

Testing It Out

After you have successfully verified and uploaded the sketch to the board, open the Serial Monitor from the menu on the left.

In order to test out the code, you could begin by stabilizing your board on a standing position in front of you (USB port facing down) and carry on by making directional UP-DOWN-RIGHT-LEFT hand gestures. Try to make your movement as clear as possible, yet subtle enough for the sensor to pick it up.

Here is a screenshot example of the sketch returning values.

Troubleshoot

Sometimes errors occur, if the code is not working there are some common issues we can troubleshoot:

• Missing a bracket or a semicolon
• Arduino board connected to the wrong port
• Accidental interruption of cable connection

Conclusion

In this tutorial we learned what a APDS9960 sensor is, how to use the one embedded in the Arduino Nano 33 BLE Sense board and the APDS9960 library, as well as how to create your own gesture detection monitor which can operate the RGB and built-in LED in various color patterns.