In this tutorial we will use an Arduino Nano 33 BLE Sense for gesture recognition, made possible by the embedded APDS9960 sensor.
We will use the sensor to print out simple hand gesture directions and control the board's RGB LED accordingly. In addition, we will program our board to blink the built-in LED and change colors to the RGB LED. according to the direction of the set gestures. The code will read simple Up-Down-Right-Left hand motions.
The goals of this project are:
The APDS9960 sensor is a multipurpose device that features advanced gesture detection, proximity detection, Digital Ambient Light Sense (ALS) and Color Sense (RGBC).
The sensor's gesture detection utilizes four directional photodiodes to sense reflected IR energy (sourced by the integrated LED) to convert physical motion information (i.e. velocity, direction and distance) into digital information.
It features:
If you want to read more about the APDS9960 sensor module see here.
The APDS9960 library allows us to use the sensor available on the board, to read gestures, color, light intensity and proximity. The library includes some of the following functions:
1begin()2end()3gestureAvailable()4readGesture()5colorAvailable()6readColor()7proximityAvailable()8readProximity()9setGestureSensitivity()10setInterruptPin()11setLEDBoost()
If you want a deeper knowledge on any of the functions of the library, you can check the Arduino reference for this library.
For the purposes of this tutorial we will only focus on the gesture readings, which are based on the detection of the movement of the hand over four photodiodes inside the sensor.
1. Setting up
Let's start by opening the Arduino Web Editor, click on the Libraries tab and search for the APDS9960 library. Then in > Examples, open the GestureSensor sketch and once it opens, you could rename it as Gesture&LEDs.
2. Connecting the board
Now, connect the Arduino Nano 33 BLE Sense to the computer and make sure that the Web Editor recognizes it, if so, the board and port should appear as shown in the image below. If they don't appear, follow the instructions to install the plugin that will allow the Editor to recognize your board.
3. Blink patterns according to hand gestures
Now we will need to modify the code on the example, in order to change the color of the RGB LED and the built-in LED respectively according to the direction of our hand gesture.
After including the
Arduino_APDS9960.h
library, we will need to configure the specified pins (22, 23, 24, LED_BUILTIN) at the beginning of the setup()
section, to behave as output:1//in-built LED2pinMode(LED_BUILTIN, OUTPUT);3//Red 4pinMode(LEDR, OUTPUT);5//Green 6pinMode(LEDG, OUTPUT);7//Blue8pinMode(LEDB, OUTPUT);
and then at the end, we need to turn all the LEDs OFF by adding the following statements
1// Turining OFF the RGB LEDs2 digitalWrite(LEDR, HIGH);3 digitalWrite(LEDG, HIGH);4 digitalWrite(LEDB, HIGH);
In the
loop()
section the if()
statement is checking that the gesture sensor is available and if it is, it reads for any incoming gesture detection. Next, in the
switch()
statement we will add different actions to be performed according to conditions, in this case the direction of the hand gesture will define those conditions.If the sensor detects motions (up, down, left or right) we can add the following code snippets between the Serial.println() and the break; in each switch case, to activate the red, blue and green colors of the RGB LED and the orange on the built-in LED.
In the
GESTURE_UP
case, the RGB LED will glow red for a second: 1digitalWrite(LEDR, LOW); 2delay(1000);3digitalWrite(LEDR, HIGH);
In the
GESTURE_DOWN
case, the RGB LED will glow green for one second:1digitalWrite(LEDG, LOW); 2delay(1000);3digitalWrite(LEDG, HIGH);
In the
GESTURE_LEFT
case, the RGB LED will glow blue for one second: 1digitalWrite(LEDB, LOW); 2delay(1000);3digitalWrite(LEDB, HIGH);
Lastly, in the
GESTURE_RIGHT
case, the small built-in LED will glow orange for one second:1digitalWrite(LED_BUILTIN, HIGH); 2delay(1000);3digitalWrite(LED_BUILTIN, LOW);
Now the code is complete!
4. Complete code
1/*2 APDS9960 - Gesture Sensor3 This example reads gesture data from the on-board APDS9960 sensor of the4 Nano 33 BLE Sense and prints any detected gestures to the Serial Monitor.5 Gesture directions are as follows:6 - UP: from USB connector towards antenna7 - DOWN: from antenna towards USB connector8 - LEFT: from analog pins side towards digital pins side9 - RIGHT: from digital pins side towards analog pins side10 The circuit:11 - Arduino Nano 33 BLE Sense12 This example code is in the public domain.13*/14
15#include <Arduino_APDS9960.h>16
17void setup() {18 Serial.begin(9600);19 //in-built LED20 pinMode(LED_BUILTIN, OUTPUT);21 //Red22 pinMode(LEDR, OUTPUT);23 //Green24 pinMode(LEDG, OUTPUT);25 //Blue26 pinMode(LEDB, OUTPUT);27 28 while (!Serial);29 if (!APDS.begin()) {30 Serial.println("Error initializing APDS9960 sensor!");31 }32 // for setGestureSensitivity(..) a value between 1 and 100 is required.33 // Higher values makes the gesture recognition more sensible but less accurate34 // (a wrong gesture may be detected). Lower values makes the gesture recognition35 // more accurate but less sensible (some gestures may be missed).36 // Default is 8037 //APDS.setGestureSensitivity(80);38 Serial.println("Detecting gestures ...");39 // Turining OFF the RGB LEDs40 digitalWrite(LEDR, HIGH);41 digitalWrite(LEDG, HIGH);42 digitalWrite(LEDB, HIGH);43}44void loop() {45 if (APDS.gestureAvailable()) {46 // a gesture was detected, read and print to serial monitor47 int gesture = APDS.readGesture();48 switch (gesture) {49 case GESTURE_UP:50 Serial.println("Detected UP gesture");51 digitalWrite(LEDR, LOW);52 delay(1000);53 digitalWrite(LEDR, HIGH);54 break;55 case GESTURE_DOWN:56 Serial.println("Detected DOWN gesture");57 digitalWrite(LEDG, LOW);58 delay(1000);59 digitalWrite(LEDG, HIGH);60 break;61 case GESTURE_LEFT:62 Serial.println("Detected LEFT gesture");63 digitalWrite(LEDB, LOW);64 delay(1000);65 digitalWrite(LEDB, HIGH);66 break;67 case GESTURE_RIGHT:68 Serial.println("Detected RIGHT gesture");69 digitalWrite(LED_BUILTIN, HIGH);70 delay(1000);71 digitalWrite(LED_BUILTIN, LOW);72 break;73 default:74 break;75 }76 }77}
After you have successfully verified and uploaded the sketch to the board, open the Serial Monitor from the menu on the left.
In order to test out the code, you could begin by stabilizing your board on a standing position in front of you (USB port facing down) and carry on by making directional UP-DOWN-RIGHT-LEFT hand gestures. Try to make your movement as clear as possible, yet subtle enough for the sensor to pick it up.
Here is a screenshot example of the sketch returning values.
Sometimes errors occur, if the code is not working there are some common issues we can troubleshoot:
In this tutorial we learned what a APDS9960 sensor is, how to use the one embedded in the Arduino Nano 33 BLE Sense board and the APDS9960 library, as well as how to create your own gesture detection monitor which can operate the RGB and built-in LED in various color patterns.