Creating a Flash-Optimized Key-Value Store

This tutorial explains how to create a Flash-optimized key-value store using the Flash memory of the Portenta H7.

AUTHOR: Giampaolo Mancini, Sebastian Romero

Overview

This tutorial explains how to create a Flash-optimized key-value store using the Flash memory of the Portenta H7. It builds on top of the Flash In-Application Programming tutorial.

Goals

In this tutorial you will learn how to use the Mbed OS TDBStore API to create a Key value store in the free space of the microcontroller's internal Flash.

Required Hardware and Software

Portenta H7 (ABX00042), Portenta H7 Lite (ABX00045) or Portenta H7 Lite Connected (ABX00046)
USB-C® cable (either USB-A to USB-C® or USB-C® to USB-C®)
Arduino IDE 1.8.10+ or Arduino Pro IDE 0.0.4+ or Arduino CLI 0.13.0+

Instructions

MbedOS APIs for Flash Storage

The core software of Portenta H7 is based on the Mbed OS operating system, allowing developers to integrate the Arduino API with the APIs exposed by Mbed OS.

Mbed OS has a rich API for managing storage on different mediums, ranging from the small internal Flash memory of a microcontroller to external SecureDigital cards with large data storage space.

In this tutorial, you are going to save a value persistently inside the Flash memory. That allows to access that value even after a reset of the microcontroller. You will retrieve some information from a Flash block by using the FlashIAPBlockDevice and the TDBStore APIs. You will use the

FlashIAPBlockDevice

class to create a block device on the free space of the Flash and you will create a Key-Value Store in it using the

TDBStore

API.

Important: The TBStore API optimizes for access speed, reduce wearing of the flash and minimize storage overhead. TBStore is also resilient to power failures. If you want to use the Flash memory of the microcontroller, always prefer the TDBStore approach over a direct access to the FlashIAP block device.

1. The Basic Setup

Begin by plugging in your Portenta board to the computer using a USB-C® cable and open the Arduino IDE. If this is your first time running Arduino sketch files on the board, we suggest you check out how to set up the Portenta H7 for Arduino before you proceed.

2. Create the Structure of the Program

Let's program the Portenta with a sketch. You will also define a few helper functions in a supporting header file.

Create a new sketch named
```
FlashKeyValue.ino
```
Create a new file named
```
FlashIAPLimits.h
```
to store the helper functions in a reusable file.

Note: Finished sketch its inside the tutorials library wrapper at:
Examples > Arduino_Pro_Tutorials > Creating a Flash-Optimized Key-Value Store > FlashKeyValueStore

3. Populate the Helper Functions

First let's add the helper functions to the

FlashIAPLimits.h

header. This will determine the available Flash limits to allocate the custom data.

1/**
2Helper functions for calculating FlashIAP block device limits
3**/
4
5// Ensures that this file is only included once
6#pragma once 
7
8#include <Arduino.h>
9#include <FlashIAP.h>
10#include <FlashIAPBlockDevice.h>
11
12using namespace mbed;
13
14// A helper struct for FlashIAP limits
15struct FlashIAPLimits {
16  size_t flash_size;
17  uint32_t start_address;
18  uint32_t available_size;
19};
20
21// Get the actual start address and available size for the FlashIAP Block Device
22// considering the space already occupied by the sketch (firmware).
23FlashIAPLimits getFlashIAPLimits()
24{
25  // Alignment lambdas
26  auto align_down = [](uint64_t val, uint64_t size) {
27    return (((val) / size)) * size;
28  };
29  auto align_up = [](uint32_t val, uint32_t size) {
30    return (((val - 1) / size) + 1) * size;
31  };
32
33  size_t flash_size;
34  uint32_t flash_start_address;
35  uint32_t start_address;
36  FlashIAP flash;
37
38  auto result = flash.init();
39  if (result != 0)
40    return { };
41
42  // Find the start of first sector after text area
43  int sector_size = flash.get_sector_size(FLASHIAP_APP_ROM_END_ADDR);
44  start_address = align_up(FLASHIAP_APP_ROM_END_ADDR, sector_size);
45  flash_start_address = flash.get_flash_start();
46  flash_size = flash.get_flash_size();
47
48  result = flash.deinit();
49
50  int available_size = flash_start_address + flash_size - start_address;
51  if (available_size % (sector_size * 2)) {
52    available_size = align_down(available_size, sector_size * 2);
53  }
54
55  return { flash_size, start_address, available_size };
56}

4. Make the Key Store Program

Go to

FlashKeyValue.ino

and include the libraries that you need from MBED and your header helper (

FlashIAPLimits.h

). The

getFlashIAPLimits()

function which is defined in the

FlashIAPLimits.h

header takes care of not overwriting data already stored on the Flash and aligns the start and stop addresses with the size of the Flash sector. You can use those calculated limits to create a block device and a

TDBStore

on top of them.

1#include <FlashIAPBlockDevice.h>
2#include <TDBStore.h>
3
4using namespace mbed;
5
6// Get limits of the In Application Program (IAP) flash, ie. the internal MCU flash.
7#include "FlashIAPLimits.h"
8auto iapLimits { getFlashIAPLimits() };
9
10// Create a block device on the available space of the FlashIAP
11FlashIAPBlockDevice blockDevice(iapLimits.start_address, iapLimits.available_size);
12
13// Create a key-value store on the Flash IAP block device
14TDBStore store(&blockDevice);
15
16// Dummy sketch stats data for demonstration purposes
17struct SketchStats {
18  uint32_t startupTime;
19  uint32_t randomValue;
20  uint32_t runCount;
21};

In the

setup()

function at the beginning you will have to wait until the Serial port is ready and then print some info about the FlashIAP block device (

blockDevice

1void setup()
2{
3  Serial.begin(115200);
4  while (!Serial);
5
6  //  Wait for terminal to come up
7  delay(1000);
8
9  Serial.println("FlashIAPBlockDevice + TDBStore Test");
10
11  // Feed the random number generator for later content generation
12  srand(micros());
13
14  // Initialize the flash IAP block device and print the memory layout
15  blockDevice.init();  
16  Serial.print("FlashIAP block device size: ");
17  Serial.println(blockDevice.size());
18  Serial.print("FlashIAP block device read size: ");
19  Serial.println(blockDevice.get_read_size());
20  Serial.print("FlashIAP block device program size: ");
21  Serial.println(blockDevice.get_program_size());
22  Serial.print("FlashIAP block device erase size: ");
23  Serial.println(blockDevice.get_erase_size());
24  // Deinitialize the device
25  blockDevice.deinit();

After that, initialize the TDBstore (our storage space), set the key for the store data (

stats

), initialize the value that you will save

runCount

and declare an object to fetch the previous values (

previousStats

1// Initialize the key-value store
2  Serial.print("Initializing TDBStore: ");
3  auto result = store.init();
4  Serial.println(result == MBED_SUCCESS ? "OK" : "Failed");
5  if (result != MBED_SUCCESS)
6    while (true); // Stop the sketch if an error occurs
7
8  // An example key name for the stats on the store
9  const char statsKey[] { "stats" };
10
11  // Keep track of the number of sketch executions
12  uint32_t runCount { 0 };
13
14  // Previous stats
15  SketchStats previousStats;

Now that you have everything ready, let's retrieve the previous values from the store and update the store with the new values.

1// Get previous run stats from the key-value store
2  Serial.println("Retrieving Sketch Stats");
3  result = getSketchStats(statsKey, &previousStats);
4
5  if (result == MBED_SUCCESS) {
6    Serial.println("Previous Stats");
7    Serial.print("\tStartup Time: ");
8    Serial.println(previousStats.startupTime);
9    Serial.print("\tRandom Value: ");
10    Serial.println(previousStats.randomValue);
11    Serial.print("\tRun Count: ");
12
13    Serial.println(previousStats.runCount);
14
15    runCount = previousStats.runCount;
16
17  } else if (result == MBED_ERROR_ITEM_NOT_FOUND) {
18    Serial.println("No previous data was found.");
19  } else {
20    Serial.println("Error reading from key-value store.");
21    while (true);
22  }
23
24  // Update the stats and save them to the store
25  SketchStats currentStats { millis(), rand(), ++runCount };  
26  result = setSketchStats(statsKey, currentStats);
27  
28  if (result == MBED_SUCCESS) {
29    Serial.println("Sketch Stats updated");
30    Serial.println("Current Stats");
31    Serial.print("\tStartup Time: ");
32    Serial.println(currentStats.startupTime);
33    Serial.print("\tRandom Value: ");
34    Serial.println(currentStats.randomValue);
35    Serial.print("\tRun Count: ");
36
37    Serial.println(currentStats.runCount);
38  } else {
39    Serial.println("Error while saving to key-value store");
40    while (true);
41  }
42}

To finish the sketch, create

getSketchStats

and

setSketchStats

functions at the bottom of the sketch (after the

setup()

and

loop()

The

getSketchStats

function tries to retrieve the stats values stored in the Flash using the key

key

and returns them via the

stats

pointer parameter. Our

SketchStats

data struct is very simple and has a fixed size. You can therefore deserialize the buffer with a simple

memcpy

The

setSketchStats

function stores the

stats

data and assigns the key

key

to it. The key will be created in the key-value store if it does not exist yet.

1// Retrieve SketchStats from the key-value store
2int getSketchStats(const char* key, SketchStats* stats)
3{
4  // Retrieve key-value info
5  TDBStore::info_t info;
6  auto result = store.get_info(key, &info);
7
8  if (result == MBED_ERROR_ITEM_NOT_FOUND)
9    return result;
10
11  // Allocate space for the value
12  uint8_t buffer[info.size] {};
13  size_t actual_size;
14
15  // Get the value
16  result = store.get(key, buffer, sizeof(buffer), &actual_size);
17  if (result != MBED_SUCCESS)
18    return result;
19
20  memcpy(stats, buffer, sizeof(SketchStats));
21  return result;
22}
23
24// Store a SketchStats to the the k/v store
25int setSketchStats(const char* key, SketchStats stats)
26{
27  return store.set(key, reinterpret_cast<uint8_t*>(&stats), sizeof(SketchStats), 0);  
28}

5. Results

Upload the sketch, the output should be similar to the following:

1FlashIAPBlockDevice + TDBStore Test
2FlashIAP block device size: 1572864
3FlashIAP block device read size: 1
4FlashIAP block device program size: 32
5FlashIAP block device erase size: 131072
6Initializing TDBStore: OK
7Retrieving Sketch Stats
8Previous Stats
9        Startup Time: 12727
10        Random Value: 1514801629
11        Run Count: 13
12Sketch Stats updated
13Current Stats
14        Startup Time: 4285
15        Random Value: 2133170025
16        Run Count: 14

Note that the Flash memory will be erased when a new sketch is uploaded.

Push the reset button to restart the sketch. The values of the stats have been updated.

Previous Stats

which is retrieved from the key-value store now contains values from the previous execution.

Conclusion

You have learned how to use the available space in the Flash memory of the microcontroller to create a key-value store and use it to retrieve and store data.

It is not recommended to use the Flash of the microcontroller as the primary storage for data-intensive applications. It is best suited for read/write operations that are performed only once in a while such as storing and retrieving application configurations or persistent parameters.

Next Steps

Learn how to retrieve a collection of keys using TDBStore iterators via
```
iterator_open
```
and
```
iterator_next
```
Learn how to create an incremental TDBStore set sequence via
```
set_start
```
,
```
set_add_data
```
and
```
set_finalize
```
Learn how to use the 16 MB QSPI Flash on the Portenta H7