Portenta H7
Creating a Flash-Optimized Key-Value Store

Creating a Flash-Optimized Key-Value Store

This tutorial explains how to create a Flash-optimized key-value store using the Flash memory of the Portenta H7.
AUTHOR: Giampaolo Mancini, Sebastian Romero

Overview

This tutorial explains how to create a Flash-optimized key-value store using the Flash memory of the Portenta H7. It builds on top of the Flash In-Application Programming tutorial.

Goals

In this tutorial you will learn how to use the Mbed OS TDBStore API to create a Key value store in the free space of the microcontroller's internal Flash.

Required Hardware and Software

Instructions

MbedOS APIs for Flash Storage

The core software of Portenta H7 is based on the Mbed OS operating system, allowing developers to integrate the Arduino API with the APIs exposed by Mbed OS.

Mbed OS has a rich API for managing storage on different mediums, ranging from the small internal Flash memory of a microcontroller to external SecureDigital cards with large data storage space.

In this tutorial, you are going to save a value persistently inside the Flash memory. That allows to access that value even after a reset of the microcontroller. You will retrieve some information from a Flash block by using the FlashIAPBlockDevice and the TDBStore APIs. You will use the 

FlashIAPBlockDevice
class to create a block device on the free space of the Flash and you will create a Key-Value Store in it using the 
TDBStore
API.

Important: The TBStore API optimizes for access speed, reduce wearing of the flash and minimize storage overhead. TBStore is also resilient to power failures. If you want to use the Flash memory of the microcontroller, always prefer the TDBStore approach over a direct access to the FlashIAP block device.

1. The Basic Setup

Begin by plugging in your Portenta board to the computer using a USB-C® cable and open the Arduino IDE. If this is your first time running Arduino sketch files on the board, we suggest you check out how to set up the Portenta H7 for Arduino before you proceed.

2. Create the Structure of the Program

Let's program the Portenta with a sketch. You will also define a few helper functions in a supporting header file.

  • Create a new sketch named 
    FlashKeyValue.ino
  • Create a new file named 
    FlashIAPLimits.h
    to store the helper functions in a reusable file.

Note: Finished sketch its inside the tutorials library wrapper at:
Examples > Arduino_Pro_Tutorials > Creating a Flash-Optimized Key-Value Store > FlashKeyValueStore

3. Populate the Helper Functions

First let's add the helper functions to the 

FlashIAPLimits.h
header. This will determine the available Flash limits to allocate the custom data.

1/**
2Helper functions for calculating FlashIAP block device limits
3**/
4

5// Ensures that this file is only included once
6#pragma once 
7

8#include <Arduino.h>
9#include <FlashIAP.h>
10#include <FlashIAPBlockDevice.h>
11

12using namespace mbed;
13

14// A helper struct for FlashIAP limits
15struct FlashIAPLimits {
16  size_t flash_size;
17  uint32_t start_address;
18  uint32_t available_size;
19};
20

21// Get the actual start address and available size for the FlashIAP Block Device
22// considering the space already occupied by the sketch (firmware).
23FlashIAPLimits getFlashIAPLimits()
24{
25  // Alignment lambdas
26  auto align_down = [](uint64_t val, uint64_t size) {
27    return (((val) / size)) * size;
28  };
29  auto align_up = [](uint32_t val, uint32_t size) {
30    return (((val - 1) / size) + 1) * size;
31  };
32

33  size_t flash_size;
34  uint32_t flash_start_address;
35  uint32_t start_address;
36  FlashIAP flash;
37

38  auto result = flash.init();
39  if (result != 0)
40    return { };
41

42  // Find the start of first sector after text area
43  int sector_size = flash.get_sector_size(FLASHIAP_APP_ROM_END_ADDR);
44  start_address = align_up(FLASHIAP_APP_ROM_END_ADDR, sector_size);
45  flash_start_address = flash.get_flash_start();
46  flash_size = flash.get_flash_size();
47

48  result = flash.deinit();
49

50  int available_size = flash_start_address + flash_size - start_address;
51  if (available_size % (sector_size * 2)) {
52    available_size = align_down(available_size, sector_size * 2);
53  }
54

55  return { flash_size, start_address, available_size };
56}

4. Make the Key Store Program

Go to 

FlashKeyValue.ino
and include the libraries that you need from MBED and your header helper (
FlashIAPLimits.h
). The 
getFlashIAPLimits()
function which is defined in the 
FlashIAPLimits.h
header takes care of not overwriting data already stored on the Flash and aligns the start and stop addresses with the size of the Flash sector. You can use those calculated limits to create a block device and a 
TDBStore
on top of them.

1#include <FlashIAPBlockDevice.h>
2#include <TDBStore.h>
3

4using namespace mbed;
5

6// Get limits of the In Application Program (IAP) flash, ie. the internal MCU flash.
7#include "FlashIAPLimits.h"
8auto iapLimits { getFlashIAPLimits() };
9

10// Create a block device on the available space of the FlashIAP
11FlashIAPBlockDevice blockDevice(iapLimits.start_address, iapLimits.available_size);
12

13// Create a key-value store on the Flash IAP block device
14TDBStore store(&blockDevice);
15

16// Dummy sketch stats data for demonstration purposes
17struct SketchStats {
18  uint32_t startupTime;
19  uint32_t randomValue;
20  uint32_t runCount;
21};

In the 

setup()
function at the beginning you will have to wait until the Serial port is ready and then print some info about the FlashIAP block device (
blockDevice
).

1void setup()
2{
3  Serial.begin(115200);
4  while (!Serial);
5

6  //  Wait for terminal to come up
7  delay(1000);
8

9  Serial.println("FlashIAPBlockDevice + TDBStore Test");
10

11  // Feed the random number generator for later content generation
12  srand(micros());
13

14  // Initialize the flash IAP block device and print the memory layout
15  blockDevice.init();
16  Serial.printf("FlashIAP block device size: %llu\r\n", blockDevice.size());
17  Serial.printf("FlashIAP block device read size: %llu\r\n", blockDevice.get_read_size());
18  Serial.printf("FlashIAP block device program size: %llu\r\n", blockDevice.get_program_size());
19  Serial.printf("FlashIAP block device erase size: %llu\r\n", blockDevice.get_erase_size());
20  // Deinitialize the device
21  blockDevice.deinit();

After that, initialize the TDBstore (our storage space), set the key for the store data (

stats
), initialize the value that you will save 
runCount
and declare an object to fetch the previous values (
previousStats
).

1// Initialize the key-value store
2  Serial.print("Initializing TDBStore: ");
3  auto result = store.init();
4  Serial.println(result == MBED_SUCCESS ? "OK" : "Failed");
5  if (result != MBED_SUCCESS)
6    while (true); // Stop the sketch if an error occurs
7

8  // An example key name for the stats on the store
9  const char statsKey[] { "stats" };
10

11  // Keep track of the number of sketch executions
12  uint32_t runCount { 0 };
13

14  // Previous stats
15  SketchStats previousStats;

Now that you have everything ready, let's retrieve the previous values from the store and update the store with the new values. 

1// Get previous run stats from the key-value store
2  Serial.println("Retrieving Sketch Stats");
3  result = getSketchStats(statsKey, &previousStats);
4

5  if (result == MBED_SUCCESS) {
6    Serial.println("Previous Stats");
7    Serial.print("\tStartup Time: ");
8    Serial.println(previousStats.startupTime);
9    Serial.print("\tRandom Value: ");
10    Serial.println(previousStats.randomValue);
11    Serial.print("\tRun Count: ");
12

13    Serial.println(previousStats.runCount);
14

15    runCount = previousStats.runCount;
16

17  } else if (result == MBED_ERROR_ITEM_NOT_FOUND) {
18    Serial.println("No previous data was found.");
19  } else {
20    Serial.println("Error reading from key-value store.");
21    while (true);
22  }
23

24  // Update the stats and save them to the store
25  SketchStats currentStats { millis(), rand(), ++runCount };  
26  result = setSketchStats(statsKey, currentStats);
27  
28  if (result == MBED_SUCCESS) {
29    Serial.println("Sketch Stats updated");
30    Serial.println("Current Stats");
31    Serial.print("\tStartup Time: ");
32    Serial.println(currentStats.startupTime);
33    Serial.print("\tRandom Value: ");
34    Serial.println(currentStats.randomValue);
35    Serial.print("\tRun Count: ");
36

37    Serial.println(currentStats.runCount);
38  } else {
39    Serial.println("Error while saving to key-value store");
40    while (true);
41  }
42}

To finish the sketch, create 

getSketchStats
and 
setSketchStats
functions at the bottom of the sketch (after the 
setup()
and 
loop()
).

The 

getSketchStats
function tries to retrieve the stats values stored in the Flash using the key 
key
and returns them via the 
stats
pointer parameter. Our 
SketchStats
data struct is very simple and has a fixed size. You can therefore deserialize the buffer with a simple 
memcpy
.

The 

setSketchStats
function stores the 
stats
data and assigns the key 
key
to it. The key will be created in the key-value store if it does not exist yet.

1// Retrieve SketchStats from the key-value store
2int getSketchStats(const char* key, SketchStats* stats)
3{
4  // Retrieve key-value info
5  TDBStore::info_t info;
6  auto result = store.get_info(key, &info);
7

8  if (result == MBED_ERROR_ITEM_NOT_FOUND)
9    return result;
10

11  // Allocate space for the value
12  uint8_t buffer[info.size] {};
13  size_t actual_size;
14

15  // Get the value
16  result = store.get(key, buffer, sizeof(buffer), &actual_size);
17  if (result != MBED_SUCCESS)
18    return result;
19

20  memcpy(stats, buffer, sizeof(SketchStats));
21  return result;
22}
23

24// Store a SketchStats to the the k/v store
25int setSketchStats(const char* key, SketchStats stats)
26{
27  return store.set(key, reinterpret_cast<uint8_t*>(&stats), sizeof(SketchStats), 0);  
28}

5. Results

Upload the sketch, the output should be similar to the following:

1FlashIAPBlockDevice + TDBStore Test
2FlashIAP block device size: 1572864
3FlashIAP block device read size: 1
4FlashIAP block device program size: 32
5FlashIAP block device erase size: 131072
6Initializing TDBStore: OK
7Retrieving Sketch Stats
8Previous Stats
9        Startup Time: 12727
10        Random Value: 1514801629
11        Run Count: 13
12Sketch Stats updated
13Current Stats
14        Startup Time: 4285
15        Random Value: 2133170025
16        Run Count: 14

Note that the Flash memory will be erased when a new sketch is uploaded.

Push the reset button to restart the sketch. The values of the stats have been updated. 

Previous Stats
which is retrieved from the key-value store now contains values from the previous execution.

Conclusion

You have learned how to use the available space in the Flash memory of the microcontroller to create a key-value store and use it to retrieve and store data.

It is not recommended to use the Flash of the microcontroller as the primary storage for data-intensive applications. It is best suited for read/write operations that are performed only once in a while such as storing and retrieving application configurations or persistent parameters.

Next Steps

Tutorial Toolbox

Portenta H7

Contribute to Arduino

Join the community and suggest improvements to this article via GitHub. Make sure to read out contribution policy before making your pull request.
CONTRIBUTE

Missing something?

Check out our store and get what you need to follow this tutorial.
VISIT OUR STORE

Suggest Changes

The content on docs.arduino.cc is facilitated through a public GitHub repository. You can read more on how to contribute in the contribution policy.
EDIT THIS PAGE