This year I was gifted a Badger 2040W. This device is essentially a Pi Pico attached to a black and white e-ink display. The device is supposed to be used as a wearable badge. My use case was a little different however. I wanted a home automation controller.
This culminated in the restful-badger project, which permits an arbitrary number of RESTful ’tiles’ to be configured. Each tile can control and get state over HTTP.
The problem
One of the major design decisions for the software was that the tiles should be configurable. I chose JSON as the data format for my tiles, a given tile may look something like this:
{
"name": "office",
"image_idx": 1,
"action_request": {
"method": "POST",
"endpoint": "/v2/meross/office",
"json_body": "{\"code\": \"toggle\"}"
},
"status_request": {
"method": "POST",
"endpoint": "/v2/meross/office",
"json_body": "{\"code\": \"status\"}",
"key": "onoff",
"on_value": "1",
"off_value": "0"
}
}
A major downside of using JSON however is that it is expensive to parse on an embedded system. But there is a way to have our cake and eat it too. A way to avoid the runtime cost of parsing JSON but still use it.
Enter the humble byte array
Something that C is very good at parsing is bytes. If we could encode our JSON into a structured byte array and feed those bytes to the microcontorller instead, most of the heavy lifting would be done before we hit runtime. All that would be left to do on our microcontroller is some book keeping to make sure the right bytes end up in the correct place.
The correct place being a couple of c structs, which will look like this:
typedef struct RESTFUL_REQUEST_DATA_ {
char *method;
char *endpoint;
char *json_body;
char *key;
char *on_value;
char *off_value;
} RESTFUL_REQUEST_DATA;
typedef struct TILE_ {
char *name;
char image_idx;
RESTFUL_REQUEST_DATA *action_request;
RESTFUL_REQUEST_DATA *status_request;
} TILE;
Most programming languages can work with bytes, so we can leverage a high level language such as python to encode the JSON file.
We have 2 types of data to encode in our example JSON, an integer and several strings. Integer encoding is trivial and involves simply appending the integer to our byte array directly:
For strings we must capture both the length and the contents of the string in the byte array to be able to effectively book keep later on in C:
Building on these concepts we can now encode our entire JSON example:
Using our binary data
We now have a binary format that contains all the information we need to be able to reconstruct the important parts of the JSON in C, but how do we get it into our C program? One method is to simply hard code it.
By adding a little bit of printing logic to our python code we can output c syntax for a char byte array:
Decoding the byte array
To decode the byte array in C on our microcontorller, we must traverse it whilst keeping track of our position. We can do this by simply incrementing a variable that we will call ptr.
We still only have 2 types of data to concern ourselves with. In the case of a single byte value, such as image_idx, extraction is simple:
static const char tile_data[102] = {
0x06, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x01, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x74, 0x6f, 0x67, 0x67, 0x6c, 0x65, 0x22, 0x7d, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x7d, 0x05, 0x6f, 0x6e, 0x6f, 0x66, 0x66, 0x01, 0x31, 0x01, 0x30
};
uint ptr = 7;
char image_idx = tile_data[ptr++];
To reconstruct a string however, we must use the string length provided in tile_data to determine how many bytes to malloc and copy:
static const char tile_data[102] = {
0x06, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x01, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x74, 0x6f, 0x67, 0x67, 0x6c, 0x65, 0x22, 0x7d, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x7d, 0x05, 0x6f, 0x6e, 0x6f, 0x66, 0x66, 0x01, 0x31, 0x01, 0x30
};
uint ptr = 0;
uint8_t str_size = (uint8_t)tile_data[ptr++]; // First byte denotes string size
char *name = (char *)malloc(str_size * sizeof(char) + 1); // Allocate memory
strncpy(name, &tile_data[ptr], str_size); // Copy to alloced memory
name[str_size] = '\0'; // Null terminate
ptr += str_size;
We will perform this same string action many times so it makes sense to refactor it into a function call:
char make_str(char **dest, const char *src) {
uint8_t str_size = (uint8_t)*src++; // First byte denotes string size
*dest = (char *)malloc(str_size * sizeof(char) + 1); // Allocate memory
strncpy(*dest, src, str_size); // Copy to alloced memory
(*dest)[str_size] = '\0'; // Null terminate
return str_size + 1;
}
int main() {
static const char tile_data[102] = {
0x06, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x01, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x74, 0x6f, 0x67, 0x67, 0x6c, 0x65, 0x22, 0x7d, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x7d, 0x05, 0x6f, 0x6e, 0x6f, 0x66, 0x66, 0x01, 0x31, 0x01, 0x30
};
uint ptr = 0;
char *name;
ptr += make_str(&name, (char *)&tile_data[ptr]);
}
Piecing this all together we end up with the following code that can decode our packed tile data into c structs:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pico/stdlib.h"
typedef struct RESTFUL_REQUEST_DATA_ {
char *method;
char *endpoint;
char *json_body;
char *key;
char *on_value;
char *off_value;
} RESTFUL_REQUEST_DATA;
typedef struct TILE_ {
char *name;
char image_idx;
RESTFUL_REQUEST_DATA *action_request;
RESTFUL_REQUEST_DATA *status_request;
} TILE;
static const char tile_data[102] = {
0x06, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x01, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x74, 0x6f, 0x67, 0x67, 0x6c, 0x65, 0x22, 0x7d, 0x04, 0x50, 0x4f, 0x53, 0x54, 0x11, 0x2f, 0x76, 0x32, 0x2f, 0x6d, 0x65, 0x72, 0x6f, 0x73, 0x73, 0x2f, 0x6f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x12, 0x7b, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x7d, 0x05, 0x6f, 0x6e, 0x6f, 0x66, 0x66, 0x01, 0x31, 0x01, 0x30
};
char make_str(char **dest, const char *src) {
uint8_t str_size = (uint8_t)*src++;
*dest = (char *)malloc(str_size * sizeof(char) + 1);
strncpy(*dest, src, str_size);
(*dest)[str_size] = '\0'; // Null terminate
return str_size + 1;
}
void make_tile() {
uint ptr = 0;
TILE *tile = (TILE *)malloc(sizeof(TILE));
// Name
ptr += make_str(&tile->name, (char *)&tile_data[ptr]);
// Image idx
tile->image_idx = tile_data[ptr++];
// Action Request
RESTFUL_REQUEST_DATA *action_request = (RESTFUL_REQUEST_DATA *)malloc(sizeof(RESTFUL_REQUEST_DATA));
tile->action_request = action_request;
ptr += make_str(&action_request->method, (char *)&tile_data[ptr]);
ptr += make_str(&action_request->endpoint, (char *)&tile_data[ptr]);
ptr += make_str(&action_request->json_body, (char *)&tile_data[ptr]);
action_request->key = NULL;
// Status Request
RESTFUL_REQUEST_DATA *status_request = (RESTFUL_REQUEST_DATA *)malloc(sizeof(RESTFUL_REQUEST_DATA));
tile->status_request = status_request;
ptr += make_str(&status_request->method, (char *)&tile_data[ptr]);
ptr += make_str(&status_request->endpoint, (char *)&tile_data[ptr]);
ptr += make_str(&status_request->json_body, (char *)&tile_data[ptr]);
ptr += make_str(&status_request->key, (char *)&tile_data[ptr]);
ptr += make_str(&status_request->on_value, (char *)&tile_data[ptr]);
ptr += make_str(&status_request->off_value, (char *)&tile_data[ptr]);
printf("Name: %s\nImage Index: %d\nAction Request:\n Method: %s\n Endpoint: %s\n JSON Body: %s\nStatus Request:\n Method: %s\n Endpoint: %s\n JSON Body: %s\n Key: %s\n On Value: %s\n Off Value: %s\n",
tile->name,
tile->image_idx,
tile->action_request->method, tile->action_request->endpoint, tile->action_request->json_body,
tile->status_request->method, tile->status_request->endpoint, tile->status_request->json_body, tile->status_request->key, tile->status_request->on_value, tile->status_request->off_value
);
}
int main() {
stdio_init_all();
make_tile();
return 0;
}
Get the preprocessor to do it
We have now managed to convert a JSON file into a byte array, hardcode it and then decode it. We can go further however. If we are using a build tool such as CMake. We can simply instruct it to run the python script on our behalf:
set(JSON_FILEPATH "${PROJECT_SOURCE_DIR}/config/tiles.json" CACHE STRING "Location of tiles json")
execute_process(COMMAND "${PROJECT_SOURCE_DIR}/tools/json_to_c_array.py" "-f" "${JSON_FILEPATH}" OUTPUT_VARIABLE "TILE_DATA")
target_compile_definitions(badger PRIVATE
TILE_DATA=${TILE_DATA}
)
This symbol can then be referenced in our C code and replaced with our bytes by the preprocessor at build time:
static const char tile_data[] = {
TILE_DATA
};
The techniques discussed in this blog post were utilized and expanded upon to create restful-badger . Hopefully, someone will find them useful for their own project. Happy coding!