Why Spangap exists

(Skip to getting started if you’re impatient.)

It’s one of those things that got out of hand. It started with network camera firmware that I wanted to create for the Seeed Studio Sense board. The existing software consisted of very crude proofs of concept that were only that: barely a web interface, slow and incomplete. When I started work I soon ran into some familiar problems I’ve battled with before when building things for the ESP32.

Multitasking: the obstacles

The ESP-IDF framework that most other ESP software (including the Arduino environment) lies on top of contains a fork of RTOS, a preemptive multi-tasking OS. But it’s not like Linux or other more featured operating systems. Partly this is because the hardware doesn’t have memory management: tasks can never be isolated from one another as they share the same addressable memory. But within that constraint, RTOS allows for multiple tasks to run concurrently.

However, it does not provide a lot of the niceties a modern programmer is used to. There’s no kernel and just a small set of bare methods for tasks to communicate. They can define queues of fixed size objects that other tasks can send them, “give” or “take” binary flags called mutexes, share ring buffers and exchange nudges called notifications — and that’s essentially the extent of it. So right off the bat, having tasks interact meaningfully is likely to involve lots of custom engineering.

But even if that were not the issue, one of the larger problems appears when a task is waiting for other tasks. If there’s only one thing the outside world could want to tell a task, it can just wait for that one queue or stream to have data, or for that one notification to come in, or flag to be returned. While it waits, it is blocking, which sounds bad but is actually a good thing. Blocking means the task uses 0% CPU until the one thing it waits for happens. This allows other tasks to use CPU cycles, and ESP-IDF’s power management can even put the CPU to sleep if no other tasks need to do anything.

To be useful in a more complex interplay however, tasks generally need to wait for any number of things: some process might continue, some other task might want something, the user might press a button, a setting might change in the web interface. Lots of things can happen and the programmer has no way of knowing what will happen first. So to avoid blocked tasks waiting for things that will never happen, programmers have to resort to polling: having a task wait for only 10 ms or so at a time, then checking on everything that might have happened and going back to sleep for the next cycle.

And then when there are five of these tasks, the CPU is in use all the time. Meanwhile a chain of tasks that depend on one another have to wait for the event to cascade back and forth: two tasks depending on an intermediate one are now a minimum of 60 ms round-trip apart.

And then there’s the obscure stuff. For example: ESP32s often have two types of RAM — the DRAM that’s built into the CPU itself, and PSRAM that’s hooked up to it via an SPI bus. This PSRAM might also be in the same chip package, but it’s slightly slower and, crucially, it is briefly inaccessible when the chip talks to its flash memory that shares the same bus. Typically an ESP32-S3 has a few hundred kilobytes of DRAM and multiple (usually 8 or 16) megabytes of PSRAM. Long story short: park files in flash, and all tasks that use PSRAM for their stacks will crash when they access it. If a few tasks need to access files, the system runs out of DRAM and the only way out is to combine multiple tasks back into fewer and fewer — and we’re back to square one.

What was needed was a way for tasks to communicate, where they would not need polling and where all the moving parts were programmed once instead of re-implemented for every two tasks that needed to talk. Once communication was solved, one task could solve our obscure problem by having a stack in the rare DRAM, talking to the flash filesystem as a service to other tasks.

I’ve realized that all these problems were essentially solvable before. But now, as I really wanted proper multitasking for my camera, I was getting accustomed to using LLM coding assistants, so I was a little less intimidated by the sheer amount of work this would involve. Don’t get me wrong: there’s plenty of ways to use these tools to create terrible code. But when I kept a close eye, I could now watch over the architecture and babysit the machines that did most of the gruntwork — which would have made coding this to its present state take a few years, instead of a few months.

ITS

First order of business was a way to have tasks open and close two streams, one in each direction, and having all tasks block waiting for the same primitive: an RTOS notification. Want to write to a stream to some other task? Also notify. Want to send a queue message? Also notify. That way, all tasks could block waiting for notifications and nothing needs to poll.

That library, called ITS — for Inter-Task Streaming, a misnomer as it can now do much more than just help tasks set up streams — worked well, and it forms the basis upon which Spangap is built.

These streams allowed for proper separation of concerns and spreading of the workload. One task is talking to the network stack, another is serving HTTP(S) files, yet another does the WebRTC packing and unpacking and yet another picks up the camera data. It took a bit of work, but now I could record while the browser was streaming a different video from SD card, while typing at a command line and watching the ESP log in another window within the main browser window.

The value store

In the process, I ran into the need for a value store that was a little more featured than ESP-IDF’s built-in flash-backed key-value store. I synchronized this to the web framework’s value store using snippets of patch JSON travelling up and down a data channel on the WebRTC link I now had with the browser. Now one could change a setting in a reactive page in the browser, and the code on the ESP that subscribed to this value would get a callback to change a value.

At first it was only ITS that was going to be the common shared thing that others could also use. But as I went, I figured that most of the work in this camera firmware was actually going to be duplicated by anyone that wanted to make something that uses the browser to do or see things on the ESP.

Meanwhile I was getting more and more interested in playing with Reticulum for radio mesh networking. ESP32s are small, low-power and cheap, so they make excellent mesh nodes. But the people building firmware for embedded Reticulum routing — and those coding for all other mesh protocols — had clearly run into the same issues I was solving: one massive main loop polling everything, often running at a few Hz, barely extensible and incapable of supporting responsive interfaces.

And that, in a nutshell, is how early April’s “I should be able to multi-task on this cool microcontroller” exploded into the universe it is as I write this in June of 2026.