Time series database (TSDB) for Zephyr RTOS

Introduction:

In many of our microcontroller-based projects, efficiently storing and managing configurations and data from various real-world sources is crucial. A common example is collecting environmental data, such as temperature and humidity, and sending it to a backend server for visualization and analysis. To ensure accuracy and context, we need to store device configurations alongside the acquired data, each tagged with a timestamp to differentiate and organize the information in a real-world timeline.

In this context, a time series database (TSDB) becomes invaluable. A TSDB is designed to handle time-stamped data, making it ideal for tracking changes over time and analyzing trends. By using a TSDB, we can easily store, retrieve, and process data points, enabling sophisticated data analysis and decision-making based on historical and real-time data.

Challenges When Using Zephyr RTOS for Time Series Data

When working with Zephyr RTOS, handling data storage and retrieval for applications like environmental monitoring can be challenging. The constraints of embedded systems, including limited memory and storage, require careful consideration of how to manage data efficiently. Let’s explore the options we have and the associated challenges:

1. Raw Flash Read/Write

One approach to managing data in Zephyr RTOS is through raw flash read/write operations. This method, when implemented correctly, offers speed and high performance, as it allows direct access to the flash memory. However, it comes with significant drawbacks:

• Metadata Management: Raw flash operations require meticulous handling of metadata to keep track of records, which adds complexity to the implementation.
• Flash Wear Leveling: Without proper wear leveling mechanisms, flash memory can degrade over time, leading to potential data loss or device failure, especially in long-term deployments.

2. Filesystem: LittleFS

LittleFS, a filesystem designed for embedded systems, provides a more structured approach to data management in Zephyr RTOS. It leverages Zephyr’s rich set of filesystem APIs, making it easier to implement file-based storage:

• Pros: The use of files and directories makes it simpler to visualize and organize data. The APIs abstract away many of the low-level details, providing a more user-friendly interface.
• Cons: Despite its simplicity, LittleFS still requires management of metadata for file operations. Moreover, the overhead of managing a filesystem can be non-trivial in resource-constrained environments.

3. Sending Data Directly to a Server

An alternative to local storage is to send data immediately to a backend server for storage and processing:

• Pros: This approach eliminates the need for managing local storage, reducing the complexity of the device firmware.
• Cons: The primary drawback is the reliance on a constant network or radio connection. For battery-powered devices, maintaining such connectivity is impractical, leading to potential data loss during periods of no connectivity.

Porting a Real-Time Series Database

In more capable systems like those running Linux, we can utilize sophisticated time series databases (TSDBs) that offer advanced querying and data management capabilities. However, the constraints of Zephyr RTOS, including its low memory footprint and limited storage capacity, present a unique challenge. After exploring various options, we identified SQLite3 as a potential solution due to its lightweight nature and minimal resource requirements.

Porting SQLite3 to Zephyr RTOS

SQLite3 is a highly reliable, self-contained SQL database engine. It is known for its lightweight design, making it an appealing choice for embedded systems. However, it lacked native support for Zephyr RTOS. To address this, we embarked on the ambitious task of porting SQLite3 to Zephyr RTOS.

• POSIX Compliance: Zephyr RTOS offers a set of POSIX APIs, albeit limited, which provided a foundation for the porting effort. We extended these APIs to cover the necessary functionality required by SQLite3.
• Challenges: The primary challenge was ensuring that SQLite3 could operate within the stringent resource constraints of typical Zephyr deployments. This required careful optimization and testing to maintain performance while minimizing memory usage.

Conclusion: The Journey and Beyond

Porting SQLite3 to Zephyr RTOS was an adventurous journey. We successfully adapted the database engine to work within the constraints of an embedded RTOS, paving the way for more advanced data handling capabilities in Zephyr-based applications. However, while SQLite3 provided a powerful solution, it was not the most viable or optimal choice for all use cases, particularly those requiring minimal resource consumption.

Recognizing the need for a lighter solution, we ventured further into the realm of lightweight time series databases. Our efforts culminated in the development and porting of a super lightweight TSDB, with a RAM footprint of just ~600 bytes. This new TSDB is designed specifically for the constraints of Zephyr RTOS and is on its way to being integrated into the mainline Zephyr project.

This journey, filled with technical challenges and innovative solutions, exemplifies the exciting possibilities within the realm of embedded systems. We invite you to follow our continued exploration and contribution to the Zephyr community, as we push the boundaries of what is possible with low-resource devices. Stay tuned for more updates and detailed insights into our work with Zephyr RTOS and embedded databases!