Building SoC Soft Real-Time Systems

The ever-increasing compute capabilities of SoC ("system on chip") devices that now match top-of-the-line desktop systems of a decade ago create unprecedented opportunities for ubiquitous embedded and "edge" computing. Embedded hardware can now take on a greater share of data processing. However, systems that do not run operating environments specifically designed for real-time applications (such as RTOS) are hampered by tail latencies associated with the overhead of running "consumer" OS (such as Linux). While tail latencies can be reduced by scaling up hardware specifications and restricting the process space, such devices can best serve as "soft" real-time systems, outside of critical "hard" real-rime applications where system response should be known with certainty.

Design and implementation of hard real-time systems unencumbered by OS and user-space software will be the subject of a future post (please check back in later). In this article I discuss a continuous environmental monitoring project based on the RP3B platform. I have completed this project as an example of a high-availability "soft" real-time system that can handle highly heterogeneous data streams.

In my implementation, an RP3B is used to take periodic PM2.5 and PM10 particulate matter measurements (concentrations of particles less than 2.5 and 10 microns, respectively, see EPA Particulate Matter (PM) Basics) as well as temperature and humidity readings. The readings are captured into a NoSQL database for data processing, and displayed in real time using an OLED display and 3 LED indicators (green for normal conditions, yellow for borderline-high, and red for high particle concentrations -- see the image and video on the left). By the time of this post, the system had been in continuous operation for over 13 months.

RP3B interacts with peripherals via GPIO using the TI 1-wire protocol for the DHT11 temperature and humidity sensor, serial emulation and the custom OEM protocol for the SDS011 particulate matter sensor, and the i2c protocol for the OLED display. Numerous examples exist in public domain for interfacing with these devices using both "computer-on-chip" and micro-controller systems. The main challenge is the integration of multiple peripherals into a high-reliability low-latency system despite the significant dispersion in response and processing times for different sensors, devices, and signals.

I solve this problem by implementing a dedicated interrogator or output process for each device. The interrogators buffer continuous data streams for later retrieval, and perform inter-process data exchange via Unix file-system sockets and my custom data-exchange protocol. The SDS011 interrogator periodically re-starts the particulate matter sensor every 5 minutes, taking 30 measurements in one minute, then shuts it down. The temperature and humidity process takes continuous readings and streams them to the data integration process when requested. The data integration process displays status on the 4-line OLED display via i2c, and controls three air-quality indicator LEDs. While SDS011 measurements are in progress, one of the LEDs is blinking to indicate the quality of the latest reading, switching to the corresponding constant state when the interrogation stops (click the image on the left for a video). The data integration process captures readings into a non-relational NoSQL database if they change in excess of a user-specified variance parameter. An analytics workstation on the local network randomly queries the database.

Note the dimming of the display when the SDS011 is in operation. This effect had developed about 6 months into the continuous operation of the device, and was due to the increased power consumption of the SDS011 internal fan that shares the bus with the OLED display. Display brightness returns to normal as soon as the particulate sensor fan is shut off.

Example of high PM concentration readings