Using a “Run To Completion Task” model in embedded programming

This is about the architecture or high-level structure of an embedded app.  You don’t always need an RTOS, or you can implement a very simple one.

References

I read these sources prior to this post.   This post is largely a hasty summary of these links.

Super Simple Tasker

Real Time For the Masses

Fearless Concurrency in your microcontroller

Context

Embedded programming is event/interrupt driven.

Embedded programming is multiprocessing:  devices are processors too.

Some chips e.g. Nordic NRF52 chips have event and task notions implemented in hardware.

Most chips have a NVIC (Nested, Vectored Interrupt Controller).

Traditional advice about ISR’s:

Interrupt service routines (ISR) should be kept short.   The references above dispute that dictum.

Traditional, typical software architecture

Your ISR’s are short and set flags (events).  Your main loop is an event loop:

main() 
  while(true)
    if (eventFlagA)
        taskA()
    if (eventFlagB)
        taskB()
    wait for event (WEV or WFI instructions on ARM)

Tasks are just functions.  They run in the main context (not in an ISR) where interrupts are enabled but no interrupt is being serviced.  Tasks are interruptible by any interrupt.  Interrupts come from concurrent device operations that your software Tasks start.  (And some chips call device operations “tasks”, that eventually generate “events” that trigger interrupts.)

Another example might be:

main()
   while true
      taskA()
      sleep(duration)
      taskB()
      sleep(duration)

sleep(duration)
   start Alarm
   while()
      wait for event
      if event is Alarm event
          return
      else
          handle other event, set flags, etc.

Again, your code dispatches ( an if sequence) on events.  You might need several sleep functions, each anticipating a different set of possible events.

Run to completion task architecture

main()
   initialize things
   schedule(taskA)
   while (true)
       wait for Event

taskA()
    do something useful
    schedule(taskB)

taskB()
   do something useful
   start a task on a device  (to generate event for taskC)

taskC()
   service device event
   schedule(taskA)
   
etc....

Here each task runs to completion.  That means it doesn’t sleep.  It does not mean that it can’t be interrupted by a higher priority task.  Each task is-a  ISR for an event/interrupt.

Here, it is important that each task runs to completion and insures that some other task is scheduled or that some device task is running and will generate an event/interrupt that will in turn schedule another task.  I.E. you must insure that you don’t sleep forever.

As discussed above for SST, the NVIC (Nested, Vectored Interrupt Controller) is the dispatcher.  Your code does not dispatch, the HW does (or an “OS” scheduler.)

You can schedule many tasks to run at the same priority at the same time in the future.  When they run, they execute pseudo-concurrently.  One will run, to completion, when the interrupt triggers, from its scheduled alarm.  The interrupt from the other task’s scheduled alarm will trigger at the same time, but since it is at the same priority, will be pended and be serviced only when the first task completes.

Difference between this and SST

SST described above has a broader notion of Task:

  • implemented in an “OS” i.e. by a Scheduler
  • has-a priority
  • that other Tasks can schedule, i.e. put in a list of ready tasks
  • that does not run on events, only when the Scheduler jumps to it.

But the “OS” scheduler can often be simpler: just set an Alarm on a HW timer, arranging that the task runs in response the the alarm, as the ISR of the HW timer.  You can use such a simpler scheduler when you don’t need the notion of a set of prioritized, software tasks.  The HW interrupt priorities can still insure that time critical code is run (to service devices) but your lowest priority tasks (the logic of your main application) don’t need to be structured as a set, each having different priority.

When you want to schedule for immediate, next execution, you can just pend the Alarm interrupt, or use a SWI?

Wrapping your mind around it

In the traditional architecture, the code structure shows the most likely (anticipated) sequence of events, and code executed in response (e.g. taskA, taskB.)  It might be easier for readers to grasp.

In the RTC Task architecture, it is harder for readers to see the sequence.  But it is easier to interpret the code as the “actions” for state machine transitions.  You can label each task to indicate its event, e.g. “onEventA.”

When to use this architecture

As discussed above for SST, when your application is a state machine.

 

 

 

 

Advertisements

Upgrading tools for next version of a radio chip NRF52810

A log of process debugging when I started using a newer version of the Nordic radio SoC ship:  nrf52810.

I am using:

  • Eclipse
  • gnu arm compiler
  • gnu mcu plugin for eclipse
  • nrfjprog
  • JLink

I found the need to upgrade nrfjprog and JLink.  When a newer chip comes out, old versions of the tools don’t understand the new chip.

The first symptom is that nrfjprog doesn’t recognize the chip and says something like “the connected chip is not the right family” even though the nrf52810 is in the 52 family and you specified –family NRF52.  You need to upgrade to the latest nrfjprog version.

When I overwrote the existing nrfjprog directory, I seemed to have problems with nrfjprog not finding the “DLL”, meaning the JLink shared library .so.   I seemed to fix that by deleting the directory and installing fresh.

The next symptom might be, if you use an old debug configuration with device equal to nrf52832, is that the debugger flashes the chip, but then your target program crashes.  You need to specify the correct device in the debug configuration.

The next symptom is that you specify in an Eclipse debug configuration, under the Device tab, that the device is nrf52810_xxAA, but the debugger gives “Failed to set device (nRF52810_xxAA). Unknown device selected?”  You need to upgrade to the latest JLink version.  That involves:

  • download a JLink (say .deb) installer and run it
  • update Eclipse settings for JLink

See https://gnu-mcu-eclipse.github.io/debug/jlink/ for one way to do it (in Window>Preferences>SEGGER J-Link.

Change the jlink_path variable in Eclipse say from “JLink_v60i”to “JLinkv630f”.  In the Debug config>Debugger tab>Executable field, choose the “Variables” button.  Expect a list of variables.  Scroll to and select  “jlink_path” and choose “Edit variables” button.  Expect another list.  Scroll to an select “jlink_path” and choose “Edit…” button.  Change value, and choose OK, OK, etc.

Also, you might need to change the “jlink_gdbserver” variable from “JLinkGDBServer” to “JLinkGDBServerCLExe”.  The name has changed, by the addition of “CLExe”?  Note that JLinkGDBServerExe has a GUI, but the GNU MCU plugin says to use the command line version JLinkGDBServerCLExe

After you do all that, you should be able to execute a debug configuration and flash your app to the device.

My app still crashes.  Now I find that I need a different softdevice for this chip, S112 instead of S132.  I need to change my build config:

  • use headers for S112
  • use linker script for S112

 

Why change to nrf52810?

For me, because it is cheaper in dollars per unit.  Fanstel sells a module BT832A for less than $4, which is cheaper than the Waveshare nrf518122 that I was using.   I am already developing and testing on the nrf52DK which uses the NRF52 family nrf52832, and which is the dev kit that support nrf52810.

Here is a comparison of the family members.

For me, the main difference is: has less memory (192k or ROM instead of 256k.)  So I will need to change my build process to include optimization for small size, and to build for a different Softdevice.  That will be costly in development time for a new build process.

I can use the nrf52810 because my app is small, doesn’t use many periherals (none that are missing on the 52810), and doesn’t use many BT features (only one connection.)

 

 

 

Modern build systems for embedded (radios)

About

I ramble about my experiences with build systems for developing for embedded systems.  I use radio, ARM, SoC’s (Nordic NRF5x family), but the discussion might apply more generally.  I try to provide links, so you can explore.

Summary: I gradually migrated to CLI command line tools, using more recent tools instead of make.

Background

History of what build systems I have used:

Those tools are in order of age,  ease of use, and speed.

For example, make is:

  • ancient
  • hard to use
  • slow

while Meson is (?):

  • recent
  • easy to use
  • fast

 

Factors to consider for a build system in this context

  • supports cross compilation
  • integrated with your IDE
  • handles frequent changes to underlying SDK (Nordic’s)
  • handles very many target embedded systems (platforms)

You might have many targets if you  have a long-lived product that you will upgrade with new hardware:

  • different vendors e.g. Nordic vs Silicon Labs)
  • chip families NRF51x/52x
  • versions within a family e.g.  52832, 52810, 52840
  • devices and protocols that you use (i.e. Softdevice with Bluetooth, or other, or proprietary protocols.)

Makefiles

They get very large.  They are not easy to understand or modify.   But the ones Nordic provides are templates:  you will need to modify them, and re modify them whenever you change the SDK.

Eclipse CDT managed build

Eclipse projects are distinguished by kind:

  • “CDT managed build”: Eclipse generates makefiles.
  • “Makefile managed build”: you provide and edit a makefile

CDT managed build uses a GUI.  You must learn the structure of the GUI instead of the structure of a text configuration file (the makefile.)

Very few people seem to use it in the niche of embedded radios.

Nordic provides a tutorial for  using Eclipse IDE for development, but the examples don’t use Eclipse “CDT managed build”, just “makefile managed build.”  Again, that means you must edit the makefiles.

CMake and the Nordic SDK

Several people have implemented and published their own CMake scripts (macros) specific to the Nordic SDK:

Apparently Nordic realizes the benefit of newer build systems.  Nordic CMake macros:  IMO they are not very modular or extendable.

I found that my own scripts are fragile wrt SDK version.

There are a plethora of options for integration with the IDE:

The fact that there is a plethora is discouraging.  It requires much reading to decide which to choose.  It tells me there is no settled easiest way.

I disliked the projects generated by CMake.  They were too different from my usual way of structuring a project.

Often I just used Eclipse as an editor, debugger, and  version control wrapper, and built using the command line.

Meson

(I haven’t used Meson yet, I am still exploring it.)

One comparison of CMake and Meson.

Apparently Meson supports cross compiling: their documentation.

Meson is written in Python.  That is encouraging, that they used a high-level, modern language to implement what is a very complex task.  I would guess that one might even be able to understand any exceptions in the Meson implementation.

Apparently integration with IDE is in the future.

There is a Eclipse plugin editor for meson files.

The big picture

You need a mental model of the build process.  You acquire that mental model after long experience.  Learning and discarding build systems can help you form that mental model.  The newer build systems more clearly represent the model.  For example, using make you must remember what a “dependency” line looks like (foo: bar) while in newer build systems, a dependency might be labeled with text that says “dependency.”

 

 

 

 

 

Nordic Timers

This is an overview of timers on Nordic radio SoC.

On the NRF52 family of radio SoC, there are these Timer devices:

  • RTC: low power, and low frequency
  • Timer: high power, and high frequency

The RTC is misnamed.  RTC is Nordic’s  acronym for “Real-Time Counter” but more generally RTC means “real time clock” and provides calender like functions: time measured in years and days.  The Nordic RTC is only 24-bit, so  it typically rolls over frequently, say in minutes.  Whereas most people expect a RealTimeClock to never roll over, practically speaking.

On the NRF52, there are three RTC’s, RTC0, RTC1, and RTC2.

  • RTC0 is used by the Softdevice
  • RTC1 is used by the Nordic app_timer module
  • RTC2 is not generally used by the Nordic SDK

So your app should use RTC2 as the first choice, so that if your app also uses Softdevice or app_timer, there is no conflict.

Nordic’s app_timer module implements virtual Timers.  That is, many Timers are implemented in software on one real Timer (i.e. one compare register of RTC1.)

Timers might better be called Alarms.  A Timer is implemented on a Clock and an Alarm, where a Clock is a counter that rolls over and an Alarm is a compare register that generates an event/interrupt when the compare register matches the clock.

Each of the RTC’s has a counter and multiple compare registers.  Thus you can implement many real Timers on each RTC.

A real Timer is implemented on hardware devices like the RTC.  It consumes no cpu cycles while it is running, and few cpu instructions to set it up.

Because virtual Timers are implemented partially in software, it takes a few more cpu cycles than a real Timer.  But that’s not important.  What is important is that the virtual Timers be implemented correctly.  There are outstanding bug reports on Nordic’s app_timer implementation.  And the test suite is no public.

Other software kits such as freeRTOS implement virtual Timers.  These might be more likely to be correct since they are open source and widely used?

Algorithms for setting an Alarm on a Clock are not trivial but are well understood.  It requires careful use of modulo arithmetic.  Modulo arithmetic is “clock like”: results of operations roll over, loosely speaking.  Correctly setting an alarm on a clock also requires use of Lamport’s rule.  Finally, a hardware implementation of a clock has limitations that the software (the Timer driver) must account for: a compare register cannot be set too near the current counter value, else the hardware event might not occur in a timely fashion.

 

 

Pragmatics of nano power radios

This is a brief note about high level concerns with nano power radios, solar powered without batteries.

Don’t rely on this, study it yourself, especially until I add proper links.  Some of it is just crude notes, even speculation.

Other References

A note at Mouser about ultra low power mcu design.

Context: nano power

The power supply:

  • provides low average current, around 1uA
  • has no large reserve
  • is is expected to provide zero current often (say every night)

For example:

  • solar power with a capacitor
  • no battery
  • indoor light
  • solar panel smaller than a credit card

Overview

  • radio is duty-cycled
  • a voltage monitor/power supervisor and load switch chip provides clean reset/boot
  • boot sequence must be short and monitor mcu Vcc
  • use a power budget for design
  • use synchronization algorithms
  • testing is hard
  • over voltage
  • energy harvesting

Duty-cycled radio

The radio is sleeping most of the time.  When sleeping, a low-power timer runs to wake the system.  The sleeping radio cannot wake the system when it receives.

Example: the system may sleep for a few seconds, and be awake (with radio on) for about a millisecond.  That is, the duty cycle is around 1000.

Voltage monitor/Load switch

A microprocessor (in a radio SoC) needs a fast-rising voltage to boot cleanly.  Otherwise it may enter a state where it consumes power without booting. (Fibrillating?)  It may be in that state for a long time.  The solution is to use an external voltage monitor aka power supervisor aka reset chip.  E.g. TPS3839 (ultra-low power of 150nA.)

You can’t just connect the voltage monitor to the reset line of the mcu.  Otherwise, the mcu will still consume power while its reset line is held in RESET state. (Between the time voltage is high enough for the voltage monitor to have active outputs say 0.6V and the time the voltage is high enough to run the mcu say 1.8V.)  An mcu may draw a fraction of a milliamp while held in reset.

So the voltage monitor drives a high-side load switch that switches power (Vcc or Vdd) to the mcu.  I use the TPS22860.  (You can switch ground i.e. low-side with a NMOS mosfet but it’s not so easy to design your own high-side switch.  You can’t switch the low-side of an mcu because many pins may leak to ground?)

Voltage monitor hysteresis and boot sequence

The voltage monitor asserts its Out (sometimes call Not Reset) at a certain threshold voltage but then unasserts if the voltage falls below the threshold a certain amount called the hysteresis.  While the mcu is booting, it must not use so much current that Vcc falls below the hysteresis.  The boot sequence typically does a bare minimum, then checks Vcc, and sleeps until Vcc is much beyond the the minimum.  That is, allowing time for the ‘challenged’ power supply to catch up and store a reserve.  Only then does the software proceed to use the radio, duty-cycled.

You could use a voltage monitor with higher hysteresis.  But they don’t seem to make them.  The hysteresis of the TPS3839 is only 0.05V.  You can play tricks with a diode/capacitor on the input of the voltage monitor to make it seem to have a higher hysteresis (to delay longer before un asserting.)  And there are application notes on the web about adding hysteresis to voltage monitors.  But they seem to apply to older voltage monitor designs, and don’t seem to apply to the ultra-low power TPS3839 (which samples Vcc.)

Also, you could design your own voltage monitor with more hysteresis.  For example, see the Nordic solar powered sensor beacon.  That uses a few mosfets to provide a 0.2V hysteresis (say booting at 2.4V and resetting at 2.2V).  Unfortunately, they don’t seem to have exactly documented how the design works.

Power Budget

A power budget calculates the average current of a system, given certain phases of certain durations, where each phase uses certain devices/peripherals.

Here the main phases are:

  • sleeping (say 1.5uA for 1 second)
  • radio and mcu on (say 6 mA for 1 milli second)

You can almost ignore any phases where only the mcu is active, it should be a small portion of your budget.

A discussion at Digikey.

Synchronization algorithms

These make your units wake at the same time, so they can communicate with each other.

A beacon is usually unsynchronized.  The thing that hears a beacon (e.g. a cell phone) has  enough power to listen a long time.  You also might not need to synchronize if you have a “gateway” that is always powered and listening.  (See Zigbee.)

This seems to still be a research topic, there is much literature to read and few open source code examples.

Testing is hard

With such a challenged, nanopower supply, testing is hard.  A bug may exhaust power so that the system brown out resets, losing information about what happened.

Most hardware debuggers make the target consume more power than the power supply can provide?  TI seems to have ultra-low power debugging tools, but I haven’t studied them.

You can implement fault/exception handlers that write to non-volatile flash so that you can subsequently connect to a debugger and read what happened.   Default handlers typically just infinite loop (which will brown out reset.)  Typical handlers will do a soft reset.  Unless your app makes a record or communicates that, you might not even know the system reset itself.

Agililent (formerly Hewlett-Packard) sells expensive instruments for monitoring power consumption.  These may tell you you when (in relation to other events) you are consuming more power than you expect, but not exactly why.

Over voltage

A solar cell is a current source, and provides a variable voltage.  Voc is voltage open circuit (when your capacitor is fully charge.)  It can exceed the Vmin of your radio (typically 3.6V.)

Voltage regulators (such as shunt regulators) that prevent that are themselves current wasters.

You can choose a solar panel whose Voc is less than the Vmin, but there are few choices in that range (Voc < 3.6V, Vope around 2.4V, for indoor light.)  Or you can require that your solar panel never be exposed to strong light.

I haven’t found a zener diode that would clamp the voltage to 3.6V, and not leak much, at such nano currents.

Energy Harvesting

This is another buzzword, but good to search on.  It often means: with a single coin cell battery.

Energy harvesting chips are available.  They solve some problems you might not have, such as over-voltage protection, or voltage boosting.

It often refers to other power sources such as heat or vibration.  Those power sources are usually even smaller than solar (light) power, but solar power is episodic (diurnal.)

Solar power in different setting differs by orders of magnitude.  Direct sun is ten times stronger than outdoor, blue-sky shade, which is ten times more than strong indoor light, which is ten timer more than  dim indoor light.

 

 

Writing custom libraries for Energia (Arduino)

This is just about the pragmatics of: where do I put source files so that they are a shared library?

Custom: one you write yourself.

Library: a set of C++ source files (.h and .cpp) that you want to share among projects.

The simplified Energia/Arduino view

Outside the simplified Energia/Arduino world, libraries would be in a separate, shared directory and they would be pre-compiled into an object and separately linked into your projects.  In the Energia/Arduino world, that is all hidden.

Also, in the Energia world, a library seems to be a zipped directory of source files that follow some conventions that identify the version and documentation of the library.   So you can share the library.  I don’t know what the conventions are.  But if you are going to share your custom library, you should follow the conventions, and zip it up.  Then others can use the simplified user interface for installing zipped libraries.  Here, I don’t bother with the zipping.

Creating a custom library

Briefly, you just need to create your source files in the place that Energia looks.

Find where your sketchbook directory is:  In Energia choose “Sketch>Show Sketch Folder.”  Expect a file browser dialog (the Finder on the Mac) to show you the directory.

You will see a sub directory named “libraries”, and it will probably be empty.  (I don’t know where Energia keeps all the other pre-installed libraries.)

In that directory, create a directory with the name of your library e.g. “PWM”.

In the “PWM” directory, create your .h (and maybe .cpp) files, e.g. “pwm.h”

Now switch back to Energia and select “Sketch>Include Library>”   Expect a hierarchal menu to appear.  Expect to see “PWM” in the “Contributed libraries” section of the menu.

You can also choose “Sketch>Include Library>Manage Libraries”.  Expect a browser kind of window to open.  You should be able to browse to a line saying “PWM version unknown INSTALLED”.  (In my opinion, this should not be called “Manage Libraries” because it seems all you can do is view a list of the libraries.)

(Note that Energia expects at least one source file in your library directory.  Until then, Energia may give an error “Invalid library found in….”)

Referencing the library

In your main sketch “#include <pwm.h>”

Then define an instance of the PWM class and call its methods.

Developing and managing your library

You can just edit the files in place, using another editor.   When you use Energia to “verify” the main sketch that uses the library, it will recompile your changed library.

By managing I mean: copy the files out of the sketchbook folder to a safer, more shared place.  The sketchbook is in /Users/foo/Documents/sketchbook (on a Mac).  I prefer to put them under source control in a “git” folder, or in the “Dropbox” folder, so when I am done developing, I copy the library folder somewhere else.

I suppose you could use git in that directory, and when you are done, commit and push that repository to a your shared (master) repository on github.

Brief Summary

A library is just a named directory in the directory “sketchbook/libraries”.  You can create a library yourself using a file browser and editor.

0xFFFFFFFE, reclaim_reent() error symptoms of embedded programming

This is a report of one cryptic symptom (and possible) fixes you might encounter when you are embedded programming.  I report it because I have experienced it more than once and always forget what it means.

When you are trying to flash your embedded mcu,  the debugger seems to download to the chip, the debugger starts  and then stops showing a stack trace something like this:

0xFFFFFFFE
reclaim_reent()

Usually you expect the debugger to stop at main() and wait for you to tell the debugger to run (but that depends on whether you have configured your IDE and debugger to breakpoint at main.)

It might mean (not a program bug, a process error):

  • your linker script <foo>.ld describes the memory of your chip incorrectly
  • you haven’t erased the chip’s ROM yet

About the latter.  I am not sure, but modules you buy might already be flashed with a program such as a DFU bootloader, and are configured to protect a debugger from overwriting that code in ROM.  For example, on the Nordic NRF51, to remove the protection and erase all of ROM so that you can then use the debugger:

 nrfjprog --recover --family NRF51