Real-time programming notes

Most bugs in real-time systems are not crashes. They are a 4 µs overrun that happens eleven times an hour, on one machine out of forty, when the network is busy. The system looks fine. It is fine, in the sense that a bridge that sags occasionally is fine, until the day the sag matters.

So the whole discipline is really two habits: knowing what your deadline is, and refusing to average. Everything below is in service of those two habits.

Real-time is a bound, not a speed

A system is real-time if correctness depends on when the result arrives, not only on the value. That is the entire definition. A 60 fps renderer is fast and not real-time; a 10 Hz valve controller that always answers within 3 ms is real-time and slow.

The useful split is by consequence, not by technique:

Two words carry the rest of the vocabulary. Worst-case execution time (WCET) is the longest the work can take, not the longest you have seen. Jitter is the spread around the intended moment — the difference between a loop that runs every 1.000 ms and one that runs every 1.000 ms ± 80 µs. A system with a 900 µs average and a 4 ms worst case is a 4 ms system that happens to be fast most of the time.

The budget

Write it down before writing code. A 1 kHz loop has 1,000 µs between activations, and that number is a container for everything else:

The reserve is the design. If the sum of the parts equals the period, the system is already late — it just has not been observed yet. Sixty percent utilisation is a reasonable target for a forgiving loop and far too generous for a hard one.

Utilisation is the first sanity check, not the proof. For a set of periodic tasks with C_i execution and T_i period, U = Σ(C_i/T_i), and if U > 1 no scheduler can save you. Below 1, feasibility depends on priority assignment and blocking — rate-monotonic scheduling is optimal among fixed priorities up to about 69% utilisation for large task sets, and exact response-time analysis is what you actually need above that.

Where determinism dies

Latency rarely comes from the code you wrote. It comes from the machine noticing something else. Ordered roughly by how often it is the answer:

The last row is not a joke. A loop that reads a configuration file from a network-mounted home directory at startup will hang for exactly as long as the file server does, and it will do it once, at 3 a.m., in a way that looks like a scheduler bug. Anything on the hot path should be on a local disk, in memory, or better still, decided long before the loop starts.

What the kernel will and will not promise

Much of PREEMPT_RT is in mainline now — the realtime preemption work landed across recent releases, with the bulk of it around 6.12 — and it changes the character of the kernel substantially: spinlocks become sleeping locks, most interrupt handlers become threads, and the worst-case scheduling delay on an idle machine drops from milliseconds to tens of microseconds. What it does not do is make a general-purpose distribution into a control platform by itself. That is the other half of the work, below.

Priorities are a promise you can break

SCHED_FIFO threads run until they block or yield, and a higher-priority thread preempts immediately. That is the contract, and it has two consequences people forget. A busy SCHED_FIFO thread at priority 99 with no blocking call will lock the machine — including the shell you would use to kill it. And the kernel enforces a budget on real-time threads by default: sched_rt_runtime_us is 950,000 of every 1,000,000 µs, so RT tasks get 95% of each second and everything else gets the rest.

That throttle is why a runaway loop usually does not take the machine down, and why a control loop occasionally sees a 50 ms stall if it tries to use more than its share. If you disable it with sched_rt_runtime_us=-1, you are taking responsibility for liveness with no safety net: keep the loop blocking every cycle, add a watchdog thread at a lower RT priority, and make sure the machine can still be reached.

SCHED_OTHER     CFS. Throughput-oriented. A correct choice for
                nothing with a deadline.
SCHED_FIFO      Fixed priority, run until block. The workhorse for
                a single control thread.
SCHED_RR        Same, with a timeslice. Useful when several equal
                priority threads must share a core.
SCHED_DEADLINE  runtime / deadline / period, with admission control.
                The scheduler refuses an infeasible task set -- which
                is a feature, and occasionally a surprise at 2 a.m.

SCHED_DEADLINE is underused. Expressing a task as “2 ms of CPU every 10 ms, deadline 10 ms” is a clearer statement of intent than a priority number, and the kernel's admission control turns an overloaded design into a failed sched_setattr() at startup rather than a mystery at runtime. The constraint is runtime ≤ deadline ≤ period, and the sum of utilisations must fit the core.

Priority inversion, concretely

Three threads: a low-priority logger holds a mutex while writing to disk, a high-priority control thread needs that mutex, and a medium-priority task is runnable. The control thread blocks on the mutex, the logger cannot run because the medium task is using the CPU, and the highest-priority thread in the system is waiting on the lowest. Nothing is broken; the design is.

The fix is priority inheritance: while the high-priority thread waits, the holder inherits its priority and gets scheduled. In C, that is one attribute, and it must be set on the mutex before anyone uses it:

pthread_mutexattr_t attr;
pthread_mutexattr_init(&attr);
pthread_mutexattr_setprotocol(&attr, PTHREAD_PRIO_INHERIT);
pthread_mutex_init(&lock, &attr);
pthread_mutexattr_destroy(&attr);

On a PREEMPT_RT kernel, much of the locking inside the kernel has this property already. In your own code, nothing does unless you ask. And the better fix is usually structural: if the high-priority thread never shares a lock with anything, there is no inversion to inherit from. That is the argument for the lock-free queue in §07.

Isolation: what to put on the kernel command line

A real-time thread does not need a fast machine. It needs a machine that stops doing other things to it. On x86-64, a kernel command line that reflects that intent looks like this:

preempt=full
isolcpus=2-7
nohz_full=2-7
rcu_nocbs=2-7
irqaffinity=0,1
intel_idle.max_cstate=1
processor.max_cstate=1
intel_pstate=passive
nosmt
tsc=reliable
nowatchdog

Pin the loop to an isolated core, and pin it explicitly — sched_setaffinity() or taskset -c 4. Do not rely on isolcpus to do the placement for you; it restricts, it does not assign. Then check that no interrupt landed there:

grep -E 'CPU|intr' /proc/interrupts | head -20
cat /proc/irq/*/smp_affinity_list | sort | uniq -c
ps -eLo pid,tid,psr,cls,rtprio,comm --sort=-rtprio | head
chrt -p $(pgrep control_loop)
cat /sys/devices/system/cpu/cpu4/cpuidle/state*/disable

Memory: the allocator is in your hot path

Dynamic allocation is not banned because it is slow. It is banned because its worst case is not bounded: a malloc() may take a lock, ask the kernel for pages, and fault them in; a free() may return memory to the kernel and trigger TLB shootdowns on other cores. All of that is fine in a web server and unacceptable in a loop with a 1 ms deadline.

The discipline is: allocate everything at startup, take a pool of fixed-size blocks for anything that must be created at runtime, and prove to yourself that the hot path contains no malloc, no free, no std::string, no printf, and no file I/O. Then lock the pages down so the kernel cannot decide to evict them:

#include <sys/mman.h>

if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0)
    die("mlockall: raise RLIMIT_MEMLOCK or run with CAP_IPC_LOCK");

/* the stack grows down into unmapped memory if you let it */
setrlimit(RLIMIT_STACK, &(struct rlimit){ .rlim_cur = 1u<<20,
                                           .rlim_max = 1u<<20 });

/* and ask for the pages back from the kernel now, not at t+9h */
if (madvise(pool, pool_bytes, MADV_WILLNEED) != 0)
    warn("madvise");

Transparent huge pages deserve their own line. They improve throughput and they introduce khugepaged, a kernel thread that occasionally decides to compact memory in the background. For a latency-sensitive process, either use explicit huge pages (hugetlbfs, reserved at boot and therefore population-free) or opt out of the transparent path for your buffers. The middle ground — leaving it on and hoping — is how a 40 µs outlier gets into a system that has been clean for a month.

Floating point, and the cost that hides

An experienced application programmer reaches for float without thinking, because on a desktop the arithmetic is free. On a board it is not, and the bill arrives in three places that a profiler will not show you.

The instruction, when there is no FPU. A Cortex-M without an FPU, or an ARM build with -mfloat-abi=soft, turns every multiply into a call to __aeabi_fmul. A control law that was four instructions becomes four function calls. That is 10× to 100×, and it is invisible unless you look at the disassembly rather than the source — the C is identical.

Denormals. This is the one that is genuinely hard to find. A filter that decays toward zero eventually enters the denormal range, below about 1.2e-38 for float, where most cores handle the value in microcode rather than silicon. Individual operations become 50–100× slower. The signature is a loop that is clean for a minute and then degrades as its state decays, which looks like a thermal problem and is not. Fix it by setting flush-to-zero and denormals-are-zero in MXCSR, or by clamping the state to zero when it goes small. -ffast-math sets FTZ for you and also lets the compiler assume no NaNs, which is a separate decision you should make deliberately.

Context switches. The kernel has to preserve the FPU state of a thread that used it. On x86 that state is large; with AVX-512 it is very large, and the kernel may fall back to XSAVE of the whole extended block. A thread that uses one vector register per cycle pays for all of them at every switch. The usual mitigation is arch_prctl(ARCH_SET_CPUID)-adjacent trickery or simply not using wide vectors in the loop, but the simplest fix is the one below.

Fixed point in the loop, floating point at the edges. Scale by a power of two so the conversion is a shift, and keep the range analysis in your head where you can see it:

#include <stdio.h>
#include <stdint.h>
#include <time.h>

#define N 20000000
#define Q 16                 /* 16.16 fixed point */
#define K 3                  /* filter gain, in Q */

/* one-pole low pass: y += (x - y) * k */
static int64_t fixed_loop(int32_t x) {
    int64_t y = 0;
    for (long i = 0; i < N; i++)
        y += ((((int64_t)x << Q) - y) * K) >> Q;
    return y;
}

static double float_loop(float x) {
    float y = 0.0f, k = (float)K / (1 << Q);
    for (long i = 0; i < N; i++)
        y += (x - y) * k;
    return y;            /* returns 0 or a denormal, depending */
}

static double now(void) {
    struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t);
    return t.tv_sec + t.tv_nsec * 1e-9;
}

int main(void) {
    double t0 = now(); volatile int64_t a = fixed_loop(1 << Q);
    double t1 = now(); volatile double  b = float_loop(1.0f);
    double t2 = now();
    printf("fixed  %8.2f ns/iter  (%lld)\n", (t1-t0)/N*1e9, (long long)a);
    printf("float  %8.2f ns/iter  (%f)\n",  (t2-t1)/N*1e9, b);
    return 0;
}

Build it twice. On the Pi: gcc -O2 fpbench.c -o fpbench uses the hardware FPU, and gcc -O2 -mfloat-abi=soft fpbench.c -o fpbench-soft calls the emulation routines. On x86-64 there is no soft-float ABI worth testing, so the interesting comparison there is float against fixed, and denormal against normal — start float_loop at 1e-30f instead of 1.0f and watch the number move.

Linux itself does not use floating point, and this is why. Kernel entry would have to save and restore FPU state on every syscall and interrupt, and an interrupt handler cannot assume it has an FPU context to run in. Everything in the kernel that looks like it needs a float — load averages, fixed-point scaling, calendar arithmetic — is integer, and drivers follow the same rule. When your code runs in a kernel context (eBPF aside), the FPU may not be reachable at all.

Exceptions are not a branch

A cache miss costs about a hundred nanoseconds. A throw costs hundreds of nanoseconds to tens of microseconds, and the spread is the problem more than the number. Unwinding is not a jump: the runtime allocates the exception object, then walks the frame tables, calling destructors and catch handlers as it goes.

"Zero-cost exceptions" means the cost is zero when nothing is thrown. That is a real and useful property, and it is why the tables exist — but it says nothing about the path that matters here. When it throws, you pay for the table walk, the allocation, and in a multi-threaded program a lock in the emergency exception pool. Order of magnitude: somewhere between 100× and 1,000× a status return, with a long tail.

The larger problem is the state. An exception unwinds through frames that have already done half their work. A function that wrote three of five registers, updated a filter history and then failed leaves the caller holding a structure that is neither old nor new. From the catch site you cannot know which invariant still holds, and the compiler will not tell you. In a control loop you also cannot roll back: the DAC write already happened, the frame already went on the bus.

The discipline is the one embedded code has always used: every function that can fail returns a status, and the caller decides. Where that becomes verbose, split the operation into a validate phase that can fail and a commit phase that cannot — check the buffer has room, then write it.

typedef enum { OK = 0, E_FULL, E_RANGE, E_IO } status;

status push(sample_t s)
{
    uint32_t h = atomic_load_explicit(&head, memory_order_relaxed);
    uint32_t t = atomic_load_explicit(&tail, memory_order_acquire);
    if (((h + 1u) & RB_MASK) == t) return E_FULL;      /* decide   */
    if (s.value > LIMIT)            return E_RANGE;
    buf[h & RB_MASK] = s;                              /* commit   */
    atomic_store_explicit(&head, h + 1u, memory_order_release);
    return OK;
}

/* and let the compiler enforce the policy in the hot path:
   gcc -fno-exceptions -fno-rtti -fno-threadsafe-statics  */

-fno-exceptions also removes the hidden failure modes you inherited: std::stoi, vector::at, std::stod and friends throw, and with exceptions off that becomes a call to std::terminate. Better to know that at compile time than at 3 a.m. In C++ code that must stay exception-safe at the boundary, keep the ban inside the loop and allow RAII in the setup path — but say which is which in a comment, because the next person will not guess.

The stack, and why it is the fastest allocator

Stack allocation is a subtract from a register. There is no lock, no free lists, no metadata, no fragmentation, and the memory is almost always already in cache because the frames around it were just used. Release is equally free: the pointer moves back at scope exit, deterministically, with no collector and no free() to take a lock behind your back.

So the rule is not "avoid the stack". It is "use the stack deliberately, and bound it". Everything that lives for one cycle — the scratch buffer, the formatted telemetry frame, the small fixed array of samples — belongs on the stack. Everything that must outlive the cycle, or whose size is not a compile- time constant, belongs in a pool allocated at startup (§06).

The bound is the part people skip. A real-time thread usually does not get the default 8 MB main-thread stack; it gets what you gave it at creation, often 64–256 KB. That is plenty for fixed buffers and nothing at all for recursion or a variable-length array whose length came off the network.

#define STACK_BYTES (256u * 1024u)
#define GUARD_BYTES (8u * 1024u)

static void *stack_region;

pthread_attr_t attr;
pthread_attr_init(&attr);
posix_memalign(&stack_region, 4096, STACK_BYTES);
pthread_attr_setstack(&attr, stack_region, STACK_BYTES);
pthread_attr_setguardsize(&attr, GUARD_BYTES);

/* touch every page now, so no cycle ever takes a stack fault */
for (char *p = stack_region; p < (char *)stack_region + STACK_BYTES; p += 4096)
    *p = 0;

pthread_create(&rt_thread, &attr, cycle, NULL);

The guard page turns an overflow into a SIGSEGV at the moment it happens instead of silent corruption of the thread next door. And the pre-touch matters because mlockall(MCL_FUTURE) only locks pages that were mapped when they were mapped: a stack that grows at runtime can fault in a page that is not locked, in the middle of a cycle, nine hours in.

Two habits to break while you are here. alloca() inside a loop is a stack leak with no diagnostic. And recursion in a cycle is unbounded by construction — the depth depends on data, so the WCET depends on data, which means you do not have a WCET.

volatile is not a synchronisation primitive

volatile tells the compiler that an object can change outside the program's control, so it must perform every access you wrote, in the order you wrote it, and must not cache the value in a register across them. That is all it does. It is not a memory barrier. It is not atomic. It does not order accesses to other variables. And it provides no mutual exclusion whatsoever.

On a device driver this is not pedantry. The compiler is entitled to elide a store to a variable it can prove is never read, and a status register that is never written is a peripheral that never starts. Equally, a polling loop on a non-volatile flag can be hoisted entirely out of the loop, leaving a tight branch that never re-reads memory — an infinite loop that the source says cannot happen.

/* hardware: volatile is required and sufficient for the access itself */
typedef struct {
    volatile uint32_t CR;      /* control     */
    volatile uint32_t SR;      /* status      */
    volatile uint32_t DR;      /* data        */
} uart_regs;

#define UART ((uart_regs *)0x40011000u)

void putc_(char c) {
    while (!(UART->SR & 0x80)) { }     /* must re-read SR every iteration */
    UART->DR = (uint32_t)c;
}

/* threads: volatile is wrong, atomic is right */
static _Atomic bool stop_requested;

while (!atomic_load_explicit(&stop_requested, memory_order_relaxed))
    do_cycle();

Inside a PREEMPT_RT kernel volatile becomes even less useful — the kernel's own locks and the compiler's barriers do the ordering, and a volatile handshake between threads is still a data race. In userspace, a spin on a volatile flag across cores will compile, run, and occasionally produce a value from before the other core wrote it. Use atomic_load_explicit and be explicit about the ordering you actually need.

Inheritance and the indirect call

A virtual call is two dependent loads and an indirect branch: load the vtable pointer from the object, load the function pointer from the vtable, jump. The cost is a few nanoseconds when everything is hot, which is why the advice sounds like a micro-optimisation. In a cycle with a fixed budget it is not the mean that matters — it is that the target is not known to the compiler, so it cannot inline, cannot hoist the loop invariants across the call, and cannot bound the callee's execution time for you.

The variance comes from the branch predictor. One call site with one target predicts perfectly. A call site that alternates between two implementations in a tight loop defeats the predictor, and each miss is fifteen to twenty cycles plus a pipeline flush — on a path you have budgeted to 90 µs, in a place where you cannot see it in the source.

The deeper cost is what a class hierarchy invites: heap allocation for the objects, reference counting or shared_ptr copies, destructors that run at the end of scope and take locks, and dynamic_cast, which walks type_info at runtime and is far more expensive than the virtual call it accompanies.

Three ways out, in order of preference for a hot path:

• A function table resolved at startup. One indirect call per cycle, at a site that always uses the same entry — the predictor is happy, and the table is a const object you can inspect.
• Templates or CRTP. Static polymorphism: the call is direct and inlinable, and the cost is compile time and code size, not runtime.
• final plus LTO. If you must keep the class, marking the override final lets GCC and Clang devirtualise the call when the dynamic type is provable. It is not a guarantee, but it is free to try and -fdump-lang-class will tell you what survived.

typedef struct filter filter;
struct filter {
    int32_t state;
    void  (*step)(filter *, int32_t);   /* one target, chosen at init */
};

static void lpf_step(filter *f, int32_t x) { f->state += (x - f->state) >> 4; }
static void hp_step (filter *f, int32_t x) { /* ... */ }

static filter f = { .state = 0, .step = lpf_step };   /* resolved once */

None of this says inheritance is wrong. It says that on a path with a microsecond budget, a construct whose cost depends on the branch predictor's mood is a construct you should be able to point at and justify. The same argument applies to std::function, which adds a possible allocation to the indirect call, and to callbacks registered from a config file — the call site is the same, the predictability is worse.

Two pieces of code

A single-producer, single-consumer ring

This is the shape almost every real-time design converges on: the RT thread never blocks, never allocates, and never waits for anyone. It pushes samples into a ring; a lower-priority thread drains them, formats them and writes them out. If the consumer falls behind, samples are dropped deliberately — that is a policy decision, and it is better than back-pressure into a deadline.

#include <stdatomic.h>
#include <stdbool.h>
#include <stdint.h>

#define RB_SIZE 1024u                  /* must be a power of two */
#define RB_MASK (RB_SIZE - 1u)

typedef struct {
    _Alignas(64) _Atomic uint32_t head;  /* producer writes only */
    _Alignas(64) _Atomic uint32_t tail;  /* consumer writes only */
    _Alignas(64) int32_t slot[RB_SIZE];
} rb_t;

bool rb_push(rb_t *rb, int32_t v)
{
    uint32_t h = atomic_load_explicit(&rb->head, memory_order_relaxed);
    uint32_t t = atomic_load_explicit(&rb->tail, memory_order_acquire);
    if (h - t == RB_SIZE)                /* unsigned wraparound is fine */
        return false;                    /* full: drop, do not wait */
    rb->slot[h & RB_MASK] = v;
    atomic_store_explicit(&rb->head, h + 1, memory_order_release);
    return true;
}

bool rb_pop(rb_t *rb, int32_t *out)
{
    uint32_t t = atomic_load_explicit(&rb->tail, memory_order_relaxed);
    uint32_t h = atomic_load_explicit(&rb->head, memory_order_acquire);
    if (h == t)
        return false;                    /* empty */
    *out = rb->slot[t & RB_MASK];
    atomic_store_explicit(&rb->tail, t + 1, memory_order_release);
    return true;
}

Three things are doing the work. The acquire load of the other index orders the read of the slot after it, so the consumer cannot see a half-written element. The release store orders the write of the slot before the index becomes visible. And _Alignas(64) puts the two indices on separate cache lines, because sharing one is a coherence ping-pong between the two cores — a correctness-preserving bug that costs a few hundred nanoseconds per operation.

This is single-producer by construction. Two producers need a CAS loop and a much more careful argument; two consumers need a different algorithm entirely. If you find yourself adding a second producer, stop and re-read the code, because the invariant is not incidental.

A cyclic loop that tells you when it is late

Sleep until an absolute time, never for an interval. A relative sleep accumulates drift, and worse, it hides lateness: the loop that oversleeps by 200 µs and then sleeps 1,000 µs again is quietly running at 833 Hz. Absolute deadlines either keep up or report that they did not.

#define PERIOD_NS 1000000L   /* 1 kHz */

static void ts_add_ns(struct timespec *t, long ns)
{
    t->tv_nsec += ns;
    while (t->tv_nsec >= 1000000000L) {
        t->tv_nsec -= 1000000000L;
        t->tv_sec  += 1;
    }
}

static long ts_diff_ns(const struct timespec *a, const struct timespec *b)
{
    return (a->tv_sec - b->tv_sec) * 1000000000L + (a->tv_nsec - b->tv_nsec);
}

void run_loop(void)
{
    struct sched_param sp = { .sched_priority = 80 };
    pthread_setschedparam(pthread_self(), SCHED_FIFO, &sp);

    struct timespec next;
    clock_gettime(CLOCK_MONOTONIC, &next);

    for (;;) {
        ts_add_ns(&next, PERIOD_NS);

        do_cycle();                     /* bounded, no allocation */

        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);

        struct timespec now;
        clock_gettime(CLOCK_MONOTONIC, &now);
        long late = ts_diff_ns(&now, &next);
        if (late > 0) {                 /* we are already past the deadline */
            overruns++;
            if (late > worst_late_ns)
                worst_late_ns = late;
        }
    }
}

CLOCK_MONOTONIC rather than CLOCK_REALTIME, always: a wall-clock adjustment from NTP must not move a deadline. And count the overruns in the running system, not only in the lab — one integer and one comparison, published in the telemetry frame, is the cheapest observability you will ever add.

Languages and compilers

C and C++ are the default because their costs are visible: no collector, no runtime, allocation only where you wrote it. That is the whole advantage, and it is worth being explicit that it is a statement about predictability rather than about speed. In C++, the constructs to keep out of the loop are the ones that allocate or take a global lock behind your back: std::string, std::function, std::shared_ptr copies, std::map inserts, exceptions, and iostreams. A -fno-exceptions -fno-rtti build makes the compiler enforce part of that policy for you.

Rust is a good fit for the same reason, in its no_std form: no allocator unless you bring one, no collector, and the borrow checker pushes the ownership question into the design where it belongs. Managed languages are not disqualified by being slow — they are disqualified by having a stop-the-world event whose timing you do not control. That is a property of the runtime, not of the benchmark.

Measure the tail

The measurement discipline is the part that separates a real-time system from a fast one. Four rules, in order of how often they are broken:

1. Measure the maximum, and the high percentiles. The mean is a comfort blanket. p99.9 and max are the numbers that decide whether the deadline holds.
2. Measure for hours, under load. A ten-minute idle run on a cold machine finds nothing. Run a CPU stressor on the housekeeping cores, push the network, fill the page cache, and leave it running over lunch.
3. Measure on the target. A desktop with a different CPU, a different idle driver and a different thermal envelope is a different system. Virtual machines need their own accounting, because steal time is invisible from inside.
4. Measure end to end. A timestamp taken at the top of do_cycle() cannot see the interrupt that delayed the wake-up. Toggle a GPIO pin at the start and end of the cycle and put a scope on it; that measurement includes everything, and it cannot lie to you.

# 8 threads, RT priority 95, 200 us interval, 1 hour,
# histogram with 400 us range, mlockall
cyclictest -m -S -p95 -i200 -h400 -D 1h -q

# the same thing while the housekeeping cores are busy
stress-ng --cpu 4 --cpu-method matrixprod --timeout 1h &

# who woke late, and why
perf sched record -- sleep 30
perf sched latency

cyclictest measures the timer path, not your application. Its value is as a floor: if the kernel cannot wake a thread within 40 µs on an idle isolated core, nothing you write upstairs will meet a 200 µs deadline. When the number is good and the application still misses, the problem is in the application, and perf sched latency and the ftrace wakeup_rt tracer are where to look next.

A review checklist

Short enough to run against a patch, specific enough to produce an argument:

• Is the deadline written down, with units, next to the code that has to meet it?
• Does the hot path allocate, free, lock a non-PI mutex, log, or open a file?
• Is every loop bounded by a constant, not by a value from the network?
• Is the sleep absolute, and does the code detect that it woke late?
• Is memory locked, and is the stack bounded and pre-touched?
• Is the thread pinned, isolated, and at a priority that nothing else shares?
• Could a lower-priority task hold a resource this thread needs? If so, is it PI?
• What happens on the first cycle after a 200 ms stall — does it catch up by running twice, or skip?
• Does the telemetry path back-pressure the loop, or drop?
• Was the worst case measured on this hardware, in this build, under load, for longer than an hour?
• Does the hot path touch floating point, and can those values decay into denormal range?
• Can any call inside the cycle throw, and if it did, what state would be left half-written?
• Is every per-cycle buffer on the stack, bounded, and pre-touched?
• Is each shared or hardware-visible variable either volatile (MMIO, signal) or _Atomic (threads) — and never mistaken for the other?
• Does the cycle make an indirect call, and is its target the same one every time?

Most of these are cheap to answer and expensive to skip. The tenth one is the only one that cannot be done from a desk.

Reading

• Buttazzo, Hard Real-Time Computing Systems — scheduling theory with the proofs kept close to the examples.
• Liu, Real-Time Systems — the response-time analysis that the utilisation test approximates.
• Kopetz, Real-Time Systems: Design Principles for Distributed Embedded Applications — the systems view, especially on time and fault containment.
• Drepper, What Every Programmer Should Know About Memory — still the clearest explanation of why the cache decides your worst case.
• Kernel documentation: Documentation/scheduler/sched-rt-group.rst and Documentation/timers/, plus man 7 sched.
• man cyclictest — read the options rather than copying a command line from a blog post, including this one.