Soft, Hard and

“Ruby” Hard Real Time

®
Approaches with Linux
Gilad Ben­Yossef
Codefidence Ltd, CTO

1
 Real Time: A Definition

● “A real-time system is one in which the

correctness of the computations not only
depends upon the logical correctness of the
computation but also upon the time at which
the result is produced.
● If the timing constraints of the system are not
met, system failure is said to have occurred."
Donald Gillies, quoted on Usenet comp.realtime FAQ

2
The Path to Real Time in Linux

● POSIX Real Time Scheduling Domains

● Priority Inversion
● Locking Memory
● Timer Frequency
● Sources Of Latency
● Scheduling Latency
● Interrupt Latency
● Real Time Linux Benchmarks
3
 Linux Priorities

Nice Real Time priority

level 99
-20 98
-19 97 Real time processes
-18 ... SCHED_FIFO
SCHED_RR
.. 4
16. 3
17 2
18 1
19 Non real-time processes
0 SCHED_OTHER
4
 Changing Real Time Priorities

int sched_setscheduler(pid_t pid, int policy,

const struct sched_param *p);
struct sched_param {
int sched_priority
};
● sched_setscheduler sets the scheduling
policy and priority identified by pid.
– Policy is SCHED_FIFO, SCHED_RR,
SCHED_OTHER or SCHED_BATCH.

5
 Thread Level Priorities

● In Linux, each thread has separate real time

priority.
● When creating threads with the
pthread_create() system call, an attribute
structure can be passed.
● Various thread attributes can be used to set
scheduling domain and priority.
● The relevant attributes can also be changed
during run time

6
 Sched Policy
Thread Attribute: schedpolicy
● Select the scheduling policy for the thread: one of
SCHED_OTHER (regular, non-realtime scheduling),
SCHED_RR (realtime, round-robin) or SCHED_FIFO
(realtime, first-in first-out).
● Default value: SCHED_OTHER.
● The real time scheduling policies SCHED_RR and
SCHED_FIFO are available only to processes with
superuser privileges.
● The scheduling policy of a thread can be changed after
creation with pthread_setschedpolicy(3).

7
 Sched Param

Thread Attribute: schedparam

● Contain the scheduling parameters (essentially, the
scheduling priority) for the thread.
● Default value: priority is 0.
● This attribute is not significant if the scheduling policy
is SCHED_OTHER; it only matters for the real time policies
SCHED_RR and SCHED_FIFO.
● The scheduling priority of a thread can be changed after
creation with pthread_setschedparam(3).

8
 Inherit Sched

Thread Attribute: inheritsched

● Indicate whether the scheduling policy and scheduling
parameters for the newly created thread are determined by
the values of the schedpolicy and schedparam attributes
(PTHREAD_EXPLICIT_SCHED) or are inherited from the
parent thread (value PTHREAD_INHERIT_SCHED).
● Default value: POSIX says PTHREAD_EXPLICIT_SCHED,
but at least some version of Linux do
PTHREAD_INHERIT_SCHED.

9
 Priority Inversion

2. High Priority task preempts low

priority task
99
3. Hi Priority task block on mutex
Task Priority

50
4. Medium Priority task preempts low
priority task and high priority task

3
1. Low Priority task takes mutex

Time

10
 Priority Inheritance

● The common solution to priority inversion is

called Priority Inheritance
● A task which holds a lock, automatically inherits
the priority of the highest priority task which
contends the lock.
– Until the lock is released, of course.
● Correct implementation hard, performance
impact non trivial, not a sliver bullet.
● Linux implementation was.. difficult.

11
 PI-Futex
● Interface through the Fast User-space
muTEX mechanism.
– Linux 2.6 application mutex support code.
● Supports user-space mutexes only.
– Similar kernel functionality in PREEMPT_RT.
● Introduced in 2.6.18
● A patched Glibc is needed right now
– Patch merging into mainline Glibc right now.

12
 How fork() seems to work

A complete copy
Father Process is created of the
father process.

Both parent and

child continue
execution from the
same spot.

Copy
Father Process Child Process

13
 How fork() really works

Child process gets a

RW copy of stack and
Father file descriptors.
Memory
Process Copy On Write
reference is used for
the memory.

Father RO RO Child
Memory
Process Process

14
 What happens during write?

When write is
attempted,
a single memory
Original Memory
page is copied
RO
and references
updated.
This is called:
“breaking the Child
CoW”. Process
RW

RO
Father
Process

15
 Locking Memory

int mlockall(int flags);
● Disables paging for all pages mapped into
the address space of the calling process.
● Flags are:
– MCL_CURRENT: Lock all pages which are currently
mapped into the address space of the process.
– MCL_FUTURE : Lock all pages which will become
mapped into the address space of the process in the
future.
– Use both for max. effect: MCL_CURRENT| MCL_FUTURE

16
 Stack Pre-Faulting
● Linux user space stacks are auto expanding.
– Use more stack then allocated to a process? kernel
gets an exception and allocates more stack for you.
– Can turn stack access to multiple context switch
and memory allocation with non deterministic
latency.
● We need to pre-fault stack pages.
– Call a dummy function that allocates an automatic
variable big enough for your entire future stack
usage and write to it, after you've mlocked memory.

17
 Timer frequency

● Linux timer interrupts are raised every HZ of

second.
● HZ is configurable during kernel build:
100, 250 (i386 default) or 1000.
See kernel/Kconfig.hz.
● Compromise between system responsiveness
and global throughput.
● Caution: not any value can be used. Constraints
apply!

18
The Effect of Timer Frequency
Requested sleep time, in ms

2 HZ=100
0

1
0

1 1 2
0 5 0
Real sleep time, in ms

19
The Effect of Timer Frequency cont.
Requested sleep time, in ms

2 HZ=1000
0

1
0

1 1 2
0 5 0
Real sleep time, in ms

20
 High-Res Timers

● High-res timers use non RTC interrupt timer

sources to deliver timer interrupt between
ticks.
● Allows POSIX timers and nanosleep() to be
as accurate as the hardware allows
● This feature is transparent.
– When enabled it makes these timers much more
accurate than the current HZ resolution.
– Around 1usec on typical hardware.

21
 Sources Of Latency

PIC Context Switch Finding ISR

saving registers...
ISR

Interrupt Latency

Interrupt Context
ISR Scheduler Task
Latency Switch

Preemption Latency

22
 The Linux O(1) Scheduler
● The kernel maintains 2 priority arrays:
the active and the expired array.
● Each array contains 140 entries each with a
queue of processes with the same priority.
● The arrays are implemented such that it is
possible to pick the queue with the runnable
task with the highest priority in constant time
– Whatever the number of runnable tasks is.
● Let's see how this helps us...

23
 Choosing and Expiring Tasks
● The scheduler finds the highest priority with a
runnable task and
– Executes the first task with that priority.
– Non real time tasks are run until they exhaust their
time slice and moved to the expired array.
Real time tasks are run until they yields the CPU [*].
● This is done until there are no more tasks in
the array, then the two arrays are swapped
and we start over.
● Scheduling is O(1).

24
 Kernel Preemption Options
● Preemption means a high priority task taking
the place of a low priority one.
– User space code is always preemptive.
● In kernel mode (when running system call):
– PREEMPT_NONE
● None preemptive kernel code.
– PREEMPT_VOLUNTARY
● Voluntary preemption points.
– PREEMPT
● Non critical section preemptive kernel.

25
 Soft Real Time

● “Deadline exists, but glitches do not

propagate”
● Typically 1 millisecond cycle time.
● Vanilla Linux is well suited:
– Preemptive kernel, O(1) scheduler, high
resolution timers, priority inheritance futex
support
– Good hardware design helps:
interrupt queues, DMA ring buffers, FIFOs.

26
Hard and “Ruby” Hard Real Time

● “Deadline exists, glitch are fatal”

● Typically Sub millisecond cycle times.
● Can be done with Linux, two approaches
available:
– Nano-kernel
● Run Linux under a nano Hard real time kernel.
– PREEMPT_RT RT Patch
● Change Linux to meet Hard Real Time needs.

27
Nano Kernel Approach

Linux RTOS
RT Tasks

RT Nano-Kernel

Hardware
28
 Nano Kernel Approach

● Nano-kernels come in many flavors:

– FSMLabs WindRiver RTLinux
– RTAI project
– Adeos I-Pipe
– Jaluna VirtualLogix VLX
● Problems:
– Real time tasks written in non Linux API.
– Limited Linux task support.
– Linux and RT task integration can be difficult.

29
 PREEMPT_RT

● Fully preemptive kernel by Ingo Molnar

– Kernel config option PREEMPT_RT
– Patched 2.6
– Preemptible critical sections
– Preemptible interrupt handlers
– Preemptible "interrupt disable" sequences
– Kernel locks priority inheritance
– Deferred operations
– Latency-reduction measures
30
 Vanilla Linux Contexts

Interrupt Handlers
Interrupt
Context
SoftIRQs

Kernel

Regular
tasklets

tasklets
Timers
Space
Hi prio

Stack
Net
...

Scheduling
Process
Points User Context
Thread

User Space Kernel

Thread

31
PREEMPT_RT Linux Contexts

SO_NODELAY Interrupt Handlers

Process
Thread Kernel

Net Stack

Tasklets
Timers
Space

Scheduling Kernel
Points Threads

User Space

32
 Interface Changes
● Spinlocks and local_irq_save() no longer
disable hardware interrupts.
● Spinlocks no longer disable preemption.
● Raw_ variants for spinlocks and
local_irq_save() preserve original meaning
for SO_NODELAY interrupts.
● Semaphores and spinlocks employ priority
inheritance
● Deferred operations API.

33
 Linux RT Benchmarking

● The setup:
– Dell PowerEdge SC420 machine with a Pentium 4
2.8MHz CPU, 256 Mb of RAM.
– Kernel version for all tests: Linux 2.6.12.
– Running various load generators.
● LMbench, LTP, flood ping, dd...
– 500,000 to 650,000 interrupts via parallel port.
– Measure response time on another machine.

34
Interrupt Response Times

Technology Avg Max Min

Vanilla Linux 2.6.12 Kernel 6.8 555.6 5.6

PREEMPT_RT V0.7.51-02 7.3 70.5 5.6

Adeos/ Ipipe-0.7 7.6 50.5 5.7

Numbers are for time interrupt generation till receive of

answer by the logger over parallel port (round trip), in
micro-seconds.

35
Any Questions?

Gilad Ben­Yossef
CTO
Codefidence Ltd.

gilad@codefidence.com

36
 Copyright Notice

● © 2007 Codefidence Ltd.

● © 2007 Michal Opdenacker, Free Electrons
● Released under a CC-by-sa 2.0 License.

37