1

Debugging embedded Linux

power management
Kernel developer point of view
ELCE, Barcelona, Nov. 7 2!2
"ero Kri#to $t%&ri#to'ti.com(
2
Content# ) *ntroduction

Debugging tools / methods for PM

Kernel power management features

Typical power management problems / bugs

3
Common +, arc-itecture
Kernel
Userspace
puidle go!" pufre# go!"
loc$ %& 'eg %&
(ardware
PM ore
)rch PM 'eg Dri!ers l$ Dri!ers
pufre# dri!er
De!fre# go!" pufre# go!
De!fre# dri!er puidledri!er
Dri!ers
Dri!ers
Dri!ers
*
Debugging tool# ) met-od# for +,
+
Di#abling &ernel feature#
Disable ,-%./0PM
1 .f bug remains2 complain to someone else3

Disable unnecessary dri!ers

1 ,nly enable minimal set li$e timers2 console2 .22 etc"2 depends on arch
which are needed
Disable PM features
1 ,nly enable a single PM feature at a time and attempt to reproduce the
problem
Pros4
1 /ood for isolating a PM related problem if don5t ha!e any $ind of clue about
what is wrong

ons4
1 'ather slow and difficult to use
6
.tre## te#ting
7cripts that do some PM related operations in a tight loop
1 89ample4 suspend loop with a wa$eup from suspend e!ery :;;ms

7hould be random enough so that the bug producing pattern is

e9ecuted
Pros4
1 an be !ery useful in reproducing some problems that ta$e typically a long
time to occur with normal use

ons4
1 Difficult to figure out what operations to actually e9ecute in the stress testing
script
<
"racing /print& ) low level 012"3
Populate enough debug print$s to the code being e9amined
.f possible2 can also dedicate a custom interface for debug traces
Pros4
1 8asy / #uic$ to use

ons4
1 Typically alters code e9ecution time =especially if using serial port>2 and may
hide the actual problem
1 -ot usable from !ery low le!el code =print$>
1 Print$4s are cached and may not be printed out before a crash happens =e"g"
during suspend>
?
4+*5 ) LED trace
)dd control to some /P., / @8D signal from certain points in code
1 89ample4 @8D is turned on when PU is running2 disabled when idle

.f possible to use multiple signals2 can pro!ide a binary coded trace

!alue from $ernel
Pros4
1 7ingle /P., / @8D control typically does not consume so much time as to
alter e9ecution times drastically =!s" U)'T>
1 an e!en use multiple de!ices with @8D72 and if using stress testing script2
can immediately see if some de!ices ha!e crashed or not
1 Useful in case debugging code areas where debugger / print$ is not usable
ons4
1 Most li$ely only a few a!ailable =what to traceA>
B
Debugger
Useful in de!eloping new code2 and sometimes can see where $ernel
has crashed
&ith PM code2 typically need brea$points
1 7tatic / dynamic
1 %a$e brea$point =infinite loop in code2 reCprogram P after stopped>
'eal hac$ers don5t use/need debugger though

Pros4
1 &ell2 debugger is always a debugger
ons4
1 ommunication with PU is probably bloc$ed during low power operating
modes
:;
Buffered trace# ) #tati#tic#
Trace information collected from $ernel side into a ring buffer
Dumped out with a console operation through e"g" debugfs
Typical uses4 cpuidle tracing2 power state usage statistics
Kernel tracepoints seem to be a good tool for this2 and it is easy to add
new tracepoints in case something is missing

Pros4
1 Minor impact to e9ecution times =no slow (& components accessed>
ons4
1 ,nly useful in debugging misbeha!ior =crash pre!ents later dump>
::
Example6 tracepoint# wit- idle !)2
89ecute following script in target de!ice4
D3/bin/sh
traceCcmd reset
sleep :;
traceCcmd start Ce power
sleep :;
traceCcmd stop
traceCcmd e9tract
:2
Example6 tracepoint# wit- idle 2)2
opy resulting trace"dat o!er to host and process it with $ernelshar$
:3
Exporting debug functionalit7 to
u#er#pace
Pro!ide a testing )P. to userspace to read / write hardware registers
directly =on memory mapped registers can use /de!/mem>
8nhance e9isting debug interfaces by adding write functionality in addition
to e9isting readConly )P.s
1 89ample4 regulator fw micro!olt nodes
)dd completely new interfaces where nothing e9ists currently
Pros4
1 (a!ing as much of the functionality a!ailable to userspace as possible ma$es
it easy to write test scripts
1 an dynamically create new test cases
ons4
1 Might not be possible to upstream these
:*
Kernel +, feature#
:+
.u#pend
89ecuted from command line =echo mem E /sys/power/state>
Disables all dri!ers manually
1 Disables also trace3
Tools for debugging4
1 Trace =limited>
1 /pios
1 Debugger =with brea$points>
Tric$s4
1 Pre!ent low le!el PM entry so that hardware is mostly ta$en out from debug
process2 reCenable once 7& wor$s
:6
Cpuidle
Fit complicated as can e9ecute multiple different states based on
system status
Tools for debugging4
1 Traces =limited> / gpios / ring buffer
1 Debugger can be used with brea$points
Debug information from userspace4
1 /sys/de!ices/system/cpu/G/cpuidle/G
Tric$s4
1 89port )P. to userspace to HforceI a certain state always
:<
2egulator#
Userspace )P. a!ailable at /sys/class/regulator
8asy to chec$ the status of regulator framewor$ against hardware
status by using multimeter etc"

Tric$s4
1 89port write capability for micro!olt nodes to userspace
1 89port regulator enable / disable to userspace
:?
Cloc& framewor&
,ngoing wor$ within @inaro to get a common cloc$ framewor$ into the
$ernel
Part of the code e9ists already but integration missing to most of the
platforms

Tools for debugging4

1 Traces
1 'egister dumps !s" cloc$ framewor$ status

Userspace interface4
1 /sys/$ernel/debug/cl$/G
1 Pro!ides info for cloc$ rates2 usecounts2 flags etc"
1 8asy to twea$ to allow manual enable of cloc$s from userspace
:B
Cpufre8
Pretty easy to debug2 as typically does not bloc$ any $ernel
functionality2 and has nice )P.s readily a!ailable from userspace
an usually trace through e!erything
1 May ha!e a critical section that re#uires more comple9 debugging
Debug information from userspace4
1 /sys/de!ices/system/cpu/G/cpufre#/G
1 /sys/class/regulator/G
1 /sys/$ernel/debug/cl$/G if a!ailable
Tric$s4
1 7elect cloc$ fre#uency manually with 5ondemand5 go!ernor by writing to
5scaling0min/ma90fre#5 nodes
2;
Devfre8
De!ice specific DJ%72 relati!ely similar to cpufre#
7hould also be possible to trace through e!erything
)dds e9tra 5de!fre#5 directory under de!ice sysfs node
2:
"7pical +, problem#
22
Bootloader madne## in#anit7
8!erybody uses a different bootloader
@ots of features inside the bootloaders2 which typically lea!e hardware
enabled after use KE pre!ents PM

Luite often it might not be e!ident that bootloader is causing PM

problems
%irst thing to do when someone complains to you about PM issues2 as$
them what bootloader they are using
1 .f not the same you ha!e KE its their problem
23
Device cra#-
De!ice dies completely2 either with or without a crash dump
.f with dump2 Must decipher the crash dump to figure out what happened
.f silent hang2 try to pinpoint where the crash happens
1 Disable ,-%./0PM
1 Disable PM options one by one
1 Disable dri!ers to get a 5minimal $ernel5
1 )dd traces to code
1 )dd brea$points to potential crash locations etc"
2*
Device malfunction
7ome dri!er starts misbeha!ing after a while
an ta$e a long time to reproduce
Maybe difficult to pinpoint the actual problem
.f you are luc$y2 might pro!ide mysterious crash dumps related to the
component in #uestion

7tress testing scripts might be useful

89ample problems4
1 De!ice stops responding to serial console after a while2 but the $ernel /
interrupts still wor$
1 Memory corruption
2+
*ncrea#ed power con#umption /!)93
Power source good initial indicator
1 e"g" battery dies too #uic$ly compared to what it should be

hec$ if cpuidle / suspend wor$ properly and set the de!ice to proper
state
1 7ysfs status for cpuidle
1 timer0stats
1 'egulator status
1 &hate!er else is e9ported to userspace from (& point of !iew
/ood if you ha!e a wor$ing / nonCwor$ing case where you can
compare the system state
26
*ncrea#ed power con#umption : -ardware
/2)93
(ardware problems usually force higher power use than planned
1 May need to disable some power sa!ing techni#ues
e"g" some regulator must always be 5enabled5
1 7ome (& pulls are incorrectly designed and consume e9tra power
Might be possible to reduce impact in some cases with software twea$s
1 8"g" align e9ternal pull !s" 7o configurable pull config
May need to reCdesign hardware
2<
*ncrea#ed power con#umption % u#er#pace
/9)93
Typical culprit for consuming too much power
7ome process is using too much resources for e9ecution
1 Pre!ents cpuidle completely =cpu load> or partially =timer usage>
hec$ out 5top5 or something similar for PU load

/proc/timer0stats is good for figuring out timers that are used too often
2?
.ome reference#
Powertop
1 Parses timer N process N interrupt info
1 https4//;:"org/powertop/

Powerdebug
1 Parses regulator N cloc$ framewor$ N sensor data
1 https4//wi$i"linaro"org/&or$ing/roups/PowerManagement/Doc/PowerDebug

Tracepoints
1 Kernel source4 include/trace/e!ents/power"h
1 Debugfs4 /sys/$ernel/debug/tracing/'8)DM8
Parsers4 =target> traceCcmd KE =host> $ernelshar$2 pytimechart
2B
"-an& 7ou;