How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

Issue on page 1 Environment on page 1 Resolution on page 1 1. Introduction on page 1 2. Disable the irqbalance service on page 3 3. Make a list of numeric interrupts on page 3 4. Determine the correct IRQ SMP affinity mask on page 4 5. Set the IRQ SMP affinity mask on page 5 6. Make IRQ SMP Affinity Mask Settings Persistent on page 5 7. Assign Desired Application(s) to the Isolated Processors on page 6 Examples on page 6 Application already running on page 6 Start application with specified CPU affinity on page 7 More information on page 7

Issue
How do I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

Environment
Red Hat Enterprise Linux 4 Red Hat Enterprise Linux 5 Red Hat Enterprise Linux 6 Note: Older releases do not have all of the capabilities discussed in this article.

Resolution
1. Introduction In some situations it may be desirable to configure a server equipped with multiple processors so that a subset of the total available processors are reserved for use only by the applications specifically assigned to them, and have all other server processes -- including the servicing of hardware interrupts and kernel maintenance tasks such as periodic flushing of the kernel's routing cache -- handled by the remaining processors. This can be useful for

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 1

How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

some applications which are highly timing-sensitive, or when it is desired otherwise that an application always have one or more processors available to it.

While Red Hat Enterprise Linux is not designed to serve as a real-time operating system, it is possible to effectively dedicate a subset of processors to servicing one or more applications, which may help improve performance.

For applications requiring realtime operating system support, and deterministic latency, Red Hat Enterprise Linux MRG-Realtime is recommended. For more information on Red Hat Enterprise Linux MRG, consult your Red Hat sales representative or visit the Red Hat Enterprise Linux MRG product overview page at http://www.redhat.com/mrg/.

1. Add the appropriate isolcpus kernel boot command line parameter 2. Edit the kernel command line in the /boot/grub/grub.conf file for the desired kernel, listing each processor to be isolated separated by commas.

For example, to isolate CPU cores 5-8 (processor ID numbers 4-7), add isolcpus=4,5,6,7 to the kernel command line options.

If the original kernel stanza in /boot/grub/grub.conf reads:

title Red Hat Enterprise Linux Server (2.6.18-194.el5) root (hd0,0) kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=/ initrd /initrd-2.6.18-194.el5.img

the modified stanza would read:

title Red Hat Enterprise Linux Server (2.6.18-194.el5) root (hd0,0) kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=/ isolcpus=4,5,6,7 initrd /initrd-2.6.18-194.el5.img

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 2

How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

2. Disable the irqbalance service The irqbalance service periodically redistributes the servicing of hardware interrupt request (IRQ) signals among available processors to equally balance the load. The simplest way to prevent the irqbalance daemon from distributing IRQ service tasks to the isolated processors is to disable the irqbalance service.
service irqbalance stop chkconfig --level 12345 irqbalance off

3. Make a list of numeric interrupts Examine the output of /proc/interrupts, noting each numeric interrupt listed. The interrupts listed are those which have been active since booting the system. The Symmetric MultiProcessing (SMP) affinity will need to be set for each of these to prevent the isolated CPUs from being assigned to them by the kernel.

Example output:

[root@localhost ~]$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 90 0 0 0 1 0 0 0 IO-APIC-edge timer 1: 3 2 1 3 1 1 1 2 IO-APIC-edge i8042 4: 0 0 0 1 0 1 0 1 IO-APIC-edge 6: 0 0 1 0 1 0 0 0 IO-APIC-edge floppy 7: 0 0 0 0 0 0 0 0 IO-APIC-edge parport0 8: 0 0 1 0 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 1 0 1 0 1 0 1 IO-APIC-edge i8042 14: 0 0 0 0 0 0 0 0 IO-APIC-edge ata_piix 15: 0 0 0 0 0 0 0 0 IO-APIC-edge ata_piix 16: 168930 168814 168690 168740 168737 168665 168801 168558 IO-APIC-fasteoi firewire_ohci, nvidia 17: 32 36 39 42 36 29 36 35 IO-APIC-fasteoi HDA Intel 18: 11423 11423 11531 11517 11551 11495 11379 11440 IO-APIC-fasteoi eth0 20: 87758 87742 87837 87780 87764 87794 87861 88025 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:us 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 23: 6528 6656 6574 6590 6583 6688 6595 6612 IO-APIC-fasteoi uhci_hcd:usb5, ahci NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 3832781 3410057 3135853 1628674 1521292 1545106 1187711 1247285 Local timer interrupts RES: 96836 80994 40640 31803 50957 35632 23327 18266 Rescheduling interrupts CAL: 17378 17232 17098 19941 22691 22167 22265 22218 function call interrupts TLB: 36208 36502 26155 23681 29545 23042 21444 17505 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts ERR: 0

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 3

How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

Note that the IRQ numbers seen will likely vary from server to server, and can also change on a given server if hardware is added or removed, interrupts previously unused become active, or if options affecting IRQ routing are changed in the server BIOS or on the kernel command line.

Furthermore, inactive interrupts should also be considered. By listing the contents of the / proc/irq virtual directory, additional interrupt numbers which do not display in the output of / proc/interrupts can also be identified. This would be useful in the case of a device which is normally inactive, but is used occasionally.

Finally, note that IRQ numbers 0 and 2 are special, and should be omitted from the list of numeric interrupts to set an SMP affinity mask for. The kernel will not allow these two to be changed anyway.

So in our example, we will need to set an SMP affinity mask for active IRQs 1, 4, 6-9, 12, 14, 15-18, and 20-23. For the sake of example, we will also assume that IRQ numbers 3, 5, 7, 10, 11, 13, and 19 are also seen when listing the contents of the /proc/irq virtual directory, and have decided for simplicity to have processors 1-4 (ID's 0-3) handle all hardware interrupt service.

Therefore all numeric interrupt numbers between 1-23 (omitting IRQ 0 and 2) will need to have their SMP affinity masks set. 4. Determine the correct IRQ SMP affinity mask For simplicity we will work with decimal masks. To determine the correct mask, the decimal values represented by the bit number representing each processor we wish to allow a hardware interrupt to execute on are added together. The following table can be used:
Zero-based CPU ID: 7 Decimal Value: 128 6 64 5 32 4 16 3 8 2 4 1 2 0 1

For systems with a larger number of processors, the decimal value doubles with each subsequently higher processor number. For example CPU ID 8 = 256, CPU ID 9 = 512, etc.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 4

How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

In our example, we wish to allow all hardware interrupts be serviced by CPU's 1-4 (CPU ID's 0-3), so we add 1 + 2 + 4 + 8 to arrive at a decimal mask of 15.

/etc/sysconfig/irqbalance provides a variable IRQBALANCE_BANNED_CPUS that provides a similar function, but takes arguments in hexadecimal notation. For example on a 2-socket, quad-core machine (8 total cores), to exclude CPU cores 0 and 1 from irqbalancing, the variable would be:
IRQBALANCE_BANNED_CPUS=fc

On a 4-socket, quad-core machine (16 total cores), the same exclusion would be:
IRQBALANCE_BANNED_CPUS=fc0

5. Set the IRQ SMP affinity mask To set the affinity mask, the calculated mask needs to be echoed into /proc/irq/NUMBER/ smp_affinity, where NUMBER is the IRQ number to be set. For example:
# echo "15" > /proc/irq/7/smp_affinity

This should be repeated for each numeric interrupt number.

Correct behavior can be verified by monitoring the output of /proc/interrupts. It should be noticed with repeated viewing of /proc/interrupts output that IRQ service counts do not increment on processors excluded by the SMP affinity mask.

Note that with some hardware platforms, interrupts can still be serviced by CPUs excluded by the defined affinity mask, but this should be recognized as a hardware or BIOS/firmware implementation problem.

6. Make IRQ SMP Affinity Mask Settings Persistent To make the customized IRQ SMP affinity settings persistent across server reboots, the settings will need to be placed in a file which executes at each server boot, such as /etc/rc.d/ rc.local. While these settings are kernel tunable parameters, they exist outside of /proc/sys and therefore cannot be added to /etc/sysctl.conf.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 5

How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

It is normal to see the isolated processors initially handle some hardware interrupts after a server boot, since the kernel may direct some interrupt work to the isolated processors until the script file containing the custom IRQ SMP affinity settings is executed.

7. Assign Desired Application(s) to the Isolated Processors Application developers can use the sched_setaffinity and sched_getaffinity system function calls directly within an application to set and retrieve the application's CPU affinity, respectively.

Applications can also be set to execute only on a specific processor, or group of processors, using the taskset utility, which is packaged for Red Hat Enterprise Linux in the schedutils RPM.

taskset can be used to modify the CPU affinity of running processes if the program's process id (pid) is known, and can also be used to launch a command or program using the CPU affinity specified on the command line. taskset will accept CPU masks in the form of numeric CPU IDs, as well as in the form of hexadecimal notation bitmasks. For simplicity the examples below will utilize numeric CPU IDs.

Examples
Application already running To set an application which is already running, having pid number 12345, to execute on CPU cores 5-8 (CPU ID's 4-7):
# taskset -p -c 4-7 12345

or this would be equally valid syntax:

# taskset -p -c 4,5,6,7 12345

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 6

How can I dedicate one or more CPUs on a multi-processor system for exclusive use by one or more specific applications?

Start application with specified CPU affinity

# taskset -c 4-7 /opt/foobar1.3/my-special-program

Note: The CPU affinity of a child process created via the fork system function call is inherited from its parent process, so if the application launches several related processes via fork, they will also run on the isolated CPUs.

More information
For more information, refer to the taskset and sched_setaffinity man pages.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 7