unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

Middleware Administration Search Blog

unix administration @ gurkulindia.com

Home Announcements Beginners zone Beginners Lessons Career Guidance Experts Zone Cloud Computing Network Design Scripting Server Security SUN CLUSTERS Veritas Cluster Services ( VCS ) Learning Intermediate Zone Redhat Linux Learning Linux Disk Management Linux LVM Linux Networking Linux Performance Linux Troubleshooting Redhat Linux Kernel Solaris Admin Blog for Unix Admin Storage Administration SAN Reference Docs Solaris 10 Zones & SMF Solaris Access Control Solaris Best Practices Solaris Booting Solaris Disk Management Solaris DNS Solaris How-to Solaris Installation Solaris Kernel Solaris Networking Solaris NFS Solaris NIS Solaris Packages & Patching Solaris Performance Solaris Tips Solaris Troubleshooting Solaris User Authentication solaris X86 Storage Configurations SUN Hardware Troubleshooting Flow charts Veritas Admin VxVM Learning VxVM Troubleshooting Unix Book Subscribe

Solaris Performance: Identify memory usage by a specific process

1 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

Connect us to Contact -

Gurkulindia .com

Ramkumar Ramadevu

Understanding how much memory individual processes are using, and what types of memory are in use is both a simple and difficult question to answer. Simple because there are a number of tools that can report the memory usage per process, but difficult to explain, as processes can share memory in a number of ways; This post purposed to tell the user a little about process memory usage, and explain how they can determine how much of what types of memory are is use by a given process. Steps to Follow The Simple Method to observe memory usage by process (Solaris 8 and later) prstat for realtime monitoring, prstat -s rss prstat -s size Depending on the memory type the user is interested in. rss is a close estimate of the physical memory in use, whereas size is a close estimate of total virtual memory in use by the process (Physical + Swap). For point in time viewing below command will show the highest memory users, sorted by RSS (Physical memory in use). # ps -efly | sort +7n S UID PID PPID C PRI NI RSS SZ WCHAN STIME TTY TIME CMD S root 2 0 0 0 SY 0 0 Jul 18 0:00 pageout S root 3 0 0 0 SY 0 0 Jul 18 318:02 fsflush T root 0 0 0 0 SY 0 0 Jul 18 3:17 sched R root 233 1 0 40 20 632 1060 Jul 18 0:10 /usr/lib/utmpd S root 400 398 0 40 20 696 2084 Jul 18 0:00 /usr/sadm/lib/smc/bin/smcboot S root 401 398 0 40 20 696 2084 Jul 18 0:00 /usr/sadm/lib/smc/bin/smcboot S root 3112 193 0 50 20 768 5696 19:40:39 console 0:00 sort +7n S root 228 7 0 40 20 788 1072 Jul 18 console 0:00 -sh O root 3111 193 0 40 20 808 1260 19:40:39 console 0:00 ps -efly S root 118 1 0 40 20 844 1272 Jul 18 0:00 /usr/lib/power/powerd S root 219 7 0 40 20 908 1696 Jul 18 0:01 /usr/lib/saf/sac -t 300 S root 234 1 0 40 20 944 2236 Jul 18 0:00 /usr/sbin/cron S root 1 0 0 40 20 1064 2004 Jul 18 0:10 /sbin/init S root 236 219 0 40 20 1072 1800 Jul 18 0:01 /usr/lib/saf/ttymon S root 184 1 0 40 20 1136 2212 Jul 18 0:00 /usr/sbin/keyserv S root 355 1 0 40 20 1224 3332 Jul 18 0:00 /usr/lib/ssh/sshd

2 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

S root 457 1 0 40 20 1252 2036 Jul 18 0:02 /usr/lib/snmp/snmpdx -y -c /etc/snm S root 104 1 0 40 20 1300 2136 Jul 18 0:00 /usr/lib/sysevent/syseventd S daemon 178 1 0 40 20 1356 2512 Jul 18 0:03 /usr/sbin/rpcbind S root 398 1 0 40 20 1388 2088 Jul 18 0:00 /usr/sadm/lib/smc/bin/smcboot

This is great for viewing things from a systemic perspective. The more detailed per process view What if we want to see more detail on a single process? It depends on the OS and what you mean by memory:

See example output below: $ /usr/proc/bin/pmap 1701 1701: /bin/ksh 00010000 184K read/exec /usr/bin/ksh 0004C000 8K read/write/exec /usr/bin/ksh 0004E000 40K read/write/exec [ heap ] EF600000 592K read/exec /usr/lib/libc.so.1 EF6A2000 24K read/write/exec /usr/lib/libc.so.1 EF6A8000 8K read/write/exec [ anon ] EF6C0000 16K read/exec /usr/platform/sun4u/lib/libc_psr.so.1 EF6D0000 16K read/exec /usr/lib/libmp.so.2 EF6E2000 8K read/write/exec /usr/lib/libmp.so.2 EF700000 448K read/exec /usr/lib/libnsl.so.1 EF77E000 32K read/write/exec /usr/lib/libnsl.so.1 EF786000 24K read/write/exec [ anon ] EF790000 32K read/exec /usr/lib/libsocket.so.1 EF7A6000 8K read/write/exec /usr/lib/libsocket.so.1 EF7A8000 8K read/write/exec [ anon ] EF7B0000 8K read/exec/shared /usr/lib/libdl.so.1 EF7C0000 112K read/exec /usr/lib/ld.so.1 EF7EA000 16K read/write/exec /usr/lib/ld.so.1 EFFFC000 16K read/write/exec [ stack ] total 1600K

In the above example, the first column is the address of the segment (Within that processes address space, NOT a systemwide virtual address), the second is the size of the mapping at that address, the third is the permissions and the fourth tells where the segment comes from (library, binary, etc.).

An even more exciting view of the processes memory usage is the pmap -x output. # pmap -x $$ 529: ksh -o vi

3 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

Address Kbytes Resident Shared Private Permissions Mapped File 00010000 200 200 200 read/exec ksh 00052000 8 8 8 read/write/exec ksh 00054000 32 32 24 8 read/write/exec [ heap ] FF180000 688 632 632 read/exec libc.so.1 FF23C000 32 32 32 read/write/exec libc.so.1 FF270000 16 16 16 read/exec libc_psr.so.1 FF280000 568 536 536 read/exec libnsl.so.1 FF31E000 40 40 40 read/write/exec libnsl.so.1 FF328000 24 16 16 read/write/exec libnsl.so.1 FF340000 16 16 16 read/exec libmp.so.2 FF354000 8 8 8 read/write/exec libmp.so.2 FF370000 8 8 8 read/write/exec [ anon ] FF380000 40 40 40 read/exec libsocket.so.1 FF39A000 8 8 8 read/write/exec libsocket.so.1 FF3A0000 8 8 8 read/exec libdl.so.1 FF3B0000 160 160 160 read/exec ld.so.1 FF3E6000 16 16 16 read/write/exec ld.so.1 FFBEC000 16 16 8 8 read/write/exec [ stack ] total Kb 1888 1792 1744 48

This output includes a great deal of detail, including the size of the mapped segment, how much if it is RSS (Real memory), how much of that is shared, the sizes and permissions of the segment. It is the output above that can be compared over time to see if a given process is growing, or leaking memory.

Shared memory

Though the above output is excellent in terms of seeing how much of what type of memory is mapped per process, its still not the complete answer, as EVERY processes that maps the same shared resource will include that segment in its RSS.

This means that if, for example, 5 processes mapped the same 5GB segment, it might appear to the casual observer that there is actually 25GB of memory in use by those 5 processes, whereas in reality its the same 5GB mapped into 5 different address spaces!

Shared memory segments, and the number of processes attached can be viewed using ipcs -ma

Which will return something that should resemble this: $ ipcs -ma

4 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

IPC status from Server1 as of Monday April 30 16:17:47 SGT 2011 T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Shared Memory: m 553648156 02116 rw-rw- ms157374 staff bobby staff 0 64528 8470 8470 11:21:35 11:41:05 11:21:35

The most interesting column from the above output is SEGSZ, which is the size of the segment in bytes. Shared memory makes things very hard to directly attribute memory usage to individual processes, as every process should indeed report that it is using that memory, however, which process should it be attributed to? It is possible in many cases that the creator of the segment no longer exists. The summary is that for a quick and dirty view, prstat or ps are great alternatives. If you really need to understand which processes are using what memory, it will require a combination of pmap -x, ipcs and some reasonable Application knowledge to understand which processes would be attached to given shared memory segments. An example case follows on a system with 4GB of memory, running Oracle This can be observed in the startup messages from the database: SQL> ORACLE instance started. Total System Global Area 1291845632 bytes Fixed Size 1978400 bytes Variable Size 335548384 bytes Database Buffers 939524096 bytes Redo Buffers 14794752 bytes Database mounted. Database opened.

The SGA is located in shared memory so that multiple processes can attach to, map and use that segment in their own address space. We can see this shared memory in the output of ipcs -mbop # ipcs -mbpo IPC status from Server1 as of Tue Oct 2 08:33:43 SGT2011 T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID Shared Memory: m 1 0x8f430250 rw-r oracle oper 35 1291853824 14195 14274

Note that the shared memory segment size is 1291853824 bytes, approximately 1.1GB.

Below command shows the multiple oracle processes that are attached to this segment. # prstat -s rss PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 14203 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11 14207 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11 14205 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11 14209 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11 14221 oracle 1403M 1347M sleep 47 4 0:00:01 0.0% oracle/1 14211 oracle 1413M 1347M sleep 47 4 0:00:00 0.0% oracle/11 14213 oracle 1402M 1347M sleep 47 4 0:00:00 0.0% oracle/11

5 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

14219 oracle 1402M 1346M sleep 47 4 0:00:00 0.0% oracle/1 14197 oracle 1402M 1345M sleep 47 4 0:00:00 0.0% oracle/1 14272 oracle 1401M 1345M sleep 47 4 0:00:00 0.0% oracle/1 14227 oracle 1401M 1345M sleep 47 4 0:00:00 0.0% oracle/1 14215 oracle 1400M 1345M sleep 47 4 0:00:01 0.0% oracle/1 14217 oracle 1400M 1345M sleep 47 4 0:00:00 0.0% oracle/1 14225 oracle 1401M 1345M sleep 47 4 0:00:00 0.0% oracle/1 14223 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1 14265 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1 14274 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1 14199 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1 14201 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1 709 noaccess 293M 150M sleep 59 0 0:03:28 0.0% java/55 9791 root 208M 75M sleep 59 0 0:00:19 0.0% iscsitgtd/11 Total: 67 processes, 331 lwps, load averages: 0.03, 0.04, 0.02

Recall that this system has only 4GB of memory: $ prtdiag System Configuration: Sun Microsystems sun4v SPARC Enterprise T2000 Memory size: 3968 Megabytes <> # pmap -x 14203 | grep -v lib 14203: ora_dbw0_orcl Address Kbytes RSS Anon Locked Mode Mapped File 0000000100000000 100128 99400 - r-x oracle 00000001062C6000 808 808 320 rwx oracle 0000000106390000 448 448 448 rwx [ heap ] 0000000106400000 8192 8192 8192 rwx [ heap ] 0000000380000000 1261576 1261576 1261576 rwxsR [ ism shmid=0x1 ] <>

Note above that the ism (Intimate Shared Memory) shmid (Shared Memory Identifier) is 01. This can be directly matched to the ipcs output seen previously, mapping the shmid to the ID column in the ipcs -mbop output. Thus, the processes can be directly associated with the shared segment, and memory usage better understood. If desired, the size of the shared memory segment can be subtracted from the processes total memory usage, to provide an idea of the individual processes memory usage, not including the shared memory segment(s).

Share to a Friend

6 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

Email

Facebook

Share

Digg

Reddit

Share

we respect your email privacy

Check confirmation email for download link

Posted by Ramdev Comment it You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Comment
Name (required) Mail (required) Website

Pages 1

7 of 8

3/17/2012 4:19 PM

unix administration @ gurkulindia.com Solaris Performance: Identify...

http://gurkulindia.com/main/2011/02/solaris-performance-identify-mem...

Copyright 2009 unix administration @ gurkulindia.com. All rights reserved.

Gurkulindia.com

8 of 8

3/17/2012 4:19 PM