Further instructions for rootvg filesystems

In order to cut number of tickets sent by CC to 2nd level Unix, we have been officially
asked to consider and perform the following instructions:

To check if a filesystem is mirrored or not, refer to the following document Notes Link
To check if a filesystem is inside rootvg use: lsvg -l rootvg |grep my_file_sytem
Bellow you find some general guidelines for filesystems belonging to rootvg volume group on AIX
systems:
If the following log files are too big, rotate them
/var/adm/wtmp
/var/adm/sulog
/var/adm/cron/log
/opt/IBM/SCM/client/jacclient.log
/scmfs/IBM/SCM/client/jacclient.log
log rotation means that the original log file is compressed (eg. .gz) and truncated to zero length. The
point of log file rotation is to prevent log files from growing indefinitely.
The way to rotate logs is by command "gzip"
It is always important to gzip the original file with parameter -c because it keeps original files unchanged.
Example:
gzip -9 -c /var/adm/wtmp > /var/adm/wtmp$(date +%Y%m%d).gz && > /var/adm/wtmp
-9 is level of compression ( 9 is highest one)
> redirection of the original one to compressed one with actual time stamp
&& means: if previous command finished successfully then perform next one
the last " > " will empty the original file

Before launching this command check for free space in filesystem and if there is not
enough space use another filesystem to temporary store the compressed copy (i.e.
/tmp/some_log_file.log.gz). After the compression move back the compressed file.
USEFUL TIP: du -m /var | sort -n -r | head -n 10
in /var
2. . if directory with httpd logs (usually
Rotate as shown above in the example.

<---- shows the 10 biggest files

/var/log/httpd/ ) is too big, rotate old logs

3. check /tmp file system and try to clean up core dumps (always check by "file" command if it's core
dump), java heapdumps etc. and files older than uptime (since the last time server was started)
Use this command to find the cores in a given filesystem ls /tmp | egrep -i 'core|dump' | xargs du -ms

core files removing

You should perform this additional check before removing some core file.
file <corefile>
(example: file /usr/local/bin/core)
Expected output is:
/usr/local/bin/core: AIX core file 32-bit, lsof
If there is not present the output similiar to red part (AIX core dile or AIX core dump) or
if the blue part is directory then DO NOT remove such core file as it can be integral
part of some application.
Also check if file is not being used at the moment..for example: fuser /usr/local/bin/core
it will produce a process ID then use ps -ef | grep -i PID to check when it was last
accessed.
the way to remove a file is rm -f /nameOFfile
4. if /home directory is too big, find the user with the biggest homedir (command finger) and ask him for
clean up
To check space ls /home | xargs du -ms (you may have to be root to perform the du command)
This will list files and their sizes. If some ID of an administrator seems too big compared to others, use the
F command to know who it is and contact him (sametime or email) to inquire about the extra space and if
possible to free some space.
example: f CZY9AEHB
You may also find sap or DB application users..etc with big sizes..in this case you may ask the
responsible team about it or you may need to open a ticket to the responsible team depending on the
situation.
5. if there are free PPs in rootvg don't hesitate to extend file system by chfs command
(example: sudo chfs -a size=+64M /home) or get root access and use (smitty chfs) command
6. if there are no free PPs in rootvg try to take space from other file systems (namely /appoggio, /kit_aix)
sudo chfs -a size=-512M /appoggio
or by taking root access and using smitty chfs
or you may take space from other filesystems within rootvg volume group but make sure you do not take
too much space from the other filesystem so that it doesnt go below threshold

7. if there is no space in /appoggio and /kit_aix, delete files older than 3 months
These filesystems contain temporary installation files which can be deleted if newer files are present so
the 3 months rule is not strict. It differs from one system to another.
find /appoggio -xdev -mtime +90 -type f -exec rm -f {} \;

<--- deletes files older than 3 months

Open ticket only if some file system is growing continuously or there is some problem with queue
deamon or mail queue
/var/spool/mqueue
/var/spool/qdaemon

Mountpoint

Used for

Instructions

/tmp

system
(appl. temp.)

Find and remove old unused files (files older than 60 days). Check by
lsof or fuser commands if they are not in use at the moment !

/nmon
/NMON

system
(monitoring)

Compress the files older than 7 days if necessary or add some space to
filesystem if needed. Usually these filesystems should be automaticaly
cleaned up. If there are sored also very old files you should inform the
BackOffice that log retention is not working properly.

/home
/utenti
/utentiIBM
etc.

home dirs

/cdrom
/CDROM

media
mountpoint

This filesystem may be used as mountpoint for CD/DVD installation

media. In such case the filesystem can be reported as full (no write
permissions).

/swbase

applications

Find and remove core files. Pay attention as these filesystems contain
the installed software !

Find and remove core dump files, compress old *.tar files. Extend the
filesystem if needed,

AIX:
Mountpoint
/
/usr
/opt

Used for

system
(binaries)

Instructions
Find and remove core dump files. You can extend the filesystem if it's
necessary (i.e. after some installation of SW).
Pay attention to the type of filesystem as JFS filesystems can not be
reduced online later so you can waste the free space !

/var

system
(variables)

Find and remove core dump files. If needed you can compress to backup
copy and empty these files:
/var/adm/wtmp
/var/adm/sulog
/var/adm/cron/log
Check for the size of /var/spool/mqueue if there are plenty of files and
size it's more than 100 MB open the ticket to BackOffice.
Do not extend this filesystem if not absolutely necessary !

/kit_aix
/ML
etc.

system
(updates)

These filesystems are frequently used to perform the APARs, updates,

etc. Do not remove anything in these filesystems. You can expand them
if they are JFS2 type, otherwise consult the solution with BackOffice.

AIX shared filesystems (HACMP cluster)

Any shared filesystem operation can be done only using the smitty hacmp fastpath.
DO NOT resize shared filesystem using the chfs command as this can lead to incorrect
cluster operation later.
Pay attention also to the filesystems in HACMP/XD clusters as these cluster are very
often using different ways how to ensure that data are consistent on both cluster nodes.
For HACMP/XD try to run:

lsdev | grep i remote

If you will see Remote Physical Volume Client or Remote physical volume server do
not try to resize shared FS and send ticket to BO AIX for solution. Wrong operations
over these filesystems can crash the cluster or damage data !
HP-UX:
Mountpoint

Used for

/
/stand

system
(binaries)

/var

system
(variables)

Instructions
Find and remove core dump files.
Do not try to extend these filesystems !
Find and remove core dump files.
Do not remove anything under /var/adm/sw !

SunOS/Solaris:
Mountpoint
/

/var

Used for
system
(root)

system
(variables)

Instructions
Find and remove core dump files.
Do not try to extend this filesystem !
You can try to check for files older than 3 months in
/var/crash/<machine_name> and delete them.
Files /var/adm/messages.[0-9] can be compressed.
Do no remove anything under the /var/run !

Linux:
Mountpoint

Used for

system
(root)

/var

system
(variables)

Instructions
Find and remove core dump files.
Try to check for old .tar, .gz, .tgz, .Z files in /var/tmp they can be
removed if not in use.
Some system log files in /var/log can be compressed to backup copy, if
you are not sure contact the BackOffice for advise.

******************************************************************************
************
!!! everytime when you want to enlarge FS you must check if there is GLVM cluster. !!!

1) Use command lslpp -L | grep -i cluster|grep -i xd

2)Then use command lsdev -Cc disk

If there will be some "remote disks" or smth, it means that cluster is glvm

Application filesystems & other tasks

WebSphere/Apache/Tomcat filesystems
The BO team for middleware is taking care not only for the application itself, but they
also manage the problems related with it. Therefore the filesystem used by mentioned
services should be managed by CC_SSOTEBOWBSPH group. If you are not sure if the
filesystem is used by such application:
/WebSphere/ - filesystem used obviously by WAS
/appl/
/swbase/
/Appl/
/usr/Apache/
etc.
- try to ps eaf | grep i websphere
ps eaf | grep i httpd
and check if there is the filesystem name contained in the output

core files removing

You should perform this additional check before removing some core file.
file <corefile>
(example: file /usr/local/bin/core)
Expected output is:
/usr/local/bin/core: AIX core file 32-bit, lsof
If there is not present the output similiar to red part (AIX core dile or AIX core dump) or
if the blue part is directory then DO NOT remove such core file as it can be integral
part of some application.
File compression/rotation

Pay attention to files you are going to compress ! You should be absolutely sure that it's
not going to harm operating system or application functionality. Basically there is
possible to compress .tar files as most of them are installation packages or some
backup archives and compression will not afftect their content.
If you want to compress some log files which are used by system/application (and you
know it's possible to compress it) you should use compression to another copy:
gzip -9 -c some_log_file.log > some_log_file.log.gz && > some_log_file.log
Before launching this command check for free space in filesystem and if there is not
enough space use another filesystem to temporary store the compressed copy (i.e.
/tmp/some_log_file.log.gz). After the compression move back the compressed file.
Basic usefull commands
show how much space is used:
AIX
df g
Linux
df h
HP-UX
bdf
Solaris
df kh
show files bigger than 50 MB:
find /full/file/system/ -xdev -type f -size +100000 -exec ls -l {} \;
show biggest directories (hp-ux du -skx ; solaris du -skd ):
du -smx /full/file/system/* | sort n