KNOWLEDGE BASE

Knowledge Base Article: 000173061

How to address/approach Unified File and Celerra Performance issues (000173061)
Version:10

Audience: Level 40 = Employees and Partners

Original Create Date: Thu Apr 16 10:32:04 GMT 2015 Article Type: How To

Channels: Customer , Internal App

First Published: Wed Nov 06 19:52:17 GMT 2013

Validation Status: Final Approved

Originally Created By: Wesley Bush

Last Modified: Thu Apr 16 10:40:37 GMT 2015

Last Published: Thu Apr 16 10:40:37 GMT 2015

Summary: This How-to article is the starting point for every NAS performance isssue, be they VNX or Celerra.

Goal:

How to address/approach Unified File and Celerra Performance issues

Issue:

Every performance latency issue reported against any VNX-File or Celerra NAS system:
VNX performance issue
NAS performance issue
Celerra performance issue
CIFS performance issue
NFS performance issue
MPFS performance issue
NDMP performance issue
iSCSI performance issue

Environment:
Product:VNX VG10
Product:VNX VG2
Product:VNX VG8
Product:VNX5100
Product:VNX5200
Product:VNX5300
Product:VNX5400
Product:VNX5500
Product:VNX5600
Product:VNX5700
Product:VNX5800
Product:VNX7500
Product: VNX7600
Product: VNX8000
Product: Celerra NS
Product: Celerra NS-120
Product: Celerra NS-480
Product: Celerra NS-960
Product: Celerra NS-G2
Product: Celerra NS-G8
Product: Celerra NS350/NS350S
Product: Celerra NS40
Product: Celerra NS40 DC NEBS Gateway
Product: Celerra NS40 DC NEBS Integrated
Product: Celerra NS40 FC Integrated
Product: Celerra NS42
Product: Celerra NS500-AUX
Product: Celerra NS500/NS500G
Product: Celerra NS600/NS600G
Product: Celerra NS600S/NS600GS
Product: Celerra NS700/NS700S
Product: Celerra NS701G/NS701GS
Product: Celerra NS704/NS704G
Product: Celerra NS80
Product: Celerra NS80G
Product: Celerra NS20
Product: Celerra NS20 FC Integrated
Product: Celerra NS20 Integrated
Product: Celerra NS22
Product: Celerra NX4

Product: Celerra NX4 Dual Data Mover Enclosure

Product: Celerra NX4 Integrated Storage Array
Product: Celerra NX4 Single Data Mover Enclosure
Cause:

Change:

Resolution:

1.
2.
3.
4.
5.
6.
7.

Section 1 : Clarifying the issue ( This is the most important part of the process )
Specific protocols involved ? -- NFS (version), CIFS (version), NDMP, iSCSI, FTP
Is the issue related to a specific export/spare, filesystem/lun ?
What time of day does the issue occur ?
What is the active workload at the time ? login, logout, backup, batch jobs
What hosts are involved ? Backup, VMs, workstations, Servers get as much detail on the applications as possible.
What does the Network config look like ? Specific interface/NIC ?
Are network interfaces isolated on separate nics (ie: replication traffic on it's own single nic)
Note: There is no such thing as too much information.
Section 2 : Enumeration of the environment
1. The Celerra/VNX-FILE component.
Gather the collect_support materials.
/nas/tools/collect_support_materials
Identify features in use; Replication, Dedupe, NDMP, CIFS, NFS, MPFS, etc
Review the network port configuration. Review file system layout from the collect support
2.The Backend Array (Clariion/Symm/VNX-Block).
Gather SPcollects, see Navisphere (or Unisphere) Help article "Generating and transferring SP log files" or:
See EMC Knowledgebase 5253 for instructions on how to obtain SPcollects on a CLARiiON running Navisphere.
See EMC Knowledgebase 13969 for instructions on how to obtain SPcollects on a VNX or CLARiiON running Unisphere.
3.The Network(Customer).
See EMC Knowledgebase 34570 for instructions on how to handle the Global Network Services.
4.The Client Host(s)/Host Environment.
EMC Support link will search for the Grab utilities for different OS types.
Download the utility and the README file. Follow the README file to run the utility for the host operating system chosen.
Alternatively, many of these grabs can also be downloaded from ftp://ftp.emc.com/pub/emcgrab/.

Note: During the initial stages some of the details of the customer network and environment may not be immediately available. If it is necessary to escalate to
necessary.

1.
2.
3.
4.
5.

Section 3 : Collection guidance

Base NAS, Array, Host and infrastructure Collections
Network statistics
Array statistics
Offline Data Mover Statistics - see section 6
Online Data Mover Statistics - see section 5
Link to VNX/File Tiage KBA (189439)

Section 4 : What is available for VNX-File/Celerra performance data.

1.Server_stats is the keystone to VNX-File/Celerra analysis
run "server_stats server_2 -list > stats.list"
Note : running this command generates a list of every statistic that is available a the code this
data mover is running. The first several are the typical ones to use when monitoring a system
online. ( run "grep "\-std" stats.list" to get just the standard system stats, the top talkers -- cifs.client and nfs.client -- will not be listed )
2./nas/tools/profile_slot

This command writes its output to a file on the datamover, as specified with the -w flag. The output of this command is a summary of where each cpu is spend
on "pcsample" or "function". -- the function method is the most frequently used
Section 5 : Live monitoring

Link to KBA for online perf analysis of VNX-File/Celerra

1. At a bare minimum you should have a terminal session running "server_stats server_X -m basic-std -i 30 -te no" at all times. This statistic provided you with CP
a feel for the system activity.
2.

2. With intermittant issues the top talker stats ( cifs.client and nfs.client ) should be running.
3. Save the outputs to a file on one of the file systems built on the internal ide drive of the Control Staton. ( I typically use /celerra/ccc and create a subdir called st
4. You can process the files through "awk" to insert the timestamp at the first field of each line then interrogate the fields of interest in the output of various stats o
Example:
awk '{ if ( $1 !~ /^..:..:../ && NR <= 3 ) { print($0) } ; if ( NR > 3 ) { if ( $1 ~ /^..:..:../ ) { tval=$1 ; print($0) } ; if ( $1 !~ /^..:..:../ ) { print(tval,substr($0,10)) } } }'
Note: this assumes that there are 3 header lines, if there is a different number you will need to update the comparison values in the "if" statements for the built
Section 6 : Offline data analysis

1. The offline data Analysis is performed by senior specialist in either support or engineering and requires a full suite of statistics in the csv format.
2. The folowing statistics must be collected for every performance issue, concurrently with the online collection and analysis, while it is active, and provided with p
does not result in resolution. Every set of csv stats must have details on the times when the issue was active and when normal response times were experienc
3. Escalations must include the various collects and grabs and array statistics.

Always pay strict attention to file system space on the control station as the collection of these csv formatted stats will result in large volumes of data in relative
The server_stats command to be run is as follows :
Celerra 5.6 Family of Code :
server_stats server_2 -summary basic,nfs,cifs,caches -table net,dvol,fsvol,nfs,cifs -interval 10 -format csv -count - > /tmp/stats.txt
Celerra 6.0 & VNX 7.0 Code Families :

server_stats server_X -monitor

basic-std,caches-std,nfs-std,nfs.v2,nfs.v3,nfs.v4,cifs-std,cifs.smb1,cifs.smb2,metaVolumes-std,diskVolumes-std,netDevices-std,store.ioTimeHist,store.scsiBus
-interval 10 -format csv > some-meaningful-file-name.csv

VNX 7.1 Family of Code :

server_stats server_X -monitor

basic-std,caches-std,cifs-std,cifsOps-std,nfs-std,nfsOps-std,metaVolumes-std,diskVolumes-std,netDevices-std,kernel.cpu.utilization,kernel.memory,snap,rep,fs
-vis eng -interval 10 -format csv > some-meaningful-file-name.csv
VNX 8.1 Family of Code :

server_stats server_X -monitor

basic-std,caches-std,cifs-std,cifs.smb1,cifs.smb2,nfs-std,nfs.v3,nfs.v4,metaVolumes-std,diskVolumes-std,netDevices-std,store.diskVolume,store.ioTimeHist,sto
csv > some-meaningful-file-name.csv
note: if cifs is not started on 8.1 the command will not run if any cifs options are included, in earlier code level the server_stats command will start and may fail
started on the mover ).
Additional KB articles to analyze active performance issues.
Link to VNX/File Triage KBA (189439)
Link to KBA for online perf analysis of VNX-File/Celerra
GCS Global Performance Standard Operating Procedure(SOP)
What to look at when having a performance issue on the NAS
How to gather the necessary information for a CLARiiON or VNX performance analysis
Notes:

Notes
(Employees
and
Partners):
Notes
(Restricted):

Product:

VNX/VNXe Family, VNX Application Protection Suite, VNX Event Enabler, VNX1 Series, VNX Storage Array for Mainframe, VNX VG10, VNX VG2, VNX VG8,
VNX5700, VNX5800, VNX7500, VNX7600, VNX8000, VNX for DLm2000, VNX for DLm6000, Celerra, Celerra NS, Celerra NS-120, Celerra NS-480, Celerra N
Celerra NS40, Celerra NS40 DC NEBS Integrated, Celerra NS40 FC Integrated, Celerra NS42, Celerra NS500/NS500G, Celerra NS600/NS600G, Celerra NS6
NS701G/NS701GS, Celerra NS704/NS704G, Celerra NS80, Celerra NS80G, Celerra NS20, Celerra NS20 FC Integrated, Celerra NS20 Integrated, Celerra NS
Celerra NX4 Integrated Storage Array, Celerra NX4 Single Data Mover Enclosure

Shared:

Yes

Originally
Created By:

Wesley Bush