Sei sulla pagina 1di 50

9087VC Ports :-

HTTP = 80 ; HTTPS =443 ;


Heart Beat Port =902 [Communication Port]
Web Server port = 8086 ; SNMP = 161[Incom] ,162 [Out] ;
TCP/ IP=7 FTP=20,21 SSH = 22 ; Telnet =23
NTP = 123 ; DNS=53 ; DHCP=67
Software iscsi = 3260 ; V Motion = 8000 ;
HA = 2050 – 2250 , 8042 – 8045 ; RDP = 3389 ; LDAP = 389 ;

TO Configure N/W Ing From Esx service Console Command Line:-

***Ensure The Network Adapter you want to use is currently connected


with the command……..
# esxcfg–nics– –l

***List Current Virtual Switches With The Command……………


#esxcfg–vswitch –l

***Create a New V-Switch With the Command…………..


#esxcfg–vswitch –a vswitch1

***Create a Service Console Port Group On this New V-Switch………


#esxcfg–vswitch –A”service console” vswitch1

***Up-link vmnic1 to the new Virtual switch …………..


#esxcfg–vswitch –l vmnic1 vswitch1

***if you need to assign a vlan…………………


esxcfg–vswitch –v <vlan> –p”service console” vswitch0

***Verify the new V-Switch configuration….


………………………………………………………………………………………
………………
***
Vmotion
It Is One Of The Vc Feature.
• Moving Running Virtual Machines From One Esx Host To Another Esx
Host Without Down Time Is Called Vmotion.
Vmotion Works By Changes In Memory Bitmap

Why We Go For Vmotion


1. Host Maintenance Purpose
• While Updating Esx Host Updates.
• While Increasing Service Console Memory Up To 800 Mb.
• While Adding Any Hardware Like: Memory, Cpu, Disk And Nic.
In The Above All Conditions Esx Host Reboot Is Required, So We Have
To Place Esx Host Into Maintenance Mode.
2. Continuous Vm Operations
Insufficient Resources In Our Vm’s In That Scenario For Vm Continuous
Operations Purpose Better To Move Vm’s From One Esx Host To Another
Esx Host.
3. Better H/W Utilization
Insufficient Resources In Our Esx Host In That Scenario Better To Move
Vm’s From One Host To Another Host.
While Moving Vm’s From One Host To Another Host We Have To Meet
Some Conditions.
(1) CpuCompatibilty (Cpu Should Be Same Family Either Intel (Or) Amd).
(2) Storage Should Be Common (Or) Shared Storage.
(3) Resources Should Be Available In Destination Host.
(4) Physical N/W Should Be Same Network.
(5) Create Vmkernel Port Group In Both Source & Destination Host’s.
(6) No Access Of Mountable Devices Ex: Cd, Floppy.
(7) No Permission ToVmx File.
Svmotion
Moving Running Virtual Machine Disk Files From One Shared Storage To
Another Shared Storage Is Called Svmotion.
HA:
• It Is One Of The Vc Feature And Fail Over Mechanism.
• Used For Continuous Availability For Vm’s
• While Creation Of Cluster In Vc We Have A Option Ha Cluster.
• Vc Allows Virtual Machines To Migrate & Restart On Another Esx
Host When The Esx Host Fails.
While MigratingVms From One Host To Another Host We Have To
Meet Some Conditions.
(1) CpuCompatibilty (Cpu Should Be Same Family Either Intel (Or) Amd).
(2) Storage Should Be Common (Or) Shared Storage.
(3) Resources Should Be Available In Destination Host.
(4) Physical N/W Should Be Same Network.
(5) Create Vmkernel Port Group In Both Source & Destination Host’s.
(6) Host Should Be Configured With Fqdn
(7) Dns Should Be Configured With Host Name.
HA CONFIGURATION
• First Login IntoVc.
• Right Click The Data Center.
• Select New Cluster Option.
• Here We Have Two Options One Is Ha (Failover Mechanism) &
Second One Is Drs
(Load Balancing Mechanism).
• Now Enable Ha Cluster-->After That We Will Get A Wizard For Max
Host Failures Default Is 1 , Maximum Is 4.
• Vm Priority Default Is ---Medium.
• Isolation Response Default Is -----Power Off.

Is Ha Dependent On Virtual Center:


Only For Install
How does HA know to restart a VM from a dropped Host?
Storage lock will be removed from the metadata
What is the Maximum Host Failure allowed in a cluster?
4
How Many IP Addresses required for Cluster Nodes?
HA Cluster 32 IP Addresses &DRS Cluster 32 IP Addresses.

DRS :
• It is one of the VC feature and its main function is Load Balancing
Mechanism.
• It balances the Virtual Machine Load across Hosts in the Cluster.
• While creation of cluster in VC we have option to create DRS cluster.
• In DRS cluster we can create resource pools.
• We can create resource pools either in standalone hosts (or)
Clustered hosts.
• We can create 512 Resource pools under Host level, 128 under
clustered.
DRS Conditions.
(1) CpuCompatibilty (Cpu Should Be Same Family Either Intel (Or) Amd).
(2) Storage Should Be Common (Or) Shared Storage.
(3) Resources Should Be Available In Destination Host.
(4) Physical N/W Should Be Same Network.
(5) Create Vmkernel Port Group In Both Source & Destination Hosts.
(6) Host Should Be Configured With Fqdn
(7) Dns Should Be Configured With Host Name.
DRS CONFIGURATION
• First Login into VC .
• Right Click the Data Center.
• Select New Cluster Option.
• Here we have Two Options one is HA (Failover Mechanism)&Second
one is DRS
(Load Balancing Mechanism).
• Now Enable DRS cluster.-->after that we will get a wizard for .
• Automation Level Recommended is –Fully Automation.
• Coming to Migration Threshold Recommended is 3 (or) 4 Star.
• Select DRS Rules (Affinity/In-Affinity).
• Select VM automation level –Partial Automation.
• Select Vm Swap file Location VM Directory Recommended.
• Click on Finish.
WHAT IS VIRTUALIZATION
• Virtualization is a technology that allows you to transform hardware
into software
• Virtualization allows multiple Virtual Machines to run concurrently
on a single computer
WHAT IS VM
Virtual Machine is a computer which looks like a Physical Machine
but it is software Generated Machine. It contains discrete set of files
like
(a) Virtual Machine Configuration file --- vmname.vmx.
(b) Virtual Machine Disk file --- vmname.vmdk.
(c) Virtual Machine Bios File --- vmname.nvram.
(d) Virtual Machine Log file --- vmname.log.
(e) Virtual Machine Swap file --- vmname.vswp.
(f) Virtual Machine Snapshot List ---
vmname.vmsd.
(g) Virtual Machine Snapshot Main File --- delta.vmdk
(h) Virtual Machine Snapshot Data ---
Vmname.vmsn
(i) Virtual Machine Extend Config ---
Vmname.Vmxf
(j) Virtual Machine Template --- Vmname.Vmtx.
Benefits:
(i) It is Easy to Move and Copy.
(ii) It is Easy to Manage.
(iii) It encapsulated a set of files
(iv) VMotion is one of the Main Features.

VMWARE TOOLS
After installing guest o.s we must install VMware tools to that
particular vm. SinceTools improve mouse performance, install device
drivers, memory management, Time synchronization and ability to
gracefully shutdown.
HOW ESX SERVER WORKS (OR) HOW DOES VIRTUALIZATION
WORKS.
• Virtualization uses either hosted or hypervisor architecture.
• A host-based virtualization system requires an operating system
(such as Windows or Linux) to be installed on the computer. Ex:
Workstation 7.0
• ESX Server is a bare-metal hypervisor system which does not require
an operating system. The hypervisor is an operating system, ESX
Server works based on X86 Architecture.
• ESX 3.5 Server manages up to 170 Virtual Machines.

Why the Companies Build Cloud Infrastructure in VMware?


• Shared IT Resources.
• Dynamic Resource Allocation
• Integrated Virtual Infrastructure Management Features.
• Application & Infrastructure Management Tools.
• Disaster Recovery & Patch and Update Manager Features.
• H/w Support & Guest Operating System Support

What is the difference between VMware &Hyper-V


VMWARE HYPER-V
70-100 MB foot Print for ESX More than 2GB required
Server
Optimized with H/w Vendors Generic windows Drivers
VmwareVstorage VMFS Lacks of Integrated clustered file
system
Ability to reclaim unused No ability to reclaim unused &
&Dup.memory Duplicate Memory
Direct Driver Mode I/O Bootle neck in Parent O.S
Virtual Security Technology Nothing Comparable.

DIFFERENCE BETWEEN VI3 & VSPHERE 4.0


VIRTUAL VIRTUAL
INFRASTRUCTURE 3.5 INFRASTRUCTURE 4.0
ESX 3.X & Virtual Center 2.x ESX 4.0 & Virtual Center 4.0
It supports for 32 Bit as well as 64 Bit It supports only for 64 Bit.
We can create upto 170 VM’s for We can create upto 320 VM’s for Each
each Host Host.
Virtual Center 2.5 allows 200 Host’s Virtual Center 4.0 allows 500 Host’s &
& 2000 Vm’s 5000 Vm’s
Virtual Infrastructure Client 2.5 VSphere Client 4.0
In 3.5 we are using Standard Switch In 4.0 Standard Switch as well as Virtual
Distributed Switch
Down Time for Changing No DownTime because it contains HOT
CPU,Memory,Disk,NIC. Add’s feature.
Virtual Center Consolidate Backup It contains Data Recovery function
(VCB)
3.5 Update Manager is 2.5 4.0 Update Manager is 4.0
HA with Minimum Downtime HA+FaultTolerence so No Down Time.
VM Version is 4. VM Version 7.

Diff Between VMware ESX & VMware ESXi


1) ESX Supports to Boot from SAN but ESXi does not support to SAN.
2) Remote console applications like Dell DRAC, HP ILO, IBM RSA, and
FSC IRMC S2 are supported with ESXi.
3) ESX supports to Web Access but ESXi does not support.
4) ESX Supports to SNMP but ESXi does not support to SNMP.
5) ESX Supports to Active Directory Integration but ESXi does not
support to AD.
6) ESX Service Console is a standard Linux. It does not contain Service
Console.
7) ESX Host supports to VC Features but ESXi does not support to VC
Features.
Difference between TOP and ESX TOP Commands
• VMware ESXTOP Command is used for Storage Monitoring
• ESX TOP show the Service Console Performance (or) ESX HOST
performance.
• Monitors overall storage performance per HBA
• Monitors storage performance per LUN
• Monitors storage performance per VM
• Top command is used to shows how many users are logged in to
Virtual Centre.

How to restart ESX Management Agent


#service mgmt-vmware restart

How to check the state of VM using command line


#vmware-cmd<vmpath>getstate.

How to shutdown a VM using command line


#vmware-cmd<vmpath>stop (or) stop hard.

Service console size.


Service console default size is 272 MB default, maximum 800 mb.
What is VI Client
• It is a primary user component to enter into Virtual Infrastructure.
• It is a Graphical user Interface used to Configure ESX Host and
Manage Virtual machines.
• One of the main tab in VI Client is Host Configuration. It contains 2
Sections, H/W & S/W Configurations.
• H/W section contains Processors, Memory, Storage, Networking
(Number of Switches), Storage Adapters (HBA information) &
Network Adapters (No. of Physical Adapters Information).
• S/W Section contains ESX License features, Time Configuration
(NTP), DNS and Routing, VM startup/shutdown/swap file location,
security profile (Firewall Settings) & Host resource allocation (CPU,
Memory and Disk).
• VI client also contains VM Performance, user &groups permissions.
What is V C
• It is Centralized Management Tool, VC is installed on windows
machine to centrally manage Multiple ESX Hosts & VM’s
• H/W Requirements: CPU: 2.0 GHZ (or) higher Intel, AMD. Memory:
2GB minimum. Disc: 560MB Minimum, 2 GB Recommended. NIC:
10/100 GB.
• S/W Requirement:- 32 bit version, win-2000 with sp4, win-2003 with
sp1,sp2.
• VC contains advanced features like VMotion, SVmtion, HA & DRS.
• During VC installation, the components installed by default are: VI
Client, License Server, Database Server, VMware Update Manager,
VMware Converter Enterprise and VCserver.

VC AGENT,HOW IT WILL WORK?


• The VirtualCenter Agent, also referred to as vpxa or the VMware-
vpxa service, is what allows a VirtualCenter Server to connect to a
ESX host.
• Specifically, vpxa is the communication conduit to the hostd, which in
turn communicates to the ESX kernel.
• #service VMware-vpxa start.
You don’t have virtual center and you need to copy a running VM,
what to do?
1. Copy .vmx& .Vmdk both files and paste it in target machine
2. Right click on .vmx fileclick on add inventory option.
WHAT IS VMKERNEL?
The VMkernel allows the virtual machines as well as the service
console access to the system’s hardware.

DATA CENTER
• After installing VC we can create DC.
• DC is a primary container of Virtual Center Inventory Objects, such as
Hosts, VMs
• From DC we can add & organize Clusters, Hosts and Folders.
• VC server contains multiple Data Centers.
• Inventory objects can interact within Data Centers but interaction
across Data Centers is limited.
RESOURCE POOL
• A Pool of Resources reserved for a Virtual Machine
• We can Add Resource Pools in Host Level & Clustered Level
• In Host Level we can add up to 512 Resource pools & in Clustered
level 128.
• While creation of a Resource Pool, it provides different options like
Reservation, Share, Limited, Unlimited and Extent Options.

TEMPLATE
• It is one of the virtual center features.
• Template is Master copy of the Virtual Machine.
• By Template we can deploy ‘N’ number of Virtual Machines.

MULTIPLE VM DEPLOYMENT BY TEMPLATE


(1) Login into VC by VI Client.
(2) First we need to take approval from Client.
(3) Prepare Project plan for VM Deployment
(4) Collect Resources Information from Manager Ex: CPU, Memory,
Disk and NIC.
(5) Before we perform Deployment we need to download sysprept file
and prepare Custom Specifications.
(6) Go To Inventory and select Virtural Machine & Template Option.
(7) Right Click Template select Deploy VM from Template option.
(8) Provide VM Name.
(9) Select Host Name/ClusterClick on Next.
(10) Now Select Data Store
(11) Now we need to select Guest Customization Option.
a. Do not Customization
b. Customization with Customization wizard.
c. Customization with Existing Customization Specifications.
(12) Click on Finish.

CLONE
• It is one of the VC feature.
• Clone is a duplicate copy of virtual machine.
• It is used in Testing & Development Environment.

CLONE VIRTUAL MACHINE


• First we need to take Approval from Manager/Client and Prepare
Project Plan.
• Login into VC by VI Client.
• Right click VM select Edit Settings Uncheck Connect at Power on
Option.
• Now right click VM select CloneProvide Clone VM Name.
• Select Host/Cluster to store Clone VM.
• Select Data Store  Click on Next.
• Now we need to select Guest Customization Option.
(a) Do not Customization
(b) Customization with Customization wizard.
(c) Customization with Existing Customization Specifications.
• Click on Finish
• Login into Clone VM Provide Network SettingsPower on VM.
• Now Select Primary VM enable Connect at Power on Option.

SNAPSHOT
• It is one of the VC feature.
• It is a pre reserved state of the Virtual Machine.
• It is used in Testing & Development Environment.
• When we need to revert back the same state of VM without any
changes in VM it is very useful.
• Snapshot is used for UPDATES & CLONE Is used for
UPGRADATION.

LOCKDOWN MODE
It is used to Prevents administrators from trying to directly manage
ESX Hosts that are already being managed by a Virtual Center.

Maps: It shows relationship between VMs, hosts, data stores & networks
VMWARE ESX NETWORKING CONCEPTS
• Networking features of ESX Server allow virtual machines to
communicate with other virtual machines within the same box and
with outside world.
• Allows the service console to communicate and allow the VMkernel to
take advantage of IP-based storage and VMotion.

There are three types of network connections


• Service console port:access to ESX Server management network
• VMkernel port: access to VMotion, iSCSI and/or NFS/NAS networks
• Virtual machine port group: access to VM networks
• More than one connection type can exist on a single virtual switch.
• We can create maximum 127 VSwithes for ESX Server.
• Each VSwitch contains maximum 1016 Virtual NIC’s (or) Ports.
• VSwitch is an interface between service console to uplink (physical
switch).
• In VM Network port group, VMs have IP addresses but in Service
Console port group Service Console have IP.

There are three network policies


• Security, Traffic Shaping, NIC Teaming. Applies on Vswitch level &
port group level.
• Security policy contains Promiscuous mode (Reject), Mac address
Changes (Accept) & Forged Transmit (Accept).
• Traffic Shaping policy contains peak bandwidth, average bandwidth &
burst size.
• NIC Teaming contains Load balancing (Port Based, MAC Based, IP
Based), Network Failover detection.
• Virtual LANs (VLANs): allow the creation of multiple logical LANs
within or across physical network segments, it contains 4096 port
groups.
• Distributed Switches: Individual host level Virtual Switches are
abstracted into single large Distributes Switches. It contains
advanced features like Network VMotion, PVLAN’s and Bidirectional
Traffic Shaping. Distributed Switches are Independent of the number
of hosts.
• Standard Switches: Standard Switches are Individually managed &
configured by Hosts and VMs.
VIRTUAL SWITCH CREATION

(1) Login ESX Host by VI Client.


(2) Go to ESX Host Configuration Tab & Select Networking Option in
H/W Pane.
(3) Select Add Networking Option.
(4) It will Generate Network Connection Type Wizard.
(5) Select any One of Virtual Switch i.e.
(a) Service Console
(b) VM Kernel
(c) VM Networking
(6) While creation of virtual Switch we required Network Label ID,
VLAN ID.
(7) Select Create Virtual Switch Option.
(8) Click on Finish.

HOW TO CREATE PORT GROUPS

(1) Login ESX Host by VI Client.


(2) Go to ESX Host Configuration Tab & Select Networking Option in
H/w Pane.
(3) Click on Appropriate VSwitch Properties.
(4) Select VSwitch click on Add.
(5) Select any One of the Connection type.
(a) Service console Port Group
(b) VM Kernel Port Group.
(c) VM Network Port Group.
(6) While creation of Port groups we required Network Lable Id, VLAN
ID & IP Address.
(7) Click on Finish.

VMWARE ESX STORAGE CONCEPTS


• Storage option gives you the flexibility to set up your storage based
on your cost, performance, and manageability requirements
• Shared storage is useful for disaster recovery, high availability and
moving VMs between ESX Servers.
• VMware provide Fibre Channel SAN storage, Iscsi SAN storage,
VMFS data store and NFS data store to ESX Servers

Storage SAN, NAS, ISCSI Differences

INTERF TRAN PROTOC BOOTE BO VMOTI HA VC VMF PERFO


ACE S OL SX OT ON / B S RMANC
VM DR E
S
FC FCHBA LUN FC/SCSI YES YES YES YE YE YES HIGH
SAN S S

ISC IScsiHB LUN IP/SCSI YES YES YES YE YE YES MEDIU


SI A S S M
NAS NIC FOL/ TCP/IP NO YES YES YE YE NTF MEDIU
FIL S S S M
LOC NIC SA/ TCP/IP YES YES NO N YE YES MEDIU
AL SCSI O S M

Fibre Channel SAN:


• A high-speed SCSI transport protocol used for Storage Area
Networking (SAN)
• Multiple switches inter connect “fabric” in a Fibre Channel SAN.
• It Support to Boot ESX.
• It Support to Create a VMFS.
• It hold VMs’ files, ISO images, and templates
• It supports to VMotion, DRS, HA & VCB.
• FC SAN is recognized by the VMkernel during the boot sequence, it
scans 256 LUN’S at Boot time.
• Performance is high.
• It contains LUN addressing & Partition Addressing. i.e.
<”VMHBAID: TARGETID: LUNID”>&<”VMHBAID: TARGETID: LUNID:
PARTITIONID”>

ISCSI SAN:
• It is IP based storage.
• It depends on TCP/IP Protocol
• It support to Boot ESX using H/W Initiator Only.
• Using hardware initiator only
• It supports to create a VMFS.
• it hold VMs’ files, ISO images, and templates
• It supports to VMotion, DRS, HA & VCB.
• Performance is high.
• Between ISCSI & ESX we are using authentication protocol CHAP.
• Both service console and VMkernel need to access the ISCSI storage
• It contains S/W ISCSI & H/W ISCSI.
• S/W ISCSI = NIC + drivers (does not supports to BOOT ESX)
• H/W ISCSI =HBA + Drivers. (Supports for BOOT ESX.)
• To enable S/ W ISCSI
• Enable s/w Iscsi Client
• Create vmkernel Port group
• Enable iscsi storage group (or) Adapter.

Problem: Cannot Access ISCSI Storage


VMKernel port configured
Second Service Console port required
Is CHAP authentication required

VMFS DATASTORE
• Each virtual machine’s files are located in its own subdirectory
• VMFS is a VMware Files System to build esx Cloud Infrastructure
• We can create VMFS and NFS data stores for the ESX Server.
• A VMFS data store is used to hold templates, ISO images and the
files that make up a VM
• The size of a VMFS can be dynamically extended
• VMFS volumes are accessible in the service console
• If the master extent is lost, it could cause data loss on the entire
VMFS

Network Attached Storage(NAS):


• Storage shared over the network at a file system level
• Lower cost, lesser infrastructure investment required than Fibre
Channel
• It does not support to Boot ESX
• It does not supports to virtual center features like VMotion, DRS, HA
& VCB.
• NAS Uses two protocols i.e. NFS (Network File System) & SMB.
• ESX Server supports NFS only

HOW TO ADD STORAGE


• Login into Esx Host by VI Client and go to Host Configuration Tab.
• It contains Two Panes - H/W Pane & S/W Pane.
• Go to H/W Pane, Select Storage option and click on add storage.
• It Generate a Add Storage Wizard.
• Select Storage Type
a. Disk/Lun
b. NFS.
• If we select DISK/LUN option, then choose VMFS data store volumes,
FC SCSI (or) Local SCSI
• If we select NFS OPTION. Then Choose Shared folder Under N/W
Connection.
• If we will select DISK/LUN then select SAN, LUN and Device
Identifiers.
• If we will select NFS then provide server FQDN/IP, Folder name &
Data Store Name.
• Click on Finish.

How to Enable S/W ISCSI? (OR) HOW TO CONFIGURE ISCSI.


• Enable S/W ISCSI Client in ESX Host Configuration Tab.
• Create VMKernal Port Group.
• While Creation of VMKernel Port Group we required Network Label
Id, VLAN Id & IP Address
• Enable ISCSI Storage Group (or) Adapters.
• While Enable ISCSI Storage Adapter we need to Enable CHAP.

Diff between Software ISCSI & Hardware ISCSI?


H/W ISCSI: ISCSI HBA & Drivers. Supports Boot ESX.
S/W ISCSI: NIC + Drivers. Does not Support Boot ESX.

How many ISCSI targets will ESX support?


8 for 3.01, 64 for 3.5 and 4.0
How Many Fiber Channel targets in ESX?
256 (128 on Install)

ISCSI Software initiator port number


3260.
PORT NUMBERS
HTTP-80, HTTPS-443, NTP-123, FTP-21, SSH-22, SNMP-161,162,
SMTP-25
Communication Port (OR) Heartbeat Port (OR) HA Agent (OR) VC Agent-
902.
LICENSE SERVER—27000 and 27010,
TCP/IP—7, UDP-17,
VMOTION-8000, LDAP-389, UPDATE MANAGER—80, 9000-9100
S/W ISCSI Client – 3260,
VMware HA Traffic (or) AAM Agent— 2050-2250, 8042-8045.
Web Service HTTP:8080, Web Service HTTPS: 8443,
NFS—111, SSL—636

WINDOWS PORT NUMBERS:-

KERBEROS—88, DNS—53, LDAP—389, SPOP3—995, POP3—110,


POP2—109, IMAP4—143
FTP—20 and 21, TELNET—23, IP-0, ICMP-1, GGP-3, TCP-6, EGP-8,
NFSD-2049,
PUP—12, UDP—17, RDP—27, RUD—66, DOMAIN (or) DNS (or) Name
Server – 53,
BootPC (or) DHCP Client - 68, BootPS (or) DHCPS - 67, Gopher – 70,
RTelnet -107, Auth -113,
NNTP - 119, NTP - 123, BGP - 179, IRC - 194, IPX - 213, KPASSWD -
464, LOGIN/WHO - 513
CMD - 514, NETWALL - 533, KLOGIN - 543, KSHELL - 544, LDAPS - 636,
DOOM - 666,
KERBEROS ADMIN - 749, KPOP - 1109, PHONE - 1167, MS-SQL-S -
1433, WINS - 1512, MAN - 9535.
ITIL PROCESS

 Information Technology Infrastructure Library.


 It Contains 5 Modules.
(1) Service Strategy.
(2) Service Design
(3) Service Transactions
(4) Service Operations.
(5) Continues Operation Improvements.
 In this my Role is Service Operations i.e. Request Fulfillment.

What Is Your Challenging Troubleshootings Vm / Host / Vc


VmlevelTroubleshootings
Fail To Power On VmCause: Because of insufficient resources and no
space in data store.
• First we need to increase VM resources (or) Data store resources.
• In this situation Verify Data store resources utilization information by
using vdf-h
• Verify service console resource utilization by using df-h commands.
• Then freeze disk by deleting Temp files, unnecessary information and
unused ISO images.
• Then it will fix.

VM Blue Screen Of Death (BSOD) (OR) HUNG Status


First verify whether it is pinging or not. If it pinging then it’s a false alert.
Suppose if it is not pinging then
• Login into VC and verify the status of VM by using VMConsole
Option.
• If there is any Hung Status (or) BSOD then Reset the Virtual
Machine. It will fix the issue.
• Suppose we are unable to fix the Issue, then login into ESX Service
Console and type command #VMware –cmd<Appropriate VM
path> Start/Starthard.
• Then it will fix the Issue.

Virtual Machine Monitor Faults: (VMM)


• We need to verify vmware.log files.
• Check VMware Tools Version
• Suppose VM is in hung status then type #VM-Support. This will
extract all Log Files from VC.
• Finally Guest O.S Problems like Incorrect O.S Patches are Applied.

HOST LEVEL TROUBLESHOOTINGS


Esx Firewall Not Responding
• In this case we can’t ping ESX Host.
• We can’t connect via SSH (or) VI Client
• ESX will not respond in VC Management Server.
Solutionis Reset the Firewall by using the command #esxcfg-firewall –r

VMWARE HOSTD SERVICE IS NOT RESPONDING


• Hostd Management service is the main communication channel
between ESX Host and VMKernel. If Hostd fails, then ESX Host will
disconnect from VC.
• So Verify the Hostd Service by using#service mgmt-VMware status.
• If it is in stop state then restart by using#service mgmt-VMware
start.

HOSTD FAILS TO START


• First we need to check Network File System (NFS) (or) Server
Message Blocks (SMB) mounts on ESX Host.
• Check firewall directory i.e. service.xml.

VM fails to CLONE in VC
• First Login into appropriate HOST Service Console.
• We need to restart VC server service #service vmware-vpxa restart

ESX HOST HALTED (OR) NOT RESPONDING


• In this case increase the Service Console Memory.
• Verify the Shared Storage Connectivity.
• Verify the Disk space in ESX Host.
• Verify the VC Server Versions.
• Verify Environmental Factors like any power outage, Nic unplugged,
Thermal outage and any Attached (or) detached Hardware devices.

ESX HOST GOES DOWN (PSOD)


1. Host failures mainly depend on Hardware & Memory.
2. Like VMKernel& Service Console.
3. PSOD Comes by VMKernel Crash.
1st Scenario:
• First verify whether it is Pinging (or) Not if it is Pinging then it’s False
alert
• If it is not Pinging then Login into ILO (Integrated Light Out) (If it is HP
Box).
• Verify the ESX Server status by using Remote Console Option, if it is
in PSOD (or) Any Hung Status (or) Not Responding.
• Go to Power Management Option Reset the ESX Host then it will fix
the Issue.
nd
2 Scenario:
• For Suppose we are unable to fix the Issue by ILO then call to Data
Center Person and inform him to check the ESX Server Status is any
Poweroutage,NIC unplugged & any Thermal Sensor outage. After
that it will fix the Issue.

3rd Scenario:
RCA:
• After Rebooting the ESX Host it will generate one core dump file
i.e.”VMKernel Dump” under ROOT Directory.
• By using #VMKdump-l <VMKernel Dump>. We can Extract Log File
from Core Dump.
• At the end of this file we will be able to see what had happen to the
server.
• Otherwise contact VMware Support team.
• If we are unable to find RCA here then go to ILO IML (Integrated
Management Logs) here we can find RCA.
• Otherwise got Event Viewer based on Time stamp we can find RCA.
• Otherwise go to HP System Manager & HP System Inside Manager
we can find RCA.
• In Windows, we use third party tool WINDBG UTILITY to extract the
file.

VIRTUAL CENTER ISSUES


VC CAN NOT LOGIN & ADD HOST TO VC FAILS
• Verify VC server agent service VPXA by using
• # service vmware-vpxa status if it is in stop status then start
• #service vmware-vpxa start/restart.
• Check HOSTD service start by using #service mgmt-vmware start.

ADDING HOST TO HA CLUSTER FAILES ACCESS DENIED


• We have to add permissions to HA by using HA Agent service i.e.
AAM.
• By using AAM>addnodeesxservername
• AAM>createuser root esxservername Node PERM_ALL.

VC server fails to start after you replace Default SSL certificate


• To resolve this Issue by changing database password.
• First we need to stop VC server service.
• Change VC directory default location i.e.C:/Windows.
• Run the VPXD.exe –p command. It ask password
• Enter new password then restart VC server service.

Esx Commands:-
Login to the service console [Alt +F1] or putty
# vmware-cmd -l = To view the
VMs List
# vmware-cmd<VM Path> start = To Start the
VM
# vmware-cmd<VM Path> start hard = To Start the
VM
# vmware-cmd<VM Path> stop = To Stop the
VM
# vmware-cmd<VM Path> stop hard = To Stop the
VM
Here, VM Path = /VMFS/Volumes/Datastore name/VM
name/Vmname.vmx

VI Deployment
-How to virtualize the datacenter if you have 400+ Physical servers
-How to Deploy 100+ Physical /Virtual servers
-How to prepare a plan for new projec [50+ servers]
-How to create a two test machines on friday[Weekend]

How to change permit root login no to yes


Login to ESX Service console[Alt+F1]
#vi /etc/ssh/sshd_config [Press Enter]
In the file content go to the second page we can find
"permit root login no"
To Edit Settings:- Press I or Insert Key
Change the permit root login no to yes
TO save Settings :- Press Esc Key and type ":wq" for save and quit
To restart SSH Service: - #service sshd restart
Note:- If there is any network issue needs to restart the network service
#service network restart

Unable to add the ESX host to VC/ ESX host is showing as not
responding-Steps to be followed:-
1.First verify whethere the ESX Host is ping or not.
If it is not pinging need to resolve the network issue.
If it is pinging go to the next step.
2.Check the ESX Host name or IP Address and Login credentials whether
it is correct or not.
3.Verify the ESX Host Agent service status
#service mgmt-vmware status
If it is not running
#service mgmt-vmware restart = To restart the host agent service
4.still problem persists,check for vpxa service status
#service vmware-vpxa status
If it is not running needs to start the service
#service vmware-vpxa start/restart
5.It will resolve the issue.

ESXis not responding & ESX is showing as disconnected-Steps to be


followed
1) Ping the ESX whether it is pinging or not
2) verifyesx Login Credentials
3) Verify Hostd service is running or not by using following command
#service mgmt-vmware status
4) If the service is not running need to restart the host agent service
#service mgmt-vmware restart

Recent Big Issue:


One of ESXi4.1 host has down and become operational again with in
few seconds.

RCA:- Root cause analysis


-At the time of host down VM backups are running that is the reason host
hardware utilization is continuously increases finally it went down.
-When i select the host -->hardware status tab it is poped up an error like
Host monitoring service is not running.
-For this issue we have to restart the VC Server service & VC Webaccess
service
-Even after restarting the vc services it does not work hence i have plan for
migrate the particular host vms to another host then restart host agent
services or host reboot however VMotion got failed for these vms
-Finally i got an approval from my client to power off all the production
vms& esx03
-I have migrated all vms to the another host using cold migration option&
restarted the esxi
-Now new issue is it does allow to login and poped up an error is
"Authentication denied; direct console access has been disabled by
administrator for hostname"
This occurs when total lockdown mode is enabled.
To resolve this: VMware support has suggested reinstalling the ESXi host.
1) Rebuilt the ESX host.
2) Configured Network connections.
3) Enabled NTP Server.

HA Recent Issue
HA is by far the most dependant on DNS, this was fixed by using host files.
Anything that needs to resolve based on a host name will need a host file
(HA, vMotion, adding hosts, etc).
vCenter may need to be adjusted as well. Add the host file that we saved to
this location like we did before by pasting it into the host file.
C:\Windows\System32\Drivers\etc\hosts
That will allow vCenter to resolve the hosts as well. This should take care
of DNS in the environment as long as there is no additional growth. If your
customer needs more assistance with DNS, then that would be up to him to
fix it in his environment. These adjustments should fix the issue with the
hosts and vCenter for you. Although this is a supported setup, it does make
some thing more difficult. I really hope this clearifies some of the issues
that had in the environment. Please let me if it does not. For now, would
you like to close the case?
Looking forward to hearing from you. You can contact me at 1877-
4VMware, opt 4, opt 2, then the SR/Case number. If I am unavailable,
please ask for the next available TSE. Have a great day!

How VMotion works


-Users currently accessing VM A on esx01
-Initiate migration of VM A from esx01 to esx02 -while VM A is up and
running
-Pre-copy memory from esx01 to esx02
-Log ongoing memory changes into a memory bitmap on esx01
-Quiesce virtual machine on esx01
-Copy memory bitmap to esx02
-Copy VM’s remaining memory (as listed in memory bitmap) from esx01
-Start VM A on esx02
-Users now access VM A on esx02
-Delete VM A from esx01

VC is not responding & VC is showing as disconnected ---Steps to be


followed
1)Ping the vc IP whether it is pinging or not
2)verify VC Login Credentials
3)VerifyVPxa service is running or not by using following command
#service vmware-vpxa status or Start->Run-->Services.msc-->verify Virtual
center server is in started state or not
4)If the service is not running need to restart the VC agent service
#service vmware-vpxa restart
Or Start->Run-->Services.msc-->Start the Virtual center server service

How to create a Portgroup:-


Login to the viclient -->select the esx host-->go to the configuration tab--
>under configuration tab -->select networking-->in the right pane select
vswitch properties-->in the vswith properties wizard click on add--> choose
appropriate network connect type and assign neccessary details click on
finish.
Minimum requirement to create a Service console portgroup/vmkernal
port group:-
-Network label name
-VlanID[Optional]
-Ip Settings

Minimum requirements to create a vm network portgroup:-


-Network label name
-VLAN ID [Optional]

BACK UP

Differential:- Backup selected files only if they were created or modified


since the previous backup,but . does not mark them as
backup.
Incremental:-Backup selected files only if they were created or modified
since the previous backup.
**As a VMware Administrator, Need to verify the All Physical and
virtual machines Backup status.

Verifying Backup Reports:-

1)Success:- If the Backup report status is showing as success needs to


send a mail to team...
Verified the backup report and all clients backup is completed
successfully.

2)Failed:- If the backup status is failed for any client need to check the
following options
--First verify the BAclient service is running or not,if it is not
running start the service and
inform to backup team
I have restarted the baclientservice,could you please rerun
the backup.
--Need to verify the server status during backup schedule time.
--Need to check the Backup Adapter is properly defined or not
--If there is any issue with Baclientsoftware,Reinstall it.
--If there is any routing issue ,add the appropriate route using
route add command
start-->run-->cmd -->route add like "Route add 10.232.8.96 mask
255.255.255.224 10.232.185.XX –p"

3) Missed: -- If the backup status is missed for any server need to verify
the following options
---Verify whether the server is existing or not.If it is not in
network since week time,just
update the same info to team/customer
---Contact the backup team wheter the server is added to
proper group or not.
---check for the server naming convention
4)Started:- if the bakcup status is showing as started need to follow the
following steps
---Check the server backup size
---verify is there any scheduled jobs[Any application
maintenance] during backup. If you
find any info update the same status to customer/team
Note:- If you are unable to resolve any backup issue, finally raise a service
request to the backup
team to investigate the issue
UPDATE MANAGER: How to upgrade ESX 3.5 to ESX 4
1) Login to VC 4 and enable VMware update mgr client.
2) Download ESX 4 iso or ESX 4 zip and placed into a common location
3) Create a baseline for ESX Host upgrade
4) Upload esxiso or zip to baseline
5) Go to the Inventory select the appropriate ESX Host
6)scan the inventory,choose the baseline and verify the non compliance
7)Remidate ESX Host
Note:- Baseline = The set of patches
VMware Converter 4.0: Pre Conversion steps for P2V Conversion
Prepare a project plan for the P2V Converter and mention all actionable
items like converter tool
name, how many cpu’s, mem, how much amount of disk and how many
nics?
-Need to choose the which is best candidate- Depends upon the
solarwindsperfmance reports we
need to choose which is under utilized system that is the best candidate to
convert P2V
-Need to uninstall the Antivirus on the Physical server
-Take the benchmarks of the Physical server [System name, IP
Settings, Drive details, Persistant
routes ..etc.,]
-Choose any of the converter tool Ex:-VMware standalone Converter
4.0
-Install the converter on the physical server
-Converter Steps
-Specify Source
-Specify Destination
-View/Edit Options
-Ready to complete
Note:- In view/Edit options need to uncheck the Network "connected at
Power on" option
-It allows to edit the C drive size.

Post Conversion Steps:-


-Power on the vm
-Install VMware Tools
-Assign IP Settings
-Join the system to domain
-Install the Antivirus
-Uninstall the Hardware Agents

Ex:- HP System Manager for HP Box & Dell OPen Manage for Dell Boxes.
etc.,
-Need to verify the VM Services and all are up and running or not
-Verify the All Benchmarks
-Place the VM is in Production

How the Conversion work


-File level Cloning or Block level Cloning
Difference Between ESX &ESXi
http://www.vmware.com/products/vsphere/esxi-and-esx/compare.html

Firmware Updates :-
Firmware is a combination of software and hardware.
Normally, For any Hardware related issues recommend solution is first
verify the firmware version.
If it is running with older version.we needs to update it to lattest version.
Update Procedure:-1.
As per the HP support team, Get the download link and download the
respective firmware ,placed in to the local system. Install it manually.
Procedure:- 2. Once you download the firmware *.iso file, Placed in to
the any utility server and Access the affected system-iLo --->Mount the iso
using administration tab--> Go to the power mgmt option -->reset the
system--->it will start booting from iso --->monitor the firmware upgrade
process from Remote console option-->once the upgradation is completed.
Restart the server and check thelattest firmware version.
Procedure:-3 Download the approriate firmware version and placed into
HP SIM server-->Go to all server section-->select the respective server--
>Go to the deploy-->Deploy Drivers,Firmware and agents--->Install
software and firmware --->Mount the iso-->click on finish--->it will start the
installation process -->needs to monitor from integrated remote console
option.
once the upgradation is completed. Restart the server and check the
lattest firmware version.

HowTo upgrade esx 4.0 to 4.1 command line


Do you want to upgrade your ESX 4.0 to 4.1, but VUM just doesn’t work.
Here is an easy way to fix that…use command line…
- download the upgrade-4.0-4.1.zip file from VMware
- upload it to one of your data store
- ssh into your esx server
- run the upgrade
Esxupdate update –bundle=/vmfs/volumes/XXX/upgrade-4.0-4.1.zip
NOTE: make sure you put your esx into “mantainence mode” first.

ESX Server Building Process:-


Pre-Installation Steps:-
-Verify Hardware Compatibile List
-Configure HP-iLO,DELL-DRAC,IBM-RSA
-Download the ESX iso file
-Get the IP Settings from Networking Team
-Get the LUN Details from Storage Team
-Connect to Server using ilO/DRAC/RSA
-Install VMware ESX server
Post -Installation Steps:-
-Change the Permit root login No to yes
-Enable NTP
-Enable SNMP
-Increase service console memory -- Configure Host file
[/etc/hosts]

Server Health status

-Depends upon the server availability report & Performance report we can
easily verify the server health status.
Windows Alerts &VMware Alerts --- Server UP/Down

ESX server is down/PSOD- Purpule Screen of death/Host is not


responding/hostis disconnected
1) Verify whether the ESX server is pinging or not.
If the ESX server is not pinging [Request timed out]
2) Access the server via aleternate IP Ex:- HP -iLoIP,DEll DRAC
IP,Bladeencloser IP
and go to the remote console verify the server status. If it is showing
as hang state or PSOD or unexepected error. Recomended solution
is need to reset the machine using Power mgmtption
If the ESX Server is UP we need look for Root cause[RCA-Root
Cause Analysis]
3)Suppose Unable to resolve the issue via alternate server access Need to
contact respective Datacenter person and inform the same and
suggest for manual reboot.
If the ESX Server is UP we need look for Root cause[RCA-Root
Cause Analysis]
4) In case still problem persists it may problem with hardware or
environmental issue.
If the problem with hardware need to contact the appropriate vedor
like HP,DELL& IBM.
I f the problem with power outage need to wait until the power is up
and share the update to next level.

RCA-Root cause Analaysis for PSOD :-


-Review the logs from the PSOD dump After the system is rebooted the
core dump is placed in the /root directory
-Use vmkdump –l <core_dump_file>to extract the vmkernellog from the
core dump
-A vmware-log.1 file is extracted from the dump
-Near the end of this file you will be able to see what had happened on the
system

RCA for BSOD:-


-After the system[physical or vitrual] is restarted.It will create a memory
dump file under c:\windows
-The memory dump file will be "memory.dmp"
-To analyze this file we can use "WinDBG utility"-windows debugger utility
-If i am unable to analyze this file i can contact next level or Microsoft
support for more help.

If the ESX Server is Pinging still we are unable to add the host to VC
...it says not responding or disconnected.
-Verify the ESX Server name or IP Address and Login credentials
-Still problem persists Verify the ESX host mgmt service
#service mgmt-vmware status = To verify the ESX host agent service
status
If it is in stopped state need to start the service
#service mgmt-vmware start or restart
-However Unable to add to VC--- Apply the following command
#service vmware-vpxa status = To verify the VC Agent service
status
If it is in stopped state need to start the service.
If there is any network issue need to restart the network service
#service network restart
It will fix the issue.
Still unable to resolve the issue depends upon the error message just
research in kb.vmware.com
we can find the resolution steps and easily fix the issue.

Troubleshooting steps for the Service down Issue:-


•First verify the particular server IP whether it is pinging or not.
Step:1
•Login to the server go the services.msc. (Or )
•Try to view or manage services on a remote server by using the services
snap-in (services.msc)
•Start the affected service.
Step:2
- Go to the services.msc-->selcect affected service-->properties -->general
tab-->check for actual service
path and verify whether the required file exists or not.
-If it is not present copy from the working machine ,start the service.it will
fix the issue.
Step:3
-If we are unable to access server RDP check the server hardware IP List
in Troom[Team Website].
--Find out the alternate server access details like Virtual center IP, ILO IP,
Blade Encloser IP, DRAC IP & ESX IP etc.,.
--Login to appropriate option verify the server status using remote console
option, try to resolve the issue.
--If we are unable to fix from remote console. Go to the power management
option and reset the server. (Please take the necessary approval before
rebooting the server as per the escalation procedure).
--Once the server has rebooted successfully.
--Log in to the server start the affected service.
--It will fix the issue.
--Suppose if we are unable to fix from the iLo/DRAC/Blade encloser call to
the operations helpdesk and inform about the failure. They direct to the
appropriate datacenter person. (Helpdesk contact details are available in T-
room or On-Call Calendar)
--Once the issue has resolved we need to look for the root cause.
RCA-Root Cause Analysis or RFO-Reason for Outage:-
--Log in to server event viewer verify whether it is system error or
application error with the time frame.
--Find the appropriate Application logs ex: - Sql server Error logs.
Please add your explanation if anything I missed.
----------------------------------------------------------------------------------------------------
-----------------------------------
PERFORMANCE ISSUES --------ESX Host CPU/ Mem over 80% ,90%
To Resolve this
1)First verify the all esx virtual machines,is there any backup running on
them.
If yes just update to team -- Currently VMs backup is in progress that is
the reason esx host cpu consumes more than 80%
We will just keep an eye until backup finish.
2)Still problem persists we need to Migrate a vm to another esx using
vmotion technique.
3)Finallyesx host cpu is back on normal state
----------------------------------------------------------------------------------------------------
----------------------------
Physical Server CPU/Mem Over 80% ,90%
1) First login to the particular server go to the taskmanager[Start-->RUN--
>Taskmgr] verify the processes. which process is consumes more than
80%.
2) Depends upon open the process name ,just Google with process name
we can find the resolution steps.
3)Still problem persists kill any unused process.
4)Itwil reset the issue
Note:- If you want to more dig in to the processes we can use
"processexplorer.exe tool"
If it is a vm apply same as above steps and finally while running the vm.it
allows to upgrade the additional CPU/Mem using hot add feature
----------------------------------------------------------------------------------------------------
---------------------------
ESX Server disk [/vmfs] over 80% ,90%
1)Log in to vc/ESX verify all esx VMs ,is there any snapshots were created
or not.If we remove the old snapshots ,issue got reset.
2)Suppose still problem persists we need to migrate running virtualmachine
files from one data store to another datastore using storage vmotion
technique.
3)Finally it will reset.
Physical Server disk over 80%,90%
Microsoft Patches Link:-
http://www.microsoft.com/technet/security/bulletin/ms11-Jul.mspx

Network Latency Issues :-


- First check for any scheduled jobs[AV Scanning,Backups& Application
Maintenance] are running
on the system[Physical,VM& ESX] and same information explain to client.
-Once scheduled jobs are completed verify for network utilization whether it
is back to normal state
or not
- Still Problem persists needs to raise a service request to Networking team
Network Driver Issues:-
-Verify the current driver version-->Start-->RUN-->Devmgmt.msc---
>Network adapter-- >
Properties-->Driver tab-->check the driver details and download the lattest
drivers from
approriate website and placed into particular system.
-Before Installing nicdrivers,check for NIC Teaming is enabled or not
-If yes,Firstdislove or drop the NIC Teaming using HP network
configuration Utility.
-Then proceed with the installation.
-Most of the times,Driver issues resolved by applying the lattest version of
drivers.

Infrastructure Management Team


1.WMG - Workstation Management Group
2.SMG - Server management Group -wintel Team
3.NMG - Network Management Group
4.SOC - Security operation center
5. DB Admins - Database adminstrators
6. Backup Administrators
7. Storage Administrators
8. Software Developers
9. ASG - Application Support Group
10. Assets Team
11. VMware Admins
12. Unix Admins…………. etc.,

Monitoring Tools:
1)WhatsUP Gold v14.4--> System UP/DOWN & Service UP/ Down Status
Monitoring.
2)Solarwinds v9.1 SP5 --> For Performance Monitoring.
3)VC Alarms--->To monitor Virtual Infrastructure [ESX &VMs]
Minimum Requirements to monitor a device[server,Switch,Router
...etc.,]
• System should be online[Pinging]
• Needs to be configure SNMP –To Monitor performance and service
up/down status
• E-mail Notification List---Respective Team Mail ID
• Mail server details----Mail Server Name & Port Number[SMTP-25]
• Alert Severity---like Sev1,2,3,4& Notes

How to configure SNMP using windows 2003


Start --->RUN-->Appwiz.cpl---> In the Add or Remove Programs wizard -
>Click on Add or Remove Components-->
In this wizard select Management & Monitoring Tools --->click on details---
>Select SNMP-->Next --->Finish.

Note:-While Configuring SNMP it requires Windows 2003 server CD or


i386 folder.

To Verifysnmp configuration:-
Log in to the server-->RUN-->Services.msc--->verify the snmp service
status -->it should be in started state and-->snmp service properties --
>security tab-->check for community string [Default = public or any other]--
>add the whats up server ip to snmp security tab to accept the packets.

How to configure SNMP using windows 2008


Login to windows 2008 server-->server manager-->Features--->Right click
select add features--->in the add features wizard --->select snmp services--
->Next --->Finish

How to configure SNMP in ESX :


Login to service console
#Service snmpd stop
To Edit the snmpd.conf file under /etc/snmp/snmpd.conf as below:
#VI /etc/snmp/snmpd.conf
In the file content Needs to be add monitoring server ipEX:-Solarwinds
server IP Address
Restart the snmpd service with the following command:
#Service snmpd start
So that the snmpd daemon starts upon every reboot, run following
command:
#Chkconfigsnmpd on
Run the following command to ensure that the SNMP ports are open before
sending traps to the management station.
#esxcfg-firewall -e snmpd
Server Models
DELL POWEREDGE 1950, 2650, 2850, 2950
DELL POWEREDGE R710, R810
HP proliant DL360G5, DL360G6, DL380G5, DL580G5, DL380 G6
*ProLiant DL (Density Line)
* ProLiant ML (Maximum Expansion)
* ProLiant BL (Blades)
* ProLiant SL (Scalable Line)
IBM X-Series
SAN BOXES
EMC CX300 = EMC Clarion 300
EMC CX400 = EMC Clarion 400

STORAGE : RAID
Simple Volume : Space From Single Hard Disk Forms Simple Volume
Spanned Volume :Free Space Can Span More Than One Hard Disk And
Appears As Single Volume
Stripped Volume: Space Selection From Each Hard Disk Is Identical In
Size
Mirrored Volume: Is An Additional Hard Disk Used For Imaging Primary
Hard Disk To Protect
The Data In The Second Disk When First Disk Fails.
*Redundancy Array Inexpensive Disks[Raid] : It Is A Standard Used For
" Protecting The Data"
Raid0 : Striping Without Parity Offers No Fault Tolerance---[Minimum 2
Disks And Maximum 32]
Raid1: Mirroring Offers Fault Tolerance, Doesn’t Offer Performance,
Requires Down Time.
[Minimum & Maximum 2 Disks]
Raid5: Striping With Parity Offers Highest Fault Tolerance,Offers
Availability And
Performance I.E No Down Time.
[Minimum 3 And Maximum 32]
Raid 10: Mirroring And Striping [Minimum 4 And Maximum 32]

SEP Antivirus Console Monitoring


SEP Mgmt Console 11.0 Installation
Q) SEP Clients 11.0 installtion manually
Ans) Login to the SEP MGMT Console-->Go to admin section-->In the
right pane-->select Install
Packages--->export approrpriate package like either a 32-bit or 64-
bit and placed in the local
drive. This package we can use for manuall installation on any of the
server.
Q) SEP Clients 11.0 installtionRemotely
Start -->Programs-->select Migration and Deployment wizard --
>choose Deployment wizard
--->specify a group name or existing client name-->Click on finish--
>Push Deployment wizard
-->assign computer name or ip -->it will start installation remotely.

Q) Verifying SEP Client updates


Ans-1) Open SEP MGMT Console-->Go to Monitor section-->In the right
pane select "logs" tab -->
use default settings --> click on view log -->It will show all SEP Client
Update information.
Ans-2) Open SEP Mgmt Console-->Go to the Clients section-->In the right
pane we can find all client
update info.
*** Suppose any client is not getting updated needs to be update
remotely.

Q)What is importance of Migration & Deployment Wizard


For Symantec clients Installations and For Symantec Clients
upgrade from 10.1 to 11.0

Q) How to troubleshoot Not updating clients?


Ans-1) Start --> Run-->LUALL(Live Update all)
Ans-2) Login to SEP MGMT Console-->select Clients section-->In the
right pane -->select affected
server-->Right click -->choose Run command on client option-->update
content--->It will
start updating remotely.
Note:- No restart for this update client installation
Q) How to create a scheduled report using SEP MGMT Console?
Open SEP MGMT Console-->select Report Section-->first verify the
quick report option which
one is required and then go to scheduled report option--->click on add and
create as per the
required time frame.

SEP MGMT Console Options:-


HOME --> To Verify SEP Securtiy and Virus defination status
Monitors--> To verify the all clients info and exporting the all clients
data
Report--> To prepare a scheduled report for daily,weekly and monthly.
Policy--> To exclude the any critical services,files..etc.,
Clients---> To verify the all clients info and Managing the clients
Admin---> To create a admin accounts and downloading Installation
packages for both 32 & 64bit.

SEP links:-
http://www.symantec.com/business/products/family.jsp?familyid=end
pointsecurity
http://www.symantec.com/business/security_response/definitions/do
wnload/detail.jsp?gid=savce

How to update the symantec not updated clients

Option :-1Login to symantecmgmt console -->Go to the clients -->In the


right pane click on my company or Default group -->we can find all the
client details--->choose the affected client Right click --->choose "Run
command on client" option-->update content
-It will update the client automatically
-If there is any issue need to restart affect client system -symantec
management client service.
-If there is any issue while restarting symantec management client service
Start-->RUN--->"smc -stop" and again "smc -start".

Option :-2 For manual updatation Start-->Run-->LUALL

Option :-3Using syslink.xml


-Login SEP MGMT Console-->go to the clients section-->In the right pane
select any default group-->Right click choose export communication
settings-->it will export the "syslink.xml file" .
-login to affected server-->stop the symantecmgmt client service [start--
>RUN-->SMC -STOP]-->Go to the "C:\Program Files\Symantec\Symantec
Endpoint Protection" copy the syslink.xml file--->start the symantecmgmt
client service. [Command is SMC -START]
-Finally, it will reporting to the SEP MGMT Console and get the new virus
definations.

How to Build SEP MGMT Console


-First prepare a project plan for this task and mention all actionalitems.EX:-
SEP MGMT console version and client software version details.
-Download the SEP MGMT Software and place it in to a respective server
-Install the SEP MGMT console and apply the licenses.
-Login to the SEP server -->start-->programs-->symantecmgmt console--
>Migration and deployment wizard-->Install symantec client software on all
network servers
-Monitor the all serverssymantec client status from the mgmt console
-If there is any issue we can able to resolve automatically from the console
or manually.
-For best practises Needs to be prepare a scheduled report for all the
servers with latest virus defination status.
Path for Virus definations
C:\Program Files\Common Files\Symantec Shared\VirusDefs
………………………………………………………………………………………
……………………..
VC linked mode
Linked mode allows for multiple vCenter servers to share information
between them. This could be very helpful in managing multiple vCenter
servers at different locations or branch offices. It could also be helpful in
setting up a vCenter server in a DR site to be managed with a single
vSphere client.
Linked mode uses a ADAM (Active Directory Application Mode) database
to replicate it’s configuration between multiple vCenter servers. Some of
the things stored in the ADAM database include:
• Connection information (IP addresses and ports)
• Certificates and thumbprints
• License Information
• User Roles

Firmware Updates :-
Before Updating firmware updates recomend is need to verify the all
network connections.
Suppose if there is any network team first we need to dislove/drop the nic
team/network team then login to ilo for HP Box DRAC for DELL Box.
Steps to be followed:
1) First download the firmware image[HP/DELL] and place it into any of the
location
2)while upgrading firmware recomend is need to login to HP-ilo or DELL-
DRAC and perform the updates
3)For example In iLo go to the adiministration tab mount the firmware
image
4) Reset the physical server using powermgmt option
5) It will automatically boot from the firmware image and monitor the update
process from the remote console option.
6)once update is complete reboot the machine and team the network
connections

How to Dislove/Drop the Network connection using HP network


configuration utility:-
Login to the respective server -->go to the HP Network configuration utility--
>select two nics and click on drop/dislove -->it will be disloving the nework
connections.
If you want to team the nics -->open HP Network configuration utility --
>select two nics and click on team-->It will be team the nework
adaptors/nics.

Website down Issue


Login to the IIS console -->select websites sections--->In the right pane
select the appropriate website -->right click start the site -->it will resolve
the issue.
still problem persists go to the default App Pool --->select the approrite site
pool--->right click restart--->test the site --->it will fix the issue.
suppose still no luck needs to restart the IIS Service & www Service.

WSUS Script:--If any of the server not pointing to wsus console- steps
to be followed
-First verify whether it is ping from WSUS console or not.if it is not pinging
resolve the network issue.
-If it is pinging -->login to the client server-->Start-->Run-->cmd--->type
:gpupdate /force
-It will refresh the group polices.
-If you want to verify the group policy information ,apply the following
command
Start-->RUN-->type: gpresult
reg delete
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v
SusClientId /f
net stop wuauserv
net start wuauserv
wuauclt /resetauthorization /detectnow
wuauclt /reportnow
If any of the server not pointing to wsusconsole : steps to be
followed
-First verify whether it is ping from WSUS console or not.if it is not pinging
resolve the network issue.
-If it is pinging -->login to the client server-->Start-->Run-->cmd--->type
:gpupdate /force
-It will refresh the group polices.
-If you want to verify the group policy information ,apply the following
command
Start-->RUN-->type: gpresult
wuauclt.exe /detectnow
reg delete
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v
SusClientId /f
net stop wuauserv
net start wuauserv
wuauclt /resetauthorization /detectnow
wuauclt /reportnow

FLEXIBLE SINGLE/SCHEMA MASTER OPERATIONS Roles [FSMO]


Domain Naming Master,
Is Responsible For Addition Or Removal Of Domains And Maintaining
Unique Domain
Names, Only One Domain Naming Master In Entire Forest.
Schema Master
Is Responsible For Over All Management, Structure And Design Of
Schema
Only One Schema Master In Entire Forest
Pdc Emulator
Is Responsible For Providing Backward Compatibility For NtBdcs In Mixed
Mode. It
Acts Like A Pdc For Bdcs. It Updates The Password Changes,
Synchronizes Time
Between Dcs, Only One Pdc Emulator Per Domain.

Infrastructure Master
Is Responsible For Updating User And Group Membership Information And
Updating
Global Catalog, Only One Infrastructure Master Per Domain
Rid Master
Relative Identifier Is Responsible For Assigning Unique Ids To The Objects
Created
In The Ad, Only One Rid Master Per Domain

DHCP:---“Dynamic Host Configuration Protocol”:--


Assigns IpsTo The Clients Requested Dynamically
OrAutomatically
PROCESS OF DHCP (DORA):-
Discover : The Client Discovers Dhcp
Offer : The Dhcp Server Offers A Group Of Ips To The Clients To Pick Any
Request : The Client Selects An Ip And Request Dhcp To Confirm It
Acknowledgement : The Dhcp Server Makes A Confirmation By Sending
An Dhcpack To The Client

SCOPE
Range Of Ip Addresses
DHCP Lease period - 8 days

If The Client Is Unable To Contact Dhcp Server, What Happens----


Obtains AnIp From Apipa
APIPA:
Automatic Private Ip Addressing, When A Client Machine Unable To
Contact Dhcp Then Apipa Assigns An Ip To The Client. This Enables The
N/W Availability.

Q) What Are TheIp Ranges Provided By APIPA


A. 169.254.0.0 TO 169.254.255.255

Q)How to resolve if the client is unable to contact DHCP Server


A) Log in to the affected system start-->RUN--->CMD-->ipconfig = To verify
the ip address
-If the ip address is not getting from dhcp server-->Just disable and enable
the network adapter-->it will acquires new ip
-Suppose it is showing an apipa range ip we need to follow the below steps
Step:- 1
Start-->RUN-->cmd--->ipconfig
ipconfig /release = Release the IP address for the specified adapter
ipconfig /renew = Renew the IP address for the specified adapter
Step :-2 If the issue is reoccuring apply below commands
ipconfig /flushdns =Purges the DNS Resolver cache.
ipconfig /registerdns =Refreshes all DHCP leases and re-registers DNS
names
ipconfig /displaydns =Display the contents of the DNS Resolver Cache.
Step :-3 Still problem persists apply below command
netshwinsoc reset = To reset all windows catalog --Require a reboot
What Are The Reserved Ip Addresses/Private Ip Range
Class A 10.0.0.1 TO 10.255.255.254
Class B 172.16.0.1 TO 172.31.255.254
Class C 192.168.0.1 TO 192.168.255.255
Subnet Mask :-
An IP address has two components, the network address and the host
address. A subnet mask separates the IP address into the network and
host addresses.
Default Subnet mask:-
Class A 255.0.0.0
Class B 255.255.0.0
Class C 255.255.255.0
Default Gateway:-
The default gateway often connects a local network to the Internet,
although internal gateways for local networks also exist.
Subnetting:- One network divided into multiple smaller networks

Exchange 2007 introduces five roles to the Exchange organization.


Edge Transport :-
The Edge Transport server is responsible for all mail entering[Inbound] or
leaving[Outbound] the Exchange organization.
Hub Transport :-
The Hub Transport role is responsible for all internal mail flow. This role is
similar to the bridgehead server in an Exchange 2000/2003 organization. In
fact it originally was called the Bridgehead Role until it was changed.
Mailbox Role:-
The Mailbox role holds the Exchange databases within which the user
mailboxes are contained. It is also home to the Public Folder databases if
you enabled Public Folders. (They are not enabled by default in Exchange
2007)
Client Access :-
-The Client Access Role is similar to the role a Front-End server would play
in an Exchange 2000/2003 organization.
-The Client Access server handles all connections whether they come from
an application such as Outlook 2003 or 2007, Outlook Express, or any
other MAPI, POP3 or IMAP4 client
Unified Messaging:-
The Unified Messaging role is responsible for merging your VOIP
infrastructure with your Exchange organization. What does this allow for?
combined voice, fax, and mail in one inbox
access to voice, fax and mail via multiple interfaces
NTERVIEW QUESTIONS
1. How many servers they are managing – Physical and Virtual
2. Team Size and role
3. VMware – version
4. Migration experience
5. AD experience
6. Patch tools used
7. Snap shots – what date gets stored in snap shots
8. HA – High Availability - functions
9. Outage
10. DRS – Distributed Research Scheduling – Load Balancing
11. Microsoft Cluster server
12. Thick Disk – How to convert Normal Disk to Thick Disk
13. Balooning
14. Templating
15. Cloning
16. GUID in Windows OS
17. Syscript
18. Service Console
19. V Center
20. Connectivity in VMware – SAN Environment
21. VLAN tagging
22. Core Services of Virtual Center
23. Basic files required for a VMware machine
24. Memory dumps
25. How to set up a Virtual machine – Memory requirements

FT -195 Page

How an Upgrade from windows 2003 to Windows Server 2008

An upgrade to Windows Server 2008 can only be performed by launching the installation
process from within the currently installed operating system. It is not possible to perform an upgrade by
booting from the installation media

Require downtime for this task


BIRLASOFT

1. Explain how vmotion works?


Login to VC -->Select the vm-->Right click select migrate -->choose Change host-->It will create a pre-
copy of vm [Memory bit map in source esx] and it will sends to the destination host where the resources
are available.
once memory bit map reach the destination, it will power on from the destination host and deleted
from the source host.

2. What will happen if no space on datastore few vms are running?


Login into VCenter select the appropriate datastore -->Right click Browse datastore-->verify that how
many vms are running and have a look for any old snapshots were exists on that datastore.
If there is any old snapshots and remove from vms using snapshot manager.
Still problem persists move running vm files to another datastore using svmotion technique.
If there is no sufficient space on additional datastores in that scenario create a new lun/datastore move
the vm files using SVMotion feature.
it will fix the issue.

3. VM performance is bad, how to investigate?

4. VI client cant connect to virtual center, what to check?


Verify the VC Server service

5. User requested more space on the VM virtual disk, how you manage?

6. What is vmfs?
It is a clustered filesystem.

7. You dont have virtual center and you need to copy a running VM, what to do?

8. What are possible P2V methods?


1)VMware converter
2)VMwarevc consolidation
3)P2v Assistant
4)Any third party tools like MS p2v,Nover Plate spin migrate

VMware Converter 4.0


MS P2V
Plate spin migrate
Vmware consolidation option
ESX host import machine
VMware P2V Assistant
9. Explain backup solution for VMs running on esx
Currently in my environment we are using IBM TSM-Tivoli storage manger and we have to install BA
Client software on all the clients to take the backup of VMs

10. Why esxi not esx?


-More Secure os compare to esx
-Lower Maintenance
-Best performance
-32 mb foot print size
---------------------------------------------------------------

HP COMPANY

VMM

VMM (virtual machine monitor)

Software that is responsible for virtualizing the CPUs. One VMM runs in kernel space for each running
virtual machine.
HA,DRS,VMOTION,STORAGE VMOTION,RESOURCE POOL,SHARES,ALL CONDITIONS
VC AGENT,HOW IT WILL WORK
HOST CONCEPTS
TOP&ESX TOP
-the top command is a system monitor tool that produces a frequently-updated list of processes.
-esx top = Display esx server resource utilization

STORAGE CMDS
NETWORKING CMDS
HOST ERROR LOGS
/var/log/vmware/hostd.log
/var/log/messages
/var/log/vmkernel
/var/log/vmksummary.txt = Availability report for esx
/var/log/vmkwarning
/var/log/vmware/vpx/vpxa.log = Virtual center Agent log

NORMAL LOGS
VMKERNEL
HOST INSTALLATION ERRORS
-Hardware compatability issues.

ERROR SCENARIOS
-No network supported
CONNECTIONS BETWEEN ESX & VC
STORAGE CONCEPTS &DIFFERENCES ,ALL INFO..NAS,SAN
TKTING TOOL
MONITORING TOOLS
SECURITY HARDENING
CHANGE MGMT,SERVICE REQUEST
ITIL&VCP INFO
VMKNOD
CLUSTERING,HOW YOU ADD HOSTS
P2V TOOLS,AND 3-PARTY TOOLS
HOW TO DO THE CPU UTILIZATIONS AND DISK UTILIZATIONS
PORTNO'S,LICENCE SERVER 27000,27010;VC HEARTBEAT 902
SNMP
VM'S STORAGE LOCATION
IPTABLES
WATCHDOG
------------------------------------------------------------------------------------------------------------------------------------------

IBM Interview

In vc how many hosts,vm's----?


-25 esx hosts 200+ vm's

vswitch -ports,portgroups
Service console portgrup
VMkernalPortgroup
VMnetworkportgroup
HA port no?
8042-8045,2050 to 2250

vmkernal
In ESX/ESXi, a high-performance operating system that occupies the virtualization layer and manages
most of the physical resources on the hardware, including memory, physical processors, storage, and
networking controllers.

is HA work with outVMkernal ?


NO

Datastore --how u add,how to extent ?

how to extend vmdk size and how to extend c: and d: for a vm ?

Q)complete vm corrupted
-If there is any snapshot -->we can revert to previous state of vm
-If there is any backup of vm-->we can restore
-suppose there is no backup and no snapshot...we need to create a new vm with same configurations

Q)evc cluster-- Enhanced VMotion compatibility


VMware Enhanced VMotion Compatibility (EVC)—available in VMware Infrastructure 3 beginning with
version 3.5 Update 2—facilitates VMotion between different CPU generations, taking advantage of Intel
Flex Migration and AMD-V Extended Migration technologies.

EVC does not allow for migration with VMotion between Intel and AMD processors.

http://www.vmguru.nl/wordpress/2009/06/vmware-evc-cluster-what-is-that/

Q)cpucompatability

Q)storagevmotion

Q)vmotion failures

Q)host logs

Q)host upgrade from 3 to 4

Q)host updates
Download Patches
Create/attach baselines
SCAN Inventory
Review Complience
Remediate Hosts or VMs

Q)vm backups

Q)p2v,v2v

Q)shrink is available or not for vmdk file


Yes,It's possible
Q)how many luns for esx host
256
Q)how to extent datastore

Q)update manager

Q)snapshot

Q)snapshot files and uses

------------------------------------------------------------------------------------------------------------------------------------------

COGNIZENT
windows
RAID 0,5,6
RAID
how to configure RAID
site& site links
AD -after deleting OU how to recover
DHCP
DNS-records
FSMO ROles
PDC emulator
Infrastructure master
Global catalog
how to install Firmware
Ports:SMTP,DNS,DHCP,UPDATE MGR,ISCSI,

VMware
updatemgr port no
how to upgrade HOST
How to upgrade VM
How to upgrade VM with out using update manger
Snapshot...sizes ....SNAP SHOT MGR
Deploy the vm..from ISO & Template
HA &drs
P2V
ESX DOWN
VM DOWN
VM Notresponding unable to reboot
Hostd is not running how to fix

Exchange
what is system attendent
cluster ....how to configure
vmware with MSCS
------------------------------------------------------------------------------------------------------------------------------

IBM -Chennai
Update Manager
ESX Pre & Post Installation
P2v-Pre,Post steps
Difference Betwee VI3 &vSphere 4
PSOD,BSOD,
Template
Clone
Snapshot

windows
FSMO
AD,dns& DHCP
-----------------------------------------------------------------------------------------------------------------------------------------

SIFI TECHNOLOGIES
SIFI Technologies
personal details
salary
d0b
relocation
---------

HA
DRS
STORAGEVMOTION
Difference between vmfs3 and vmfs3.33
Max vm ram
max vm swap
maxluns per esx 128,
maxcpus per vm--4,8
MAX vmfs SIZE
EVC Cluster
CPU MASK
share
limit
Reservation
Portgroups
how many max extent per esx
Difference b/w vi client and vc
esx 4 license
DRS
-----------------------------------------------------------------------------------------------------------------------------------------

UCB- Drugs company...L3 Interview


Memory Dump
Page file
min requirements for Memory Dump
performance Issues
-CPU
-Mem
Bottle neck
Capacity Management
Tools---for issues
-Perfmon
-core debugeg
-Processor dump
-Drwatson Logs
what is Memory Leak Problem
Process explorer
IBM

1. Have u done p2v Migrations? How many times?


2. Physical Server Model, Ram, CPU Cores...
3. How to Extend C drive size, Step by step/Command Line?
4. P2v Process
5. After adding Drive it will automatically shows???
6. How many clusters u have?
7. What is snap shot, can we move snap shot from one host to another? How?
8. Clone, Template
9. SV Motion, why we need sv motion?
10. RDM, Adding RDM Explain
11. Import & Export of VM's
12. Alarms
13. Resource pool
14. Esx Server Crash, How you respond?
15. If vm performance bad then wt u will do?
16. AD, installing AD, requirements
17. FSMO Roles, if PDC goes down Impact?
18. How to transfer roles? Seizing
19. TCP IP subnet/ Default Gate way
20. Which service you will stop in p2v Process?
21. VCB
22. v Motion Back ground process
23. Can you Add Ram to vm while power on?
------------------------------------------------------------------------------------------------------------------------------------------

Emphasis-

1. Can u tell me u r professional experience?


2. How many Esx servers u r maintains?
3. To install an Esx host wt are Minimum requirements 4. VC fails wt will happen?
5. HA, DRS, VCB
6. Esxi&Esx Differences
7. How can u say Esxi safer than Esx?
8. Vmotion uses which Network port group?
9. What kernel using in Esx ?
10. ITIL Process
11. V motion pre requisites & Post Requisites 12. HA requirements 13. Resource pools 14. Have u done
Esx update 3.5 to 4.0 explain Step by step 15. If VC Fails Wt will happen to VM's?
16. How can u update VC 2.5 to 4.0?
17. Ticketing Tools
18. If u want to migrate host what process u will follow?
19. Why u want to change present job?
20. If we don’t give a VM tasks then wt you will do?
21. Have you worked on Citrix/VMware Lab Manager/Thin App???
22. What are the new features (AD) in 2008 server?
23. Can windows machine understand VMFS file?
24. What are the main files in a VM?
25. Wt is SV Motion?
26. What are the differences between web Access & VC 27. Affinity, Infinity Rules 28. How you will
restart Web Access Service?
29. VM Patches
30. R u working in house project? r u working in Production environment?
31. What is the function of Service Console?
32. Have you worked on Windows Cluster?
---------------------------------------------------------------------------------------------------------------------------------------

HCL
1. How Many Max Host Failures in a cluster?
2. How do know where vm migrating if Cluster fail?
3. How many iScsi targets in a cluster?
4. How VMotion works? backend process
5. P2V Migration
6. FSMO Roles, IF PDC EMULAT Fails?
7. Port Groups
8. If mac address change is in Reject wt happen?
9. Linux Console Version
10. RDM
11. Datastores
12. How can you configure diffrent types of datastores on ESX 2.5
13. How can u deploy vmsFrom template?
14. Snap shot
15. What is the most difficult VMWare related issue you ever faced?
16. VCB
------------------------------------------------------------------------------------------------------------------------------------------

PATNI
1. V - Motion post/pre requirements, back end process 2. How can u create HA? DRS 3. What is the
importance of storeage in vmotion4. How can u add storage from commandline?
5. If ESX Crashed what u will tell to client? Manager?
6. What resorces can u add from esxcfg-vswif-l?
7. How can u change IP address of service console?
8. NIC Teaming?
9. If Physical NIC fails? then
10. Windows Clustering
11. DHCP? how to configure DHCP in Linux?
12. DNS ?
13. TCP/IP?
14. Importing & Exporting VMS how can u do this?
15. Port Groups
16. NTP Client Configuration
17. When u will put a host in to maintanence?
18. ITIL Process Change request?
19. Upgrade process? inurenvr
20. Diffrence between template & Clone
------------------------------------------------------------------------------------------------------------------------------------------
Sify Technologies

Family backround
Hobbies
HA
DRS
FT
EVC
Service console PG Down
Limitions
Supported application
Virtualization
------------------------------------------------------------------------------------------------------------------------------------------
CSS- Windows
Difference between windows 2000 & 2003
IIS -Browse..websitedown,home page content
DNS -Zones-F/W ,Reverse look up zones
RP,ARP
DHCP Reservation
Monitoring tools
Whatsup
Solarwinds
Ticketing tools
Server down
BSOD
Escalation Process
Team size --8
Service Down
IIS service name
Performance issue
cpu over 100% how to resolve
ilo
drac
unable to connect RDP
Request time out
Destination host unreachable
Destination Net unreachable
HP SIM
how to access servers console...Vision APP
Patching---wsus 3.0
how many servers 1100+
what is cluster load balancing
what type of servers r u using in your environment

Potrebbero piacerti anche