3/9/13

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

search this site...

Home Cloud Cool Tools HomeLab VMUG About me VMware vSphere Best Practices

Networking configuration for ESX or ESXi Part 3

Posted by Artur on May 29, 2012 in ESX, Featured, vSphere, vSphere 4, vSphere 5 | 19 comments Today, third part, this time ESX(i) host has 10 pNICs (1Gbps) on Standard Switches (vSS)
Scenario #1 10 NICs (1Gbps 2 x quad port adapters and 2 on-board ports) standard Switch for each type of traffic

In scenario I have to design network for 5 different type of traffic. Each of the traffic has different vLAN ID which will help to utilize all NICs for more than one traffic, optimize pNIC utilization and have network secured. 1. 2. 3. 4. 5. mgmt VLANID 10 vMotion vLANID 20 VM network vLANID 30 VM Backup vLANID 40 DMZ vLANID 50

When you dont have Enterprise Plus vSphere license the only way to configure virtual networking is vSS. In a diagram below, mgmt (Service Console or vmk MORE IN ESX, FEATURED, VSPHERE, VSPHERE 4, VSPHERE 5 (82 OF 146 port) and vMotion were placed on common vSwitch0 with active passive approach (in vSphereARTICLES) can use only on vmnic), Active and Stand by state 4 vMotion vSphere 10 HA DRS deepdive both network is set in a portgroups. On physical ports, where both pNICs are connected two vLANs must be trunked (vLAN4.1 and 20) cause we needfor FREE available on each port, such as in case of failover traffic from both networks will carry over one port.last day

Make sure that connection between physical switches are configured to carry all VMware specific traffic.

Other networks, have their own dedicated vSwitches, each vSwitch has at least 2 NICs connected to two physical switches and all vmnics are in Active state (see table below for details). Below configuration follows virtual networking best practices in terms of: hardware redundancy 2 physical switches, at least two pNIC per vSwitch, failover each virtual network has at least two vmnics available security separate vLAN for each traffic (e.g vMotion is not encrypted), vSwitch security options set to Reject capacity each network has preserve bandwidth capacity (sending traffic over separate physical NIC)

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

1/9

3/9/13

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

ESX ESXi networking configuration for 10 nics vSwitch settings (applicable for all vSwitches) Promiscuous mode Reject MAC address changes Reject Forget Transmits Reject Load balancing = route based on the originating virtual port ID (default) Network failover detection link status only Notify switches Yes Failback No

vmnic location
vmnic0 on board vmnic1 on board vmnic2 quad NIC 1 vmnic3 quad NIC 1 vmnic4 quad NIC 1 vmnic5 quad NIC 1 vmnic6 quad NIC 2 vmnic7 quad NIC 2 vmnic8 quad NIC 2 vmnic9 quad NIC 2

vSwitch portgroup
vswitch3 vswitch3 backup VM backup VM

state

vLANID pSwitch
30 30 10, 20 40 50 50 10, 20 40 50 50 Switch1 Switch2 Switch1 Switch1 Switch1 Switch1 Switch2 Switch2 Switch2 Switch2

active active active in mgmt vSwtich0 mgmt/vMotion passive in vMotion vSwitch1 DMZ active vswitch2 VM network active vswitch2 VM network active active in vMotion vSwtich0 mgmt/vMotion passive in mgmt vSwitch1 DMZ active vswitch2 VM network active vswitch2 VM network active

If you have questions regarding particular case scenario, put question in comments and I will be glad to help you Next post, further this week, will describes scenario with 10 pNIC but using vSS together with vDS (mixed virtual networking configuration approach)

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

2/9

3/9/13

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

UPDATE:
Network configuration 10 x 1Gbps for vSphere 5.1

10x1Gbps vSphere 5.1 vDS Above is my recommended network configuration for vSphere 5.1 with Enterprise Plus license. As you know one of the cooles new features in vSphere 5.1 is backup possibility of the Virtual Distributed Switches. In case you lost vCenter Database and there is no way to restore it you can easily restore vDS config into new DB awesome. No risk of loosing network after vCenter DB lost and all network types including mgmt vMotion can run on single Virtual Distributed Switch. All vLAN has to be trunk on all physical switch ports.
vSphere network 10x1Gbps : vDS
vDS vDS1 vDS1 vDS1 vDS1 vDS1 teaming options LBT LBT LBT LBT LBT portgroup name mgmt vMotion VMnetwork storage FT active dvuplink ALL ALL ALL ALL ALL standby dvuplink none none none none none

vDS

> <

Network configuration 10 x 1Gbps for vSphere 5.x and vSphere 4.x

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

3/9

3/9/13

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

10x1Gbps vSphere 5.x and vSphere 4.X mixed vSS and vDS My recommended network configuration for vSphere 5.X and vSphere 4.X with Enterprise Plus license. In above config vMotion and mgmt run on Virtual Standard Switch and Active/Passive vmnic configuration, where Storage, VM and FT traffic utilize Virtual Distributed Switch. The reason of heaving mgmt traffic on vSS is, in case of vCenter database lost you wont loose possibility to change ESXi/ESX host networking (N/A on vSphere 5.1 and above).
vSphere network 10x1Gbps : Mixed vSS and vDS
active dvuplink or vmnic vmnic0 vmnic1 vmnic2-vmnic9 vmnic2-vmnic9 vmnic2-vmnic9 standby dvuplink or vmnic vmnic1 vmnic0 none none none

vDS\vSwitch vSwitch0 vSwitch0 vDS1 vDS1 vDS1

teaming options Explict failover Explict failover LBT LBT LBT

portgroup name mgmt vMotion VMnetwork storage FT

NIOC shares

Mixed vSS and vDS

> <

See links below for different networking configuration ESX and ESXi networking configuration for 4 NICs on standard and distributed switches ESX and ESXi networking configuration for 6 NICs on standard and distributed switches ESX and ESXi networking configuration for 10 NICs on standard and distibuted switches ESX and ESXi networking configuration for 4 x10 Gbps NICs on standard and distributed switches ESX and ESXi networking configuration for 2 x 10 Gbps NICs on standard and distributed switches

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

4/9

3/9/13
0 Like

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization
3 0 7 0 0

18 comments
Leave a message...

Discussion

Com m unity

Share

C h u ck
0

2 months ago Reply Share

Lets say in the VM LAN I had 3 port groups would I want to use the Active/Standby approach or just leave all vmnics as active??

Artu r Krzyw d zi n s ki M o d

Chuck 2 months ago

if you have Enterprise Plus license - keep Active/Active with vDS and NIOC but if you don't have it then depends on priority and bandwidth requirements for each PG I would use Active\Standby to prioritize it, if all traffics have same priority stay with Active/Active.
0

Reply

Share

H a rry
0

2 months ago Reply Share

How will be network configuration with 10 NICs (1Gbps) with Enterprise Plus license

Artu r Krzyw d zi n s ki M o d
0

Harry 2 months ago

I will post 10 NIC and E+ configuration by the end of the day tomorrow, stay tuned
Reply

Share

H a rry

Artur Krzyw dzinski 2 months ago

Hi, You still not posted the configuration.

0

Reply

Share

Artu r Krzyw d zi n s ki M o d

Harry 2 months ago

Post updated
0

Reply

Share

Be n

3 months ago

Hi, I am wondering why for vSwitch0 and vSwitch1, the vmnic(s) are not being used in a running number way? As in, vSwitch0 - vmnic2 and vmnic 3, while vSwitch1 - vmnic6 and vmnic7. Could you please enlight me on this part? Thank you!
0

Reply

Share

Artu r Krzyw d zi n s ki M o d

Ben 3 months ago

Hi Ben, it is due to hardware redundancy: on board NIC's - vmnic0, vmnic1 first quad port - vmnic2, vmnic3, vmnic4, vmnic5 second q port - vmnic6, vmnic7, vmnic8, vmnic9 If you have any additional question feel free to ask
0

Reply

Share

Ke n

6 months ago

Thank you for this informative post. Just one quick question, I thought we can not have two active connection to to pswitch from one vswitch, like the one on vm backup. I tried this without luck. Can you explain in couple of sentences please. Thank you in advance
0

Reply

Share

Go p i n a th

9 months ago

One more doubt,

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/ 5/9

3/9/13

One more doubt,

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

With same scenario of above, if there is 2 Top of rack switches connected between ESXi hosts and core switch, I will connect the esx hosts pnics to the top of rack switches and the top of rack switches to the core switch in mesh topology. 1- In this case, i believe there wont be any issue, just like the previous post ? 2- Do I need to enable the STP and portfast in all switches (top of rack and core switch)? thanks Gopi
0

Reply

Share

Go p i n a th

9 months ago

Great Help !!! Thanks for your timely help. I am designing Vsphere 5, with HP 3PAR and HP C7000 blade center for a bank. The network consultant told there will be a packet drop in this design, so i confused. Thanks for your advice and quick response.
0

Reply

Share

Artu r Krzyw d zi n s ki M o d

Gopinath 9 months ago

wait wait :-) you haven't mention that this is for blades :-) do you have Virtual connect or pass-through adapters ? 10Gb or 1Gb NIC's ?
0

Reply

Share

Go p i n a th

Artur Krzyw dzinski 8 months ago

With same scenario of first post, assume if there is 2 Top of rack switches connected between ESXi hosts and core switch, I will connect the esx hosts pnics to the top of rack switches/HP virtual connect and the top of rack switches/HP virtual connect to the core switch in mesh topology. The uplinks from the VC to the Top of rack and the uplinks from top of rack to core switch are configured as ether channel trunks. 1 - First case, The esx host (blade) is connected to the > HP Virtual connect > then top of rack > then to the core switch a- In this case, i believe there wont be any issue, just like the very first post ? b- Do I need to enable the STP and portfast in all switches (top of rack, VC and core switch)? 2 - In the second case, the normal one - ESX > connected to top of rack > then connected to the core switch. a- In this case, i believe there wont be any issue, just like the very first post ? b- Do I need to enable the STP and portfast in all switches (top of rack, and core switch)? so please let me know the above and what are the other things to consider also? Thanks Gopi
0

Reply

Share

Go p i n a th

9 months ago

hello I have 2 pswitches Cisco 6500 series as core switches, and i have vsphere 5 with enterprise plus license. I have one esxi host with 4 PNICS, the 2 core switches are interconnected via etherchannel trunk and we are not using stacking and stack cable. Scenario-1 in the esxi host, i have created 1 vswitch and 4 pnics are attached to it. 2 pnics are connected to the pswitch1 and other 2 pnics are connected to the pswitch2. the vswitch teaming policy is selected as (Load balancing = route based on the originating virtual port ID (default)). 5 virtual machines are connected to one VM port group,and running inside the vswitch 1 - will this work ? 2 - will i get the pswitch redundancy? 3 - any duplicate mac address issue occurs inside the 2 core switch? 4 - any packet drops occur in the event of one physical switch failure? scenario-2 the same setup as above, now i need to use a distributed switch with LBT teaming policy. vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/
6/9

3/9/13

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization the same setup as above, now i need to use a distributed switch with LBT teaming policy.

1 - will this work ? 2 - will i get the pswitch redundancy? 3 - any duplicate mac address issue occurs inside the 2 core switch? 4 - any packet drops occur in the event of one physical switch failure? Please help me, i am really confused with this.
0

Reply

Share

Artu r Krzyw d zi n s ki M o d

Gopinath 9 months ago

Hi, thanks for comment Both scenarios will work without any problems, you will have full redundancy, I suggest to use LBT, is really cool feature and works fantastic. Answers: 1 - yes 2 - yes 3 - no 4 - no Cheers Artur
0

Reply

Share

Sta n J

9 months ago

Hey Artur. I have also one proposal. As you also marked the topic as vSphere 5 related (upgrade expected) I would recommend you to move also one NIC port from the VM network to the Mgmt/vMotion group. This decision ofc depends on number of expected VMs (related to expected nr of vmotion migrations). You will have in this case environment more prepared for vSphere 5 from vmotion perspective and 3 NICs should be standardly enough for Prod traffic. Hi Sander, I'm also not familiar with NEN security certification. Can you briefly describe it or dirrect us to some documentation? Also if I understood properly what you are proposing require two NICs (to have redundancy).
0

Reply

Share

s a n d e rd a e m s

10 months ago

What about the "DMZ" network, if you need a fully NEN certified DMZ network? In that situation you need a separate PCI NIC adapter (separate bus) to split the network traffic, in case you combine 1 dual/quart port adapter with LAN/DMZ connections it's "possible" to sniff the traffic/packets. To configure this redundant you need to add a second physical NIC adapter in the host to connect the physical DMZ network switches. If you don't need a NEN certified DMZ and you mean "DMZ" as a different subnet with VLAN ID.. why don't you add the two DMZ network adapters to the LAN vSwitch for more bandwidth/redundancy and trunk the VLAN ID's together?
0

Reply

Share

Artu r Krzyw d zi n s ki M o d

sanderdaems 10 months ago

I'm not security guru yet, unfortunately, I don't know what NEN is. But is always good to know how to improve design, if I would place DMZ (to be NEN certified) on to on board NIC - will my design fill out NEN requirements ?
0

Reply

Share

ALSO ON VMWAREMINE BLOG

What's this?

Spotkanie VMUG Polska: Zarejstruj si ju dzi!

2 comments 12 days ago

Cloud solution based on Nexenta with KVM and CloudStack

9 comments 22 days ago

Artur Krzywdzinski no, calkiem fajnie sie zapowiada, trzeba z Maciejem zagadac zeby organizowal w czasie jak bedziemy w

Marek Lubinski well iops differ, but we do lots of them :) and our iops are quite big in size, therefore we max out

C o m m e n t fe e d

Su b s cri b e vi a e m a i l

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

7/9

3/9/13

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

Follow Me !!!!

Sponsors

social networks

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

8/9

3/9/13
Find us on Facebook

Networking configuration for vSphere 4 ESX or vSphere 5 ESXi | VMwaremine - Mine of knowledge about virtualization

vmwaremine.com
Like 49 people like vmwaremine.com.

vmwaremine.com on Facebook

Tweets
Artur Krzywdzinski @artur_ka

Follow 16h

Cool Tools RVtools 3.5 available for download shar.es/enBhT

Follow

Artur Krzywdzinski @artur_ka VMware Workstation 9.0.2 now available

22h

+21

Tweet to @artur_ka

Meta
Register Log in Entries RSS Comments RSS WordPress.org

Powered by WordPress | Designed by Elegant Themes Follow

Follow VMwaremine Mine of knowledge about virtualization

Get every new post on this blog delivered to your Inbox. Join other followers:
Enter email address Sign me up!

vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/

9/9