The physical network design decisions govern the physical layout and use of VLANs. They also include decisions on jumbo frames and on some other network-related requirements such as DNS and NTP.

The design uses 4 spine switches with 40 GbE ports. As a result, each leaf switch must have 4 uplink ports capable of 40 GbE. 

The resulting environment supports fault tolerance and compensates for oversubscription, as follows.

Fault Tolerance

In case of a switch failure or scheduled maintenance, switch fabric capacity reduction is 25% with four spine switches.

Oversubscription

Oversubscription can occur within a leaf switch. To compute the oversubscription for a leaf switch, use this formula. 

Total bandwidth available to all connected servers / aggregate amount of uplink bandwidth

The compute rack and the management/edge rack have 19 ESXi hosts. Each ESXi host has one 10 GbE port connected to each ToR switch, creating up to 190 Gbps of bandwidth. With four 40 GbE uplinks to the spine, you can compute oversubscription as follows (see Oversubscription in the Leaf Switches).

190 Gbps (total bandwidth)  / 160 Gbps (uplink bandwidth) =  1.2:1
Oversubscription in the Leaf Switches
The environment compensates for oversubscription

Routing protocols

Base the selection of the external routing protocol on your current implementation or on available expertise among the IT staff. Take performance requirements into consideration. Possible options are OSPF, BGP and IS-IS.

DHCP proxy

The DHCP proxy must point to a DHCP server by way of its IPv4 address. See the Planning and Preparation documentation for details on the DHCP server. 

Physical Network Design Decisions

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-PHY-NET-001

Racks are connected using a leaf-and-spine topology and Layer 3 connectivity.

A Layer 3 leaf-and-spine architecture supports scale out while maintaining failure isolation.

Layer 2 traffic is reduced to within the pod.

SDDC-PHY-NET-002

Only the management and shared edge and compute hosts have physical access to the external network by way of VLANs.

Aggregating physical cabling and network services to the management and shared edge and compute rack reduces costs.

Workloads in compute pods located in compute racks have to use network virtualization (NSX for vSphere) for external network connectivity.

SDDC-PHY-NET-003

Each rack uses two ToR switches. These switches provide connectivity across two 10 GbE links to each server.

This design uses two 10 GbE links to provide redundancy and reduce overall design complexity.

Requires two ToR switches per rack which can increase costs.

SDDC-PHY-NET-004

Use VLANs to segment physical network functions.

Allow for Physical network connectivity without requiring large number of NICs.

Segregation is needed for the different network functions that are required in the SDDC. This segregation allows for differentiated services and prioritization of traffic as needed.

Uniform configuration and presentation is required on all the trunks made available to the ESXi hosts.

Additional design decisions deal with static IP addresses, DNS records, and the required NTP time source.

Additional Network Design Decisions

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-PHY-NET-005

Assign Static IP addresses to all management nodes of the SDDC infrastructure.

Configuration of static IP addresses avoid connection outages due to DHCP availability or misconfiguration.

Accurate IP address management must be in place.

SDDC-PHY-NET-006

Create DNS records for all management nodes to enable forward, reverse, short and FQDN resolution.

Ensures consistent resolution of management nodes using both IP address (reverse lookup) and name resolution.

None

SDDC-PHY-NET-007

Use an NTP time source for all management nodes.

Critical to maintain accurate and synchronized time between management nodes.

None

IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-frame payload from 1500 bytes to the jumbo frame setting increases the efficiency of data transfer. Jumbo frames must be configured end-to-end, which is easily accomplished in a LAN. When you enable jumbo frames on an ESXi host, you have to select an MTU that matches the MTU of the physical switch ports.

The workload determines whether it makes sense to configure jumbo frames on a virtual machine. If the workload consistently transfers large amounts of network data, configure jumbo frames if possible. In that case, the virtual machine operating systems and the virtual machine NICs must also support jumbo frames.

Using jumbo frames also improves performance of vSphere vMotion.

Note

VXLANs need an MTU value of at least 1600 bytes on the switches and routers that carry the transport zone traffic.

Jumbo Frames Design Decisions

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-PHY-NET-008

Configure the MTU size to 9000 bytes (Jumbo Frames) on the portgroups that support the following traffic types.  

NFS

vSAN

vMotion

VXLAN

vSphere Replication

Setting the MTU to 9000 bytes (Jumbo Frames) improves traffic throughput.

In order to support VXLAN the MTU setting must be increased to a minimum of 1600 bytes, setting this portgroup also to 9000 bytes has no affect on VXLAN but ensures consistency across portgroups that are adjusted from the default MTU size.

When adjusting the MTU packet size, the entire network path (VMkernel port, distributed switch, physical switches and routers) must also be configured to support the same MTU packet size.