When performing network configuration, you have to consider the traffic and decide how to isolate vSAN traffic.

Consider how much replication and communication traffic is running between hosts. With VMware vSAN, the amount of traffic depends on the number of VMs that are running in the cluster, and on how write-intensive the I/O is for the applications running in the VMs. 

Isolate vSAN traffic on its own Layer 2 network segment. You can do this with dedicated switches or ports, or by using a VLAN. 

The vSAN VMkernel port group is created as part of cluster creation. Configure this port group on all hosts in a cluster, even for hosts that are not contributing storage resources to the cluster. 

The following diagram illustrates the logical design of the network.

VMware vSAN Conceptual Network
The vSAN traffic flows in its own Layer 2 network segment. You can allocate this segement this by using dedicated switches or ports, or by using a VLAN.

VMware recommends that solutions use a 10 Gb Ethernet connection for use with vSAN to ensure the best and most predictable performance (IOPS) for the environment. Without it, a significant decrease in array performance results.

Note

vSAN all-flash configurations are supported only with 10 GbE.

Network Speed Selection

Design Quality

1Gb

10Gb

Comments

Availability

o

o

Neither design option impacts availability.

Manageability

o

o

Neither design option impacts manageability.

Performance

Faster network speeds increase vSAN performance (especially in I/O intensive situations).

Recoverability

Faster network speeds increase the performance of rebuilds and synchronizations in the environment. This ensures that VMs are properly protected from failures.

Security

o

o

Neither design option impacts security.

 Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.

Network Bandwidth Design Decision

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-001

Use only 10 GbE for VMware vSAN traffic.

Performance with 10 GbE is optimal. Without it, a significant decrease in array performance results.

The physical network must support 10 Gb networking between every host in the vSAN clusters.

vSAN supports the use of vSphere Standard Switch or vSphere Distributed Switch. The benefit of using vSphere Distributed Switch is that it supports Network I/O Control which allows for prioritization of bandwidth in case of contention in an environment.

This design uses a vSphere Distributed Switch for the vSAN port group to ensure that priority can be assigned using Network I/O Control to separate and guarantee the bandwidth for vSAN traffic.

Virtual switch type affects performance and security of the environment.

Virtual Switch Types

Design Quality

vSphere Standard Switch

vSphere Distributed Switch

Comments

Availability

o

o

Neither design option impacts availability.

Manageability

The vSphere Distributed Switch is centrally managed across all hosts, unlike the standard switch which is managed on each host individually.

Performance

The vSphere Distributed Switch has added controls, such as Network I/O Control, which you can use to guarantee performance for vSAN traffic.

Recoverability

The vSphere Distributed Switch configuration can be backed up and restored, the standard switch does not have this functionality.

Security

The vSphere Distributed Switch has added built-in security controls to help protect traffic.

Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.

Virtual Switch Design Decision

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-002

Use the existing vSphere Distributed Switch instances in the management clusters.

Provide guaranteed performance for vSAN traffic in case of contention by using existing networking components. 

All traffic paths are shared over common uplinks.

VMware vSAN supports jumbo frames for vSAN traffic. 

A VMware vSAN design should use jumbo frames only if the physical environment is already configured to support them, they are part of the existing design, or if the underlying configuration does not create a significant amount of added complexity to the design.

Jumbo Frames Design Decision

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-003

Configure jumbo frames on the VLAN dedicated to vSAN traffic.

Jumbo frames are already used to improve performance of vSphere vMotion and NFS storage traffic.

Every device in the network must support jumbo frames.

VMware recommends isolating VMware vSAN traffic on its own VLAN. When a design uses multiple vSAN clusters, each cluster should use a dedicated VLAN or segment for its traffic. This approach prevents interference between clusters and helps with troubleshooting cluster configuration.

VLAN  Design Decision

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-004

Use a dedicated VLAN for vSAN traffic for each vSAN enabled cluster.

VLANs ensure traffic isolation.

 VLANs span only a single pod.

 A sufficient number of VLANs are available within each pod and should be used for traffic segregation.

VMware vSAN requires that IP multicast is enabled on the Layer 2 physical network segment that is used for intra-cluster communication. All VMkernel ports on the vSAN network subscribe to a multicast group using Internet Group Management Protocol (IGMP).

A default multicast address is assigned to each vSAN cluster at the time of creation. IGMP (v3) snooping is used to limit Layer 2 multicast traffic to specific port groups. As per the Physical Network Design, IGMP snooping is configured with an IGMP snooping querier to limit the physical switch ports that participate in the multicast group to only vSAN VMkernel port uplinks. In some cases, an IGMP snooping querier can be associated with a specific VLAN. However, vendor implementations might differ.