Simplify Hybrid Cloud Deployments with Overlay Networking

Grazed from Sys Con Media. Author: Brandon Hoff.

IT networks have reached a juncture where they must transition from static, host-centric, vLAN centric infrastructures, to modern, efficient programmable network fabrics that allow enterprises to leverage capacity on demand. They must also have the ability to move virtual machine (VM) workloads within the data center, between data centers, and into the external cloud. However, this VM mobility places new requirements on data center networks to efficiently support multi-tenant environments.

As organizations make the shift from a virtualization-driven data center toward an infrastructure designed for cloud computing, they must shift their focus to application consolidation at scale for multiple customers in multi-tenant data centers. While virtualization has reduced the cost and time required to deploy a new application from weeks to minutes, driving costs down from thousands of dollars to a few hundred, reconfiguring the network for a new or migrated virtual workload still takes approximately a week and can cost thousands of dollars…

To scale existing networking technology for multi-tenant infrastructures, organizations must deploy new solutions that enable VM communication and migration across Layer 3 boundaries without impacting connectivity. At the same time, they must also ensure isolation for thousands of logical network segments and maintain existing customer VM addresses and Media Access Control (MAC) addresses as VMs migrate anywhere. It is time to virtualize the network. Overlay Networks, along with software-defined networking (SDN) and other network services, are all key building blocks in network virtualization.

Cloud Computing Growth and Data Center Consolidation
Enterprise adoption of public and private cloud computing models continues unabated, with IDC forecasting server revenue to grow to $57.4 billion in 20151, and cloud services spending to increase to $60 billion this year, a growth of 26 percent2.

Additionally, 80 percent of respondents to a recent study by Enterprise Strategy Group3 indicated having three or more data centers worldwide, and 54 percent have started or completed data center consolidation projects.

This trend toward data center consolidation is a key driver of new requirements for multi-tenant, cloud-based private infrastructures that are manageable and scalable.

The ability to ensure optimized growth of cloud computing infrastructure, and achieve ROI on infrastructure investment, rests on these fundamental tenets – based on optimization of three elements of data center infrastructures:

Server – The cloud is a natural evolution of server virtualization, with advances including powerful multi-core processors, server clusters/pods and standardized low cost servers. Collectively, they offer a number of options to design an optimized cloud infrastructure.
Storage – Storage virtualization, de-duplication, on-demand and simplified provisioning, disaster recovery and scalable file storage to support the various availability, reliability, and scalability requirements of the various workloads in the hybrid cloud.
Network – Forrester Research reports that 50 percent of companies that didn’t invest in the network suffered from degradation of services or had noticeable increases in operational costs4. Advances in Network Interface Cards (NICs) are driving the optimization of data center networking for the cloud infrastructure. The NIC has evolved into a networking device that sits between the vSwitch and the physical network that can apply policies to different network flows. Advanced NICs also provide a rich suite of network services that optimizes network flows between CPU cores and Ethernet ports.

Cloud Network Scalability Challenges
The current practice of host-centric segmentation based on vLANs/L2 subnets, with server clusters separated by L3 physical IP networks, limit today’s multi-tenant cloud infrastructure from meaningful network scaling for longer term growth. These limitations include security, capacity constraints, limitations due to the Spanning Tree Protocol (STP) protocol, and increased network management costs.

Limitations of STP
STP effectively limits the number of VMs in a vLAN or IP network segment to 200-300. STP also creates a condition where links between switches are underutilized because redundant links in the networking infrastructure are effectively turned off to avoid connectivity loops. Plus, STP has limited ability to provide multipathing for networking resiliency due to the STP’s requirement for a single active path between network nodes. While there are proposals to expand the L2 domain, none of these proposals meet the scalability requirements of the cloud.

Limitations using today’s IEEE 802.1Q vLAN grouping standards
VMs belonging to a group, whether departmental, line of business or corporate function, require administrators to tag them with vLAN identifiers for L2 segment isolation. However, the IEEE 802.1Q standard, with its 12-bit namespace for a vLAN Identifier (VID), allows only 4,094 vLAN IDs. Cloud providers have the goal of providing cloud services to thousands of customers. Since customers will have multiple vLANs in their infrastructure, vLAN technology limits a cloud provider to only a few hundred customers. vLANs cannot scale to meet the requirements of the cloud.

Overall, current physical networks with vLANs and STP do not support the elastic and dynamic requirements of building data centers for the hybrid cloud. As more and more workloads are consolidated on servers and the number of cores on each server keeps growing, a rack with 40 servers can hold 800 or more virtual workloads. Within a rack, the servers must be configured into at least four vLANs, and probably more. Since L2 can be only stretched between a handful of racks, the L2 network limits the ability of VMs to migrate beyond a handful of racks.

Capacity expansion across server racks or server clusters
A single rack with multiple host servers, or multiple server racks residing in a common cluster or pod, effectively provides a small virtualized environment for a single tenant. Each server rack may constitute an L2 subnet, while multiple host server racks in a cluster will likely be assigned their own L3 network.

Elastic demand for additional computing resources by the tenant may require more servers or VMs on a different rack, within a different L2 subnet, or on a different host cluster in a different L3 network. These additional VMs need to communicate with the application’s VM on the primary server rack or cluster.

Additionally, administrators may need to migrate VMs to another server cluster to achieve inter-host workload mobility and optimize usage of the underlying server hardware resources. This may be driven by a number of scenarios, including:

Migration from, and shut down of, underutilized hosts to consolidate on another more fully utilized host to reduce energy costs.
Decommissioning of one or more older hosts and bringing up workloads on newly added infrastructure.

In each scenario, IT needs to stretch the tenant’s L2 subnet to connect the servers or VM since VM-to-VM communications and/or VM migration between hosts currently requires the hosts to be in the same L2 subnet.

Infrastructure capital expenditures for over provisioning
It is common for enterprise IT to over-provision a data center by as much as 10 percent to provide spare capacity for server failover. This over-provisioning practice means data center managers allocate one spare physical host server for every 11, based on an approximate 23:1 consolidation ratio with the L2 domain limited by STP of 254 devices (network nodes). Over-provisioning can span the entire infrastructure and cover servers, racks, power and cooling, networking, administration and management, and some storage.

Goal: Reduce the costs of Networking in Virtualized and Cloud Environments
It is no secret that server virtualization has drastically reduced the cost and time to deploy a new server workload. At a recent Open Networking Summit, VMware asserted5 that before virtualization, it would take 10 weeks and cost approximately $10,000, plus software costs, to deploy a new server workload. With virtualization, a new server workload can be deployed in a few minutes at a deployment cost of approximately $300, plus software costs. However the network hasn’t been virtualized, nor has it really changed during the last 10 years. Because the network hasn’t been virtualized, it takes approximately five days and can cost $1,800 to reconfigure the network devices to support a new or migrated virtual workload, including routers, IP and MAC addresses, firewalls, intrusion detection systems, etc. The use of overlay networks is the first step toward network virtualization and promises to significantly reduce the costs of network management for virtualized workloads and hybrid cloud deployments.

Overlay Networking is a new networking technology that reduces the cost of managing data center networks and cloud networks, and enables VMs to migrate across L3 networks without penalty, and management tools that are integrated into hypervisor management tools to reduce the complexity and cost of management.

Overlay Networking
The Internet Engineering Task Force (IETF) is currently evaluating two different "overlay networking" standards to overcome these challenges: Network Virtualization using Generic Routing Encapsulation (NVGRE) initiated by Microsoft and others and Virtual eXtensible Local Area Networks (VXLAN) initiated by VMware and others. Both standards are designed to enable the efficient and fluid movement of virtual resources across cloud infrastructures in order to pave the way for large scale and cloud scale VM deployments. Let’s take a closer look at what’s driving the need for overlay networking and the options being considered within the IETF to build the data plane for virtual networks. For the most part, these standards focus on server based encapsulation and leave the management and control plane features as a separate work effort.

Overcoming Cloud Network Scalability Challenges with Overlay Networking
Microsoft, VMware and Emulex, along with other industry leaders, are working to define a vision for a new, virtualized overlay network architecture that enables the efficient and fluid movement of virtual resources across cloud infrastructures. This will help pave the way for large scale and cloud VM deployments.

NVGRE and VXLAN are two emerging overlay networking standards – both providing L2 overlay schemes over an L3 network – which enable the connection between two or more L3 networks, while giving the appearance that they share the same L2 subnet. This enables inter-VM communications or VM migrations across L3 networks while operating as if they belonged to the same L2 subnet.

Comparing NVGRE and VXLAN
NVGRE Ethernet Frame Encapsulation
NVGRE is designed to encapsulate Ethernet L2 Frames in GRE to enable the creation of virtualized L2 subnets that can span physical L3 IP networks. In this environment, a unique 24-bit ID called a Tenant Network Identifier (TNI) is added to the L2 Ethernet frame using the lower 24 bits of the GRE Key field. This new 24-bit TNI enables more than 16 million L2 (logical) networks to operate within the same administrative domain, a scalability improvement of many orders of magnitude over the 4,094 vLAN segment limit prescribed by the IEEE.

Essentially, NVGRE is a tunneling scheme; using the GRE routing protocol as defined by RFC 2784 and extended by RFC 2890. Each TNI is associated with an individual GRE tunnel and uniquely identifies, as its name suggests, a cloud tenant’s unique virtual subnet. NVGRE thus isn’t a new standard, per se, since it uses the already established GRE protocol between hypervisors.

VXLAN Ethernet Frame Encapsulation
VXLAN also encapsulates Ethernet L2 Frames in UDP to enable the creation of virtualized L2 subnets that can span physical L3 IP networks. Each L2 overlay network, mentioned above, is interchangeably also called a VXLAN Segment. A unique 24-bit segment ID called a VXLAN Network Identifier (VNI) is added. Like the TNI referenced above, this new 24-bit VNI enables more than 16 million VXLAN segments to operate within the same administrative domain, a scalability improvement of many orders of magnitude over the 4,094 vLAN segment limit discussed above.

The L2 frame with VXLAN encapsulation is then encapsulated with an outer UDP Header, outer IP header and finally an outer Ethernet Frame. As a result of this encapsulation, think of VXLAN as another tunneling scheme. As currently designed, VMware ESXi hosts will act as the VXLAN Tunnel End Points (VTEP). The VTEPs encapsulate the virtual machine traffic in a VXLAN header, and later strip it off and present the destination virtual machine with the original L2 packet.

Summary
Overlay Networks solves current networking problems for virtualized and cloud environments caused by the limited number of vLANs that can be deployed as well as the limitations of STP. Overlay networks promises to reduce data center network expenses, enable VMs to migrate across L3 boundaries without losing connectivity, and provide an integrated management platform for the cloud.

Given the cost and complexity of using traditional networks to deploy and manage virtual servers, overlay network protocols such as NVGRE and VXLAN can reduce the time and cost to deploy and manage a virtual workload. They provide reduced complexity, improved security and VM mobility to make hybrid cloud deployments as simple to operate and deploy as software-as-a-service (SaaS) solutions are today.