Home Search Center Intelligent Model Selection IP Encyclopedia

What Is an Overlay Network?

Overlay networks are virtual logical networks constructed on an underlay network using network virtualization technologies. Although different overlay networks share devices and lines on an underlay network, services on overlay networks are decoupled from physical networking and interconnection technologies on the underlay network. With the help of the multi-instance technology — core networking technology adopted by the SD-WAN and data center solutions, overlay networks can serve not only different services (such as multiple departments) of the same tenant but also different tenants.

Why Do We Need Overlay Networks?

Overlay networks are logical networks established on an underlay network. We need overlay networks due to the limitations of the underlay network.

Underlay Network

An underlay network, as the name implies, is the underlying physical infrastructure of overlay networks.

As shown in the following figure, an underlay network is a physical network consisting of multiple types of devices, which is responsible for data packet transmission between networks.

Typical underlay network
Typical underlay network

On an underlay network, devices such as switches, routers, load balancers, and firewalls can be interconnected. However, routing protocols must be used to ensure IP connectivity between these devices.

The underlay network can be a Layer 2 or Layer 3 network. A typical example of Layer 2 underlay network is an Ethernet network, on which VLANs are created. The Internet is a typical Layer 3 underlay network. The Open Shortest Path First (OSPF) or Intermediate System to Intermediate System (IS-IS) protocol is used for route control in an autonomous system (AS), while the Border Gateway Protocol (BGP) is used for route transfer and interconnection between ASs. As technologies advance, underlay networks can also be constructed using Multiprotocol Label Switching (MPLS), which is a wide area network (WAN) technology running between Layer 2 and Layer 3.

However, traditional network devices forward data packets based on hardware. An underlay network constructed based on traditional network devices has the following problems:

  • Hardware devices forward data packets based on destination IP addresses. Therefore, packet forwarding is highly dependent on transmission paths.
  • When services are added or changed, existing underlying network connections need to be modified. The reconfiguration is time-consuming.
  • The Internet cannot satisfy security requirements of private communications.
  • Network slicing and segmentation are complex, and cannot implement on-demand allocation of network resources.
  • Multi-path forwarding is complex, and underlying networks cannot be integrated to implement load balancing.

Overlay Network

To get rid of the limitations of underlay networks, virtual overlay networks can be created over underlay networks using network virtualization technologies.

Overlay network topology
Overlay network topology

Devices on overlay networks are interconnected through logical links, constituting overlay topologies.

Tunnels are established between interconnected overlay devices. When sending a data packet, a device adds a new IP header and a tunnel header to the data packet and shields the inner IP header. The data packet is then forwarded based on the new IP header. When the data packet is received by another device, the device removes the outer IP header and tunnel header to obtain the original data packet. In this process, overlay network devices are unaware of the underlay network.

Overlay networks support various network protocols and standards, including virtual extensible local area network (VXLAN), Network Virtualization using Generic Routing Encapsulation (NVGRE), single spanning tree (SST), GRE, Network Virtualization over Layer 3 (NVO3), and Ethernet Virtual Private Network (EVPN).

With the introduction of software-defined networking (SDN) technology, overlay networks with the controller deployed have the following advantages:

  • Traffic transmission does not depend on specific lines. Overlay networks use tunneling technologies to flexibly select different underlying links and use multiple methods to ensure stable traffic transmission.
  • Different virtual topologies can be established on overlay networks as required without the need to modify the underlying network.
  • Encryption approaches can be used to ensure security of private traffic on the Internet.
  • Network slicing and segmentation are supported. Different services can be separated to achieve optimal allocation of network resources.
  • Multi-path forwarding is supported. On overlay networks, traffic can be transmitted from the source to the destination through multiple paths to implement load balancing and maximize the bandwidth utilization of links.

What Are the Examples of Overlay Networks?

Overlay networks are widely used in SD-WAN and data center solutions. The overlay network topology varies according to underlay network architectures.

Data Center Overlay Network

As the data center architecture evolves, most data centers use the spine-leaf architecture to construct underlay networks and use VXLAN technology to implement interconnection of overlay networks. Service packets are transmitted on VXLAN overlay networks, and are decoupled from physical bearer networks.

Data center overlay network
Data center overlay network

Leaf and spine nodes are fully meshed so that equal-cost multi-path routing (ECMP) paths are available to ensure high availability of the network.

Leaf nodes function as access nodes to connect various network devices on the underlay network to the VXLAN network. Leaf nodes are also edge devices on the overlay network and function as VXLAN tunnel endpoints (VTEPs).

Spine nodes are core nodes of a data center network, which provide high-speed IP forwarding and connect to leaf nodes through high-speed interfaces.

SD-WAN Overlay Network

The underlay network of SD-WAN is based on WAN and uses hybrid links to implement interconnection between the headquarters, branch sites, and cloud sites. The logical topologies of overlay networks are built to meet interconnection requirements in different scenarios.

Figure 1-5 SD-WAN overlay network (using the hub-spoke networking an example)

An SD-WAN network consists of customer-premises equipment (CPEs), which are classified into edge devices and gateways.

  • Edge: egress device of an SD-WAN site.
  • Gateway: a device that connects SD-WAN sites and other networks (for example, legacy VPNs).

Multiple types of overlay networks can be built based on the enterprise network scale, number of hub sites, and inter-site communication requirements.

  • Hub-spoke networking: This networking is applicable to enterprises that have one or two data centers. Branches access services deployed in the headquarters or data centers through the WAN. A small amount of traffic is transmitted between branches, or branches do not need to communicate with each other. Traffic between branches traverses the headquarters or data centers.
  • Full-mesh networking: This networking is applicable to small enterprises with a small number of sites or large enterprises whose branches need to collaborate with each other. Collaborative services of large enterprises, for example, high-value applications including VoIP and video conferencing, have high requirements on network performance such as the packet loss rate, delay, and jitter. To meet requirements of such services, branches are recommended to directly communicate with each other.
  • Hierarchical networking: This networking features a clear network structure and excellent scalability and is therefore applicable to enterprises that have a large number of sites or multinational enterprises with sites widely distributed across countries or regions.
  • Mutli-hub networking: This networking is applicable to enterprises that have multiple data centers and deploy service servers in each data center to provide services for branches.
  • PoP networking: When carriers or MSPs provide SD-WAN access services for enterprises, some enterprises may have both legacy branch sites and SD-WAN sites, which need to communicate with each other. In this networking, the Interworking Gateway (IWG) can be deployed to enable communication between SD-WAN sites and legacy MPLS VPN sites for multiple enterprise tenants.

Overlay Network vs. Underlay Network

The following table compares overlay and underlay networks.

Table 1-1 Underlay network vs. overlay network


Underlay Network

Overlay Network

Data transmission

Data is transmitted through network devices such as routers and switches.

Data is transmitted over virtual links between nodes.

Packet encapsulation and overhead

Packet encapsulation is performed at Layer 2 and Layer 3.

Data packets need to be encapsulated based on the source and destination, resulting in extra overheads.

Packet control



Deployment time

Rollout of new services involves a large number of configurations, which is time-consuming.

When new services are deployed, only the virtual network topology structure needs to be modified, enabling fast service rollout.

Multi-path forwarding

Due to low scalability, multi-path forwarding is required, which increases the overhead and network complexity.

Multi-path forwarding on virtual networks is supported.


The scalability is poor. After the underlying network is built, it is difficult to add new devices.

The scalability is high. For example, a VLAN supports a maximum of 4096 discriminators, whereas a VXLAN provides a maximum of 16 million discriminators.


Ethernet switching, VLAN, and routing protocols (OSPF, IS-IS, and BGP)


Multi-tenant management

NAT- or VRF-based isolation is required, which is a great challenge on large-scale networks.

Overlapping IP addresses of multiple tenants can be managed.

About This Topic
  • Author: Ding Heng
  • Updated on: 2021-11-18
  • Views: 16548
  • Average rating:
Share link to