What Is iReliable?

Service Continuity is Paramount, Requiring High Network Reliability

In certain industries, a single hour of DC downtime will bring huge economic losses. For example, one hour of downtime in the finance industry will cause a loss of more than US$6 million. Therefore, DCNs need to improve the overall reliability to ensure zero service interruption.

Impact of IT system service interruption

Rapidly Advancing AI Technologies Demand Highly Reliable Networks

With the rapid development of AI technologies, especially the emergence of large-scale models, the demand for computing power has exploded. As models have advanced from the early small-scale models to today's models with hundreds of billions or even trillions of parameters, the training process requires handling massive amounts of data and complex computations, placing extremely high demands on the scale and performance of DC computing power. At the same time, as more AI application scenarios emerge, inference demands are also growing rapidly. In particular, fields such as intelligent customer service, smart security, and autonomous driving require the real-time inference to ensure application effectiveness. For example, in autonomous driving scenarios, vehicles must analyze sensor data and make decisions almost instantly, which requires DCs to provide low-latency, high-concurrency inference services.

In scenarios such as high-performance computing and AI training, network packet loss rates and latency must meet extremely stringent standards, as even minimal packet loss can degrade task performance or delay completion. Moreover, since the cloud carries enterprises' core services, the network must offer extremely high reliability. Any human error or slow fault response could result in significant economic losses. To mitigate these risks, networks must evolve beyond traditional monitoring to incorporate autonomous capabilities—predicting, detecting, and repairing faults proactively.

Network-Level Reliability Issues

When silent faults occur on the network, they can cause session-level service anomalies, such as link failures, forwarding entry exceptions, forwarding component malfunctions, ports in the physical Up state but unable to forward traffic, and configuration errors. These types of faults are often referred to in the industry as the "cancer" of network operations: they present no obvious symptoms, trigger no alarms, and are difficult to locate.

Currently, the industry mainly relies on traffic analyzers to collect data and detect anomalies, and depends on manual operations for troubleshooting and fault repair. However, this traditional approach is inefficient, taking anywhere up to several hours, which severely impacts service continuity.

Device-Level Reliability Issues

With the continuous acceleration of service provisioning and the introduction of virtualization technologies (such as VMs and containers), the uncertainty of network traffic has significantly increased. However, current network O&M methods still mainly rely on the traditional NMS and CLI. Minute-level network monitoring is no longer sufficient to meet service requirements for responsiveness within seconds. This passive fault detection mode often results in low O&M efficiency.

Fault locating today largely depends on expert experience, supported by auxiliary tools for segment-by-segment delimitation, per-flow analysis, and packet capture. This approach is not only inefficient but also fails to meet the need for rapid response.

Link failover upon link faults

In the typical Clos networking shown in the preceding figure, there are multiple paths between switches. When a link fails, a series of operations—such as port status detection, route convergence, and forwarding path switchover—are performed to switch traffic from the faulty link to the standby link. The entire process takes seconds to complete. When it comes to AI scenarios, however, each communication is completed within milliseconds, which means multiple rounds of communication will have been completed within seconds. Evidently, using the conventional route convergence approach will severely compromise the AI-powered computing efficiency.

In addition to the preceding challenges, traffic is interrupted for seconds during the traditional M-LAG upgrade, which cannot meet service continuity requirements.

Link-Level Reliability Issues

In traditional DCNs, aggregated links are typically established between individual devices. If a link or peer device fails, the affected device or server can no longer communicate with its counterpart. To meet stringent reliability requirements, the network must be able to withstand the complete failure of one device by ensuring that service traffic continues to be forwarded seamlessly through another.

Conventional storage networks often rely on Fibre Channel (FC), with current bandwidths commonly at 32 Gbit/s or 64 Gbit/s. In intra-city dual–center deployments, achieving 400GE interconnection would require the deployment of ten or even dozens of FC links, resulting in significant cost inefficiencies. By contrast, Ethernet technologies can deliver 100GE and 400GE interconnection bandwidths, but they face new challenges in long-distance transmission. Increased latency over extended distances directly undermines the effectiveness of backpressure mechanisms within the DC. For example, in a standard intra-city DC spanning 70 kilometers, the transmission latency typically exceeds 1 millisecond. At this level, existing storage backpressure mechanisms within the DC become mostly ineffective, making it impossible to guarantee lossless long-distance transmission.

DPCF

DPCF is a technology that addresses silent faults by enabling automatic fault detection without analyzers or manual intervention. It applies to scenarios such as routing blackholes and ARP entry anomalies—faults that are traditionally undetectable by automated means. Based on preset policies, DPCF performs rapid rerouting to achieve path switching within seconds. In critical domains such as finance, storage, and supercomputing, this ensures service recovery within seconds, delivering efficiency thousands of times greater than conventional approaches in the industry.

Network fault awareness

As shown in the following figure, during TCP traffic transmission, the sender sends data packets to the receiver. After receiving data packets, the receiver returns an ACK packet to the sender, indicating receipt of data prior to the ACK number. If the sender receives no ACK packet within a specific period of time, it retransmits the data packets.

The sender creates a flow table for TCP traffic matching ACL rules. If the interval at which the sender retransmits the data packets exceeds the configured fault detection interval, it is determined to be a timeout fault. In this way, network fault awareness is achieved.

TCP acknowledgment mechanism

Network fault recovery

In the traditional Layer 3 network architecture shown in the following figure, servers are connected using independent IP addresses. Leaf switches are deployed as independent Layer 3 gateways to forward Layer 2 and Layer 3 traffic. Spine switches are deployed as independent Layer 3 devices and are connected to leaf switches to implement ECMP load balancing.

This network architecture mainly applies to lossless scenarios such as finance, storage, and supercomputing. If a silent fault occurs on the network, services are interrupted for a long time and upper-layer services are severely affected. For online transaction applications, if packets are continuously lost, the transaction fails or in some cases the connection to the peer protocol stack times out entirely. This severely impacts application performance. After DPCF is deployed, the device can automatically detect faults if service traffic fails to be forwarded. It then quickly switches services to other ECMP members, ensuring service recovery within seconds.

Traditional three-layer networking for DPCF

Active/Standby or Active-active Egress DR

Active/Standby egresses: In the multi-PoD scenario, egresses can be deployed in multiple DCs. These egresses work in active/standby mode, which can be shared by multiple DCs.

Active and standby egresses in the multi-PoD scenario

Active-active egresses: In the multi-PoD scenario, egresses can be deployed in multiple DCs. These egresses work in active-active mode to implement load balancing. Traffic of a DC is preferentially forwarded through the local egress. If a fault occurs, the traffic can be transmitted through the egress of the other DC.

Active-active egresses in the multi-PoD scenario

Across multiple DCs and PoDs, network-level active/standby or active-active egress DR is used to implement automatic service switchover, ensuring service continuity.

M-LAG Hitless Upgrade

M-LAG hitless upgrade allows traffic to be switched to the backup link before an M-LAG member device is upgraded. This prevents traffic loss and ensures service continuity during the upgrade.

As shown in the following figure, Leaf1 and Leaf2 form an M-LAG and connect to the network through dynamic routing protocols, and the server is dual-homed to the M-LAG.

M-LAG hitless upgrade

The process for performing an M-LAG hitless upgrade on Leaf1 and Leaf2 is as follows:

1. On Leaf1, adjust the route cost, adjust the route advertisement priority, and set the downlink interface to Down, so that traffic on Leaf1 is switched to Leaf2.
2. After confirming that traffic on Leaf1 is switched to Leaf2, upgrade Leaf1.
3. After Leaf1 is upgraded, restore its route cost and route advertisement priority, and set its downlink interface to Up, so that traffic is switched back to Leaf1.
4. Repeat the preceding steps to upgrade Leaf2 to complete the M-LAG upgrade.

DPFR

DPFR technology implements millisecond-level convergence and provides local or remote fast convergence based on the data plane. It implements rapid fault detection, local and remote fast fault convergence, fault notification generation, reception and relay processing, and entry aging. In key application scenarios such as online transaction, DPFR enables application-unaware fast fault convergence. This ensures that service performance is not compromised in case of a link fault.

DPFR will be detailed in the following two typical scenarios.

• Scenario 1: local fast convergence, with multiple next hops involved. In the following figure, after the traffic from Server3 to Server1 reaches Leaf2, the next hop of Leaf2 can be Spine1 or Spine2. When the Leaf2 -> Spine2 path fails, traffic can be rapidly switched to the Leaf2 -> Spine1 path on the local side.
  
  Local fast convergence of DPFR
• Scenario 2: remote fast convergence, with only one next hop involved. In the following figure, Server1 accesses Server3 through the path Leaf1 -> Spine2 -> Leaf2. When the traffic reaches Spine2, only one link is available for forwarding the traffic to Leaf2. In this scenario, if the Spine2 -> Leaf2 path is faulty, Spine2 sends a notification packet to Leaf1. Leaf1 then locally switches the traffic to the Leaf1 -> Spine1- > Leaf2 path.
  
  Remote fast convergence of DPFR

M-LAG

Traditional aggregated links are typically established between individual devices. If a link or peer device fails, the affected device or server can no longer communicate with its counterpart. M-LAG addresses this by aggregating interfaces on different devices in an M-LAG into a single logical interface. In this way, even if a device or link fails, the aggregated link can still forward traffic, ensuring reliable data transmission. In addition to improving link reliability, M-LAG provides the following benefits:

• Virtualizes both M-LAG member devices into one logical device, which provides a loop-free Layer 2 topology. This simplifies the logical networking and improves bandwidth efficiency.
• Allows both M-LAG member devices to be upgraded separately, improving upgrade efficiency and minimizing service interruptions.

Given these benefits, M-LAG is recommended for scenarios that require high network reliability and the shortest possible service interruptions during a device upgrade.

M-LAG

Long-Distance Lossless Algorithm

Spatiotemporal variables are added to the short-distance iLossless algorithm, providing the four-dimensional iLossless-Data Center Interconnect (DCI) algorithm that addresses packet loss in long-distance transmission. Convolutional prediction based on big data determines random traffic and responds to traffic changes in advance, ensuring lossless Ethernet transmission over long distances. In addition, Huawei's CloudEngine series switches support MACsec encryption, which ensures zero leakage of cross-DC data transfers.

Long-distance lossless algorithm

Automotive Industry: Zero Packet Loss in Network Upgrades, Ensuring Continuous Production

A car manufacturer's production network needs to operate 24/7 without interruption, where every minute of downtime results in losses of tens of thousands of dollars. In traditional network models, the customer's network department had to repeatedly coordinate operation windows with the business department. Each device upgrade inevitably caused 20–100 ms of packet loss, severely impacting service continuity and failing to meet high reliability requirements.

Huawei's iReliable technology achieved zero service interruption during link switching and zero packet loss during network upgrades, which received high praise from the customer. This technology significantly improved network reliability, reduced the workload and risks for maintenance staff during service cutovers, and provided strong assurance for continuous and stable production.

Banking Industry: Ensuring Service Continuity with Financial-Grade Reliability

Before live network reconstruction, a bank faced challenges including aging core network devices, complex O&M, and difficult service continuity assurance. Other vendors' devices on the live network were approaching end-of-life, with insufficient vendor support, leading to declining hardware reliability and longer fault repair cycles. All of this threatened service continuity. At the same time, traditional networks relied on CLI-based configuration and manual troubleshooting. Network expansion or service changes would take several weeks, which failed to keep pace with the rapid iteration needs of financial services. Traditional networks also suffered from low manual troubleshooting efficiency (measured in hours), resulting in long fault repair cycles that threatened service continuity.

Huawei enabled smooth migration from the traditional network to Huawei's SDN using migration tools. Huawei's iReliable technology enhanced reliability across the entire network, achieving millisecond-level convergence in the event of any node failure. This ensured zero interruption to financial services and met the stringent requirements of high-concurrency transactions and real-time clearing.