IP Encyclopedia > Graceful Restart

What Is Graceful Restart?

As an ordered and controllable control-plane protocol restart technology, GR is widely used in scenarios such as active/standby switchover and system upgrade. It helps implement NSF at the protocol plane and ensures normal data forwarding when the control-plane protocol is restarted, preventing key services on the network from being affected.

Contents

Why Is Graceful Restart Required?

Currently, various value-added services, such as Internet Protocol television (IPTV) and video conferencing, have been widely deployed. Network interruption may affect a large number of services and cause great losses. Hence, the reliability of the basic network for carrying services has attracted much attention. To improve the reliability of network devices and implement non-stop data forwarding, modern high-performance network devices must meet certain requirements in terms of hardware, software, and protocols.

Framework model of the modern high-performance network device

The hardware requirements are as follows:

Dual main control board redundancy: Two main control boards are configured in redundancy mode. That is, one functions as the active main board (AMB) and is in the working state, and the other functions as the standby main board (SMB) and is in the backup state. Once the AMB fails, an active/standby switchover is performed, and the SMB takes over the work of the faulty AMB and restarts the control plane.

Distributed structure: The control plane and forwarding plane are allocated to different components and have corresponding memory and processors. The control-plane processor is located on the main control board and is responsible for route learning and calculation. Whereas the forwarding-plane processors are located on the interface boards and are responsible for data packet forwarding, such as IP and Multiprotocol Label Switching (MPLS) packet forwarding. The distributed structure ensures that the interface board is not restarted and the forwarding information table (FIB) entries are not withdrawn during the active/standby switchover of the main control board. The data thereby is forwarded according to the original forwarding table obtained from the main control board.

The software requirements are as follows:

When the AMB is working properly, it backs up information such as configuration information and interface status information to the SMB.

The protocol requirements are as follows:

When the protocol on the control plane is restarted, the neighbor relationship does not flap. Take a routing protocol as an example. Generally, when a router fails, neighbors at the routing protocol plane detect that their neighbor relationships go down and then up again after a period of time. This is neighbor relationship flapping. The flapping triggers the devices on the entire network to recalculate routes, causing route flapping. Furthermore, route blackhole may occur on the restarted device within a period of time, or the neighbors may switch data services from the restarted device to other devices, which greatly reduces the network reliability.

To resolve the protocol flapping problem on the entire network caused by neighbor relationship flapping during control-plane protocol restart, GR is introduced.

As an ordered and controllable control-plane protocol restart technology, GR helps implement NSF on the protocol plane, minimizes the impact of the protocol restart, and is widely used in scenarios such as active/standby switchover and system upgrade scenarios. It brings the following benefits:

Retains the neighbor relationship during the protocol restart, ensuring uninterrupted service forwarding.
Keeps the protocol flapping range within neighbors connected to the restarted device so that the flapping does not affect the entire network.
Collaborates with neighboring devices to implement fast protocol convergence and restore local forwarding information.

What Roles Are Involved in a Graceful Restart?

GR involves the following roles:

GR restarter: A GR restarter is a GR-enabled device on which an active/standby switchover takes place. The switchover can be triggered by an administrator or a fault. During the switchover, the GR restarter notifies its neighbors of the switchover and requests them to maintain neighbor relationships with it.
GR helper: A GR helper – a neighbor of the GR restarter – assists the GR restarter during the restart. A GR helper must be able to identify GR signaling. When the GR restarter performs an active/standby switchover, the GR helper maintains the neighbor relationship with the GR restarter. After the active/standby switchover is complete, the GR helper assists the GR restarter in restoring network topology information.

GR session: A GR session is responsible for negotiation between a GR restarter and a GR helper. The process includes protocol restart notification and information exchanging during a protocol restart. The GR restarter and GR helper can use a GR session to notify each other of their GR capability.
GR timer: A GR helper starts the GR timer when finding that the GR restarter is down and keeps the topology information or routes obtained from the GR restarter until the timer expires.

How Does Graceful Restart Work?

To support GR, the following control-plane network protocols are extended:

Routing protocols: Border Gateway Protocol (BGP), Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), and more.
MPLS-related protocols: Label Distribution Protocol (LDP) and Resource Reservation Protocol (RSVP).

VPN GR: VPN GR is an integrated application of technologies such as BGP GR, IGP GR, and LDP GR in VPN scenarios.

This section uses BGP as an example to describe the working process of BGP GR.

BGP Extension for GR

To implement GR, RFC 4724 extends BGP as follows:

A new Graceful Restart Capability (BGP-CAP) is added to the BGP Open messages. A BGP speaker can use this capability to instruct its peers to retain its forwarding status during a BGP restart. The length of BGP-CAP is variable. It consists of the Restart Flags field, Restart Time field, and a 0 to 63 <Address Family Identifier, Subsequent Address Family Identifier, Flags for Address Family> 3-tuples.

The following shows the format of the Graceful Restart Capability.

+--------------------------------------------------+          
| Restart Flags (4 bit)                           | 
+--------------------------------------------------+ 
| Restart Time in seconds (12 bit)                |
+--------------------------------------------------+ 
| Address Family Identifier (16 bit)              |
+--------------------------------------------------+ 
| Subsequent Address Family Identifier (8 bit)    | 
+--------------------------------------------------+
| Flags for Address Family (8 bit)                |
+--------------------------------------------------+ 
| ...                                              |
+--------------------------------------------------+ 
| Address Family Identifier (16 bit)              |
+--------------------------------------------------+ 
| Subsequent Address Family Identifier (8 bit)    | 
+--------------------------------------------------+
| Flags for Address Family (8 bit)                |
+--------------------------------------------------+

The following table describes the meanings of the fields.

Table 1-1 Description of BGP-CAP fields

Field	Description
Restart Flags	The most significant bit is defined as the restart state bit (R bit for short), indicating whether the BGP speaker is restarted. If the BGP speaker is restarted, the peer should send route update messages to the BGP speaker instead of waiting for an End-of-RIB (EOR) message from the speaker.
Restart Time in seconds	The maximum time for re-establishing a BGP session after the restart
<Address Family Identifier, Subsequent Address Family Identifier, Flags for Address Family>	Address Family Identifier (AFI): The address family that supports GR
	Subsequent Address Family Identifier (SAFI): Used together with AFI to identify the address family that supports GR.
	Flags for Address Family: The most significant bit is defined as the forwarding state bit (F bit for short), indicating whether the forwarding status of the routes with the specified AFI and SAFI is retained on the interface board during the last BGP restart. This bit is very important for implementing non-stop forwarding. If the system does not retain the route forwarding entries on the interface board during the restart, even if BGP has the GR capability, non-stop forwarding cannot be achieved.

A new type of message, End-of-RIB (EOR) message, is added for the BGP speaker to notify the BGP peer that the initial route update process is complete after the BGP session is established.
Two important timers are defined to help implement BGP GR.
Restart timer: indicates the period during which a GR helper retains the topology information or routes obtained from the GR restarter after the GR helper finds that the GR restarter is down.

EOR timer: the maximum time during which the GR restarter and GR helper wait for the EOR information from each other. It can also be considered as the maximum time for the peer to send route updates for route synchronization.

If the GR restarter does not receive an EOR message from the GR helper within the EOR timer, the GR restarter performs BGP route calculation based on the existing routes.

If the GR helper does not receive an EOR message from the GR restarter within the EOR timer, the GR helper ages the routes related to the GR restarter.

Working Process

As shown in the following figure, DeviceA and DeviceB establish a BGP peer relationship and both support BGP GR. When DeviceA restarts in GR mode, DeviceB assists DeviceA in completing GR. DeviceA is called the GR restarter, and DeviceB is called the GR helper.

BGP GR process

The working process of BGP GR is as follows:

During the initial BGP peer relationship establishment, DeviceA and DeviceB inform each other that they are GR-capable and negotiate GR capabilities.
After an active/standby switchover is performed on DeviceA, its interface board is not restarted, the routing table and forwarding table remain unchanged, and the forwarding on DeviceA is normal. After DeviceB detects that the TCP connection with DeviceA is interrupted, DeviceB retains the routes and forwarding entries related to DeviceA during the restart, and adds the stale flag to the entries so that other devices cannot detect the switchover on DeviceA. Then, DeviceB starts the Restart timer and waits for the re-establishment of the connection.
After DeviceA restarts, DeviceB and DeviceA re-establish a TCP connection. DeviceA sets the R flag to 1 in the BGP Open message, indicating that DeviceA has just restarted, and the F flag to 1, indicating that the address family (identified by AFI and SAFI) remains in the forwarding state. DeviceB then deletes the Restart timer and starts the EOR timer.
Once DeviceA and DeviceB re-establish a BGP peer relationship, DeviceB resends route update messages to DeviceA. After the update is complete, DeviceB sends an EOR message to DeviceA. DeviceA receives and processes the route update messages from the peer, and starts the EOR timer.
DeviceA performs BGP route calculation when either of the following conditions is met:
- EOR messages from all peers are received and the EOR timer is deleted.
- The EOR timer times out but DeviceA fails to receive EOR messages from all peers.
After route calculation is complete, DeviceA sends route update messages to each peer. After the route update is complete, DeviceA sends an EOR message to each peer.
DeviceB exits the GR helper state and ages the routes related to DeviceA when either of the following conditions is met:
- DeviceB receives the EOR message from DeviceA, and the EOR timer is deleted.
- The EOR timer times out but DeviceB fails to receive the EOR message from DeviceA.
After DeviceB exits the GR process, it deletes the retained stale routes learned from DeviceA and continues the BGP process.