TOC |
|
The purpose of this document is to guide the design of congestion notification in any lower layer or tunnelling protocol that encapsulates IP. The aim is for explicit congestion signals to propagate consistently from lower layer protocols into IP. Then the IP internetwork layer can act as a portability layer to carry congestion notification from non-IP-aware congested nodes up to the transport layer (L4). Following these guidelines should assure interworking between new lower layer congestion notification mechanisms, whether specified by the IETF or other standards bodies.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”
This Internet-Draft will expire on March 21, 2014.
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
1.
Introduction
1.1.
Scope
2.
Terminology
3.
Modes of Operation
3.1.
Feed-Forward-and-Up Mode
3.2.
Feed-Up-and-Forward Mode
3.3.
Feed-Backward Mode
3.4.
Null Mode
4.
Feed-Forward-and-Up Mode: Guidelines for Adding Congestion Notification
4.1.
IP-in-IP Tunnels with Tightly Coupled Shim Headers
4.2.
Wire Protocol Design: Indication of ECN Support
4.3.
Encapsulation Guidelines
4.4.
Decapsulation Guidelines
4.5.
Sequences of Similar Tunnels or Subnets
4.6.
Reframing and Congestion Markings
5.
Feed-Up-and-Forward Mode: Guidelines for Adding Congestion Notification
6.
Feed-Backward Mode: Guidelines for Adding Congestion Notification
7.
IANA Considerations (to be removed by RFC Editor)
8.
Security Considerations
9.
Conclusions
10.
Acknowledgements
11.
Comments Solicited
12.
References
12.1.
Normative References
12.2.
Informative References
Appendix A.
Outstanding Document Issues
Appendix B.
Changes in This Version (to be removed by RFC Editor)
TOC |
Explicit Congestion Notification (ECN [RFC3168] (Ramakrishnan, K., Floyd, S., and D. Black, “The Addition of Explicit Congestion Notification (ECN) to IP,” September 2001.)) is defined in the IP header (v4 & v6) to allow a resource to notify the onset of queue build-up without having to drop packets, by explicitly marking a proportion of packets with the congestion experienced (CE) codepoint.
ECN removes nearly all congestion loss and it cuts delays for two main reasons: i) it avoids the delay when recovering from congestion losses, which particularly benefits small flows, making their completion time predictably short [RFC2884] (Hadi Salim, J. and U. Ahmed, “Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks,” July 2000.); and ii) as ECN is used more widely by end-systems, it will gradually remove the need to configure a degree of delay into buffers before they start to notify congestion (the cause of bufferbloat). The latter delay is because drop involves a trade-off between sending a timely signal and trying to avoid impairment, whereas ECN is solely a signal not an impairment, so there is no harm triggering it earlier.
Some lower layer technologies (e.g. MPLS, Ethernet) are used to form large subnetworks with IP-aware nodes only at the edges. These networks are often designed so that it is rare for interior queues to overflow. However, this has often only been possible because the original design of TCP did not scale, and fixes (e.g. [RFC1323] (Jacobson, V., Braden, B., and D. Borman, “TCP Extensions for High Performance,” May 1992.)) proved hard to deploy. Now that modern operating systems are finally capable of saturating inerior links, even the buffers of well-provisioned interior switches will need to signal episodes of queuing. However, the above benefits of ECN can only be fully realised if support for ECN is added to the relevant subnetwork technology, as well as IP. When a lower layer queue drops a packet it does not just drop at that layer; the packet disappears from all layers. In contrast, when a lower layer marks a packet with ECN, the marking needs to be explicitly propagated up the layers.
Propagation of ECN is defined for MPLS [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.), and is being defined for TRILL [trill‑rbridge‑options] (Eastlake, D., Ghanwani, A., Manral, V., and C. Bestler, “RBridges: Further TRILL Header Extensions,” June 2012.), but it remains to be defined for a number of other subnetwork technologies.
Similarly, ECN propagation is yet to be defined for many tunnelling protocols. [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) defines how ECN should be propagated for IP-in-IP [RFC2003] (Perkins, C., “IP Encapsulation within IP,” October 1996.) and IPsec [RFC4301] (Kent, S. and K. Seo, “Security Architecture for the Internet Protocol,” December 2005.) tunnels. However, as Section 9.3 of RFC3168 pointed out, ECN support will need to be defined for other tunnelling protocols, e.g. L2TP [RFC2661] (Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G., and B. Palter, “Layer Two Tunneling Protocol "L2TP",” August 1999.), GRE [RFC1701 (Hanks, S., Li, T., Farinacci, D., and P. Traina, “Generic Routing Encapsulation (GRE),” October 1994.), RFC2784 (Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, “Generic Routing Encapsulation (GRE),” March 2000.)], PPTP [RFC2637] (Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, W., and G. Zorn, “Point-to-Point Tunneling Protocol,” July 1999.) and GTP [GTPv1 (3GPP, “GPRS Tunnelling Protocol (GTP) across the Gn and Gp interface,” .), GTPv1‑U (3GPP, “General Packet Radio System (GPRS) Tunnelling Protocol User Plane (GTPv1-U),” .), GTPv2‑C (3GPP, “Evolved General Packet Radio Service (GPRS) Tunnelling Protocol for Control plane (GTPv2-C),” .)].
The purpose of this document is to guide the addition of congestion notification to any subnet technology or tunnelling protocol, so that lower layer equipment can signal congestion explicitly and it will propagate consistently into encapsulated (higher layer) headers, otherwise the signals will not reach their ultimate destination.
Incremental deployment is the most tricky aspect when adding support for ECN. The original ECN protocol in IP [RFC3168] (Ramakrishnan, K., Floyd, S., and D. Black, “The Addition of Explicit Congestion Notification (ECN) to IP,” September 2001.) was carefully designed so that a congested buffer would not mark a packet (rather than drop it) unless both source and destination hosts were ECN-capable. Otherwise its congestion markings would never be detected and congestion would just deteriorate further. However, to support congestion marking below the IP layer, it is not sufficient to only check that the two end-points support ECN; correct operation also depends on the decapsulator at each subnet egress faithfully propagating congestion notifications to the higher layer. Otherwise, a legacy decapsulator might silently fail to propagate any ECN signals from the outer to the forwarded header. Then the lost signals would never be detected and again congestion would deteriorate further. The guidelines given later require protocol designers to carefully consider incremental deployment, and suggest various safe approaches for different circumstances.
Of course, the IETF does not have standards authority over every link layer protocol. So this document gives guidelines for designing propagation of congestion notification across the interface between IP and protocols that may encapsulate IP (i.e. that can be layered beneath IP). Each lower layer technology will exhibit different issues and compromises, so the IETF or the relevant standards body must be free to define the specifics of each lower layer congestion notification scheme. Nonetheless, if the guidelines are followed, congestion notification should interwork between different technologies, using IP in its role as a 'portability layer'.
Therefore, the capitalised term 'SHOULD' or 'SHOULD NOT' are often used in preference to 'MUST' or 'MUST NOT', because it is difficult to know the compromises that will be necessary in each protocol design. If a particular protocol design chooses to contradict a 'SHOULD (NOT)' given in the advice below, it MUST include a sound justification.
It has not been possible to give common guidelines for all lower layer technologies, because they do not all fit a common pattern. Instead they have been divided into a few distinct modes of operation: feed-forward-and-upward; feed-upward-and-forward; feed-backward; and null mode. These modes are described in Section 3 (Modes of Operation), then in the following sections separate guidelines are given for each mode.
This document updates the advice to subnetwork designers about ECN in Section 13 of [RFC3819] (Karn, P., Bormann, C., Fairhurst, G., Grossman, D., Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. Wood, “Advice for Internet Subnetwork Designers,” July 2004.).
TOC |
This document only concerns wire protocol processing of explicit notification of congestion and makes no changes or recommendations concerning algorithms for congestion marking or for congestion response (algorithm issues should be independent of the layer the algorithm operates in).
The question of congestion notification signals with different semantics to those of ECN in IP is touched on in a couple of specific cases (e.g. QCN [IEEE802.1Qau] (Finn, N., Ed., “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks - Amendment 13: Congestion Notification,” March 2010.)) and with schemes with multiple severity levels such as PCN [RFC6660] (Briscoe, B., Moncaster, T., and M. Menth, “Encoding Three Pre-Congestion Notification (PCN) States in the IP Header Using a Single Diffserv Codepoint (DSCP),” July 2012.)). However, no attempt is made to give guidelines about schemes with different semantics that are yet to be invented.
The semantics of congestion signals can be relative to the traffic class. Therefore correct propagation of congestion signals could depend on correct propagation of any traffic class field between the layers. In this document, correct propagation of traffic class information is assumed, while what 'correct' means and how it is achieved is covered elsewhere (e.g. [RFC2983] (Black, D., “Differentiated Services and Tunnels,” October 2000.)) and is outside the scope of the present document.
Note that these guidelines do not require the subnet wire protocol to be changed to accommodate congestion notification. Another way to add congestion notification without consuming header space in the subnet protocol might be to use a parallel control plane protocol.
This document focuses on the congestion notification interface between IP and lower layer protocols that can encapsulate IP, where the term 'IP' includes v4 or v6, unicast, multicast or anycast. However, it is likely that the guidelines will also be useful when a lower layer protocol or tunnel encapsulates itself (e.g. Ethernet MAC in MAC [IEEE802.1Qah] (IEEE, “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks—Amendment 6: Provider Backbone Bridges,” August 2008.)) or when it encapsulates other protocols. In the feed-backward mode, propagation of congestion signals for multicast and anycast packets is out-of-scope (because it would be so complicated that it is hoped no-one would attempt such an abomination).
TOC |
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.).
Further terminology used within this document:
- Protocol data unit (PDU):
- Information that is delivered as a unit among peer entities of a layered network consisting of protocol control information (typically a header) and possibly user data (payload) of that layer. The scope of this document includes layer 2 and layer 3 networks, where the PDU is respectively termed a frame or a packet (or a cell in ATM). PDU is a general term for any of these. This definition also includes a payload with a shim header lying somewhere between layer 2 & 3.
- Transport:
- The end-to-end transmission control function, conventionally considered at layer-4 in the OSI reference model. Given the audience for this document will often use the word transport to mean low level bit carriage, whenever the term is used it will be qualified, e.g. 'L4 transport'.
- Encapsulator:
- The link or tunnel endpoint function that adds an outer header to a PDU (also termed the 'link ingress', the 'subnet ingress', the 'ingress tunnel endpoint' or just the 'ingress' where the context is clear).
- Decapsulator:
- The link or tunnel endpoint function that removes an outer header from a PDU (also termed the 'link egress', the 'subnet egress', the 'egress tunnel endpoint' or just the 'egress' where the context is clear).
- Incoming header:
- The header of an arriving PDU before encapsulation.
- Outer header:
- The header added to encapsulate a PDU.
- Inner header:
- The header encapsulated by the outer header.
- Outgoing header:
- The header forwarded by the decapsulator.
- CE:
- Congestion Experienced [RFC3168] (Ramakrishnan, K., Floyd, S., and D. Black, “The Addition of Explicit Congestion Notification (ECN) to IP,” September 2001.)
- ECT:
- ECN-Capable Transport [RFC3168] (Ramakrishnan, K., Floyd, S., and D. Black, “The Addition of Explicit Congestion Notification (ECN) to IP,” September 2001.)
- Not-ECT:
- Not ECN-Capable Transport [RFC3168] (Ramakrishnan, K., Floyd, S., and D. Black, “The Addition of Explicit Congestion Notification (ECN) to IP,” September 2001.)
- ECN-PDU:
- A PDU that is part of a feedback loop within which all the nodes that need to propagate explicit congestion notifications back to the Load Regulator are ECN-capable. An IP packet with a non-zero ECN field implies that the endpoints are ECN-capable, so this would be an ECN-PDU. However, ECN-PDU is intended to be a general term for a PDU at any layer, not just IP.
- Not-ECN-PDU:
- A PDU that is part of a feedback-loop within which some nodes necessary to propagate explicit congestion notifications back to the load regulator are not ECN-capable.
- Load Regulator:
- For each flow of PDUs, the transport function that is capable of controlling the data rate. Typically located at the data source, but in-path nodes can regulate load in some congestion control arrangements (e.g. admission control or policing nodes). Note the term "a function capable of controlling the load" deliberately includes a transport that doesn't actually control the load but ideally it ought to (e.g. a sending application without congestion control that uses UDP).
- Congestion Baseline:
- The location of the function on the path that initialised the values of all congestion notification fields in a sequence of packets, before any are set to the congestion experienced (CE) codepoint if they experience congestion further downstream. Typically the original data source at layer-4.
TOC |
This section sets down the different modes by which congestion information is passed between the lower layer and the higher one. It acts as a reference framework for the following sections, which give normative guidelines for designers of explicit congestion notification protocols, taking each mode in turn:
- Feed-Forward-and-Up:
- Nodes feed forward congestion notification towards the egress within the lower layer then up and along the layers towards the end-to-end destination at the transport layer. The following local optimisation is possible:
- Feed-Up-and-Forward:
- A lower layer switch feeds-up congestion notification directly into the ECN field in the higher layer (e.g. IP) header, irrespective of whether the node is at the egress of a subnet.
- Feed-Backward:
- Nodes feed back congestion signals towards the ingress of the lower layer and (optionally) attempt to control congestion within their own layer.
- Null:
- Nodes cannot experience congestion at the lower layer except at ingress nodes (which are IP-aware or equivalently higher-layer-aware).
TOC |
Like IP and MPLS, many subnet technologies are based on self-contained protocol data units (PDUs) or frames sent unreliably. They provide no feedback channel at the subnetwork layer, instead relying on higher layers (e.g. TCP) to feed back loss signals.
In these cases, ECN may best be supported by standardising explicit notification of congestion into the lower layer protocol that carries the data forwards. It will then also be necessary to define how the egress of the lower layer subnet propagates this explicit signal into the forwarded upper layer (IP) header. It can then continue forwards until it finally reaches the destination transport (at L4). Then typically the destination will feed this congestion notification back to the source transport using an end-to-end protocol (e.g. TCP). This is the arrangement that has already been used to add ECN to IP-in-IP tunnels [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.), IP-in-MPLS and MPLS-in-MPLS [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.).
This mode is illustrated in Figure 1 (Feed-Forward-and-Up Mode). Along the middle of the figure, layers 2, 3 & 4 of the protocol stack are shown, and one packet is shown along the bottom as it progresses across the network from source to destination, crossing two subnets connected by a router, and crossing two switches on the path across each subnet. Congestion at the output of the first switch (shown as *) leads to a congestion marking in the L2 header (shown as C in the illustration of the packet). The chevrons show the progress of the resulting congestion indication. It is propagated from link to link across the subnet in the L2 header, then when the router removes the marked L2 header, it propagates the marking up into the L3 (IP) header. The router forwards the marked L3 header into subnet 2, and when it adds a new L2 header it copies the L3 marking into the L2 header as well, as shown by the 'C's in both layers (assuming the technology of subnet 2 also supports explicit congestion marking).
Note that there is no implication that each 'C' marking is encoded the same; a different encoding might be used for the 'C' marking in each protocol.
Finally, for completeness, we show the L3 marking arriving at the destination, where the host transport protocol (e.g. TCP) feeds it back to the source in the L4 acknowledgement (the 'C' at L4 in the packet at the top of the diagram).
_ _ _ /_______ | | |C| ACK Packet (V) \ |_|_|_| +---+ layer: 2 3 4 header +---+ | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 | | +---+ | ^ | | | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 | | +---+ +---+ | ^ | +---+ +---+ | | | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| source subnet A router subnet B dest __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ | | | | | | | | |C| | | |C| | | |C|C| Data________\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / layer: 4 3 2A 4 3 2A 4 3 4 3 2B header
Figure 1: Feed-Forward-and-Up Mode |
Of course, modern networks are rarely as simple as this text-book example, often involving multiple nested layers. For example, a 3GPP mobile network may have two IP-in-IP (GTP) tunnels in series and an MPLS backhaul between the base station and the first router. Nonetheless, the example illustrates the general idea of feeding congestion notification forward then upward whenever a header is removed at the egress of a subnet.
Note that the FECN (forward ECN) bit in Frame Relay and the explicit forward congestion indication (EFCI [ITU‑T.I.371] (ITU-T, “Traffic Control and Congestion Control in B-ISDN,” March 2004.)) bit in ATM user data cells follow a feed-forward pattern. However, in ATM, this is only as part of a feed-forward-and-backward pattern at the lower layer, not feed-forward-and-up out of the lower layer—the intention was never to interface to IP ECN at the subnet egress. To our knowledge, Frame Relay FECN is solely used to detect where more capacity should be provisioned [Buck00] (Buckwalter, J., “Frame Relay: Technology and Practice,” 2000.).
TOC |
Ethernet is particularly difficult to extend incrementally to support explicit congestion notification. One way to support ECN in such cases has been to use so called 'layer-3 switches'. These are Ethernet switches that bury into the Ethernet payload to find an IP header and manipulate or act on certain IP fields (specifically Diffserv & ECN). For instance, in Data Center TCP [DCTCP] (Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., and M. Sridharan, “Data Center TCP (DCTCP),” October 2010.), layer-3 switches are configured to mark the ECN field of the IP header within the Ethernet payload when their output buffer becomes congested. With respect to switching, a layer-3 switch acts solely on the addresses in the Ethernet header; it doesn't use IP addresses, and it doesn't decrement the TTL field in the IP header.
_ _ _ /_______ | | |C| ACK packet (V) \ |_|_|_| +---+ layer: 2 3 4 header +---+ | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 | | +---+ | ^ | | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 | | +--^+ +---+ | | +---+ +---+ | | | | | *| | | | | | | | | | |L2 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| source subnet E router subnet F dest __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ | | | | | | | |C| | | | |C| | | |C|C| data________\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / layer: 4 3 2 4 3 2 4 3 4 3 2 header
Figure 2: Feed-Up-and-Forward Mode |
By comparing Figure 2 (Feed-Up-and-Forward Mode) with Figure 1 (Feed-Forward-and-Up Mode), it can be seen that subnet E (perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and-forward mode by notifying congestion directly into L3 at the point of congestion, even though the congested switch does not otherwise act at L3. In this example, the technology in subnet F (e.g. MPLS) does support ECN natively, so when the router adds the layer-2 header it copies the ECN marking from L3 to L2 as well.
TOC |
In some layer 2 technologies, explicit congestion notification has been defined for use internally within the subnet with its own feedback and load regulation, but typically the interface with IP for ECN has not been defined.
For instance, for the available bit-rate (ABR) service in ATM, the relative rate mechanism was one of the more popular mechanisms for managing traffic, tending to supersede earlier designs. In this approach ATM switches send special resource management (RM) cells in both the forward and backward directions to control the ingress rate of user data into a virtual circuit. If a switch buffer is approaching congestion or congested it sends an RM cell back towards the ingress with respectively the No Increase (NI) or Congestion Indication (CI) bit set in its message type field [ATM‑TM‑ABR] (Cisco, “Understanding the Available Bit Rate (ABR) Service Category for ATM VCs,” June 2005.). The ingress then holds or decreases its sending bit-rate accordingly.
_ _ _ /_______ | | |C| ACK packet (X) \ |_|_|_| +---+ layer: 2 3 4 header +---+ | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 | | +---+ | ^ | | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 | | +---+ +---+ | | +---+ +---+ | | | | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2 | | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| source subnet G router subnet H dest __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ later | | | | | | | | | | | | | | | | |C| | data________\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) / 4 3 2 4 3 2 4 3 4 3 2 _ /__ |C| Feedback control \ |_| cell/frame (V) 2 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier | | | | | | | | | | | | | | | | | | | data________\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / layer: 4 3 2 4 3 2 4 3 4 3 2 header
Figure 3: Feed-Backward Mode |
ATM's feed-backward approach doesn't fit well when layered beneath IP's feed-forward approach—unless the initial data source is the same node as the ATM ingress. Figure 3 (Feed-Backward Mode) shows the feed-backward approach being used in subnet H. If the final switch on the path is congested (*), it doesn't feed-forward any congestion indications on packet (U). Instead it sends a control cell (V) back to the router at the ATM ingress.
However, the backward feedback doesn't reach the original data source directly because IP doesn't support backward feedback (and subnet G is independent of subnet H). Instead, the router in the middle throttles down its sending rate but the original data sources don't reduce their rates. The resulting rate mismatch causes the middle router's buffer at layer 3 to back up until it becomes congested, which it signals forwards on later data packets at layer 3 (e.g. packet W). Note that the forward signal from the middle router is not triggered directly by the backward signal. Rather, it is triggered by congestion resulting from the middle router's mismatched rate response to the backward signal.
In response to this later forward signalling, end-to-end feedback at layer-4 finally completes the tortuous path of congestion indications back to the origin data source, as before.
TOC |
Often link and physical layer resources are 'non-blocking' by design. In these cases congestion notification may be implemented but it does not need to be deployed at the lower layer; ECN in IP would be sufficient.
A degenerate example is a point-to-point Ethernet link. Excess loading of the link merely causes the queue from the higher layer to back up, while the lower layer remains immune to congestion. Even a whole meshed subnetwork can be made immune to interior congestion by limiting ingress capacity and careful sizing of links, particularly if multi-path routing is used to ensure even worst-case patterns of load cannot congest any link.
TOC |
Feed-forward-and-up is the mode already used for signalling ECN up the layers through MPLS into IP [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.) and through IP-in-IP tunnels [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.). These RFCs take a consistent approach and the following guidelines are designed to ensure this consistency continues as ECN support is added to other protocols that encapsulate IP. The guidelines are also designed to ensure compliance with the more general best current practice for the design of alternate ECN schemes given in [RFC4774] (Floyd, S., “Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field,” November 2006.).
The rest of this section is structured as follows:
TOC |
A common pattern for many tunnelling protocols is to encapsulate an inner IP header with shim header(s) then an outer IP header. In many cases the shim header(s) always have to be tightly coupled to the outer IP header because they are not sufficient as outer headers in their own right. In such cases the shim header(s) and the outer IP header are always added (or removed) in the same operation. Therefore, in all such tightly coupled IP-in-IP tunnelling protocols, the rules in [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) for propagating the ECN field between the two IP headers SHOULD be applied directly.
Examples of tightly coupled IP-in-IP tunnelling protocols where [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) can be applied directly are:
TOC |
This section is intended to guide the redesign of any lower layer protocol that encapsulate IP to add native ECN support at the lower layer. It reflects the approaches used in [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) and in [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.). Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) or [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.) will already satisfy this guidance.
A lower layer (or subnet) congestion notification system:
Note that there is no need for all interior nodes within a subnet to be able to mark congestion explicitly. A mix of ECN and drop signals from different nodes is fine. However, if any interior nodes might generate ECN markings, guideline 2 above says that all relevant egress node(s) SHOULD be able to propagate those markings up to the higher layer.
In IP, if the ECN field in each PDU is cleared to the Not-ECT (not ECN-capable transport) codepoint, it indicates that the L4 transport will not understand congestion markings. A congested buffer must not mark these Not-ECT PDUs, and therefore drops them instead.
The mechanism a lower layer uses to distinguish the ECN-capability of PDUs need not mimic that of IP. All the above guidelines say is that the lower layer system, as a whole, should achieve the same outcome. For instance, ECN-capable feedback loops might use PDUs that are identified by a particular set of labels or tags. Alternatively, logical link protocols that use flow state might determine whether a PDU can be congestion marked by checking for ECN-support in the flow state. Other protocols might depend on out-of-band control signals.
The per-domain checking of ECN support in MPLS [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.) is a good example of a way to avoid sending congestion markings to transports that will not understand them, without using any header space in the subnet protocol.
In MPLS, header space is extremely limited, therefore RFC5129 does not provide a field in the MPLS header to indicate whether the PDU is an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are allowed to set explicit congestion indications without checking whether the PDU is destined for a transport that will understand them. Nonetheless, this is made safe by requiring that the network operator upgrades all decapsulating edges of a whole domain at once, as soon as even one switch within the domain is configured to mark rather than drop during congestion. Therefore, any edge node that might decapsulate a packet will be capable of checking whether the higher layer transport is ECN-capable. When decapsulating a CE-marked packet, if the decapsulator discovers that the higher layer (inner header) indicates the transport is not ECN-capable, it drops the packet—effectively on behalf of the earlier congested node (see Decapsulation Guideline 1 in Section 4.4 (Decapsulation Guidelines)).
It was only appropriate to define such an incremental deployment strategy because MPLS is targeted solely at professional operators, who can be expected to ensure that a whole subnetwork is consistently configured. This strategy might not be appropriate for other link technologies targeted at zero-configuration deployment or deployment by the general public (e.g. Ethernet). For such 'plug-and-play' environments it will be necessary to invent a failsafe approach that ensures congestion markings will never fall into black holes, no matter how inconsistently a system is put together. Alternatively, congestion notification relying on correct system configuration could be confined to flavours of Ethernet intended only for professional network operators, such as IEEE 802.1ah Provider Backbone Bridges (PBB).
QCN [IEEE802.1Qau] (Finn, N., Ed., “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks - Amendment 13: Congestion Notification,” March 2010.) provides another example of how to indicate to lower layer devices that the end-points will not understand ECN. An operator can define certain 802.1p classes of service to indicate non-QCN frames and an ingress bridge is required to map arriving not-QCN-capable IP packets to one of these non-QCN 802.1p classes.
TOC |
This section is intended to guide the redesign of any node that encapsulates IP with a lower layer header when adding native ECN support to the lower layer protocol. It reflects the approaches used in [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) and in [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.). Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) or [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.) will already satisfy this guidance.
TOC |
This section is intended to guide the redesign of any node that decapsulates IP from within a lower layer header when adding native ECN support to the lower layer protocol. It reflects the approaches used in [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) and in [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.). Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) or [RFC5129] (Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” January 2008.) will already satisfy this guidance.
A subnet egress SHOULD NOT simply copy congestion notification from outer headers to the forwarded header. It SHOULD calculate the outgoing congestion notification field from the inner and outer headers using the following guidelines. If there is any conflict, rules earlier in the list take precedence over rules later in the list:
TOC |
In some deployments, particularly in 3GPP networks, an IP packet may traverse two or more IP-in-IP tunnels in sequence that all use identical technology (e.g. GTP).
In such cases, it would be sufficient for every encapsulation and decapsulation in the chain to comply with RFC 6040. Alternatively, as an optimisation, a node that decapsulates a packet and immediately re-encapsulates it for the next tunnel MAY copy the incoming outer ECN field directly to the outgoing outer and the incoming inner ECN field directly to the outgoing inner. Then the overall behavior across the sequence of tunnel segments would still be consistent with RFC 6040.
Appendix C of RFC6040 describes how a tunnel egress can monitor how much congestion has been introduced within a tunnel. A network operator might want to monitor how much congestion had been introduced within a whole sequence of tunnels. Using the technique in Appendix C of RFC6040 at the final egress, the operator could monitor the whole sequence of tunnels, but only if the above optimisation were used consistently along the sequence of tunnels, in order to make it appear as a single tunnel. Therefore, tunnel endpoint implementations SHOULD allow the operator to configure whether this optimisation is enabled.
When ECN support is added to a subnet technology, consideration SHOULD be given to a similar optimisation between subnets in sequence if they all use the same technology.
TOC |
The guidance in this section is worded in terms of framing boundaries, but it applies equally whether the protocol data units are frames, cells or packets.
Where framing boundaries are different between two layers, congestion indications SHOULD be propagated on the basis that a congestion indication on a PDU applies to all the octets in the PDU. On average, an encapsulator or decapsulator SHOULD approximately preserve the number of marked octets arriving and leaving (counting the size of inner headers, but not added encapsulating headers).
The next departing frame SHOULD be immediately marked even if only enough incoming marked octets have arrived for part of the departing frame. This ensures that any outstanding congestion marked octets are propagated immediately, rather than held back waiting for a frame no bigger than the outstanding marked octets—which might involve a long wait.
For instance, an algorithm for marking departing frames could maintain a counter representing the balance of arriving marked octets minus departing marked octets. It adds the size of every marked frame that arrives and if the counter is positive it marks the next frame to depart and subtracts its size from the counter. This will often leave a negative remainder in the counter, which is deliberate.
TOC |
The guidance in this section is primarily applicable to encapsulation of IP packets in Ethernet headers. However, it generalises to encapsulation by other subnet technologies with no native support for explicit congestion notification. It is unlikely to be applicable or necessary for IP-in-IP encapsulation, where feed-forward-and-up mode based on [RFC6040] (Briscoe, B., “Tunnelling of Explicit Congestion Notification,” November 2010.) would be more appropriate.
Marking the IP header while switching at layer-2 (by using a layer-3 switch) seems to represent a layering violation. However, it can be considered as a benign optimisation if the guidelines below are followed. Feed-up-and-forward is certainly not a general alternative to implementing feed-forward congestion notification in the lower layer, because:
Nonetheless, configuring a layer-3 switch to look for an ECN field in an encapsulated IP header is a useful optimisation. If the implementation follows the guidelines below, this optimisation does not have to be confined to a controlled environment such as within a data centre; it could usefully be applied on any network—even if the operator is not sure whether the above issues will never apply:
TOC |
It can be seen from Section 3.3 (Feed-Backward Mode) that congestion notification in a subnet using feed-backward mode has generally not been designed to be directly coupled with IP layer congestion notification. The subnet attempts to minimise congestion internally, and if the incoming load at the ingress exceeds the capacity somewhere through the subnet, the layer 3 buffer into the ingress backs up. Thus, a feed-backward mode subnet is in some sense similar to a null mode subnet, in that there is no need for any direct interaction between the subnet and higher layer congestion notification. Therefore no detailed protocol design guidelines are appropriate. Nonetheless, a more general guideline is appropriate:
The feed-backward approach at least works beneath IP, where the term 'works' is used only in a narrow functional sense because feed-backward can result in very inefficient and sluggish congestion control—except if it is confined to the subnet directly connected to the original data source, when it is faster than feed-forward. It would be valid to design a protocol that could work in feed-backward mode for paths that only cross one subnet, and in feed-forward-and-up mode for paths that cross subnets.
In the early days of TCP/IP, a similar feed-backward approach was tried for explicit congestion signalling, using source-quench (SQ) ICMP control packets. However, SQ fell out of favour and is now formally deprecated [RFC6633] (Gont, F., “Deprecation of ICMP Source Quench Messages,” May 2012.). The main problem was that it is hard for a data source to tell the difference between a spoofed SQ message and a quench request from a genuine buffer on the path. It is also hard for a lower layer buffer to address an SQ message to the original source port number, which may be buried within many layers of headers, and possibly encrypted.
Quantised congestion notification (QCN—also known as backward congestion notification or BCN) [IEEE802.1Qau] (Finn, N., Ed., “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks - Amendment 13: Congestion Notification,” March 2010.) uses a feed-backward mode structurally similar to ATM's relative rate mechanism. However, QCN confines its applicability to scenarios such as some data centres where all endpoints are directly attached by the same Ethernet technology. If a QCN subnet were later connected into a wider IP-based internetwork (e.g. when attempting to interconnect multiple data centres) it would suffer the inefficiency shown Figure 3 (Feed-Backward Mode).
TOC |
This memo includes no request to IANA.
TOC |
If a lower layer wire protocol is redesigned to include explicit congestion signalling in-band in the protocol header, care SHOULD be take to ensure that the field used is specified as mutable during transit. Otherwise interior nodes signalling congestion would invalidate any authentication protocol applied to the lower layer header—by altering a header field that had been assumed as immutable.
The redesign of protocols that encapsulate IP in order to propagate congestion signals between layers raises potential signal integrity concerns. Experimental or proposed approaches exist for assuring the end-to-end integrity of in-band congestion signals, e.g.:
Given these end-to-end approaches are already being specified, it would make little sense to attempt to design hop-by-hop congestion signal integrity into a new lower layer protocol, because end-to-end integrity inherently achieves hop-by-hop integrity.
TOC |
Following the guidance in the document enables ECN support to be extended to numerous protocols that encapsulate IP (v4 & v6) in a consistent way, so that IP continues to fulfil its role as an end-to-end interoperability layer. This includes:
Guidelines have been defined for supporting propagation of ECN between Ethernet and IP on so-called Layer-3 Ethernet switches, using a 'feed-up-an-forward' mode. This approach could enable other subnet technologies to pass ECN signals into the IP layer, even if they do not support ECN natively.
Finally, attempting to add ECN to a subnet technology in feed-backward mode is deprecated except in special cases, due to its likely sluggish response to congestion.
TOC |
Thanks to Gorry Fairhurst for extensive initial reviews. Michael Welzl pointed out that lower layer congestion notification signals may have different semantics to those in IP.
Bob Briscoe was part-funded by the European Community under its Seventh Framework Programme through the Trilogy project (ICT-216372) for initial drafts and through the Reducing Internet Transport Latency (RITE) project (ICT-317700) subsequently. The views expressed here are solely those of the authors.
TOC |
Comments and questions are encouraged and very welcome. They can be addressed to the IETF Transport Area working group mailing list <tsvwg@ietf.org>, and/or to the authors.
TOC |
TOC |
[RFC2119] | Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” BCP 14, RFC 2119, March 1997 (TXT, HTML, XML). |
[RFC3168] | Ramakrishnan, K., Floyd, S., and D. Black, “The Addition of Explicit Congestion Notification (ECN) to IP,” RFC 3168, September 2001 (TXT). |
[RFC3819] | Karn, P., Bormann, C., Fairhurst, G., Grossman, D., Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. Wood, “Advice for Internet Subnetwork Designers,” BCP 89, RFC 3819, July 2004 (TXT). |
[RFC4774] | Floyd, S., “Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field,” BCP 124, RFC 4774, November 2006 (TXT). |
[RFC5129] | Davie, B., Briscoe, B., and J. Tay, “Explicit Congestion Marking in MPLS,” RFC 5129, January 2008 (TXT). |
[RFC6040] | Briscoe, B., “Tunnelling of Explicit Congestion Notification,” RFC 6040, November 2010 (TXT). |
TOC |
[ATM-TM-ABR] | Cisco, “Understanding the Available Bit Rate (ABR) Service Category for ATM VCs,” Design Technote 10415, June 2005. |
[Buck00] | Buckwalter, J., “Frame Relay: Technology and Practice,” Pub. Addison Wesley ISBN-13: 978-0201485240, 2000. |
[DCTCP] | Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., and M. Sridharan, “Data Center TCP (DCTCP),” ACM SIGCOMM CCR 40(4)63--74, October 2010 (PDF). |
[GTPv1] | 3GPP, “GPRS Tunnelling Protocol (GTP) across the Gn and Gp interface,” Technical Specification TS 29.060. |
[GTPv1-U] | 3GPP, “General Packet Radio System (GPRS) Tunnelling Protocol User Plane (GTPv1-U),” Technical Specification TS 29.281. |
[GTPv2-C] | 3GPP, “Evolved General Packet Radio Service (GPRS) Tunnelling Protocol for Control plane (GTPv2-C),” Technical Specification TS 29.274. |
[I-D.ietf-conex-abstract-mech] | Mathis, M. and B. Briscoe, “Congestion Exposure (ConEx) Concepts and Abstract Mechanism,” draft-ietf-conex-abstract-mech-07 (work in progress), July 2013 (TXT). |
[I-D.moncaster-tcpm-rcv-cheat] | Moncaster, T., “A TCP Test to Allow Senders to Identify Receiver Non-Compliance,” draft-moncaster-tcpm-rcv-cheat-01 (work in progress), June 2007 (TXT). |
[IEEE802.1Qah] | IEEE, “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks—Amendment
6: Provider Backbone Bridges,” IEEE Std 802.1Qah-2008, August 2008 (HTML). (Access Controlled link within page) |
[IEEE802.1Qau] | Finn, N., Ed., “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks - Amendment 13: Congestion Notification,” IEEE Std 802.1Qau-2010, March 2010 (PDF). (Access Controlled link within page) |
[ITU-T.I.371] | ITU-T, “Traffic Control and Congestion Control in B-ISDN,” ITU-T Rec. I.371 (03/04), March 2004. |
[RFC1323] | Jacobson, V., Braden, B., and D. Borman, “TCP Extensions for High Performance,” RFC 1323, May 1992 (TXT). |
[RFC1701] | Hanks, S., Li, T., Farinacci, D., and P. Traina, “Generic Routing Encapsulation (GRE),” RFC 1701, October 1994 (TXT). |
[RFC2003] | Perkins, C., “IP Encapsulation within IP,” RFC 2003, October 1996 (TXT, HTML, XML). |
[RFC2637] | Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, W., and G. Zorn, “Point-to-Point Tunneling Protocol,” RFC 2637, July 1999 (TXT). |
[RFC2661] | Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G., and B. Palter, “Layer Two Tunneling Protocol "L2TP",” RFC 2661, August 1999 (TXT). |
[RFC2784] | Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, “Generic Routing Encapsulation (GRE),” RFC 2784, March 2000 (TXT). |
[RFC2884] | Hadi Salim, J. and U. Ahmed, “Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks,” RFC 2884, July 2000 (TXT). |
[RFC2983] | Black, D., “Differentiated Services and Tunnels,” RFC 2983, October 2000 (TXT). |
[RFC3540] | Spring, N., Wetherall, D., and D. Ely, “Robust Explicit Congestion Notification (ECN) Signaling with Nonces,” RFC 3540, June 2003 (TXT). |
[RFC4301] | Kent, S. and K. Seo, “Security Architecture for the Internet Protocol,” RFC 4301, December 2005 (TXT). |
[RFC6633] | Gont, F., “Deprecation of ICMP Source Quench Messages,” RFC 6633, May 2012 (TXT). |
[RFC6660] | Briscoe, B., Moncaster, T., and M. Menth, “Encoding Three Pre-Congestion Notification (PCN) States in the IP Header Using a Single Diffserv Codepoint (DSCP),” RFC 6660, July 2012 (TXT). |
[trill-rbridge-options] | Eastlake, D., Ghanwani, A., Manral, V., and C. Bestler, “RBridges: Further TRILL Header Extensions,” draft-ietf-trill-rbridge-options-07 (work in progress), June 2012 (TXT). |
[vxlan] | Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingam-dutt-dcops-vxlan-04 (work in progress), May 2013 (TXT). |
TOC |
TOC |
- From briscoe-02 to 03:
- Scope section:
- Added dependence on correct propagation of traffic class information
- For the feed-backward mode, deemed multicast and anycast out of scope
- Ensured all guidelines referring to subnet technologies also refer to tunnels and vice versa by adding applicability sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and 5.
- Added Security Considerations on ensuring congestion signal fields are classed as immutable and on using end-to-end congestion signal integrity technologies rather than hop-by-hop.
- From briscoe-01 to 02:
- Added authors: JK & PT
- Added
- Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim Headers"
- Section 4.5 "Sequences of Similar Tunnels or Subnets"
- roadmap at the start of Section 4, given the subsections have become quite fragmented.
- Section 9 "Conclusions"
- Clarified why transports are starting to be able to saturate interior links
- Under Section 1.1, addressed the question of alternative signal semantics and included multicast & anycast.
- Under Section 3.1, included a 3GPP example.
- Section 4.2. "Wire Protocol Design":
- Altered guideline 2. to make it clear that it only applies to the immediate subnet egress, not later ones
- Added a reminder that it is only necessary to check that ECN propagates at the egress, not whether interior nodes mark ECN
- Added example of how QCN uses 802.1p to indicate support for QCN.
- Added references to Appendix C of RFC6040, about monitoring the amount of congestion signals introduced within a tunnel
- Appendix A: Added more issues to be addressed, including plan to produce a standards track update to IP-in-IP tunnel protocols.
- Updated acks and references
- From briscoe-00 to 01:
- Intended status: BCP (was Informational) & updates 3819 added.
- Briefer Introduction: Introductory para justifying benefits of ECN. Moved all but a brief enumeration of modes of operation to their own new section (from both Intro & Scope). Introduced incr. deployment as most tricky part.
- Tightened & added to terminology section
- Structured with Modes of Operation, then Guidelines section for each mode.
- Tightened up guideline text to remove vagueness / passive voice / ambiguity and highlight main guidelines as numbered items.
- Added Outstanding Document Issues Appendix
- Updated references
TOC |
Bob Briscoe | |
BT | |
B54/77, Adastral Park | |
Martlesham Heath | |
Ipswich IP5 3RE | |
UK | |
Phone: | +44 1473 645196 |
EMail: | bob.briscoe@bt.com |
URI: | http://bobbriscoe.net/ |
John Kaippallimalil | |
Huawei | |
5340 Legacy Drive, Suite 175 | |
Plano, Texas 75024 | |
USA | |
EMail: | john.kaippallimalil@huawei.com |
Pat Thaler | |
Broadcom Corporation | |
5025 Keane Drive | |
Carmichael, CA 95608 | |
USA | |
EMail: | pthaler@broadcom.com |