< draft-ietf-conex-concepts-uses-01.txt   draft-ietf-conex-concepts-uses-02a.txt >
CONEX T. Moncaster, Ed. ConEx B. Briscoe, Ed.
Internet-Draft Moncaster Internet Consulting Internet-Draft BT
Intended status: Informational J. Leslie, Ed. Intended status: Informational R. Woundy, Ed.
Expires: September 15, 2011 JLC.net Expires: January 8, 2012 Comcast
B. Briscoe July 07, 2011
BT
R. Woundy
Comcast
D. McDysan
Verizon
March 14, 2011
ConEx Concepts and Use Cases ConEx Concepts and Use Cases
draft-ietf-conex-concepts-uses-01 draft-ietf-conex-concepts-uses-02
Abstract Abstract
Internet Service Providers (operators) are facing problems where Internet Service Providers (operators) are facing problems where
localized congestion prevents full utilization of the path between localized congestion prevents full utilization of the path between
sender and receiver at today's "broadband" speeds. Operators desire sender and receiver at today's "broadband" speeds. Operators desire
to control this congestion, which often appears to be caused by a to control this congestion, which often appears to be caused by a
small number of users consuming a large amount of bandwidth. small number of users consuming a large amount of bandwidth.
Building out more capacity along all of the path to handle this Building out more capacity along all of the path to handle this
congestion can be expensive and may not result in improvements for congestion can be expensive and may not result in improvements for
skipping to change at page 2, line 4 skipping to change at page 1, line 43
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 15, 2011.
This Internet-Draft will expire on January 8, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 6 2. Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Congestion Management . . . . . . . . . . . . . . . . . . . . 7 2.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 6
3.1. Existing Approaches . . . . . . . . . . . . . . . . . . . 7 2.2. Non-Goals of ConEx and Common Misconceptions . . . . . . . 8
4. Exposing Congestion . . . . . . . . . . . . . . . . . . . . . 9 3. Traffic Management . . . . . . . . . . . . . . . . . . . . . . 9
4.1. ECN - a Step in the Right Direction . . . . . . . . . . . 10 3.1. Existing Approaches . . . . . . . . . . . . . . . . . . . 9
5. ConEx Use Cases . . . . . . . . . . . . . . . . . . . . . . . 10 4. Exposing Congestion . . . . . . . . . . . . . . . . . . . . . 10
5.1. ConEx as a basis for traffic management . . . . . . . . . 10 4.1. ECN - a Step in the Right Direction . . . . . . . . . . . 12
5.2. ConEx to incentivise scavenger transports . . . . . . . . 11 5. ConEx Use Cases . . . . . . . . . . . . . . . . . . . . . . . 12
5.3. Accounting for Congestion Volume . . . . . . . . . . . . . 12 5.1. Inform the Operator's Traffic Management . . . . . . . . . 12
5.4. ConEx as a form of differential QoS . . . . . . . . . . . 12 5.2. Consequence: Incentivise Scavenger Transports . . . . . . 13
5.5. Partial vs. Full Deployment . . . . . . . . . . . . . . . 13 5.3. Consequence: User-Controlled Intra-Class Quality of
6. Statistical Multiplexing over Differing Timescales . . . . . . 14 Service (QoS) . . . . . . . . . . . . . . . . . . . . . . 14
6.1. ConEx Objectives for This Issue . . . . . . . . . . . . . 15 5.4. Other Use-Cases . . . . . . . . . . . . . . . . . . . . . 15
6.2. ConEx as a Solution . . . . . . . . . . . . . . . . . . . 16 6. Deployment Arrangements . . . . . . . . . . . . . . . . . . . 15
6.3. Additional Support Using other Measures and Mechanisms . . 16 6.1. Incremental Deployment Features of the Protocol
7. Other issues . . . . . . . . . . . . . . . . . . . . . . . . . 17 Mechanism . . . . . . . . . . . . . . . . . . . . . . . . 15
7.1. Congestion as a Commercial Secret . . . . . . . . . . . . 17 6.2. Per-Network Deployment Concepts . . . . . . . . . . . . . 16
7.2. Information Security . . . . . . . . . . . . . . . . . . . 18 6.3. Single Receiving Network Scenario . . . . . . . . . . . . 17
8. Security Considerations . . . . . . . . . . . . . . . . . . . 18 6.4. Other Initial Deployment Scenarios . . . . . . . . . . . . 18
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 7. Potential Issues or Non-Issues . . . . . . . . . . . . . . . . 18
10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19 7.1. Congestion as a Commercial Secret . . . . . . . . . . . . 18
11. Informative References . . . . . . . . . . . . . . . . . . . . 20 7.2. Self Congestion . . . . . . . . . . . . . . . . . . . . . 19
8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
8.1. Information Security . . . . . . . . . . . . . . . . . . . 20
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20
10. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 20
11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22
11.1. Contributors . . . . . . . . . . . . . . . . . . . . . . . 22
12. Informative References . . . . . . . . . . . . . . . . . . . . 23
Appendix A. Changes from previous drafts (to be removed by
the RFC Editor) . . . . . . . . . . . . . . . . . . . 24
1. Introduction 1. Introduction
The growth of "always on" broadband connections, coupled with the The power of Internet technology comes from multiplexing shared
steady increase in access speeds, have caused unforeseen problems for capacity with packets rather than circuits. Network operators
network operators and users alike. Users are increasingly seeing usually provide sufficient shared capacity, but whenever too much
congestion at peak times and changes in usage patterns (with the packet load meets too little shared capacity, congestion results.
growth of real-time streaming) simply serve to exacerbate this. Congestion appears as either increased delay, missing packets or
Operators want all their users to see a good service but are unable packets explicitly marked with ECN [RFC3168]. Referring to Figure 1,
to see where congestion problems originate. But congestion results congestion control currently relies on the transport receiver
from sharing network capacity with others, not merely from using it. detecting these 'Congestion Signals' and informing the transport
In general, today's "DSL" and cable-internet users cannot "cause" sender in 'Congestion Feedback Signals'. The sender is then meant to
congestion in the absence of competing traffic. (Wireless operators reduce its rate in response.
and cellular internet have different tradeoffs which we will not
discuss here.)
Despite its central role in network control and management,
congestion is a remarkably hard conept to define. The discussions in
[Bauer09] provide a good academic background. [RFC6077] defines it
as "as a state or condition that occurs when network resources are
overloaded, resulting in impairments for network users as objectively
measured by the probability of loss and/or delay." An economist
might define it as the condition where the utility of a given user
decreases due to an increase in network load. Common to these
definitions is the idea that an increase in load results in a
reduction of service from the network.
Congestion takes two distinct forms. The first results from the
interaction of traffic from one set of users with traffic from other
users, causing in a reduction in service (a cost) for all of them.
the second, often referred to as "self-congestion", occurs when an
increase in traffic from a single user causes that user to suffer a
worse service (for instance because their traffic is being "shaped"
by their ISP, or because they have an excessively large buffer in
their home router). ConEx is principally interested in the first
form of congestion since it involves informing those other users of
the impact you expect to have on them.
While building out more capacity to handle increased traffic is
always good, the expense and lead-time can be prohibitive, especially
for network operators that charge flat-rate feeds to subscribers and
are thus unable to charge heavier users more for causing more
congestion [BB-incentive]. The operators also face the challenge
that network traffic grows according to Moore's Law -- increasing
capacity may only be buying a few months grace before you are again
facing increasing congestion, reducing utility and customers
demanding a better service. For an operator facing congestion caused
by other operators' networks, building out its own capacity is
unlikely to solve the congestion problem. Operators are thus facing
increased pressure to find effective solutions to dealing with the
increasing bandwidth demands of all users.
The growth of "scavenger" behaviour (e.g. [LEDBAT]) helps to reduce
congestion, but can actually make the problem less tractable. These
users are trying to make good use of the capacity of the path while
minimising their own costs. Thus, users of such services may show
very heavy total traffic up until the moment congestion is detected
(at the Transport Layer), but then will immediately back off.
Monitoring (at the Internet Layer) cannot detect this congestion
avoidance if the congestion in question is in a different domain
further along the path and hence such users may get tretated as
congestion-causing users.
The ConEx working group proposes that Internet Protocol (IP) packets
will carry additional ConEx information. The exact protocol details
are not described in this document, but the ConEx information will be
sufficient to allow any node in the network to see how much
congestion is attributable to a given traffic flow. See
[ConEx-Abstract-Mech] for further details.
Changes from previous drafts (to be removed by the RFC Editor):
From draft-ietf-conex-concepts-uses-00 to -01:
Added section on timescales: Section 6
Revised introduction to clarify congestion definitions
Changed source for congestion definition in Section 2 This document provides the entry point to the set of ConEx
documentation. It motivates a new congestion exposure (ConEx) field
at the IP layer, focusing on the 'why' rather than the 'how'. A
companion document about the ConEx protocol mechanism gives the 'how'
[ConEx-Abstract-Mech]. Briefly, the idea is for the sender to
continually signal expected congestion in the headers of any data it
sends. To a first approximation the sender does this by relaying the
'Congestion Feedback Signals' back into the IP layer. They then
travel unchanged across the network to the receiver (shown as 'IP-
Layer-ConEx-Signals' in Figure 1). Certain IP layer devices can then
use this new information, for example as input to traffic management.
Other minor changes ,---------. ,---------.
|Transport| |Transport|
| Sender | . |Receiver |
| | /|___________________________________________| |
| ,-<---------------Congestion-Feedback-Signals--<--------. |
| | |/ | | |
| | |\ Transport Layer Feedback Flow | | |
| | | \ ___________________________________________| | |
| | | \| | | |
| | | ' ,-----------. . | | |
| | |_____________| |_______________|\ | | |
| | | IP Layer | | Data Flow \ | | |
| | | |(Congested)| \ | | |
| | | | Network |--Congestion-Signals--->-' |
| | | | Device | \| |
| | | | | /| |
| `----------->--(new)-IP-Layer-ConEx-Signals-------->| |
| | | | / | |
| |_____________| |_______________ / | |
| | | | |/ | |
`---------' `-----------' ' `---------'
From draft-moncaster-conex-concepts-uses-02 to Figure 1: Where the ConEx Protocol Fits in the Internet Architecture
draft-ietf-conex-concepts-uses-00 (per decisions of working group): Current traffic management solutions limit traffic based on either
bit-rate or volume. For instance, weighted fair queuing (based on
[RFC0970]) shares out bit-rate when a link is congested. However it
fails to consider how much of the time a consumer keeps the link
busy, which is the main factor that determines everyone else's bit-
rate. To try to address this issue, many network operators have
introduced volume limits. However, these tend to be either not
strict enough during congested periods, or unnecessarily strict when
traffic is light.
Removed section on DDoS mitigation use case. Because each solution only partially addresses the problem, operators
keep adding more solutions. So a data path across the Internet often
encounters numerous blockages and throttles in each network it
crosses. This clutter is actually a symptom of a deeper underlying
problem: neither bit-rate nor volume is the right metric.
Removed appendix on ConEx Architectural Elements. PLEASE NOTE: Traffic management would be so much better (and simpler) if
Alignment of terminology with the Abstract Mechanism draft has congestion were visible in packets. Then, whenever congestion was
been deferred to the next version. not present, all restraints could be removed, leaving the full
capacity available to everyone. But if some excessive users were
causing a lot of congestion, the traffic management function would
know and be able to directly limit their traffic, in order to protect
the service of other users sharing the same capacity.
From draft-moncaster-conex-concepts-uses-01 to ConEx exposes exactly the right information for this, in the IP
draft-moncaster-conex-concepts-uses-02: layer. It reveals a consumer's overall contribution to congestion,
which is a direct measure of how much one user affects others. ConEx
makes this easy to measure--as easy as counting straight volume,
except you only count the volume of packets with ConEx markings.
With the right metric visible, traffic management would only have to
be done once on a path because it would be done well.
Updated document to take account of the new Abstract Mechanism In the absence of the right metric, some operators have deployed deep
draft [ConEx-Abstract-Mech]. packet inspection (DPI) equipment; not just in the public Internet
but also in enterprise and campus networks. Their main aim has been
to identify and limit specific types of application that they
associate with heavy usage.
Updated the definitions section. With ConEx, a network operator can manage traffic without dipping
into the higher layers, because ConEx makes the relevant bulk
congestion information accessible at the IP layer. This solves two
problems that have made DPI controversial: traffic management can be
application-neutral and compatible with IPsec encryption.
Removed sections on Requirements and Mechanism. Also, because ConEx information is added explicitly at the IP layer,
it is visible to provider and consumer alike. Therefore traffic
contracts or acceptable use policies can be based on a quantifiable
metric that is open and transparent to both parties, so that it will
be sufficient to manage traffic without extra non-transparent
wriggle-room and caveats.
Moved section on ConEx Architectural Elements to appendix. To summarise so far, ConEx is designed to make it simple to do
traffic management that is transparent, neutral and free of
unnecessary restraints. Although it is not our place to make a
network provider meet these requirements, ConEx is designed to make
this the simplest service to provide.
Minor changes throughout. {ToDo: review this para when done and shorten} This introduction has
focused on traffic management as the main use for ConEx. However,
ConEx is intended as a generative technology, with a wider range of
potential uses. The document structure reflects this. Section 3
overviews existing approaches to traffic management and Section 4
explains why exposing congestion would address their limitations.
Section 5 introduces the main use-cases for ConEx: traffic
management, incentivising scavenger transports and intra-class
quality of service as well as briefly mentioning others. Then
Section 6 presents how how the above use-cases might drive deployment
of ConEx {ToDo: Trying desperately to meet the charter requirement
about deployment without adding an unrelated section.} Finally,
Section 7 discusses a number of potential issues (and non-issues)
that are often raised about ConEx, before the usual tailpiece
sections in conclusion.
From draft-moncaster-conex-concepts-uses-00 to But before all this, Section 2 introduces the basic concepts
draft-moncaster-conex-concepts-uses-01: necessary to understand ConEx, as well as dispelling a number of
common misconceptions.
Changed end of Abstract to better reflect new title 2. Concepts
Created new section describing the architectural elements of Despite its central role in network control and management,
ConEx. Added Edge Monitors and Border Monitors (other elements congestion is a remarkably hard concept to define. [Bauer09]
are Ingress, Egress and Border Policers). provides a good academic background. For the purposes of ConEx, the
definition below focuses on how congestion would be measured, rather
than precisely what it is. Our definition of congestion is then
equivalent to the loss probability (or the marking probability if ECN
is used).
Extensive re-write of Section 5 partly in response to suggestions ConEx is essentially about accountability for congestion (or blame in
from Dirk Kutscher crude language). The blame for congestion lies equally between too
little capacity and too much traffic. On the capacity side,
congestion itself measures how badly the network provider is to
blame. On the traffic side, in a shared network, the blame is split
among all the users. Congestion-volume measures how much of each
user's traffic is to blame for the congestion. Note that congestion-
volume is a property of traffic, whereas congestion is a property of
a link or a path.
Improved layout of Section 2 and added definitions of Whole Path Congestion-volume is a relatively newly defined metric that is
Congestion, ConEx-Enabled and ECN-Enabled. Re-wrote definition of central to ConEx. To grasp an intuitive feel for what congestion-
Congestion Volume. Renamed Ingress and Egress Router to Ingress volume measures, some network operators give you an allowance and
and Egress Node as these nodes may not actually be routers. only count data volume during the peak period against it. This is
equivalent to counting your congestion-volume by assuming that
congestion is 100% during the peak period and 0% otherwise.
Improved document structure. Merged sections on Exposing The congestion-volume metric is more refined but broadly equivalent,
Congestion and ECN. and still as simple to measure as the above time-of-day volume.
Imagine Alice sends 1GB while the loss-probability is a constant
0.2%. Then her contribution to congestion (or congestion-volume) is
1GB x 0.2% = 2MB. If she then sends 3GB while congestion is 0.1%,
this adds 3MB to her congestion-volume. Her total contribution to
congestion is then 2MB+3MB = 5MB.
Added new section on ConEx requirements with a ConEx Issues To measure Alice's congestion-volume no-one has to do all these
subsection. Text for these came from the start of the old ConEx multiplications and additions. It is simply a matter of counting the
Use Cases section total volume of Alice's traffic that was discarded (a queue with a
percentage loss involves multiplication inherently).
Added a sub-section on Partial vs Full Deployment Section 5.5 Finally, there is yet another way to cut the blame for congestion.
Recall that the level of congestion itself measures the provider's
blame. However, in an internetwork there are multiple providers. If
a data centre network with zero congestion is connected to an access
network and sends traffic over a link with 0.4% loss probability,
then clearly all the blame for congestion lies with the access
network. However, at another time, there might be 0.1% congestion
across the data centre and 0.7% across the access, making 0.8% end-
to-end congestion across the path.
Added a discussion on ConEx as a Business Secret Section 7.1 In order to apportion blame accordingly, ConEx is designed so that a
meter at the border can see how much of the congestion is on one side
or other, termed upstream and downstream congestion.
From draft-conex-mechanism-00 to If A and B are connected within a chain of more than two networks, A
draft-moncaster-conex-concepts-uses-00: attributes all congestion beyond B to B, and vice versa. As far as A
is concerned, B chooses who to route to, so B takes responsibility
for its choices.
Changed filename to draft-moncaster-conex-concepts-uses. 2.1. Definitions
Changed title to ConEx Concepts and Use Cases. Congestion: Congestion occurs when any user's traffic suffers
increased delay, loss or ECN marking as a result of one or more
network resources becoming overloaded. For the purposes of ConEx,
the focus is on concrete signals of congestion (ECN and loss),
therefore delay is set to one side. Congestion is measured as the
probability of loss or the probability of ECN marking, usually
expressed as dimensionless percentages.
Chose uniform capitalisation of ConEx. Congestion-volume: For any granularity of traffic (packet, flow,
aggregate, link, etc.), the volume of bytes dropped or marked in a
given period of time. Conceptually, data volume multiplied by the
instantaneous congestion each packet of the volume experienced.
Usually expressed in bytes (or MB, GB, of course).
Moved definition of Congestion Volume to list of definitions. Congestion-rate: For any granularity of traffic (packet, flow,
aggregate, link, etc.), the instantaneous rate of traffic
discarded or marked due to congestion. Conceptually, the
instantaneous bit-rate of the traffic weighted by the
instantaneous congestion it experiences. Usually expressed in
b/s.
Clarified mechanism section. Changed section title. Upstream Congestion: The accumulated level of congestion experienced
by a traffic flow thus far, relative to a point along its path.
In other words, at any point the Upstream Congestion is the
accmulated level of congestion the traffic flow has experienced as
it travels from the sender to that point. At the receiver this is
equivalent to the end-to-end congestion level that (usually) is
reported back to the sender.
Modified text relating to conex-aware policing and policers (which Downstream Congestion: The level of congestion a flow of traffic is
are NOT defined terms). expected to experience on the remainder of its path. In other
words, at any point the Downstream Congestion is the level of
congestion the traffic flow is yet to experience as it travels
from that point to the receiver. Aka. Rest-of-Path Congestion.
Re-worded bullet on distinguishing ConEx and non-ConEx traffic in Consumer: A general term representing the contractual entity such as
Section 5. a user or business or institution that uses the service of a
network provider. There is no implication that the contract has
to be commercial; for instance the consumers of a University or
enterprise network service would be students or employees and they
might well be required to comply with some form of contract or
acceptable use policy. There is also no implication that the
consumer is necessarily an end-consumer. Where two networks form
a customer-provider relationship, the term consumer is also
intended to cover the customer network.
2. Definitions Network provider: (also 'operator'): the term is not confined to
Internet service providers (ISPs) who run commercial public
networks but is also intended to generalise to operators of
enterprise, campus and other networks.
In this section we define a number of terms that are used throughout [ConEx-Abstract-Mech] gives further definitions for aspects of ConEx
the document. The key definition is that of congestion, which has a to do with protocol mechanisms.
number of meanings depending on context. The definition we use in
this document is based on the definition in [RFC6077]. This list of
definitions is supplementary to that in [ConEx-Abstract-Mech].
Congestion: Congestion occurs when any user's traffic suffers 2.2. Non-Goals of ConEx and Common Misconceptions
increased delay, loss or ECN marking as a result of one or more
network resources being overloaded.
Flow: a series of packets from a single sender to a single receiver Congestion exposure is about the transport sender exposing congestion
that are treated by that sender as belonging to a single stream to the network, not the other way round. That would not make sense
for the purposes of congestion control. NB in general this is not given by design the transport endpoints handle congestion in the
the same as the aggregate of all traffic between the sender and TCP/IP protocol suite.
receiver.
Congestion-rate: For any granularity of traffic (packet, flow, Nonetheless, it is a non-goal for IP layer devices to use this ConEx
aggregate, etc.), the instantaneous rate of traffic discarded or information to do fine-grained congestion control. That is still
marked due to congestion. Conceptually, the instantaneous bit- best done at the transport sender. There is also no expectation that
rate of the traffic multiplied by the instantaneous congestion it the information will be used by every IP router and forwarding
is experiencing. device. More likely, specific ConEx-based functions like traffic
management will be added to edge routers. This in turn should
incentivize end-system transports to be more careful about congesting
others.
Congestion-volume: For any granularity of traffic (packet, flow, Note that good behaviour at individual flow granularity is the
aggregate, etc.), the volume of bytes dropped or marked in a given intended outcome, not the forcing function--it is the end, not the
period of time. Conceptually, congestion-rate multipled by time. means. Enforcing per-flow compliance to the TCP congestion response
(or any per-flow rate enforcement) is a non-goal:
Upstream Congestion: the accumulated level of congestion experienced o It used to be commonly believed that TCP-friendliness was critical
by a traffic flow thus far along its path. In other words, at any to fairness on the Internet. However, a particular congestion
point the Upstream Congestion is the accmulated level of control algorithm doesn't determine how much data an application
congestion the traffic flow has experienced as it travels from the transfers, or how many flows it opens, or which congestion control
sender to that point. At the receiver this is equivalent to the algorithm an application uses, or how many applications the user
end-to-end congestion level that (usually) is reported back to the runs, or how many users are active at once. These factors all
sender. have a stronger impact on how great a share of a link is available
for any particular data transfer. To achieve fairness at this
broader granularity (between users, homes, sites or whole
networks) requires that individual flows in the same bottleneck
will sometimes be very unequal.
Downstream Congestion: the level of congestion a flow of traffic is o If the network forced everything to be TCP-friendly, some
expected to experience on the remainder of its path. In other important applications would not work. Worse still, this would
words, at any point the Downstream Congestion is the level of prevent deployment of higher performance replacements to TCP. It
congestion the traffic flow is yet to experience as it travels is important to allow give and take between one user's flows: some
from that point to the receiver. can be more aggressive than TCP and others less, e.g. long
transfers, following the example of LEDBAT [LEDBAT] (see
Section 5.2).
Ingress: the first node a packet traverses that is outside the Therefore, network enforcement of per-flow fairness is not only a
source's own network. In a domestic network that will be the non-goal, it would be a harmful goal in many respects.
first node downstream from the home access equipment. In an
enterprise network this is the provider edge router.
Egress: the last node a packet traverses before reaching the Capacity provisioning in relation to ConEx is another area of
receiver's network. confusion. Congestion-based traffic management is not an alternative
to good capacity provisioning. Either or both can be good practice
depending on the situation, and ConEx can provide a useful metric for
both (see Section 5.4).
ConEx-enabled: Any piece of equipment (end-system, router, tunnel However, removing congestion from _all_ parts of a network is
end-point, firewall, policer, etc) that complies with the core unlikely to be a goal. If an end-to-end transport protocol cannot go
ConEx protocol, which is to be defined by the ConEx working group. fast enough to cause congestion somewhere along the path, it is
By extension a ConEx-enabled network is a network whose edge nodes probably broken. A good transport protocol will continue to
are all ConEx-enabled. accelerate until it encounters a bottleneck--typically in an access
network (or all too often in the receiver's buffer, but that's
another story). Low levels of congestion _somewhere_ are therefore
healthy. Just to be clear though, zero congestion somewhere (e.g.
the core) is also perfectly healthy.
3. Congestion Management 3. Traffic Management
Since 1988 the Internet architecture has made congestion management Since 1988 the Internet architecture has made congestion management
the responsibility of the end-systems. The network signals the responsibility of the end-systems. The network signals
congestion to the receiver, the receiver feeds this back to the congestion to the receiver, the receiver feeds this back to the
sender and the sender is expected to try and reduce the traffic it sender and the sender is expected to try and reduce the traffic it
sends. sends. {ToDo: FQ (1985) pre-dated TCP congestion avoidance.}
Any network that is persistently highly congested is inefficient. Any network that is persistently highly congested is inefficient.
However the total absence of congestion is equally bad as it means However the total absence of congestion is equally bad as it means
there is spare capacity in the network that hasn't been used. The there is spare capacity in the network that hasn't been used. The
long-standing aim of congestion control has been to find the point long-standing aim of congestion control has been to find the point
where these two things are in balance. where these two things are in balance.
Over recent years, some network operators have come to the view that Over recent years, some network operators have come to the view that
end-system congestion management is insufficient. Because of the end-system congestion management is insufficient. Because of the
heuristics used by TCP, a relatively small number of end-machines can heuristics used by TCP, a relatively small number of end-machines can
skipping to change at page 9, line 23 skipping to change at page 11, line 15
over time -- congestion means that your traffic impacts other users, over time -- congestion means that your traffic impacts other users,
and conversely that their traffic impacts you. So if there is no and conversely that their traffic impacts you. So if there is no
congestion there need not be any restriction on the amount a user can congestion there need not be any restriction on the amount a user can
send; restrictions only need to apply when others are sending traffic send; restrictions only need to apply when others are sending traffic
such that there is congestion. such that there is congestion.
For example, an application intending to transfer large amounts of For example, an application intending to transfer large amounts of
data could use a congestion control mechanism like [LEDBAT] to reduce data could use a congestion control mechanism like [LEDBAT] to reduce
its transmission rate before any competing TCP flows do, by detecting its transmission rate before any competing TCP flows do, by detecting
an increase in end-to-end delay (as a measure of impending an increase in end-to-end delay (as a measure of impending
congestion). However such techniques rely on voluntary, altruistic congestion). Currently, such techniques rely on voluntary,
action by end users and their application providers. Operators can altruistic action by end users and their application providers.
neither enforce their use nor avoid penalizing them for congestion Operators can neither enforce their use nor avoid penalizing them for
they avoid. congestion they avoid. {ToDo remove double negatives}
The Internet was designed so that end-hosts detect and control The Internet was designed so that end-hosts detect and control
congestion. We argue that congestion needs to be visible to network congestion. We argue that congestion needs to be visible to network
nodes as well, not just to the end hosts. More specifically, a nodes as well, not just to the end hosts. More specifically, a
network needs to be able to measure how much congestion any network needs to be able to measure how much congestion any
particular traffic expects to cause between the monitoring point in particular traffic expects to cause between the monitoring point in
the network and the destination ("rest-of-path congestion"). This the network and the destination ("rest-of-path congestion"). This
would be a new capability. Today a network can use Explicit would be a new capability. Today a network can use Explicit
Congestion Notification (ECN) [RFC3168] to detect how much congestion Congestion Notification (ECN) [RFC3168] to detect how much congestion
the traffic has suffered between the source and a monitoring point, the traffic has suffered between the source and a monitoring point
but not beyond. This new capability would enable an ISP to give ("upstream congestion"), but not beyond. This new capability would
incentives for the use of LEDBAT-like applications that seek to enable an ISP to give incentives for the use of LEDBAT-like
minimise congestion in the network whilst restricting inappropriate applications that seek to minimise congestion in the network whilst
uses of traditional TCP and UDP applications. restricting inappropriate uses of traditional TCP and UDP
applications. {ToDo: remove implication that it enforces TCP
behaviour.}
So we propose a new approach which we call Congestion Exposure. We So we propose a new approach which we call Congestion Exposure. We
propose that congestion information should be made visible at the IP propose that congestion information should be made visible at the IP
layer, so that any network node can measure the contribution to layer, so that any network node can measure the contribution to
congestion of an aggregate of traffic as easily as straight volume congestion of an aggregate of traffic as easily as straight volume
can be measured today. Once the information is exposed in this way, can be measured today. Once the information is exposed in this way,
it is then possible to use it to measure the true impact of any it is then possible to use it to measure the true impact of any
traffic on the network. traffic on the network.
In general, congestion exposure gives operators a principled way to In general, congestion exposure gives operators a principled way to
skipping to change at page 10, line 24 skipping to change at page 12, line 22
it. The probability of a packet being marked increases with the it. The probability of a packet being marked increases with the
length of the queue and thus the rate of CE marks is a guide to the length of the queue and thus the rate of CE marks is a guide to the
level of congestion at that queue. This CE codepoint travels forward level of congestion at that queue. This CE codepoint travels forward
through the network to the receiver which then informs the sender through the network to the receiver which then informs the sender
that it has seen congestion. The sender is then required to respond that it has seen congestion. The sender is then required to respond
as if it had experienced a packet loss. Because the CE codepoint is as if it had experienced a packet loss. Because the CE codepoint is
visible in the IP layer, this approach reveals the upstream visible in the IP layer, this approach reveals the upstream
congestion level for a packet. congestion level for a packet.
Alas, this is not enough - ECN gives downstream nodes an idea of the Alas, this is not enough - ECN gives downstream nodes an idea of the
congestion so far for any flow. This can help hold a receiver congestion so far {ToDo: clarify} for any flow. This can help hold a
accountable for the congestion caused by incoming traffic. But a receiver accountable for the congestion caused by incoming traffic.
receiver can only indirectly influence incoming congestion, by But a receiver can only indirectly influence incoming congestion, by
politely asking the sender to control it. A receiver cannot make a politely asking the sender to control it. A receiver cannot make a
sender install an adaptive codec, or install LEDBAT instead of TCP sender install an adaptive codec, or install LEDBAT instead of TCP
congestion-control. And a receiver cannot cause an attacker to stop congestion-control. And a receiver cannot cause an attacker to stop
flooding it with traffic. flooding it with traffic.
What is needed is knowledge of the downstream congestion level, for What is needed is knowledge of the downstream congestion level, for
which you need additional information that is still concealed from which you need additional information that is still concealed from
the network. the network.
5. ConEx Use Cases 5. ConEx Use Cases
This section sets out some of the use cases for ConEx. These use {ToDo: Consider framing these three cases as stages.} This section
cases rely on some of the conceptual elements described in sets out some of the use cases for ConEx. These use cases rely on
[ConEx-Abstract-Mech]. The authors don't claim this is an exhaustive some of the conceptual elements described in [ConEx-Abstract-Mech].
list of use cases, nor that these have equal merit. In most cases The authors don't claim this is an exhaustive list of use cases, nor
ConEx is not the only solution to achieve these. But these use cases that these have equal merit. In most cases ConEx is not the only
represent a consensus among people that have been working on this solution to achieve these. But these use cases represent a consensus
approach for some years. among people that have been working on this approach for some years.
5.1. ConEx as a basis for traffic management 5.1. Inform the Operator's Traffic Management
Currently many operators impose some form of traffic management at Currently many operators impose some form of traffic management at
peak hours. This is a simple economic necessity -- the only reason peak hours. This is a simple economic necessity -- the only reason
the Internet works as a commercial concern is that operators are able the Internet works as a commercial concern is that operators are able
to rely on statistical multiplexing to share their expensive core to rely on statistical multiplexing to share their expensive core
network between large numbers of customers. In order to ensure all network between large numbers of customers. In order to ensure all
customers get some chance to access the network, the "heaviest" customers get some chance to access the network, the "heaviest"
customers will be subjected to some form of traffic management at customers will be subjected to some form of traffic management at
peak times (typically a rate cap for certain types of traffic) peak times (typically a rate cap for certain types of traffic)
[Fair-use]. Often this traffic management is done with expensive [Fair-use]. Often this traffic management is done with expensive
skipping to change at page 11, line 21 skipping to change at page 13, line 18
ConEx offers a better approach that will actually target the users ConEx offers a better approach that will actually target the users
that are causing the congestion. By using Ingress or Egress that are causing the congestion. By using Ingress or Egress
Policers, an ISP can identify which users are causing the greatest Policers, an ISP can identify which users are causing the greatest
Congestion Volume throughout the network. This can then be used as Congestion Volume throughout the network. This can then be used as
the basis for traffic management decisions. The Ingress Policer the basis for traffic management decisions. The Ingress Policer
described in [Policing-freedom] is one interesting approach that described in [Policing-freedom] is one interesting approach that
gives the user a congestion volume limit. So long as they stay gives the user a congestion volume limit. So long as they stay
within their limit then their traffic is unaffected. Once they within their limit then their traffic is unaffected. Once they
exceed that limit then their traffic will be blocked temporarily. exceed that limit then their traffic will be blocked temporarily.
5.2. ConEx to incentivise scavenger transports 5.2. Consequence: Incentivise Scavenger Transports
{ToDo: Absence of contribution to congestion is as important to know
(e.g. LEDBAT can prove it's less of a problem)}
The growth of "scavenger" behaviour (e.g. [LEDBAT]) helps to reduce
congestion, but can actually make the problem less tractable. These
users are trying to make good use of the capacity of the path while
minimising their own costs. Thus, users of such services may show
very heavy total traffic up until the moment congestion is detected
(at the Transport Layer), but then will immediately back off.
Monitoring (at the Internet Layer) cannot detect this congestion
avoidance if the congestion in question is in a different domain
further along the path and hence such users may get treated as
congestion-causing users.
{ToDo: The above has been moved here from the old introduction: need
to stitch it in, or remove duplication}
Recent work proposes a new approach for QoS where traffic is provided Recent work proposes a new approach for QoS where traffic is provided
with a less than best effort or "scavenger" quality of service. The with a less than best effort or "scavenger" quality of service. The
idea is that low priority but high volume traffic such as OS updates, idea is that low priority but high volume traffic such as OS updates,
P2P file transfers and view-later TV programs should be allowed to P2P file transfers and view-later TV programs should be allowed to
use any spare network capacity, but should rapidly get out of the way use any spare network capacity, but should rapidly get out of the way
if a higher priority or interactive application starts up. One if a higher priority or interactive application starts up. One
solution being actively explored is LEDBAT which proposes a new solution being actively explored is LEDBAT which proposes a new
congestion control algorithm that is less aggressive in seeking out congestion control algorithm that is less aggressive in seeking out
bandwidth than TCP. bandwidth than TCP.
At present most operators assume a strong correlation between the At present most operators assume a strong correlation between the
volume of a flow and the impact that flow causes in the network. volume of a flow and the impact that flow causes in the network.
This assumption has been eroded by the growth of interactive This assumption has been eroded by the growth of interactive
streaming which behaves in an inelastic manner and hence can cause streaming which behaves in an inelastic manner and hence can cause
high congestion at relatively low data volumes. Currently LEDBAT- high congestion at relatively low data volumes. Currently LEDBAT-
like transports get no incentive from the ISP since they still like transports get no incentive from the ISP since they still
transfer large volumes of data and may reach high transfer speeds if transfer large volumes of data and may reach high transfer speeds if
the network is uncongested. Consequently the only current incentive the network is uncongested. Consequently the only current incentive
for LEDBAT is that it can reduce self-congestion effects. for LEDBAT is that it can reduce self-congestion effects (see
Section 7.2).
If the ISP has deployed a ConEx-aware Ingress Policer then they are If the ISP has deployed a ConEx-aware Ingress Policer then they are
able to incentivise the use of LEDBAT because a user will be policed able to incentivise the use of LEDBAT because a user will be policed
according to the overall congestion volume their traffic generates, according to the overall congestion volume their traffic generates,
not the rate or data volume. If all background file transfers are not the rate or data volume. If all background file transfers are
only generating a low level of congestion, then the sender has more only generating a low level of congestion, then the sender has more
"congestion budget" to "spend" on their interactive applications. It "congestion budget" to "spend" on their interactive applications. It
can be shown [Kelly] that this approach improves social welfare -- in can be shown [Kelly] that this approach improves social welfare -- in
other words if you limit the congestion that all users can generate other words if you limit the congestion that all users can generate
then everyone benefits from a better service. then everyone benefits from a better service.
5.3. Accounting for Congestion Volume 5.3. Consequence: User-Controlled Intra-Class Quality of Service (QoS)
Accountability was one of the original design goals for the Internet
[Design-Philosophy]. At the time it was ranked low because the
network was non-commercial and it was assumed users had the best
interests of the network at heart. Nowadays users generally treat
the network as a commodity and the Internet has become highly
commercialised. This causes problems for operators and others which
they have tried to solve and often leads to a tragedy of the commons
where users end up fighting each other for scarce peak capacity.
The most elegant solution would be to introduce an Internet-wide
system of accountability where every actor in the network is held to
account for the impact they have on others. If Policers are placed
at every Network Ingress or Egress and Border Monitors at every
border, then you have the basis for a system of congestion
accounting. Simply by controlling the overall Congestion Volume each
end-system or stub-network can send you ensure everyone gets a better
service.
5.4. ConEx as a form of differential QoS
Most QoS approaches require the active participation of routers to Most QoS approaches require the active participation of routers to
control the delay and loss characteristics for the traffic. For control the delay and loss characteristics for the traffic. For
real-time interactive traffic it is clear that low delay (and real-time interactive traffic it is clear that low delay (and
predictable jitter) are critical, and thus these probably always need predictable jitter) are critical, and thus these probably always need
different treatment at a router. However if low loss is the issue different treatment at a router. However if low loss is the issue
then ConEx offers an alternative approach. then ConEx offers an alternative approach.
Assuming the ingress ISP has deployed a ConEx Ingress Policer, then Assuming the ingress ISP has deployed a ConEx Ingress Policer, then
the only control on a user's traffic is dependent on the congestion the only control on a user's traffic is dependent on the congestion
skipping to change at page 13, line 8 skipping to change at page 15, line 5
there are strong economic pressures to maintain a high enough data there are strong economic pressures to maintain a high enough data
rate (as that will directly influence the Quality of Experience the rate (as that will directly influence the Quality of Experience the
end-user receives. This approach removes the need for bandwidth end-user receives. This approach removes the need for bandwidth
brokers to establish QoS sessions, by removing the need to coordinate brokers to establish QoS sessions, by removing the need to coordinate
requests from multiple sources to pre-allocate bandwidth, as well as requests from multiple sources to pre-allocate bandwidth, as well as
to coordinate which allocations to revoke when bandwidth predictions to coordinate which allocations to revoke when bandwidth predictions
turn out to be wrong. There is also no need to "rate-police" at the turn out to be wrong. There is also no need to "rate-police" at the
boundaries on a per-flow basis, removing the need to keep per-flow boundaries on a per-flow basis, removing the need to keep per-flow
state (which in turn makes this approach more scalable). state (which in turn makes this approach more scalable).
5.5. Partial vs. Full Deployment 5.4. Other Use-Cases
In a fully-deployed ConEx-enabled internet, [QoS-Models] shows that Consequence: Preventing Congestion Collapse
ISP settlements based on congestion volume can allocate money to
where upgrades are needed. Fully-deployed implies that ConEx-marked
packets which have not exhausted their expected congestion would go
through a congested path in preference to non-ConEx packets, with
money changing hands to justify that priority.
In a partial deployment, routers that ignore ConEx markings and let The Internet is no less vulnerable to congestion collapses now than
them pass unaltered are no problem unless they become congested and it was in the 1980s [ref]. If a new app that implemented congestion
drop packets. Since ConEx incentivises the use of lower congestion control badly rapidly became popular, existing non-congestion-related
transports, such congestion drops should anyway become rare events. traffic controls would not prevent collapse. Under extreme
ConEx-unaware routers that do drop ConEx-marked packets would cause a congestion, ConEx-based traffic management would automatically
problem so to minimise this risk ConEx should be designed such that override host congestion control and prevent collapse.
ConEx packets will appear valid to any node they traverse. Failing
that it could be possible to bypass such nodes with a tunnel.
If any network is not ConEx enabled then the sender and receiver have Inform Inter-Operator Contracts {ToDo: Interoperation between those
to rely on ECN-marking or packet drops to establish the congestion who make different economic choices in different economic
level. If the receiver isn't ConEx-enabled then there needs to be conditions.}
some form of compatibility mode. Even in such partial deployments
the end-users and access networks will benefit from ConEx. This will
put create incentives for ConEx to be more widely adopted as access
networks put pressure on their backhaul providers to use congestion
as the basis of their interconnect agreement.
The actual charge per unit of congestion would be specified in an Inform Capacity Provisioning
interconnection agreement, with economic pressure driving that charge
downward to the cost to upgrade whenever alternative paths are
available. That charge would most likely be invisible to the
majority of users. Instead such users will have a contractual
allowance to cause congestion, and would see packets dropped when
that allowance is depleted.
Once an Autonomous System (AS) agrees to pay any congestion charges Flooding attacks
to any other AS it forwards to, it has an economic incentive to
increase congestion-so-far marking for any congestion within its
network. Failure to do this quickly becomes a significant cost,
giving it an incentive to turn on such marking.
End users (or the writers of the applications they use) will be given 6. Deployment Arrangements
an incentive to use a congestion control that back off more
aggressively than TCP for any elastic traffic. Indeed they will
actually have an incentive to use fully weighted congestion controls
that allow traffic to cause congestion in proportion to its priority.
Traffic which backs off more aggressively than TCP will see
congestion charges remain the same (or even drop) as congestion
increases; traffic which backs off less aggressively will see charges
rise, but the user may be prepared to accept this if it is high-
priority traffic; traffic which backs off not at all will see charges
rise dramatically.
6. Statistical Multiplexing over Differing Timescales This document is about concepts and use-cases, not mechanism.
However, in order to describe how the above use-cases would be
deployed, a high-level understanding of ConEx mechanism is first
given below.
Access networks are usually provisioned assuming statistical 6.1. Incremental Deployment Features of the Protocol Mechanism
multiplexing, where end-users are presumed not all to use their
maximum bandwidth simultaneously. Typically, an ISP might design
access networks with shared resources (e.g., circuits, ports,
schedulers) dimensioned in proportion to the sum of average usage by
the customers involved. Generally, ISPs monitor actual usage
averaged over some time period (typically stated in minutes) to plan
when upgrades to shared resources will be needed.
Almost always, they find that certain busy periods of the day have The ConEx mechanism document [ConEx-Abstract-Mech] goes to great
higher usage; and that actual contention for bandwidth at a shared lengths to design for incremental deployment in all the respects
resource (e.g., circuit, port, scheduler) is limited to those below. It should be referred to for precise details on each of these
periods. This leads to "economic congestion" as defined in Section points:
3.4 of [Bauer09], where traffic by one end-user imposes a "cost" of
reduced utility on other users. Sometimes, there is an extended
period between economic congestion being first observed and the
completion of upgrades. In other cases, a trend of "economic
congestion" is used by a service provider before congestion as
defined in the abstract mechanism (loss or ECN marking) occurs.
During busy periods, it has been observed that roughly 20% of the o The ConEx mechanism is essentially a change to the source, in
end-users are using 80% of the bandwidth [Varian]. We call this order to re-insert congestion feedback into the network
roughly-20% "heavy users", and the others "light users". Left to (Figure 1).
itself, this situation means that heavy users cause queues to fill at
a rate much greater than light users do. (Note that this heavy/light
categorization is for illustrative purposes since there is actually a
continuum of "heaviness" across users.) When both heavy and light
users pay the same flat rate, ISPs believe heavy users should bear
more of the "cost" of reduced utility.
When all users have unlimited access to a shared resource bottleneck, o Source-host-only deployment is possible without any negotiation
this problem is the most severe since maximum per user bandwidth is required, and individual transport protocol implementations within
that of the shared resource. In order to provide more control over a source host can be updated separately.
the maximum rate at which individual users may send, many ISPs have
deployed "traffic shapers" to limit bandwidth available to an
individual user during all time periods. Note that this limits the
per user maximum bandwidth in the sub-second timeframe of the shaper
queue. Currently, these "shapers" make no distinction between busy
periods where shared resource congestion may occur and periods when
no congestion occurs.
During a period of higher usage, a shared resource becomes the o Receiver modification may optionally improve ConEx for some
bottleneck and causes filling of a shared queue or individual user transport protocols with feedback limitations (TCP being the main
shaper queues. However, heavy users create much more queuing and example), but it is not a necessity
therefore potentially more congestion volume Section 2 as compared
with lighter users. This means that during periods of higher usage,
heavier and lighter users see comparable congestion (i.e., packet
loss or ECN marking). Thus, the overall utility (i.e., probability
of a packet not being lost or ECN marked) is reduced by the fewer
heavier users at the expense of the many lighter users.
During periods of lighter usage, heavier users will fill their o Proxies for the source and/or receiver are feasible (though not
individual shaper queues, potentially creating loss or ECN marking, necessarily straightforward)
such that TCP congestion-control does what the ISP desires and cuts
back the sending rate giving the user the expected maximum bandwidth.
6.1. ConEx Objectives for This Issue o Queues and network forwarding do not require any modification for
ConEx.
ConEx should provide better information for a provider to address the o ECN is not required in the network for ConEx. If some network
"economic congestion" problem. Specifically, ConEx should help to nodes support ECN, it can be used by ConEx.
distinguish which users cause queue-filling over a time interval
matching the economic congestion and statistical multiplexing
algorithm time scales. This can range from seconds, to minutes, to
hours. It is also desirable to distinguish "self-congestion" where
there is no contention for a shared resource bandwidth (e.g.,
circuit, port, scheduler), which is referred to as "inter-user
congestion" in the following. If this is visible to end-users, they
could use an out-of-band mechanism to "go faster" if only "self-
congestion" is limiting their throughput.
There are (at least) three approaches for addressing this issue. o ECN is not required at the receiver for ConEx. The sender should
nonetheless attempt to negotiate ECN-usage with the receiver,
given some aspects of ConEx work better the more ECN is deployed,
particularly auditing and border measurement.
1. Treat "self-congestion" the same as "inter-user congestion" since o Given ConEx exposes information for IP-layer policy devices to
they both create congestion as perceived by the flow user; use, the design does not preclude possible innovative uses of
ConEx information by other IP-layer devices, e.g. forwarding
itself
2. Signal more information to the receiver about the cause of loss o Packets indicate whether or not they support ConEx.
since the remedy may differ;
3. Process (and generate) ConEx information at the same network 6.2. Per-Network Deployment Concepts
element which implements the shaper, which has knowledge of the
configured maximum bandwidth for the users as well as local
shared resource congestion.
For the most part, these don't require any changes to the abstract Network deployment-related definitions:
mechanism; but a subcase of 2), where the traffic-shaper might use
ConEx to signal that the "congestion" is actually due to traffic-
shaping, not shared resource contention, could require additional
signaling to be defined in the ConEx protocol.
Note that during busy periods "self congestion" might not be the Internet Ingress: The first IP node a packet traverses that is
limiting factor, but there will inevitably be less-busy periods where outside the source's own network. In a domestic network that will
"self-congestion" will predominate. be the first node downstream from the home access equipment. In
an enterprise network this is the provider edge router.
6.2. ConEx as a Solution Internet Egress: The last IP node a packet traverses before reaching
the receiver's network.
Over a time period related to the statistical multiplexing or ConEx-Enabled Network: A network whose edge nodes implement ConEx
economic congestion interval (e.g., many seconds to minutes to hours) policy functions.
total up the number of bytes that have been congestion marked and the
total number of bytes sent per end-user. Compute the ratio of
congested bytes to total bytes. This measures the average rate per
user.
Quantizing users into classes using one threshold on total and and Each network can unilaterally choose to use any ConEx information
another threshold on ratio results in a grid that identifies four given by those sources using ConEx, independently of whether other
classes of user: networks use it.
+------------+-------------+-------------+ Typically, a network will use ConEx information by deploying a policy
| | Volume | function at the ingress edge of its network to monitor arriving
| Ratio | Large | Small | traffic and to act in some way on the congestion information in those
+------------+-------------+-------------+ packets that are ConEx-enabled. Actions might include policing,
| High | Heavy User | Bursty User | altering the class of service, or re-routing. Alternatively, less
+------------+-------------+-------------+ direct actions via a management system might include triggering
| Low | LEDBAT User | Light User | capacity upgrades, triggering penalty clauses in contracts or levying
+------------+-------------+-------------+ charges between networks based on ConEx measurements.
(Where "LEDBAT User" includes other Less-than-Best-Effort algorithms.)
Figure 1: Four Classes of User [I need to sleep, so I'll resort to bullets for the rest...]
Audit: what it is (checks ConEx info is never less than actual
congestion so far on the path). If ConEx protocol is adhered to,
there should be no place in a network where this is not true, so
audit can be placed wherever most convenient. But most useful
placement is after the likely congestion locations, ideally as close
to the egress as possible.
Note that Bursty and Heavy Users contribute more to congestion Typically, a network using ConEx info will deploy a ConEx policy
marking, but a Bursty user contributes less overall congestion function near the ingress edge and a ConEx audit function near the
marking and may be creating shorter periods of queue filling as egress edge. The segment of the path between a ConEx policy function
compared with heavy users. LEDBAT and light users create less to and a ConEx audit function can be considered to be a ConEx-protected
congestion marking, with LEDBAT users able to transfer more volume as segment of the path. And assuming a network covers all its ingresses
compared with light users since LEDBAT users back off before and egresses with policy functions and audit functions respectively,
congestion marking occurs. An operator might reasonably take this the network within this ring will be a ConEx-protected network.
into account in their shaping algorithms.
6.3. Additional Support Using other Measures and Mechanisms Of course, because each edge device usually serves as both an ingress
and an egress, the two functions are both likely to be present in
each edge device.
An additional congestion measure of burstiness (in addition to [Plan view diagram of a ConEx-protected segment and ConEx-protected
"congestion") would allow nodes upstream from the node implementing network here?]
the shaper to implement traffic management. This measure could be
derived from signals in the abstract mechanism but would require (a
majority) of the heavier senders and receivers to implement conex and
also would only work if loss or ECN marking occurs. Also, signaling
a measure of the burstiness (or something related to it) would
partially address the scenario where no loss or ECN marking occurs.
As an alternative, a "light weight" TCP proxy might be implemented at [We might have to explain the concept of non-negativity of congestion
the network element containing the shaper, and an upstream network earlier in the doc, so we can use it here. Could also require one of
element (e.g., an ingress router), could potentially create a ConEx those staircase diags I used in re-ECN, but that assume ECN and ConEx
control loop between these network elements to provide more optimal - I'd rather focus on drop-only cases.]
balance between heavier and lighter users during congested intervals.
This would be a closed domain where the signals could be implicitly
trusted. The burstiness measure could be communicated using TCP
extensions between these proxies.
There is also the aspect of "self congestion" where a traffic-shaper Sources can choose not to send ConEx-enabled packets. Networks are
is at the access node. Using the current mechanisms, the receiver expected to make it in senders' interests to expose congestion
cannot tell the difference between "self-congestion" and "inter-user information in packets by treating ConEx-enabled packets better (in
congestion." Adding a signal to the abstract mechanism could enable some sense) than non-ConEx packets.
a receiver to inform the sender about the cause of congestion,
enabling the sender to request that the traffic-shaper parameters
change to enable the flow to "go faster".
7. Other issues Prior to ConEx, networks have generally place constraints on incoming
traffic to reduce the chances of it causing congestion. For ConEx-
enabled traffic, networks can remove these pessimistic constraints
and solely apply constraints when the ConEx information indicates
traffic is contributing to congestion.
6.3. Single Receiving Network Scenario
Charter scenario [Description, picture and some deployment incentive
text]
Focusing on first step: why OS developers of senders would implement
and why users would send ConEx. Then, once info is there, why
network would use it.
6.4. Other Initial Deployment Scenarios
o Mobile: [conex-mobile] Couple of sentence description
(characterised by terminals and network standards all co-
ordinated.)
o Data Centre: Brief description. (Happens because under one
authority)
7. Potential Issues or Non-Issues
7.1. Congestion as a Commercial Secret 7.1. Congestion as a Commercial Secret
Network operators have long viewed the congestion levels in their Network operators have long viewed the congestion levels in their
network as a business secret. In some ways this harks back to the network as a business secret. In some ways this harks back to the
days of fixed-line telecommunications where congestion manifested as days of fixed-line telecommunications where congestion manifested as
failed connections or dropped calls. But even in modern data-centric failed connections or dropped calls. But even in modern data-centric
packet networks congestion is viewed as a secret not to be shared packet networks congestion is viewed as a secret not to be shared
with competitors. It can be debated whether this view is sensible, with competitors. It can be debated whether this view is sensible,
but it may make operators uneasy about deploying ConEx. The but it may make operators uneasy about deploying ConEx. The
skipping to change at page 18, line 18 skipping to change at page 19, line 5
provider to maintain a good level of QoS. But in reality the provider to maintain a good level of QoS. But in reality the
Content Provider will just use the feedback mechanisms in Content Provider will just use the feedback mechanisms in
streaming protocols such as Adobe Flash to monitor the congestion. streaming protocols such as Adobe Flash to monitor the congestion.
Of course some might say that the idea of keeping congestion secret Of course some might say that the idea of keeping congestion secret
is silly. After all, end-hosts already have knowledge of the is silly. After all, end-hosts already have knowledge of the
congestion throughout the network, albeit only along specific paths, congestion throughout the network, albeit only along specific paths,
and operators can work out that there is persistent congestion as and operators can work out that there is persistent congestion as
their customers will be suffering degraded network performance. their customers will be suffering degraded network performance.
7.2. Information Security 7.2. Self Congestion
make a source believe it has seen more congestion than it has
hijack a user's identity and make it appear they are dishonest at an
egress policer
clear or otherwise tamper with the ConEx markings
... {ToDo: Re-phrase as an issue}
{ToDo} Write these up properly... Congestion takes two distinct forms. The first results from the
interaction of traffic from one set of users with traffic from other
users, causing in a reduction in service (a cost) for all of them.
the second, often referred to as "self-congestion", occurs when an
increase in traffic from a single user causes that user to suffer a
worse service (for instance because their traffic is being "shaped"
by their ISP, or because they have an excessively large buffer in
their home router). ConEx is principally interested in the first
form of congestion since it involves informing those other users of
the impact you expect to have on them.
8. Security Considerations 8. Security Considerations
This document proposes a mechanism tagging onto Explicit Congestion This document proposes a mechanism tagging onto Explicit Congestion
Notification [RFC3168], and inherits the security issues listed Notification [RFC3168], and inherits the security issues listed
therein. The additional issues from ConEx markings relate to the therein. The additional issues from ConEx markings relate to the
degree of trust each forwarding point places in the ConEx markings it degree of trust each forwarding point places in the ConEx markings it
receives, which is a business decision mostly orthogonal to the receives, which is a business decision mostly orthogonal to the
markings themselves. markings themselves.
skipping to change at page 19, line 20 skipping to change at page 20, line 8
is one of the worst-kept secrets a network has, because end-hosts is one of the worst-kept secrets a network has, because end-hosts
can see congestion better than network operators can. Congestion can see congestion better than network operators can. Congestion
Exposure will enable network operators to pinpoint whether Exposure will enable network operators to pinpoint whether
congestion is on one side or the other of any border. It is congestion is on one side or the other of any border. It is
conceivable that forwarders with underprovisioned networks may try conceivable that forwarders with underprovisioned networks may try
to obstruct deployment of Congestion Exposure. to obstruct deployment of Congestion Exposure.
The Receiver Receivers generally have an incentive to under-declare The Receiver Receivers generally have an incentive to under-declare
congestion since they generally wish to receive the data from the congestion since they generally wish to receive the data from the
sender as rapidly as possible. [Savage] explains how a receiver sender as rapidly as possible. [Savage] explains how a receiver
can significantly improve their throughput my failing to declare can significantly improve their throughput by failing to declare
congestion. This is a problem with or without Congestion congestion. This is a problem with or without Congestion
Exposure. [KGao] explains one possible technique to encourage Exposure. [KGao] explains one possible technique to encourage
receiver's to be honest in their declaration of congestion. receiver's to be honest in their declaration of congestion.
The Sender One proposed mechanism for Congestion Exposure deployment The Sender One proposed mechanism for Congestion Exposure deployment
adds a requirement for a sender to advise the network how much adds a requirement for a sender to advise the network how much
congestion it has suffered or caused. Although most senders congestion it has suffered or caused. Although most senders
currently respond to congestion they are informed of, one use of currently respond to congestion they are informed of, one use of
exposed congestion information might be to encourage sources of exposed congestion information might be to encourage sources of
persistent congestion to back off more aggressively. Then clearly persistent congestion to back off more aggressively. Then clearly
skipping to change at page 19, line 42 skipping to change at page 20, line 30
congestion. This will be a particular problem with sources of congestion. This will be a particular problem with sources of
flooding attacks. "Policing" mechanisms have been proposed to flooding attacks. "Policing" mechanisms have been proposed to
deal with this. deal with this.
In addition there are potential problems from source spoofing. A In addition there are potential problems from source spoofing. A
malicious sender can pretend to be another user by spoofing the malicious sender can pretend to be another user by spoofing the
source address. Congestion Exposure allows for "Policers" and source address. Congestion Exposure allows for "Policers" and
"Traffic Shapers" so as to be robust against injection of false "Traffic Shapers" so as to be robust against injection of false
congestion information into the forward path. congestion information into the forward path.
8.1. Information Security
make a source believe it has seen more congestion than it has
hijack a user's identity and make it appear they are dishonest at an
egress policer
clear or otherwise tamper with the ConEx markings
...
{ToDo} Write these up properly...
9. IANA Considerations 9. IANA Considerations
This document does not require actions by IANA. This document does not require actions by IANA.
10. Acknowledgments 10. Conclusions
{ToDo: It is not our place to judge. Many operators choose a mix of
traffic management and capacity upgrades, whether they run a
University network or a national ISP. Rather than beating our chests
about evil traffic management, the idea is to recognise it is a
legitimate choice, understand why poor practice is widespread and
enable best practice instead.}
Alissa: Benefits of congestion exposure
o Incentivizes reduced-congestion protocols like LEDBAT
o Exposes congestion end-to-end, across network borders
o Provides transparency at every network node
o good for capacity planning
o Avoids cat-and-mouse with apps developers, other drawbacks of
application-based approaches
o Any-to-any traffic: enables Internet-wide solutions
Rich: Comparison of Re-ECN and Fair-Share
o Similarities between re-ECN and FairShare
* "Protocol agnostic" and "application agnostic"
* Acts on current network conditions and recent user traffic
* Compatible with Internet standards and supportive of Internet
innovation
o Advantages of re-ECN over FairShare
* Re-ECN enables end-to-end congestion management
* Re-ECN signals the presence of congestion to user applications
and to all devices on the network path
* Re-ECN enables additional user and/or application responses to
congestion
* Re-ECN enables DDoS mitigation and other capabilities
Charter: It is believed that the CONEX mechanism will be useful as a
generative technology that can be applied as a key element of
congestion management solutions in a wide variety of use cases.
However, the CONEX WG will initially focus on one use case, where the
end hosts and the network that contains the destination end host are
CONEX-enabled but other networks need not be. CONEX information can
assist the network operator's traffic management and, for example,
incentivize LEDBAT-like applications. Congestion-based billing is
not within the scope of the WG.
11. Acknowledgments
Bob Briscoe is partly funded by Trilogy, a research project (ICT- Bob Briscoe is partly funded by Trilogy, a research project (ICT-
216372) supported by the European Community under its Seventh 216372) supported by the European Community under its Seventh
Framework Programme. The views expressed here are those of the Framework Programme. The views expressed here are those of the
author only. author only.
The authors would like to thank the many people that have commented The authors would like to thank the many people that have commented
on this document. Bernard Aboba, Mikael Abrahamsson, Joao Taveira on this document. Bernard Aboba, Mikael Abrahamsson, Joao Taveira
Araujo, Steve Bauer, Caitlin Bestler, Steven Blake, Louise Burness, Araujo, Marcelo Bagnulo Braun, Steve Bauer, Caitlin Bestler, Steven
Alissa Cooper, Philip Eardley, Matthew Ford, Ingemar Johansson, Mirja Blake, Louise Burness, Ken Carlberg, Alissa Cooper, Nandita
Kuehlewind, Dirk Kutscher, Zhu Lei, Kevin Mason, Michael Menth, Chris Dukkipati, Philip Eardley, Wes Eddy, Matthew Ford, Ingemar Johansson,
Morrow, Hannes Tschofenig and Stuart Venters. Please accept our Georgios Karagiannis, Mirja Kuehlewind, Dirk Kutscher, Zhu Lei, Kevin
apologies if your name has been missed off this list. Mason, Matt Mathis, Michael Menth, Chris Morrow, Tim Shepard, Hannes
Tschofenig and Stuart Venters. Please accept our apologies if your
name has been missed off this list.
11. Informative References 11.1. Contributors
[BB-incentive] MIT Communications Futures Program (CFP) and The following co-authored this document, and Toby & John also co-
Cambridge University Communications Research edited it through most of its life:
Network, "The Broadband Incentive Problem",
September 2005. Toby Moncaster
Moncaster Internet Consulting
Dukes
Layer Marney
Colchester CO5 9UZ
UK
EMail: toby@moncaster.com
John Leslie
JLC.net
10 Souhegan Street
Milford, NH 03055
US
EMail: john@jlc.net
Dave McDysan
Verizon
22001 Loudon County Pkwy
Ashburn, VA 20147
US
EMail: dave.mcdysan@verizon.com
12. Informative References
[Bauer09] Bauer, S., Clark, D., and W. Lehr, "The [Bauer09] Bauer, S., Clark, D., and W. Lehr, "The
Evolution of Internet Congestion", 2009. Evolution of Internet Congestion", 2009.
[ConEx-Abstract-Mech] Briscoe, B., "Congestion Exposure (ConEx) [ConEx-Abstract-Mech] Mathis, M. and B. Briscoe, "Congestion
Concepts and Abstract Mechanism", Exposure (ConEx) Concepts and Abstract
draft-ietf-conex-abstract-mech-00 (work in Mechanism", draft-ietf-conex-abstract-mech-02
progress), March 2011. (work in progress), July 2011.
[Design-Philosophy] Clarke, D., "The Design Philosophy of the
DARPA Internet Protocols", 1988.
[Fair-use] Broadband Choices, "Truth about 'fair usage' [Fair-use] Broadband Choices, "Truth about 'fair usage'
broadband", 2009. broadband", 2009.
[Fairer-faster] Briscoe, B., "A Fairer Faster Internet
Protocol", IEEE Spectrum Dec 2008 pp38-43,
December 2008.
[KGao] Gao, K. and C. Wang, "Incrementally Deployable [KGao] Gao, K. and C. Wang, "Incrementally Deployable
Prevention to TCP Attack with Misbehaving Prevention to TCP Attack with Misbehaving
Receivers", December 2004. Receivers", December 2004.
[Kelly] Kelly, F., Maulloo, A., and D. Tan, "Rate [Kelly] Kelly, F., Maulloo, A., and D. Tan, "Rate
control for communication networks: shadow control for communication networks: shadow
prices, proportional fairness and stability", prices, proportional fairness and stability",
Journal of the Operational Research Journal of the Operational Research
Society 49(3) 237--252, 1998, <http:// Society 49(3) 237--252, 1998, <http://
www.statslab.cam.ac.uk/~frank/rate.html>. www.statslab.cam.ac.uk/~frank/rate.html>.
[LEDBAT] Shalunov, S., "Low Extra Delay Background [LEDBAT] Shalunov, S., Hazel, G., Iyengar, J., and M.
Kuehlewind, "Low Extra Delay Background
Transport (LEDBAT)", Transport (LEDBAT)",
draft-ietf-ledbat-congestion-01 (work in draft-ietf-ledbat-congestion-06 (work in
progress), March 2010. progress), May 2011.
[Malice] Briscoe, B., "Using Self Interest to Prevent
Malice; Fixing the Denial of Service Flaw of
the Internet", WESII - Workshop on the
Economics of Securing the Information
Infrastructure 2006, 2006, <http://
wesii.econinfosec.org/draft.php?paper_id=19>.
[Padhye] Padhye, J., Firoiu, V., Towsley, D., and J.
Kurose, "Modeling TCP Throughput: A Simple
Model and its Empirical Validation", ACM
SIGCOMM Computer Communications Review Vol
28(4), pages 303-314, May 1998.
[Policing-freedom] Briscoe, B., Jacquet, A., and T. Moncaster, [Policing-freedom] Briscoe, B., Jacquet, A., and T. Moncaster,
"Policing Freedom to Use the Internet Resource "Policing Freedom to Use the Internet Resource
Pool", RE-Arch 2008 hosted at the 2008 CoNEXT Pool", RE-Arch 2008 hosted at the 2008 CoNEXT
conference , December 2008. conference , December 2008.
[QoS-Models] Briscoe, B. and S. Rudkin, "Commercial Models [RFC0970] Nagle, J., "On packet switches with infinite
for IP Quality of Service Interconnect", BTTJ storage", RFC 970, December 1985.
Special Edition on IP Quality of Service vol
23 (2), April 2005.
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie,
B., Deering, S., Estrin, D., Floyd, S., B., Deering, S., Estrin, D., Floyd, S.,
Jacobson, V., Minshall, G., Partridge, C., Jacobson, V., Minshall, G., Partridge, C.,
Peterson, L., Ramakrishnan, K., Shenker, S., Peterson, L., Ramakrishnan, K., Shenker, S.,
Wroclawski, J., and L. Zhang, "Recommendations Wroclawski, J., and L. Zhang, "Recommendations
on Queue Management and Congestion Avoidance on Queue Management and Congestion Avoidance
in the Internet", RFC 2309, April 1998. in the Internet", RFC 2309, April 1998.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black,
"The Addition of Explicit Congestion "The Addition of Explicit Congestion
Notification (ECN) to IP", RFC 3168, Notification (ECN) to IP", RFC 3168,
September 2001. September 2001.
[RFC6077] Papadimitriou, D., Welzl, M., Scharf, M., and [RFC6077] Papadimitriou, D., Welzl, M., Scharf, M., and
B. Briscoe, "Open Research Issues in Internet B. Briscoe, "Open Research Issues in Internet
Congestion Control", RFC 6077, February 2011. Congestion Control", RFC 6077, February 2011.
[Re-Feedback] Briscoe, B., Jacquet, A., Di Cairano-
Gilfedder, C., Salvatori, A., Soppera, A., and
M. Koyabe, "Policing Congestion Response in an
Internetwork Using Re-Feedback", ACM SIGCOMM
CCR 35(4)277--288, August 2005, <http://
www.acm.org/sigs/sigcomm/sigcomm2005/
techprog.html#session8>.
[Savage] Savage, S., Wetherall, D., and T. Anderson, [Savage] Savage, S., Wetherall, D., and T. Anderson,
"TCP Congestion Control with a Misbehaving "TCP Congestion Control with a Misbehaving
Receiver", ACM SIGCOMM Computer Communication Receiver", ACM SIGCOMM Computer Communication
Review , 1999. Review , 1999.
[Varian] Varian, H., "Congestion pricing principles", [conex-mobile] Kutscher, D., Mir, F., Winter, R., Krishnan,
Technical Plenary 78th IETF Meeting, July . S., and Y. Zhang, "Mobile Communication
Congestion Exposure Scenario",
draft-kutscher-conex-mobile-00 (work in
progress), March 2011.
[re-ecn-motive] Briscoe, B., Jacquet, A., Moncaster, T., and Appendix A. Changes from previous drafts (to be removed by the RFC
A. Smith, "Re-ECN: A Framework for adding Editor)
Congestion Accountability to TCP/IP",
draft-briscoe-tsvwg-re-ecn-tcp-motivation-02
(work in progress), October 2010.
Authors' Addresses From draft-ietf-conex-concepts-uses-00 to -01:
Toby Moncaster (editor) Added section on timescales: Section 6
Moncaster Internet Consulting
Dukes
Layer Marney
Colchester CO5 9UZ
UK
EMail: toby@moncaster.com Revised introduction to clarify congestion definitions
John Leslie (editor) Changed source for congestion definition in Section 2
JLC.net
10 Souhegan Street
Milford, NH 03055
US
EMail: john@jlc.net Other minor changes
Bob Briscoe From draft-moncaster-conex-concepts-uses-02 to
draft-ietf-conex-concepts-uses-00 (per decisions of working group):
Removed section on DDoS mitigation use case.
Removed appendix on ConEx Architectural Elements. PLEASE NOTE:
Alignment of terminology with the Abstract Mechanism draft has
been deferred to the next version.
From draft-moncaster-conex-concepts-uses-01 to
draft-moncaster-conex-concepts-uses-02:
Updated document to take account of the new Abstract Mechanism
draft [ConEx-Abstract-Mech].
Updated the definitions section.
Removed sections on Requirements and Mechanism.
Moved section on ConEx Architectural Elements to appendix.
Minor changes throughout.
From draft-moncaster-conex-concepts-uses-00 to
draft-moncaster-conex-concepts-uses-01:
Changed end of Abstract to better reflect new title
Created new section describing the architectural elements of
ConEx. Added Edge Monitors and Border Monitors (other elements
are Ingress, Egress and Border Policers).
Extensive re-write of Section 5 partly in response to suggestions
from Dirk Kutscher
Improved layout of Section 2 and added definitions of Whole Path
Congestion, ConEx-Enabled and ECN-Enabled. Re-wrote definition of
Congestion Volume. Renamed Ingress and Egress Router to Ingress
and Egress Node as these nodes may not actually be routers.
Improved document structure. Merged sections on Exposing
Congestion and ECN.
Added new section on ConEx requirements with a ConEx Issues
subsection. Text for these came from the start of the old ConEx
Use Cases section
Added a sub-section on Partial vs Full Deployment (Section 5.5)
Added a discussion on ConEx as a Business Secret Section 7.1
From draft-conex-mechanism-00 to
draft-moncaster-conex-concepts-uses-00:
Changed filename to draft-moncaster-conex-concepts-uses.
Changed title to ConEx Concepts and Use Cases.
Chose uniform capitalisation of ConEx.
Moved definition of Congestion Volume to list of definitions.
Clarified mechanism section. Changed section title.
Modified text relating to conex-aware policing and policers (which
are NOT defined terms).
Re-worded bullet on distinguishing ConEx and non-ConEx traffic in
Section 5.
Authors' Addresses
Bob Briscoe (editor)
BT BT
B54/77, Adastral Park B54/77, Adastral Park
Martlesham Heath Martlesham Heath
Ipswich IP5 3RE Ipswich IP5 3RE
UK UK
Phone: +44 1473 645196 Phone: +44 1473 645196
EMail: bob.briscoe@bt.com EMail: bob.briscoe@bt.com
URI: http://bobbriscoe.net/ URI: http://bobbriscoe.net/
Richard Woundy
Richard Woundy (editor)
Comcast Comcast
Comcast Cable Communications Comcast Cable Communications
27 Industrial Avenue 27 Industrial Avenue
Chelmsford, MA 01824 Chelmsford, MA 01824
US US
EMail: richard_woundy@cable.comcast.com EMail: richard_woundy@cable.comcast.com
URI: http://www.comcast.com URI: http://www.comcast.com
Dave McDysan
Verizon
22001 Loudon County Pkwy
Ashburn, VA 20147
US
EMail: dave.mcdysan@verizon.com
 End of changes. 112 change blocks. 
508 lines changed or deleted 677 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/