There is a need for 100 Gb/s data transport, or 100 Gb Ethernet (100 GbE), for both local area networks (LANs) and wide area networks (WANs). Standardization related to 100 Gb/s is a key next step for Ethernet to continue in its role as a ubiquitous end-to-end protocol. This article describes recent standardization efforts, and activities planned for the future. 100 Gb is a work in progress and this article provides an early snapshot of the standards effort being undertaken mostly by the IEEE 802.3 and its High Speed Study Group (HSSG), which recently approved a project authorization request (PAR) 802.3ba document. In addition, the International Telecommunication Union (ITU-T) is involved in the definition of 100 GbE parameters for transport over the WAN by using the Optical Transport Network (OTN) rates standardized so far. It is critical that 100 GbE efforts progress in an orderly manner to ensure a robust and healthy overall network infrastructure.

The quickly rising tide of new bandwidth requirements driven by IP traffic, IP video, IPTV broadband and transaction-intensive Web 2.0 has caused carriers to purchase growing numbers of 10G interfaces on routers, carrier Ethernet switches and dense wavelength division multiplexing (DWDM) systems [1–3]. Equipment is now being deployed with 40G interfaces, and some long-haul DWDM-routes now carry 40G wavelengths.

Today, two thrusts are driving optical network infrastructure, architectures and applications. First, global carriers need to provide a ubiquitous network infrastructure to support all current (and, hopefully, future) services and telecommunications needs of residential and business users. And, second, the carriers must support the increasing demands of the scientific community for networks handling large-scale data transport and processing.

The need for 100G Ethernet is growing because of such applications. End users continue to add 10 Gb/s service pipes to meet the demand, and carriers need a better way to aggregate such links. While the standardization effort recently approved within IEEE is definitely forward-looking, there is a real need to develop a standardized networking solution at 100 GbE, and to develop a server-interconnect standardized solutions for 40 GbE. These two applications complement one another.

The increase of Internet traffic and the introduction of triple-play services are forcing carriers to increase network capacity at moderate cost. Introduction of electronic time-division multiplexed (ETDM) signals at higher channel bit rates reduces the cost-per-transmitted-bit due to the system's lower power consumption, smaller footprint and reduced management effort and complexity. Improved performance of electronic and optoelectronic components has allowed research on ETDM bit rates beyond 40 Gb/s, which is currently the highest standard channel bit rate for optical telecommunications networks. Recent progress in high-speed ETDM technology for 80 Gb/s and beyond, and development of high-speed ETDM transmission technologies, allow the telecommunications industry to begin discussions on standardizing emerging technologies, such as 100 Gb/s. Currently, high-speed electronics makes ETDM systems possible with line rates of 80-85 Gb/s, and even 100 Gb/s, which is expected to be the next generation of Ethernet in global data communications.

There are many standards development organizations (SDOs) and other fora working on 100G and beyond. Such organizations include the IEEE, the Alliance for Telecommunication Industry Solutions (ATIS), and the International Telecommunication Union (ITU-T). Non-SDOs interested in 100G are the Optical Internetworking Forum (OIF), the Ethernet Alliance, the Road to 100G Alliance, and the Optoelectronics Industry Development Association (OIDA). Each of them is working in some fashion to empower 100G as a key optical networking technology. Using or leveraging existing technology will definitively help to achieve a global 100 Gb/s industry standard.

Standardization Drivers and Issues

There have been indications within the carrier sector that 100G is desperately needed, and so requires standardization for interoperability. A generic architecture is needed to accommodate the fact that bandwidth growth will always exceed the fastest interface we know how to build and standardize in the timeframe required by the marketplace. A direct relationship exists between standards and traffic growth. With a new standard written every 2 to 3 years, the data rate has had to reflect a factor of ten increase in that time to cope with traffic growth. Industry responded with FE (fast Ethernet), GbE (gigabit Ethernet) and 10 GbE to accommodate the traffic growth. This does not seem to be the case with the next-generation standard for 100G. Any high-speed standardization effort must also be compatible with DWDM applications and functionality.

It is noted that industry cost targets for 100 GbE are running 4 to 5 times the cost of discrete 10G PHYs (LAN/PHY specifically). This provides a good starting point for the 100G PHY. As always, keeping capital and operational costs as low as possible plays a significant role in adoption of the technology.

Bandwidth growth in the service provider industry is a known fact. Studies have shown annual growth of 15 percent in the metro space and around 10 percent in the core network space. These are conservative estimates and are always subject to disruptive business anomalies [3]. Developing a standard interface that targets the WAN via router-based DWDM interfaces is important to the industry as a whole. Timing for the 100 Gb/s interface is part of the industry's technology update cycle.

The IEEE HSSG has received unprecedented input from the carrier/service-provider and data-center end-user communities. These end users have participated in the discussions and made known their requirements. Today, a large percentage of these end users are using the Nx10G link aggregation group (LAG), with many starting to push the limits of this technology. Numerous end users have clearly stated that within the 2 to 3 year standards development schedule they will have exhausted the limits of LAG, and are already looking to interim, even proprietary solutions.

A number of other key points of benefit to end users drive the standardization effort, such as:

• The clear market need for 100 GbE.
• A standard for 100 GbE needed to replace the pre-standard/proprietary implementations that are less acceptable in the marketplace.
• Non-standard 40/80 GbE solutions will only tend to slow the development and adoption of 100 GbE.
• An industry standard is the best way to make worldwide deployment of 100 GbE cost effective.

Also, concerning standardization efforts, it should not be forgotten that 40G and 100G do not align gracefully. End users have already seen issues with LAN-FI/WAN-FI bandwidth mismatches when there's only a minor speed difference, and do not want to go through the growing pains of a mixed 40/100 Gb/s environment. End user hopes are focused on a standard for 100G being available as soon as possible, or100 G will be competing with 160 Gb/s very shortly.

Some end users feel that 100 GbE standardization is late, and many are working with vendors that have pre-standard 100 GbE plans. Because the need is so great, proprietary solutions may be satisfactory for a while as long as the links on the outer edges of the fabric are standardized.

IEEE HSSG/ITU-T and Optical Transport Networks
The IEEE 802.3 HSSG has been active in defining the Media Access Control (MAC) layer parameters and LAN physical interface specifications for the 100GbE Physical Interface (PHY). This is key to ensuring the standardized physical interconnection of Ethernet network elements within enterprise and data centers, and their connection to wide area network (WAN) optical transport systems.

The Optical Transport Network (OTN) is the future transport platform for all types of digital information exchange. The ITU-T has standardized OTN under the G. (G dot) series of recommendations as frame structures (G.709), architectures (G.872), and management functions (G.798). They are some of the global standards in use today. A multiplexed hierarchy of transport containers of optical data units (ODUs) is organized into optical transport units (OTUs) that provide the basis for generic framing procedure (GFP)-Framed, or GFP-Transparent data services. The OTU containers, as they are referred to, are numbered 1–4 and reflect the following data rates:

1. OTU1/ODU1 for 2.5 Gb/s
2. OTU2/ODU2 for 10 Gb/s
3. OTU3/ODU3 for 40 Gb/s
4. OTU4/ODU4 for 120 Gb/s

Development of the OTU4 container specification is currently under way in ITU-T SG 15 [4] and is the model for the 100 Gb/s effort in the IEEE HSSG. The OTU4 containers will be able to transparently transport nine, 10 GbE signals, or one 100 GbE signal. A high-level summary of related and complementary IEEE and ITU-T activities on 100 GbE standardization is shown in Table 1.

Recent IEEE HSSG Activities
The IEEE 802.3 HSSG just recently achieved (July 2007) a key milestone when the IEEE 802 Executive Committee (EC) approved the pre-submission of the P802.3ba PAR to the New Standards Committee (NESCOM) for consideration at the December 2007 IEEE Standards Association Standards Board (SASB) meeting. This will remain on the agenda subject to November 2007 802 EC approval.

Additionally, the 802 EC has approved extension of the HSSG work to the interconnection of equipment satisfying the distance requirements of the intended applications. The project's purpose is to extend the 802.3 protocols to operating speeds of 40 Gb/s and 100 Gb/s and provide a significant increase in bandwidth while maintaining maximum compatibility with the installed base of 802.3 interfaces and the previous investment in research and development, and principles of network operation and management.
This project is necessary to provide for applications requiring bandwidth beyond existing capabilities. These include data centers, Internet exchanges, high- performance computing, and video-on-demand delivery. Network aggregation and end-station bandwidth requirements are increasing at different rates, and call for two distinct speeds.

In the July 2007 closing plenary, the IEEE 802.3 group voted to adopt the following objectives (and associated PAR no. P802.3ba) for a Task Force to develop 100G and 40G Ethernet standards. Highlights related to 100GbE are:

• Support of full-duplex operation only
• Preservation of the 802.3 Ethernet frame format utilizing the 802.3 MAC
• Preservation of the minimum and maximum frame size of the current 802.3 standard
• Support of a bit error rate (BER) better than or equal to 10–12 at the media access controller/physical signalling (MAC/PLS) service interface
• Appropriate support for OTN
• Support for a MAC data rate of 100 Gb/s
• Physical layer specifications that support 100 Gb/s operation over:
• At least 40 km on single mode fiber (SMF)
• At least 10 km on SMF
  At least 100 m on optical multimode 3 (OM3) multimode fiber (MMF)
• At least 10 m over a copper cable assembly

The sample architecture shown in Fig. 1 depicts a 100 GbE MAC, denoted as 100 GE, with ten times 10G physical coding sublayer (PCS) I/O for 10-km single mode fiber. The challenge for the 100 GbE 10-km SMF standards effort is to enable the development with little development risk of first-generation transceivers using existing technology, and also to enable follow-on transceivers with significantly reduced cost, power and size as new technologies become available. The choice of wavelength operating point, spacing, and count should accommodate existing and future technologies. It is desirable for the 100 GbE 10-km standards activities to be incrementally extendable to 40 Gb/s. Both 100 Gb/s and 40 Gb/s links are critical for linking servers and switches. The server/switch connection is today mostly at 40 Gb/s due to software and bus limitations, while switch/switch connection runs at 100 Gb/s since the bandwidth needs to match the increasing node performance.
Reaching consensus at their last meeting in July 2007, HSSG members agreed to move 40 Gb/s and 100 Gb/s standardization efforts forward on PAR P802.3ba. The group was reacting to five criteria:

• Economic feasibility-includes such items as known cost factors, reliable data, reasonable cost performance, and installation cost.
• Broad market potential-includes a broad set of applications and multiple vendors and end users, as well as a balanced cost that considers LANs versus attached stations.
• Compatibility-considers the 802 series of standards and that standards developed shall be in conformance with the IEEE 802.1 architecture, management, and interworking documents as follows: 802. Overview and Architecture, 802.1D, 802.1q, and parts of 802.1f. Any variances in conformance that emerge shall be thoroughly disclosed and reviewed with 802. Each standard in the IEEE 802 family of standards shall include a definition of managed objects that are compatible with systems-management standards.
• Distinct identity-this effort will be substantially different from other IEEE 802 standards and will address one unique solution per problem (not two solutions per problem) and will be user friendly for selection of the relevant specification.
• Technical feasibility-the specification must demonstrate system feasibility, proven technology, reasonable testing and there is confidence in its reliability.

Each of the five criteria above have a distinct set of identifiers that are to be used as a guide to help justify each specific criteria, and provide a framework for the work to be done.

Assuming that the EC (which already approved pre-submission of P802.3ba) and other formal approvals go through as expected, a single Task Force will move forward to develop a joint 40 Gb Ethernet and 100 Gb Ethernet standards document. The first Task Force meeting would be in January 2008. A possible timeline for development of the standard is May 2008, last new proposal; May 2009, last technical change; and May 2010, standard completed.

ITU-T Activities on 100GbE
At the last ITU-T Study Group (SG) 15's plenary meeting in June 2007 many contributions addressed the work on 100 Gb technology for standardization. The basic effort has been for the new bit rate and the format for 100+ Gb/s to be incorporated into ITU-T Recommendation G.709.

As part of this effort, Study Group SG 15/Q11 suggested making the population of the forward error control (FEC) bytes required (not optional) for the new higher bit rate for the OTU4 interface under consideration. This was eventually adopted. There have been proposals to use 32-byte interleaved RS(255,239) for FEC codecs for OTU4 in G.709, instead of the 16-byte interleaving used in the lower bit-rate OTN interfaces. However, the subject matter experts were not convinced of the merit of making the format for ODU4/OTU4 different from that of the lower bit-rate interfaces.

Three contributions were briefly discussed that contained specific ODU4/OTU4 bit-rate proposals. Table 2 reflects the multiplexing structures and bit rates discussed. The 174.261 Gb/s bit rate was included for completeness but not considered a serious possibility for consideration. There was general agreement with the proposal that the bit rate for the new ODU4/OTU4 be kept as close as possible to the 100GbE bit rate (which is still undefined). However, there was no intention to imply support for two-stage multiplexing.
There was also general agreement that optical-interface recommendations should be considered in the development of an ODU4/OTU4 specification. Close coordination with SG 15/Q6 is also envisioned to ensure that any specification takes into account the physical aspects, which are far more critical as the bit rate gets higher.

There is concern in ITU-T about the IEEE 802.3 HSSG definition of 40GbE, since it may not fit into an ODU3 frame structure. With the recent approval by the IEEE HSSG to include 40 GbE as part of the effort, ITU-T SG 15 sent a liaison to the IEEE 802.3 HSSG requesting selection of 40GbE signals that fit within an ODU3 transport structure.
The ITU-T SG15 expressed interest in the additional flexibility that ODU4/OTU4 mapping schemes might offer, but possible management issues were of concern. There were also proposals for the adoption of time slots within the OPU for mapping of multilane 100GbE. But because the IEEE had not agreed on the specification of a multilane 100GbE signal, these proposals had no clear support.

In summary, in standardization of proposed rates for OTU4, ITU-T will closely coordinate its effort with the activities of 802.3 HSSG. The standardization of 100G transceivers has not yet begun, and should be considered as soon as possible to optimize the adoption of 100 Gb/s.

Transmission Media for 100GbE
Standardization efforts are concentrated around proven and deployed fiber technology, explained in more detail in [5]. Both single-mode and multimode fiber interfaces have been considered as candidates for inclusion in the final document.
The technical feasibility and market potential for a 100 Gb/s copper interconnect has been demonstrated by industry. Up to a 5-m reach is consistent with actual intra/interrack distances. Up to a 10-m reach is consistent with high performance computing (HPC) clusters. A 100 GbE standard should permit a high-speed copper interconnect to address these reach requirements [6]. To facilitate next-generation Ethernet, all parts of the system must be considered. Data center links, typically implemented with copper cable, must be included to fully facilitate next- generation Ethernet.

Copper interconnect technology is currently available operating at 10 Gb/s per differential pair. Also, 100 Gb/s copper Ethernet within data centers must be feasible for cost-effective deployment. Current interconnect hardware providing 10-Gb/s serial links can be deployed in parallel for copper 100 GbE by using high-performance components and adjusting the signaling scheme. Standardizing 100 Gb/s is doable with a small number of aggregate lanes (five at 20 Gb/s or four at 25 Gb/s), and by using existing hardware that employs signaling schemes other than NRZ (such as PAM-4 or duobinary).

A number of existing and developing physical-layer specifications should be considered for higher-speed copper Ethernet, and this is part of the standardization effort in IEEE 802 HSSG per PAR P802.3ba.

Thinking Beyond 100GbE
Various analyses of traffic growth and its impact on interface development have noted that interface speeds do not keep up with Internet traffic growth. It has been also noted that 40 GbE would be a good standard to have right now, while 100 Gb/s is needed before 2009. Some industry experts believe that a standard describing 1 Tb/s should even be available before the 2012–2013 timeframe. Such a need may vary among the telecom carriers, which have different topologies and geographies. In addition, standards should include high-speed transmission over DWDM.

Additional Reading
[1] B. Trubey, "Future Market Potential for 100G Ethernet, An MSO Perspective by Time Warner Cable," IEEE 802.3 Plenary meeting, HSSG subcommittee, Mar. 2007, Orlando, Fla.
[2] http://www.ieee802.org/3/hssg/public/index.html

JOHN MCDONOUGH is an industry veteran with more than 25 years of supervisory, managerial, and technical experience in both the North American and international telecommunications industry. He spent a number of years working for both carriers and vendors. Some of his previous posts included senior technical management positions with Nynex (now Verizon) and Cisco Systems. At the same time, McDonough was very active in different industry standards forums, where among other posts, he served as vice president of the Optical Internetworking Forum's board of directors; chair and vice chair of the Alliance for Telecommunication Industry Solutions (ATIS) Technical Subcommittee T1M1 (now TMOC); vice chair of ATIS's OPTXS Optical Hierarchical Interface Working Group; and vice chair of the internal NGN Network Management Subcommittee. He joined NEC Corporation of America in 2006 and serves as the company's principal representative in industry forums dealing with optical networking.