Lawrence Berkeley National Laboratory Team Achieves 10.6 Gigabits/Second Data Throughput in 10-Gigabit Ethernet Test

BERKELEY, CA -- Although there has been a lot of discussion recently about 10-Gigabit Ethernet capability, actually achieving that level of performance in the real world has been difficult. Until now.

On July 2, 2002, a team from Lawrence Berkeley National Laboratory, which operates some of the world's most powerful computing, data storage and networking resources for the U.S. Department of Energy, teamed with Force10 Networks (switches), SysKonnect (network interfaces), FineTec Computers (clusters), Quartet Network Storage (on-line storage) and Ixia (line rate monitors) to assemble a demonstration system that runs a true scientific application to produce data on one 11-processor cluster, then sends the resulting data across a 10-Gigabit Ethernet connection to another cluster, where it is rendered for visualization.

The result? The team was able to sustain 10.6 gigabits/sec aggregated across two 10-gigabit interfaces, 9.8 gigabits per second on one interface and 960 megabits per second on the other. The measurements were taken from fiber optic taps using Ixia 400 performance analyzers with 10-Gigabit Ethernet interfaces. A total of 58 terabytes of data were transferred over 12 hours of pre-demonstration testing and the demo itself.

With the IEEE's adoption of Standard 802.3ae for 10-Gigabit Ethernet equipment in June, the speed of Ethernet operations has increased by an order of magnitude -- at least on paper. But achieving that 10-fold increase in actual Ethernet performance remains a challenge that can be met only with leading-edge equipment and expertise.

The system was built as a prelude to Berkeley Lab's entry into the High-Performance Bandwidth Challenge at the SC2002 conference of high-performance computing and networking, to be held in November in Baltimore, Maryland. Berkeley Lab teams have won the High-Performance Bandwidth Challenge for two consecutive years. At the SC2001 conference held last November, the LBNL team took top honors, moving data across the network at a sustained rate of 3.3 gigabits in a live computational steering/visualization demonstration involving the Albert Einstein Institute's "Cactus" simulation code (www.cactuscode.org) and Berkeley Lab's Visapult parallel visualization system.

The demonstration was originally put together to demonstrate real-world applications of 10-Gig E capability for a conference scheduled for June. However, the conference was delayed and the Berkeley Lab team decided to put on a public demonstration before taking the system apart and returning the loaned equipment to the vendors.

"The demo turned out to be really successful. Force 10 loaned us the switches, FineTec donated enough computers to make it interesting and we worked with SysKonnect to get very high performance from their network interfaces," said network engineer Mike Bennett. "Quartet provided the network storage for storing the data to be visualized and Ixia supplied the monitoring equipment. The result is we proved that 10-Gig E is a reality, not just a bunch of back-of-the-envelope calculations."

According to Bennett, most demonstrations of 10-Gig E to date have been done to showcase interoperability of components made by different vendors, which is the aim of the IEEE standard. That standard doesn't mean, however, that a system will achieve peak performance.

"What we are demonstrating is that it does work in the real world," Bennett said.

John Shalf, a member of the Berkeley Lab Visualization Group, said that 10-Gig E capability is important for scientific applications.

"Codes like Cactus can easily consume an entire supercomputer, like the 3,328-processor IBM SP at our National Energy Research Scientific Computing Center, or NERSC. The Cactus team ran the code at NERSC for 1 million CPU-hours, or 14 CPU-years, performing the first-ever simulations of the inspiraling coalescence of two black holes," Shalf said.

A high-bandwidth connection allows users to keep up with the huge data production rates of such simulations -- about a terabyte per time step -- and ensure that the code is running properly. Otherwise, mistakes may not be detected until the run is finished -- wasting lots of computer cycles generating bad data.

Remote monitoring and visualization require a system that can provide visualization capability over wide area network connections without compromising interactivity or the simulation performance. The team used Visapult, developed by Wes Bethel of LBNL's Visualization Group for DOE's Next Generation Internet/Combustion Corridor project several years ago. Visapult allows users to use a desktop workstation to perform interactive volume visualization of remotely computed datasets without downsampling of the original data. It does so by employing the same massively parallel distributed memory computational model employed by the simulation code in order to keep up with the data production rate of the simulation. It also uses high performance networking in order to distribute its computational pipeline across a WAN so as to provide a remote visualization capability that is decoupled from the cycle time of the simulation code itself.

To achieve the 10.6 gigabits per second performance, George "Chip" Smith of the team had to work with SysKonnect to overcome a problem resulting from running Linux on the clusters. "When you run Linux with the SysKonnect card, the libraries in the kernel for the SysKonnect cards have a default behavior and run with an average line rate of 600-700 megabits per second," Smith said. "Working with Syskonnect, I was able to change one of the libraries in the kernel and using a recent virtual Ethernet interface module, I was able to get 950 to 1000 megabits off the single interfaces. This enabled us to run this demonstration with one-third fewer machines than it would have without the work on the kernel."

Bennett said the main obstacle to achieving even better performance wasn't the lack of bandwidth, but rather the lack of resources, including the number of machines in each cluster.

"One of the most exciting things is that it scales. If we would have had 50 boxes in the cluster, we could have delivered 50 gigabits," Bennett said. "Now that we've done 10 Gig, it's time to start looking at 100."

Background Information on the Demonstration

The Vendors

About Force10 Networks

Force10 Networks, Inc. is a leader in scalable, high-performance Ethernet solutions. Founded in May 1999, Force10 Networks has raised $168 million from leading venture capital firms New Enterprise Associates, USVP, and Worldview Technology Partners. Force10 Networks is headquartered in Milpitas, California. For more information, call 408-571-3500 or visit the web site at www.force10networks.com.

About FineTec Company

Fine Tec Company was founded in 1992 in Silicon Valley. Our focus is to provide Service Contracts and Network & Computing Equipments to clients such as government agencies, schools and corporations. Our strength is building powerful and reliable our own brand of workstations/servers and storage equipments. Our goal is to provide quick and excellent services and quality products.

About SysKonnect

SysKonnect, a member of the Marvell family, develops, manufactures, supports and markets worldwide server connectivity products for today's client/server environments worldwide. Many years of experience uniquely position us to supply high-end network interface technologies that optimally meet the needs of e-commerce, financial, health care. Imaging and mission-critical business applications. SysKonnect has offices in Germany, the U.K. and in San Jose, CA,

About Ixia

Ixia (Nasdaq: XXIA) delivers powerful, distributed, multiport traffic generators, and performance/conformance analyzers for wire-speed verification of optical networking equipment, LAN, MAN, WAN multi-layer switches and routers, and sophisticated routing protocols. Its products utilize a variety of interfaces—Packet Over SONET, BERT, 10 Gigabit Ethernet, Gigabit Ethernet, 10/100/1000 Mbps Ethernet, and USB. Ixia's network operations applications address the industry's growing requirements for a higher level of control over network optimization, traffic engineering, traffic profiling, and security. Ixia's analysis solutions are distinguished by their accuracy, reliability, high port density, support for emerging protocol standards, conformance adherence, and adaptability to the industry's constant evolution.

About Quartet Network Storage

Quartet Network Storage, Inc. (based in Silicon Valley) designs and develops high-performance scalable networked storage system products. Converging four key technology areas (servers + SAN + NAS + networking), Quartet builds fast and highly interoperable NAS engines to drive data access through multi-gigabit-speed networks. Quartet's Opus-1 product is the first to uniquely fuse the best of IP networking with Fibre-channel RAID connectivity.

The Hardware

The demonstration will consist of two powerful Linux clusters of FineTec computers powered by AMD MP CPUs and using Kingston Value RAM, each with Gigabit Ethernet interfaces from SysKonnect. The clusters are connected together via a pair of Force10 E1200 switch/routers, which are connected over a 10-Gigabit Ethernet interface. One cluster of dual-CPU Linux PCs will run the parallel Cactus simulation code (www.cactuscode.org) and feed data to another cluster of PCs, which run the Visapult parallel volume rendering application.

The Applications

The combination of two scientific applications—Cactus, used to simulate the collisions of black holes and neutron stars, and Visapult, which does real-time parallel volume rendering of the data provided by the simulation code while the simulation code is running on the cluster—will be able to utilize the entire bandwidth of a 10 Gigabit Ethernet network.

The Problems Being Attacked

These applications were designed to solve a specific problem. Einstein's general theory of relativity consists of equations that are among the most complex in the world of physics, containing millions of terms, if fully expanded. The General Relativity Group at Germany's Albert Einstein Institute has developed the Cactus Code for solving these equations on supercomputers in order to simulate the most extreme of astrophysical phenomena, such as the collision of two black holes and the gravitational waves that radiate from that event.

The Cactus simulation codes run on some of the largest supercomputers in the world, including the 3,328-processor IBM SP supercomputer at the U.S. Department of Energy's National Energy Research Scientific Computing Center (NERSC) at Berkeley Lab. NERSC's IBM is the fifth most powerful supercomputer in the world. The simulations are so large, that it is nearly impossible to use traditional visualization tools to see and understand the results of these simulations.

Visapult was developed by the Berkeley Lab/NERSC visualization group under the Next Generation Internet (NGI) program to solve this problem. Visapult is a distributed parallel volume renderer that allows us to use distributed memory/cluster supercomputers and high performance networking resources that are on the same order of scale as the supercomputers that these massive simulation codes consume.

The Visapult code was demonstrated at the SC2000 conference and helped the Berkeley Lab team win the first-ever Bandwidth Challenge award by connecting over the wide area network to a distributed parallel network file system (DPSS). The following year at SC 2001, Visapult connected directly to a running Cactus simulation code in order to provide live visualization and remote steering of the calculation. Cactus also won a Gordon Bell Award for high performance distributed computing during the same event. This permitted a remote user, connected by a wide area high performance network, to interactively view multi-terabyte data streams produced by these large simulations.