High energy physicists set new record for network data transfer

Building on eight years of record-breaking developments, and on the restart of the Large Hadron Collider (LHC), an international team of high-energy physicists, computer scientists, and network engineers led by the California Institute of Technology (Caltech) joined forces to capture the Bandwidth Challenge award for massive data transfers during the SuperComputing 2009 (SC09) conference held in Portland, Oregon.

Caltech's partners in the project include scientists from Michigan (UM), Fermilab, Brookhaven National Laboratory, CERN, San Diego (UCSD), Florida (UF and FIU), Brazil (Rio de Janeiro State University, UERJ, and the São Paulo State University, UNESP), Korea (Kyungpook National University, KISTI), Estonia (NICPB) and Pakistan (NUST).

Caltech's exhibit at SC09 by the High Energy Physics (HEP) group and the Center for Advanced Computing Research (CACR) demonstrated applications for globally distributed data analysis for the LHC at CERN. It also demonstrated Caltech's worldwide collaboration system, EVO (Enabling Virtual Organizations; http://evo.caltech.edu), developed with UPJS in Slovakia; its global-network and grid monitoring system MonALISA (http://monalisa.caltech.edu); and its Fast Data Transfer application (http://monalisa.caltech.edu/FDT), developed in collaboration with the Politechnica University (Bucharest). The CACR team also showed near-real-time simulations of earthquakes in the Southern California region, experiences in time-domain astronomy with Google Sky, and recent results in multiphysics multiscale modeling.
 
The focus of the exhibit was the HEP team's record-breaking demonstration of storage-to-storage data transfer over wide area networks from two racks of servers and a network switch-router on the exhibit floor. The high-energy physics team's demonstration, "Moving Towards Terabit/Sec Transfers of Scientific Datasets: The LHC Challenge," achieved a bidirectional peak throughput of 119 gigabits per second (Gbps) and a data flow of more than 110 Gbps that could be sustained indefinitely among clusters of servers on the show floor and at Caltech, Michigan, San Diego, Florida, Fermilab, Brookhaven, CERN, Brazil, Korea, and Estonia.
 
Following the Bandwidth Challenge, the team continued its tests and established a world-record data transfer between the Northern and Southern hemispheres, sustaining 8.26 Gbps in each direction on a 10 Gbps link connecting São Paulo and Miami.

By setting new records for sustained data transfer among storage systems over continental and transoceanic distances using simulated LHC datasets, the HEP team demonstrated its readiness to enter a new era in the use of state-of-the-art cyber infrastructure to enable physics discoveries at the high energy frontier, while demonstrating some of the groundbreaking tools and systems they have developed to enable a global collaboration of thousands of scientists located at 350 universities and laboratories in more than 100 countries to make the next round of physics discoveries.
  
"By sharing our methods and tools with scientists in many fields, we hope that the research community will be well-positioned to further enable their discoveries, taking full advantage of current networks, as well as next-generation networks with much greater capacity as soon as they become available," says Harvey Newman, Caltech professor of physics, head of the HEP team, colead of the U.S. LHCNet, and chair of the U.S. LHC Users Organization. "In particular, we hope that these developments will afford physicists and young students throughout the world the opportunity to participate directly in the LHC program, and potentially to make important discoveries."

The record-setting demonstrations were made possible through the use of fifteen 10 Gbps links to the SC09 booth provided by SCinet, together with National Lambda Rail (11 links including six dedicated links to Caltech) and CENIC, Internet2 (two links), ESnet, and Cisco. The Caltech HEP team used its dark-fiber connection to Los Angeles provided by Level3 and a pair of DWDM optical multiplexers provided by Ciena Corporation to light the fiber with a series of 10G wavelengths to and from the Caltech campus in Pasadena. Ciena also supported a portion of the Caltech traffic with a single serial 100G wavelength running into the SC09 conference from the Portland Level3 PoP, operating alongside other links into SC09 from Portland. Onward connections to the partner sites included links via Esnet and Internet2 to UCSD; FLR to University of Florida as well as FIU and Brazil; MiLR to Michigan; Starlight and USLHCNet to CERN; AMPATH, together with RNP and ANSP, to Brazil via Southern Light; GLORIAD and KREONet to Korea; and Internet2 and GEANT3 to Estonia.
 
The network equipment at the Caltech booth was a single heavily populated Nexus 7000 series switch-router provided by Cisco and a large number of 10 gigabit Ethernet server-interface cards provided by Myricom. The server equipment on the show floor included five widely available Supermicro 32 core servers using Xeon quad core processors with 12 Seagate SATA disks each, and 18 Sun Fire X4540 servers, each with 12 cores and 48 disks provided by Sun Microsystems.

One of the features of next-generation networks supporting the largest science programs-notably the LHC experiments-is the use of dynamic circuits with bandwidth guarantees crossing multiple network domains. The Caltech team at SC09 used Internet2's recently announced ION service-developed together with ESnet, GEANT and in collaboration with US LHCNet-to create a dynamic circuit between Portland and CERN as part of the bandwidth-challenge demonstrations.