Data Transfer Nodes Yield Results

The ability to reliably move and share data around the globe is essential to scientific collaboration, that’s why three Department of Energy (DOE) Scientific Computing Centers—Argonne and Oak Ridge Leadership Computing Facilities, and the National Energy Research Scientific Computing Center (NERSC)—have teamed up to focus on optimizing wide area network (WAN) transfers.

This ongoing effort began several years ago when each site deployed dedicated transfer nodes (DTNs), optimized for carrying data between the DOE facilities. Today, engineers from each site continue to meet regularly with DOE’s Energy Sciences Network staff (ESnet) to develop strategies for optimizing bandwidth performance. This collaboration is called the Data Transfer Working Group, and their effort is having a huge impact on scientific research. 

“We typically compute at one facility and analyze data at another,” says Chesley McColl, Associate Scientist at National Oceanic and Atmospheric Administration’s (NOAA’s) Earth System Research Laboratory. “The DTNs save us so much time. In fact, we recently moved an 80 terabyte dataset from the high performance storage system at Oak Ridge to NERSC in just two months, without the DTNs this would have taken as long as six months.”

Jeff Porter of NERSC’s Outreach, Software and Programming Group and the Relativistic Nuclear Collisions Department in the Lawrence Berkeley National Laboratory’s Nuclear Science Division, notes that because the DTNs mounted on NERSC’s filesystems proved so useful to other researchers, the PDSF user community decided to mount one on their cluster too. PDSF is a networked distributed computing cluster designed primarily to meet the detector simulation and data analysis requirements of Physics, Astrophysics and Nuclear Science collaborations. Although PDSF is housed at NERSC, it is different from the center’s other systems because it does not rely on allocations. Instead, each collaboration or group receives shares in the batch system proportional to their contributions to shared resources like compute nodes or staffing.

“NERSC’s DTNs are important benchmarks for our transfer nodes. We know that NERSC has a vested interest in monitoring WAN transfers, and so we run tests of those nodes and compare them to the PDSF transfer node to make sure that our hardware is working well,” says Porter.

 Because the perfSONAR network monitoring applications are deployed on all of the DTNs, users can now identify “choke points” in their data movements and work with NERSC staff to fix it. “In the past, there have been issues with WAN transfers to PDSF and finding the problem was like finding a needle in a haystack,” adds Porter. “The great thing about the DTNs is that, with a well defined and maintained configuration, if there is a problem we can look at the transfer and system logs, identify the problem and fix it.”

Globus Online Enabled

This past year, NERSC also enabled Globus Online on their DTNs. This hosted service automates tasks associated with moving files between sites, or “endpoints,” including retrying failed transfers, recovering from faults automatically whenever possible, and reporting status. So users simply log into the system, choose a start and end point, hit transfer and the system does the rest.

“Our scientists are familiar with tools like scp and GridFTP, but with Globus Online we can offer a much simpler and faster method for moving data. Globus Online actually makes web-based data syncing an easy, nearly trivial process, so you don’t have to be an IT or middleware expert to move your files,” says David Skinner, of NERSC’s Software and Programming Group.

 “Globus online saves so much time and trouble, you don’t need any system admin skills, it just works,” says McColl, who notes that she often uses this tool to move data between NOAA and NERSC.