Moving Massive Amounts of Data across Any Distance Efficiently

Chin Fang from Zettar gave this talk at the Rice Oil & Gas Conference. “The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally.”

SLAC Researchers use ESnet for Record Speed Transfer of 1 Petabyte of Data

“The innovative and versatile nature of ESnet’s OSCARS enabled us to design a unique 5000-mile 100 Gbps loop, which our ESnet colleagues quickly helped establish in 2015,” said Zettar Founder and CEO Chin Fang. “We were thus able to have the data storage clusters and the data transfer nodes co-located in the same rack although they were at both ends of the loop. Physically they were separated by inches, but digitally they are separated by 5000miles. Further the clusters themselves have a self-imposed cap of 80 Gbps.”

Big Data over Big Distance: Zettar Moves a Petabyte over 5000 Miles in 29 Hours

Today AIC announced a world-record in data transfer: one petabyte in 29 hours encrypted data transfer, with data integrity checksum unconditionally enabled, over a distance of 5000 miles. The average transfer rate is 75Gbps, or 94% utilization of the available bandwidth of 80Gbps. “Even with massive amounts of data, this test confirmed once more that it’s completely feasible to carry out long distance, fully encrypted and checksum-ed data transfer at nearly the line-rate, over a shared and production network.”

How Zettar Transferred 1 Petabyte of Data in Just 34 Hours Using AIC Servers

In the world of HPC, moving data is a sin. That may be changing. “Just a few weeks ago, AIC announced the successful completion of a landmark, 1-petabyte transfer of data in 34 hours, during a recent test by Zettar that relied on the company’s SB122A-PH, 1U 10-bay NVMe storage server. The milestone was reached using a unique 5000-mile 100Gbps loop which is a SDN layer over a shared, production 100G network operated by the US DOE’s ESNet.”