OpenIO Achieves Record 1 Tbps Write Speed

Print Friendly, PDF & Email

French startup OpenIO recently announced record performance with Criteo, the advertising platform for the open Internet. OpenIO demonstrated the performance and scalability of their object storage solution by deploying their technology on a cluster of more than 350 physical servers provided by Criteo. The benchmark made it possible to cross the symbolic threshold of writing up to one terabit of data per second, and even to exceed it since the useful throughput observed is 1.372 Tbps.

Since the creation of OpenIO, we have placed the issue of performance at the very center of what we do,” explains Laurent Denel, CEO and co-founder of OpenIO. “Once we solved the problem of hyper-scalability, it became clear that data would be manipulated more intensively than in the past. This is why we designed an efficient solution, capable of being used as primary storage for video streaming (as is the case with our customer Dailymotion), or to serve increasingly large datasets for Big Data use cases.”

In recent years, due to the exponential growth in the volume of data collected by companies, their choices of storage technologies have often been guided by two criteria: the price per gigabyte and scalability – the ability to easily increase the capacity of a storage platform to create its “data lake” without creating data silos.

It is now possible to extract value from this data, and this extraction is becoming one of the main growth drivers for many companies. Also, the concern today is less to optimize the cost of archiving data than to make their processing possible by voracious Machine Learning and Deep Learning algorithms. After the era of Data Archiving and Data Sharing, we are entering the era of Data Processing.

The high performance that OpenIO achieved on Criteo machines under production conditions, confirms the design of the OpenIO object storage software for new data uses, in particular the massive exploitation of data by AI algorithms on Big Data / HPC clusters. Inaugurating the #TbpsChallenge with this record, OpenIO invites other market players to put their technology to the test (on commodity servers and tuned for production!).

Criteo is pleased to have been a part of this benchmark, which gave OpenIO the opportunity to demonstrate how their technology can maintain consistent high performance while adding new nodes to scale massively” said Stuart Pook, Senior Site Reliability Engineer at Criteo. “They were able to achieve a write rate close to the theoretical limits of the hardware we made available.”

Criteo handles large volumes of data and has built a unique Big Data platform with several thousand nodes. Criteo’s team of engineers provided more than 350 machines from its infrastructure, recently racked in one of their datacenters, but not yet put into their production. An invaluable opportunity for OpenIO, because this cluster made up of commodity storage servers allows them to reach an order of magnitude that is truly ‘hyper-scale’.

About OpenIO

Founded in 2015, OpenIO develops a hyper-scalable, open source object storage solution that is ideal for Big Data, AI and HPC use cases. OpenIO’s grid of nodes architecture and unique ConscienceGrid technology enable our object store to scale instantly without rebalancing data and while maintaining consistent high performance. OpenIO supports S3 and can be deployed on-premise, cloud-hosted or at the edge. It can run on any hardware and has already attracted more than 40 customers worldwide, which include among others, Dailymotion, the CEA, and the IIJ (Internet Initiative Japan) service provider.

Visit OpenIO at booth #367 at SC19 in Denver.

Download the detailed benchmark report on crossing the Terabit threshold

Sign up for our insideHPC Newsletter