Creators of MLPerf Launch MLCommons Consortium for Machine Learning Benchmarks, Metrics, Datasets, Models and Best Practices

Print Friendly, PDF & Email

SAN FRANCISCO – Dec. 3, 2020 – Today, open engineering consortium MLCommons launched an industry-academic partnership to accelerate machine learning innovation and broaden access to this critical technology for the public good.

MLCommons will focus on:

  • Benchmarks and Metrics – that deliver transparency and a level playing field for comparing ML systems, software, and solutions, e.g., MLPerf, for machine learning training and inference performance.
  • Datasets and Models – that are publicly available and can form the foundation for new capabilities and AI applications, e.g. People’s Speech, the world’s largest public speech-to-text dataset.
  • Best Practices – e.g. MLCube, a set of common conventions that enables open and frictionless sharing of ML models across different infrastructure and between researchers and developers around the globe.

The non-profit organization initially formed as MLPerf has assembled a founding board that includes representatives from Alibaba, Facebook AI, Google, Intel, NVIDIA and Professor Vijay Janapa Reddi of Harvard University; and a broad range of more than 50 founding members. The founding membership includes over 15 startups and small companies that focus on semiconductors, systems, and software from across the globe, as well as researchers from universities like U.C. Berkeley, Stanford, and the University of Toronto.

In its announcement, the organization said MLCommons will advance development of, and access to, the latest AI and Machine Learning datasets and models, best practices, benchmarks and metrics. An intent is to enable access to machine learning solutions such as computer vision, natural language processing, and speech recognition by as many people, as fast as possible.

“MLCommons has a clear mission – accelerate Machine Learning innovation to ‘raise all boats’ and increase positive impact on society,” said Peter Mattson, President of MLCommons. “We are excited to build on MLPerf and extend its scope and already impressive impact, by bringing together our global partners across industry and academia to develop technologies that benefit everyone.”

“Machine learning is a young field that needs industry-wide shared infrastructure and understanding,” said David Kanter, Executive Director of MLCommons. “With our members, MLCommons is the first organization that focuses on collective engineering to build that infrastructure. We are thrilled to launch the organization today to establish measurements, datasets, and development practices that will be essential for fairness and transparency across the community.”

Today’s launch of MLCommons in partnership with its founding members will promote global collaboration to build and share best practices – across industry and academia, software and hardware, from nascent startups to the largest companies. For example, MLCube enables researchers and developers to easily share machine learning models to ensure portability and reproducibility across a wide range of infrastructure, so that innovations can be easily adopted and fuel the next wave of technology.

The opportunities to apply machine learning to benefit everyone are endless; from communication, to healthcare, to making driving safer. To foster the ongoing development, implementation, and sharing of Machine Learning and AI technologies, and to measure progress on quality, speed, and reliability, the industry requires a universally agreed upon set of best practices and metrics.

MLCommons is focused on building these tools for the entire ML community. A cornerstone asset within MLCommons is MLPerf, the ML benchmark suite that measures system performance for real applications. With MLPerf, MLCommons is promoting industry wide transparency and making like-for-like comparisons possible.

Machine learning and AI require high quality datasets, as they are foundational to the performance of new capabilities. To accelerate innovation in ML, MLCommons is committed to the creation of large-scale, high-quality public datasets that are shared and made accessible to all.

An early example of such an initiative for MLCommons is People’s Speech, the public speech-to-text dataset in multiple languages that will enable better speech-based assistance. MLCommons has collected more than 80,000 hours of speech with the goal of democratizing speech technology. With People’s Speech, MLCommons will create opportunities to extend the reach of advanced speech technologies to many more languages and help to offer the benefits of speech assistance to the entire world population rather than confining it to speakers of the most common languages.

About MLCommons

MLCommons is an open engineering consortium with a mission to accelerate machine learning innovation, raise all boats and increase its positive impact on society. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding member partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

The MLCommons founding members are from leading companies, including Advanced Micro Devices, Inc., Alibaba Co., Ltd., Arm Limited & Its Subsidiaries, Baidu Inc., Cerebras Systems, Centaur Technology, Inc., Cisco Systems, Inc., Ctuning Foundation, Dell Technologies, d-Matrix Corp., Facebook AI, Fujitsu Ltd, FuriosaAI, Inc., Gigabyte Technology Co., LTD., Google LLC, Grai Matter Labs, Graphcore Limited, Groq Inc., Hewlett Packard Enterprise, Horizon Robotics Inc., Inspur, Intel Corporation, Kalray, Landing AI, MediaTek, Microsoft, Myrtle.ai, Neuchips Corporation, Nettrix Information Industry Co., Ltd., Nvidia Corporation, Qualcomm Technologies, Inc., Red Hat, Inc., SambaNova Systems, Samsung Electronics Co., Ltd, Shanghai Enflame Technology Co., Ltd, Syntiant Corp., Tenstorrent Inc., VerifAI Inc., VMind Technologies, Inc., Xilinx, Gungdong Oppo Mobile Telecommunications Corp., Ltd (Zeku Technology (Shanghai) Corp. Ltd.) and researchers from the following institutions: Harvard University, Indiana University, Stanford University, University of California, Berkeley, University of Toronto, and University of York. Additional MLCommons membership at launch includes LSDTech.

For additional information on MLCommons and details on becoming a member of the organization, please visit http://mlcommons.org/ or contact membership@mlcommons.org.