Interview: A*STAR to Spotlight Singapore at ISC High Performance

The ISC High Performance conference brings out the very best in international supercomputing, and this year will be no exception with a host of content focused on HPC in Asia. Singapore has been a hotbed of HPC activity of late, so we caught up with Marek Michalewicz, who was recently named Chief Executive Officer at A*STAR Computational Resource Centre.

insideHPC: What is the mission of A*STAR and what kind of  supercomputing resources do you have on hand?

Marek Michalewicz, A*STAR

Marek Michalewicz, A*STAR

Marek Michalewicz: A*STAR or Agency for Science, Technology and Research is the largest Government funded research organization in Singapore with fourteen research Institutes and several Research Consortia, grouped thematically under the Biomedical Research Council and the Science and Engineering Research Council. It employs over 5300 staff. A*STAR official mission states: “We advance science and develop innovative technology to further economic growth and improve lives.

A*STAR Computational Resource Centre (A*CRC) serves all HPC and related storage and networking needs of A*STAR. We have an interesting assortment of hardware: older HP and Fujitsu clusters with 1000 and 4,500 cores each, an IBM Power 7 cluster with ~1000 cores, an SMP SGI UV 1000 with ~2600 cores and 15 TBytes shared memory, and the smallest Supercomputer in the World: BlueGene Q (single mid-plane with 8000 cores). It is perhaps the only half-filled BlueGene Q system I am aware of, but having this computer allowed A*STAR scientist to reach previously impossible level of parallelism in running their codes. Our BlueGene Q has a very high utilization, reaching 100% almost all the time, and codes like NAMD scale easily to the entire machine. 

Back in 2009 we had rather modest resources on some 10 TFLOPS combined in almost ten small systems, since then we have been following Moore’s Law, each time doubling all of our combined resources every two years or so – except when we got special systems, like Power 7 or UV 1000 – where size was not the primary consideration. However, over the last three years we effectively had a “freeze” on new acquisitions in lead-up time to the newly formed National Supercomputer Centre (NSCC). During my time with A*CRC HPC resources grew from 10 TFLOPS in 2009 by about a factor of two each 2 years, reaching 220 TFLOPS in 2013, and, with the new Supercomputer to be bought within the next six weeks, well over 1 PFLOPS this year.

insideHPC: What is new in Singapore and at A*STAR?

Marek Michalewicz: This year Singapore celebrates it’s 50th Anniversary. A per capita income of less than USD$500 half-a-century ago grew to more than USD $50,000 today, an improvement of more than a hundred times. Singapore will be celebrating Science@50 to mark half a century of excellent science and its contribution to economic growth and positive societal outcomes for Singapore.

In October, we will be officially opening Phase 2A of Fusionopolis, the counterpart campus to Biopolis. With this development these two combined campuses, only about 1.5km apart, will accommodate about 9,000 scientists. 

Also in early October we will inaugurate the National Supercomputer Centre with opening refitted NSCC datacentre and a Petascale supercomputer. The tender for this new machine has just closed few days ago and currently we are evaluating the bids. 

I can assure you – it is a very exciting time for Supercomputing in Singapore. 

The new, so-called “fourth University,” the Singapore University of Technology and Design, created jointly with MIT, has just moved to its brand new beautiful campus in the West part of the island.

Reporting on quite recent, interesting events, in March we ran the first “Supercomputing Frontiers 2015” conference here. We were very fortunate to attract a truly stellar group of Supercomputing thought leaders. Among our keynote speakers were Jack Dongarra, John Gustafson, Thomas Sterling, Satoshi Matsuoka, CS Chang, Jeroen Tromp, Scott Klasky, Rick Stevens, Robert Harrison and many others. It was a small, single track meeting which allowed good discussions and interactions among the participants. The select papers will be published in International Journal of Supercomputing Frontiers and Innovations in the later part of this year. We plan to continue this conference as a regular event on the calendar.

Last year, for the first time, Singaporean students entered the international Student Cluster Competitions. A*CRC took active role in promoting the competitions at local Universities and in coaching and teams’ preparations.

Two teams, from National University of Singapore (NUS) and Nanyang Technological University (NTU) entered ASC14 competition in China and the NTU team placed the second among 82 other teams. At SC14 in New Orleans, another NUS team placed sixth among 12 teams. Now a fourth Singapore student team in less then a year, this time from NTU, qualified among the sixteen finalist of ASC15 from the field of 153 teams, and will be competing in Taiyuan City, Shanxi Province, China in May. 

insideHPC: You had an elaborate demo at SC14 in New Orleans. Can you describe the demo and what has happened since?

Marek Michalewicz: Last year we started the InfiniCortex project, to demonstrate global range connectivity between HPC class systems using InfiniBand. 

In about six months leading to SC14 we managed, with incredible engagement of almost thirty organisations, to build the first prototype of InfiniCortex. It connected three continents: Australia, Asia, and North America; four countries: Australia, Japan, the US and Singapore. 

The nodes of InfiniCortex emanating from Singapore, reached National Computing Infrastructure in Canberra, Australia, Tokyo Tech in Japan  and Georgia Tech in Atlanta, with end-point at the Expo hall at SC14. Not only we demonstrated trans-Pacific 100Gbps InfiniBand connectivity but we were fortunate to have an excellent group of collaborators from ORNL, Rutgers University, U. Tennessee, Princeton plasme Physics Laboratory, Stony Brook and Georgia Tech. With support and involvement of Scott Klasky and his collaborators from ORNL and elsewhere we run codes and ADIOS enabled workflows concurrently across continents.

Some of the best examples were C.S. Chang’s (Princeton Plasma Physics Lab) plasma code which simulates and investigates electrostatic turbulence and effects present in torus-shaped fusion systems, Tahsin Kurc’s (Stony Brook) code to rapidly analyze tissue slide images (40Kx40K to 120Kx120K pixels in resolution) in order to quantitatively assess disease condition. Matthew Wolf (Georgia Tech) has implemented NAMD code with staging of perts of computations in Singapore, Georgia Tech and finally visualisation of the results at SC14 expo floor. Our Australian partners, Jakub Chrzeszczyk and Andrew Howard and coleagues, demonstrated InfiniCloud – on-demand provisioning of arbitrary scale HPC Cloud instances, enabled with InfiniBand fabric, and available 30,000 km away (we had to cross Pacific twice: AU-USA-Singapore to use the undersea cables provisioned for this experiment!).

The NCI team demonstrated trans-continental genomics workflows on both our very large SMP machine in Singapore and on their InfiniCloud instances in Canberra. They have demonstrated 30-fold reduction of transfer time of genomics data files using 30,000km InfiniBand over traditional TCP/IP: for a 1.2 TBytes file the time 0f 12.5 hours was reduced to 28 minutes. 

We are continuing this work with ORNL/Georgia Tech, ANU/NCI and TokyoTech partners – but our network had to be scaled down to 10Gbps . The 100Gbps trans-Pacific link was donated free of charge by Tata Communication specifically for SC14. I need to acknowledge special role of Obsidian Strategics – creators of Longbow InfiniBand range extenders who provided us with dozens of pieces of demo equipment, and DDN who brought in almost half PBytes Luster storage to our demo booth at SC14.

This year we have enlarged our partnership to include University of Reims and University of Lille in France and Poznan Supercomputing and Networking Centre in Poland as well as ICM in Warsaw, Poland. We are beginning to test applications on a 10 Gbps InfiniBand link going directly from A*CRC in Singapore, through trans-Pacific undersea cables, USA, trans-Atlantic to Amsterdam, and Poznan, Poland. 

Again, I must thank fantastic teams of ESnet and GEANT partners and engineers who are making this possible. 

Our InfiniCortex project has been noticed in Singapore and just a week ago it scored three awards: 2015 A*STAR Award for Innovation, Ministry of Trade and Industry 2015 Gold Award for Innovative Projects, and FutureGov Singapore Award in Technology Leadership category.

insideHPC: What are the biggest computational challenges in the Singapore region?

Marek Michalewicz: The issues of national concern in Singapore are urban development and related problems of very densely populated island, water supplies and water quality, well-being of the citizens, aging population and related health and medical services, mass urban transportation, and sustained and profitable economic growth. All of those issues require deep scientific or engineering insight and require computational models and scenario studies at great scale. For this reason, in recent years, with funding provided by the National Research Foundation, several new substantial research centres were established: Future Cities Labs (in partnership with ETH Zurich), Earth Observatory of Singapore, Singapore Centre on Environmental Life Science Engineering (meta-genomics related to water purification), Photovoltaic Centre and several others. The National Supercomputer Centre will be able to provide computational resources to all these institutes. 

There are also serious industrial players with large local R&D facilities who require supercomputing resources, especially in aerospace and in bio-medical sectors. Examples include Rolls-Royce, Proctor & Gamble, Novartis and many others.

insideHPC: What are your plans for ISC High Performance in July?

Marek Michalewicz: It will be a busy week, but I would like to highlight three things in particular:

  • On Monday, July 13th I will be chairing a session titled “Understanding Urban Development through HPC.” There will be two very interesting talks on how computational sciences, big data analysis and sensing are helping to shape urban development in Singapore and in Fukuoka, Japan.
  • At “HPC in Asia” session on Wednesday, July 15th, Prof TAN Tin Wee, the Chairman of A*CRC  will give a short report on Singapore scene and recent developments. By that time we will be able to share who the winning NSCC bidder is, as well as the details of our new Petascale system.

  • A*CRC, together with the nascent NSCC, will host an expo booth, where, together with our new French and Polish partners, we will demonstrate expanded InfiniCortex – with several new applications running over it. We will also have InfiniBand routing implemented using Obsidian Crossbow devices and BGFC software – which will enable us to design and build a real topology of globally connected, concurrent HPC resources. NB: We invite any interested party to work with us on the InfiniCortex project so it will reach a critical mass and ignite a truly global “chain-reaction.”

Registration is now open for ISC High Performance, which takes place July 12-16 in Frankfurt.

Sign up for our insideHPC Newsletter.