Will the Cloud steal my job?

In this special guest feature, Dr Rosemary Francis from Ellexus describes why the HPC Community has little to fear from the rise of Cloud Computing.

Rosemary Francis, CEO of Ellexus

As HPC moves to the cloud should I be worried about my job? This is a question we often hear from HPC cluster managers and system administrators.

In general the answer is: No. There is no cause for concern.

On the surface, it is a fair question. Moving storage from in-house to an external body could be viewed as taking work out of the company. It is a common misconception about the move from big in-house compute systems to the cloud that once the compute moves off-prem there will be nothing to manage.

In fact, the opposite is true. It is common for administrators to manage HPC clusters that are in a different building or even a different country to them; why do we think it’s going to be any different with the cloud? A cloud infrastructure needs just as much management as in-house infrastructure, perhaps more. The possibility of misusing the cloud is a different problem, but with a lot of cross-over skill required.

The sheer number of options in the cloud means that even after the initial set-up, storage teams will have to tune the environment continuously. AWS alone has eight different types of storage that vary hugely in terms of storage size, bandwidth, peak performance and other more complex metrics. It’s not a ‘one set-up’ solution. If you’ve spent time optimizing your on-prem cluster for each different run or set of user expectations, you’re just going to have to do the same in the cloud.

Misuse could also be far costlier on the cloud compared to on-prem, as the pay-as-you-go pricing of cloud compute means costs can spiral very quickly. A recent white paper from Enterprise Management Associates found businesses waste 40-50% of cloud computing spend through bad practices such as unused instances and storage, bad financial decisions related to an expensive cloud region or suboptimal application placement due to unknown workload performance requirements. Surely if this is true, organizations will be far keener to employ administrators to reign in these costs?

In my opinion, ‘the cloud’ is a damaging term. It suggests something magical that we have little control over. If instead of calling it cloud we perhaps started to refer to it as “off-site compute” we would all have a much more realistic relationship with the future.

There’s no doubt that cloud adoption is speeding up. IDG’s Enterprise Cloud Computing Survey 2016 found that for 2017, cloud will account for 28% of total IT budgets. By 2018, a typical IT department will have 60% of applications in the cloud. A lot of organisations are looking to create a hybrid environment; a cluster in-house alongside storage in the cloud that can expand and contract as demands vary.

The survey did also highlight, however, that on average it took 15 months to migrate applications to the cloud and 90% of users experience difficulties due to complexity. This isn’t a quick move and it takes the same expertise to get it right as has always been needed.

If these predictions do come true, HPC storage could soon look very different and people’s jobs will evolve with it. But they will almost certainly not go away.