The Progress of OpenStack in Technical Computing
For a growing number within the high-performance computing (HPC) community, the future of cloud computing lies with an open-source cloud operating stack that controls pools of compute, storage, and networking resources throughout a data centre, via a dashboard or orchestration API. Dubbed OpenStack, this collection of software tools enables administrators to maintain control, while empowering users to select resources through a web GUI or programmatically. As organisations seek to evaluate the potential of OpenStack within engineering design and technical computing environments, the question arises as to its suitability for this purpose.
Continuing the work of virtualisation technologies pioneered by companies like VMware, OpenStack has been viewed by some as being a cheap ‘drop-in’ replacement for VMware – a view that that Nathan Harper, IT Systems Lead at the Centre for Modelling and Simulation (CFMS), is keen to correct. ‘It’s important to note that the participants themselves in OpenStack don’t see the technologies as being a replacement for VMware,’ he says, ‘and VMware itself is an active participant in OpenStack, with the company’s vSphere available to use as an underlying infrastructure.’
VMware’s strength has always been in enterprise virtualisation. In addition to offering a great deal of resiliency and simple, straight-forward set-up, the environment is extremely user-friendly. VMware’s technology is very well supported by both hardware and software vendors and has become a defacto standard. The prohibitive factor for some organisations, however, is one of cost. Vmware’s proprietary licensing doesn’t scale easily and so adding additional capacity can become costly.
It is at this point – where organisations are questioning for they can make virtualisation work for them – that technologies like OpenStack have entered the market. Many aspects have been accrued by the project – orchestration, for example, ensures that rather than creating a virtual machine, organisations can create an entire environment at the push of a button, and do so repeatedly. Another benefit of orchestration is auto-scaling, which enables resources to be automatically added to an application when the load goes up, and removes those resources when the load goes down – all without human intervention. In addition to eliminating the burden of resource planning, this approach frees organisations from having to buy or rent additional infrastructure for use during peak periods of demand.
‘OpenStack offers greater degrees of control and flexibility in terms of IT infrastructure, from very typical enterprise environments right through to HPC,’ Harper comments. ‘At CFMS, we’re particularly interested in OpenStack because there are many people looking to invest in the software and its flexibility, but there is the question of whether it’s necessarily appropriate for everything. In particular, is it appropriate to discard your traditional in-house HPC service and replace it with one based on OpenStack?
‘Those are some of the considerations people really need to bear in mind. And they’re reasonable questions to ask because if an organisation is going to invest in a technology, the more of their infrastructure that technology can manage, the better. The issue is that it isn’t appropriate for everyone.’ Harper added that in order to tackle that question, a first port of call would be to speak with people who have already deployed it. ‘As an open source group of technologies, OpenStack comes with that open source mentality that means people are generally very happy to discuss it. Already in the UK, there’s a thriving user group for people who are looking closely at these technologies,’ he continued. ‘Furthermore, there are areas of research that university teams are working on that are being enabled by OpenStack, particularly areas like life sciences where they’re looking for repeatability. OpenStack can certainly help solve some of those challenges.’
For organisations wishing to evaluate OpenStack technologies for themselves, the initial barriers of entry are relatively low. Companies like Mirantis are heavily involved in the OpenStack environment, and provide users with the ability to set up their own tests. Within roughly one hour of downloading Mirantis’ technology, people can have a working virtual OpenStack test environment within which to evaluate how well it works with specific applications, workflows, processes and protocols. ‘The drawback is that as soon as people want to start delving more deeply into it, they find that OpenStack is definitely a more complicated beast compared to the likes of VMware vSphere,’ said Harper, emphasising that this is because it’s not just a single piece of technology – it’s a group of interconnected projects, which makes it far more complicated to set-up and run in a more production-like environment. ‘One of the core considerations for any organisation looking to deploy OpenStack is whether they have the relevant skills in-house, or whether they’ll need to work with an external organisation. These skilled people are very much in demand as there aren’t many true OpenStack experts out there, particularly in the UK, and there are lots of organisations interested in their services.’
Within traditional HPC environments, should an organisation want to run a specific application on an environment different to the currently deployed operating system, it becomes far more of a burden on the system administrator. By using OpenStack, however, users can – to a point – select what they want from a drop-down menu and click ‘deploy’. But again the question is whether the technology is the right fit. ‘If you speak to companies like VMware or Red Hat, they’ll tell you that virtualisation is 97-98% based on the performance of bare metal, and as there’s practically no drawback, why not use it?’ Harper commented. ‘What they don’t tell you is that the performance level relates to RAM and CPU, but when it comes to network performance or disk performance you won’t see figures nearly that high. Although this is not a problem for single node/single core ‘throughput-type’ computing, but when working with large scale HPC applications where networking performance and latency is key, that can become critical.’
According to Harper, there have been some very interesting conversations about how to deal with those issues. Mellanox, for example, is active in both HPC and OpenStack and as such has put technologies into its software stack and hardware to help solve some of these problems. One such technology is SR-IOV, a single root IO virtualisation that helps provide a virtual machine with accelerated access to the network hardware. Traditionally this would have been achieved with software – and would have included all the associated overheads that would entail. This challenge is also being tackled by a consortium of UK universities that are looking to deploy OpenStack for scientific computing. These universities have been testing the appropriateness of the platform for varying types and levels of workload. The verdict, thus far, has been that by using the accelerated performance some of the virtualisation overhead can be removed, though it does become more challenging to do that deployment. This need for end users to be experts does in fact step away from the concept of anyone being able to pick up and use the technology, and this goes against what many are hoping OpenStack will eventually achieve.
‘OpenStack is great in that people think they can simply choose a specific machine with so many CPUs, RAM, disks, etc., and go away and use it,’ Harper explained. ‘To a certain extent this is true but in much the same way as you can currently do that with Amazon’s cloud service. Getting it to work well is the challenge, and that’s where you need specialist knowledge.’
In addition to deploying an OpenStack environment, CFMS has been working with OpenStack partners to guide them in selecting the appropriate technology to deploy. It’s important to note that at this point, the Centre still views OpenStack for HPC as a work in progress. ‘There are some real technological hurdles that need to be addressed before OpenStack makes real sense,’ Harper commented. ‘These hurdles are being looked at as the OpenStack’s consortium involves all the right people, from the hardware providers and software providers to those doing the middleware, and they all have the same goal – to ensure the technology succeeds in every area. That’s something we at CFMS are also very much involved in and we now have a test and development environment that we’re using for both general private cloud and HPC use.
In certain areas, OpenStack is very mature but in others is not quite there yet. But the future is looking very bright from the technology’s point of view, and there are some very large enterprises investing significant amounts of money and effort into helping OpenStack achieve its potential. ‘Our role at CFMS is to bridge the two worlds of general enterprise IT and HPC to ensure OpenStack is everything we hope it will be,’ said Harper. ‘One of the most exciting elements to this is that we’re not just talking about a new technology, it’s a new way of thinking. In many ways, cloud enabled a shift in the way organisations work and the same can be said of OpenStack, but the crucial difference is that OpenStack can be deployed in-house, providing ultimate control over the environment.
‘Looking to the future, I think the question isn’t when OpenStack will be ready for the industry, but rather when the industry will be ready for OpenStack. For many, the adoption of new technologies or shift cultures is like trying to turn a fuel tanker! It’s often a slow and somewhat precarious situation. Even with evangelists and enthusiasts within the organisation, getting security policies updated, forging relationships with new vendors, and a whole host of other activities that need to be completed before OpenStack can truly be evaluated can put some off making the change,’ Harper commented, adding that in order to accelerate the process it makes perfect sense for them to partner with organisations who have already gone through deployment.
‘At CFMS we will continue working with OpenStack as the potential here is truly exciting,’ Harper concluded. ‘Whether it will fully realise that potential is another question, but it’s one that people will only be able to answer if they continue to look and invest in the technology. If it all comes together, it really could be a game-changer for the industry.’
