BeeGFS, the parallel file system, originally developed by the Fraunhofer Institute for performance-critical applications, has now been released as a free open source product.
Parallel file systems have long been heralded as critical to the future of HPC applications because a parallel file systems performance scales as the number of servers and disks in the system increases. This means that the systems can be scaled from small clusters up to enterprise-class systems with thousands of nodes depending on the performance required by the users.
The release of the open source edition of BeeGFS, previously FhGFS (Fraunhofer Gesellschaft file system) has been long awaited by the BeeGFS community, as it allows users to finally get into the underlying source code and begin optimising it for their specific requirements.
The first official announcement to make BeeGFS open source came during the International Supercomputing Conference (ISC) 2013. This was in the context of the European exascale project DEEP-ER, which is designing and testing solutions to address the extreme I/O requirements of exascale computing. In an exascale system, the different software and hardware layers have to work together very efficiently to achieve maximum scalability. Thus, making the sources of BeeGFS available is one logical step to enabling efficient integration of all layers of an exascale compute stack.
DEEP-ER is a European Commission project that was started in October 2013. DEEP-ER extends the Cluster-Booster architecture of the DEEP project that uses different system architectures to handle different parts of computing jobs, which is then all brought together through a highly scalable I/O system with the addition of an efficient mechanism to recover application tasks that fail due to hardware errors.
However, it is not just the DEEP-ER project that has been awaiting the release of BEEGFS as an open source product, explained Sven Breuner, CEO of ThinkParQ, the company behind BeeGFS. ‘While some of our users are just happy with the fact that BeeGFS is so easy to install and doesn’t require much attention, others really want to understand exactly what is happening under the hood to further optimise the runtime of their applications, improve their monitoring or port it to other platforms like BSD. Also, being able to build BeeGFS for non-x86 architectures like ARM and Power is another important aspect that the community has been waiting for.’
There are other projects which are also addressing the requirements of exascale computing such as the ExaNest project, which aims to explore the use of ARM technologies as part of a potential exascale HPC system. Thus, the BeeGFS team is also participating in ExaNeSt, a new European exascale project, which is specifically focused on getting the ARM ecosystem ready for performance-critical workloads. ‘Although BeeGFS can already run out of the box on ARM systems today, this project will give us the opportunity to make sure that we can deliver the maximum performance on this architecture as well.’, stated Bernd Lietzow, BeeGFS’ head representative on the ExaNeSt project.
The ExaNeSt (Exascale Networking and Storage) project is funded by the European Commission to develop and prototype evolved solutions for interconnection networks, persistent storage, and cooling, specifically on ARM systems. The project started in December 2015. Real-world HPC applications will be used and tuned by the project team to evaluate and improve the developed solutions.
For people interested in learning more about BeeGFS, there is an upcoming user meeting on May 18 and 19 in Kaiserslautern, Germany. The Early Bee deadline for registration ends on 4 May.
