BeeGFS

BeeGFS is a parallel le system, developed and op- (#56 on installation), and the Abel[8] cluster at the Uni-
timized for high-performance computing. BeeGFS versity of Oslo, Norway (#96 on installation).
includes a distributed metadata architecture for scal-
ability and exibility reasons. Its most important
aspect is data throughput. BeeGFS is developed
at the Fraunhofer Institute for Industrial Mathematics
2 Key concepts & features
(ITWM) in Kaiserslautern, Germany and was initially
known under the name FhGFS, short for Fraunhofer When developing BeeGFS, Fraunhofer ITWM aimed for
Gesellschaft File System (or Fraunhofer FS). three key concepts with the software: scalability, exibil-
ity and good usability.
On ISC'14 in Leipzig, Fraunhofer presented the new
name for the rst time to the public, even though the re- BeeGFS runs on any Linux machine and consists of sev-
naming process had begun with the founding of a Fraun- eral components that include services for clients, meta-
hofer spin-o. data servers and storage servers. In addition, there is
a service for the management host as well as one for a
The software can be downloaded and used free of charge
[2] graphical administration and monitoring system.
from the projects website.

1 History & usage

BeeGFS started in 2005 as an in-house development at
Fraunhofer ITWM to replace the existing le system on
the institutes new compute cluster and to be used in a
production environment.
In 2007, the rst beta version of the software was an-
nounced during ISC07 in Dresden, Germany and intro-
duced to the public during SC07 in Reno, NV. One year
later the rst stable major release became available.
In 2014, Fraunhofer started its spin-o. The new
company called ThinkParQ[3] will take over sales, cus-
tomer service, and professional support. In this process,
FhGFS was renamed and became BeeGFS.[4] While
ThinkParQ will take over customer support and sales,
Fraunhofer ITWM will continue to drive the further de-
velopment of the software. The focus will stay on im-
provements and new features for HPC systems and appli-
cations.
Due to the nature of BeeGFS being free of charge, it is
unknown how many active installations there are. How-
ever, there are around 100 customers worldwide that
use BeeGFS and are supported by Fraunhofer. Among
those are academic users such as universities and research Bee
facilities[5] as well as for-prot companies in nance or System Overview
the oil & gas industry.
Notable installations include several TOP500 computers
such as Loewe-CSC[6] cluster at the Goethe University To run BeeGFS, at least one instance of the metadata
of Frankfurt, Germany (#22 on installation), the Vienna server and the storage server is required. But BeeGFS
Scientic Cluster[7] at the University of Vienna, Austria allows multiple instances of each service to distribute the

1
2 3 BENCHMARKS

load from a large number of clients. The scalability of age) and a three tier memory: 1 TB RAM, 20 TB SSD,
each component makes sure the system itself is scalable. 120 TB HDD. Single node performance on the local le
File contents are distributed over several storage servers system without BeeGFS is 1,332 MB/s (write) and 1,317
using striping, i.e. each le is split into chunks of a given MB/s (read).
size and these chunks are distributed over the existing The nodes are equipped with 2x Intel Xeon X5660, 48
storage servers. The size of these chunks can be dened GB RAM, 4x Intel 510 Series SSD (RAID 0), Ext4, QDR
by the le system administrator. In addition, also the Inniband and run Scientic Linux 6.3, Kernel 2.6.32-
metadata is distributed over several metadata servers on a 279 and FhGFS 2012.10-beta1.
directory level, with each server storing a part of the com-
plete le system tree. This approach allows fast access on
the data.
Clients as well as metadata or storage servers can be
added into an existing system without any downtime. The
client itself is a lightweight kernel module that does not
require any kernel patches. The servers run on top of
an existing local le system. There are no restrictions to
the type of underlying le system as long as it supports
POSIX; recommendations are to use ext4 for the meta-
data servers and XFS for the storage servers. Both servers
run in userspace.
Also there is no strict requirement for dedicated hardware
for individual services. The design allows a le system ad-
ministrator to start the services in any combination on a
given set of machines and expand in the future. A com-
mon way among BeeGFS users to take advantage of this
is combining metadata servers and storage servers on the
same machines.
BeeGFS supports various network-interconnects with dy-
namic failover such as Ethernet or Inniband as well
as many dierent Linux distributions and kernels (from
2.6.16 to the latest vanilla). The software has a simple Throughput
setup and startup mechanism using init scripts. For users
who prefer a graphical interface over command lines, a
Java-based GUI (AdMon) is available. The GUI pro-
vides monitoring of the BeeGFS state and management
of system settings. Besides managing and administrating
the BeeGFS installation, this tool also oers a couple of
monitoring options to help identifying performance issues
within the system.

2.1 BeeGFS on-demand

BeeGFS on-demand allows the creation of BeeGFS on

a set of nodes with one single command line. Possible
use cases for the tool are manifold; a few include setting
up a dedicated parallel le system for a cluster job, cloud
computing or for fast and easy temporary setups for test-
ing purposes.

3 Benchmarks
The following benchmarks have been performed on
Fraunhofer Seislab, a test and experimental cluster at Creates
Fraunhofer ITWM with 25 nodes (20 compute + 5 stor-
 3

6 References
[1] Latest stable BeeGFS release. December 28, 2016. Re-
trieved December 28, 2016.

[2] BeeGFS End-User License Agreement (EULA)".

Fraunhofer ITWM. February 22, 2012. Retrieved March
15, 2014.

[3] ThinkParQ website. Retrieved March 17, 2014.

[4] Rich Brueckner (March 13, 2014). Fraunhofer to Spin

O Renamed BeeGFS File System. insideHPC. Re-
trieved March 17, 2014.

[5] FraunhoferFS High-Performance Parallel File System.

ClusterVision eNews. November 2012. Archived from the
original on March 17, 2014. Retrieved March 17, 2014.

[6] "... And Fraunhofer. StorageNewsletter.com. June 18,

2010. Retrieved March 17, 2014.

[7] VSC-2. Top500 List. June 20, 2011. Retrieved March

17, 2014. IOPS

[8] Abel. Top500 List. June 18, 2012. Retrieved March 17,
2014.

4 BeeGFS and exascale [9] DEEP-ER Project Website. Retrieved March 17, 2014.

Fraunhofer ITWM is participating in the Dynamic-

Exascale Entry Platform Extended Reach (DEEP-ER)
project of the European Union,[9] which addresses the
problems of the growing gap between compute speed and
I/O bandwidth, and system resiliency for large-scale sys-
tems.
Some of the aspects that BeeGFS developers are working
on under the scope of this project are:

support for tiered storage,

POSIX interface extensions,

fault tolerance and high availability (HA), and

improved monitoring and diagnostic tools.

The plan is to keep the POSIX interface for backward

compatibility but also allow applications more control
over how the le system handles things like data place-
ment and coherency through API extensions.

5 See also

Distributed le system

List of le systems, the distributed parallel le sys-

tem section
4 7 TEXT AND IMAGE SOURCES, CONTRIBUTORS, AND LICENSES

7 Text and image sources, contributors, and licenses

7.1 Text
BeeGFS Source: https://en.wikipedia.org/wiki/BeeGFS?oldid=757072240 Contributors: Bearcat, Bender235, Qwertyus, Janfrode, Smack-
Bot, Frap, Kvng, Iridescent, Keith D, Someguy1221, KathrynLybarger, ImageRemovalBot, Mild Bill Hiccup, Niceguyedc, Blanchardb,
Tarheel95, Dsimic, Addbot, Yobot, AnomieBOT, W Nowicki, Natisto, Sweet xx, John Cline, Frogging frog, ClueBot NG, BG19bot,
ScotXW, Tobias.goetz, InternetArchiveBot, Ant70wiki and Anonymous: 20

7.2 Images
File:BeeGFS_Logo.png Source: https://upload.wikimedia.org/wikipedia/en/e/ed/BeeGFS_Logo.png License: Fair use Contributors:
http://www.fhgfs.com/cms/ Original artist: ?
File:BeeGFS_System_Architecture_Overview.svg Source: https://upload.wikimedia.org/wikipedia/en/8/80/BeeGFS_System_
Architecture_Overview.svg License: CC-BY-SA-3.0 Contributors:
Created it with Corel Draw
Previously published: Published in print only so far
Original artist:
Tobias.goetz
File:FhGFS_file_create_benchmark.png Source: https://upload.wikimedia.org/wikipedia/commons/8/83/FhGFS_file_create_
benchmark.png License: CC BY-SA 3.0 Contributors: Diagram made using benchmark data provided by Fraunhofer ITWM Original
artist: Tobias.goetz
File:FhGFS_iops_benchmark.png Source: https://upload.wikimedia.org/wikipedia/commons/9/93/FhGFS_iops_benchmark.png Li-
cense: CC BY-SA 3.0 Contributors: Diagram made using benchmark results by Fraunhofer ITWM Original artist: Tobias.goetz
File:FhGFS_read_write_throughput_benchmark.png Source: https://upload.wikimedia.org/wikipedia/commons/f/f4/FhGFS_read_
write_throughput_benchmark.png License: CC BY-SA 3.0 Contributors: Diagram using benchmark data produced by Fraunhofer ITWM
Original artist: Tobias.goetz

7.3 Content license

Creative Commons Attribution-Share Alike 3.0