Oracle Cloud Infrastructure’s Era 2 cloud is shifting the needle for high-performance file servers within the cloud but once more. Oracle Cloud prospects can now use Cluster Networking to recover from 130 GB per second throughput in a cloud-based file server.

As a result of BeeGFS on Oracle Cloud Infrastructure can feed the beast for high-performance computing (HPC), synthetic intelligence, and machine studying workloads, we plan to incorporate this file server in our HPC market picture.

We just lately revealed the Deploy a BeeGFS File System on Oracle Cloud Infrastructure weblog to display the best and most simple option to deploy BeeGFS filesystem on Oracle Cloud Infrastructure. In that weblog publish, we lined methods to deploy BeeGFS on Oracle Cloud Infrastructure utilizing transmission management protocol (TCP). Now, we’re sharing what an incredible enchancment our HPC naked steel Compute form (BM.HPC2.36) with distant direct reminiscence entry (RDMA) cluster community can ship to the efficiency of your BeeGFS filesystem cluster.

BeeGFS working on Oracle HPC compute shapes use and drastically advantages from RDMA over converged ethernet, also called RoCE. RoCE is a key function of our HPC cluster community providing on Oracle Cloud Infrastructure. Shifting the BeeGFS server-client block communication away from the normal 25-GbE networks and onto the quick and dependable 100-Gbps RDMA infrastructure offers a big efficiency enhance for lowered latency and elevated throughput. At Oracle Cloud Infrastructure, we offer Oracle Linux UEK working system pictures, which have RDMA OFED libraries preinstalled and configured for our prospects to make use of.

By turning on the help for RDMA in BeeGFS utilizing the VERBS API for knowledge switch between a storage or metadata server and shopper, prospects can additional drive the latency down and the throughput up. To allow RDMA through the BeeGFS set up course of, use the next instructions. You will discover extra particulars on the set up course of within the Install Commands Walkthrough.

1. Set up libbeegfs-ib bundle to allow RDMA based mostly on OFED ibverbs API.

Meta node:


yum set up beegfs-meta libbeegfs-ib -y


Storage node:


yum set up beegfs-storage libbeegfs-ib -y


2. Replace beegfs-client-autobuild.conf to incorporate path to put in kernel driver modules.

Shopper node:


sed -i 's|^buildArgs=-j8|buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/embrace|g'/and so forth/beegfs/beegfs-client-autobuild.conf


On Oracle Linux UEK Working System:


yum set up -y elfutils-libelf-devel



sed -i -e '/ifeq.*compat-2.6.h/,+Three s/^/# /'/choose/beegfs/src/shopper/client_module_7/supply/Makefile


3. BeeGFS Shopper rebuild:

Shopper node:


/and so forth/init.d/beegfs-client rebuild



We created a BeeGFS converged cluster with 16 HPC naked steel nodes, every with 6.Four TB native NVME and 100-Gbps RDMA cluster networking enabled. A converged BeeGFS cluster is a setup the place completely different BeeGFS companies (server and shopper) run on identical compute nodes. We use 14 nodes for the file server, offering 89.6 TB of whole storage.

BeeGFS Tuning

On shopper nodes, we up to date the chunk or stripe dimension to 1m, as an alternative of 512ok default worth.


beegfs-ctl --setpattern --chunksize=1m --numtargets=4 /mnt/beegfs


The next BeeGFS parameters are required tuning for RDMA:

  • tuneNumWorkers

  • tuneBindToNumaZone

  • connRDMABufSize

  • connRDMABufNum

  • connMaxInternodeNum

For extra particulars, confer with How To Configure and Test BeeGFS with RDMA.

IOZone Configuration

For IO throughput benchmarking, we used IOzone, an open supply file system benchmarking utility. We ran IOzone concurrently on all 16 shopper nodes. We configured it to run write workload with report/switch dimension of 2m (-r2m), fsync (-e) enabled, Stonewall disabled (-x), 32 threads and processes per shopper node (-t32). We examined the IO throughput for a variety of file sizes utilizing a complete of 512 threads throughout 16 shopper nodes.


As you possibly can see within the following graph, we will rise up to 140 GB/s mixture write throughput from a 14 OSs node filesystem.

We automated the deployment of a converged BeeGFS BeeGFS ON Demand utilizing RDMA on Oracle Cloud Infrastructure utilizing the Terraform template. Quickly, we’re together with the provisioning of a BeeGFS BeeOND cluster in our HPC market picture at no extra cost.

An architecture diagram of the BeeGFS BeeOND RDMA Cluster.

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *