Farouk Salem

FLESnet should tolerate failures of individual compute nodes and should continue to distribute and build timeslices without overly sacrificing the achieved aggregate throughput and latency in large scale deployments. This requires all input nodes to consistently adhere to a new schedule and distribution schema on time and might require the redistribution of not yet entirely built timeslices. We implemented such a fault-tolerance mechanism in our Data-Flow Scheduler that detects and dynamically (re-)assigns the load of failed nodes to the remaining compute nodes. We show the influence of failures on the aggregate bandwidth and latency of timeslice building on our OmniPath testbed.

Minimizing the latency is essential for FLESnet to achieve good aggregated bandwidth and we already improved the throughput and lowered the latency with our Data-Flow Scheduler. However, there is still a gap between the achieved and the maximally achievable bandwidth. For timeslice building, we found that FLESnet performs two RDMA writes for each contribution. This congests the network unnecessarily and increases the latency, especially in large deployments. We, therefore, optimized FLESnet to need only one RDMA request per contribution. We show how the aggregated bandwidth is increased when the scheduler is used. We also discuss the performance of the scheduler on an Infiniband cluster.

Achieving efficient many-to-many communication on a given network topology is a challenging task when many data streams from different sources have to be scattered concurrently to many destinations with low variance in arrival times. In such scenarios, it is critical to saturate but not to congest the bisectional bandwidth of the network topology in order to achieve a good aggregate throughput. When there are many concurrent point-to-point connections, the communication pattern needs to be dynamically scheduled in a fine-grained manner to avoid network congestion (links, switches), overload in the node’s incoming links, and receive buffer overflow. Motivated by the use case of the Compressed Baryonic Matter experiment (CBM), we study the performance and variance of such communication patterns on a Cray XC40 with different routing schemes and scheduling approaches. We present a distributed Data Flow Scheduler (DFS) that reduces the variance of arrival times from all sources at least 30 times and increases the achieved aggregate bandwidth by up to 50%.

Achieving efficient many-to-many communication on a given network topology is a challenging task when many data streams from different sources have to be scattered concurrently to many destinations with low variance in arrival times. In such scenarios, it is critical to saturate but not to congest the bisectional bandwidth of the network topology in order to achieve a good aggregate throughput. When there are many concurrent point-to-point connections, the communication pattern needs to be dynamically scheduled in a fine-grained manner to avoid network congestion (links, switches), overload in the node’s incoming links, and receive buffer overflow. Motivated by the use case of the Compressed Baryonic Matter experiment (CBM), we study the performance and variance of such communication patterns on a Cray XC40 with different routing schemes and scheduling approaches. We present a distributed Data Flow Scheduler (DFS) that reduces the variance of arrival times from all sources at least 30 times and increases the achieved aggregate bandwidth by up to 50 %.

FLES is the first-level event selector of the CBM project. For the timeslice building, the FLESnet software uses
tightly coupled processes. Unfortunately, slow processes and network congestion cause a serious performance bottleneck, especially in large systems. We, therefore, developed a scheduling mechanism to minimize the synchronization overhead and enhance the scalability. We highlight the main components of our new scheduling mechanism and we show how it enhances the performance of FLESnet on large deployments.

FLES is the first-level event selector of the CBM project. The system prototype, FLESnet, natively uses InfiniBand for communication, which limits the portability to other interconnects that may be of interest when the actual analysis cluster is built. We adopt FLESnet to Libfabric, a generic networking framework, and explain how this framework is used for time-slice building with efficient, zero-copy data transfers. We discuss preliminary benchmarking results of the new implementation using the Cray GNI interconnect with Intel Xeon Phi processors.