Overview

HPC clusters have changed little since the mid-1990s, when the first Beowulf cluster was built at NASA. HPC users still rely on dated, difficult-to-learn interfaces, and HPC workflows are hard to port to cloud environments, which assume different hardware and a different scheduling model.

Fuzzball was created to ease the use and administration of traditional HPC clusters, and to overcome the challenges of running HPC workloads in the cloud.

At its heart, Fuzzball is a container orchestration platform built for discrete jobs rather than for the long-running microservices that tools like Kubernetes are designed to run. A workflow can include long-running services when its jobs need them (a database or an inference server, for example), but the unit of work is the job: a task that runs to completion and produces a result.

Fuzzball has a web UI and a Command Line Interface (CLI) for submitting and monitoring workflows. Workflows are specified in YAML documents called Fuzzfiles, which the web UI’s workflow editor can generate for you.

Every workflow is container based, and Fuzzball abstracts hardware details, so workflows are portable and reproducible. They can run on-prem or in the cloud.

Fuzzball also has an administrative layer for managing settings and permissions at the organization, group, and individual user levels.

The Fuzzball solution is composed of two parts:

Orchestrate runs on a central server node and accepts requests from users in the form of Fuzzfiles. It allocates computational resources for the jobs in a workflow, downloads containers, sets up storage volumes, and initiates jobs. Orchestrate itself runs on Kubernetes.

Substrate runs on the compute nodes. It is a purpose-built container platform that works with Orchestrate to set up the environment for each job and run it. Substrate runs directly on the host operating system rather than inside Kubernetes. Conceptually, it adopts much of the same design philosophy as the HPC container platform Apptainer (formerly known as Singularity).

Fuzzball is API driven, and its web UI covers the everyday work of building, submitting, and monitoring workflows. Users do not need to open a terminal emulator, use the secure shell (ssh) utility, understand login nodes versus compute nodes, learn a batch scheduling system, or manage files with Linux commands. This lowers the barrier for users who are new to HPC.

Users who prefer text-based interaction can use the CLI instead.

Fuzzball is container based and abstracts hardware and storage, so workflows are reproducible and can be shared with colleagues even if they work on a different cluster.

The same design that makes Fuzzball workflows reproducible makes them portable between on-prem and cloud environments. Provisioning and allocating compute nodes is abstracted away in both, leaving users free to concentrate on their workflows.

Containers are a common deployment method for scientific software, AI frameworks, and modeling suites. Because Fuzzball is container based, users can often get the software they need by providing a URI. If the software does not already exist in a container, building one is usually less work than installing and maintaining the software on an HPC system.

Fuzzball provides integrated MPI support for MPICH and OpenMPI, which cover the vast majority of use cases. Users of distributed programs are therefore not limited to the MPI implementation installed on the system, and need not recompile their programs or wait for administrators to update the software stack. This makes it practical to run containerized MPI software across multiple nodes.

Fuzzball lets owners and users be managed at the organization, group, or individual user level. In traditional HPC, user groups are managed through the tools available in the Linux distribution and services like Active Directory. Integrating them is usually left to the cluster administrators, and one-off solutions are common. File system access must be managed with Linux groups, and users can accidentally set permissions that lock them out of their own data or expose it to others. Fuzzball removes these administrative burdens.

Fuzzball extends batch scheduling to include provisioning cloud resources on demand, so a workflow becomes a flexible set of instructions that can be executed wherever the resources are.