Skip to content

Lambda's Managed Slurm health checks#

Managed Slurm continuously validates the health of your cluster so problems are found and fixed before they reach your jobs. Validation happens at two levels:

  • Passive checks constantly verify every node's hardware and connectivity, integrated directly with the Slurm scheduler.
  • Active checks run deeper GPU diagnostics and stress tests on idle capacity, catching subtle degradation without ever touching your workloads.

Together with auto-remediation, these checks keep your jobs landing on healthy hardware, automatically.

Passive checks: every node, continuously#

Every node runs a comprehensive health check every few minutes, wired into Slurm's native health check mechanism. Each pass verifies the components your jobs depend on:

  • GPUs: overall device health, driver responsiveness, and GPU-interconnect status, powered by NVIDIA DCGM.
  • Network: the high-speed fabric links on every node are present, up, and running at full speed.
  • Storage: the shared filesystems (/home and /data) and local disks are mounted and healthy.
  • System: CPU, memory, and core operating system health.

A node that fails any check is automatically removed from scheduling before it can affect new jobs. See auto-remediation for what happens next.

Active checks: deep diagnostics on idle GPUs#

Passive monitoring catches faults; active diagnostics catch degradation. On a regular schedule, the cluster runs deep GPU diagnostics and sustained stress tests, including NVIDIA DCGM diagnostics and GPU load tests, to surface issues that only appear under pressure, such as thermal problems or subtle memory faults.

Active checks are designed to be invisible to your work:

  • They run as ordinary Slurm jobs in a dedicated, lowest-priority partition.
  • They schedule only onto idle GPUs.
  • The moment one of your jobs needs the hardware, the diagnostic is preempted immediately and returns to the queue.

Results are published as cluster metrics, where they drive auto-remediation automatically.

You might notice these jobs in squeue, or the dedicated partition in sinfo. That's the cluster testing itself, at no cost to your capacity.

Run diagnostics yourself

Ready-made diagnostic and stress-test jobs, including a deep GPU diagnostic and a GPU burn test, are available in /data/examples, alongside compute, storage, and network benchmarks. Submit them like any other job, for example to spot-check the hardware before kicking off a long training run.

Per-job GPU statistics#

Health data doesn't stop at the node level. For every GPU job, the cluster records per-GPU statistics (utilization, memory, and energy) for each node the job used, and saves them next to your job's output in <JOBID>.gpu_stats/. It's an easy first stop when a run behaves unexpectedly.

Health at a glance#

The Slurm console shows the same picture the checks see: node states, health check pass/fail history and failure detail, and live GPU telemetry (temperature, memory, and power) for every node in the cluster. Lambda monitors the same signals around the clock as part of Managed Slurm.