Skip to content

Managed Slurm quick start#

This guide takes you from a fresh user account to running multi-node GPU jobs on a Managed Slurm cluster. For an overview of how Lambda's Managed Slurm works, and how it compares to an unmanaged deployment, see Using Lambda's Managed Slurm.

Before you begin#

Start in the Slurm console, opened from the Lambda Cloud console: it's the fastest first look at your new cluster, and where accounts are managed. To follow this guide, you need:

  • The IP address of your cluster's login node, available in the Lambda Cloud console.
  • A user account on the cluster. Administrators can create accounts in seconds, from the Slurm console or from the command line. See Creating and removing MSlurm users.

Connect to your cluster#

SSH into the login node:

ssh <USERNAME>@<LOGIN-NODE-IP>

The login node is your home base: edit code, manage data, and submit jobs from here. Compute nodes stay dedicated to scheduled work, which keeps them healthy and predictable for everyone's jobs.

Node-to-node connectivity is automatic

The first time you log in, the cluster configures SSH between nodes for your account. Multi-node jobs and tools that rely on node-to-node SSH work out of the box, with no key setup needed. While one of your jobs is running on a node, you can also SSH from the login node directly into that node to inspect it:

ssh <NODE-NAME>

Get your bearings#

A few commands give you a complete picture of the cluster:

Command What it shows
sinfo -N -l Every node, its partition, and its state.
scontrol show partition Partitions, their limits, and their defaults.
squeue -u $USER Your pending and running jobs.
scontrol show topology The cluster's network topology.

Jobs run in the cluster's default partition unless you select another with --partition. Scheduling is topology-aware: Slurm knows how nodes connect to the network fabric and places multi-node jobs to maximize communication performance.

Note

You might notice a low-priority partition dedicated to automated health checks, along with its jobs in squeue output. These diagnostics run only on idle capacity and yield immediately when your jobs need the nodes. See A healthy cluster, automatically.

Know your storage#

Path Scope Best for
/home Shared across all nodes Code, environments, checkpoints
/data Shared across all nodes Datasets, shared tools, examples
/scratch Local NVMe on each compute node Fast temporary space during a job

Your home directory is the same on every node, so your code, virtual environments, and outputs follow your jobs wherever they're scheduled.

Note

Stage data on local storage before running I/O-heavy jobs. Reading directly from shared storage during a job can bottleneck on I/O. Treat /scratch as ephemeral: copy results back to shared storage before your job ends.

Run your first jobs#

srun --gpus=1 nvidia-smi -L

srun runs a command on a compute node and streams the output back to your terminal. It's the quickest way to test something on a GPU.

sbatch --wrap 'hostname'

sbatch queues a script (here, a one-liner) and returns immediately; output lands in slurm-<JOBID>.out. Your cluster also ships with ready-made job scripts in /data/examples:

sbatch /data/examples/hostname.sbatch

Each example is a readable template. Take a look inside to see how a job script is put together.

salloc --gpus=1

salloc reserves resources and drops you into a shell on the allocated compute node. Run commands directly, then press Ctrl+D to release the allocation.

Watch and manage your work:

squeue -u $USER        # your queue at a glance
scancel <JOBID>        # cancel a job

Every job is recorded in the cluster's accounting database. Review your history and resource usage anytime:

sacct -o JobID,JobName,Partition,AllocTRES%35,State,Elapsed

Request GPUs; the rest is proportional

Defaults are tuned so that asking for GPUs is usually all you need: CPU cores and memory are allocated automatically in proportion to the GPUs you request. Override them anytime with options such as --cpus-per-task and --mem.

Set up your software#

Clusters come ready for GPU development out of the box:

  • CUDA toolkit, including the nvcc compiler.
  • NCCL and HPC-X MPI, ready in your default environment for distributed workloads.
  • Everyday tools such as Python, Git, and common editors and utilities.
  • Lmod modules for additional toolchains. List them with module avail.

For example, manage Python projects and virtual environments with uv:

module load uv
uv venv ~/venvs/demo && source ~/venvs/demo/bin/activate

See Managing software using Lmod for more on modules.

No root required

Between modules, uv-managed Python environments, and containers, everyday workflows don't need sudo, and Managed Slurm keeps nodes clean and consistent by reserving system-level changes for cluster administrators and Lambda. Need a system package installed? Ask your cluster administrator, or reach out to Lambda support.

Use containers#

Run any container image straight through Slurm. Pyxis and Enroot are preconfigured, and the NVIDIA Container Toolkit ensures containers get GPU access:

srun --gpus=1 --container-image=nvidia/cuda:12.8.1-runtime-ubuntu22.04 \
     nvidia-smi -L

Slurm pulls the image, starts the container on a compute node, and runs the command. No image management is needed on your part. /data/examples includes container-based job scripts too, such as a two-node NCCL test that runs entirely in containers (pyxis_2x_all_reduce.sbatch).

Prefer another workflow? Apptainer and Podman (with a Docker-compatible docker CLI) are also available on compute nodes.

Check your cluster's performance#

The examples directory includes benchmark jobs you can run as-is. To verify multi-node GPU communication across the cluster's GPUDirect RDMA-capable fabric, submit the provided NCCL all-reduce test:

sbatch /data/examples/full_cluster_all_reduce.sbatch

The job builds NVIDIA's nccl-tests in /data/$USER and runs a cluster-wide all-reduce; results land in ~/<JOBID>.nccl-test.out. The same directory covers the rest of the stack: gemm.sbatch measures each GPU's raw compute (TFLOPS), fio.sbatch benchmarks the shared and local filesystems, and further variants and GPU stress tests sit alongside them.

After any GPU job finishes, look for a <JOBID>.gpu_stats/ directory next to your job's output: the cluster automatically records per-job GPU statistics (utilization, memory, and energy) for every node the job used.

Profile at full depth#

Profiling isn't an afterthought here: every user gets full access to GPU hardware performance counters, no root required. Run NVIDIA Nsight Compute (ncu) on your own kernels under srun, or start from the ready-made example:

sbatch /data/examples/ncu.sbatch

The job profiles a small GPU kernel and saves a report you can open in the Nsight Compute UI. CPU-side profiling with perf works out of the box too, so host-side bottlenecks (data loading, launch overhead) are as easy to chase as device-side ones.

See your cluster in the browser#

Every Managed Slurm cluster includes the Slurm console, with sign-in linked to your Lambda identity. From one place, you can:

  • Watch cluster health live: node states, job flow, and GPU temperature, memory, and power for every node.
  • Work with your jobs: watch the queue, browse history and per-user accounting, drill into any job's output and GPU metrics, or submit new jobs without leaving the browser.
  • Review access at a glance, from login activity to a full audit trail of cluster changes.
  • Manage users, SSH keys, and Slurm accounts. See Managing users from the Slurm console.

For the full tour, see the Slurm console.

A healthy cluster, automatically#

Managed Slurm watches the hardware so you don't have to:

  • Continuous node health checks are wired directly into Slurm and run on every node every few minutes. A node that fails a check is automatically taken out of scheduling before it can affect your jobs, and automated remediation works to return it to service.
  • Deep GPU diagnostics and stress tests run continuously in the background, using only idle capacity in a dedicated low-priority partition that yields instantly to user jobs.
  • Cluster telemetry, covering hardware, GPU, and scheduler metrics, is collected continuously and monitored by Lambda. Lambda coordinates patches, upgrades, and repairs with you according to your SLA.

Get help#