# Managed Slurm quick start This guide takes you from a fresh user account to running multi-node GPU jobs on a Managed Slurm cluster. For an overview of how Lambda's Managed Slurm works, and how it compares to an unmanaged deployment, see [Using Lambda's Managed Slurm](https://docs.lambda.ai/managed-slurm/). ## Before you begin Start in the [Slurm console](https://docs.lambda.ai/managed-slurm/slurm-console/), opened from the Lambda Cloud console: it's the fastest first look at your new cluster, and where accounts are managed. To follow this guide, you need: - The IP address of your cluster's login node, available in the Lambda Cloud console. - A user account on the cluster. Administrators can create accounts in seconds, from the Slurm console or from the command line. See [Creating and removing MSlurm users](https://docs.lambda.ai/managed-slurm/#creating-and-removing-mslurm-users). ## Connect to your cluster SSH into the login node: ``` ssh @ ``` The login node is your home base: edit code, manage data, and submit jobs from here. Compute nodes stay dedicated to scheduled work, which keeps them healthy and predictable for everyone's jobs. Node-to-node connectivity is automatic. The first time you log in, the cluster configures SSH between nodes for your account. Multi-node jobs and tools that rely on node-to-node SSH work out of the box, with no key setup needed. While one of your jobs is running on a node, you can also SSH from the login node directly into that node to inspect it: ``` ssh ``` ## Get your bearings A few commands give you a complete picture of the cluster: | Command | What it shows | | --- | --- | | `sinfo -N -l` | Every node, its partition, and its state. | | `scontrol show partition` | Partitions, their limits, and their defaults. | | `squeue -u $USER` | Your pending and running jobs. | | `scontrol show topology` | The cluster's network topology. | Jobs run in the cluster's default partition unless you select another with `--partition`. Scheduling is topology-aware: Slurm knows how nodes connect to the network fabric and places multi-node jobs to maximize communication performance. **Note**: You might notice a low-priority partition dedicated to automated health checks, along with its jobs in `squeue` output. These diagnostics run only on idle capacity and yield immediately when your jobs need the nodes. See [A healthy cluster, automatically](https://docs.lambda.ai/managed-slurm/quickstart/#a-healthy-cluster-automatically). ## Know your storage | Path | Scope | Best for | | --- | --- | --- | | `/home` | Shared across all nodes | Code, environments, checkpoints | | `/data` | Shared across all nodes | Datasets, shared tools, examples | | `/scratch` | Local NVMe on each compute node | Fast temporary space during a job | Your home directory is the same on every node, so your code, virtual environments, and outputs follow your jobs wherever they're scheduled. **Note**: Stage data on local storage before running I/O-heavy jobs. Reading directly from shared storage during a job can bottleneck on I/O. Treat `/scratch` as ephemeral: copy results back to shared storage before your job ends. ## Run your first jobs [Run a command with `srun`](https://docs.lambda.ai/managed-slurm/quickstart/#run-a-command-with-srun) [Submit a script with `sbatch`](https://docs.lambda.ai/managed-slurm/quickstart/#submit-a-script-with-sbatch) [Work interactively with `salloc`](https://docs.lambda.ai/managed-slurm/quickstart/#work-interactively-with-salloc) ``` srun --gpus=1 nvidia-smi -L ``` `srun` runs a command on a compute node and streams the output back to your terminal. It's the quickest way to test something on a GPU. ``` sbatch --wrap 'hostname' ``` `sbatch` queues a script (here, a one-liner) and returns immediately; output lands in `slurm-.out`. Your cluster also ships with ready-made job scripts in `/data/examples`: ``` sbatch /data/examples/hostname.sbatch ``` Each example is a readable template. Take a look inside to see how a job script is put together. ``` salloc --gpus=1 ``` `salloc` reserves resources and drops you into a shell on the allocated compute node. Run commands directly, then press `Ctrl` + `D` to release the allocation. Watch and manage your work: ``` squeue -u $USER # your queue at a glance scancel # cancel a job ``` Every job is recorded in the cluster's accounting database. Review your history and resource usage anytime: ``` sacct -o JobID,JobName,Partition,AllocTRES%35,State,Elapsed ``` Request GPUs; the rest is proportional. Defaults are tuned so that asking for GPUs is usually all you need: CPU cores and memory are allocated automatically in proportion to the GPUs you request. Override them anytime with options such as `--cpus-per-task` and `--mem`. ## Set up your software Clusters come ready for GPU development out of the box: - **CUDA toolkit**, including the `nvcc` compiler. - **NCCL** and **HPC-X MPI**, ready in your default environment for distributed workloads. - **Everyday tools** such as Python, Git, and common editors and utilities. - **Lmod modules** for additional toolchains. List them with `module avail`. For example, manage Python projects and virtual environments with `uv`: ``` module load uv uv venv ~/venvs/demo && source ~/venvs/demo/bin/activate ``` See [Managing software using Lmod](https://docs.lambda.ai/managed-slurm/#managing-software-using-lmod) for more on modules. No root required. Between modules, `uv`-managed Python environments, and containers, everyday workflows don't need `sudo`, and Managed Slurm keeps nodes clean and consistent by reserving system-level changes for cluster administrators and Lambda. Need a system package installed? Ask your cluster administrator, or reach out to [Lambda support](https://docs.lambda.ai/managed-slurm/quickstart/#get-help). ## Use containers Run any container image straight through Slurm. Pyxis and Enroot are preconfigured, and the NVIDIA Container Toolkit ensures containers get GPU access: ``` srun --gpus=1 --container-image=nvidia/cuda:12.8.1-runtime-ubuntu22.04 \ nvidia-smi -L ``` Slurm pulls the image, starts the container on a compute node, and runs the command. No image management is needed on your part. `/data/examples` includes container-based job scripts too, such as a two-node NCCL test that runs entirely in containers (`pyxis_2x_all_reduce.sbatch`). Prefer another workflow? **Apptainer** and **Podman** (with a Docker-compatible `docker` CLI) are also available on compute nodes. ## Check your cluster's performance The examples directory includes benchmark jobs you can run as-is. To verify multi-node GPU communication across the cluster's GPUDirect RDMA-capable fabric, submit the provided NCCL all-reduce test: ``` sbatch /data/examples/full_cluster_all_reduce.sbatch ``` The job builds NVIDIA's `nccl-tests` in `/data/$USER` and runs a cluster-wide all-reduce; results land in `~/.nccl-test.out`. The same directory covers the rest of the stack: `gemm.sbatch` measures each GPU's raw compute (TFLOPS), `fio.sbatch` benchmarks the shared and local filesystems, and further variants and GPU stress tests sit alongside them. After any GPU job finishes, look for a `.gpu_stats/` directory next to your job's output: the cluster automatically records per-job GPU statistics (utilization, memory, and energy) for every node the job used. ## Profile at full depth Profiling isn't an afterthought here: every user gets full access to GPU hardware performance counters, no root required. Run NVIDIA Nsight Compute (`ncu`) on your own kernels under `srun`, or start from the ready-made example: ``` sbatch /data/examples/ncu.sbatch ``` The job profiles a small GPU kernel and saves a report you can open in the Nsight Compute UI. CPU-side profiling with `perf` works out of the box too, so host-side bottlenecks (data loading, launch overhead) are as easy to chase as device-side ones. ## See your cluster in the browser Every Managed Slurm cluster includes the Slurm console, with sign-in linked to your Lambda identity. From one place, you can: - Watch cluster health live: node states, job flow, and GPU temperature, memory, and power for every node. - Work with your jobs: watch the queue, browse history and per-user accounting, drill into any job's output and GPU metrics, or submit new jobs without leaving the browser. - Review access at a glance, from login activity to a full audit trail of cluster changes. - Manage users, SSH keys, and Slurm accounts. See [Managing users from the Slurm console](https://docs.lambda.ai/managed-slurm/#managing-users-from-the-slurm-console). For the full tour, see [the Slurm console](https://docs.lambda.ai/managed-slurm/slurm-console/). ## A healthy cluster, automatically Managed Slurm watches the hardware so you don't have to: - **[Continuous node health checks](https://docs.lambda.ai/managed-slurm/health-checks/)** are wired directly into Slurm and run on every node every few minutes. A node that fails a check is automatically taken out of scheduling before it can affect your jobs, and [automated remediation](https://docs.lambda.ai/managed-slurm/auto-remediation/) works to return it to service. - **Deep GPU diagnostics and stress tests** run continuously in the background, using only idle capacity in a dedicated low-priority partition that yields instantly to user jobs. - **Cluster telemetry**, covering hardware, GPU, and scheduler metrics, is collected continuously and monitored by Lambda. Lambda coordinates patches, upgrades, and repairs with you according to your SLA. ## Get help - Lambda's support team is available 24/7. See [Support](https://docs.lambda.ai/public-cloud/1-click-clusters/support/) for scope and contact details. - [Using Lambda's Managed Slurm](https://docs.lambda.ai/managed-slurm/) covers user management, job submission, and Lmod in more depth. - SchedMD's [Slurm documentation](https://slurm.schedmd.com/documentation.html) and [man pages](https://slurm.schedmd.com/man_index.html) cover every Slurm command and option.