# Lambda's Managed Slurm auto-remediation

At GPU-cluster scale, occasional hardware trouble is a fact of life. What matters is what happens next. Managed Slurm pairs its [health checks](https://docs.lambda.ai/managed-slurm/health-checks/) with an automated remediation loop that detects, isolates, and repairs unhealthy nodes, usually before you notice anything at all.

## The remediation loop

1. **Detect.** [Passive and active health checks](https://docs.lambda.ai/managed-slurm/health-checks/) validate every node continuously. Their results are published as cluster metrics, watched by both the remediation system and Lambda's 24/7 monitoring.

2. **Isolate.** A node that fails validation is immediately taken out of scheduling. No new work lands on it, and queued jobs simply flow to healthy nodes.

3. **Remediate.** A remediation agent on every node watches those health metrics and acts on its own node: drain, reboot, and return to service. A node rejoins the pool only after it passes its health checks again, never before.

4. **Escalate.** If a node can't recover on its own, hands-on break/fix with Lambda's data center operations team is initiated automatically, and Lambda sees the repair or replacement through according to your SLA.

## What you'll notice

Usually, nothing: capacity dips briefly while a node is repaired, then returns automatically. If you look closely, you might see a node marked as drained or down in `sinfo`, or its state change in the [Slurm console](https://docs.lambda.ai/managed-slurm/slurm-console/), with no action needed on your part.

Make long jobs resumable

For multi-day training runs, checkpoint regularly and make your batch jobs requeueable. If a node fault ever interrupts a job, Slurm can requeue it automatically, and it resumes from your latest checkpoint on healthy hardware.