Slurm Beginner's Tutorial

Slurm is the workload manager that turns your resource request and commands into scheduled work on Deceema. In this hands-on tutorial, you will submit a real job, follow it through the queue, read its logs, diagnose common states, and build a small multi-job workflow.

The five-word workflow

Prepare → request → submit → observe → verify. Keep this loop in mind and Slurm will feel much less mysterious.

What You Will Learn

By the end of this tutorial, you will be able to:

• explain why substantial computation belongs on compute nodes—not login nodes;
• match a Deceema project account with its permitted QoS;
• request CPU, memory, GPU, and time without confusing those resources;
• write, validate, and submit a batch script;
• monitor pending and running jobs without submitting duplicates;
• inspect logs, final states, exit codes, and resource usage;
• cancel only the intended job;
• use interactive jobs, arrays, and dependencies; and
• collect the evidence needed to troubleshoot or request support.

Before You Begin

Log in to Deceema and confirm that Slurm commands are available:

$ command -v sbatch squeue sacct scancel

Review your Deceema access approval or account details in Deceema Admin. Write down one valid project and its corresponding QoS. You will replace PROJECT_CODE and QOS_NAME with those values throughout this tutorial. If the pairing is unclear, ask your project owner or contact Support; do not guess it or copy another user's values.

Keep the pair together

A project code supplied with --account must be used with a QoS permitted for that project. If you belong to several projects, choose the pair that should fund and govern this particular job.

1. Build the Right Mental Model

Deceema is shared infrastructure. The login node is your control desk: use it to edit scripts, organize files, submit jobs, and inspect results. Compute nodes are where CPU-, memory-, and GPU-intensive work runs after Slurm grants an allocation.

You on a login node
        |
        | sbatch job.sh
        v
   Slurm queue  ---- waits for a valid scheduling opportunity
        |
        | allocates requested resources
        v
   Compute node ---- runs commands and writes logs/results

Term	Plain-language meaning
Job	A resource request plus commands to execute.
Job script	A Bash file containing #SBATCH requests and commands.
Account	The Deceema project charged for the work.
QoS	The permitted service policy associated with that project.
Task	A process or distributed rank requested with --ntasks.
CPU per task	CPU cores assigned to each task.
Wall time	The maximum elapsed time the job may run.
Job ID	The unique number Slurm returns for an accepted job.
State	Slurm's current or final description of the job lifecycle.

Submitting a job does not mean it starts immediately. It means Slurm accepted the request and assigned a job ID; the job may wait in PENDING until its requirements can be satisfied.

2. Create a Training Workspace

Keep scripts, logs, inputs, and results separate:

$ mkdir -p "$HOME/training/slurm-beginner"/{scripts,logs,data,results}
$ cd "$HOME/training/slurm-beginner"
$ pwd

Slurm does not create missing parent directories for log paths. Creating logs/ before submission prevents an avoidable failure.

3. Write Your First Batch Job

Create scripts/hello-job.sh:

#!/bin/bash
#SBATCH --job-name=slurm-hello
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err

set -euo pipefail

module purge
module load deceema

echo "Job ID: ${SLURM_JOB_ID}"
echo "Job name: ${SLURM_JOB_NAME}"
echo "Compute host: $(hostname)"
echo "Started: $(date -Is)"
srun hostname
sleep 10
echo "Finished: $(date -Is)"

Replace both placeholders with the matching values from your approved access.

Read the script in two layers

The #SBATCH lines describe the allocation. The remaining lines describe what happens inside that allocation.

Directive	What it requests
--job-name	A recognizable name for queue and log output.
--account	The Deceema project that owns the work.
--qos	The permitted scheduling policy to use.
--time	A five-minute wall-time limit in HH:MM:SS form.
--ntasks	One process or rank.
--cpus-per-task	One CPU core for that process.
--mem	512 MiB of memory for the job.
--output	Standard output; %x becomes the job name and %j the job ID.
--error	Standard error, using the same unique naming pattern.

Directives are literal

Put #SBATCH directives immediately after the shebang and before the first executable command. Shell variables in directives are not expanded; use concrete values or pass options to sbatch at submission time.

4. Check Before You Submit

Ask Bash to parse the script without executing it:

$ bash -n scripts/hello-job.sh

No output means Bash found no syntax error. Also review the resource lines:

$ grep '^#SBATCH' scripts/hello-job.sh

This catches many mistakes cheaply, but it cannot confirm that an account, QoS, module, or application is valid on Deceema.

5. Submit and Save the Job ID

Submit the job and capture its ID automatically:

$ JOB_ID=$(sbatch --parsable scripts/hello-job.sh)
$ echo "$JOB_ID"
123456

--parsable is useful in scripts because it returns a machine-friendly job identifier. Keep the ID: it connects the queue entry, logs, accounting record, and any later support request.

6. Watch the Job Lifecycle

Inspect only your job:

$ squeue --jobs="$JOB_ID"

For a compact live view of all your work:

$ squeue --user="$USER" --format="%.18i %.20j %.2t %.10M %.6D %R"

Common states include:

Code	State	Meaning
PD	Pending	Accepted and waiting for a scheduling condition.
R	Running	Allocated resources and executing.
CG	Completing	Work ended; Slurm is finishing cleanup.
CD	Completed	Finished with a zero exit code.
F	Failed	Ended unsuccessfully.
CA	Cancelled	Cancelled by a user or administrator.
TO	Timeout	Exceeded the requested time limit.
OOM	Out of memory	Consumed more memory than the allocation allowed.

Completed jobs normally disappear from squeue; use sacct for their history.

Understand pending reasons

When a job is pending, the final squeue column explains why. Frequent reasons include Resources, Priority, Dependency, or an account/QoS limit.

$ squeue --jobs="$JOB_ID" --format="%.18i %.2t %.10M %R"
$ scontrol show job "$JOB_ID"
$ squeue --start --jobs="$JOB_ID"

An estimated start time can change as the queue changes. A pending job is not automatically broken, so inspect its reason before editing or resubmitting it.

7. Read the Logs

After the job starts, list its uniquely named files:

$ ls -lh logs/slurm-hello-"$JOB_ID".*
$ less logs/slurm-hello-"$JOB_ID".out
$ less logs/slurm-hello-"$JOB_ID".err

An empty error log is encouraging, but it is not proof that the scientific result is correct. Check the output for the expected hostname, timestamps, and application results.

Follow a running log

Use tail -f logs/slurm-hello-${JOB_ID}.out to follow new lines. Press ++ctrl+c++ to stop following the file; this does not cancel the job.

8. Inspect the Final Record

Use accounting after the job leaves the queue:

$ sacct --jobs="$JOB_ID" \
    --format=JobID,JobName,State,ExitCode,Elapsed,AllocCPUS,ReqMem,MaxRSS

Interpret the evidence together:

• State=COMPLETED and ExitCode=0:0 mean Slurm observed a successful exit;
• a non-zero application exit code normally produces a failed job state;
• TIMEOUT means the time request was too short or the program ran too long;
• OUT_OF_MEMORY means the workload exceeded its memory allocation; and
• valid output files are still the best evidence that the intended work succeeded.

MaxRSS may appear on a job step rather than the top-level record, so inspect all rows returned for the job.

9. Cancel Safely

If a test is wrong or no longer needed, inspect it and cancel that exact ID:

$ squeue --jobs="$JOB_ID"
$ scancel "$JOB_ID"
$ sacct --jobs="$JOB_ID" --format=JobID,State,ExitCode

Avoid broad cancellation patterns until you fully understand which jobs they match. Quoting the intended job ID is the safest beginner habit.

10. Request the Right Resources

Resource requests are limits and scheduling requirements—not performance buttons. More resources help only when the application can use them.

CPU processes and threads

Use --ntasks for separate processes or distributed ranks. Use --cpus-per-task for threads used by each process.

# One process with eight threads
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8

For threaded software, connect the application to the allocation:

export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}"
srun ./threaded-program

For an MPI-style workload, follow the application's Deceema-specific launch instructions; a typical resource shape uses multiple tasks:

#SBATCH --ntasks=4
#SBATCH --cpus-per-task=1

Memory and time

Request enough memory and time for a successful run, then use accounting data to improve the next request:

#SBATCH --time=02:00:00
#SBATCH --mem=8G

Wall time is a maximum, not a promise that the job will consume two hours. Overstated requests may reduce scheduling opportunities; understated requests can end in TIMEOUT or OUT_OF_MEMORY.

GPUs

Request a GPU only when the application is GPU-enabled:

#SBATCH --gres=gpu:1

A GPU request does not convert CPU-only code into GPU code. Confirm that your framework detects the allocated device and records that fact in the log.

11. Run an Interactive Job

Interactive allocations are useful for short debugging, environment checks, and exploratory commands that need compute resources:

$ srun --account=PROJECT_CODE --qos=QOS_NAME \
    --time=00:10:00 --ntasks=1 --cpus-per-task=1 --mem=512M \
    --pty bash
$ hostname
$ echo "$SLURM_JOB_ID"
$ exit

Replace the placeholders first. The shell begins only after resources are allocated. Type exit when finished so the shared resources are released. Use a batch job for long, repeatable, or unattended work.

12. Deliberately Learn from a Failure

Create a harmless job that exits with a non-zero code:

#!/bin/bash
#SBATCH --job-name=learn-failure
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=00:02:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=256M
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err

set -euo pipefail
echo "This failure is intentional." >&2
exit 7

After replacing the placeholders, submit it and investigate:

$ FAILURE_ID=$(sbatch --parsable scripts/expected-failure.sh)
$ sacct --jobs="$FAILURE_ID" --format=JobID,State,ExitCode
$ cat logs/learn-failure-"$FAILURE_ID".err

You should see a failed state, a non-zero exit code, and the diagnostic line in the error log. This is the same three-part investigation to use for real jobs.

13. Process Many Similar Inputs with an Array

Arrays submit related tasks from one script. First create four tiny inputs:

$ printf 'alpha\nbeta\ngamma\ndelta\n' > data/items.txt

Create scripts/array-job.sh:

#!/bin/bash
#SBATCH --job-name=array-demo
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=256M
#SBATCH --array=0-3%2
#SBATCH --output=logs/%x-%A_%a.out
#SBATCH --error=logs/%x-%A_%a.err

set -euo pipefail

mapfile -t items < data/items.txt
item="${items[SLURM_ARRAY_TASK_ID]}"
printf '%s\n' "${item^^}" > "results/${item}.txt"
echo "Task ${SLURM_ARRAY_TASK_ID} processed ${item}"

Submit and inspect the array:

$ ARRAY_ID=$(sbatch --parsable scripts/array-job.sh)
$ squeue --array --jobs="$ARRAY_ID"
$ sacct --jobs="$ARRAY_ID" --format=JobID,JobName,State,ExitCode
$ ls -lh results logs/array-demo-"$ARRAY_ID"_*

0-3 creates four array tasks and %2 permits at most two to run at once. %A is replaced by the parent array ID and %a by the task index. Arrays are ideal when the same program independently processes many inputs.

14. Connect Jobs with Dependencies

Dependencies let Slurm start one job only after a condition on another job is met. Create a summary script:

#!/bin/bash
#SBATCH --job-name=array-summary
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=256M
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err

set -euo pipefail
cat results/*.txt > results/summary.txt
wc -l results/summary.txt

Submit the array, then submit the summary with afterok:

$ ARRAY_ID=$(sbatch --parsable scripts/array-job.sh)
$ SUMMARY_ID=$(sbatch --parsable \
    --dependency="afterok:${ARRAY_ID}" scripts/summarize.sh)
$ echo "Array: $ARRAY_ID  Summary: $SUMMARY_ID"
$ squeue --jobs="$ARRAY_ID,$SUMMARY_ID" --format="%.18i %.2t %R"

The summary becomes eligible only after every required array task completes successfully. Other useful dependency types include afterany for cleanup or reporting regardless of success and aftercorr for corresponding array tasks.

15. Troubleshoot Systematically

Start with the job ID and move from scheduler evidence to application evidence:

$ squeue --jobs=JOB_ID --format="%.18i %.2t %.10M %R"
$ scontrol show job JOB_ID
$ sacct --jobs=JOB_ID \
    --format=JobID,State,ExitCode,Elapsed,AllocCPUS,ReqMem,MaxRSS
$ less logs/JOB-NAME-JOB_ID.out
$ less logs/JOB-NAME-JOB_ID.err

Symptom	Check first	Constructive next step
Job remains pending	Reason from squeue or scontrol	Verify account/QoS and wait if the reason is resources or priority.
Job fails immediately	Error log and exit code	Check paths, modules, permissions, and command spelling.
TIMEOUT	Elapsed time and application progress	Optimize, checkpoint, or request a justified longer limit.
OUT_OF_MEMORY	MaxRSS, program logs, input size	Reduce memory use or make an evidence-based larger request.
CPU allocation is idle	Application thread/process settings	Match software parallelism to tasks and CPUs per task.
GPU allocation is idle	Framework device detection	Use GPU-enabled software and verify device selection.
Output file is missing	Working directory and log paths	Create parent directories and use explicit paths.
Dependent job stays pending	Dependency reason and parent state	Inspect every prerequisite; afterok requires success.

When requesting support, include the job ID, submission time, account and QoS, exact command, relevant logs, and the result you expected. Do not include passwords, access tokens, or private keys.

16. Complete the Beginner Challenge

Build a miniature reproducible workflow without copying values blindly:

1. Create a new project directory with scripts/, logs/, data/, and results/ subdirectories.
2. Write a batch job that records the job ID, hostname, start time, end time, loaded modules, and one verifiable result.
3. Validate it with bash -n.
4. Submit it with the correct Deceema account/QoS pair and save the job ID.
5. Observe its pending or running state and explain the reason field.
6. Confirm its final state and exit code with sacct.
7. Read both logs and verify the result independently.
8. Explain how you would right-size its time, CPU, and memory requests next time.

If you can do all eight steps, you can run and diagnose a basic Deceema job without treating the scheduler as a black box.

Command Cheat Sheet

Goal	Method
Find your approved project and QoS	Review your access approval or Deceema Admin account details.
Validate Bash syntax	bash -n SCRIPT
Submit and capture an ID	JOB_ID=$(sbatch --parsable SCRIPT)
View one active job	squeue --jobs="$JOB_ID"
View your active jobs	squeue --user="$USER"
Explain a job in detail	scontrol show job "$JOB_ID"
Inspect job history	sacct --jobs="$JOB_ID"
Cancel one job	scancel "$JOB_ID"
Start an interactive allocation	srun ... --pty bash

Common Beginner Mistakes

Mistake	Better habit
Running substantial work on a login node	Submit it through Slurm.
Copying somebody else's account or QoS	Use the matching pair from your approved access.
Assuming submission means immediate execution	Expect valid jobs to spend time pending.
Resubmitting while a job is pending	Inspect the reason before taking action.
Requesting resources “just in case”	Measure completed jobs and right-size.
Requesting CPUs the program never uses	Configure application parallelism explicitly.
Forgetting to create the log directory	Create it before submission.
Looking only at job state	Validate logs and scientific output as well.
Losing the job ID	Capture it with sbatch --parsable.

Readiness Checklist

• I can explain the difference between a login node and a compute node.
• I can match my Deceema project code with its permitted QoS.
• I understand tasks, CPUs per task, memory, GPUs, and wall time.
• I can validate, submit, monitor, inspect, and cancel one job safely.
• I can interpret common states, pending reasons, and exit codes.
• I check both scheduler records and application output.
• I can launch a short interactive allocation and release it when finished.
• I can describe when arrays and dependencies are useful.
• I can collect enough evidence for a useful support request.

Continue Learning

The Jobs on Deceema guide goes further into production CPU and GPU templates, reproducibility, storage I/O, failure diagnosis, and operational best practices.

Open Jobs on Deceema

Official Slurm References

These links open in a new tab so you can keep the tutorial beside the official reference material:

• Slurm quick start
• sbatch reference
• squeue reference
• sacct reference
• Job state codes
• Pending reason codes
• Job exit codes
• Job arrays
• Slurm command index