SLURM-POWERED COMPUTING

Jobs on Deceema

Turn a reproducible script into scheduled computation. Describe what your workload needs, let Slurm place it on Deceema, and use evidence from each run to make the next one faster and more efficient.

Submit your first job Diagnose a job

From Idea to Result

• 1. Describe

  Put resource requests and executable commands in a versioned job script.

• 2. Submit

  Send the script to Slurm and record the job ID it returns.

• 3. Observe

  Follow the job through the queue, inspect its state, and preserve its logs.

• 4. Improve

  Compare the request with actual behavior, then right-size the next run.

Deceema uses Slurm to schedule shared compute resources. Instead of running a demanding workload directly on a login node, you describe the resources it needs and submit it as a job. Slurm assigns suitable resources when they become available and gives the run a unique job ID.

Login nodes are for orchestration

Use a login node to inspect files, prepare scripts, submit jobs, and check results. Run compute-intensive or long-lived work through Slurm so it uses allocated compute resources.

Before You Submit

Confirm your project and Quality of Service (QoS) against your Deceema access approval or account details in Deceema Admin. Ask your project owner or Support if the correct pairing is unclear.

Then answer these questions:

• Which project should account for the work?
• Which QoS corresponds to that project?
• Is the application serial, threaded, multi-process, or GPU-enabled?
• How much time, memory, CPU, and GPU capacity does one run need?
• Where are the input data and output directory?
• Can the workload restart from a checkpoint if interrupted?

If this is your first submission, complete Getting Started before continuing.

Your First Production-Ready Job

Create the output directory before submitting because Slurm opens output and error files before the script begins:

$ mkdir -p logs

Save this template as analysis.sh:

#!/bin/bash
#SBATCH --job-name=analysis
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err

set -euo pipefail

module purge
module load deceema

echo "Job ${SLURM_JOB_ID} started on $(hostname) at $(date -Is)"

srun ./analysis --input data/input.dat --output results/output.dat

echo "Job ${SLURM_JOB_ID} finished at $(date -Is)"

Replace PROJECT_CODE, QOS_NAME, the resource values, and the final command with settings appropriate to your account and workload.

Placeholders are not valid settings

Do not submit the example unchanged. PROJECT_CODE and QOS_NAME must be replaced with the matching project and QoS from your approved access.

Read the resource request

Directive	Meaning
--job-name=analysis	Gives the job a recognizable name.
--account=PROJECT_CODE	Selects the project allocation.
--qos=QOS_NAME	Selects the Quality of Service associated with that project.
--time=01:00:00	Sets a one-hour wall-clock limit.
--ntasks=1	Requests one process or task.
--cpus-per-task=4	Assigns four CPU cores to that task.
--mem=8G	Requests 8 GiB of memory for the job.
--output / --error	Separates standard output and error into identifiable files.

In output filenames, %x expands to the job name and %j to the job ID.

A crucial script rule

Keep every #SBATCH directive before the first executable line. Slurm stops processing directives after it encounters the first non-comment, non-whitespace line. It also reads directive values literally, so shell variables such as $ACCOUNT are not expanded inside #SBATCH lines.

Submit and Capture the Job ID

Submit the script:

$ sbatch analysis.sh
Submitted batch job 123456

The job ID—123456 in this example—is the key to every later action. Submission means Slurm accepted the script; it does not mean the job has started.

For automation, request only the identifier:

$ JOB_ID=$(sbatch --parsable analysis.sh)
$ printf 'Submitted %s\n' "$JOB_ID"

Follow a Job Through the Queue

List your jobs:

$ squeue --user="$USER"

Inspect one job in detail:

$ scontrol show job JOB_ID

For a pending job, ask Slurm for an estimated start time when an estimate is available:

$ squeue --start --jobs=JOB_ID

Estimates can change as the queue, priorities, and available resources change.

Common states

Code	State	Meaning
PD	Pending	Waiting for scheduling conditions or resources.
R	Running	The job currently has allocated resources.
CG	Completing	Work ended and Slurm is completing cleanup.
CD	Completed	The job finished successfully from Slurm's perspective.
F	Failed	The job ended unsuccessfully.
TO	Timeout	The job exceeded its requested time limit.
CA	Cancelled	The job was cancelled by a user or system action.
OOM	Out of Memory	The workload exceeded its memory allocation.

State names and codes shown by your Slurm installation are the authoritative source for a specific job.

Understand a Pending Job

Pending is a normal scheduling state, not automatically a fault. Inspect the reason field in squeue or the job details from scontrol show job JOB_ID.

Common categories include:

• Resources: Suitable compute resources are currently in use.
• Priority: Other queued work has higher scheduling priority.
• Dependency: A required earlier job has not satisfied its condition.
• Project or QoS limits: The request has reached a configured usage or job limit.
• Unavailable request: The requested combination of time, CPU, memory, GPU, or other constraints cannot currently be placed.

Do not repeatedly cancel and resubmit an unchanged pending job. That loses its queue history without fixing the request. First understand the reported reason.

Inspect a Completed Job

When accounting data is available, review the job's result and resource use:

$ sacct --jobs=JOB_ID \
    --format=JobID,JobName,State,ExitCode,Elapsed,AllocCPUS,ReqMem,MaxRSS

An exit code is displayed as status:signal. A value of 0:0 usually means the batch script returned successfully without termination by a signal. That does not prove the scientific result is correct—validate expected output too.

Read the logs:

$ less logs/analysis-JOB_ID.out
$ less logs/analysis-JOB_ID.err

Replace JOB_ID with the numeric identifier in the actual filename.

Request the Right Resources

Good resource requests are large enough to succeed and small enough to schedule efficiently.

CPU and tasks

• Use --ntasks for separate processes, such as ranks in a parallel workload.
• Use --cpus-per-task for threads used by one process.
• Confirm that the application is configured to use what you request.
• More CPU does not guarantee a faster result; measure scaling with a representative workload.

Memory

Request enough memory for peak use, not only the input file size. Libraries, in-memory transformations, buffers, and parallel workers all add overhead. After a successful representative run, compare ReqMem with available peak usage such as MaxRSS when accounting collection provides it.

Time

Allow a sensible margin beyond the measured run time. Very large time requests can be harder to schedule, while requests that are too short end in TIMEOUT. Build checkpointing into long workloads where the application supports it.

GPUs

Request a GPU only for software that can use it. Add the resource directive to an appropriate job template:

#SBATCH --gres=gpu:1

A GPU request does not automatically make CPU-only code run on a GPU. Load and configure a GPU-enabled application, then verify that it uses the accelerator.

GPU Job Template

#!/bin/bash
#SBATCH --job-name=gpu-analysis
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=02:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --gres=gpu:1
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err

set -euo pipefail

module purge
module load deceema

srun python train.py --config config/train.yaml

Adjust CPU, memory, time, GPU, project, QoS, environment, and application settings from measured requirements—not from the example alone.

Run Many Similar Tasks with an Array

Job arrays are ideal when the same workflow must run independently over many inputs. This example processes 100 entries while allowing at most 10 array tasks to run at once:

#!/bin/bash
#SBATCH --job-name=dataset
#SBATCH --account=PROJECT_CODE
#SBATCH --qos=QOS_NAME
#SBATCH --time=00:30:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --array=0-99%10
#SBATCH --output=logs/%x-%A_%a.out
#SBATCH --error=logs/%x-%A_%a.err

set -euo pipefail

module purge
module load deceema

INPUT=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" inputs.txt)
srun ./process-one "$INPUT"

%A represents the array's main job ID, %a the array task index, and SLURM_ARRAY_TASK_ID exposes the current index to the script. The %10 limit prevents more than ten tasks from running simultaneously.

Tip

Test one input successfully before submitting a large array. A mistake in one script can otherwise become hundreds of identical failures.

Build Pipelines with Dependencies

Dependencies allow work to advance only when an earlier stage reaches the required outcome:

$ PREP_ID=$(sbatch --parsable prepare.sh)
$ ANALYSIS_ID=$(sbatch --parsable --dependency="afterok:${PREP_ID}" analysis.sh)
$ sbatch --dependency="afterok:${ANALYSIS_ID}" summarize.sh

afterok releases the dependent job only after the earlier job completes successfully. Other dependency types serve different workflows; choose them deliberately and monitor the whole chain.

Use an Interactive Allocation

For short exploration or debugging that genuinely needs compute resources, request an interactive shell through Slurm:

$ srun --account=PROJECT_CODE --qos=QOS_NAME \
    --time=00:30:00 --ntasks=1 --cpus-per-task=2 --mem=4G \
    --pty bash

Replace the placeholders and resource values before running the command. Exit the shell when finished so the allocation is released:

$ exit

Interactive allocations are useful for focused debugging, not for leaving unattended work running. Convert successful exploratory commands into a batch script for reproducibility.

Cancel Work You No Longer Need

Cancel one job:

$ scancel JOB_ID

Cancel only your pending jobs:

$ scancel --user="$USER" --state=PENDING

Before cancelling multiple jobs, inspect the target set with squeue. Avoid broad filters that could stop unrelated project work.

Make Every Run Reproducible

For each production run, preserve:

• The exact job script.
• Application version and loaded modules.
• Configuration and command-line arguments.
• Input data identifiers or checksums.
• Slurm job ID and submission time.
• Standard output and error.
• Code revision or container image identifier where applicable.
• A short statement describing the expected result.

Useful script metadata:

echo "Job ID: ${SLURM_JOB_ID}"
echo "Host: $(hostname)"
echo "Started: $(date -Is)"
module list 2>&1

Never print passwords, tokens, private keys, or other secrets into job logs.

Design Storage and I/O Deliberately

Slurm schedules computation; it does not move your input files automatically. Use paths that are accessible from the allocated compute resources.

• Create one output directory per run.
• Keep source data separate from generated data.
• Avoid many tasks writing to the same output file.
• Avoid producing huge collections of tiny files when a structured format or archive is more suitable.
• Save checkpoints and final results to the intended Deceema storage location.
• Estimate input, intermediate, checkpoint, log, and output space before large runs.

See the Storage guide for data layout, transfer, integrity, capacity, and cleanup practices.

Diagnose a Failed Job

Use a disciplined order so the evidence remains clear.

1. Record the job

$ scontrol show job JOB_ID
$ sacct --jobs=JOB_ID --format=JobID,State,ExitCode,Elapsed,ReqMem,MaxRSS

2. Read both logs

Start at the first meaningful error, not only the final line. Application errors can cause several later failures that hide the original problem.

3. Interpret the result

The job is OUT_OF_MEMORY

Compare requested memory with available peak usage, then inspect whether input size, concurrency, or an application change increased demand. Test a representative correction before scaling up.

The job reached TIMEOUT

Check how much useful progress was made. Request a measured time limit, reduce the workload per job, or enable application checkpointing where supported.

The application says command not found

Confirm the required modules or environment are loaded in the batch script, not only in your interactive shell. Record module list in the log.

The job completed but output is missing

Verify working and output paths, application exit behavior, available storage, and permissions. Remember that Slurm's completed state alone does not validate the scientific output.

The job remains pending

Inspect the pending reason. Verify account and QoS, then assess whether the resource combination is realistic. Do not repeatedly resubmit identical copies.

4. Reproduce at the smallest scale

Use one task, a small input, and a short time limit where appropriate. A small, repeatable failure is faster to diagnose and safer to share with support.

Ask for Job Support

Include the evidence needed to reproduce and route the issue:

Job support details

Deceema username:
Project code and QoS:
Job ID:
Submission date, time, and time zone:
Job state and exit code:
Requested time, tasks, CPUs, memory, and GPUs:
Expected result:
Actual result:
First meaningful error:
Has this workload succeeded before?:
Recent code, data, module, or configuration changes:
Attachments: [job script and sanitized relevant logs]

Remove credentials, tokens, private keys, and confidential data before sharing scripts or logs.

Open Support Check Service Status

Submission Checklist

• The script uses the project and QoS listed in my approved access.
• The resource request matches a measured or defensible workload need.
• Input paths exist and the output directory is writable.
• The output and error directory exists before submission.
• The application environment is loaded inside the script.
• The script records enough context to reproduce the run.
• No credentials or sensitive values can appear in commands or logs.
• I know how to monitor, cancel, and inspect the job afterward.

Official Slurm References

• Submit batch jobs with sbatch
• View queued jobs with squeue
• Inspect jobs with scontrol
• Review accounting data with sacct
• Understand job states
• Work with job arrays