SERVICE HEALTH · INCIDENTS · MAINTENANCE

Deceema Service Status¶

Know what is working, understand what is affected, and take the right next step. This page is your guide to Deceema service health, planned maintenance, incident updates, and user-side checks.

Run the quick checks Report a problem

About live status

This documentation page explains how Deceema service status is communicated; it is not itself a live telemetry dashboard. Consult the approved Deceema announcement or status channel for current incident and maintenance bulletins. If no bulletin is visible, do not assume that a problem is already known—run the checks below and report it when needed.

Start Here¶

When something does not behave as expected, answer three questions:

What is affected? Login, jobs, storage, the portal, an application, or network access?
Who is affected? Only you, everyone in your project, or several users?
When did it begin? Record the date, time, and time zone of the first failure.

Those answers separate a local configuration problem from a wider service incident and give the Deceema team a useful head start.

Access and Login

Authentication, SSH access, login nodes, accounts, projects, and QoS visibility.

Check access
Scheduler and Compute

Slurm submission, queue processing, CPU and GPU nodes, and job execution.

Diagnose jobs
Storage and Data

Home and project paths, capacity, permissions, data access, and transfer.

Check storage
Portal and Applications

Browser access, interactive sessions, hosted applications, VMs, and containers.

Check the portal

Services Covered¶

Status communications may cover one service, several related components, or the whole platform.

Service area	What it includes	A useful first signal
Identity and access	Authentication, account access, projects, and QoS visibility	Can affected users sign in and use their approved project details?
Login service	SSH connectivity and login-node sessions	Can users establish a new session?
Slurm scheduler	Job submission, queue state, and allocation processing	Does `sbatch` accept a small valid job and return a job ID?
Compute	CPU nodes, GPU nodes, and job execution	Are jobs starting, or are many valid jobs waiting for the same reason?
Storage	Home and project filesystems, capacity, and data access	Can users list and read the expected path?
Deceema Portal	Web access and interactive-session launch	Does the portal load, and can a minimal session launch?
Hosted applications	Platform applications, VMs, and containers	Is one application affected or every hosted service?
Network and transfer	Platform connectivity and movement of data	Is the problem tied to one location, path, or transfer method?

Understand Status Labels¶

A concise status label communicates impact quickly; the accompanying update should always define the affected service and scope.

Status	Meaning	What users should do
Operational	The service is operating normally based on available observations.	Continue working and report unexpected behavior with evidence.
Degraded performance	The service is available, but slower or less reliable than expected.	Avoid duplicate submissions, preserve job IDs, and follow updates.
Partial outage	Some functions, users, nodes, or applications are unavailable.	Use a published workaround only when it applies to your case.
Major outage	A core service is unavailable to most or all users.	Pause repeated retries and monitor the official incident update.
Maintenance	Planned work may reduce or interrupt availability.	Save work, respect the maintenance window, and avoid starting jobs that overlap it.

Operational does not mean every job will succeed

A healthy platform can still receive an invalid script, encounter an application error, or keep a large request pending while resources are busy. Use the job ID, state, reason, logs, and exit code to tell the difference.

Follow the Incident Lifecycle¶

Status updates should evolve as the team learns more:

Investigating → Identified → Mitigating → Monitoring → Resolved

Phase	What it tells you
Investigating	The issue is confirmed or suspected, and its cause or scope is still being determined.
Identified	The likely cause and affected services are understood.
Mitigating	Work is under way to restore service or reduce impact.
Monitoring	A fix or workaround is in place and stability is being observed.
Resolved	Service has returned to the expected operating state.

A useful update contains the affected component, user impact, current phase, start or detection time, next update time when known, and any safe workaround. An estimated restoration time should be treated as an estimate—not a promise.

Example incident update¶

Example only — not a live incident

Slurm scheduler · Degraded performance · Investigating

Some users may experience delayed job submission or queue updates. Jobs already running are not currently known to be affected. The team is investigating; avoid submitting duplicate copies of the same job. First observed: 14:20 WAT. Next update: 15:00 WAT.

This example is useful because it distinguishes submission from execution, describes the observed impact, gives users a safe action, and anchors the update in time.

Prepare for Planned Maintenance¶

Maintenance notices should state the date, time zone, affected services, expected impact, and any action users must take. Before a window begins:

check whether a running job could overlap the window;
save portal and interactive-session work to persistent storage;
stop non-essential transfers before a storage or network maintenance window;
record active job IDs and important paths;
avoid launching work that depends on a service scheduled to be unavailable;
follow published shutdown or resubmission instructions; and
wait for the completion notice before assuming normal service has returned.

Do not rely on an interactive session as permanent storage

Save important work to the appropriate Deceema storage path before maintenance. A browser tab, terminal session, or running process is not a durable copy of your data.

Check Your Path in Three Minutes¶

Use the smallest safe check that represents the failing path. Do not create a storm of retries or submit many duplicate jobs.

1. Establish context¶

$ date -Is
$ hostname
$ whoami

Record the output that is relevant to the problem. Never share passwords, multi-factor codes, private keys, access tokens, or other secrets.

2. Check scheduler visibility¶

$ squeue --user="$USER"
$ sinfo

If a job is affected, inspect that exact job:

$ squeue --jobs=JOB_ID --format="%.18i %.20j %.2t %.10M %R"
$ scontrol show job JOB_ID
$ sacct --jobs=JOB_ID --format=JobID,JobName,State,ExitCode,Elapsed

A pending reason such as resources, priority, dependency, account, or QoS can explain a valid wait without indicating a platform outage.

3. Check the exact storage path¶

$ ls -ld AFFECTED_PATH
$ df -h AFFECTED_PATH

Use the path involved in the failure. Do not recursively scan a large project, change permissions, or delete files simply to test availability.

4. Check the portal path¶

For a portal or hosted-application problem, record:

the application and page or action that failed;
whether the portal itself loads;
whether a new private browsing session behaves the same way;
the requested session resources, if relevant;
the browser name and version; and
the exact date, time, time zone, and visible error.

Is It the Platform or My Workload?¶

No single signal proves the answer, but scope and repeatability are powerful.

Observation	More likely	Next action
Several users and unrelated projects lose the same service at the same time	Shared service issue	Check official updates and report the common scope if it is not acknowledged.
One script fails consistently while other jobs run normally	Script, environment, input, or application issue	Inspect the job's logs and exit code; try the smallest representative case.
One user cannot use a project that teammates can access	Account or project membership issue	Compare approved memberships and follow access support guidance.
Many jobs are pending for `Resources`	Normal contention may be involved	Keep one correct submission, right-size it, and monitor rather than duplicating it.
Every tested path on one filesystem stalls or fails for several users	Storage service issue	Stop repeated heavy checks and report the exact filesystem and time.
One portal application fails while the portal and other applications work	Application-specific issue	Capture the application name, launch details, and visible error.

Report an Unlisted Issue¶

If the issue is not acknowledged—or your impact differs from the published update—send a focused report through the approved Deceema support channel.

Service issue report

Subject: [Service area] Brief description

First observed: [date, time, and time zone]
Affected service:
Scope: [only me / project team / multiple users]
Research or operational impact:

Expected result:
Actual result:
Exact error:
Smallest safe steps that reproduce it:

Username:
Project code, if relevant:
Job ID, if relevant:
Hostname or application, if relevant:

Checks already completed:
Relevant sanitized logs or screenshot:

Security incidents take a different path

If you suspect compromised credentials, unauthorized access, or exposed sensitive data, stop the affected activity and contact the approved Deceema security or urgent-support channel immediately. Do not post secrets or sensitive evidence in a public status conversation.

Open Support Review Jobs on Deceema

After Service Is Restored¶

When an incident or maintenance window is marked resolved:

Start with a small representative check rather than your largest workload.
Inspect jobs that were pending, interrupted, cancelled, or timed out.
Verify outputs before continuing a dependent workflow.
Resume transfers and submissions gradually.
Report any remaining symptom with its new timestamp and previous incident context.

“Resolved” means the service is expected to be available again; it does not guarantee that interrupted application work automatically resumed or produced valid results.

Status-Ready Checklist¶

I know which service and action are affected.
I recorded the first failure time and time zone.
I know whether the problem affects only me or multiple users.
I checked the relevant job ID, path, or application without flooding the service.
I preserved useful logs and exact error messages.
I removed secrets and sensitive data from anything I plan to share.
I checked the approved Deceema status or announcement channel.
I can describe the research or operational impact clearly.