DECEEMA DATA INFRASTRUCTURE

Storage Built for Discovery

Keep data close to the compute that transforms it. Deceema provides shared, high-capacity storage for research inputs, active workloads, and valuable results.

~100 TBcurrent shared storage capacity

Deceema currently operates approximately 100 TB of storage. This is shared infrastructure: available capacity changes as projects create, process, archive, and remove data. Good data stewardship keeps the platform responsive and useful for everyone.

Think in Data Lifecycles

Storage is most effective when every file has a purpose and a destination.

• Bring Data In

  Transfer deliberately, verify integrity, and place data where the project can use it.

• Work with Structure

  Separate source data, code, temporary files, logs, and results so each is easy to understand.

• Compute Efficiently

  Design file access for shared HPC storage and avoid unnecessary I/O.

• Keep What Matters

  Preserve valuable results, archive completed work appropriately, and remove data that no longer serves the project.

Know Where You Are Working

Your Deceema home directory is:

/hpc/home/$USER

$USER represents your Deceema username. Use your home directory for personal configuration, scripts, source code, and other user-specific files that belong with your account.

Project datasets and shared outputs should live in the storage location assigned to the project. Confirm the correct location with your project owner or the Deceema team before moving significant data.

Shared capacity is not a personal allocation

The approximately 100 TB figure describes Deceema's current shared storage capacity. It does not mean that every account or project can consume 100 TB. Follow the allocation and retention expectations for your project.

Design a Project Layout

A predictable directory structure makes collaboration, automation, and cleanup safer. One possible starting point is:

Example project layout

project/
├── README.md
├── data/
│   ├── raw/
│   └── processed/
├── src/
├── config/
├── jobs/
├── logs/
├── results/
└── archive/

Directory	Purpose
data/raw/	Original source data. Treat it as read-only where practical.
data/processed/	Derived inputs that can be recreated from source data and code.
src/	Scripts, notebooks, and application source.
config/	Versioned parameters and environment configuration without secrets.
jobs/	Slurm job scripts used to produce results.
logs/	Standard output, error logs, and diagnostic records.
results/	Reviewed outputs intended for analysis, sharing, or preservation.
archive/	Completed material retained according to project policy.

Add a short README.md at the project root explaining ownership, data sources, important commands, naming conventions, and retention decisions.

Understand Your Usage

Check the size of your home directory:

$ du -sh "$HOME"

See which top-level directories use the most space:

$ du -h --max-depth=1 "$HOME" | sort -h

Inspect the filesystem's currently reported capacity:

$ df -h "$HOME"

Note

df reports filesystem-wide capacity, not necessarily your account or project's allocation. Use it as context, not as proof that you may consume all space shown.

Running du across a very large directory tree can take time and generate substantial metadata activity. Start with the smallest relevant path.

Plan Capacity Before a Large Run

Estimate more than the input dataset. A workload may also create intermediate files, checkpoints, logs, temporary copies, and multiple result versions.

Use this simple planning model:

required space = input + intermediate data + output + checkpoints + safety margin

Before a data-intensive run, ask:

• How large is the source dataset?
• Will tasks create a copy or transformed version of each input?
• How many jobs can write output concurrently?
• How often are checkpoints written, and how many are retained?
• Which outputs are reproducible and which are irreplaceable?
• What will be removed or archived when the work finishes?

If the expected footprint is substantial or uncertain, discuss it with the Deceema team before launching the workload.

Transfer Data Reliably

Use a transfer method approved for your Deceema environment. For resumable command-line transfers between accessible locations, rsync is often useful:

$ rsync -avh --partial --progress SOURCE/ DESTINATION/

The trailing slash on SOURCE/ means “copy the contents of this directory.” Review both paths carefully before starting a large transfer.

Verify integrity

For important files, calculate a checksum at the source and destination:

$ sha256sum FILE

Matching checksum values provide evidence that the file arrived unchanged. Record checksums in a manifest when transferring a collection of important research files.

Keep credentials out of transfer commands

Do not embed passwords, tokens, private keys, or other secrets in commands, scripts, filenames, or logs. Use the approved authentication method for the transfer service.

Make HPC I/O Work for You

Shared storage performs best when applications use it deliberately.

• Avoid millions of tiny files when possible. Opening and listing many small files can create more overhead than processing their contents.
• Use one output directory per run. This prevents jobs from overwriting one another and makes results easier to trace.
• Do not let many tasks append to one file. Coordinate output through the application or write task-specific files and combine them afterward.
• Write checkpoints intentionally. Choose an interval that protects useful progress without filling storage with redundant snapshots.
• Keep logs useful. Capture enough detail to reproduce failures, then compress, archive, or remove noisy logs according to project policy.
• Separate temporary and durable data. Do not keep reproducible intermediate files longer than the workflow needs them.

For scheduled workloads, document the input and output paths directly in the job script or its accompanying configuration. See Jobs on Deceema.

Tame Collections of Small Files

If a workflow produces many small files, packaging them can simplify transfer and long-term handling:

$ tar -cf collection.tar DIRECTORY/

List an archive before extracting it:

$ tar -tf collection.tar

Create archives only when they improve the workflow. A large archive is not a substitute for a backup, and changing one file can require recreating the archive.

Collaborate Without Losing Control

Before sharing a directory, identify:

• The owning project and responsible project lead.
• Who needs read, write, or administrative access.
• Whether the data has confidentiality or handling requirements.
• How membership changes will be managed.
• What happens to the data when the project ends.

Inspect ownership and permissions without changing them:

$ ls -ld PATH

Do not make a directory broadly writable as a quick fix. If ownership or access is incorrect, capture the path and permission output and contact Support.

Capacity Is Not the Same as Backup

Do not assume a file is backed up merely because it is stored on Deceema. Confirm the protection, retention, and recovery policy for the specific storage location your project uses.

For irreplaceable data:

• Keep an approved independent copy where policy permits.
• Record checksums and provenance.
• Test that important files can be read.
• Document who owns recovery decisions.
• Do not rely on a running job, portal session, or container as permanent storage.

Clean Up with Confidence

Deletion should be deliberate and reviewable.

1. Confirm the exact directory and project owner.
2. Identify which data is original, reproducible, or irreplaceable.
3. Check whether collaborators or active jobs still use it.
4. Archive or transfer material that must be retained.
5. Verify the retained copy before removing the source.
6. Record what was removed and why.

A fast cleanup can become permanent data loss

Never run a recursive deletion command against an unresolved variable, wildcard, home directory, project root, or path you have not inspected.

Troubleshoot Storage Problems

I see “No space left on device”

Capture the exact error, affected path, time, df -h output for that path, and the smallest useful du summary. The cause may involve shared capacity, an allocation limit, or another filesystem constraint; do not assume which one from the message alone.

I see “Permission denied”

Run ls -ld PATH and confirm the project code, expected access, and whether teammates are affected. Do not apply broad permission changes. Include the sanitized output in a support request.

A directory is unexpectedly slow

Record the path, operation, time, approximate file count and sizes, and whether jobs are accessing it concurrently. Test the smallest safe example and check Service Status.

A file is missing

Stop processes that could overwrite related data. Confirm the exact path, filename, last known time, owning project, and actions performed before it disappeared. Contact support promptly and do not create replacement files at the same path until recovery options are understood.

My job creates too much data

Stop launching additional copies, identify the job IDs and output path, and estimate the current growth rate. Review which intermediate files can be reduced, combined, or omitted before resuming.

Request Storage Support

Include this information with a storage request:

Storage support details

Deceema username:
Project code:
Affected path:
Date, time, and time zone:
Expected result:
Actual result and exact error:
Approximate data size and file count:
Affected job IDs, if any:
Does the issue affect other project members?:
Relevant df, du, or ls output:
Recent transfers, jobs, or permission changes:

Sanitize the details before sending them. Never attach credentials, tokens, private keys, or confidential data.

Open Support Check Service Status

Storage Checklist

• I know which location should hold personal and project data.
• My project has enough capacity for input, intermediates, output, and a safety margin.
• Important transfers are verified with checksums.
• Permissions give access only to the people who need it.
• Job output is organized by run and does not overwrite source data.
• I understand the retention and recovery policy for this location.
• Completed work has been retained or removed according to project policy.