Structure of SCITAS filesystem
- The structure and purpose of each filesystem is described here File systems
- $HOME and $WORK are shared across the site, while $SCRATCH is local to each machine.
- For $SCRATCH Automatic deletion of files older than 2 weeks may happen without notice.
- Production jobs should run on $SCRATCH.
What to do when CPU time and WALL time are significantly different?
- $SCRATCH mount a GPFS parallel filesystem which is designed to perform well with parallel I/O.
- In certain cases a big number of files is produced at runtime. This IO pattern puts stress on the $SCRATCH and can cause hardware failure from time to time.
- A proper solution would require the usage of external libraries like HDF5 or ADIOS. Those libraries gives certain flexibility in the way data are saved/handled.
- A workaround would be to use the local filesystem $TMPDIR
- $TMPDIR is visible only once resources are allocated. You can try to query the value of $TMPDIR after the login :
- The variables $TMPDIR, $WORK and $SCRATCH are set by the SLURM prescheduler
How to use the $TMPDIR in your simulations
- The following example show how to set the $TMPDIR with Quantum-ESPRESSO(QE).
- QE relies on fortran namelists to read certain parameters used during the simulation.
- The only change that has to be done to a standard pw input is related to the outdir in the &CONTROL namelist. For example, in the input below the outdir is set equal to fakeoutdir:
- The submission script would look like
- After the sed command the CONTROL namelist looks like:
- For a single 100GB file, all results in MB/s, <write into TMPDIR> : <copy from TMPDIR to /scratch>: