This explains the new job queue priority, which has been activated on April 11th. 2016.
- Default walltime for jobs: 4:00:00
- New queue for "short" jobs with a walltime less than 12 hours. Nodes reserved: 56.
- Priority bonus if jobs are submitted to a project account.
- Fairshare has been disabled. No priority penalty for submitting jobs.
- A continuesly running process elevates the priority for small batches of jobs, regardless of the users priority.
- A new 'priority' tool
The primary reason for changing the scheduling model, is to improve the experience of using the cluster.
When you submit a job at the GenomeDK cluster, it is now given one of the following initial priorities:
- 10000 - regular user (no project account is specified)
- 20000 - regular project (a project account is specified using -A for sbatch/srun)
- 30000 - high priority project
Example of specifying a project account: sbatch -A DanishPanGenome jobscript.sh
At regular intervals (~once per minute) select jobs will be elevated to an extra high priority level (50000).
The reason for the elevated level, is to give each waiting user a fair chance to start at least one job within a reasonable time, even if other users have thousands of jobs already waiting in queue.
At least one job for each user are permitted to wait at the elevated level. Up to 24 jobs for each user are permitted there, if they are very small. Every minute the system will try to elevate more jobs and every minute each old, elevated job gets one additional priority point.
Once the job has been elevated, its expected start time will be much sooner.
Here is a detailed description on how jobs are picked for elevation:
- Jobs are picked strictly in order of priority.
- The elevated jobs+running jobs of a user must not together ask for more than 24 cores.
- The elevated jobs+running jobs of a user must not together ask for more than 256 GB of memory.
Use the new 'priority' command to list your queued jobs and to give insights into when your jobs will run.
[runef@fe1 runef]# priority -a
POS JOBID AGE(minutes) PRIORITY USER NAME
1 23964764 25 50000 xxx1 PCA
2 23964449_[30-999] 66 20066 xxx2 cmds
3 23964767 7 10007 xxx3 dispatch