Case Status
Log In

Wiki

Options

 
Job Queue Priority
  • RSS Feed

Last modified on 11-04-2016 15:12 by User.

Tags:

Job Queue Priority

 

This explains the new job queue priority, which has been activated on April 11th. 2016.

Main changes:

  • Default walltime for jobs: 4:00:00
  • New queue for "short" jobs with a walltime less than 12 hours. Nodes reserved: 56.
  • Priority bonus if jobs are submitted to a project account.
  • Fairshare has been disabled. No priority penalty for submitting jobs.
  • A continuesly running process elevates the priority for small batches of jobs, regardless of the users priority.
  • A new 'priority' tool

 

The primary reason for changing the scheduling model, is to improve the experience of using the cluster.

 

When you submit a job at the GenomeDK cluster, it is now given one of the following initial priorities:

  • 10000 - regular user (no project account is specified)
  • 20000 - regular project (a project account is specified using -A for sbatch/srun)
  • 30000 - high priority project

 

Example of specifying a project account: sbatch -A DanishPanGenome jobscript.sh

 

At regular intervals (~once per minute) select jobs will be elevated to an extra high priority level (50000).

The reason for the elevated level, is to give each waiting user a fair chance to start at least one job within a reasonable time, even if other users have thousands of jobs already waiting in queue.

At least one job for each user are permitted to wait at the elevated level. Up to 24 jobs for each user are permitted there, if they are very small. Every minute the system will try to elevate more jobs and every minute each old, elevated job gets one additional priority point.

Once the job has been elevated, its expected start time will be much sooner.

Here is a detailed description on how jobs are picked for elevation:

  • Jobs are picked strictly in order of priority.
  • The elevated jobs+running jobs of a user must not together ask for more than 24 cores.
  • The elevated jobs+running jobs of a user must not together ask for more than 256 GB of memory.

 

Use the new 'priority' command to list your queued jobs and to give insights into when your jobs will run.

[runef@fe1 runef]# priority -a
   POS                JOBID    AGE(minutes)        PRIORITY    USER        NAME
     1             23964764              25           50000    xxx1        PCA
     2    23964449_[30-999]              66           20066    xxx2        cmds
     3             23964767               7           10007    xxx3        dispatch