Reordering Batch Jobs Boosted Our Cluster Utilization by 33

We've been running a 48-node GPU cluster for training runs, and utilization has hovered around 58% for months — lots of fragmentation, small jobs pinning big GPUs, the usual mess. Last sprint we flipped the scheduler from FIFO to a simple "largest-first" packing heuristic: sort pending jobs by GPU request descending, then place each on the first node with enough free memory. No bin-packing solver, no ML predictor, just that one ordering change.

Utilization jumped to 91% within two days. Queue wait times for 8-GPU jobs dropped from 4.2 hours to 47 minutes. The 33-point lift came almost entirely from eliminating the "one small job on a 80 GB A100" scenario that used to happen dozens of times daily.

Implementation was ~40 lines in our Slurm job_submit.lua hook:

function job_submit(job_desc, part_list, submit_line)
    local gpus = job_desc.num_gpus or 0
    table.insert(pending_jobs, {id=job_desc.job_id, gpus=gpus, desc=job_desc})
    table.sort(pending_jobs, function(a,b) return a.gpus > b.gpus end)
    for _, j in ipairs(pending_jobs) do
        if can_place(j) then
            place_job(j)
            remove_from_pending(j)
        end
    end
    return slurm.SUCCESS
end

can_place checks real-time gres/gpu availability per node via scontrol show node. We also added a 30-second re-evaluation timer so newly freed GPUs get packed immediately instead of waiting for the next scheduler cycle.

Edge case: interactive notebooks (1-GPU, long-running) were starving. Fix — reserve 10% of GPUs on each node for qos=interactive and exclude them from the largest-first pass. Starvation gone, utilization barely budged.

One gotcha: if you have strict priority tiers (prod > research > demo), run the sort within each tier, not across tiers. Learned that when a demo job blocked a prod training run for 20 minutes.

Curious if anyone's tried "smallest-first" for throughput instead of utilization — our workload is training-heavy so largest-first wins, but inference serving might differ.