Engineering notes GPU scheduling

Tetris with GPUs

Packing elastic, preemptible jobs across GPU shapes and job lengths

Fleet AI EngineeringKueue + Ray · October 2026 · 36 min read

In a GPU-scarce world, one of the biggest opportunities is to make better use of the capacity we already have. A cluster can have GPUs sitting free while jobs wait. The available GPUs may be scattered across nodes, while the next job needs a whole group together.

We treat this as Tetris with pieces that can change size. Some jobs resize during execution. Others restart from a checkpoint with a different allocation. Jobs finish at different times, opening new gaps as others close.

Our goal is to approach full GPU allocation while preserving useful progress. Two assumptions matter: we can influence job design, and enough compatible work is waiting to run.

In one week on a 192-GPU cluster, offline evaluations used 3,444 GPU-hours that training and other jobs left free. They filled 35% of that spare capacity. This was an observation of one workload mix, not a controlled before-and-after experiment. Near-full allocation remains a target. See the measurements and their limits.

eval GPU-hours in seven days: a third of what training left unallocated: 3,444
mean GPU allocation: 70% training and other jobs, plus 10% evals in the same week: 80%
rollouts across 83 checkpoint evals, 98.8% graded: 205,189

The cluster is our board

Eight free GPUs do not necessarily fit an eight-GPU job. The job might need all eight on one node, while the free GPUs span four nodes. A larger job may also need its nodes under the same network switch.^{[6]6Kueue: Topology-Aware Scheduling}

Capacity changes over time. Jobs finish and release GPUs; higher-priority jobs arrive and reclaim them. An opportunistic allocation might last minutes or hours. Filling every slot helps only when useful progress outweighs startup, checkpointing, and recovery costs.

Allocation is capacity assigned to jobs. Utilization measures activity on those GPUs. Useful progress is completed work after retries and recovery. Good packing should improve the last measure, not just the first.

Reading the diagrams: each vertical node has eight GPU slots, filled from bottom to top. The queue shows pending admission requests; filled slots show allocations and hollow slots show free capacity. Pale q1 slots are other work outside the example. Growth requests include the existing allocation. The evaluation ledger records completed items separately from GPU admission. Counts are illustrative. Each diagram can open the matching 64-GPU scenario in the sandbox; the sandbox starts from its own scaled setup.

Figure 1Eight free GPUs. Will the whole request fit?Illustrative layout · frequency measuredOpen in sandbox #

Queue · Kueue

q0q1q2

q2
1 pod × 8 GPUs
q2
1 pod × 4 GPUs

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

Cluster setup

Starting free capacity

New job

GPUs per pod

Placement

State

Eight GPUs requestedEight GPUs requested. Try the allocation to see whether it fits.

ExistingJob A · requestJob B · short trial

Each square is one GPU. Every pod must fit on one node. A fixed request is admitted only when all its pods fit, including any same-switch constraint.

Choose a request in the queue, then admit it. Outlined nodes show a blocked fit or a successful placement. Changing the starting layout or requirements resets this example; it does not relocate running pods.

Identical GPUs and available CPU and memory are assumed. Selecting a smaller request explores a possible admission policy. This is not a simulation of every Kueue policy. Topology requirements can change the fit.

RayJob: the fixed request this diagram places(click to expand)

Job A in this diagram is a fixed gang: every pod is admitted together or none is, and each pod must fit on one node. This is one on dev: two workers of two GPUs on one node, one Workload admitted whole. The diagram places other sizes of the same shape; the counts it shows are illustrative.

Named job-a here to match the diagram; the dev run used tetris-gang.

# Fixed gang, native Kueue path: ONE Workload (PodSets head + gang) admitted all-or-nothing.
# 2 workers x 2 GPUs = 4 GPUs, both pods on one node. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-a
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: fixed-gang
    kueue.x-k8s.io/queue-name: training-lq     # Kueue manages the job (there is no default queue)
    kueue.x-k8s.io/priority-class: q1          # [shape] q: one queue priority for the whole job, head included
spec:
  suspend: true                     # created suspended; Kueue unsuspends it once the whole gang fits
  shutdownAfterJobFinishes: true    # required by Kueue
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 1800       # per admission attempt
  backoffLimit: 0
  submissionMode: HTTPMode          # no submitter pod: the driver runs on the head
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"            # [shape] no enableInTreeAutoscaling: the size never changes
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}          # no tasks on the head
      template:
        metadata:
          labels: {blog.fleet.ai/example: fixed-gang}
        spec:
          priorityClassName: c2                # c2 never preempts: the head shares nodes with the control plane
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head (no topology annotation)
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}   # app.py + out-dir from the ConfigMap
                - {name: sfs, mountPath: /mnt/sfs}         # checkpoints outlive the RayCluster
          volumes:
            - {name: app, configMap: {name: job-a-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: gang
        replicas: 2                   # [shape] fixed: replicas == minReplicas == maxReplicas
        minReplicas: 2
        maxReplicas: 2
        rayStartParams: {num-gpus: "2"}
        template:
          metadata:
            labels: {blog.fleet.ai/example: fixed-gang}
            annotations:
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname   # [shape] both pods on ONE node
          spec:
            priorityClassName: c1     # [shape] c: kube-scheduler priority of the worker pods
            nodeSelector: {workload: fleetai-training-ng-gpu}                   # -> flavor b300-training
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}

RayJob: eight GPUs as two 4-GPU pods that may sit on different nodes(click to expand)

The unconstrained twin of an eight-GPU, one-node request. The worker group has two replicas of four GPUs and podset-unconstrained-topology, so Kueue may place the pods on different nodes. The single-node request has one replica of eight GPUs and podset-required-topology: kubernetes.io/hostname. Both ran at q2so neither could preempt anything.

# The same eight GPUs as the whole-node request, as a gang of two 4-GPU pods that MAY span nodes (unconstrained topology), at q2.
# Used by multi-node/run.sh scatter: it fits in the two nodes' four free GPUs each, where the whole-node request cannot.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-two-by-four
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: multi-node
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] q2
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 900
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: |
    python -u - <<'EOF'
    import socket, time, ray
    ray.init()

    @ray.remote(num_gpus=1)
    def hold(seconds):
        time.sleep(seconds)
        return socket.gethostname()

    print("two-by-four job held 8 GPUs for 30 s on", set(ray.get([hold.remote(30) for _ in range(8)])))
    EOF
  rayClusterSpec:
    rayVersion: "2.49.2"
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: multi-node}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
    workerGroupSpecs:
      - groupName: node
        replicas: 2                   # [shape] fixed: two 4-GPU pods
        minReplicas: 2
        maxReplicas: 2
        rayStartParams: {num-gpus: "4"}
        template:
          metadata:
            labels: {blog.fleet.ai/example: multi-node}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the two pods may sit on different nodes
          spec:
            priorityClassName: c2     # [shape] c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "4"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "4"}

Measured on dev, October 23, 2026, one run.Two plain pods outside Kueue held four GPUs each, one per node; the second could not be placed until the node group's autoscaler added a second GPU node, which took 512 seconds. With four GPUs free on each of the two nodes, the single-node eight-GPU request stayed pending: Kueue's message was that the topology does not allow it to fit any of its one pod, and it waited 31 minutes. The same eight GPUs as two 4-GPU pods were admitted the moment they were submitted, with a topology assignment naming both nodes, and the job finished 63 seconds later. The waiting request was admitted 4 seconds after the first blocker ended and freed a whole node, and ran there. On October 24, eight GPU ranks in two pods on the two nodes formed one NCCL group: every rank agreed on the all-reduce sum, and a 512 MB all-reduce took 167 milliseconds across the nodes against 1.3 milliseconds among the four ranks of one pod. On October 25 the FSDP2 trainer of the checkpoint section ran across the two nodes: four ranks on one, two on the other, 15 epochs, and all 14 reshard loads matched their digests. A gang above our 8-GPU quota stays pending (a 16-GPU request waited 71 minutes), because the quota is a GitOps value that ArgoCD re-asserts within a second; run through a temporary queue with a 16-GPU quota, two whole-node pods formed one 16-rank NCCL group, with a 512 MB all-reduce taking 263 milliseconds across the nodes against 1 millisecond within one. A gang spanning more than two nodes was not tested.^{[17]17Job-shape examples, run records and claim audit, October 16 to 26, 2026}

Figure 2Play the scheduling sandbox#

The board is interactive: open this article online to play it, or open the sandbox on its own at /sandbox/scheduling.html.

Runs in this page and talks to no cluster.Open full screen

The game the rest of the article refers to: an eight-node fat tree under the same Kueue rules as our training cluster. Submit jobs of different shapes and priorities, step time, and watch admission, growth, preemption and placement. It is a model of the rules, not a measurement; the measured week is in the figure near the end.

Scope and implementation

We focus on jobs that run to completion: training, evaluation, scoring, and synthetic data generation. Serverless traffic scaling and long-running deployments are outside this post. Serving pools appear only when they support finite work with a defined completion condition.

We use Kueue for queueing and admission, and a dedicated Ray cluster for each RayJob.^{[1]1Kueue: Run a RayJob} Ray manages execution; applications supply their checkpoint, retry, and progress-tracking logic.

A RayJob's pods come in groups: one CPU head that coordinates the job, and one or more groups of GPU workers. We call each group a workgroup; Ray calls the GPU ones worker groups. Kueue tracks each admission request as a Workload. The priority names used below, q0 to q2 and c0 to c2, are defined in the next section.

With our adapter, the CPU head has its own admission. The diagrams give that workload priority q0 so lower-priority worker reclamation leaves it alone. Pod priority is separate: the example manifests use c2 for the head so it cannot preempt other CPU pods. CPU capacity is still required, and no priority protects against every failure.^{[10, 11]10Kueue Ray worker-group adapter11Fleet queue and pod priority configuration}

We use two admission paths. Native admission treats a RayJob as one workload. Our adapter separates its workgroups. The week of evaluation measurements uses the native path; the adapter has separate lifecycle tests. Both run with our patched Kueue build.

Two extensions support this design:

Our RayJob adapter (private) gives the head and each worker group separate Kueue admissions.^{[10]10Kueue Ray worker-group adapter}
Our Kueue changes (private) let jobs set the timeout before an unready allocation is requeued.^{[13]13Fleet Kueue preemption and readiness recovery}

The adapter preserves sibling allocations when one group is reclaimed. Its implementation contract also describes the application responsibilities and supported inputs.^{[10]10Kueue Ray worker-group adapter}

Two priority layers

A RayJob does not itself provide the job-level priority queue we need. Queue priority applies to a Kueue Workload and influences admission and workload preemption.^{[3, 4]3Kueue: Workload Priority Class4Kueue: Preemption} Pod priority influences Kubernetes scheduling and pod preemption after admission.^{[5]5Kubernetes: Pod Priority and Preemption}

Our configured levels are q0/c0 = 20,000, q1/c1 = 10,000, and q2/c2 = 5,000. The lower-numbered name has higher priority. These are separate controls, even when an API pairs them by policy.^{[11]11Fleet queue and pod priority configuration}

The queue uses LowerPriority reclamation. A higher-priority workload can displace a lower-priority one in that queue.^{[4, 11]4Kueue: Preemption11Fleet queue and pod priority configuration} For pods, c2 has preemptionPolicy: Never: it cannot evict other pods to make room for itself. It can still be preempted.^{[5, 11]5Kubernetes: Pod Priority and Preemption11Fleet queue and pod priority configuration}

Figure 3A higher-priority job arrivesMeasured on dev · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

1/9 · Elastic workers occupy Node B1/9 · Elastic workers occupy Node B. A fixed q1 job holds Node A. An elastic q2 group holds Node B. This is an illustrative sequence, not a measured run.

Job A · fixedJob B · elastic evalJob C · priority

Draining · still allocated

Elastic workers occupy Node B. A fixed q1 job holds Node A. An elastic q2 group holds Node B. This is an illustrative sequence, not a measured run.
The q0 job needs one whole node. Priority alone does not create capacity. First explore an application-requested scale-down to four workers; Kueue preemption itself does not shrink a workgroup.
Four workers drain; capacity is still held. Do not admit the new job onto terminating pods. Unfinished evaluation items will be retried from the application ledger.
Four free GPUs still cannot fit an eight-GPU pod. Scale-down released four GPUs. The pending job still needs eight together. Under this illustrative policy, the lower-priority workgroup is selected for whole-group preemption.
The remaining workgroup drains. The four remaining workers are terminating. Their allocation is still held; the group is not yet ready for readmission.
Node B is free; unfinished work waits. Old pods are gone. The q2 group now waits for readmission. Completed application items remain complete; unfinished items remain retryable.
The q0 job is admitted; q2 waits. The higher-priority job occupies Node B. The fixed job on Node A never moved. Admission is shown as allocation; actual pod startup is separate.
Capacity returns to the queue. The priority job completed. The elastic group can now request its four-worker target again.
Replacement workers retry unfinished items. Four replacement workers are admitted. Application recovery uses the ledger; queue readmission does not restore application state by itself.

Application scale-down and Kueue whole-workgroup preemption are separate operations. Capacity held by draining pods is unavailable. This example assumes a recoverable evaluation ledger.

Step through a possible sequence: application-requested shrink, then whole-group preemption when four free GPUs cannot fit an eight-GPU pod. This is an explanation of the distinction, not a claim that Kueue automatically shrinks jobs first.

Nodes and GPU slots stay in place. Back and Reset rewind the illustration, not a real cluster. Kueue preemption reference.

Measured on a one-node dev cluster on October 16 and 20, 2026. First run: a q1 four-GPU gang arrived while a q2 elastic eval held its GPUs and only two were free. Kueue evicted the eval's whole Workload, KubeRay deleted the entire Ray cluster, head included, within a second, and the same Workload was readmitted 0.2 seconds later at one worker, then regrew. Second set, three runs: an elastic eval grew to six workers, shrank itself to two, and then a q0 job that needs the whole eight-GPU node arrived. Kueue evicted the eval's whole Workload and readmitted it when the q0 job finished. Not measured: the sizes drawn here (the dev node has eight GPUs in all, the figure draws a four-GPU gap). Run records.

RayJob: the elastic eval this walkthrough preempts(click to expand)

The group that gives way is Job B, an elastic eval at q2 whose workers hold one GPU each, the shape of this manifest: it grows and shrinks one worker at a time, between zero and six, and any q1 or q0 arrival may evict it. The arrival in the diagram is Job C, a q0job that needs a whole eight-GPU node; it is not shown as a manifest, and the diagram's sizes are illustrative.

Named job-b here to match the diagram; the dev run used tetris-elastic.

# Elastic eval, native Kueue path (how Fleet runs offline evals): ONE Workload whose size changes
# through Workload slices, one 1-GPU worker at a time, between 0 and 6. Lines marked [shape] define it.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-b
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: elastic
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: elastic}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: job-b-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: eval
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: elastic}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Requeueing and recovery

Our tested configuration allows 30 minutes for startup readiness and five minutes to recover after losing readiness. A readiness-timeout eviction uses a requeue backoff starting at 60 seconds and capped at 600 seconds.^{[12]12Fleet Kueue readiness configuration example} Priority preemption does not wait for that readiness-recovery window; readmission still depends on capacity and policy.^{[10, 13]10Kueue Ray worker-group adapter13Fleet Kueue preemption and readiness recovery}

Recovery follows admission scope. Native job admission can suspend and recreate the Ray cluster.^{[1]1Kueue: Run a RayJob} Worker-group admission recovers the affected group. The extension does not retry arbitrary terminal application failures or restore checkpoints. Those remain application responsibilities.^{[10, 13]10Kueue Ray worker-group adapter13Fleet Kueue preemption and readiness recovery}

Three shapes for one GPU workgroup

A GPU workgroup can contain many workers across several nodes. The CPU coordination workgroup is additional. We classify GPU shapes by whether they can resize and what must remain together.

Fixed gang

A fixed gang needs its complete allocation. If a job requires four nodes, two are insufficient. It is admitted together and, when preempted, releases the group together. Whole-node or topology requirements describe its placement; a fixed execution size is what makes it a fixed gang.^{[6, 10]6Kueue: Topology-Aware Scheduling10Kueue Ray worker-group adapter}

Figure 4A fixed gang leaves and returns togetherOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

RunningRunning. Both eight-GPU pods are required.

Job A · training · 16/16

Running. Both eight-GPU pods are required.
Preempted. The whole GPU allocation is released and the job waits to be readmitted.
Readmitted. Restore a checkpoint or restart work at the same size.

The entire GPU workgroup is the admission and reclamation unit. Recovery waits for the same required size.

RayJob: the fixed gang this diagram draws(click to expand)

A fixed gang as it ran on dev: two workers of two GPUs on one node, the head on CPU quota, one Workload that is admitted whole and released whole. The lines marked [shape] define the shape. The diagram draws the same shape with two eight-GPU pods; the sizes differ, the all-or-nothing admission does not.

Named job-a here to match the diagram; the dev run used tetris-gang.

# Fixed gang, native Kueue path: ONE Workload (PodSets head + gang) admitted all-or-nothing.
# 2 workers x 2 GPUs = 4 GPUs, both pods on one node. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-a
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: fixed-gang
    kueue.x-k8s.io/queue-name: training-lq     # Kueue manages the job (there is no default queue)
    kueue.x-k8s.io/priority-class: q1          # [shape] q: one queue priority for the whole job, head included
spec:
  suspend: true                     # created suspended; Kueue unsuspends it once the whole gang fits
  shutdownAfterJobFinishes: true    # required by Kueue
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 1800       # per admission attempt
  backoffLimit: 0
  submissionMode: HTTPMode          # no submitter pod: the driver runs on the head
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"            # [shape] no enableInTreeAutoscaling: the size never changes
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}          # no tasks on the head
      template:
        metadata:
          labels: {blog.fleet.ai/example: fixed-gang}
        spec:
          priorityClassName: c2                # c2 never preempts: the head shares nodes with the control plane
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head (no topology annotation)
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}   # app.py + out-dir from the ConfigMap
                - {name: sfs, mountPath: /mnt/sfs}         # checkpoints outlive the RayCluster
          volumes:
            - {name: app, configMap: {name: job-a-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: gang
        replicas: 2                   # [shape] fixed: replicas == minReplicas == maxReplicas
        minReplicas: 2
        maxReplicas: 2
        rayStartParams: {num-gpus: "2"}
        template:
          metadata:
            labels: {blog.fleet.ai/example: fixed-gang}
            annotations:
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname   # [shape] both pods on ONE node
          spec:
            priorityClassName: c1     # [shape] c: kube-scheduler priority of the worker pods
            nodeSelector: {workload: fleetai-training-ng-gpu}                   # -> flavor b300-training
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}

Elastic workers

An elastic group changes its replica count within explicit minimum and maximum bounds. Ray autoscaling drives demand.^{[8]8Ray: KubeRay Autoscaling} Growth creates a cumulative Kueue slice requesting the full new count, with the existing allocation transferred when its successor is admitted. Workers already admitted can continue while growth waits. Increments need not be one worker.^{[2, 14]2Kueue: Elastic Workloads14Adapter elastic controller}

Ordinary scale-down removes workers and then releases their quota. Kueue preemption is different: the adapter drains the selected workgroup. It does not automatically shrink that group first.^{[14]14Adapter elastic controller}

Figure 5Grow, shrink, or preempt a workgroupMeasured on dev · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

Two workers are admittedTwo workers are admitted. Each worker uses one GPU. Another job holds the other two GPUs in this four-GPU example.

2/2 running2 GPUs held0 free in slice

Ordinary scale-down removes selected workers. Kueue preemption drains the whole selected workgroup. Striped squares are draining pods; hollow squares show unallocated capacity within each group, not extra physical GPUs.

Four GPUs total. Change demand, release capacity, or try workgroup preemption.

Back restores the previous illustration. Reset returns to two admitted workers. Implementation reference.

Measured on a one-node dev cluster on October 16 and 20, 2026: an elastic eval grew from one to six 1-GPU workers in five resize steps. Each step was one new Workload slice that replaced the last, admitted within 0.1 seconds of the autoscaler's request. Whole-workgroup preemption was observed in a separate run. On October 20 scale-down was run: with the application asking for one task at a time, idle workers left after the 30-second idle timeout, taking the eval from six workers to two within about a minute. How long a pod spends draining was timed from pod events in the October 22 runs: nine workers removed by scale-down alone were gone in a median of 1.2 seconds (eight within 3 seconds, the slowest in 17), workers deleted with their cluster took 0.8 to 12 seconds, and a head pod took about 30 seconds, its grace period. Not measured: the four-GPU sizes drawn here. Run records.

RayJob: the elastic workgroup this diagram grows and shrinks(click to expand)

One Workload whose size changes through Workload slices, one 1-GPU worker at a time, between zero and six. Growth and shrinking are the workers the diagram adds and removes (its four-GPU example is smaller than this manifest's six-worker limit); Kueue preemption drains the whole group, as the diagram shows.

Named job-a here to match the diagram; the dev run used tetris-elastic.

# Elastic eval, native Kueue path (how Fleet runs offline evals): ONE Workload whose size changes
# through Workload slices, one 1-GPU worker at a time, between 0 and 6. Lines marked [shape] define it.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-a
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: elastic
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: elastic}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: job-a-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: eval
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: elastic}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Shaped elastic units

Some applications need several GPUs per execution unit. A replica in a four-GPU pod naturally scales in four-GPU increments. A multi-pod unit needs additional coordination to stay useful when its workers change.

Ray placement groups reserve application resource bundles. Pending bundles can trigger autoscaling.^{[7, 8]7Ray: Placement Groups8Ray: KubeRay Autoscaling} The adapter admits worker-group PodSets; it does not translate placement groups into atomic multi-pod admission or reclamation units. Kubernetes placement still needs its own constraints.^{[15]15Our RayJob template: elasticity and placement-group contract}

Our current adapter supports unconstrained topology-aware scheduling for elastic groups. Fixed groups can request required or preferred topology. Constrained multi-node elastic units remain a design and validation gap.^{[15]15Our RayJob template: elasticity and placement-group contract}

Figure 6A four-GPU replica needs a complete four-GPU slotOpen in sandbox #

Queue · Kueue

q0q1q2

Job A · model poolq2
2 pods × 4 GPUs

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

One replicaOne replica. Only three GPUs are free on the suitable node. The next four-GPU pod waits.

Job A · model pool · 4/8

One replica. Only three GPUs are free on the suitable node. The next four-GPU pod waits.
Two replicas. The other job releases one GPU, giving the second pod four contiguous slots on the node.

Here each unit is one pod. Grouping multiple pods into an atomic elastic unit requires additional support.

RayJob: the 4-GPU units this diagram places(click to expand)

The unit of growth is one 4-GPU pod, so the job holds four or eight GPUs, never five, six or seven. Each unit is reserved by a placement group, which is why a four-GPU replica needs a complete four-GPU slot, as the diagram draws it.

Named job-a here to match the diagram; the dev run used tetris-shaped.

# Shaped elastic, native Kueue path: like ../elastic, but the unit of growth is one 4-GPU pod,
# so the job holds 4 or 8 GPUs, never 5, 6 or 7. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-a
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: shaped-elastic
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] q2
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] size changes are Workload slices
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: shaped-elastic}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: job-a-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: unit
        replicas: 1                   # [shape] start with one unit (a 0-worker start gets no TAS placement)
        minReplicas: 0
        maxReplicas: 2                # [shape] at most 2 units = 8 GPUs
        rayStartParams: {num-gpus: "4"}   # [shape] one unit = one pod = 4 GPUs (numOfHosts stays 1)
        template:
          metadata:
            labels: {blog.fleet.ai/example: shaped-elastic}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] elastic: unconstrained only
          spec:
            priorityClassName: c2     # [shape] c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
                  limits: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}

Combining GPU workgroups

One RayJob can combine these shapes. Separate Workloads let a flexible group give way while another keeps its pods. The application must tolerate that imbalance: a trainer waiting for generated data may retain its allocation without making progress. The adapter does not infer dependencies between groups or checkpoint their actors.^{[10]10Kueue Ray worker-group adapter}

Figure 7What does a four-GPU arrival reclaim?Measured on dev · small RL run · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

Job configuration

Admission scope

State

All 12 GPUs are allocatedAll 12 GPUs are allocated. The head, trainer, and rollouts share one q2 Workload. Native admission gives this whole RayJob one queue priority.

CPU head running · q212/12 GPUs allocated

Job A · trainerJob A · rolloutsJob B · incoming

A stepped comparison of admission scope and priority. Native mode uses one q2 Workload. Adapter mode uses q1 for the trainer, q2 for rollouts, and q0 for the CPU head.

All 12 GPUs are in use: eight for training and four for rollouts. Introduce a four-GPU job at q1.

CPU capacity and valid pod placement are assumed. The original request stays at 12 GPUs. Native mode gives the whole job one queue priority; the adapter gives each group its own. Retaining trainer pods does not guarantee progress. See the adapter contract and lifecycle validation.

RayJob: the same workgroups admitted separately, through the adapter(click to expand)

The diagram opens in native admission: the head, trainer and rollouts share one q2Workload, so an arrival tears the whole RayJob down. This manifest is the adapter version the diagram's second mode draws: one RayJob, three Kueue Workloads (a CPU head at q0, a fixed trainer at q1, an elastic rollout group at q2), so a q1arrival can reclaim only the rollouts. The diagram's sizes are illustrative; this trainer is one worker of four GPUs, with up to four 1-GPU rollout workers. The application must tolerate the imbalance.

Named job-a here to match the diagram; the dev run used tetris-rl.

# Decoupled RL through the Fleet kueue-ray-adapter: one RayJob, THREE Kueue Workloads
# (head, trainer, rollout), each with its own priority. Trainer: fixed 1 x 4 GPUs at q1.
# Rollouts: elastic 0..4 x 1 GPU at q2. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-a
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: decoupled-rl
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: decoupled-rl}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head is its own Workload; q0 so no group outranks it
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: job-a-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trainer
        replicas: 1                   # [shape] fixed: 1 x 4 GPUs (replicas == min == max)
        minReplicas: 1
        maxReplicas: 1
        rayStartParams:
          num-gpus: "4"
          resources: '"{\"trainer\": 1}"'    # custom Ray resource: trainer ranks only fit here
        template:
          metadata:
            labels: {blog.fleet.ai/example: decoupled-rl}
            annotations:
              ray/kueue-priority-class: q1                                     # [shape] trainer Workload: q1
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname  # fixed groups may require a domain
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
                  limits: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
      - groupName: rollout
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 4                # [shape] ... and grows to 4 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"rollout\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: decoupled-rl}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] rollout Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Follow one eval

We keep online evaluation small so it can guide training without delaying every step. Larger checkpoint evaluations run as separate, finite jobs. A CPU head tracks the remaining items. GPU workers load the checkpoint with vLLM and process them. The measured jobs used one GPU per worker.

Admit: one pool, two priorities

Consider an illustrative full cluster: training holds 172 GPUs and an eval holds 20. A training job requests eight more. Under our queue policy, its q1 request can preempt the q2 eval. Another q2 eval would wait. Reclamation follows the workload boundary; it need not release exactly eight GPUs.

Place: topology-aware eval workers

Eval and training workers compete for the same physical GPUs. Our current configuration accounts for them in the same topology-aware resource flavor. A flavor identifies eligible nodes; topology-aware scheduling assigns pods to capacity on those nodes. Eval workers can use any suitable node, while training can impose stricter placement constraints.^{[6]6Kueue: Topology-Aware Scheduling}

Grow: elastic by admission

Pending work creates Ray resource demand. The autoscaler requests more workers, and Kueue admits the larger allocation through a replacement workload slice. Existing workers can keep running while growth waits. When demand falls, Ray removes idle workers and their quota is released.^{[2, 8]2Kueue: Elastic Workloads8Ray: KubeRay Autoscaling}

In our tested native topology-aware configuration, each GPU group starts with one replica. That gives the initial slice a GPU topology assignment. This bootstrap setting is separate from the minimum replica count used during execution. It is a constraint of this configuration, not a general rule for all Ray jobs. Every job also needs an explicit LocalQueue label. In our configuration there is no default LocalQueue and unlabelled jobs are not managed by Kueue, so they would run outside its quota.

Lose capacity: the design and one job

Native whole-job preemption can replace the RayCluster, including its CPU head. A worker readiness failure follows a different path: our configuration allows five minutes for recovery before eviction. Neither path restores application state automatically.^{[1, 12]1Kueue: Run a RayJob12Fleet Kueue readiness configuration example}

Resume: requeue logic

Readmission waits for quota and suitable placement. In three traced evals, the wait after recorded preemption ranged from under a minute to nearly three hours. These are observed waits, not a recovery guarantee. When the entrypoint restarts, it reads the ledger and retries work without a committed result.

RayJob: an elastic eval with a per-job readiness-recovery window(click to expand)

The elastic eval's shape with fleet.ai/readiness-recovery-timeouton the head and worker templates. The value must be the same on every pod template; otherwise Kueue falls back to the cluster default. The scenario script stalls one worker's readiness agent with SIGSTOP, so the pod goes unready while its tasks keep running.

# Per-job readiness-recovery window, native Kueue path: the elastic eval's shape (ONE q2 Workload resized through
# slices, 0..6 x 1 GPU) with fleet.ai/readiness-recovery-timeout on every pod template. When an admitted Workload
# loses readiness (a worker pod goes unready), Kueue waits this long for it to recover before it evicts the Workload
# and requeues it. The scenario stalls one worker's readiness agent and records what each window costs.
# Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-recov
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: recovery-timeout
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: recovery-timeout}
          annotations:
            fleet.ai/readiness-recovery-timeout: "5m"     # [shape] per-job recovery window: how long an unready allocation waits before it is requeued
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-recov-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: eval
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: recovery-timeout}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
              fleet.ai/readiness-recovery-timeout: "5m"              # [shape] the same value on every PodSet, or Kueue uses the cluster default
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Measured on the dev cluster, one run per case.We stalled one worker's readiness agent in an elastic eval (240 ids, up to six GPUs) under a 15-second and a five-minute recovery window. Kueue saw the readiness loss about a minute after the stall, the probe period. At 15 seconds it evicted the whole Workload 12 seconds later, or at once when the stall was never lifted; the job was running again about 100 seconds after the eviction, every id was committed once, and the job finished about eleven minutes after the stall. At five minutes nothing was evicted: the Ray autoscaler removed the unready worker about ten seconds after Kueue noticed, the Workload became ready again, and throughput did not drop. For an elastic job the window is a race against the autoscaler. The stalled agent left the worker's tasks running, so this measures the recovery policy, not the loss of in-flight work. A fifth run gave the head template a 45-second window and the worker template 15 seconds: nothing was evicted, so the 15-second window was not applied. The autoscaler heals the worker before either 45 seconds or the five-minute default, so the run cannot say which of the two Kueue fell back to.

What survives: the ledger

The ledger lives outside the RayCluster. Each result has a unique key for job, task, and repetition. A committed result is counted once. That does not prove exactly-once execution: a worker may finish an item and disappear before committing it, so the computation may run again.

We count graded, ungraded, and failed rollouts in the denominator. Over the measured week, those shares were 98.8%, 0.5%, and 0.7%. Failed work cannot improve the reported score by disappearing from it.

Use cases

A use case can fit several shapes. The choice depends on placement needs, interruption cost, and the recovery behavior the application supports. The expandable manifests come from our dev cluster. They require its queues, priority classes, resource flavors, storage, and application ConfigMaps; they are examples to adapt, not standalone install instructions.

Reserved training

Training protected from lower-priority reclamation uses the highest job priority. Lower-priority work cannot reclaim it under the configured queue policy.^{[11]11Fleet queue and pod priority configuration} It may use a fixed gang or a shaped elastic design. Reservation is a policy; checkpoints are still needed for failure recovery.

Figure 8Priority protects an existing allocationOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

RunningRunning. Two eight-GPU pods hold the training allocation.

Trainer · q0 · 16/16

Running. Two eight-GPU pods hold the training allocation.
q1 job waits. The q0 trainer keeps its allocation. The incoming q1 job needs capacity elsewhere or later.

A lower-priority arrival cannot reclaim this q0 allocation under the configured queue policy. Failures still require recovery.

RayJob: a fixed gang, reserved by running it at q0(click to expand)

The same fixed gang as it ran on dev at q0: all six steps finished in one admission. A reserved run changes two things from the q1 manifest: the job label kueue.x-k8s.io/priority-class: q0 and the workers' priorityClassName: c0. Nothing else in the shape changes.

Named job-a here to match the diagram; the dev run used tetris-gang.

# Variant: the same fixed gang run at q0 (reserved training). Only the two priority lines differ.
# Fixed gang, native Kueue path: ONE Workload (PodSets head + gang) admitted all-or-nothing.
# 2 workers x 2 GPUs = 4 GPUs, both pods on one node. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: job-a
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: fixed-gang
    kueue.x-k8s.io/queue-name: training-lq     # Kueue manages the job (there is no default queue)
    kueue.x-k8s.io/priority-class: q0          # [shape] q0: no other job can reclaim it under the configured queue policy
spec:
  suspend: true                     # created suspended; Kueue unsuspends it once the whole gang fits
  shutdownAfterJobFinishes: true    # required by Kueue
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 1800       # per admission attempt
  backoffLimit: 0
  submissionMode: HTTPMode          # no submitter pod: the driver runs on the head
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"            # [shape] no enableInTreeAutoscaling: the size never changes
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}          # no tasks on the head
      template:
        metadata:
          labels: {blog.fleet.ai/example: fixed-gang}
        spec:
          priorityClassName: c2                # c2 never preempts: the head shares nodes with the control plane
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head (no topology annotation)
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}   # app.py + out-dir from the ConfigMap
                - {name: sfs, mountPath: /mnt/sfs}         # checkpoints outlive the RayCluster
          volumes:
            - {name: app, configMap: {name: job-a-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: gang
        replicas: 2                   # [shape] fixed: replicas == minReplicas == maxReplicas
        minReplicas: 2
        maxReplicas: 2
        rayStartParams: {num-gpus: "2"}
        template:
          metadata:
            labels: {blog.fleet.ai/example: fixed-gang}
            annotations:
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname   # [shape] both pods on ONE node
          spec:
            priorityClassName: c0     # [shape] c0: kube-scheduler priority of the worker pods
            nodeSelector: {workload: fleetai-training-ng-gpu}                   # -> flavor b300-training
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}

Preemptible training and SFT

Checkpoint restart provides elasticity when training can resume with a different number of complete units. Each execution may still run as a gang. These expensive allocations can sit above opportunistic work. PyTorch Distributed Checkpoint can reshard saved state when loading at a different world size.^{[9]9PyTorch: Getting Started with Distributed Checkpoint} A checkpoint alone does not prove that changing the training size works.

Figure 9Change training size through recoveryMeasured on dev · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

16 GPUs16 GPUs. Two training units run with periodic checkpoints.

Trainer · q1 · 16/16

16 GPUs. Two training units run with periodic checkpoints.
Checkpoint retained. The whole allocation is lost. Previously saved training state remains durable.
8 GPUs. Restore at one supported training unit. Changing size requires application support.

Each execution is a gang. Resuming at a smaller size requires checkpoint recovery that supports a different number of training units.

RayJob: a checkpointing fixed gang at q1(click to expand)

Two workers of two GPUs on one node (required hostname topology), the head on CPU quota, one Workload admitted whole. The driver writes a checkpoint every step and resumes from it on restart. On dev it ran to the end once, and once a q0 job that needed a whole node preempted it after step 3: the Workload was readmitted 95 seconds later on a new Ray cluster, resumed from the step-3 checkpoint, and finished the remaining three steps.

# Fixed gang, native Kueue path: ONE Workload (PodSets head + gang) admitted all-or-nothing.
# 2 workers x 2 GPUs = 4 GPUs, both pods on one node. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-gang
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: fixed-gang
    kueue.x-k8s.io/queue-name: training-lq     # Kueue manages the job (there is no default queue)
    kueue.x-k8s.io/priority-class: q1          # [shape] q: one queue priority for the whole job, head included
spec:
  suspend: true                     # created suspended; Kueue unsuspends it once the whole gang fits
  shutdownAfterJobFinishes: true    # required by Kueue
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 1800       # per admission attempt
  backoffLimit: 0
  submissionMode: HTTPMode          # no submitter pod: the driver runs on the head
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"            # [shape] no enableInTreeAutoscaling: the size never changes
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}          # no tasks on the head
      template:
        metadata:
          labels: {blog.fleet.ai/example: fixed-gang}
        spec:
          priorityClassName: c2                # c2 never preempts: the head shares nodes with the control plane
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head (no topology annotation)
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}   # app.py + out-dir from the ConfigMap
                - {name: sfs, mountPath: /mnt/sfs}         # checkpoints outlive the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-gang-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: gang
        replicas: 2                   # [shape] fixed: replicas == minReplicas == maxReplicas
        minReplicas: 2
        maxReplicas: 2
        rayStartParams: {num-gpus: "2"}
        template:
          metadata:
            labels: {blog.fleet.ai/example: fixed-gang}
            annotations:
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname   # [shape] both pods on ONE node
          spec:
            priorityClassName: c1     # [shape] c: kube-scheduler priority of the worker pods
            nodeSelector: {workload: fleetai-training-ng-gpu}                   # -> flavor b300-training
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "2"}

RayJob: an elastic trainer that checkpoints with torch.distributed.checkpoint(click to expand)

The elastic eval's shape with the shared volume mounted on the workers, because each rank writes its own shard. The driver trains in three-step epochs at whatever world size is ready, loads the newest checkpoint resharded to it, and saves a new one.

# Elastic trainer with distributed checkpoints, native Kueue path: ONE q2 Workload resized through slices,
# 0..6 x 1 GPU. The driver trains in epochs at whatever world size is ready and saves a torch.distributed.
# checkpoint after each, so every restart (and every growth step) loads it resharded to a new rank count.
# Workers mount the shared volume because each rank writes its own shard. Lines marked [shape] define it.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-ckpt
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: checkpoint-resize
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: checkpoint-resize}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-ckpt-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: rank
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: checkpoint-resize}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] ranks write their checkpoint shards here
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}

Measured on dev, October 21, 2026. An elastic trainer of up to six 1-GPU ranks checkpointed with PyTorch Distributed Checkpoint every three steps. The world size grew from one to three as ranks joined; at step 12 a q1 gang evicted the whole pool, and the restarted driver loaded the checkpoint that three ranks had written onto a single rank, then grew back through two, three, four and five to six. Every one of the 19 loads matched the expected values exactly, and the run finished all 60 steps. That run used small tensors on the CPU.^{[17]17Job-shape examples, run records and claim audit, October 16 to 26, 2026}

RayJob: real FSDP2 training with model and optimizer state checkpointed across rank counts(click to expand)

The same shape with real state: a small GPT (about 3.4 million parameters) trains with FSDP2 and AdamW on the ranks' GPUs, in ten-step epochs on a fixed global batch. A checkpoint holds the sharded parameters and the sharded optimizer moments, and a digest of both, summed in full, is compared after every load.

# Real training with distributed checkpoints, native Kueue path: ONE q2 Workload resized through slices, 0..6 x
# 1 GPU. A small GPT trains with FSDP2 and AdamW across the ranks that are ready, and the checkpoint (parameters
# and optimizer state) is reloaded resharded after every resize and after a Kueue eviction. Workers mount the
# shared volume because each rank writes its own shard. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-rckpt
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: real-checkpoint
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: real-checkpoint}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-rckpt-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: rank
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-checkpoint}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] ranks write their checkpoint shards here
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}

Measured on dev, October 22, 2026. On B300 GPUs the world size grew from one to four ranks, then a q1 gang evicted the whole pool at step 100 and the restarted driver loaded the four-rank checkpoint onto a single rank. It then grew back through two, three, four and five to six ranks and finished all 900 steps. All 89 loads matched their digests, and the loss stayed at its noise floor, about 1.0, across every resize. A load took 0.2 seconds on one rank and 5 to 7 seconds when the rank count changed. A larger model is measured next, and a rank group spanning two nodes after that.

Measured on dev, October 22, 2026, at a size where saving takes time. The same shape with a 1.14-billion-parameter model: 13.66 GB per checkpoint, sharded parameters and AdamW moments. The world size grew from one to four ranks, a q1 gang evicted the pool, and the restarted driver loaded the four-rank checkpoint onto one rank, then grew back to six. All 11 loads matched their digests. A save took 70 seconds on one rank, 38 on two and about 20 on six. A load onto more ranks was faster: 17.8 seconds onto two ranks, 9.8 onto six, and 2.7 when six ranks reloaded their own six-rank checkpoint. The four-rank checkpoint loaded onto one rank in 6.5 seconds. The loss sat at the entropy of a uniform guess, about 10.4, so this run measures checkpoints, not learning. One run.

Sweeps and filler jobs

Small trials fit gaps left by larger allocations. Short trials can restart; longer ones may checkpoint. A sweep can use independent gangs or an elastic trial-worker pool. This is backfilling fragmented capacity; it does not necessarily consolidate GPUs for a larger waiting job.

Figure 10Small trials fill scattered gapsOpen in sandbox #

Queue · Kueue

q0q1q2

6 trialsq2
1 GPU per trial

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

Six gapsSix gaps. Two GPUs are free on Node A and four on Node B.

Existing allocationIndependent trialFree GPU

Six gaps. Two GPUs are free on Node A and four on Node B.
Six trials running. Six one-GPU trials fill the gaps. Existing allocations stay in place.

Six independent 1-GPU trials use six free slots. Existing allocations stay in place; this backfills capacity without consolidating it.

RayJob: one sweep trial, a q2 job with a single 1-GPU worker(click to expand)

One worker of one GPU, held for 60 seconds at q2 and c2, so it waits for a gap instead of making one. On dev, six of these were submitted behind a four-GPU q1 gang on an eight-GPU node: four were admitted at once, the other two waited and were admitted when the first four finished, and nothing was preempted.

# One sweep trial: an independent q2 RayJob with a single 1-GPU worker (native path) that holds its GPU for
# 60 seconds. sweep/run.sh stamps out six of them (tetris-trial-1 ... tetris-trial-6) beside a q1 gang.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-trial-1
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: sweep
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: fills gaps, never takes them
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 900
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: |
    python -u - <<'EOF'
    import socket, time, ray
    ray.init()

    @ray.remote(num_gpus=1)
    def hold(seconds):
        time.sleep(seconds)
        return socket.gethostname()

    print("trial held 1 GPU for 60 s on", set(ray.get([hold.remote(60)])))
    EOF
  rayClusterSpec:
    rayVersion: "2.49.2"
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: sweep}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
    workerGroupSpecs:
      - groupName: trial
        replicas: 1                   # [shape] fixed: one 1-GPU pod
        minReplicas: 1
        maxReplicas: 1
        rayStartParams: {num-gpus: "1"}
        template:
          metadata:
            labels: {blog.fleet.ai/example: sweep}
            annotations:
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname   # [shape] 1 GPU
          spec:
            priorityClassName: c2     # [shape] c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

RayJob: an elastic pool of checkpointed trials(click to expand)

The elastic eval's shape used for training trials: a trial is six 15-second steps, each a 1-GPU task, checkpointed after every step. On dev a q1 gang evicted the whole pool when eight trials had finished and four more were under way; the restarted driver resumed those four from their checkpoints, and the pool executed exactly the 180 steps it needed. Run again without checkpoints it executed 182, because these trials are so short that checkpointing saved two steps. The saving grows with the length of a trial.

# Elastic trial pool, native Kueue path: the elastic eval's shape (ONE q2 Workload resized through slices,
# 0..6 x 1 GPU) used as a pool of training trials. Each trial is a chain of short 1-GPU steps; the driver
# writes a checkpoint after every step, so a trial that was cut off by a Kueue preemption resumes from its
# last step instead of step 0. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-pool
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: trial-pool
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: trial-pool}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-pool-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trial
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: trial-pool}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Bounded serving pools

Pools serving custom checkpoints can process finite evaluation or rollout work on spare GPUs. Replicas may use one GPU, a tensor-parallel pod, or a coordinated multi-node group. There is no training state to restore, but interrupted requests need retries and replacement replicas must load weights.

Figure 11Serve a finite batch, then release the poolOpen in sandbox #

Queue · Kueue

q0q1q2

TP4 poolq2
2 pods × 4 GPUs

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

Batch readyBatch ready. Six requests for one custom checkpoint await the pool.

TP4 pool · q2 · 0/8

Batch ready. Six requests for one custom checkpoint await the pool.
Processing. Two four-GPU replicas process the finite batch. Tensor parallelism stays within each pod.
Pool released. All six results are complete. The pool exits and releases eight GPUs.

Each TP4 replica occupies one 4-GPU pod. A disrupted replica must reload weights, and unfinished requests need retries.

RayJob: 4-GPU units with a placement group per unit(click to expand)

Each unit is one 4-GPU pod reserved by a STRICT_PACK placement group, so the group grows and shrinks four GPUs at a time. A serving pool runs its tensor-parallel replica inside each unit; the run used a placeholder task per unit and went 4 → 8 → 4 GPUs.

# Shaped elastic, native Kueue path: like ../elastic, but the unit of growth is one 4-GPU pod,
# so the job holds 4 or 8 GPUs, never 5, 6 or 7. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-shaped
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: shaped-elastic
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] q2
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] size changes are Workload slices
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: shaped-elastic}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-shaped-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: unit
        replicas: 1                   # [shape] start with one unit (a 0-worker start gets no TAS placement)
        minReplicas: 0
        maxReplicas: 2                # [shape] at most 2 units = 8 GPUs
        rayStartParams: {num-gpus: "4"}   # [shape] one unit = one pod = 4 GPUs (numOfHosts stays 1)
        template:
          metadata:
            labels: {blog.fleet.ai/example: shaped-elastic}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] elastic: unconstrained only
          spec:
            priorityClassName: c2     # [shape] c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
                  limits: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}

RayJob: a bounded serving pool behind a retrying router(click to expand)

A q0 head runs the router and a load generator; the replicas are an elastic q2group of up to six 1-GPU workers, each loading "weights" for 20 seconds and then answering a one-second request. On dev a warm, burst and cool load grew the pool to five replicas. A q1 gang then evicted the replica Workload while the head and router stayed up, and the pool was empty for about three minutes. Of 1,935 requests, 1,872 succeeded; 63 waited past the 300-second deadline because the backlog drained only two requests per second faster than new ones arrived. All five replicas reloaded their weights on readmission. Nothing was generated and no real weights were loaded.

# Bounded serving pool through the Fleet kueue-ray-adapter: one RayJob, TWO Kueue Workloads (head, replica
# group). The replicas are elastic 0..6 x 1 GPU at q2; the head runs the router and a load generator and
# stays up when the replica Workload is evicted. Each replica "loads weights" for 20 s before it serves.
# Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-serve
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: serving
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: serving}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head (router and load generator) is its own Workload, q0
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-serve-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: replica
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 6                # [shape] ... and grows to 6 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"replica\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: serving}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] replica Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

RayJob: a serving pool whose replicas load and run a real model(click to expand)

The same pool with the placeholders replaced. Each replica installs torch and transformers, loads Qwen2.5-0.5B-Instruct from the shared volume onto its GPU, and answers each request by generating 32 tokens. The replica pods mount the shared volume; without that mount no replica can fetch the weights and none ever becomes ready.

# Bounded serving pool with a real model, through the Fleet kueue-ray-adapter: one RayJob, TWO Kueue Workloads (head, replica
# group). The replicas are elastic 0..6 x 1 GPU at q2; the head runs the router and a load generator and
# stays up when the replica Workload is evicted. Each replica "loads weights" for 20 s before it serves.
# Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-rserve
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: real-serving
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: real-serving}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head (router and load generator) is its own Workload, q0
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-rserve-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: replica
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 6                # [shape] ... and grows to 6 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"replica\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-serving}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] replica Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] the Hugging Face cache with the weights; every replica loads from it
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}

Measured on dev, October 24, 2026, one run, transformers. Replicas fetched the weights in 0.5 to 6.8 seconds and loaded them in 5.6 to 12.3 seconds; the first was ready about two minutes after the job started, mostly the per-pod torch install. A q1 gang evicted the replica Workload while the head and router stayed up: the pool was empty for 5.5 minutes, then five replicas reloaded their weights (ten loads in all). Of 1,935 requests, 1,374 succeeded and 561 waited past the 300-second deadline; 36 were retried. The median latency of 100 to 200 seconds is the backlog, not the model. With vLLM replicas the pool came back 8 minutes 10 seconds after its eviction: each replica installs about 8 GB of packages and loads in 55 to 72 seconds. On the B300 that needed vLLM's bundled FlashAttention backend, because its default FlashInfer path wants to compile kernels and the image has no compiler.

Offline evaluation

Run minimal online evaluation during training, then evaluate checkpoints extensively in opportunistic jobs. A durable ledger records completed items and supports retries. Small models fit independent workers; larger models may use one eight-GPU replica or two four-GPU replicas, where supported.

Figure 12Keep completed items; retry unfinished workOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

EvaluatingEvaluating. Four items are complete, one is running and one is pending.

Evaluation · q2 · 2/2

Evaluating. Four items are complete, one is running and one is pending.
Retry queued. The interrupted item returns to the retry queue. All four completed results remain.
Resumed. Two workers process the retry and pending item. Assignment is not completion.
Complete. All results are recorded. The finite job releases its workers.

Assignment is not completion: the interrupted evaluation item returns to the retry queue while completed results remain recorded.

RayJob: an elastic ledger eval, one 1-GPU worker at a time(click to expand)

A CPU head owns the work list and a ledger file on a shared volume; 1-GPU workers start at one and scale to six under the Ray autoscaler, each resize one Kueue workload slice. On restart the head skips every id already in the ledger. The file stands in for the Postgres table the measured evals used: it records each id once but has no key enforcing it.

# Elastic eval, native Kueue path (how Fleet runs offline evals): ONE Workload whose size changes
# through Workload slices, one 1-GPU worker at a time, between 0 and 6. Lines marked [shape] define it.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-elastic
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: elastic
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: elastic}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-elastic-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: eval
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: elastic}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

RayJob: an eval whose workers load a real model(click to expand)

The elastic eval's shape with a real model: each of up to six workers installs torch and transformers, loads Qwen2.5-0.5B-Instruct from the shared volume onto its GPU, and answers prompts. The first worker stages the weights on the volume; later loads read them from there.

# A real model-loading eval, native Kueue path: the elastic eval's shape (ONE q2 Workload resized through
# slices, 0..6 x 1 GPU) where each worker loads Qwen2.5-0.5B-Instruct from the shared volume onto its GPU and
# generates answers. Workers mount the shared volume for the weights. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-reval
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: real-eval
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: real-eval}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-reval-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: eval
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-eval}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] ranks write their checkpoint shards here
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}

Measured on dev, October 22, 2026. A q1 gang evicted the pool after 125 of 480 answers. Four prompts that were in flight went back on the list, the restarted driver skipped everything already committed, and all 480 answers ended up in the ledger once. From eviction to the first new answer took 186 seconds: about 92 for readmission and a new head, 58 for a worker to install torch and load the model, and 36 for the first answer, which includes GPU warm-up. The load itself took 1.5 seconds from the shared volume. The torch install is a cost of the stock image, which a prebuilt image would remove. A 7B model is measured next.

Measured on dev, October 23, 2026, with Qwen2.5-7B-Instruct (about 15 GB of weights). A q1 gang evicted the pool after 69 of 240 answers and the restarted driver skipped them; all 240 answers ended up in the ledger once. The first evaluator took 23 seconds to fetch the weights and 10.4 to load them. After the eviction the weights were already on the shared volume: the fetch took 0.2 seconds and the load 5.2. From eviction to the first new answer took 198 seconds: about 101 to readmit and start a new driver, 60 for an evaluator to install torch and load the model, and 37 for the first answer. Loading was not the bottleneck at this size; readmission and the per-pod install were. The weights were deleted from the shared volume afterwards. One run.

Distillation

Student training and teacher inference can have separate GPU workgroups. Teacher capacity can give way first, while buffering or coordination handles reduced throughput. The student may also resize when its recovery path supports it. On dev, with placeholder work, evicting the teacher did not stop the student (see the manifest below).

Figure 13Teacher and student have separate allocationsMeasured on dev · small real run · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

Both runningBoth running. The teacher supplies targets or logits to an application buffer for student training.

Teacher · q2 · 8/8Student · q1 · 8/8

Both running. The teacher supplies targets or logits to an application buffer for student training.
Student retained. The teacher gives way first. The student may need to pause when buffered data runs out.
Teacher returns. Teacher replicas reload weights and resume generating targets.

Example priorities let teacher capacity give way first. An application buffer and coordination handle changes in teacher throughput. Retained student pods do not guarantee training progress.

RayJob: student fixed at q1, teacher elastic at q2(click to expand)

The decoupled-RL shape with the roles renamed: a fixed student, an elastic teacher, and a buffer of 48 teacher labels between them. On dev a q1gang evicted only the teacher Workload, which was back about three minutes later. The student kept stepping every ten seconds throughout: twelve steps on buffered labels, then six steps that replayed labels it had already used because the buffer had run dry. All 90 steps finished: 82 on fresh labels, 6 on replayed ones, and the first 2 before the teacher had produced anything. Whether replayed labels are acceptable is the application's decision.

# Distillation through the Fleet kueue-ray-adapter: one RayJob, THREE Kueue Workloads (head, student,
# teacher), each with its own priority. Student: fixed 1 x 4 GPUs at q1. Teacher: elastic 0..4 x 1 GPU at
# q2, producing labels into a buffer the student consumes. The same shape as ../decoupled-rl with the roles
# renamed; the app is what differs. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-distill
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: distillation
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: distillation}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head is its own Workload; q0 so no group outranks it
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-distill-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: student
        replicas: 1                   # [shape] fixed: 1 x 4 GPUs (replicas == min == max)
        minReplicas: 1
        maxReplicas: 1
        rayStartParams:
          num-gpus: "4"
          resources: '"{\"student\": 1}"'    # custom Ray resource: the student only fits here
        template:
          metadata:
            labels: {blog.fleet.ai/example: distillation}
            annotations:
              ray/kueue-priority-class: q1                                     # [shape] student Workload: q1
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname  # fixed groups may require a domain
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
                  limits: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
      - groupName: teacher
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 4                # [shape] ... and grows to 4 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"teacher\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: distillation}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] teacher Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

RayJob: distillation with real models, teachers elastic at q2, student fixed at q1(click to expand)

The same shape with real models. Up to four teacher workers hold Qwen2.5-3B-Instruct and answer multiplication questions greedily; a fixed one-GPU student, Qwen2.5-0.5B-Instruct, takes an AdamW step on each batch of 32 teacher answers, right or wrong. It is sequence-level distillation: cross-entropy on the teacher's answer tokens, not logit matching.

# Real distillation through the Fleet kueue-ray-adapter: one RayJob, THREE Kueue Workloads (head q0, student q1, teachers q2).
# Qwen2.5-3B-Instruct teacher workers label "What is A times B?" on elastic GPUs (0..4 x 1 GPU); a fixed 1-GPU student
# (Qwen2.5-0.5B-Instruct) trains on every label. Same shape as ../real-rl with the roles renamed: the trainer group is the student,
# the rollout group the teacher. Lines marked [shape] differ from ../decoupled-rl.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-rdist
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: real-distillation
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: real-distillation}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head is its own Workload; q0 so no group outranks it
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-rdist-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trainer
        replicas: 1                   # [shape] fixed: one 1-GPU worker (replicas == min == max)
        minReplicas: 1
        maxReplicas: 1
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"trainer\": 1}"'    # custom Ray resource: only the trainer fits here
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-distillation}
            annotations:
              ray/kueue-priority-class: q1                                     # [shape] trainer Workload: q1
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname  # fixed groups may require a domain
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] the trainer publishes policies here; rollout workers load them
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
      - groupName: rollout
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 4                # [shape] ... and grows to 4 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"rollout\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-distillation}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] rollout Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] the trainer publishes policies here; rollout workers load them
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}

Measured on dev, October 24, 2026, one run. Held-out greedy accuracy on 160 fixed questions went from 0.113 for the student to 0.219 after 1,200 steps on 69,216 teacher labels, against 0.406 for the teacher; 37% of the labels were correct. A q1 gang evicted the teacher Workload 4 seconds after it arrived. The student already had no labels buffered, because it consumes them faster than four teachers write them, so it stood still until the teachers returned 4 minutes 37 seconds later; reloading the 3B weights then took about 3 seconds from the shared volume, against 21 the first time, and the student finished. A shorter 200-step run reached 0.194. The task is narrow and the run is single.

Decoupled RL

Trainer and rollout workgroups have different interruption costs. Separate admissions allow rollout capacity to be reclaimed first. Rollouts track unfinished work in a ledger; the trainer uses checkpoints. Keeping its pods alive is valuable only if the application handles fewer rollouts.

Figure 14Reclaim rollout capacity while retaining the trainerMeasured on dev · small RL run · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

Both runningBoth running. The trainer and rollout workers have separate GPU Workloads.

Trainer · q1 · 8/8Rollouts · q2 · 4/4

Both running. The trainer and rollout workers have separate GPU Workloads.
Trainer retained. All rollout pods leave. The ledger retains unfinished work for retry. Trainer progress depends on buffered rollout data.
Rollouts return. Replacement workers retry unfinished rollout work.

Kueue preempts the entire rollout Workload in this example. The adapter preserves the trainer allocation; continued training depends on available rollout data.

RayJob: head q0, trainer fixed at q1, rollouts elastic at q2, through the adapter(click to expand)

The annotation ray/kueue-admission-scope: worker-group opts the job into the adapter, and each template carries its own ray/kueue-priority-class. On dev a q1 arrival preempted only the rollout Workload; head and trainer kept their pods. In four more runs the application first shrank the rollouts from eight workers to two; the arrival then evicted the rollout Workload at that size and it was readmitted when the arrival finished.

# Decoupled RL through the Fleet kueue-ray-adapter: one RayJob, THREE Kueue Workloads
# (head, trainer, rollout), each with its own priority. Trainer: fixed 1 x 4 GPUs at q1.
# Rollouts: elastic 0..4 x 1 GPU at q2. Lines marked [shape] define the shape.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-rl
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: decoupled-rl
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: decoupled-rl}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head is its own Workload; q0 so no group outranks it
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-rl-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trainer
        replicas: 1                   # [shape] fixed: 1 x 4 GPUs (replicas == min == max)
        minReplicas: 1
        maxReplicas: 1
        rayStartParams:
          num-gpus: "4"
          resources: '"{\"trainer\": 1}"'    # custom Ray resource: trainer ranks only fit here
        template:
          metadata:
            labels: {blog.fleet.ai/example: decoupled-rl}
            annotations:
              ray/kueue-priority-class: q1                                     # [shape] trainer Workload: q1
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname  # fixed groups may require a domain
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
                  limits: {cpu: "16", memory: 128Gi, nvidia.com/gpu: "4"}
      - groupName: rollout
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 4                # [shape] ... and grows to 4 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"rollout\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: decoupled-rl}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] rollout Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Mechanism validated on GPUs, October 16, 2026. On the dev cluster a fixed trainer at q1 kept its pods while a q1 arrival preempted, and later restored, its q2 rollouts (see the validation section).^{[17]17Job-shape examples, run records and claim audit, October 16 to 26, 2026} Trainer and rollouts were placeholders in that run; a small real RL loop follows.

RayJob: a small real RL loop, head q0, trainer q1, rollouts q2(click to expand)

Qwen2.5-0.5B-Instruct learns to multiply a three-digit number by a two-digit one by expert iteration: rollout workers sample four answers per question from the newest policy on the shared volume, the driver scores them, and a one-GPU trainer takes an AdamW step on the newest 1,024 correct answers, publishing weights every four steps. After 40 steps a q1 gang arrives and Kueue evicts only the rollouts. This is rejection-sampling fine-tuning, not a policy-gradient method, on one narrow task.

# A small REAL decoupled RL loop through the Fleet kueue-ray-adapter: one RayJob, THREE Kueue Workloads (head q0, trainer q1,
# rollouts q2). Qwen2.5-0.5B-Instruct learns "What is A times B?" by expert iteration: rollout workers sample answers on
# elastic GPUs (0..4 x 1 GPU), a fixed 1-GPU trainer fine-tunes on the correct ones and publishes the weights to the
# shared volume, and rollout workers reload them. Lines marked [shape] differ from ../decoupled-rl.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-rrl
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: real-rl
    kueue.x-k8s.io/queue-name: training-lq     # the ONLY kueue.x-k8s.io/* key allowed on an adapter job
  annotations:
    ray/kueue-admission-scope: worker-group    # [shape] opt in: one Workload per group; set at create, immutable
    # No kueue.x-k8s.io/priority-class label and no kueue.x-k8s.io/elastic-job annotation: the adapter
    # rejects both. Priority is per pod template below; the adapter manages elastic slices itself.
spec:
  # No suspend: the adapter gates pods instead of suspending the job (suspend: true is rejected).
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode                     # a K8sJobMode submitter would run outside Kueue
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # needed once any group has minReplicas != maxReplicas
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: real-rl}
          annotations:
            ray/kueue-priority-class: q0       # [shape] the head is its own Workload; q0 so no group outranks it
        spec:
          priorityClassName: c2                # c2 never preempts on the shared CPU nodes
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-rrl-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trainer
        replicas: 1                   # [shape] fixed: one 1-GPU worker (replicas == min == max)
        minReplicas: 1
        maxReplicas: 1
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"trainer\": 1}"'    # custom Ray resource: only the trainer fits here
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-rl}
            annotations:
              ray/kueue-priority-class: q1                                     # [shape] trainer Workload: q1
              kueue.x-k8s.io/podset-required-topology: kubernetes.io/hostname  # fixed groups may require a domain
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "8", memory: 64Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] the trainer publishes policies here; rollout workers load them
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
      - groupName: rollout
        replicas: 0                   # [shape] elastic: starts at 0 (allowed on the adapter path) ...
        minReplicas: 0
        maxReplicas: 4                # [shape] ... and grows to 4 x 1 GPU
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"rollout\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: real-rl}
            annotations:
              ray/kueue-priority-class: q2                             # [shape] rollout Workload: q2, evicted first
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic groups: unconstrained only
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                volumeMounts:
                  - {name: sfs, mountPath: /mnt/sfs}        # [shape] the trainer publishes policies here; rollout workers load them
            volumes:
              - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}

Measured on dev, October 23, 2026, one run. Over 300 optimizer steps the held-out greedy accuracy (160 fixed questions) rose from 0.125 to 0.188; sampled accuracy of the published policies rose from 0.09 at step 4 to about 0.2 from step 40 on. The q1 arrival evicted the rollout Workload 95 seconds after it was submitted; the trainer kept its GPU and went on stepping on buffered answers until the buffer ran dry at step 108, then waited. The rollouts were back about 3 minutes 20 seconds after the eviction and reloaded the newest policy in about two seconds. A first attempt with a learning rate ten times higher and no warmup made the policy worse (0.125 to 0.087), so this is a tuned run, not a first try. With 160 questions and one run, the gain is real but small; it shows the loop works through an eviction, not that this recipe is good. The same loop with Qwen2.5-1.5B-Instruct, three times the parameters, ran on October 26: held-out accuracy went from 0.300 to 0.325, which is within the noise of 160 questions, and the sampled accuracy of the published policies rose from 0.26 to about 0.33 by step 40 and stayed there. The q1 arrival evicted the rollouts 10 seconds after it was submitted; the trainer stepped on buffered answers from step 47 to 80, then waited, and the rollouts were back about 4.5 minutes after the eviction. It is a larger model, not a production-scale RL stack.

Fully elastic RL

Both workgroups resize: the trainer through supported checkpoint recovery, and rollouts according to their execution shape. Policy decides which group releases capacity first. The application coordinates the new balance. On dev it has run end to end with placeholder work, including through reclamation of both groups.

Figure 15Coordinate a smaller trainer and rollout poolMeasured on dev · placeholder work · sizes differOpen in sandbox #

Queue · Kueue

q0q1q2

No waiting work

Cluster · 8 nodes × 8 GPUs

q0q1q2Free

Actions

State

12 GPUs12 GPUs. Two four-GPU training units and four rollout workers are running.

Trainer · q1 · 8/8Rollouts · q2 · 4/4

12 GPUs. Two four-GPU training units and four rollout workers are running.
Trainer restarting. The application stops training for checkpoint recovery and reduces rollout demand to two workers.
6 GPUs. Restore one supported training unit. Both groups now use six GPUs in total.

This application policy reduces rollout demand and shrinks the trainer before the arrival, so nothing is evicted. Kueue does not automatically perform this coordinated resize.

RayJob: trainer and rollouts both elastic(click to expand)

Both groups have minReplicas below maxReplicas, so each is its own elastic Workload. On dev the trainer grew from one to four ranks one slice at a time while four rollout workers ran, and the job finished all 60 steps (168 rollouts in the ledger). A second run added two arrivals: a q1 gang evicted only the rollout Workload, and a later q0job that needed the whole node evicted the trainer's four ranks as well. The driver logged the lost ranks, retried the step, and when the node freed up both groups regrew and training resumed from its saved step. It finished all 120 steps with 284 rollouts, none duplicated. That run set STEPS=120 on the head (the scenario script injects it) so the job outlives both arrivals; the manifest above is the 60-step one. The figure above draws four-GPU training units; the runs used one-GPU ranks.

# Variant: fully elastic RL through the adapter. Same as rayjob.yaml except the trainer is elastic
# too: 0..4 one-GPU ranks at q1, added one at a time by the app. Elastic groups may only use
# unconstrained topology, so the trainer gives up "all ranks on one node". Lines marked [shape] differ.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-rl-elastic
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: decoupled-rl
    kueue.x-k8s.io/queue-name: training-lq
  annotations:
    ray/kueue-admission-scope: worker-group
spec:
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: decoupled-rl}
          annotations:
            ray/kueue-priority-class: q0
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              env:                                       # [shape] tell app.py: up to 4 ranks of 1 GPU
                - {name: TRAINER_GPUS, value: "1"}
                - {name: TRAINER_RANKS, value: "4"}
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-rl-elastic-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trainer
        replicas: 0                   # [shape] elastic trainer: 0..4 x 1 GPU
        minReplicas: 0
        maxReplicas: 4
        rayStartParams:
          num-gpus: "1"               # [shape] one rank per 1-GPU pod
          resources: '"{\"trainer\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: decoupled-rl}
            annotations:
              ray/kueue-priority-class: q1
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic: unconstrained only
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
      - groupName: rollout
        replicas: 0
        minReplicas: 0
        maxReplicas: 4
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"rollout\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: decoupled-rl}
            annotations:
              ray/kueue-priority-class: q2
              kueue.x-k8s.io/podset-unconstrained-topology: "true"
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

RayJob: the same job, resized by the application before the arrival(click to expand)

The fully elastic job at 120 steps, plus a control file on the shared volume that the driver reads between steps. Told trainer_ranks=2, rollout_slots=0, it stops its rollouts, kills the extra trainer ranks between steps, and lets the Ray autoscaler release their pods. The manifest also creates the arrival: a q0 job that needs six GPUs.

# Coordinated resize through the kueue-ray-adapter: the fully elastic RL job (trainer 0..4 x 1 GPU at q1, rollouts 0..4 x
# 1 GPU at q2, head q0) with a driver that reads a control file and shrinks itself ahead of a reclamation: it stops
# submitting rollouts and releases trainer ranks, so a q0 job that needs six GPUs fits without Kueue evicting anything.
# The control file stands in for a notification from the scheduler. Lines marked [shape] differ from ../decoupled-rl.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-coord
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: coordinated-resize
    kueue.x-k8s.io/queue-name: training-lq
  annotations:
    ray/kueue-admission-scope: worker-group
spec:
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true
    autoscalerOptions: {idleTimeoutSeconds: 30}
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: coordinated-resize}
          annotations:
            ray/kueue-priority-class: q0
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              env:                                       # [shape] tell app.py: up to 4 ranks of 1 GPU
                - {name: TRAINER_GPUS, value: "1"}
                - {name: TRAINER_RANKS, value: "4"}
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}
          volumes:
            - {name: app, configMap: {name: tetris-coord-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: trainer
        replicas: 0                   # [shape] elastic trainer: 0..4 x 1 GPU
        minReplicas: 0
        maxReplicas: 4
        rayStartParams:
          num-gpus: "1"               # [shape] one rank per 1-GPU pod
          resources: '"{\"trainer\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: coordinated-resize}
            annotations:
              ray/kueue-priority-class: q1
              kueue.x-k8s.io/podset-unconstrained-topology: "true"     # [shape] elastic: unconstrained only
          spec:
            priorityClassName: c1
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
      - groupName: rollout
        replicas: 0
        minReplicas: 0
        maxReplicas: 4
        rayStartParams:
          num-gpus: "1"
          resources: '"{\"rollout\": 1}"'
        template:
          metadata:
            labels: {blog.fleet.ai/example: coordinated-resize}
            annotations:
              ray/kueue-priority-class: q2
              kueue.x-k8s.io/podset-unconstrained-topology: "true"
          spec:
            priorityClassName: c2
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Measured on dev, October 22, 2026, one run per arm, placeholder trainer. The job filled the node (a four-rank trainer, four rollout workers) and a q0 job needing six GPUs arrived. Uncoordinated, Kueue evicted both groups 16 seconds after the arrival, readmitted them about two minutes later, and the trainer lost four ranks and retried its step. Coordinated, the driver was told first: about a minute later the job held two GPUs, the arrival was admitted 8 seconds after it was submitted, and nothing was evicted. Once the arrival finished, both groups regrew within about 90 seconds. The cost moved rather than vanished: with no rollouts the trainer had nothing to train on for those minutes, which a real trainer would also feel.

Capabilities and shapes

Possible configurations. Counts include GPU workgroups only; the CPU workgroup is additional.
Use case	GPU groups	Possible shapes	Job capabilities
Reserved training	1	Fixed gang; shaped elastic	Checkpoint after failure; top-priority policy
Preemptible training / SFT	1	Fixed gang; shaped elastic if restart supports it	Checkpoint at a supported size; resharded across rank counts, with model and optimizer state on GPUs
Sweeps and filler	1 per job	Small gangs; elastic trial pool	Rerun trial or checkpoint
Bounded serving	1	Elastic workers; shaped replicas	Reload weights; retry requests; placeholder run and real 0.5B runs, with transformers and with vLLM
Offline evaluation	1	Elastic workers; shaped replicas	Ledger and retries; recovery measured with 0.5B and 7B models
Distillation	2	Fixed or elastic student; elastic teacher	Separate priorities; checkpoint and inference recovery; placeholder run and a real 3B-to-0.5B run
Decoupled RL	2	Gang trainer; elastic or shaped rollouts	Separate priorities; checkpoint and ledger; small real RL loop measured through an eviction
Fully elastic RL	2	Shaped elastic trainer and elastic rollouts	Independent resizing; coordinated recovery; run through reclamation of both groups; coordinated resize measured

What we have validated

On October 16, 2026, small jobs on one eight-GPU B300 node exercised admission, resizing, and preemption. They reserved real GPUs but ran placeholder work. They did not measure model training throughput.

Fixed gang: two two-GPU workers completed in one admission.
Elastic workers: one worker grew to six; existing workers retained their pods.
Four-GPU units: allocation grew from four to eight GPUs, then shrank to four.
Native preemption: the RayCluster was replaced, and the restarted job recovered committed ledger results.
Adapter preemption: rollout workers were reclaimed while the CPU head and fixed trainer retained their pods.
October 20 follow-up: an eval that shrank itself to two workers lost its whole Workload to a q0 job needing the full node, then was readmitted when that job finished (3 of 3 runs). A q0 fixed gang ran, and a q1 gang preempted by q0 resumed from its checkpoint. Six q2 trials queued behind a q1 gang. Fully elastic trainer and rollouts ran 60 steps. Rollouts shrunk through the adapter were then preempted and restored (4 of 4 runs).
October 21 follow-up: an elastic trainer checkpointed with PyTorch Distributed Checkpoint and loaded it resharded across 1 to 6 ranks, including straight after a whole-pool eviction (19 of 19 loads exact). A fixed student kept stepping while its elastic teacher was evicted for about three minutes. The fully elastic RL job survived a q1 and a q0 arrival and finished 120 steps. A pool of checkpointed trials resumed after an eviction with no repeated steps. A serving pool kept its router up while its replicas were evicted for about three minutes, then reloaded weights and recovered.
October 22 follow-up: a small GPT trained with FSDP2 and AdamW had its model and optimizer state checkpointed and reloaded across 1 to 6 ranks on GPUs, including four ranks onto one after a whole-pool eviction (89 of 89 loads matched). An eval whose workers load a real 0.5B model recovered from an eviction in 186 seconds and finished all 480 answers once.
October 22 to 23 follow-up: per-job readiness-recovery windows of 15 seconds and five minutes were run with a stalled worker agent (four runs); a fully elastic RL job told ahead of a q0 arrival shrank without any eviction, against an uncoordinated run that lost both groups; a 1.14-billion-parameter model checkpointed and resharded across 1 to 6 ranks (11 of 11 loads matched); a 7B eval recovered from an eviction in 198 seconds with all 240 answers once; a small real RL loop improved held-out accuracy from 0.125 to 0.188 through an eviction of its rollouts; and on two GPU nodes, a one-node eight-GPU request waited while the same eight GPUs as two 4-GPU pods were admitted across both nodes.
October 24 follow-up: eight GPU ranks in two pods on two nodes formed one NCCL group and agreed on an all-reduce; a real 3B-to-0.5B distillation raised the student from 0.113 to 0.219 held-out accuracy through an eviction of its teachers (back after 4 min 37 s); and a serving pool whose replicas load a real 0.5B model was evicted, sat empty for 5.5 minutes, and reloaded its weights.
October 25 to 26 follow-up: an FSDP2 trainer ran across two nodes (four ranks on one, two on the other) and resharded its checkpoint 14 times, all matching; a 16-GPU gang of two whole-node pods formed one NCCL group through a temporary 16-GPU queue (the real 8-GPU quota kept it pending for 71 minutes); and the small RL loop with a 1.5B model ran through the same eviction.

In the single dev preemption run, the restarted eval made its first new ledger commit 97 seconds after preemption. There were 99 seconds between the last commit before preemption and the first one after it. Images were cached, and the tasks were placeholders. A later run with a real 0.5-billion-parameter model is in the evaluation section.^{[17]17Job-shape examples, run records and claim audit, October 16 to 26, 2026}

Single-node tests cannot establish multi-node placement behavior; the two-node tests above cover placement, NCCL collectives up to 16 ranks, and a small FSDP2 model trained and resharded across two nodes. The tests do not establish a gang over more than two nodes, atomic multi-pod elasticity, or an RL run at production scale. The examples above show the tested manifests and label variants that have not run in that role.^{[18]18Adapter elastic reconciliation tests}

Measuring the packing result

Near-full allocation remains a target. Record both allocation and completed work so that retries and recovery cannot hide behind a busy GPU graph.

Measured: one week of training with elastic evals. A cluster of 192 B300 GPUs; Kueue 0.19.4, KubeRay 1.4.2, Ray 2.49.2; seven UTC days, September 10–16, 2026. Training ran at q1 and evals at q2 on single-GPU workers. The cluster was never full: the week peaked at 183 of 192 GPUs allocated. Method and limitations are below.

Ten-minute Prometheus samples of GPU limits on running pods, averaged over the seven measured days. Both columns come from the same week: the first counts training and other non-eval pods; the second adds eval pods. These are not before-and-after runs.
Measure	Training + other jobs	All jobs, including evals
Mean GPUs allocated, of 192	134 (70%)	154 (80%)
Share of samples above 90% allocated	0.2%	29.3%
GPU-hours unallocated	9,755	6,311
Eval GPU-hours	0	3,444

This is not a controlled comparison of two policies under the same demand. It is one week in which evals filled 35% of the capacity training left free, with the eval work itself reported in the offline-evaluation section above. When training and other jobs held at least 170 GPUs, evals averaged 2.2 GPUs. That is a binned average, not a per-job response guarantee.

Figure 16Seven days of GPU allocation, evals stacked on training#

Training and other jobs
Elastic evals
Unallocated

No sample selected.

September 2026, UTC day boundaries, 1,007 ten-minute samples of GPU limits on running pods. Hover or focus the chart for a sample's values.

View the daily numbers and the full binning

Per UTC day: mean GPUs by series, the eval peak, and the rollouts the ledger records as finished that day.
Day	Eval GPUs, mean	Eval GPUs, max	Training and other, mean	Unallocated, mean	Rollouts finished
Sep 10	12.4	35	151.5	28.1	5,577
Sep 11	23.8	56	126.6	41.6	17,376
Sep 12	24.0	80	122.4	45.6	19,469
Sep 13	29.4	73	138.7	23.9	28,898
Sep 14	9.0	76	127.0	56.1	27,741
Sep 15	24.8	52	138.3	29.0	85,503
Sep 16	20.4	60	132.7	39.0	20,625

Eval allocation binned by what training held in the same ten-minute sample.
Training held (GPUs)	Samples	Left free by training, mean	Eval GPUs, mean
≥ 170	59	19.9	2.2
160–170	70	29.0	3.5
140–160	414	46.0	20.4
120–140	219	62.3	25.3
≤ 120	245	92.4	25.7

GPU columns come from Prometheus samples, which stop at 23:44 on day 7; rollouts are ledger rows finished in each UTC day.

Ten-minute Prometheus samples of GPU limits on running pods over the seven measured UTC days. Evals, the top band, fill toward the 192 line when training drops and shrink to almost nothing when it climbs; the eval peak was 80. Both bands together averaged 154 of 192 GPUs, against 134 for training and other jobs. Hover for values.

RayJob: the kind of eval this chart measures(click to expand)

The evals in the top band ran as elastic ledger evals, which is the shape of this manifest. Most of that week's eval GPU-hours used the earlier flavor without topology awareness, so this is the shape they ran in, not the exact spec of that week.

# Elastic eval, native Kueue path (how Fleet runs offline evals): ONE Workload whose size changes
# through Workload slices, one 1-GPU worker at a time, between 0 and 6. Lines marked [shape] define it.
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: tetris-elastic
  namespace: fleet-train-jobs
  labels:
    blog.fleet.ai/example: elastic
    kueue.x-k8s.io/queue-name: training-lq
    kueue.x-k8s.io/priority-class: q2          # [shape] lowest q: any q1 or q0 arrival may evict it
  annotations:
    kueue.x-k8s.io/elastic-job: "true"         # [shape] Kueue admits each size change as a new Workload slice
spec:
  suspend: true
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 0
  activeDeadlineSeconds: 3600
  backoffLimit: 0
  submissionMode: HTTPMode
  entrypoint: python -u /home/ray/app/app.py
  rayClusterSpec:
    rayVersion: "2.49.2"
    enableInTreeAutoscaling: true              # [shape] the Ray autoscaler changes the worker count
    autoscalerOptions: {idleTimeoutSeconds: 30}   # an idle worker (and its GPU) goes back after 30 s
    headGroupSpec:
      rayStartParams: {num-cpus: "0"}
      template:
        metadata:
          labels: {blog.fleet.ai/example: elastic}
        spec:
          priorityClassName: c2
          nodeSelector: {workload: fleetai-training-ng-cpu}   # -> flavor cpu-head
          containers:
            - name: ray-head
              image: docker.io/rayproject/ray:2.49.2-py312-cpu@sha256:de04957cc0a2f30563389ab945b5f96358f601ae76bd49755f5727adbcb5e7ba
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              volumeMounts:
                - {name: app, mountPath: /home/ray/app}
                - {name: sfs, mountPath: /mnt/sfs}         # the ledger outlives the RayCluster
          volumes:
            - {name: app, configMap: {name: tetris-elastic-app}}
            - {name: sfs, persistentVolumeClaim: {claimName: sfs-shared}}
    workerGroupSpecs:
      - groupName: eval
        replicas: 1                   # [shape] start with one worker: a 0-worker start gets no TAS placement
        minReplicas: 0                # [shape] may shrink to zero when idle
        maxReplicas: 6                # [shape] never more than 6 workers = 6 GPUs
        rayStartParams: {num-gpus: "1"}   # [shape] the unit of growth: one 1-GPU pod
        template:
          metadata:
            labels: {blog.fleet.ai/example: elastic}
            annotations:
              kueue.x-k8s.io/podset-unconstrained-topology: "true"   # [shape] the only TAS mode elastic jobs may use
          spec:
            priorityClassName: c2     # [shape] c2: preemptionPolicy Never
            nodeSelector: {workload: fleetai-training-ng-gpu}
            tolerations: [{key: workload, operator: Equal, value: fleetai-training-ng-gpu, effect: NoSchedule}]
            containers:
              - name: ray-worker
                image: docker.io/rayproject/ray:2.49.2-py312-gpu@sha256:0e53859fe223857409d8e99458d0b056e98c3319c8ec0f3232bf61669b492052
                resources:
                  requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}

Recorded GPU utilization averaged 81% on eval pods and 51% on other GPU pods. This describes activity while pods held GPUs; it does not measure training speed or prove that the scheduling policy caused the difference. Training demand was lower than in the preceding week, so these results do not establish a week-over-week allocation improvement. Cluster-wide utilization was about 55% in both weeks; training and other jobs averaged 157 allocated GPUs in the preceding week and 134 in the measured week.

Measurement method and evidence limits

The chart contains 1,007 ten-minute samples from September 10–16, 2026, UTC. We sum GPU limits on Running pods, split eval pods from other pods, and average the sampled counts. Estimated GPU-hours are the sum of those counts multiplied by ten minutes. Brief changes between samples may be missed.

The GPU-hour estimate weights these samples as 167 hours and 50 minutes. The original Prometheus data aged out of retention; the retained series is what supports the allocation figures. Ledger totals and utilization summaries were recorded separately and cannot be recomputed from this chart.

Download the chart data (CSV). The file contains the same aggregate counts shown here, with UTC timestamps. It includes no job names or per-task records.

Most eval GPU-hours used the earlier non-topology-aware flavor. This is neither a benchmark of the adapter nor a controlled comparison of topology policies. The detailed internal source record (private) requires repository access.

Some gaps remain unusable. Complete units must fit, topology restricts placement, and short allocations may end before model loading pays off. Backfilling must also leave a path for large jobs to start. The aim is more completed work from the GPUs we have, with recovery costs measured alongside the capacity we fill.

If you run this

What we would set and watch before running evals as a tier on a shared GPU cluster.

Validate your exact versions. Our results use a patched Kueue build. Elasticity with topology-aware scheduling is version-sensitive; enabling a feature flag alone does not reproduce these results.^{[2, 15]2Kueue: Elastic Workloads15Our RayJob template: elasticity and placement-group contract}
Account for shared GPUs together. Queue admission and physical placement must agree about which capacity is available. Give every job an explicit queue and verify its assigned nodes.
Choose the smallest useful execution unit. A four-GPU replica can fit gaps an eight-GPU replica cannot. Total free GPU count does not prove that a multi-node unit can fit its topology.
Choose an interruption policy. Set queue and pod priorities deliberately. Decide which workgroup can give way first, and how its peers cope with lower throughput.
Test recovery with real interruptions. Keep committed results or checkpoints outside the RayCluster. Verify restart behavior, duplicate handling, and recovery at every supported worker count.
Measure completed work and waiting time. Track startup, retries, and checkpoint costs alongside allocation. Check that small filler jobs do not keep a large waiting job from starting.

Evals are the most interruptible GPU work on our cluster, so we run them as its preemptible tier. Before you size a reservation for evals, ask what they could give back instead.

References

Numbered citations link claims to their sources. Official documentation explains the underlying tools. Our implementation notes and test records describe supported behavior and validation limits. Public documentation may describe a newer release than our tested configuration.

Scheduling, execution, and recovery

Kueue: Run a RayJob — Native RayJob admission, queue selection, and suspension.
Kueue: Elastic Workloads — Workload slices and changing resource requirements.
Kueue: Workload Priority Class — Queue priority, independent of pod priority.
Kueue: Preemption — Policies for reclaiming workload allocations.
Kubernetes: Pod Priority and Preemption — Pod scheduling priority and the meaning of preemptionPolicy: Never.
Kueue: Topology-Aware Scheduling — Node hierarchies and placement constraints.
Ray: Placement Groups — Resource bundles and placement strategies.
Ray: KubeRay Autoscaling — Scaling from task, actor, and placement-group resource demand.
PyTorch: Getting Started with Distributed Checkpoint — Saving distributed state and resharding it when loading at a different world size. We measured it on our own cluster with FSDP2 and AdamW state; see the run records.

Our implementation and validation

These links pin the source and configuration behind the claims above. The adapter and our Kueue changes live in their own repositories; our cluster configuration, run records and the measured week live in our infrastructure repository. All three are private for now and need access.

Kueue Ray worker-group adapter (private) — Supported inputs, independent admissions, eviction, and application responsibilities.
Fleet queue and pod priority configuration (private) — Our q0–q2 and c0–c2 values and preemption policies.
Fleet Kueue readiness configuration example (private) — Portable Helm values with startup and recovery timeouts, plus the requeue backoff limits.
Fleet Kueue preemption and readiness recovery (private) — Per-Workload recovery timeouts, admission scope, and application responsibilities.
Adapter elastic controller (private) — Cumulative growth, ordinary scale-down, and whole-workgroup draining on preemption.
Our RayJob template: elasticity and placement-group contract (private) — Supported topology modes and the limits of multi-pod shaped elasticity.
Adapter lifecycle validation, September 16, 2026 (private) — The historical one-node test and its limits. It does not establish training performance or multi-node packing results.
Job-shape examples, run records and claim audit, October 16 to 26, 2026 (private) — The RayJob manifests in this article and the records of their runs on the dev training cluster (one GPU node, two for the multi-node tests): admission, resizing, preemption, recovery windows, real-model evals, training and serving, and a claim-by-claim audit of what is and is not verified.
Adapter elastic reconciliation tests (private) — Automated checks for growth bounds, quota accounting, eviction, and sibling preservation.

Title inspiration

The title was inspired by Andrej Karpathy's Deep Reinforcement Learning: Pong from Pixels (2016).

Test cluster: 192 B300 GPUs. Kueue 0.19.4; KubeRay 1.4.2; Ray 2.49.2. Ledger and Prometheus aggregates: seven UTC days on that cluster in September 2026. The September 16 lifecycle trace falls inside that window but is not used to compute the weekly aggregates.

Disclaimer: Tetris is used here only as a GPU scheduling analogy. Fleet has no relationship or affiliation with Tetris or its rights holders. This article is not sponsored or endorsed by them.