Introduction
Managing GPU workloads in a Kubernetes cluster is a challenge that many AI teams face today. The team behind Nova AI recently published a detailed analysis of their journey selecting a GPU task scheduler, comparing three popular open-source solutions: Volcano, Kueue, and KAI Scheduler. This article summarizes their findings from the first part of their series, highlighting the key criteria, trade-offs, and practical considerations for anyone building a similar infrastructure.
The original material, published on Habr in July 2026, explains how the developers evaluated each scheduler based on real workloads, scalability needs, and integration complexity. The project team implemented a multi-stage selection process that involved stress testing, resource fragmentation analysis, and cost modeling. This first part focuses on the initial requirements and the comparative evaluation of Volcano versus Kueue.
Why GPU Scheduling Matters
GPU resources are expensive and often scarce. Without an efficient scheduler, teams face two common problems: underutilization (GPUs sitting idle while jobs wait) or contention (jobs failing due to resource conflicts). The Nova AI team needed a scheduler that could handle heterogeneous GPU types (NVIDIA A100, H100, and upcoming B200), support gang scheduling for multi-GPU training jobs, and integrate with their existing Prometheus monitoring stack.
The Candidates: Volcano vs. Kueue
Both Volcano and Kueue are Kubernetes-native batch schedulers, but they approach GPU management differently. The table below summarizes the key differences the team identified after initial testing.
| Feature | Volcano | Kueue |
|---|---|---|
| Gang scheduling | Yes (built-in) | No (requires additional config) |
| GPU topology awareness | Yes (NUMA, PCIe) | Partial (via node labels) |
| Queue management | Hierarchical queues | Simple queues with borrowing |
| Community maturity | Mature (since 2019) | Growing (since 2022) |
| Integration with Kubeflow | Native | Via admission controller |
The team ran a series of benchmark jobs using PyTorch Distributed Data Parallel (DDP) on 8-GPU nodes. Volcano consistently reduced job completion time by 15–20% compared to default Kubernetes scheduling, thanks to its gang scheduling that prevents partial resource allocation. Kueue, while easier to deploy, showed higher fragmentation in multi-tenant scenarios — one tenant could block others by reserving GPUs but not using them fully.
Real-World Results from the Team
The developers encountered a significant issue with Kueue during peak load: when multiple users submitted large training jobs simultaneously, the scheduler sometimes allocated GPUs in a way that left small unusable fragments across nodes. This led to a 12% increase in average job wait time. Volcano’s resource reservation mechanism mitigated this by packing jobs more densely.
However, Kueue had an advantage in simplicity. The team reported that setting up Volcano required additional Custom Resource Definitions (CRDs) and a dedicated controller, while Kueue worked out of the box with standard Kubernetes RBAC. For smaller teams with limited DevOps bandwidth, Kueue might be a better starting point.
Practical Recommendations from the Article
The Nova AI team offers three actionable takeaways for anyone evaluating GPU schedulers:
- Test with your own workloads — generic benchmarks don’t capture your specific job patterns. The team ran a 72-hour stress test with mixed batch and interactive jobs.
- Monitor GPU fragmentation — use tools like DCGM Exporter and Grafana dashboards to visualize how GPUs are allocated over time.
- Consider hybrid approaches — some companies use Volcano for training jobs and Kueue for inference serving, combining the strengths of both.
What’s Next in Part 2
The article hints that Part 2 will cover KAI Scheduler in depth, including its novel approach to preemption and fair sharing across teams. The team also plans to discuss their final decision and the custom patches they contributed to the chosen scheduler.
Conclusion
Choosing a GPU task scheduler is not a one-size-fits-all decision. The Nova AI team’s experience shows that Volcano offers superior performance for large-scale distributed training, while Kueue provides simplicity for smaller deployments. The key is to align the scheduler’s capabilities with your team’s operational maturity and workload characteristics. Read the full original analysis on Habr for more technical details.
Comments