IBM-Backed llm-d 1.0 Adds Controls for AI Inference Workloads

The llm-d project released version 1.0 on Oct. 20, adding workload controls and routing features for organizations running large AI models on Kubernetes infrastructure they manage. IBM, Red Hat and Google are lead maintainers of llm-d, an open-source framework that distributes inference requests among model servers such as vLLM and SGLang. The project is hosted by the Cloud Native Computing Foundation as a sandbox project. The new controls let operators prioritize requests, share capacity among users and workloads, and shed traffic when demand exceeds capacity. Existing deployments keep their current scheduling behavior unless operators enable the new policies.

Source: TokenPost