NVIDIA Dynamo is a new modular inference framework designed for serving large language models (LLMs) in multi-node
distributed environments. It enables seamless scaling of inference workloads across GPU nodes and the dynamic allocation
of GPU workers to address traffic bottlenecks at various stages of the model pipeline.
NVIDIA Dynamo also features LLM-specific capabilities, such as disaggregated serving, which separates the context
(prefill) and generation (decode) steps of inference requests onto distinct GPUs and GPU nodes to optimize performance.
NVIDIA Dynamo includes four key innovations:
Dynamo is a flexible, component based, data center scale inference
***Smart Router**: An LLM-aware router that directs requests across large GPU fleets to minimize costly key-value (KV)
serving framework designed to meet the demands of complex use cases
cache recomputations for repeat or overlapping requests, freeing up GPUs to respond to new incoming requests
including those of Generative AI. It is designed to enable developers
***Low-Latency Communication Library**: An inference optimized library that supports state-of-the-art GPU-to-GPU
to implement and customize routing, load balancing, scaling and
communication and abstracts complexity of data exchange across heterogenous devices and networking protocols,
workflow definitions at the data center scale without sacrificing
accelerating data transfers
performance or ease of use.
***Memory Manager**: An engine that intelligently offloads and reloads inference data (KV cache) to and from lower-cost memory and storage devices using NVIDIA NIXL without impacting user experiences
> [!NOTE]
> [!NOTE]
> This project is currently in the alpha / experimental /
> This project is currently in the alpha / experimental /
> rapid-prototyping stage and we are actively looking for feedback and
> rapid-prototyping stage and we are actively looking for feedback and
> collaborators.
> collaborators.
## Quick Start
TODO add quick start guide here
## Building Dynamo
## Building Dynamo
### Requirements
### Requirements
Dynamo development and examples are container based.
Dynamo development and examples are container based.
