Unverified Commit eebc7410 authored by atchernych's avatar atchernych Committed by GitHub
Browse files

docs: Adjust the path to examples (#2056)

parent ffb54096
...@@ -68,12 +68,13 @@ pip install "ai-dynamo[all]" ...@@ -68,12 +68,13 @@ pip install "ai-dynamo[all]"
### Building the Dynamo Base Image ### Building the Dynamo Base Image
Although not needed for local development, deploying your Dynamo pipelines to Kubernetes will require you to build and push a Dynamo base image to your container registry. You can use any container registry of your choice, such as: Although not needed for local development, deploying your Dynamo pipelines to Kubernetes will require you to use a Dynamo base image to your container registry. You can use any container registry of your choice, such as:
- Docker Hub (docker.io) - Docker Hub (docker.io)
- NVIDIA NGC Container Registry (nvcr.io) - NVIDIA NGC Container Registry (nvcr.io)
- Any private registry - Any private registry
Here's how to build it: We publish our images in [nvcr.io](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/containers/vllm-runtime) and you can use them.
Alternatively you could build and push an image from source:
```bash ```bash
./container/build.sh ./container/build.sh
...@@ -83,8 +84,10 @@ docker push <your-registry>/dynamo-base:latest-vllm ...@@ -83,8 +84,10 @@ docker push <your-registry>/dynamo-base:latest-vllm
``` ```
Notes about builds for specific frameworks: Notes about builds for specific frameworks:
- For specific details on the `--framework vllm` build, see [here](examples/vllm/README.md). - For specific details on the `--framework vllm` build [read about the VLLM backend](components/backends/vllm/README.md)
- For specific details on the `--framework tensorrtllm` build, see [here](examples/tensorrt_llm/README.md). .
- For specific details on the `--framework tensorrtllm` build, see [Read about the TensorRT-LLM backend](components/backends/trtllm/README.md)
.
Note about AWS environments: Note about AWS environments:
- If deploying Dynamo in AWS, make sure to build the container with EFA support using the `--make-efa` flag. - If deploying Dynamo in AWS, make sure to build the container with EFA support using the `--make-efa` flag.
...@@ -197,8 +200,6 @@ pip install . ...@@ -197,8 +200,6 @@ pip install .
cd ../../../ cd ../../../
pip install ".[all]" pip install ".[all]"
# To test Follow the [Quickstart Guide](docs/guides/dynamo_deploy/quickstart.md)
docker compose -f deploy/metrics/docker-compose.yml up -d
cd examples/llm
dynamo serve graphs.agg:Frontend -f configs/agg.yaml
``` ```
\ No newline at end of file
...@@ -29,8 +29,10 @@ This approach allows you to install Dynamo directly using a DynamoGraphDeploymen ...@@ -29,8 +29,10 @@ This approach allows you to install Dynamo directly using a DynamoGraphDeploymen
### Basic Installation ### Basic Installation
Here is how you would install a VLLM inference backend example.
```bash ```bash
helm upgrade --install dynamo-graph ./deploy/helm/chart -n dynamo-cloud -f ./examples/vllm/deploy/agg.yaml helm upgrade --install dynamo-graph ./deploy/helm/chart -n dynamo-cloud -f ./components/backends/vllm/deploy/agg.yaml
``` ```
### Customizable Properties ### Customizable Properties
...@@ -39,7 +41,7 @@ You can override the default configuration by setting the following properties: ...@@ -39,7 +41,7 @@ You can override the default configuration by setting the following properties:
```bash ```bash
helm upgrade --install dynamo-graph ./deploy/helm/chart -n dynamo-cloud \ helm upgrade --install dynamo-graph ./deploy/helm/chart -n dynamo-cloud \
-f ./examples/vllm/deploy/agg.yaml \ -f ./components/backends/vllm/deploy/agg.yaml \
--set "imagePullSecrets[0].name=docker-secret-1" \ --set "imagePullSecrets[0].name=docker-secret-1" \
--set etcdAddr="my-etcd-service:2379" \ --set etcdAddr="my-etcd-service:2379" \
--set natsAddr="nats://my-nats-service:4222" --set natsAddr="nats://my-nats-service:4222"
......
...@@ -16,11 +16,7 @@ This guide provides instructions for setting up the Inference Gateway with Dynam ...@@ -16,11 +16,7 @@ This guide provides instructions for setting up the Inference Gateway with Dynam
[See Quickstart Guide](../../../docs/guides/dynamo_deploy/quickstart.md) to install Dynamo Cloud. [See Quickstart Guide](../../../docs/guides/dynamo_deploy/quickstart.md) to install Dynamo Cloud.
2. **Launch Dynamo Deployments** 2. **Deploy Inference Gateway**
[See VLLM Example](../../../examples/vllm/README.md)
3. **Deploy Inference Gateway**
First, deploy an inference gateway service. In this example, we'll install `kgateway` based gateway implementation. First, deploy an inference gateway service. In this example, we'll install `kgateway` based gateway implementation.
...@@ -54,7 +50,7 @@ kubectl get gateway inference-gateway ...@@ -54,7 +50,7 @@ kubectl get gateway inference-gateway
# inference-gateway kgateway True 1m # inference-gateway kgateway True 1m
``` ```
4. **Apply Dynamo-specific manifests** 3. **Apply Dynamo-specific manifests**
The Inference Gateway is configured through the `inference-gateway-resources.yaml` file. The Inference Gateway is configured through the `inference-gateway-resources.yaml` file.
......
...@@ -2,7 +2,7 @@ ...@@ -2,7 +2,7 @@
## Serving examples locally ## Serving examples locally
TODO: Follow individual examples to serve models locally. Follow individual examples under components/backends/ to serve models locally.
## Deploying Examples to Kubernetes ## Deploying Examples to Kubernetes
...@@ -38,7 +38,7 @@ export NAMESPACE=<your-namespace> # the namespace you used to deploy Dynamo clou ...@@ -38,7 +38,7 @@ export NAMESPACE=<your-namespace> # the namespace you used to deploy Dynamo clou
Deploying an example consists of the simple `kubectl apply -f ... -n ${NAMESPACE}` command. For example: Deploying an example consists of the simple `kubectl apply -f ... -n ${NAMESPACE}` command. For example:
```bash ```bash
kubectl apply -f examples/vllm/deploy/agg.yaml -n ${NAMESPACE} kubectl apply -f components/backends/vllm/deploy/agg.yaml -n ${NAMESPACE}
``` ```
You can use `kubectl get dynamoGraphDeployment -n ${NAMESPACE}` to view your deployment. You can use `kubectl get dynamoGraphDeployment -n ${NAMESPACE}` to view your deployment.
......
...@@ -167,7 +167,7 @@ docker compose -f deploy/docker-compose.yml up -d ...@@ -167,7 +167,7 @@ docker compose -f deploy/docker-compose.yml up -d
### Start Dynamo LLM Serving Components ### Start Dynamo LLM Serving Components
[Explore the VLLM Example](../examples/vllm/README.md) [Explore the VLLM Example](../components/backends/vllm/README.md)
## Local Development ## Local Development
......
...@@ -187,19 +187,6 @@ We provide a script to uninstall CRDs should you need a clean start. ...@@ -187,19 +187,6 @@ We provide a script to uninstall CRDs should you need a clean start.
## Explore Examples ## Explore Examples
Pick your deployment destination.
If local
```bash
export DYNAMO_CLOUD=http://localhost:8080
```
If kubernetes
```bash
export DYNAMO_CLOUD=https://dynamo-cloud.nvidia.com
```
If deploying to Kubernetes, create a Kubernetes secret containing your sensitive values if needed: If deploying to Kubernetes, create a Kubernetes secret containing your sensitive values if needed:
```bash ```bash
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment