As you read about each concept, it is helpful to have the [basic example](../examples/hello_world.md) up as well so you can refer back to it.
As you read about each concept, it is helpful to have the [basic example](../examples/hello_world.md) up as well so you can refer back to it.
### Defining a Service
### Defining a Service
...
@@ -49,6 +50,7 @@ A Service is a core building block for a project. You can think of it as a logic
...
@@ -49,6 +50,7 @@ A Service is a core building block for a project. You can think of it as a logic
```
```
Key configuration options:
Key configuration options:
1.`dynamo`: Dictionary that defines the Dynamo configuration and enables/disables it. When enabled, a dynamo worker is created under the hood which can register with the [Distributed Runtime](../architecture/architecture.md)
1.`dynamo`: Dictionary that defines the Dynamo configuration and enables/disables it. When enabled, a dynamo worker is created under the hood which can register with the [Distributed Runtime](../architecture/architecture.md)
2.`resources`: Dictionary defining resource requirements. The GPUs field is used for local and remote deployment. The other fields are used to determine resources when deploying to K8s.
2.`resources`: Dictionary defining resource requirements. The GPUs field is used for local and remote deployment. The other fields are used to determine resources when deploying to K8s.
3.`workers`: Number of parallel instances of the service to spin up.
3.`workers`: Number of parallel instances of the service to spin up.
...
@@ -93,7 +95,9 @@ class ServiceA:
...
@@ -93,7 +95,9 @@ class ServiceA:
```
```
#### Class-Based Architecture
#### Class-Based Architecture
Dynamo follows a class-based architecture similar to BentoML making it intuitive for users familiar with those frameworks. Each service is defined as a Python class, with the following components:
Dynamo follows a class-based architecture similar to BentoML making it intuitive for users familiar with those frameworks. Each service is defined as a Python class, with the following components:
1. Class attributes for dependencies using `depends()`
1. Class attributes for dependencies using `depends()`
2. An `__init__` method for standard initialization
2. An `__init__` method for standard initialization
3. Optional lifecycle hooks like `@async_on_start` and `@on_shutdown`
3. Optional lifecycle hooks like `@async_on_start` and `@on_shutdown`
...
@@ -104,7 +108,9 @@ Dynamo follows a class-based architecture similar to BentoML making it intuitive
...
@@ -104,7 +108,9 @@ Dynamo follows a class-based architecture similar to BentoML making it intuitive
This approach provides a clean separation of concerns and makes the service structure easy to understand.
This approach provides a clean separation of concerns and makes the service structure easy to understand.
#### Service Dependencies with `depends()`
#### Service Dependencies with `depends()`
The `depends()` function is a powerful feature that lets you create a dependency between services. When you use `depends(ServiceB)`, several things happen:
The `depends()` function is a powerful feature that lets you create a dependency between services. When you use `depends(ServiceB)`, several things happen:
1. It ensures that `ServiceB` is deployed when `ServiceA` is deployed by adding it to an internal service dependency graph
1. It ensures that `ServiceB` is deployed when `ServiceA` is deployed by adding it to an internal service dependency graph
2. It creates a client to the endpoints of `ServiceB` that is being served under the hood.
2. It creates a client to the endpoints of `ServiceB` that is being served under the hood.
3. You are able to access `ServiceB` endpoints as if it were a local function!
3. You are able to access `ServiceB` endpoints as if it were a local function!
...
@@ -142,6 +148,7 @@ self.worker_client = (
...
@@ -142,6 +148,7 @@ self.worker_client = (
This is used in some of our prebuilt examples and is a powerful way to leverage the benefits of the SDK while being able to access Dynamo's core primitives.
This is used in some of our prebuilt examples and is a powerful way to leverage the benefits of the SDK while being able to access Dynamo's core primitives.
#### Lifecycle Hooks
#### Lifecycle Hooks
Dynamo supports key lifecycle hooks to manage service initialization and cleanup.
Dynamo supports key lifecycle hooks to manage service initialization and cleanup.
##### `@async_on_start`
##### `@async_on_start`
...
@@ -156,11 +163,14 @@ async def async_init(self):
...
@@ -156,11 +163,14 @@ async def async_init(self):
self.tokenizer=awaitload_tokenizer()
self.tokenizer=awaitload_tokenizer()
self.engine=awaitinitialize_engine(self.model)
self.engine=awaitinitialize_engine(self.model)
```
```
This is especially useful for:
This is especially useful for:
- Initializing external connections
- Initializing external connections
- Setting up runtime resources that require async operations
- Setting up runtime resources that require async operations
#### `@on_shutdown`
#### `@on_shutdown`
The `@on_shutdown` hook is called when the service is shutdown handles cleanup.
The `@on_shutdown` hook is called when the service is shutdown handles cleanup.
```python
```python
...
@@ -220,6 +230,7 @@ MyService:
...
@@ -220,6 +230,7 @@ MyService:
```
```
The YAML file has a hierarchical structure:
The YAML file has a hierarchical structure:
- Top level keys are service class names
- Top level keys are service class names
-`ServiceArgs` contains parameters for the service decorator
-`ServiceArgs` contains parameters for the service decorator
- Other keys are passed as arguments to the service constructor
- Other keys are passed as arguments to the service constructor
...
@@ -322,13 +333,14 @@ The `ServiceArgs` section in YAML configuration allows you to override any param
...
@@ -322,13 +333,14 @@ The `ServiceArgs` section in YAML configuration allows you to override any param
MyService:
MyService:
ServiceArgs:
ServiceArgs:
dynamo:
dynamo:
namespace:"staging"# Override namespace
namespace:"staging"# Override namespace
resources:
resources:
gpu:4# Use more GPUs
gpu:4# Use more GPUs
workers:8# Scale up workers
workers:8# Scale up workers
```
```
This is particularly useful for:
This is particularly useful for:
- Changing resource allocations between environments
- Changing resource allocations between environments
- Modifying worker counts based on expected load
- Modifying worker counts based on expected load
- Switching between namespaces for different deployments
- Switching between namespaces for different deployments
Here's a comprehensive example showing how all these pieces fit together:
Here's a comprehensive example showing how all these pieces fit together:
...
@@ -402,6 +415,7 @@ dynamo serve service:LLMService -f prod_config.yaml --LLMService.temperature=0.9
...
@@ -402,6 +415,7 @@ dynamo serve service:LLMService -f prod_config.yaml --LLMService.temperature=0.9
```
```
The service receives the combined configuration with the command-line value taking precedence, resulting in effective configuration of:
The service receives the combined configuration with the command-line value taking precedence, resulting in effective configuration of:
-`dynamo.namespace = "prod"`
-`dynamo.namespace = "prod"`
-`resources.gpu = 4`
-`resources.gpu = 4`
-`workers = 8`
-`workers = 8`
...
@@ -422,21 +436,27 @@ The service receives the combined configuration with the command-line value taki
...
@@ -422,21 +436,27 @@ The service receives the combined configuration with the command-line value taki
Following these practices help create flexible and maintainable Dynamo services that can be easily configured for different environments and use cases.
Following these practices help create flexible and maintainable Dynamo services that can be easily configured for different environments and use cases.
### Deploying a Single Service
### Deploying a Single Service
You can deploy a single service for local development even if you have a dependancy graph defined using `depends()` using `dynamo serve --service-name <ClassName> <entrypoint> <configuration arguments>`. You can see an example of this in our multinode documentation [here](../examples/multinode.md).
You can deploy a single service for local development even if you have a dependency graph defined using `depends()` using `dynamo serve --service-name <ClassName> <entrypoint> <configuration arguments>`. You can see an example of this in our multinode documentation [here](../examples/multinode.md).
### Composing Services into an Graph
### Composing Services into an Graph
There are two main ways to compose services in Dynamo:
There are two main ways to compose services in Dynamo:
1. Use `depends()` (Recommended)
1. Use `depends()` (Recommended)
The depends() approach is the recommended way for production deployments:
The depends() approach is the recommended way for production deployments:
- Automatically deploys all dependencies
- Automatically deploys all dependencies
- Creates a static inference graph at deployment time
- Creates a static inference graph at deployment time
- Provides type hints and better IDE support
- Provides type hints and better IDE support
2. Use `.link()` (Experimental)
2. Use `.link()` (Experimental)
Our `.link()` syntax is an flexible and experimental way to compose various services. Linking allows you to compose checks at runtime and view behavior. Under the hood - we are editing the dependency graph between various services. This is useful for experimentation and development but we suggest writing a static graph for your final production deployment.
Our `.link()` syntax is an flexible and experimental way to compose various services. Linking allows you to compose checks at runtime and view behavior. Under the hood - we are editing the dependency graph between various services. This is useful for experimentation and development but we suggest writing a static graph for your final production deployment.
#### Understanding the `.link()` syntax
#### Understanding the `.link()` syntax
Lets take the example of a `Processor` component. This component can currently do 2 things:
Lets take the example of a `Processor` component. This component can currently do 2 things:
1. Process a request and send it to a `Router` to decide what worker to send it to.
1. Process a request and send it to a `Router` to decide what worker to send it to.
2. Process a request and send it to a `Worker` directly.
2. Process a request and send it to a `Worker` directly.