The test suite includes several predefined fault injection scenarios designed to validate system resilience under various failure conditions. These scenarios are configured in `scenarios.py` and can be selected via pytest parameters. Below is a description of the available scenarios:
| Scenario Name | Description | Affected Components | Timing Example |
| **client_*.log.txt** | Request/response logs for each client instance (contains JSON-formatted request details) |
| **dynamo_*/error.log** | Error logs for specific Dynamo components (e.g., Frontend, Processor, VllmWorker) |
| **dynamo_*/output.log** | Standard output logs for Dynamo components (service startup/shutdown messages) |
| **dynamo.log.txt** | Aggregate logs for Dynamo services (orchestration and initialization) |
| **etcd.log.txt** | Logs for etcd, the distributed key-value store used for service coordination |
| **nats-server.log.txt** | Logs for NATS message broker, handling inter-service communication |
| **nvidia-smi.log.txt** | GPU monitoring logs (records utilization statistics during test execution) |
| **test.log.txt** | Primary test execution log (contains fault injection timing, process management, and test status)|
| **watcher.log.txt** | Metrics collected by the watcher service (e.g., pending requests, active workers) |
### Summary Results
Results are presented in table format after each test providing summary statistics.
**Test Group:** agg-tp-2-dp-1
**Test Command:** dynamo serve graphs.agg:Frontend -f /workspace/tests/fault_tolerance/configs/agg_tp_2_dp_1.yaml --Frontend.port 8000 in /workspace/examples/llm
| Failure | Startup Time | Success | Failed | Latency Before | Latency After | Pending Before | Pending After | Violations Before | Violations After | Recovery Time |
| **Failure** | Type of fault injection applied during the test (or 'none' for baseline) |
| **Startup Time** | Time (seconds) taken for the service to become ready after initialization |
| **Success** | Number of client requests that succeeded |
| **Failed** | Number of client requests that failed or were invalid |
| **Latency Before** | Average request latency (seconds) for successful requests before fault injection |
| **Latency After** | Average request latency (seconds) for successful requests after fault injection (N/A if no fault) |
| **Pending Before** | Average number of pending requests observed before fault injection |
| **Pending After** | Average number of pending requests observed after fault injection (N/A if no fault) |
| **Violations Before** | Number of successful requests exceeding SLA latency before fault injection |
| **Violations After** | Number of successful requests exceeding SLA latency after fault injection (N/A if no fault) |
| **Recovery Time** | Time (seconds) taken for failed components to recover after fault injection |
## Example Results
The following results were obtained running on a single node with 8
L40 GPUs using "deepseek-ai/DeepSeek-R1-Distill-Llama-8B" with 8
concurrent clients each sending 100 requests.
### Aggregated Workers
#### No Redundancy
To demonstrate the failure and recovery time in the case that there is
a single instance of each process we ran a simmple "agg-tp-2-dp-1" configuration.
```mermaid
graph LR
Client["Client"]
Frontend["Frontend"]
Processor["Processor"]
Client --> Frontend
Frontend --> Processor
Processor --> DecodePool
%% Decode Worker Pool (vertical layout)
subgraph DecodePool["Decode Worker Pool"]
direction TB
subgraph Decode1["Decode 1"]
direction TB
D1GPU0["GPU 0"]
D1GPU1["GPU 1"]
end
end
%% Styling
style DecodePool stroke:#000,stroke-width:2px
```
#### Results:
**Test Group: agg-tp-2-dp-1**
**Test Command:** dynamo serve graphs.agg:Frontend -f /workspace/tests/fault_tolerance/configs/agg_tp_2_dp_1.yaml --Frontend.port 8000 in /workspace/examples/llm
| Failure | Startup Time | Success | Failed | Latency Before | Latency After | Pending Before | Pending After | Violations Before | Violations After | Recovery Time |
1. Dynamo does not currently detect and recover from direct vllm worker sub process failure. (WIP)
2. Recovery time for the decode worker itself is the largest and a decode worker failure has the largest impact (as expected)
3. Overall failure count is roughly equal to recovery time multiplied by number of clients (as expected).
#### Redundant Workers (Over Provisoned)
To demonstrate the failure and recovery time in the case that there
are multiple instances of each process (except for the frontend) we
ran a simple "agg-tp-2-dp-4" configuration.
In this case we also consider the system to be "over provisioned" for
the workload as multiple workers are not needed to maintain SLA for
the 8 clients.
```mermaid
graph LR
Client["Client"]
Frontend["Frontend"]
Processor_1["Processor 1"]
Processor_2["Processor 2"]
Client --> Frontend
Frontend --> Processor_1
Frontend --> Processor_2
subgraph DecodePool["Decode Worker Pool"]
direction LR
subgraph Decode1["Decode 1"]
direction TB
D1GPU0["GPU 0"]
D1GPU1["GPU 1"]
end
subgraph Decode2["Decode 2"]
direction TB
D2GPU0["GPU 0"]
D2GPU1["GPU 1"]
end
subgraph Decode3["Decode 3"]
direction TB
D3GPU0["GPU 0"]
D3GPU1["GPU 1"]
end
subgraph Decode4["Decode 4"]
direction TB
D4GPU0["GPU 0"]
D4GPU1["GPU 1"]
end
end
Processor_1 --> DecodePool
Processor_2 --> DecodePool
style DecodePool stroke:#000,stroke-width:2px
```
#### Results:
**Test Group:** agg-tp-2-dp-4
**Test Command:** dynamo serve graphs.agg:Frontend -f /workspace/tests/fault_tolerance/configs/agg_tp_2_dp_4.yaml --Frontend.port 8000 in /workspace/examples/llm
| Failure | Startup Time | Success | Failed | Latency Before | Latency After | Pending Before | Pending After | Violations Before | Violations After | Recovery Time |
1. Dynamo does not currently detect and recover from direct vllm
worker sub process failure. In the case of redundant workers this
results in roughtly 1/4 the requests failing after the initial 30
seconds. (WIP)
2. By immediately detecting a decode worker failure, Dynamo can limit
the failures and reroute requests to healthy workers with minimal
impact.
3. While the processor was configured with redundancy - the system was
unable to instantiate two processors successfully leading to
failure when the processor was terminated. (WIP)
#### Redundant Workers (Exact Provisioning)
To demonstrate the failure and recovery time in the case that there
are multiple instances of each process (except for the frontend) we
ran a simple "agg-tp-2-dp-4" configuration.
In this case we also consider the system to be "exact provisioned" for
the workload as we limit the max-num-seqs for each decode worker to
exactly 2. This artificially creates a scenario that results in queing
when a failur occurs before a worker is recovered.
```mermaid
graph LR
Client["Client"]
Frontend["Frontend"]
Processor_1["Processor 1"]
Processor_2["Processor 2"]
Client --> Frontend
Frontend --> Processor_1
Frontend --> Processor_2
subgraph DecodePool["Decode Worker Pool"]
direction LR
subgraph Decode1["Decode 1 (max 2 seq)"]
direction TB
D1GPU0["GPU 0"]
D1GPU1["GPU 1"]
end
subgraph Decode2["Decode 2 (max 2 seq)"]
direction TB
D2GPU0["GPU 0"]
D2GPU1["GPU 1"]
end
subgraph Decode3["Decode 3 (max 2 seq)"]
direction TB
D3GPU0["GPU 0"]
D3GPU1["GPU 1"]
end
subgraph Decode4["Decode 4 (max 2 seq)"]
direction TB
D4GPU0["GPU 0"]
D4GPU1["GPU 1"]
end
end
Processor_1 --> DecodePool
Processor_2 --> DecodePool
style DecodePool stroke:#000,stroke-width:2px
```
#### Results:
**Test Group:** agg-tp-2-dp-4
**Test Command:** dynamo serve graphs.agg:Frontend -f /workspace/tests/fault_tolerance/configs/agg_tp_2_dp_4.yaml --Frontend.port 8000 --VllmWorker.max_num_seqs 2 in /workspace/examples/llm
| Failure | Startup Time | Success | Failed | Latency Before | Latency After | Pending Before | Pending After | Violations Before | Violations After | Recovery Time |
1. Dynamo does not currently detect and recover from direct vllm
worker sub process failure. In the case of redundant workers this
results in roughtly 1/4 the requests failing after the initial 30
seconds. All requests after the initial 30 seconds would also be
subject to queing as a result and we see increased SLA
violations. (WIP)
2. By immediately detecting a decode worker failure, Dynamo can limit
the failures and reroute requests to healthy workers with minimal
impact. However during the recovery period requests are subject to
queing and as a results we see increased SLA violations.
3. While the processor was configured with redundancy - the system was
unable to instantiate two processors successfully leading to
failure when the processor was terminated. (WIP)
### Disaggregated Workers
#### No Redunancy
To demonstrate the failure and recovery time in the case of a
disaaggregated deployment with a single instance for each process in
the graph we ran a simple `disagg-p-tp-2-dp-1-d-tp-4-dp-1` configuration.
```mermaid
graph LR
Client["Client"]
Frontend["Frontend"]
Processor["Processor"]
PrefillQueue["Remote Prefill Queue"]
Client --> Frontend
Frontend --> Processor
Processor <--> DecodePool
%% Prefill Worker Pool (horizontal layout)
subgraph PrefillPool["Prefill Worker Pool"]
direction LR
subgraph Prefill1["Prefill 1"]
direction TB
P1GPU0["GPU 0"]
P1GPU1["GPU 1"]
end
end
%% Decode Worker Pool (vertical layout)
subgraph DecodePool["Decode Worker Pool"]
direction TB
subgraph Decode1["Decode 1"]
direction TB
D1GPU0["GPU 0"]
D1GPU1["GPU 1"]
D1GPU2["GPU 2"]
D1GPU3["GPU 3"]
end
end
PrefillQueue --> PrefillPool
DecodePool --> PrefillQueue
PrefillPool -.-> DecodePool
%% Styling
style PrefillPool stroke:#0066cc,stroke-width:2px
style DecodePool stroke:#000,stroke-width:2px
```
#### Results:
**Test Group:** disagg-p-tp-2-dp-1-d-tp-4-dp-1
**Test Command:** dynamo serve graphs.disagg:Frontend -f /workspace/tests/fault_tolerance/configs/disagg_p_tp_2_dp_1_d_tp_4_dp_1.yaml --Frontend.port 8000 in /workspace/examples/llm
| Failure | Startup Time | Success | Failed | Latency Before | Latency After | Pending Before | Pending After | Violations Before | Violations After | Recovery Time |
1. Dynamo does not currently detect and recover from direct vllm
worker sub process failure. In this example the vllm sub process
failure targets a prefill worker and has the same overall impact.
(WIP)
2. Prefill worker failure causes request timeout (30 sec) and in
addition during recovery time prefill requests are queued in the
prefill queue.
3. Decode worker failure is currently permanent in the disaggregated
case as the prefill worker holds references to memory and which are
not freed. This leads to total failure after fault injection.
#### Redundant Workers
To demonstrate the failure and recovery time in the case that there
are multiple instances of each process (except for the frontend and
decode worker) we ran a simple "disagg-p-tp-2-dp-2-d-tp-4-dp-1"
configuration.
```mermaid
graph LR
Client["Client"]
Frontend["Frontend"]
Processor_1["Processor 1"]
Processor_2["Processor 2"]
PrefillQueue["Remote Prefill Queue"]
Client --> Frontend
Frontend --> Processor_1
Frontend --> Processor_2
Processor_1 <--> DecodePool
Processor_2 <--> DecodePool
%% Prefill Worker Pool (horizontal layout)
subgraph PrefillPool["Prefill Worker Pool"]
direction LR
subgraph Prefill1["Prefill 1"]
direction TB
P1GPU0["GPU 0"]
P1GPU1["GPU 1"]
end
subgraph Prefill2["Prefill 2"]
direction TB
P2GPU0["GPU 0"]
P2GPU1["GPU 1"]
end
end
%% Decode Worker Pool (vertical layout)
subgraph DecodePool["Decode Worker Pool"]
direction TB
subgraph Decode1["Decode 1"]
direction TB
D1GPU0["GPU 0"]
D1GPU1["GPU 1"]
D1GPU2["GPU 2"]
D1GPU3["GPU 3"]
end
end
PrefillQueue --> PrefillPool
DecodePool --> PrefillQueue
PrefillPool -.-> DecodePool
%% Styling
style PrefillPool stroke:#0066cc,stroke-width:2px
style DecodePool stroke:#000,stroke-width:2px
```
#### Results:
**Test Group:** disagg-p-tp-2-dp-2-d-tp-4-dp-1
**Test Command:** dynamo serve graphs.disagg:Frontend -f /workspace/tests/fault_tolerance/configs/disagg_p_tp_2_dp_2_d_tp_4_dp_1.yaml --Frontend.port 8000 in /workspace/examples/llm
| Failure | Startup Time | Success | Failed | Latency Before | Latency After | Pending Before | Pending After | Violations Before | Violations After | Recovery Time |
