Skip to content

GitLab

"lib/runtime/vscode:/vscode.git/clone" did not exist on "1f07dab7bd8b0563893f820d725ca980dfb9e49f"
Unverified Commit 02c822d6 authored by Tzu-Ling Kan's avatar Tzu-Ling Kan Committed by GitHub
Browse files

feat: Using ai-perf for k8 FT tests. (#3289) 


Signed-off-by: default avatartzulingk@nvidia.com <tzulingk@nvidia.com>
parent b5782fcd
Hide whitespace changes
Inline Side-by-side
Showing with 1076 additions and 722 deletions
tests/fault_tolerance/deploy/README.md
View file @ 02c822d6
...@@ -28,11 +28,12 @@ conditions. ...@@ -28,11 +28,12 @@ conditions.
The fault tolerance test suite is designed as a set of pytest The fault tolerance test suite is designed as a set of pytest
configurations that launch typical dynamo deployments in a Kubernetes configurations that launch typical dynamo deployments in a Kubernetes
environemnt and then inject failures by terminating processes or environment and then inject failures by terminating processes or
pods. To test the recovery time and impact of failures a set number of pods. To test the recovery time and impact of failures, a set number of
clients are launched in parallel. Each client sends a set number of clients are launched in parallel using **AI-Perf (aiperf)** for load generation.
synchronous requests. Log files are stored for each pod as well as for Each client sends synthetic requests with configurable token patterns.
each client and inspected using a post-processing script. Log files are stored for each pod as well as for each client and inspected
using a post-processing script that parses AI-Perf metrics.
> [!NOTE] > [!NOTE]
> Test pass / failure is not an indication of SLA for recovery or resilience > Test pass / failure is not an indication of SLA for recovery or resilience
...@@ -82,12 +83,18 @@ Below are some representative examples of the generated scenarios: ...@@ -82,12 +83,18 @@ Below are some representative examples of the generated scenarios:
The full test matrix is generated from these parameters, creating comprehensive test coverage across all configurations. The full test matrix is generated from these parameters, creating comprehensive test coverage across all configurations.
#### Client Load #### Client Load (AI-Perf Configuration)
- **Concurrent Clients**: 10 clients by default, adjustable per scenario. - **Load Generator**: AI-Perf (`aiperf`) with synthetic token generation
- **Requests per Client**: 100 requests, simulating sustained load. - **Concurrent Clients**: 10 clients by default, adjustable per scenario
- **Input/Output Token Length**: 100 tokens for both input prompts and generated outputs. - **Requests per Client**: 150 requests per client (configurable)
- **Request Rate Limit**: Ensures clients do not overwhelm the service, with a maximum of 1 request per second per client. - **Input/Output Token Configuration**:
- Input tokens: mean=100, stddev=0 (consistent length)
- Output tokens: mean=100, stddev=0 (consistent length)
- **Concurrency**: Sequential requests (concurrency=1) per client
- **Retry Logic**: 3 retry attempts for fault tolerance
- **Streaming Support**: Optional `--streaming` flag for TTFT/ITL metrics
- **No warmup**: warmup-request-count=0 to avoid initial failures
#### Failures #### Failures
...@@ -131,83 +138,107 @@ pytest tests/fault_tolerance/deploy/test_deployment.py -s -v --namespace ${NAMES ...@@ -131,83 +138,107 @@ pytest tests/fault_tolerance/deploy/test_deployment.py -s -v --namespace ${NAMES
### Test Results Directory ### Test Results Directory
For each test scenario a directory of log files is created and post processed to summarize the test. For each test scenario a directory of log files is created and post-processed to summarize the test.
``` ```
test_fault_scenario[agg-tp-1-dp-1-none] test_fault_scenario[sglang-agg-tp-1-dp-1-frontend]
. .
├── client_0.log.txt ├── client_0/
├── client_1.log.txt │ └── attempt_0/
├── client_2.log.txt │ ├── profile_export_aiperf.json # AI-Perf metrics in JSON format
├── client_3.log.txt │ ├── profile_export_aiperf.csv # AI-Perf metrics in CSV format
├── client_4.log.txt │ ├── genai_perf.log # AI-Perf execution log
├── client_5.log.txt │ └── logs/
├── client_6.log.txt │ └── aiperf.log # Detailed AI-Perf logs
├── client_7.log.txt ├── client_1/
├── client_8.log.txt │ ├── attempt_0/ # First attempt (may fail during fault)
├── client_9.log.txt │ └── attempt_1/ # Retry attempt after failure
├── Frontend │ └── [same structure as above]
├── [client_2 through client_9...]
├── Frontend/
│ ├── fault-tolerance-test-frontend-576bd784dc-jv68q.log │ ├── fault-tolerance-test-frontend-576bd784dc-jv68q.log
│ ├── fault-tolerance-test-frontend-576bd784dc-jv68q.metrics.log │ ├── fault-tolerance-test-frontend-576bd784dc-jv68q.metrics.log
│ ├── fault-tolerance-test-frontend-576bd784dc-jv68q.previous.log │ ├── fault-tolerance-test-frontend-576bd784dc-jv68q.previous.log # Pre-restart logs
│ └── fault-tolerance-test-frontend-576bd784dc-jv68q.yaml │ └── fault-tolerance-test-frontend-576bd784dc-jv68q.yaml
├── test.log.txt ├── decode/ # Or VllmDecodeWorker for vLLM backend
└── VllmDecodeWorker │ └── [same structure as Frontend]
├── fault-tolerance-test-vllmdecodeworker-56b7bdf447-6tzqq.log └── test.log.txt
├── fault-tolerance-test-vllmdecodeworker-56b7bdf447-6tzqq.metrics.log
├── fault-tolerance-test-vllmdecodeworker-56b7bdf447-6tzqq.previous.log
└── fault-tolerance-test-vllmdecodeworker-56b7bdf447-6tzqq.yaml
``` ```
| File/Directory Name | Description | | File/Directory Name | Description |
|------------------------------------|------------------------------------------------------------------------------------------------| |------------------------------------|------------------------------------------------------------------------------------------------|
| **client_*.log.txt** | Request/response logs for each client instance (contains JSON-formatted request details) | | **client_N/attempt_M/** | AI-Perf results for client N, attempt M (supports multiple retry attempts) |
| **{Service}/*.log | Container log for pod at end of test (Frontend, VllmDecodeWroer, etc.) | | **profile_export_aiperf.json** | Complete AI-Perf metrics including latencies (P50/P90/P99), throughput, token counts |
| **{Service}/*.previous.log** | Previous container log for pod in case of crash / exit. (Frontend, VllmDecodeWroer, etc.). Empty if N/A. | | **profile_export_aiperf.csv** | Tabular format of key metrics for easy analysis |
| **{Service}/*.metrics.log** | Metrics as reported by `/metrics` for the service | | **genai_perf.log** | AI-Perf execution output (stdout/stderr) |
| **{Service}/*.yaml** | yaml for pod including status transitions | | **{Service}/*.log** | Current container log for pod (Frontend, decode, etc.) |
| **test.log.txt** | Primary test execution log (contains fault injection timing, process management, and test status)| | **{Service}/*.previous.log** | Previous container log before restart (contains pre-fault logs) |
| **{Service}/*.metrics.log** | Prometheus metrics from `/metrics` endpoint |
| **{Service}/*.yaml** | Pod specification and status transitions |
| **test.log.txt** | Primary test execution log (deployment timing, fault injection, recovery events) |
### Summary Results ### Summary Results
Results are presented in table format after each test providing summary statistics. Results are parsed from AI-Perf metrics and presented in table format after each test. The parsing script (`parse_results.py`) extracts comprehensive metrics for each scenario:
#### Per-Test Output Format
``` ```
Test Group: agg-tp-1-dp-1 ============================================================
╒═════════════════════════╤═══════════╤═══════════╤══════════╤═══════════╤══════════╤═══════════╤═══════════╤════════════╕ FAULT TOLERANCE TEST SUMMARY - AI-PERF
│ Failure │ Startup │ Success │ Failed │ Success │ Failed │ Latency │ Latency │ Recovery │ ============================================================
│ │ │ Before │ Before │ After │ After │ Before │ After │ │ ╒═══════════════════════════════════╤════════════════════════════════════════════════════╕
╞═════════════════════════╪═══════════╪═══════════╪══════════╪═══════════╪══════════╪═══════════╪═══════════╪════════════╡ │ Metric │ Value │
│ none │ 180.00 │ 1500.00 │ 0.00 │ N/A │ N/A │ 1.19 │ N/A │ N/A │ ╞═══════════════════════════════════╪════════════════════════════════════════════════════╡
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤ │ Test Directory │ test_fault_scenario[sglang-agg-tp-1-dp-1-frontend] │
│ frontend │ 181.00 │ 153.00 │ 0.00 │ 820.00 │ 527.00 │ 1.21 │ 1.18 │ 3.36 │ ├───────────────────────────────────┼────────────────────────────────────────────────────┤
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤ │ Number of Clients │ 10 │
│ frontend_pod │ 169.00 │ 140.00 │ 0.00 │ 785.00 │ 305.00 │ 1.20 │ 1.18 │ 5.39 │ ├───────────────────────────────────┼────────────────────────────────────────────────────┤
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤ │ === Deployment Metrics === │ │
│ decode_worker │ 161.00 │ 140.00 │ 0.00 │ 510.00 │ 850.00 │ 1.21 │ 1.18 │ 154.11 │ ├───────────────────────────────────┼────────────────────────────────────────────────────┤
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤ │ Startup Time │ 69.00 sec │
│ decode_worker_pod │ 181.00 │ 140.00 │ 0.00 │ 511.00 │ 849.00 │ 1.22 │ 1.18 │ 156.47 │ ├───────────────────────────────────┼────────────────────────────────────────────────────┤
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤ │ Recovery Time │ 2.00 sec │
│ vllm_decode_engine_core │ 181.00 │ 140.00 │ 0.00 │ 524.00 │ 836.00 │ 1.21 │ 1.19 │ 152.52 │ ├───────────────────────────────────┼────────────────────────────────────────────────────┤
╘═════════════════════════╧═══════════╧═══════════╧══════════╧═══════════╧══════════╧═══════════╧═══════════╧════════════╛ │ === Request Metrics === │ │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Total Requests │ 1500 │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Successful Requests │ 1470 │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Failed Requests │ 30 │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Success Rate │ 98.00% │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ === Latency Metrics (seconds) === │ │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Mean Latency │ 0.502 │
├───────────────────────────────────┼────────────────────────────────────────────────┤
│ P50 Latency │ 0.396 │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ P90 Latency │ 0.422 │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ P99 Latency │ 0.761 │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ === Throughput Metrics === │ │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Total Throughput │ 19.72 req/s │
├───────────────────────────────────┼────────────────────────────────────────────────────┤
│ Avg Client Throughput │ 1.97 req/s │
╘═══════════════════════════════════╧════════════════════════════════════════════════════╛
``` ```
| Metric Category | Metrics Included |
| Column Name | Description |
|-----------------------|-----------------------------------------------------------------------------| |-----------------------|-----------------------------------------------------------------------------|
| **Failure** | Type of fault injection applied during the test (or 'none' for baseline) | | **Deployment Metrics**| Startup Time, Recovery Time |
| **Startup** | Time (seconds) taken for the service to become ready after initialization | | **Request Metrics** | Total/Successful/Failed Requests, Success Rate |
| **Succes/nBefore** | Numoer of client requests that succeeded before fault injection | | **Latency Metrics** | Mean, P50, P90, P99 latencies (in seconds) |
| **Failed/nBefore** | Number of client requests that failed or were invalid before fault injection | | **Token Metrics** | TTFT (Time to First Token), ITL (Inter-Token Latency) when streaming enabled |
| **Success/nAftere** | Number of client requests that succeeded after fault injection | | **Throughput Metrics**| Total and per-client request throughput |
| **Latency Before** | Average request latency (seconds) for successful requests before fault injection |
| **Latency After** | Average request latency (seconds) for successful requests after fault injection |
| **Recovery Time** | Time (seconds) taken for failed components to recover after fault injection |
## Example Results ## Example Deployment Architectures
The following results were obtained running on a cluster of A100 The following architectures are tested with various failure scenarios:
nodes.
### Aggregated Workers ### Aggregated Workers
...@@ -237,32 +268,9 @@ graph LR ...@@ -237,32 +268,9 @@ graph LR
style DecodePool stroke:#000,stroke-width:2px style DecodePool stroke:#000,stroke-width:2px
``` ```
#### Results:
```
Test Group: agg-tp-1-dp-1
╒═════════════════════════╤═══════════╤═══════════╤══════════╤═══════════╤══════════╤═══════════╤═══════════╤════════════╕
│ Failure │ Startup │ Success │ Failed │ Success │ Failed │ Latency │ Latency │ Recovery │
│ │ │ Before │ Before │ After │ After │ Before │ After │ │
╞═════════════════════════╪═══════════╪═══════════╪══════════╪═══════════╪══════════╪═══════════╪═══════════╪════════════╡
│ none │ 180.00 │ 1500.00 │ 0.00 │ N/A │ N/A │ 1.19 │ N/A │ N/A │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend │ 181.00 │ 153.00 │ 0.00 │ 820.00 │ 527.00 │ 1.21 │ 1.18 │ 3.36 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend_pod │ 169.00 │ 140.00 │ 0.00 │ 785.00 │ 305.00 │ 1.20 │ 1.18 │ 5.39 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker │ 161.00 │ 140.00 │ 0.00 │ 510.00 │ 850.00 │ 1.21 │ 1.18 │ 154.11 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker_pod │ 181.00 │ 140.00 │ 0.00 │ 511.00 │ 849.00 │ 1.22 │ 1.18 │ 156.47 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ vllm_decode_engine_core │ 181.00 │ 140.00 │ 0.00 │ 524.00 │ 836.00 │ 1.21 │ 1.19 │ 152.52 │
╘═════════════════════════╧═══════════╧═══════════╧══════════╧═══════════╧══════════╧═══════════╧═══════════╧════════════╛
```
#### Summary:
1. Recovery time for the decode worker itself is the largest and a decode worker failure has the largest impact (as expected)
2. Recovery time doesn't include time for the ready probe to return `ready` so even if the process is recovered early (as in the case of the Frontend) requests may fail until the pod is probed.
#### Redundant Workers (Over Provisoned) #### Redundant Workers (Over Provisoned)
...@@ -296,30 +304,6 @@ graph LR ...@@ -296,30 +304,6 @@ graph LR
style DecodePool stroke:#000,stroke-width:2px style DecodePool stroke:#000,stroke-width:2px
``` ```
#### Results:
```
Test Group: agg-tp-1-dp-2
╒═════════════════════════╤═══════════╤═══════════╤══════════╤═══════════╤══════════╤═══════════╤═══════════╤════════════╕
│ Failure │ Startup │ Success │ Failed │ Success │ Failed │ Latency │ Latency │ Recovery │
│ │ │ Before │ Before │ After │ After │ Before │ After │ │
╞═════════════════════════╪═══════════╪═══════════╪══════════╪═══════════╪══════════╪═══════════╪═══════════╪════════════╡
│ none │ 181.00 │ 1500.00 │ 0.00 │ N/A │ N/A │ 1.18 │ N/A │ N/A │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend │ 181.00 │ 121.00 │ 0.00 │ 1373.00 │ 6.00 │ 1.21 │ 1.17 │ 4.37 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend_pod │ 182.00 │ 122.00 │ 0.00 │ 1378.00 │ 0.00 │ 1.21 │ 1.17 │ 5.24 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker │ 169.00 │ 121.00 │ 0.00 │ 1374.00 │ 5.00 │ 1.20 │ 1.18 │ 153.09 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker_pod │ 181.00 │ 125.00 │ 0.00 │ 1369.00 │ 6.00 │ 1.21 │ 1.18 │ 152.72 │
├─────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ vllm_decode_engine_core │ 182.00 │ 120.00 │ 0.00 │ 1375.00 │ 5.00 │ 1.20 │ 1.18 │ 154.75 │
╘═════════════════════════╧═══════════╧═══════════╧══════════╧═══════════╧══════════╧═══════════╧═══════════╧════════════╛
```
#### Summary:
1. By immediately detecting a decode worker failure, Dynamo can limit 1. By immediately detecting a decode worker failure, Dynamo can limit
the failures and reroute requests to healthy workers with minimal the failures and reroute requests to healthy workers with minimal
impact. impact.
...@@ -367,34 +351,6 @@ graph LR ...@@ -367,34 +351,6 @@ graph LR
style DecodePool stroke:#000,stroke-width:2px style DecodePool stroke:#000,stroke-width:2px
``` ```
#### Results:
```
Test Group: disagg-tp-1-dp-1
╒══════════════════════════╤═══════════╤═══════════╤══════════╤═══════════╤══════════╤═══════════╤═══════════╤════════════╕
│ Failure │ Startup │ Success │ Failed │ Success │ Failed │ Latency │ Latency │ Recovery │
│ │ │ Before │ Before │ After │ After │ Before │ After │ │
╞══════════════════════════╪═══════════╪═══════════╪══════════╪═══════════╪══════════╪═══════════╪═══════════╪════════════╡
│ none │ 175.00 │ 1500.00 │ 0.00 │ N/A │ N/A │ 1.99 │ N/A │ N/A │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend │ 182.00 │ 100.00 │ 0.00 │ 817.00 │ 583.00 │ 1.91 │ 2.00 │ 4.28 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend_pod │ 181.00 │ 81.00 │ 0.00 │ 1024.00 │ 395.00 │ 2.31 │ 1.96 │ 5.53 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker │ 181.00 │ 82.00 │ 0.00 │ 560.00 │ 858.00 │ 2.26 │ 1.98 │ 155.79 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker_pod │ 181.00 │ 92.00 │ 0.00 │ 566.00 │ 842.00 │ 2.21 │ 1.83 │ 174.15 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ prefill_worker │ 182.00 │ 84.00 │ 0.00 │ 1346.00 │ 70.00 │ 2.22 │ 1.49 │ 153.53 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ prefill_worker_pod │ 161.00 │ 83.00 │ 0.00 │ 1362.00 │ 55.00 │ 2.21 │ 1.51 │ 154.18 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ vllm_decode_engine_core │ 167.00 │ 81.00 │ 0.00 │ 569.00 │ 850.00 │ 2.33 │ 2.12 │ 153.81 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ vllm_prefill_engine_core │ 182.00 │ 83.00 │ 0.00 │ 568.00 │ 849.00 │ 2.24 │ 2.00 │ 153.84 │
╘══════════════════════════╧═══════════╧═══════════╧══════════╧═══════════╧══════════╧═══════════╧═══════════╧════════════╛
```
#### Summary: #### Summary:
...@@ -456,34 +412,7 @@ graph LR ...@@ -456,34 +412,7 @@ graph LR
style DecodePool stroke:#000,stroke-width:2px style DecodePool stroke:#000,stroke-width:2px
``` ```
#### Results:
```
Test Group: disagg-tp-1-dp-2
╒══════════════════════════╤═══════════╤═══════════╤══════════╤═══════════╤══════════╤═══════════╤═══════════╤════════════╕
│ Failure │ Startup │ Success │ Failed │ Success │ Failed │ Latency │ Latency │ Recovery │
│ │ │ Before │ Before │ After │ After │ Before │ After │ │
╞══════════════════════════╪═══════════╪═══════════╪══════════╪═══════════╪══════════╪═══════════╪═══════════╪════════════╡
│ none │ 181.00 │ 1500.00 │ 0.00 │ N/A │ N/A │ 1.47 │ N/A │ N/A │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend │ 182.00 │ 100.00 │ 0.00 │ 1390.00 │ 10.00 │ 1.75 │ 1.43 │ 4.32 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ frontend_pod │ 182.00 │ 91.00 │ 0.00 │ 1409.00 │ 0.00 │ 1.78 │ 1.43 │ 5.48 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker │ 182.00 │ 94.00 │ 0.00 │ 1404.00 │ 2.00 │ 1.78 │ 1.58 │ 154.30 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ decode_worker_pod │ 181.00 │ 100.00 │ 0.00 │ 1394.00 │ 6.00 │ 1.75 │ 1.57 │ 153.00 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ prefill_worker │ 172.00 │ 90.00 │ 0.00 │ 1408.00 │ 2.00 │ 1.78 │ 1.44 │ 154.68 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ prefill_worker_pod │ 174.00 │ 100.00 │ 0.00 │ 1398.00 │ 2.00 │ 1.74 │ 1.41 │ 155.59 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ vllm_decode_engine_core │ 181.00 │ 91.00 │ 0.00 │ 1403.00 │ 6.00 │ 1.79 │ 1.56 │ 157.54 │
├──────────────────────────┼───────────┼───────────┼──────────┼───────────┼──────────┼───────────┼───────────┼────────────┤
│ vllm_prefill_engine_core │ 181.00 │ 94.00 │ 0.00 │ 1404.00 │ 2.00 │ 1.77 │ 1.43 │ 154.10 │
╘══════════════════════════╧═══════════╧═══════════╧══════════╧═══════════╧══════════╧═══════════╧═══════════╧════════════╛
```
#### Summary: #### Summary:
...@@ -512,7 +441,7 @@ Then run the development container mounting the workspace and your kube config. ...@@ -512,7 +441,7 @@ Then run the development container mounting the workspace and your kube config.
### Run the tests ### Run the tests
``` ```bash
pytest tests/fault_tolerance/deploy/test_deployment.py -s -v --namespace ${NAMESPACE} --image ${IMAGE} pytest tests/fault_tolerance/deploy/test_deployment.py -s -v --namespace ${NAMESPACE} --image ${IMAGE}
``` ```
......
tests/fault_tolerance/deploy/client.py
View file @ 02c822d6
...@@ -13,14 +13,15 @@ ...@@ -13,14 +13,15 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""AI-Perf client implementation for fault tolerance testing."""
import json import json
import logging import logging
import os import os
import random import subprocess
import time import time
from copy import deepcopy from pathlib import Path
from datetime import datetime from typing import Any, Dict, List, Optional, Tuple
from typing import Any, Dict
import requests import requests
...@@ -29,194 +30,499 @@ from tests.utils.managed_deployment import ManagedDeployment ...@@ -29,194 +30,499 @@ from tests.utils.managed_deployment import ManagedDeployment
LOG_FORMAT = "[TEST] %(asctime)s %(levelname)s %(name)s: %(message)s" LOG_FORMAT = "[TEST] %(asctime)s %(levelname)s %(name)s: %(message)s"
DATE_FORMAT = "%Y-%m-%dT%H:%M:%S" DATE_FORMAT = "%Y-%m-%dT%H:%M:%S"
payload = {
"model": "",
"messages": [
{
"role": "user",
"content": "",
}
],
"max_tokens": 0,
"temperature": 0.1,
# "seed": 10,
"ignore_eos": True,
"min_tokens": 0,
"stream": False,
}
# Configure logging # Configure logging
logging.basicConfig( logging.basicConfig(
level=logging.INFO, level=logging.INFO,
format=LOG_FORMAT, format=LOG_FORMAT,
datefmt=DATE_FORMAT, # ISO 8601 UTC format datefmt=DATE_FORMAT,
) )
def _get_random_prompt(length): def get_frontend_port(
word_list = [f"{i}" for i in range(10)] managed_deployment: ManagedDeployment,
return " ".join(random.choices(word_list, k=length)) client_index: int,
deployment_spec: Any,
pod_ports: Dict[str, Any],
logger: logging.Logger,
) -> Tuple[Optional[str], Optional[int], Optional[str]]:
"""
Select a frontend pod using round-robin and setup port forwarding.
Args:
managed_deployment: ManagedDeployment instance
client_index: Client index for round-robin selection
deployment_spec: Deployment specification with port info
pod_ports: Dictionary to track existing port forwards
- Key: pod name (str)
- Value: port forward object from managed_deployment.port_forward()
logger: Logger instance
def _single_request( Returns:
url, Tuple of (pod_name, local_port, pod_instance) or (None, None, None) if failed
pod, """
payload, pods = managed_deployment.get_pods(managed_deployment.frontend_service_name)
model,
logger, port = 0
retry_attempts=1, pod_name = None
input_token_length=100, selected_pod = None
output_token_length=100,
timeout=30, # Filter ready pods and cleanup stale port forwards
retry_delay=1, pods_ready = []
):
prompt = _get_random_prompt(input_token_length) for pod in pods[managed_deployment.frontend_service_name]:
payload_copy = deepcopy(payload) if pod.ready():
payload_copy["messages"][0]["content"] = prompt pods_ready.append(pod)
payload_copy["max_tokens"] = output_token_length else:
payload_copy["min_tokens"] = output_token_length # Cleanup port forwards for non-ready pods
payload_copy["model"] = model if pod.name in pod_ports:
response = None try:
end_time = None pod_ports[pod.name].stop()
start_time = time.time() except Exception as e:
results = [] logger.debug(f"Error stopping port forward for {pod.name}: {e}")
del pod_ports[pod.name]
while retry_attempts:
start_request_time = time.time() if not pods_ready:
response = None logger.error("No ready frontend pods found")
return None, None, None
# Round-robin selection based on client index
selected_pod = pods_ready[client_index % len(pods_ready)]
pod_name = selected_pod.name
# Setup or reuse port forward
if pod_name not in pod_ports:
# Get port from deployment_spec (default: 8000)
port_value = getattr(deployment_spec, "_port", 8000)
port_forward = managed_deployment.port_forward(selected_pod, port_value)
if port_forward:
pod_ports[pod_name] = port_forward
port = port_forward.local_port
else:
logger.error(f"Failed to create port forward for pod {pod_name}")
return None, None, None
else:
# Reuse existing port forward
port = pod_ports[pod_name].local_port
logger.debug(f"Selected pod {pod_name} with local port {port}")
return pod_name, port, selected_pod
def wait_for_model_availability(
url: str,
endpoint: str,
model: str,
logger: logging.Logger,
max_attempts: int = 15,
attempt_timeouts: Optional[List[float]] = None,
) -> bool:
"""
Wait for model to be available before running AI-Perf.
Args:
url: Base URL for the service
endpoint: API endpoint path
model: Model name to test
logger: Logger instance
max_attempts: Maximum number of attempts to check availability
attempt_timeouts: List of timeout values for each attempt
Returns:
True if model is available, False otherwise
"""
if attempt_timeouts is None:
# Default: Start with 60s timeout, then gradually decrease
attempt_timeouts = [60, 60, 45, 30, 30, 20, 20, 15, 15, 15, 10, 10, 10, 10, 10]
test_url = f"{url}{endpoint}"
for attempt in range(max_attempts):
try: try:
response = requests.post( test_payload = {
url, "model": model,
json=payload_copy, "messages": [{"role": "user", "content": "test"}],
timeout=timeout, "max_tokens": 1,
"stream": False,
}
timeout_val = attempt_timeouts[min(attempt, len(attempt_timeouts) - 1)]
logger.info(
f"Testing model availability at {test_url} (attempt {attempt+1}/{max_attempts}, timeout={timeout_val}s)"
) )
end_time = time.time() response = requests.post(test_url, json=test_payload, timeout=timeout_val)
content = None if response.status_code == 200:
logger.info(f"Model '{model}' is available and responding")
# Give a bit more time for stabilization
logger.info("Model ready, waiting 5s for stabilization...")
time.sleep(5)
return True
elif response.status_code == 404:
logger.warning(
f"Model '{model}' not found (404). Response: {response.text[:200]}"
)
elif response.status_code == 400:
logger.warning(f"Bad request (400). Response: {response.text[:200]}")
else:
logger.warning(
f"Unexpected status code {response.status_code}: {response.text[:200]}"
)
try: except requests.Timeout as e:
content = response.json() logger.warning(
except ValueError: f"Model availability test timed out (attempt {attempt+1}): {e}"
pass )
except Exception as e:
logger.warning(f"Model availability test failed (attempt {attempt+1}): {e}")
if attempt < max_attempts - 1:
wait_time = 10 if attempt < 5 else 5
logger.info(f"Waiting {wait_time}s before retry...")
time.sleep(wait_time)
logger.warning(
"Could not confirm model availability after all attempts, proceeding anyway..."
)
return False
def run_aiperf(
url: str,
endpoint: str,
model: str,
pod_name: str,
port: int,
requests_per_client: int,
input_token_length: int,
output_token_length: int,
output_dir: Path,
logger: logging.Logger,
max_retries: int = 1,
retry_delay: float = 1,
) -> bool:
"""
Execute AI-Perf with specified parameters.
Args:
url: Base URL (http://localhost:port)
endpoint: API endpoint path (e.g., "v1/chat/completions")
model: Model name
pod_name: Selected pod name for logging
port: Local port number
requests_per_client: Number of requests to send
input_token_length: Input token count
output_token_length: Output token count
output_dir: Directory for AI-Perf artifacts
logger: Logger instance
max_retries: Maximum number of retry attempts (default: 1)
retry_delay: Delay in seconds between retries (default: 1)
Returns:
True if successful, False otherwise
"""
# Validate required parameters
if not model or not url or not endpoint:
logger.error(
f"Missing required parameter: model={model!r}, url={url!r}, endpoint={endpoint!r}"
)
return False
# Build AI-Perf command
cmd = [
"aiperf",
"profile",
# Model configuration (required)
"--model",
model,
# Endpoint configuration
"--url",
url,
"--endpoint",
endpoint if endpoint.startswith("/") else f"/{endpoint}",
"--endpoint-type",
"chat", # Required: tells AI-Perf the API type
# Enable streaming for TTFT and ITL metrics
"--streaming",
# Request parameters
"--request-count",
str(requests_per_client), # Required: how many requests
"--concurrency",
"1", # Optional: we set to 1 for sequential
# Token configuration
"--synthetic-input-tokens-mean",
str(input_token_length),
"--synthetic-input-tokens-stddev",
"0", # Set to 0 for consistent token counts
"--output-tokens-mean",
str(output_token_length),
"--output-tokens-stddev",
"0", # Set to 0 for consistent token counts
# Skip warmup to avoid initial failures
"--warmup-request-count",
"0",
# Output configuration
"--artifact-dir",
str(output_dir),
"--random-seed",
"100", # For reproducible results
]
# Calculate timeout (same as legacy would for all requests)
timeout = max(requests_per_client * 2 + 60, 300) # At least 5 minutes
# Log execution
logger.info(f"Starting AI-Perf for Pod {pod_name} Local Port {port}")
logger.info(f"Using model name: {model}")
# Wait for model to be available
wait_for_model_availability(url, endpoint, model, logger)
logger.info(f"Command: {' '.join(cmd)}")
# Retry logic for fault tolerance - retry FULL request count until success
max_attempts = max_retries if max_retries > 0 else 1
success = False
all_results = []
for attempt in range(max_attempts):
logger.info(
f"AI-Perf attempt {attempt + 1}/{max_attempts} with {requests_per_client} requests"
)
# Update output directory for this attempt
attempt_dir = output_dir / f"attempt_{attempt}"
attempt_dir.mkdir(parents=True, exist_ok=True)
# Use the original command but update artifact directory
cmd_attempt = cmd.copy()
artifact_dir_idx = cmd_attempt.index("--artifact-dir") + 1
cmd_attempt[artifact_dir_idx] = str(attempt_dir)
try:
result = subprocess.run(
cmd_attempt,
capture_output=True,
text=True,
timeout=timeout,
stdin=subprocess.DEVNULL, # Prevent stdin reading which can cause process suspension
)
# Save logs for this attempt
with open(attempt_dir / "genai_perf.log", "w") as f:
f.write("=== STDOUT ===\n")
f.write(result.stdout)
f.write("\n\n=== STDERR ===\n")
f.write(result.stderr)
results.append( all_results.append(
{ {
"status": response.status_code, "attempt": attempt + 1,
"result": content, "returncode": result.returncode,
"request_elapsed_time": end_time - start_request_time, "stdout": result.stdout,
"url": url, "stderr": result.stderr,
"pod": pod,
} }
) )
if response.status_code != 200: if result.returncode == 0:
time.sleep(retry_delay) logger.info(
retry_attempts -= 1 f"Attempt {attempt + 1} succeeded with all {requests_per_client} requests"
continue )
log_summary_metrics(attempt_dir, logger, pod_name, port)
success = True
break # Success - we're done!
else: else:
logger.warning(
f"Attempt {attempt + 1} failed with return code {result.returncode}"
)
logger.debug(
f"Stderr: {result.stderr[:500] if result.stderr else 'No stderr'}"
)
except Exception as e:
logger.error(f"Error in attempt {attempt + 1}: {str(e)}")
all_results.append({"attempt": attempt + 1, "error": str(e)})
# Sleep before next attempt (if not the last attempt)
if not success and attempt < max_attempts - 1:
time.sleep(retry_delay)
if success:
logger.info(
f"AI-Perf successfully completed all {requests_per_client} requests for {pod_name}"
)
else:
logger.error(f"AI-Perf failed all {max_attempts} attempts for {pod_name}")
return success
def log_summary_metrics(
output_dir: Path, logger: logging.Logger, pod_name: str, port: int
) -> None:
"""
Log summary metrics from AI-Perf results.
Args:
output_dir: Directory containing AI-Perf artifacts
logger: Logger instance
pod_name: Pod name for logging
port: Port number for logging
"""
# Look for AI-Perf output file
profile_json = output_dir / "profile_export_aiperf.json"
if not profile_json.exists():
# Try alternative names
for name in ["profile_export.json", "profile_results.json"]:
alt_path = output_dir / name
if alt_path.exists():
profile_json = alt_path
break break
except (requests.RequestException, requests.Timeout) as e: if profile_json.exists():
results.append( try:
{ with open(profile_json) as f:
"status": str(e), metrics = json.load(f)
"result": None,
"request_elapsed_time": time.time() - start_request_time, # Extract key metrics from AI-Perf format
"url": url, records = metrics.get("records", {})
"pod": pod,
} # Request count from request_count record
request_count_record = records.get("request_count", {})
request_count = (
int(request_count_record.get("avg", 0)) if request_count_record else 0
) )
time.sleep(retry_delay)
retry_attempts -= 1
continue
return { # Check for errors
"time": datetime.now().strftime("%Y-%m-%dT%H:%M:%S"), error_summary = metrics.get("error_summary", [])
"results": results, error_count = len(error_summary)
"total_time": time.time() - start_time,
"url": url, # Latency metrics (in milliseconds)
"pod": pod, request_latency = records.get("request_latency", {})
} avg_latency = request_latency.get("avg", 0) / 1000.0 # Convert to seconds
p99_latency = request_latency.get("p99", 0) / 1000.0 # Convert to seconds
# Throughput metrics
request_throughput = records.get("request_throughput", {})
throughput = request_throughput.get("avg", 0)
# Log summary
logger.info(
f"Summary: Pod {pod_name} Port {port} "
f"Requests: {request_count} "
f"Errors: {error_count} "
f"Throughput: {throughput:.1f} req/s "
f"Avg Latency: {avg_latency:.3f}s "
f"P99 Latency: {p99_latency:.3f}s"
)
# Log success rate
if request_count > 0:
success_rate = (request_count - error_count) / request_count * 100
logger.info(f"Success rate: {success_rate:.1f}%")
# Also write summary to CSV file for aggregation
csv_path = output_dir / "profile_export_aiperf.csv"
if csv_path.exists():
logger.info(f"AI-Perf results saved to {csv_path}")
except Exception as e:
logger.warning(f"Failed to parse AI-Perf metrics: {e}")
def client( def client(
deployment_spec, deployment_spec,
namespace, namespace: str,
model, model: str,
log_dir, log_dir: str,
index, index: int,
requests_per_client, requests_per_client: int,
input_token_length, input_token_length: int,
output_token_length, output_token_length: int,
max_retries, max_retries: int,
max_request_rate, retry_delay: float = 1,
retry_delay=1,
): ):
"""
Generate load using AI-Perf for fault tolerance testing.
This function sets up port forwarding to a frontend pod and uses AI-Perf
to generate synthetic requests for performance testing and fault tolerance
evaluation.
Args:
deployment_spec: Deployment specification object
namespace: Kubernetes namespace
model: Model name
log_dir: Directory for output logs and AI-Perf artifacts
index: Client index used for round-robin pod selection
requests_per_client: Number of requests to generate
input_token_length: Number of input tokens per request
output_token_length: Number of output tokens per request
max_retries: Maximum retry attempts for AI-Perf execution
retry_delay: Delay in seconds between retry attempts
"""
logger = logging.getLogger(f"CLIENT: {index}") logger = logging.getLogger(f"CLIENT: {index}")
logging.getLogger("httpx").setLevel(logging.WARNING) logging.getLogger("httpx").setLevel(logging.WARNING)
managed_deployment = ManagedDeployment(log_dir, deployment_spec, namespace) managed_deployment = ManagedDeployment(log_dir, deployment_spec, namespace)
pod_ports: Dict[str, Any] = {} pod_ports: Dict[str, Any] = {}
min_elapsed_time = (1 / max_request_rate) if max_request_rate > 0 else 0.0
try: try:
os.makedirs(log_dir, exist_ok=True) os.makedirs(log_dir, exist_ok=True)
log_path = os.path.join(log_dir, f"client_{index}.log.txt") client_output_dir = Path(log_dir) / f"client_{index}"
with open(log_path, "w") as log: client_output_dir.mkdir(parents=True, exist_ok=True)
for i in range(requests_per_client):
pods = managed_deployment.get_pods( # Add a startup delay for early clients to give model time to load
managed_deployment.frontend_service_name if index < 5:
) wait_time = 30 - (index * 5) # 30, 25, 20, 15, 10 seconds
port = 0 logger.info(
pod_name = None f"Client {index} waiting {wait_time}s for model registration..."
)
pods_ready = [] time.sleep(wait_time)
for pod in pods[managed_deployment.frontend_service_name]: # Select frontend pod and setup port forwarding
if pod.ready(): pod_name, port, selected_pod = get_frontend_port(
pods_ready.append(pod) managed_deployment=managed_deployment,
else: client_index=index,
if pod.name in pod_ports: deployment_spec=deployment_spec,
pod_ports[pod.name].stop() pod_ports=pod_ports,
del pod_ports[pod.name] logger=logger,
)
if pods_ready:
pod = pods_ready[i % len(pods_ready)] if not pod_name or not port:
if pod.name not in pod_ports: logger.error("Failed to select pod or setup port forwarding")
port_forward = managed_deployment.port_forward( return
pod, deployment_spec.port
)
if port_forward:
pod_ports[pod.name] = port_forward
if pod.name in pod_ports:
port = pod_ports[pod.name].local_port
pod_name = pod.name
url = f"http://localhost:{port}/{deployment_spec.endpoint}"
result = _single_request(
url,
pod_name,
payload,
model,
logger,
max_retries,
input_token_length=input_token_length,
output_token_length=output_token_length,
retry_delay=retry_delay,
)
logger.info(
f"Request: {i} Pod {pod_name} Local Port {port} Status: {result['results'][-1]['status']} Latency: {result['results'][-1]['request_elapsed_time']}"
)
log.write(json.dumps(result) + "\n") url = f"http://localhost:{port}"
log.flush()
if result["total_time"] < min_elapsed_time: # Get endpoint from deployment_spec (default: /v1/chat/completions)
time.sleep(min_elapsed_time - result["total_time"]) endpoint = getattr(deployment_spec, "_endpoint", "/v1/chat/completions")
success = run_aiperf(
url=url,
endpoint=endpoint,
model=model,
pod_name=pod_name,
port=port,
requests_per_client=requests_per_client,
input_token_length=input_token_length,
output_token_length=output_token_length,
output_dir=client_output_dir,
logger=logger,
max_retries=max_retries,
retry_delay=retry_delay,
)
if not success:
logger.error("AI-Perf execution failed")
except Exception as e: except Exception as e:
logger.error(str(e)) logger.error(f"Client error: {str(e)}")
finally:
for pf_name, port_forward in pod_ports.items():
try:
port_forward.stop()
logger.debug(f"Stopped port forward for {pf_name}")
except Exception as e:
logger.debug(f"Error stopping port forward for {pf_name}: {e}")
logger.info("Exiting") logger.info("Exiting")
tests/fault_tolerance/deploy/parse_results.py
View file @ 02c822d6
...@@ -13,441 +13,515 @@ ...@@ -13,441 +13,515 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""Parser for AI-Perf results in fault tolerance tests."""
import argparse import argparse
import json import json
import logging
import os import os
import re import re
from datetime import datetime from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
import pandas as pd import numpy as np
from tabulate import tabulate from tabulate import tabulate
def parse_test_log(file_path): def parse_test_log(
file_path: str,
) -> Tuple[Optional[float], Optional[List[str]]]:
"""
Parse test log for startup time and failure info.
Args:
file_path: Path to test.log.txt
Returns:
Tuple of (startup_time_seconds, failure_info)
"""
start_time = None start_time = None
ready_time = None ready_time = None
fault_time = None failure_info: Optional[List[str]] = None
start_cmd: Optional[List[str]] = None
if not os.path.isfile(file_path): if not os.path.isfile(file_path):
return None, None, None return None, None
with open(file_path, "r") as f: with open(file_path, "r") as f:
for line in f: for line in f:
line = line.strip() # Extract timestamp using regex to handle different log formats
if "Starting Deployment fault-tolerance-test with spec" in line: timestamp_match = re.search(r"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})", line)
start_time = datetime.fromisoformat(
line.split(" ")[1].replace("T", " ") # Look for deployment start
) if "Starting Deployment" in line and timestamp_match:
start_cmd = [] timestamp = timestamp_match.group(1)
elif "Deployment fault-tolerance-test is ready" in line: start_time = datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S")
ready_time = datetime.fromisoformat(
line.split(" ")[1].replace("T", " ") # Look for deployment ready
) if "Deployment fault-tolerance-test is ready" in line and timestamp_match:
elif "Injecting failure for:" in line: timestamp = timestamp_match.group(1)
fault_time = datetime.fromisoformat( ready_time = datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S")
line.split(" ")[1].replace("T", " ")
) # Look for fault injection
startup_time = ( if "Injecting failure for:" in line:
(ready_time - start_time).total_seconds() if start_time and ready_time else None # Extract failure details
) match = re.search(r"Failure\((.*?)\)", line)
return startup_time, fault_time, start_cmd if match:
failure_str = match.group(1)
parts = failure_str.split(", ")
def parse_client_logs(test_dir, expected_length=100): failure_dict = {}
all_logs = [] for part in parts:
for file in os.listdir(test_dir): key_val = part.split("=")
if file.startswith("client_") and file.endswith(".log.txt"): if len(key_val) == 2:
with open(os.path.join(test_dir, file), "r") as f: failure_dict[key_val[0]] = key_val[1]
request_number = 0
for line in f: # Build command list from failure info
request_number += 1 if failure_dict:
data = json.loads(line.strip()) failure_info = [
for result in data["results"]: failure_dict.get("pod_name", "unknown").strip("'\""),
log_entry = { failure_dict.get("command", "unknown").strip("'\""),
"time": datetime.fromisoformat( ]
data["time"].replace("T", " ")
), # Calculate startup time in seconds
"status": result["status"], startup_time = None
"request_elapsed_time": result["request_elapsed_time"], if start_time and ready_time:
"request_number": request_number - 1, startup_time = (ready_time - start_time).total_seconds()
"client": file.split("_")[1].split(".")[0],
} return startup_time, failure_info
if (
"result" in result
and result["result"] def calculate_recovery_time(
and "choices" in result["result"] failure_info: Optional[List[str]],
and result["result"]["choices"] process_logs_dir: str,
): ) -> Optional[float]:
log_entry["success"] = True """
if "content" in result["result"]["choices"][0]["message"]: Calculate recovery time by comparing last timestamp in .previous.log with first in current log.
content = result["result"]["choices"][0]["message"][ This avoids timezone issues between test.log.txt and container logs.
"content"
] Args:
elif ( failure_info: List with [pod_name, command] from fault injection
"reasoning_content" process_logs_dir: Directory containing process log files
in result["result"]["choices"][0]["message"]
): Returns:
content = result["result"]["choices"][0]["message"][ Recovery time in seconds or None if not found
"reasoning_content" """
] if not failure_info:
return None
if not content or len(content) < expected_length:
log_entry["success"] = False
else:
log_entry["success"] = False
all_logs.append(log_entry)
if len(all_logs):
df = pd.DataFrame(all_logs)
df.sort_values("time", inplace=True)
return df
return None # Determine component type from failure info (strip any quotes)
component_type = failure_info[0].strip("'\"") # e.g., "Frontend" or "decode"
component_dir = os.path.join(process_logs_dir, component_type)
if not os.path.exists(component_dir):
return None
def calculate_metrics(df, fault_time, sla=None): last_timestamp_before = None
if fault_time: first_timestamp_after = None
before_fault = df[df["time"] <= fault_time]
after_fault = df[df["time"] > fault_time] # Find the last timestamp from .previous.log (container before restart)
else: for log_file in os.listdir(component_dir):
before_fault = df if log_file.endswith(".previous.log"):
after_fault = None log_path = os.path.join(component_dir, log_file)
try:
# Existing latency metrics (only successful requests) with open(log_path, "r") as f:
successful_before = before_fault[before_fault["success"]] # Read last few lines to find last valid timestamp
avg_before = successful_before["request_elapsed_time"].mean() lines = f.readlines()
std_before = successful_before["request_elapsed_time"].std() for line in reversed(lines[-10:]): # Check last 10 lines
success_before_count = before_fault["success"].sum() if '"time":"' in line:
failure_before_count = len(before_fault) - success_before_count try:
log_entry = json.loads(line)
avg_after, std_after, success_after_count, failure_after_count = ( timestamp_str = log_entry.get("time", "")[:19]
None, last_timestamp_before = datetime.strptime(
None, timestamp_str, "%Y-%m-%dT%H:%M:%S"
None, )
None, break
) except (json.JSONDecodeError, ValueError):
if after_fault is not None and not after_fault.empty: continue
successful_after = after_fault[after_fault["success"]] except IOError as e:
avg_after = successful_after["request_elapsed_time"].mean() logging.debug(f"Could not read {log_file}: {e}")
std_after = successful_after["request_elapsed_time"].std()
success_after_count = after_fault["success"].sum() # Find the first timestamp from current container log
failure_after_count = len(after_fault) - success_after_count for log_file in os.listdir(component_dir):
if log_file.endswith(".log") and not log_file.endswith(
if sla: (".previous.log", ".metrics.log")
# SLA violations (only successful requests exceeding the SLA) ):
violations_before = (successful_before["request_elapsed_time"] > sla).sum() log_path = os.path.join(component_dir, log_file)
violations_after = ( try:
(successful_after["request_elapsed_time"] > sla).sum() with open(log_path, "r") as f:
if after_fault is not None and not after_fault.empty first_line = f.readline()
else None if first_line and '"time":"' in first_line:
) log_entry = json.loads(first_line)
else: timestamp_str = log_entry.get("time", "")[:19]
violations_before = None first_timestamp_after = datetime.strptime(
violations_after = None timestamp_str, "%Y-%m-%dT%H:%M:%S"
)
return ( except (json.JSONDecodeError, ValueError, IOError) as e:
success_before_count, logging.debug(f"Could not parse timestamp from {log_file}: {e}")
failure_before_count,
success_after_count, # Calculate recovery time from container timestamps (both in UTC)
failure_after_count, if last_timestamp_before and first_timestamp_after:
avg_before, recovery_time = (first_timestamp_after - last_timestamp_before).total_seconds()
std_before, # Sanity check - recovery should be seconds/minutes, not hours
avg_after, if recovery_time > 3600: # More than 1 hour is likely wrong
std_after, logging.warning(
violations_before, f"Recovery time {recovery_time}s seems too large, possible timezone issue"
violations_after, )
) return recovery_time
return None
def parse_process_log(log_dir, process_name): def parse_aiperf_client_results(log_dir: str) -> Dict[str, Any]:
process_ready_pattern = { """
"Frontend": re.compile(r"added model"), Parse AI-Perf results from all client directories.
"VllmDecodeWorker": re.compile(
r"VllmWorker for (?P<model_name>.*?) has been initialized" Args:
), log_dir: Directory containing client result directories
"VllmPrefillWorker": re.compile(
r"VllmWorker for (?P<model_name>.*?) has been initialized" Returns:
), Dictionary with aggregated metrics and client count
"""
all_metrics: Dict[str, Any] = {
"total_requests": 0,
"successful_requests": 0,
"failed_requests": 0,
"latencies": [],
"ttft": [], # Time to First Token
"itl": [], # Inter-Token Latency
"throughputs": [],
"p50_latencies": [],
"p90_latencies": [],
"p99_latencies": [],
"num_clients": 0,
} }
if not os.path.isdir(log_dir):
return {}
ready_times: Dict[str, List[Tuple[datetime, str, float]]] = {}
for entry in os.listdir(log_dir):
if entry.endswith(".log") and "metrics" not in entry:
replica_number = entry.split(".")[0]
if replica_number not in ready_times:
ready_times[replica_number] = []
process_start_time = None
with open(os.path.join(log_dir, entry), "r") as f:
for line in f:
line = line.strip()
if not line:
continue
# Try to parse as JSONL first
try:
json_data = json.loads(line)
# Extract timestamp and message from JSON format
if "time" in json_data:
timestamp = datetime.fromisoformat(
json_data["time"].replace("Z", "")
)
log_message = json_data.get("message", "")
else:
continue
except (json.JSONDecodeError, ValueError, KeyError):
# Fall back to readable format parsing
clean_line = re.sub(
r"\x1b\[.*?m", "", line
) # Remove ANSI codes
if not clean_line:
continue
parts = clean_line.split()
if len(parts) < 2:
continue
try:
# Parse timestamp (remove 'Z' for naive datetime)
timestamp = datetime.fromisoformat(
parts[0].replace("Z", "")
)
except ValueError:
continue
log_message = " ".join(parts[1:])
if not process_start_time:
process_start_time = timestamp
relative_time = (timestamp - process_start_time).total_seconds()
# Check for process start lines
if process_name in process_ready_pattern:
if process_ready_pattern[process_name].search(log_message):
if "previous" in entry:
location = 0
else:
location = -1
ready_times[replica_number].insert(
location, (timestamp, log_message, relative_time)
)
return ready_times
def calculate_recovery_time(test_dir, failure_type, fault_time):
if not fault_time:
return None
processes = [ # Iterate over actual client directories
"Frontend", for item in sorted(os.listdir(log_dir)):
"VllmDecodeWorker", if not item.startswith("client_") or not os.path.isdir(
"VllmPrefillWorker", os.path.join(log_dir, item)
] ):
continue
process_start = {} client_dir = Path(log_dir) / item
start_time = None all_metrics["num_clients"] += 1
for process in processes: # Look for AI-Perf results in attempt directories
starts = parse_process_log(os.path.join(test_dir, process), process) profile_json = None
if starts:
process_start[process] = starts
last_recovery_time = 0
for process, replicas in process_start.items():
for replica, container_starts in replicas.items():
for starts in container_starts:
start_time = starts[0]
recovery_time = (start_time - fault_time).total_seconds()
if recovery_time > last_recovery_time:
last_recovery_time = recovery_time
if last_recovery_time == 0:
return None
return last_recovery_time
# Check for attempt directories (attempt_0, attempt_1, etc.)
for attempt_dir in sorted(client_dir.glob("attempt_*")):
json_path = attempt_dir / "profile_export_aiperf.json"
if json_path.exists():
profile_json = json_path
break # Use the first successful attempt
def process_test_directory(test_dir, sla): if not profile_json:
if "test_fault_scenario" not in test_dir: logging.warning(f"No AI-Perf results found for {item} in {client_dir}")
return {} else:
test_name = test_dir.split("test_fault_scenario[", 1)[1].rstrip("]") try:
failure_type = test_name.split("-")[-1] with open(profile_json) as f:
test_prefix = "-".join(test_name.split("-")[:-1]) client_metrics = json.load(f)
# AI-Perf format has "records" dictionary at the top level
records = client_metrics.get("records", {})
# Extract request count (this is the total requests made)
request_count_record = records.get("request_count", {})
request_count = (
int(request_count_record.get("avg", 0))
if request_count_record
else 0
)
startup_time, fault_time, start_cmd = parse_test_log( # Check for errors in error_summary
os.path.join(test_dir, "test.log.txt") error_summary = client_metrics.get("error_summary", [])
) error_count = len(error_summary)
df = parse_client_logs(test_dir)
# Check if test was cancelled
was_cancelled = client_metrics.get("was_cancelled", False)
if was_cancelled:
error_count = request_count # Mark all as failed if cancelled
all_metrics["total_requests"] += request_count
all_metrics["successful_requests"] += request_count - error_count
all_metrics["failed_requests"] += error_count
# Extract latency from request_latency record
request_latency = records.get("request_latency", {})
if request_latency:
# Convert milliseconds to seconds for consistency
if "avg" in request_latency:
all_metrics["latencies"].append(request_latency["avg"] / 1000.0)
if "p50" in request_latency:
all_metrics["p50_latencies"].append(
request_latency["p50"] / 1000.0
)
if "p90" in request_latency:
all_metrics["p90_latencies"].append(
request_latency["p90"] / 1000.0
)
if "p99" in request_latency:
all_metrics["p99_latencies"].append(
request_latency["p99"] / 1000.0
)
# Time to first token (if available in records)
ttft = records.get("time_to_first_token", {}) or records.get("ttft", {})
if ttft and "avg" in ttft:
all_metrics["ttft"].append(ttft["avg"] / 1000.0) # Convert ms to s
# Inter-token latency (if available in records)
itl = records.get("inter_token_latency", {}) or records.get("itl", {})
if itl and "avg" in itl:
all_metrics["itl"].append(itl["avg"] / 1000.0) # Convert ms to s
# Throughput from request_throughput record
request_throughput = records.get("request_throughput", {})
req_throughput = request_throughput.get("avg", 0)
if req_throughput:
all_metrics["throughputs"].append(req_throughput)
except Exception as e:
logging.error(f"Error parsing {item} results: {e}")
return all_metrics
def print_summary_table(
log_dir: str,
num_clients: int,
startup_time: Optional[float],
recovery_time: Optional[float],
metrics: Dict[str, Any],
tablefmt: str = "grid",
sla: Optional[float] = None,
) -> None:
"""
Print formatted summary table with AI-Perf metrics.
Args:
log_dir: Test directory path
num_clients: Number of client processes
startup_time: Time to start deployment (seconds)
recovery_time: Time to recover from fault (seconds)
metrics: Aggregated metrics from AI-Perf
tablefmt: Table format for output
sla: Service level agreement for latency (optional)
"""
headers = ["Metric", "Value"]
rows = []
# Test info
rows.append(["Test Directory", log_dir])
rows.append(["Number of Clients", str(num_clients)])
rows.append(["", ""])
# Deployment metrics
rows.append(["=== Deployment Metrics ===", ""])
if startup_time:
rows.append(["Startup Time", f"{startup_time:.2f} sec"])
else:
rows.append(["Startup Time", "N/A"])
if df is None or df.empty: if recovery_time:
return None rows.append(["Recovery Time", f"{recovery_time:.2f} sec"])
( else:
success_before, rows.append(["Recovery Time", "N/A"])
failure_before, rows.append(["", ""])
success_after,
failure_after, # Request metrics
avg_before, rows.append(["=== Request Metrics ===", ""])
std_before, rows.append(["Total Requests", metrics["total_requests"]])
avg_after, rows.append(["Successful Requests", metrics["successful_requests"]])
std_after, rows.append(["Failed Requests", metrics["failed_requests"]])
violations_before,
violations_after, if metrics["total_requests"] > 0:
) = calculate_metrics(df, fault_time, sla) success_rate = (
metrics["successful_requests"] / metrics["total_requests"]
recovery_time = calculate_recovery_time(test_dir, failure_type, fault_time) ) * 100
rows.append(["Success Rate", f"{success_rate:.2f}%"])
rows.append(["", ""])
# Latency metrics
rows.append(["=== Latency Metrics (seconds) ===", ""])
if metrics["latencies"]:
mean_latency = np.mean(metrics["latencies"])
rows.append(["Mean Latency", f"{mean_latency:.3f}"])
# Check SLA if provided
if sla is not None:
sla_status = "✓ PASS" if mean_latency <= sla else "✗ FAIL"
rows.append(["SLA Status", f"{sla_status} (target: {sla:.3f}s)"])
if metrics["p50_latencies"]:
rows.append(["P50 Latency", f"{np.mean(metrics['p50_latencies']):.3f}"])
if metrics["p90_latencies"]:
rows.append(["P90 Latency", f"{np.mean(metrics['p90_latencies']):.3f}"])
if metrics["p99_latencies"]:
rows.append(["P99 Latency", f"{np.mean(metrics['p99_latencies']):.3f}"])
rows.append(["", ""])
# Token generation metrics
rows.append(["=== Token Generation Metrics ===", ""])
if metrics["ttft"]:
rows.append(
["Time to First Token (mean)", f"{np.mean(metrics['ttft']):.3f} sec"]
)
if metrics["itl"]:
rows.append(
["Inter-Token Latency (mean)", f"{np.mean(metrics['itl']):.4f} sec"]
)
rows.append(["", ""])
# Throughput metrics
rows.append(["=== Throughput Metrics ===", ""])
if metrics["throughputs"]:
total_throughput = sum(metrics["throughputs"])
rows.append(["Total Throughput", f"{total_throughput:.2f} req/s"])
rows.append(
["Avg Client Throughput", f"{np.mean(metrics['throughputs']):.2f} req/s"]
)
# Print table
print("\n" + "=" * 60)
print("FAULT TOLERANCE TEST SUMMARY - AI-PERF")
print("=" * 60)
print(tabulate(rows, headers=headers, tablefmt=tablefmt))
print("=" * 60 + "\n")
def process_single_test(
log_dir: str, tablefmt: str = "grid", sla: Optional[float] = None
) -> Dict[str, Any]:
"""
Process a single test log directory.
Args:
log_dir: Directory containing test results
tablefmt: Table format for output
sla: Service level agreement for latency (optional)
Returns:
Dictionary with test results
"""
# Parse test configuration
test_log = os.path.join(log_dir, "test.log.txt")
startup_time, failure_info = parse_test_log(test_log)
# Calculate recovery time only if fault was injected
recovery_time = None
if failure_info:
recovery_time = calculate_recovery_time(failure_info, log_dir)
# Parse AI-Perf results (also counts clients)
metrics = parse_aiperf_client_results(log_dir)
# Extract client count from metrics
num_clients = metrics.get("num_clients", 0)
# Print summary
print_summary_table(
log_dir, num_clients, startup_time, recovery_time, metrics, tablefmt, sla
)
return { return {
"test": test_prefix, "log_dir": log_dir,
"cmd": start_cmd, "num_clients": num_clients,
"failure": failure_type, "startup_time": startup_time,
"start_time": startup_time,
"success_before_requests": success_before,
"failed_before_requests": failure_before,
"success_after_requests": success_after,
"failed_after_requests": failure_after,
"avg_latency_before": avg_before,
"std_latency_before": std_before,
"avg_latency_after": avg_after,
"std_latency_after": std_after,
"violations_before": violations_before,
"violations_after": violations_after,
"recovery_time": recovery_time, "recovery_time": recovery_time,
"metrics": metrics,
} }
def main(logs_dir, tablefmt, log_paths=None, sla=None): def main(
results = [] logs_dir: Optional[str] = None,
log_paths: Optional[List[str]] = None,
tablefmt: str = "grid",
sla: Optional[float] = None,
):
"""
Main parser entry point with support for multiple log paths.
Args:
logs_dir: Base directory for logs (optional)
log_paths: List of log directories to process
tablefmt: Table format for output
sla: Service level agreement for latency (optional)
Returns:
Combined results from all processed tests
"""
# Handle different input formats
if log_paths: if log_paths:
# Process multiple log paths
all_results = []
for log_path in log_paths: for log_path in log_paths:
result = process_test_directory(log_path, sla) if logs_dir:
if result: full_path = os.path.join(logs_dir, log_path)
results.append(result) else:
elif logs_dir: full_path = log_path
for entry in os.listdir(logs_dir):
if entry.startswith("test_fault_scenario[") and os.path.isdir( if os.path.isdir(full_path):
os.path.join(logs_dir, entry) print(f"\nProcessing: {full_path}")
): results = process_single_test(full_path, tablefmt, sla)
result = process_test_directory(os.path.join(logs_dir, entry), sla) all_results.append(results)
if result:
results.append(result)
# Group results by test prefix
grouped: dict[str, list[dict[str, Any]]] = {}
commands = {}
for res in results:
test_prefix = res["test"]
if test_prefix not in grouped:
grouped[test_prefix] = []
commands[test_prefix] = res["cmd"]
grouped[test_prefix].append(res)
order = [
"none",
"frontend",
"frontend_pod",
"decode_worker",
"decode_worker_pod",
"prefill_worker",
"prefill_worker_pod",
"vllm_decode_engine_core",
"vllm_prefill_engine_core",
]
# Print grouped tables
for test_prefix, group in grouped.items():
new_group = []
for failure in order:
for res in group:
if failure == res["failure"]:
new_group.append(res)
group = new_group
if sla:
headers = [
"Failure",
"Startup",
"Success\nBefore",
"Failed\nBefore",
"Success\nAfter",
"Failed\nAfter",
"Latency\nBefore",
"Latency\nAfter",
"Violations\nBefore",
"Violations\nAfter",
"Recovery",
]
else:
headers = [
"Failure",
"Startup",
"Success\nBefore",
"Failed\nBefore",
"Success\nAfter",
"Failed\nAfter",
"Latency\nBefore",
"Latency\nAfter",
"Recovery",
]
rows = []
for res in group:
if sla:
row = [
res["failure"],
res["start_time"], # if res["start_time"] is not None else "N/A",
res["success_before_requests"],
res["failed_before_requests"],
res["success_after_requests"],
res["failed_after_requests"],
res["avg_latency_before"],
res["avg_latency_after"],
res["violations_before"],
res["violations_after"],
res["recovery_time"],
]
else: else:
row = [ print(f"Warning: {full_path} is not a valid directory, skipping...")
res["failure"],
res["start_time"], # if res["start_time"] is not None else "N/A", # If multiple tests, also print combined summary
res["success_before_requests"], if len(all_results) > 1:
res["failed_before_requests"], print("\n" + "=" * 60)
res["success_after_requests"], print("COMBINED TEST SUMMARY")
res["failed_after_requests"], print("=" * 60)
res["avg_latency_before"],
res["avg_latency_after"], total_requests = sum(r["metrics"]["total_requests"] for r in all_results)
res["recovery_time"], total_successful = sum(
] r["metrics"]["successful_requests"] for r in all_results
rows.append(row)
print(f"\nTest Group: {test_prefix}")
# print(f"\nTest Command: {commands[test_prefix]}")
print(
tabulate(
rows,
headers,
tablefmt=tablefmt,
floatfmt=".2f",
missingval="N/A",
numalign="right",
stralign="center",
) )
) total_failed = sum(r["metrics"]["failed_requests"] for r in all_results)
print("\n" + "=" * 80)
print(f"Total Tests: {len(all_results)}")
print(f"Total Requests: {total_requests}")
print(f"Total Successful: {total_successful}")
print(f"Total Failed: {total_failed}")
if total_requests > 0:
print(
f"Overall Success Rate: {(total_successful/total_requests)*100:.2f}%"
)
print("=" * 60 + "\n")
return all_results
elif logs_dir:
# Process single directory
return process_single_test(logs_dir, tablefmt, sla)
else:
print("Error: Must provide either logs_dir or log_paths")
return None
if __name__ == "__main__": if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Parse test results") # Configure logging
parser.add_argument("--log-dir", default=".", help="Path to the logs directory") logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
parser = argparse.ArgumentParser(description="Parse fault tolerance test results")
parser.add_argument( parser.add_argument(
"--format", choices=["fancy", "markdown"], default="fancy", help="Table format" "log_dir", type=str, help="Directory containing test logs and results"
) )
parser.add_argument("--sla", type=float, default=None)
args = parser.parse_args() args = parser.parse_args()
# Map format choices to tabulate formats if not os.path.isdir(args.log_dir):
tablefmt = ( logging.error(f"{args.log_dir} is not a valid directory")
"fancy_grid" if args.format == "fancy" else "pipe" exit(1)
) # Using pipe for markdown compatibility
main(args.log_dir, tablefmt, args.sla) main(args.log_dir)
tests/fault_tolerance/deploy/scenarios.py
View file @ 02c822d6
...@@ -13,11 +13,13 @@ ...@@ -13,11 +13,13 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import re
from dataclasses import dataclass from dataclasses import dataclass
from typing import Optional from typing import Dict, Optional, Pattern
from tests.utils.managed_deployment import DeploymentSpec from tests.utils.managed_deployment import DeploymentSpec
# Worker name mapping for different backends
WORKER_MAP = { WORKER_MAP = {
"vllm": { "vllm": {
"decode": "VllmDecodeWorker", "decode": "VllmDecodeWorker",
...@@ -29,6 +31,51 @@ WORKER_MAP = { ...@@ -29,6 +31,51 @@ WORKER_MAP = {
}, },
} }
# Process ready patterns for recovery detection
WORKER_READY_PATTERNS: Dict[str, Pattern] = {
# Frontend
"Frontend": re.compile(r"added model"),
# vLLM workers
"VllmDecodeWorker": re.compile(
r"VllmWorker for (?P<model_name>.*?) has been initialized"
),
"VllmPrefillWorker": re.compile(
r"VllmWorker for (?P<model_name>.*?) has been initialized"
),
# SGLang workers - look for their specific initialization messages
"decode": re.compile(
r"Model registration succeeded|Decode worker handler initialized|Worker handler initialized"
),
"prefill": re.compile(
r"Model registration succeeded|Prefill worker handler initialized|Worker handler initialized"
),
}
def get_all_worker_types() -> list[str]:
"""Get all worker type names for both vLLM and SGLang."""
worker_types = ["Frontend"]
for backend in WORKER_MAP.values():
worker_types.extend(backend.values())
# Remove duplicates while preserving order
seen = set()
result = []
for x in worker_types:
if x not in seen:
seen.add(x)
result.append(x)
return result
def get_worker_ready_pattern(worker_name: str) -> Optional[Pattern]:
"""Get the ready pattern for a specific worker type."""
return WORKER_READY_PATTERNS.get(worker_name)
def get_backend_workers(backend: str) -> Dict[str, str]:
"""Get worker mapping for a specific backend."""
return WORKER_MAP.get(backend, {})
@dataclass @dataclass
class Load: class Load:
...@@ -36,8 +83,7 @@ class Load: ...@@ -36,8 +83,7 @@ class Load:
requests_per_client: int = 150 requests_per_client: int = 150
input_token_length: int = 100 input_token_length: int = 100
output_token_length: int = 100 output_token_length: int = 100
max_retries: int = 1 max_retries: int = 3 # Increased for fault tolerance
max_request_rate: float = 1
sla: Optional[float] = None sla: Optional[float] = None
......
tests/fault_tolerance/deploy/test_deployment.py
View file @ 02c822d6
...@@ -31,7 +31,7 @@ def _clients( ...@@ -31,7 +31,7 @@ def _clients(
input_token_length, input_token_length,
output_token_length, output_token_length,
max_retries, max_retries,
max_request_rate, retry_delay=5, # Default 5 seconds between retries
): ):
procs = [] procs = []
ctx = multiprocessing.get_context("spawn") ctx = multiprocessing.get_context("spawn")
...@@ -49,7 +49,7 @@ def _clients( ...@@ -49,7 +49,7 @@ def _clients(
input_token_length, input_token_length,
output_token_length, output_token_length,
max_retries, max_retries,
max_request_rate, retry_delay,
), ),
) )
) )
...@@ -178,6 +178,5 @@ async def test_fault_scenario( ...@@ -178,6 +178,5 @@ async def test_fault_scenario(
scenario.load.input_token_length, scenario.load.input_token_length,
scenario.load.output_token_length, scenario.load.output_token_length,
scenario.load.max_retries, scenario.load.max_retries,
scenario.load.max_request_rate,
): ):
_inject_failures(scenario.failures, logger, deployment) _inject_failures(scenario.failures, logger, deployment)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment