Skip to content

GitLab

Unverified Commit ede0da16 authored by Pavithra Vijayakrishnan's avatar Pavithra Vijayakrishnan Committed by GitHub
Browse files

test: Update the test guideline (#3377) 


Signed-off-by: default avatarPavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com>
Co-authored-by: default avatarAlec <35311602+alec-flowers@users.noreply.github.com>
parent 30610e73
Hide whitespace changes
Inline Side-by-side
Showing with 193 additions and 84 deletions
tests/README.md
View file @ ede0da16
# Dynamo Testing Framework
# Dynamo Testing Guidelines
## Overview
This document provides instructions for organizing, marking, and running tests in the Dynamo project. Follow these guidelines to ensure consistency and maintainability across the test suite.
This document outlines the testing framework for the Dynamo runtime system, including test discovery, organization, and best practices.
---
## Directory Structure
## Test Organization: Where to Store Tests
```bash
tests/
├── serve/ # E2E tests
│ ├── conftest.py # test fixtures as needed for specific test area
├── conftest.py # Shared fixtures and configuration
├── utils # Common utils accross tests
└── README.md # This file
### Directory Structure
```
dynamo/
├── lib/
│ ├── runtime/
│ │ ├── src/
│ │ │ └── lib.rs # Rust code + unit tests inside
│ │ └── tests/ # Rust integration tests for runtime
│ ├── llm/
│ │ └── src/
│ │ └── lib.rs # Rust code + unit tests inside
│ │ └── tests/ # Rust integration tests for llm
│ └── ...
├── components/
│ ├── src/dynamo/
│ | └── planner/
│ │ │ └── tests/ # Python unit/integration tests for planner
│ | └── router/
│ │ │ └── tests/
│ | └── ...
│ ├── backend/
│ │ └── vllm
│ │ │ └── tests/ # Python unit/integration tests for backend
│ │ └── trtllm
│ │ │ └── tests/
│ │ └── trtllm
│ │ │ └── tests/
│ └── ...
├── tests/ # End-to-end and cross-component tests
│ ├── serve/
│ ├── kvbm/
│ ├── benchmark/
│ ├── fault_tolerance/
│ └── ...
```
- Place **unit/integration tests** for a component in its `tests/` subfolder.
- Place **end-to-end (E2E) tests** and cross-component tests in `dynamo/tests/`.
- Name test files as `test_<component>_<flow>.py` for clarity.
### Test Types and Locations
| Type | Description | Location |
|--------------|------------------------------------------|-----------------------|
| Unit | Single function/class, isolated | `<component>/tests/` |
| Integration | Interactions between modules/services | `<component>/tests/` |
| End-to-End | User workflows, CLI, API | `tests/serve/`, etc. |
| Benchmark | Performance/load | `tests/benchmark/` |
| Stress | Chaos, long-run, resource limits | `tests/fault_tolerance/` |
---
## Test Marking: How to Mark Tests
Markers are required for all tests. They are used for test selection in CI and local runs.
### Marker Requirements
- Every test must have at least one **Lifecycle** marker, and **test type** and **Hardware** markers.
- **component** markers are required as applicable.
### Marker Table
| Category | Marker(s) | Description |
|-------------------------|--------------------------|------------------------------------|
| Lifecycle [required] | pre_merge, post_merge, nightly, weekly, release | When the test should run |
| Test Type [required] | unit, integration, e2e, benchmark, stress | Nature of the test |
| Hardware [required] | gpu_0, gpu_1, gpu_2, gpu_4, gpu_8, h100 | Number/type of GPUs required |
| Component/Framework | vllm, trtllm, sglang, kvbm, planner, router | Backend or component specificity |
| Other | slow, skip, xfail | Special handling |
### Example
```python
@pytest.mark.integration
@pytest.mark.gpu_2
@pytest.mark.vllm
def test_kv_cache_multi_gpu_behavior():
...
```
## Test Discovery
Pytest automatically discovers tests based on their naming convention. All test files must follow this pattern:
### Lifecycle Marker Note
Use the marker for the earliest pipeline stage where the test must run (e.g., `@pytest.mark.pre_merge`). This ensures the test is included in that stage and all subsequent ones (e.g., nightly, release), as CI pipelines select tests marked for earlier stages.
**Example:**
If a test is marked with `@pytest.mark.pre_merge`, and the nightly pipeline runs:
```bash
test_<component_or_flow>.py
pytest -m "e2e and (pre_merge or post_merge or nightly)"
```
then this test will be included in the nightly run as well.
---
## Test Execution: How to Run Tests Locally and in CI
### Environment Setup
- Use the dev container for consistency.
- Install dependencies as specified in `pyproject.toml`.
- Set the `HF_TOKEN` environment variable for HuggingFace downloads:
```bash
export HF_TOKEN=your_token_here
```
- Model cache is located at `~/.cache/huggingface` to avoid repeated downloads.
### Running Tests
- Run all tests:
```bash
pytest
```
- Run by marker:
```bash
pytest -m "unit"
pytest -m "integration and gpu_1"
pytest -m "e2e and pre_merge"
pytest -m "benchmark and vllm"
```
- Run by component:
```bash
pytest -m planner
pytest -m kvbm
```
- Show print/log output:
```bash
pytest -s
```
- Run in container:
```bash
./container/build.sh --framework <backend>
./container/run.sh --mount-workspace -it -- pytest
./container/run.sh --mount-workspace -it -- pytest -m [optional markers]
```
- CI runs use the similar instructions as running inside the container. For example, running E2E tests as part of the nightly suite inside the Dynamo-VLLM container (which requires a single GPU) can be done with:
```bash
./container/run.sh --image $VLLM_IMAGE_NAME --name $VLLM_CONTAINER_NAME -- pytest -m "e2e and gpu_1 and (pre_merge or post_merge or nightly) "
```
#### Running tests locally outside of a container
Where:
- `component_or_flow`: The component or flow being tested (e.g., planner, kv_router)
- For e2e tests, this could be the API or simply "dynamo"
## Running Tests
To run tests outside of the development container, ensure that you have properly setup your environment and have installed the following dependencies in your `venv`:
To run all tests:
```bash
pytest
uv pip install pytest-mypy
uv pip install pytest-asyncio
```
To run only specific tests:
```bash
# Run only vLLM tests
pytest -v -m vllm
# Run only e2e tests
pytest -v -m e2e
# Run tests for a specific component
pytest -v -m planner
# Run with print statements visible
pytest -s
---
## Rust Testing: Organization and Execution
Rust tests in Dynamo are organized as follows:
- **Unit tests** are placed within the corresponding Rust source files (e.g., `lib.rs`) using `#[cfg(test)]` modules.
- **Integration tests** are placed in the crate's `tests/` directory and must be gated behind the `integration` feature.
### Test Segmentation by Features
- Use Cargo features to enable or disable groups of tests. For example:
```bash
cargo test --features planner
```
- Place all integration tests behind the `integration` feature gate. This ensures they are only run when explicitly enabled:
```bash
cargo test --features integration
cargo test --all-features
```
### Marking Slow or Special-Case Tests
- Use `#[ignore]` to mark slow or special-case tests. These tests will not run by default and must be explicitly included:
```bash
cargo test -- --ignored
```
### Example
```rust
#[cfg(test)]
mod kv_cache_tests {
#[test]
fn test_kv_cache_basic() {
// ...
}
#[test]
#[ignore]
fn test_kv_cache_long_running() {
// ...
}
}
```
## Test Markers
### CI Integration
- CI runs integration tests using either `cargo test --features integration` or `cargo test --all-features`.
- Use feature gates to control which tests are included in each CI pipeline.
Markers help control which tests run under different conditions. Add these decorators to your test functions:
---
### Frequency-based markers
- `@pytest.mark.nightly` - Tests run nightly
- `@pytest.mark.weekly` - Tests run weekly
- `@pytest.mark.pre_merge` - Tests run before merging PRs
## Additional Requirements and Troubleshooting
### Role-based markers
- `@pytest.mark.e2e` - End-to-end tests
- `@pytest.mark.integration` - Integration tests
- `@pytest.mark.unit` - Unit tests
- `@pytest.mark.stress` - Stress/load tests
- `@pytest.mark.benchmark` - Performance benchmark tests
- Tests must be deterministic; flaky tests are not permitted.
- Performance targets:
- Unit: <15 seconds per suite
- Integration: <5 minutes (premerge), <5 minutes (postmerge)
- E2E: <15 minutes (premerge).
- If a test is not running, verify the filename, markers, and folder location.
- For flaky tests, remove sources of randomness or set a fixed seed. Avoid unnecessary network calls.
- For slow tests, profile and optimize, or mark as `@pytest.mark.slow`.
- If model downloads fail, ensure `HF_TOKEN` is set and network access is available.
- If coverage is insufficient, add more tests or refactor code for better testability.
### Component-specific markers
- `@pytest.mark.vllm` - Framework tests
- `@pytest.mark.sglang` - Framework tests
- `@pytest.mark.trtllm_marker` - Framework tests
- `@pytest.mark.planner` - Planner component tests
- `@pytest.mark.kv_router` - KV Router component tests
- etc.
---
### Execution-related markers
- `@pytest.mark.slow` - Tests that take a long time to run
- `@pytest.mark.skip(reason="Example: KV Manager is under development")` - Skip these tests
- `@pytest.mark.xfail(reason="Expected to fail because...")` - Tests expected to fail
## References
- [pytest documentation](https://docs.pytest.org/en/stable/)
## Environment Setup
Tests are designed to run in the appropriate framework container built
via ```./container/build.sh --framework X``` and run via
```./container/run.sh --mount-workspace -it -- pytest```.
### Environment Variables
- `HF_TOKEN` - Your HuggingFace API token to avoid rate limits
- Get a token from [HuggingFace Settings](https://huggingface.co/settings/tokens)
- Set it before running tests: `export HF_TOKEN=your_token_here`
### Model Download Cache
The tests will automatically use a local cache at `~/.cache/huggingface` to avoid
repeated downloads of model files. This cache is shared across test runs to improve performance.
## Running tests locally outside of a container
To run tests outside of the development container, ensure that you have properly setup your environment and have installed the following dependencies in your `venv`:
```bash
uv pip install pytest-mypy
uv pip install pytest-asyncio
```
\ No newline at end of file
For further assistance, contact the Dynamo development team.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment