# Benchmarks This directory contains benchmarking scripts and tools for performance evaluation of Dynamo deployments. The benchmarking framework is a wrapper around aiperf that makes it easy to benchmark DynamoGraphDeployments or other deployments with exposed endpoints. ## Quick Start ### Benchmark a Dynamo Deployment First, deploy your DynamoGraphDeployment using the [deployment documentation](../components/backends/), then: ```bash # Port-forward your deployment to http://localhost:8000 kubectl port-forward -n svc/ 8000:8000 > /dev/null 2>&1 & # Run benchmark python3 -m benchmarks.utils.benchmark \ --benchmark-name my-benchmark \ --endpoint-url http://localhost:8000 \ --model "" # Generate plots python3 -m benchmarks.utils.plot --data-dir ./benchmarks/results # Or plot only specific benchmark experiments python3 -m benchmarks.utils.plot --data-dir ./benchmarks/results --benchmark-name my-benchmark ``` ## Features Benchmark any HTTP endpoints! The benchmarking framework supports: **Flexible Configuration:** - User-defined benchmark names using `--benchmark-name` flag - Support for single endpoint benchmarking with `--endpoint-url` flag - Customizable concurrency levels (configurable via CONCURRENCIES env var), sequence lengths, and models - Automated performance plot generation with custom benchmark names **Supported Backends:** - DynamoGraphDeployments with port-forwarded endpoints - External HTTP endpoints (for comparison with non-Dynamo backends or platforms) ## Installation This is already included as part of the Dynamo container images. To install locally or standalone: ```bash pip install -e . ``` ## Data Generation Tools This directory also includes lightweight tools for: - Analyzing prefix-structured data (`datagen analyze`) - Synthesizing structured data customizable for testing purposes (`datagen synthesize`) Detailed information is provided in the `prefix_data_generator` directory. ## Comprehensive Guide For detailed documentation, configuration options, and advanced usage, see the [complete benchmarking guide](../docs/benchmarks/benchmarking.md).