This document describes the integration strategy for combining Tokio's asynchronous runtime with Rayon's data-parallel compute capabilities in the Dynamo runtime. The core philosophy is simple:
-**Tokio** handles I/O-bound operations, waiting, and coordination
-**Rayon** handles CPU-bound operations and data parallelism
- Multiple async tasks can concurrently submit different types of work to the shared Rayon thread pool
-**Map-reduce patterns**: Aggregations over large datasets
### Decision Thresholds
- Use Rayon when processing **≥10 items** in parallel
- Use `spawn_blocking` when CPU work takes **>1ms**
- Keep Tokio for operations with **>100μs waits** between items
- Use Rayon when you can **saturate multiple CPU cores**
### Overhead Considerations
Based on benchmarks, the async bridge between Tokio and Rayon has:
-**~25μs overhead** for small tasks (due to channel communication)
-**~4% overhead** for tasks taking >2ms
-**Negligible overhead** for tasks taking >10ms
For minimal overhead when using Rayon from async context:
-**Small tasks (<100μs)**: Run directly on Tokio
-**Medium tasks (100μs-1ms)**: Use `spawn_blocking` + `pool.execute_sync()`
-**Large tasks (>1ms)**: Use `pool.execute()` for convenience
## Concurrent Usage Patterns
The key insight is that multiple async tasks can concurrently use the same Rayon thread pool with different parallelization patterns. Rayon's work-stealing scheduler efficiently distributes work regardless of the pattern used.
5. Clear separation between I/O-bound and CPU-bound work
This architecture enables building high-performance services that efficiently handle both network I/O and CPU-intensive computations without manual thread management or complex synchronization.
Ryan Olson <rolson@nvidia.com>