README.md 3.06 KB
Newer Older
1
2
# Setup OpenTelemetry POC

3
> **Note:** The core OpenTelemetry packages (`opentelemetry-sdk`, `opentelemetry-api`, `opentelemetry-exporter-otlp`, `opentelemetry-semantic-conventions-ai`) are bundled with vLLM. Manual installation is not required.
4
5

1. Start Jaeger in a docker container:
6

7
    ```bash
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
    # From: https://www.jaegertracing.io/docs/1.57/getting-started/
    docker run --rm --name jaeger \
        -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
        -p 6831:6831/udp \
        -p 6832:6832/udp \
        -p 5778:5778 \
        -p 16686:16686 \
        -p 4317:4317 \
        -p 4318:4318 \
        -p 14250:14250 \
        -p 14268:14268 \
        -p 14269:14269 \
        -p 9411:9411 \
        jaegertracing/all-in-one:1.57
    ```

1. In a new shell, export Jaeger IP:
25

26
    ```bash
27
28
29
    export JAEGER_IP=$(docker inspect   --format '{{ .NetworkSettings.IPAddress }}' jaeger)
    export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
    ```
30

31
    Then set vLLM's service name for OpenTelemetry, enable insecure connections to Jaeger and run vLLM:
32

33
    ```bash
34
35
    export OTEL_SERVICE_NAME="vllm-server"
    export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
36
    vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
37
38
39
    ```

1. In a new shell, send requests with trace context from a dummy client
40

41
    ```bash
42
43
44
45
46
47
48
    export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)
    export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
    export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
    export OTEL_SERVICE_NAME="client-service"
    python dummy_client.py
    ```

49
1. Open Jaeger webui: <http://localhost:16686/>
50
51
52
53
54
55
56
57

    In the search pane, select `vllm-server` service and hit `Find Traces`. You should get a list of traces, one for each request.
    ![Traces](https://i.imgur.com/GYHhFjo.png)

1. Clicking on a trace will show its spans and their tags. In this demo, each trace has 2 spans. One from the dummy client containing the prompt text and one from vLLM containing metadata about the request.
![Spans details](https://i.imgur.com/OPf6CBL.png)

## Exporter Protocol
58

59
60
OpenTelemetry supports either `grpc` or `http/protobuf` as the transport protocol for trace data in the exporter.
By default, `grpc` is used. To set `http/protobuf` as the protocol, configure the `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` environment variable as follows:
61

62
```bash
63
64
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
65
vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
66
67
68
```

## Instrumentation of FastAPI
69

70
OpenTelemetry allows automatic instrumentation of FastAPI.
71

72
1. Install the instrumentation library
73

74
    ```bash
75
76
77
78
    pip install opentelemetry-instrumentation-fastapi
    ```

1. Run vLLM with `opentelemetry-instrument`
79

80
    ```bash
81
    opentelemetry-instrument vllm serve facebook/opt-125m
82
83
84
85
    ```

1. Send a request to vLLM and find its trace in Jaeger. It should contain spans from FastAPI.

86
![FastAPI Spans](https://i.imgur.com/hywvoOJ.png)