feat: OpenAI compatible http service (#123)

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com> Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Neelay Shah <neelays@nvidia.com>

feat: OpenAI compatible http service (#123)
Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com> Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Neelay Shah <neelays@nvidia.com>
ffc6dde1 · Ryan Olson · GitHub · 9d6643b7 · ffc6dde1 · ffc6dde1
Commit ffc6dde1 authored Feb 10, 2025 by Ryan Olson Committed by GitHub Feb 10, 2025
20 changed files
--- a/llm/rust/triton-llm/tests/data/replays/meta/llama-3.1-8b-instruct/completions/replay.yaml
+++ b/llm/rust/triton-llm/tests/data/replays/meta/llama-3.1-8b-instruct/completions/replay.yaml
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/completion.streaming.1
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/completion.streaming.1
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/completion.streaming.2
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/completion.streaming.2
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/completion.streaming.3
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/completion.streaming.3
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/replay.yaml
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mistral-7b-instruct-v0.3/completions/replay.yaml
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/chat-completion.streaming.1
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/chat-completion.streaming.1
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/chat-completion.streaming.2
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/chat-completion.streaming.2
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/chat-completion.streaming.3
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/chat-completion.streaming.3
--- a/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/replay.yaml
+++ b/llm/rust/triton-llm/tests/data/replays/mistralai/mixtral-8x22b-instruct-v0.1/chat_completions/replay.yaml
--- a/llm/rust/triton-llm/tests/data/replays/model_replay_config.yaml
+++ b/llm/rust/triton-llm/tests/data/replays/model_replay_config.yaml
--- a/llm/rust/triton-llm/tests/http-service.rs
+++ b/llm/rust/triton-llm/tests/http-service.rs
--- a/llm/rust/triton-llm/tests/openai_completions.rs
+++ b/llm/rust/triton-llm/tests/openai_completions.rs
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__minimum_viable_request.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__minimum_viable_request.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-10.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-10.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-2.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-2.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-3.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-3.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-4.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-4.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-5.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-5.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-6.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-6.snap
--- a/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-7.snap
+++ b/llm/rust/triton-llm/tests/snapshots/openai_completions__valid_samples-7.snap