GitLab

Signed-off-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

This provides a simple example of how to write a triton-llm engine, and how to connect it to the OpenAI HTTP server.

This is the tool previously called `nio` and `llmctl`.

- **Inputs**: Text and HTTP.
- **Engines**: Echo, which streams your prompt back with a slight delay.

Build: `cargo build`

Pre-requisites: `nats-server` and `etcd` must be running locally, even though they are not yet used by `tio`.

Run with text input:
```
./target/debug/tio in=text out=echo_full --model-name test
```

Run with the triton-llm HTTP server:
```
./target/debug/tio in=http out=echo_full --http-port 8080 --model-name Echo-0B
```

List models:
```
curl localhost:8080/v1/models | jq
```

Will output
```
{
  "object": "list",
  "data": [
    {
      "id": "Echo-0B",
      "object": "object",
      "created": 1739400430,
      "owned_by": "nvidia"
    }
  ]
}
```

#### What's next

As triton-distributed gains features `tio` will be able to grow:
- When we get the pre-processor we can have token-in token-out engines. 
- When we get a pull-router we can have `in=nats` and `out=nats`.
- When we get discovery we can have dynamic engines.

Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>

Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Co-authored-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

Co-authored-by: aflowers <aflowers@nvidia.com>

Co-authored-by: aflowers <aflowers@nvidia.com>

Co-authored-by: aflowers <aflowers@nvidia.com>

Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>