- 12 Mar, 2025 13 commits
-
-
Graham King authored
Command line arguments are passed to the python engine like this: ``` dynamo-run out=pystr:my_python_engine.py -- -n 42 --custom-arg Orange --yes ``` The python engine receives the arguments in `sys.argv`. The argument list will include some standard ones as well as anything after the `--`. This input: ``` dynamo-run out=pystr:my_engine.py /opt/models/Llama-3.2-3B-Instruct/ --model-name llama_3.2 --tensor-parallel-size 4 -- -n 1 ``` is read like this: ``` async def generate(request): .. as before .. if __name__ == "__main__": print(f"MAIN: {sys.argv}") ``` and produces this output: ``` MAIN: ['my_engine.py', '--model-path', '/opt/models/Llama-3.2-3B-Instruct/', '--model-name', 'llama3.2', '--http-port', '8080', '--tensor-parallel-size', '4', '--base-gpu-id', '0', '--num-nodes', '1', '--node-rank', '0', '-n', '1'] ``` This allows quick iteration on the engine setup. Note how the `-n` `1` is included. Flags `--leader-addr` and `--model-config` will also be added if provided to `dynamo-run`. -
Ryan McCormick authored
-
Hongkuan Zhou authored
Co-authored-by:
hongkuan <hongkuanz@nvidia.com> Co-authored-by:
Anant Sharma <anants@nvidia.com>
-
ptarasiewiczNV authored
Co-authored-by: ptarasiewicz@nvidia.com <Piotr Tarasiewicz>
-
Maksim Khadkevich authored
-
Dmitry Tokarev authored
-
Neelay Shah authored
-
Anant Sharma authored
-
Maksim Khadkevich authored
-
Hongkuan Zhou authored
Co-authored-by:hongkuan <hongkuanz@nvidia.com>
-
Tanmay Verma authored
-
Ziqi Fan authored
feat: rename dynamo-sdk to dynamo; add dynamo run to call dynamo-run under the hood for unification (#104)
-
Neelay Shah authored
-
- 11 Mar, 2025 21 commits
-
-
hhzhang16 authored
-
Tanmay Verma authored
-
Neelay Shah authored
-
Ryan McCormick authored
-
Graham King authored
In https://github.com/ai-dynamo/dynamo/pull/89 `dynamo-run` was moved into a workspace. That means it builds in that workspace, so into `launch/target` not `launch/dynamo-run/target`. Update docs to match.
-
Graham King authored
If the python file raises an exception we print it like Python would. ``` $ ./target/debug/dynamo-run in=http out=pystr:~/Temp/cn47/1_e.py --model-name test Traceback (most recent call last): File "/home/graham/Temp/cn47/1_e.py", line 17, in generate raise MyException("The message") 1_e.MyException: The message ``` -
Hongkuan Zhou authored
Co-authored-by:
alec-flowers <aflowers@nvidia.com> Co-authored-by:
hongkuanz <hongkuanz@nvidia.com> Co-authored-by:
Alec <35311602+alec-flowers@users.noreply.github.com>
-
Graham King authored
- Latest from repo, many improvements - Support most of the OpenAI request features (temperature, top_p, etc) - Download models from Hugging Face if necessary
-
Neelay Shah authored
Co-authored-by:Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
-
Ryan McCormick authored
-
julienmancuso authored
-
Alec authored
-
Hongkuan Zhou authored
Co-authored-by:hongkuanz <hongkuanz@nvidia.com>
-
ptarasiewiczNV authored
-
Hongkuan Zhou authored
Co-authored-by:hongkuanz <hongkuanz@nvidia.com>
-
Tanmay Verma authored
-
Piotr Marcinkiewicz authored
-
ishandhanani authored
-
Hongkuan Zhou authored
Co-authored-by:hongkuanz <hongkuanz@nvidia.com>
-
Anant Sharma authored
-
Biswa Panda authored
-
- 10 Mar, 2025 6 commits
-
-
Anant Sharma authored
-
Ryan McCormick authored
-
Tanmay Verma authored
Co-authored-by:Shreyas Misra <shreyasm@nvidia.com>
-
Graham King authored
For the `echo` and `pystr` engines we previously required the user to pass `--model-name <x>` so we would have a name for the model. If the input is HTTP we do need this to match on the users' JSON request. If the input is Text we don't need a name. So if the input is Text and we don't already have a name for the model, give it one.
-
Harrison Saturley-Hall authored
-
Anant Sharma authored
-