Dynamic batch C++ API example (#1728)

Example of using the C++ API to run an ONNX model with dynamic batch

examples/migraphx/cpp_dynamic_batch/CMakeLists.txt

+#####################################################################################
+# The MIT License (MIT)
+#
+# Copyright (c) 2015-2023 Advanced Micro Devices, Inc. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+#####################################################################################
+cmake_minimum_required(VERSION 3.5)
+project (cpp_dynamic_batch)
+
+set (CMAKE_CXX_STANDARD 14)
+set (EXAMPLE dynamic_batch)
+
+list (APPEND CMAKE_PREFIX_PATH /opt/rocm)
+find_package (migraphx)
+
+message("source file: " ${EXAMPLE}.cpp " ---> bin: " ${EXAMPLE})
+add_executable(${EXAMPLE} ${EXAMPLE}.cpp)
+
+target_link_libraries(${EXAMPLE} migraphx::c)

examples/migraphx/cpp_dynamic_batch/README.md

+# Running ONNX model with dynamic batch
+
+
+## Description
+This examples demonstrates how to run a graph program with dynamic batch using the MIGraphX C++ API. 
+
+
+## Creating dynamic dimension objects
+`dynamic_dimension` objects are used in MIGraphX to specify a range of dimension values from a minimum value to a maximum value and optimal values that the tensor can be at model evaluation time.
+A dynamic shape is defined by a list of `dynamic_dimensions` while a static shape only has fixed dimension values.
+For example, a `dynamic_dimension` with `{min:1, max:10, optimals:{1, 4, 10}}` means that the dimension can be any value from 1 through 10 with the optimal values being 1, 4, and 10.
+Supplied optimal values may allow MIGraphX to optimize the program for those specific shapes.
+
+A fixed `dynamic_dimension` can be specified by setting the `min` and `max` to the same value (ex. `{min:3, max:3}`).
+A dynamic shape specified solely by fixed `dynamic_dimension` objects will be converted to a static shape during parsing.
+This can be useful for setting a static shape using the `set_dyn_input_parameter_shape()` method discussed later in this document.
+
+
+## Parsing
+ONNX graphs [ONNX](https://onnx.ai/get-started.html) can be parsed by MIGraphX to create a runnable program with dynamic batch sizes.
+The dynamic batch range must be specified by a `dynamic_dimension` object.
+
+One method to set the `dynamic_dimension` object works for ONNX files that only have symbolic variables for the batch dimensions:
+```
+migraphx::program p;
+migraphx::onnx_options options;
+options.set_default_dyn_dim_value(migraphx::dynamic_dimension{1, 4, {2, 4}});
+p = parse_onnx(input_file, options);
+```
+
+Another option that can run any ONNX model with dynamic batch sizes uses the dynamic input map where the entire shape of the input parameter is supplied:
+```
+migraphx::program p;
+migraphx::onnx_options options;
+migraphx::dynamic_dimensions dyn_dims = {migraphx::dynamic_dimension{1, 4, {2, 4}},
+                                         migraphx::dynamic_dimension{3, 3},
+                                         migraphx::dynamic_dimension{4, 4},
+                                         migraphx::dynamic_dimension{4, 4}};
+options.set_dyn_input_parameter_shape("input", dyn_dims);
+p = parse_onnx(input_file, options);
+```
+
+## Compiling
+Currently the MIGraphX C/C++ API requires that `offload_copy` be enabled for compiling dynamic batch programs.
+Here is a snippet of compiling a model with `offload_copy` enabled:
+```
+migraphx::compile_options c_options;
+c_options.set_offload_copy();
+p.compile(migraphx::target("gpu"), c_options);
+```
+where `p` is the `migraphx::program`.
+
+
+## Saving and Loading
+A dynamic batch MIGraphX program can be saved and loaded to/from a MXR file the same way as a fully static shape program.
+
+
+## Executing the dynamic batch model
+The compiled dynamic batch model can be executed the same way as a static model by supplying the input data as `arguments` in a `program_parameters` object.
+
+
+## Running the Example
+Your ROCm installation could be installed in a location other than the one specified in the CMakeLists.txt.
+You can set `LD_LIBRARY_PATH` or `CMAKE_PREFIX_PATH` to that location so that this program can still build.
+
+The provided example is [`dynamic_batch.cpp`](./dynamic_batch.cpp)
+To compile and run the example from this directory:
+```
+$ mkdir build
+$ cd build
+$ cmake ..
+$ make
+```
+There will now be an executable named `dynamic_batch` with the following usage:
+```
+$ ./dynamic_batch
+```

examples/migraphx/cpp_dynamic_batch/dynamic_batch.cpp

+/*
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2015-2023 Advanced Micro Devices, Inc. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#include <iostream>
+#include <fstream>
+#include <vector>
+#include <string>
+#include <algorithm>
+
+// MIGraphX C++ API
+#include <migraphx/migraphx.hpp>
+
+int main(int argc, char** argv)
+{
+    migraphx::onnx_options o_options;
+    migraphx::dynamic_dimensions dyn_dims = {migraphx::dynamic_dimension{1, 4, {2, 4}},
+                                             migraphx::dynamic_dimension{3, 3},
+                                             migraphx::dynamic_dimension{4, 4},
+                                             migraphx::dynamic_dimension{5, 5}};
+    o_options.set_dyn_input_parameter_shape("0", dyn_dims);
+    auto p = migraphx::parse_onnx("../add_scalar_test.onnx", o_options);
+    migraphx::compile_options c_options;
+    c_options.set_offload_copy();
+    p.compile(migraphx::target("gpu"), c_options);
+
+    // batch size = 2
+    std::vector<uint8_t> a(2 * 3 * 4 * 5, 3);
+    std::vector<uint8_t> b = {2};
+    migraphx::program_parameters pp;
+    migraphx::shape s = migraphx::shape(migraphx_shape_uint8_type, {2, 3, 4, 5});
+    pp.add("0", migraphx::argument(s, a.data()));
+    pp.add("1", migraphx::argument(migraphx::shape(migraphx_shape_uint8_type, {1}, {0}), b.data()));
+    auto outputs = p.eval(pp);
+    auto result  = outputs[0];
+    std::vector<uint8_t> c(2 * 3 * 4 * 5, 5);
+    if(bool{result == migraphx::argument(s, c.data())})
+    {
+        std::cout << "Successfully executed dynamic batch add\n";
+    }
+    else
+    {
+        std::cout << "Failed dynamic batch add\n";
+    }
+
+    return 0;
+}