Dynamic onnx

[stqwzr]

Small changes for creating dynamic onnx and the converting to tensorrt. Attached image input and output shapes will look like this. (Image after onnx visualization netron)

Tested with multiple batch sizes to ensure the model performs efficiently and correctly with dynamic input shapes. Also attached the screen of successfull converting to tensorrt by using trtexec only

Please review the changes and provide any feedback if needed. Thank you!

[stqwzr]

Also for converting i used this docker image nvcr.io/nvidia/tensorrt:23.06-py3

[triple-mu]

Great job!
I will test your PR this weekend.
Thanks so much!

[Egorundel]

@stqwzr Hello!
Did you rewrite the code to make it work with batch inference?

[stqwzr]

Hi @Egorundel
I added only converting part which supporting batch inference, I didn't change the inference code.

[Egorundel]

@stqwzr It's a pity, because I tried to change the inference code so that everything works with the incoming image package std::vector<cv::Mat> batchImages. But so far it has been unsuccessful, as I am confused about how to allocate the memory associated with CUDA correctly.

In theory, everything is ready for it, just need to submit an image in the preprocessing and postprocessing functions and form an image vector in this way, and then submit it to the inference.

Do you have any tips on what can be done?

[stqwzr]

@Egorundel You can try with creating some flattened_batch_data (float *), where shape is (batch x channels x width x height).

#include <opencv2/opencv.hpp>
#include <vector>
#include <cuda_runtime.h>

void flattenBatch(const std::vector<cv::Mat>& input_batch, float* flattened_data) {
    int channels = input_batch[0].channels();
    int height = input_batch[0].rows;
    int width = input_batch[0].cols;

    for (size_t i = 0; i < input_batch.size(); ++i) {
        cv::Mat img = input_batch[i];
        img.convertTo(img, CV_32F); // Convert image to float if not already

        std::vector<cv::Mat> channels_vec;
        cv::split(img, channels_vec); // Split channels

        for (int c = 0; c < channels; ++c) {
            memcpy(flattened_data + i * channels * height * width + c * height * width, channels_vec[c].data, height * width * sizeof(float));
        }
    }
}

Kind of like this, btw it was generated by GPT maybe some issues, but logic should be same