[Feature Request] TensorRT support

[BlueAccords]

I think I remember somewhere you were looking into supporting tensortRT models, is that still in the backlog somewhere? or would implementing support for tensortRT require too much rework of the existing codebase?

[comfyanonymous]

Yes it's planned. There's a few more things I want to implement before. Unless something better comes out in the meantime I'm going to implement it.

[78Alpha]

Be prepared to pull your hair out if you get to it. Getting it setup on one machine is a chore, but getting it to work on someone else's is worse. Many TensorRT implementations have fallen because of it. Volta-ML and SDA-node are the larger examples.

[CyberTimon]

I would love to see tensorrt support, because SD XL is quite slow.
Also, tensorrt only seems to support max 768x768px. Do you think it is somehow possible to pass the SDXL 1024 in?

[78Alpha]

There was a pull for automatic that references the limits of TRT.

AUTOMATIC1111/stable-diffusion-webui-tensorrt#36

There is an upper limit of what it can do. As an example, if you have your batch size set to 8, you may not be able to generate dynamic images greater than 512x480 and the like.

I took note of some issues I've found during the year of TRT implementations flooding GitHub.

AUTOMATIC1111/stable-diffusion-webui-tensorrt#46 (comment)

Max dimensions I got from the calculated limit is Batch size 1, 858 x 858.

[comfyanonymous]

Yeah I'm going to go with AITemplate instead of TRT unless they add a way to replace the weights at runtime.

[CyberTimon]

Yeah AiTemplate seems good too.

[nistvan86]

SDXL models now have a TensorRT variant. https://huggingface.co/stabilityai/stable-diffusion-xl-1.0-tensorrt

[al-swaiti]

yeah its there! any idea how make it work in comfyui

[Erehr]

Aaaand new drivers just dropped promising 2X performance boost with TRT
https://www.nvidia.com/en-us/geforce/news/game-ready-driver-dlss-3-naraka-vermintide-rtx-vsr/

[78Alpha]

Still has the aforementioned issues with being dynamic (size, maximums, minimums). But, it's at least an official implementation for A1111. Hopefully that means it's working now, and on windows.

…

On Tue, Oct 17, 2023, 10:23 AM Erehr ***@***.***> wrote: Aaaand new drivers just dropped promising 2X performance boost with TRT https://www.nvidia.com/en-us/geforce/news/game-ready-driver-dlss-3-naraka-vermintide-rtx-vsr/ — Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7OOITKIRYIJOLHOCKSSLLX725H7AVCNFSM6AAAAAAVIY5CK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRWHA2TIMJWHA> . You are receiving this because you commented.Message ID: ***@***.***>

Aaaand new drivers just dropped promising 2X performance boost with TRT https://www.nvidia.com/en-us/geforce/news/game-ready-driver-dlss-3-naraka-vermintide-rtx-vsr/

and with that I'd love to see the fruits of this 2x speed boost as that could greatly improve my workflow on my 3090.

[xueqing0622]

support!
https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT

[SplitMilky]

Ok so how do i translate that support instructions for auto1111 to compfy. As I was using auto1111 but I was getting issues creating and using the optimized versions

[phineas-pta]

advanced users can try my node: https://github.com/phineas-pta/comfy-trt-test

[Dunc4n1dah0]

I just tested Nvidia's A1111 TRT extension and results are 2x faster indeed (at least for 512x768 simple generation).