Just Released: Torch-TensorRT v2.4.0

C++ runtime support in Windows Support, Enhanced Dynamic Shape support in Converters, PyTorch 2.4, CUDA 12.4, TensorRT 10.1, Python 3.12

Torch-TensorRT 2.4.0 targets PyTorch 2.4, CUDA 12.4 (builds for CUDA 11.8/12.1 are available via the PyTorch package index - https://download.pytorch.org/whl/cu118 https://download.pytorch.org/whl/cu121) and TensorRT 10.1.
This version introduces official support for the C++ runtime on the Windows platform, though it is limited to the dynamo frontend, supporting both AOT and JIT workflows. Users can now utilize both Python and C++ runtimes on Windows. Additionally, this release expands support to include all Aten Core Operators, except torch.nonzero, and significantly increases dynamic shape support across more converters. Python 3.12 is supported for the first time in this release.

Full Windows Support

In this release we introduce both C++ and Python runtime support in Windows. Users can now directly optimize PyTorch models with TensorRT on Windows, with no code changes. C++ runtime is the default option and users can enable Python runtime by specifying use_python_runtime=True

Enhanced Op support in Converters

Support for Converters is near 100% of core ATen. At this point fall back to PyTorch execution is either due to specific limitations of converters or some combination of user compiler settings (e.g. torch_executed_ops, dynamic shape). This release also expands the number of operators that support dynamic shape. dryrun will provide specific information on your model + settings support.

What's Changed

fix: FakeTensors appearing in get_attr calls by @gs-olive in #2669 feat: support adaptive_avg_pool1d dynamo converter by @zewenli98 in #2614 fix: Add cmake missing source file ref for core_lowering.passes by @Arktische in #2672 ci: Torch nightly version upgrade to 2.4.0 by @gs-olive in #2704 Add support for aten.pixel_unshuffle dynamo converter by @HolyWu in #2696 feat: support aten.atan2 converter by @chohk88 in #2689 feat: support aten.index_select converter by @chohk88 in #2710 feat: support aten.isnan converter by @chohk88 in #2711 feat: support adaptive avg pool 2d and 3d dynamo converters by @zewenli98 in #2632 feat: support aten.expm1 converter by @chohk88 in #2714 fix: Add dependencies to Docker container for apt versioning TRT by @gs-olive in #2746 fix: Missing parameters in compiler settings by @gs-olive in #2749 fix: param bug in test_binary_ops_aten by @zewenli98 in #2733 aten::empty_like by @apbose in #2654 empty_permute decomposition by @apbose in #2698 Removing grid lowering by @apbose in #2686 Selectively enable different frontends by @narendasan in #2693 chore(deps): bump transformers from 4.33.2 to 4.36.0 in /tools/perf by @dependabot in #2555 Fix upsample converter not properly registered by @HolyWu in #2683 feat: TS Add converter support for aten::grid_sampler by @mfeliz-cruise in #2717 fix: Bump torchvision version by @gs-olive in #2770 fix: convert_module_to_trt_engine by @zewenli98 in #2728 chore: cherry pick of save API by @peri044 in #2719 chore: Upgrade TensorRT version to TRT 10 EA (#2699) by @peri044 in #2774 Fix minor grammatical corrections by @aakashapoorv in #2779 feat: cherry-pick of Implement symbolic shape propagation, sym_size converter by @peri044 in #2751 feat: cherry-pick of torch.compile dynamic shapes by @peri044 in #2750 chore: bump deps for default workspace file by @narendasan in #2786 fix: Point infra branch to main by @gs-olive in #2785 "empty_like" decomposition test correction by @apbose in #2784 chore: Bump versions by @narendasan in #2787 fix: refactor layer norm converter with INormalization Layer by @zewenli98 in #2755 TRT-10 GA Support for main branch by @zewenli98 in #2781 chore(//tests): Update tests to use assertEqual by @narendasan in #2800 feat: Add support for is_causal argument in attention by @gs-olive in #2780 feat: Adding support for native int64 by @narendasan in #2789 chore: small mypy issue by @narendasan in #2803 Rand converter - evaluator by @apbose in #2580 cherry-pick: Python Runtime Windows Builds on TRT 10 (#2764) by @gs-olive in #2776 feat: support 1d ITensor offsets for embedding_bag converter by @zewenli98 in #2677 chore(deps): bump transformers from 4.36.0 to 4.38.0 in /tools/perf by @dependabot in #2766 fix: a bug in func run_test_compare_tensor_attributes_only by @zewenli98 in #2809 Fix ModuleNotFoundError in ptq by @HolyWu in #2814 docs: Example on how to use custom kernels in Torch-TensorRT by @narendasan in #2812 typo fix in doc on saving models by @laikhtewari in #2818 chore: Remove CUDNN dependencies by @zewenli98 in #2804 fix: bug in elementwise base for static inputs by @zewenli98 in #2819 Use environment for docgen by @atalman in #2826 tool: Opset coverage notebook by @narendasan in #2831 ci: Add release flag for nightly build tag by @gs-olive in #2821 [doc] Update options documentation for torch.compile by @lanluo-nvidia in #2834 feat(//py/torch_tensorrt/dynamo): Support for BF16 by @narendasan in #2833 feat: data parallel inference examples by @bowang007 in #2805 fix: bugs in TRT 10 upgrade by @zewenli98 in #2832 feat: support aten._cdist_forward converter by @chohk88 in #2726 chore: cherry pick of #2805 by @bowang007 in #2851 feat: Add support for multi-device safe mode in C++ by @gs-olive in #2824 feat: support aten.log1p converter by @chohk88 in #2823 feat: support aten.as_strided converter by @chohk88 in #2735 fix: Fix deconv kernel channel num_output_maps where wts are ITensor by @andi4191 in #2678 Aten scatter converter by @apbose in #2664 fix user_guide and tutorial docs by @yoosful in #2854 chore: Make from and to methods use the same TRT API by @narendasan in #2858 add aten.topk implementation by @lanluo-nvidia in #2841 feat: support aten.atan2.out converter by @chohk88 in #2829 chore: update docker, refactor CI TRT dep to main by @peri044 in #2793 feat: Cherry pick of Add validators for dynamic shapes in converter registration by @peri044 in #2849 feat: support aten.diagonal converter by @chohk88 in #2856 Remove ops from decompositions where converters exist by @HolyWu in #2681 slice_scatter decomposition by @apbose in #2519 select_scatter decomp by @apbose in #2515 manylinux wheel file build update for TensorRT-10.0.1 by @lanluo-nvidia in #2868 replace itemset due to numpy version 2.0 removed itemset api by @lanluo-nvidia in #2879 chore: cherry-pick of DS feature by @peri044 in #2857 feat: TS Add converter support for aten::flip by @mfeliz-cruise in #2722 ptq test error correction by @apbose in #2860 feat: Add dynamic shape support for sub by @keehyuna in #2888 feat: dynamic shapes support for sqrt and copy by @chohk88 in #2889 add dynamic shape support for aten.ops.gt and aten.ops.ge by @lanluo-nvidia in #2883 chore: cherry-pick FP8 by @peri044 in #2892 add dynamic shape support for sin/cos/cat by @lanluo-nvidia in #2887 Cancel in-progress ci build when a new commit is pushed by @lanluo-nvidia in #2903 readme by @laikhtewari in #2864 Only trigger doc gen if it is not a pytorchbot commit by @lanluo-nvidia in #2909 fix: Handle dynamic shapes in where ops by @keehyuna in #2853 chore: Dynamic support for split (#2871) into main by @peri044 in #2914 feat: C++ runtime on Windows by @HolyWu in #2806 chore: cherry pick of #2709 by @peri044 in #2850 Add dynamic shape support for layer_norm/native_group_norm/group_norm by @lanluo-nvidia in #2908 feat: dynamic shapes support for neg ops by @keehyuna in #2878 empty_stride decomposition by @apbose in #2859 empty_memory_format evaluator by @apbose in #2745 gather converter by @apbose in #2905 feat: Win/Linux Dual Compatible WORKSPACE + Upgrade CUDA + Upgrade PyT by @gs-olive in #2907 chore: add dynamic shapes section in the resnet tutorial by @peri044 in #2904 fix: Remove build artifact by @gs-olive in #2924 feat: Use a global timing cache and add a save option by @peri044 in #2898 chore: fix ValueRanges computation in symbolic nodes by @peri044 in #2918 scatter CI failures by @apbose in #2925 chore: Update layer_norm converter to use INormalizationLayer by @mfeliz-cruise in #2509 Add dynamic shape support for leaky_relu/elu/hard_sigmoid/softplus by @lanluo-nvidia in #2927 feat: Improve logging throughout the Dynamo path by @gs-olive in #2405 fix unsqueeze cannot work on more than 1 dynamic_shape dimensions by @lanluo-nvidia in #2933 feat: support native_dropout dynamo converter by @zewenli98 in #2931 feat: support aten index_put converter for accumulate=False by @chohk88 in #2880 feat: support aten.resize_ converter by @chohk88 in #2874 fix the docker build failure on main by @lanluo-nvidia in #2942 feat: Add Branches to Docker Build File by @gs-olive in #2935 add dynamic shape support for amax/amin/max/min/prod/sum by @lanluo-nvidia in #2943 fix: bug in vgg16_fp8_ptq example by @zewenli98 in #2950 Fixed layernorm when weight and bias is None in Stable Diffusion 3 by @cehongwang in #2936 chore: dynamic shape support for rsqrt/erf ops by @keehyuna in #2929 feat: dynamic shape support for tan, sinh, cosh, asin and acos by @chohk88 in #2941 fix: Repair integer inputs in dynamic shape cases by @gs-olive in #2876 Update PYTORCH to 2.4 by @lanluo-nvidia in #2953 Automate release artifacts build: usage pytorch cxx11 builder base image by @lanluo-nvidia in #2988 chore: cherrypick of #2855 by @zewenli98 in #3027 cherry pick 2740 to release2.4 branch. by @lanluo-nvidia in #3033 cherry pick from 3008 to release/2.4 by @lanluo-nvidia in #3035 assertEquals is deprecated in TestCase in Python 3.12 by @lanluo-nvidia in #3038 fix the artifacts name issue by @lanluo-nvidia in #3041

New Contributors

@Arktische made their first contribution in #2672 @aakashapoorv made their first contribution in #2779 @atalman made their first contribution in #2826 @yoosful made their first contribution in #2854

Full Changelog: v2.3.0...v2.4.0