InternVideo3: avoid slow Conv3d fallback in vision patch projection - #326

InternVideo3 vision patch embedding applies Conv3d to independent inputs whose spatial/temporal dimensions exactly match the convolution kernel. On PyTorch 2.9.1 + cuDNN 9.10.2 with BF16 contiguous tensors, this dispatches to aten::slow_conv_dilated3d, dominating inference while leaving the GPU lightly utilized.

Replace this patch projection with the mathematically equivalent F.linear, flattening the existing Conv3d weight. The Conv3d module, parameter names/shapes, bias, checkpoint loading and patch order remain intact.

PyTorch 2.9.1 explicitly disables cuDNN for contiguous FP16/BF16 3D convolutions with non-unit kernels on cuDNN 9.8–9.14, as a workaround for a known cuDNN issue: https://github.com/pytorch/pytorch/blob/v2.9.1/aten/src/ATen/native/Convolution.cpp#L417-L427 . On the same machine, profiling confirms FP16/BF16 contiguous inputs use the slow fallback, while channels_last_3d inputs and FP32 inputs use cuDNN. This is a version/dtype/layout-dependent problem, not a claim that all Conv3d implementations are slow. Current PyTorch main no longer contains this exact guard; it was not runtime-tested here. This change avoids the unnecessary convolution backend dependency entirely.

Validation:

• Executed the modified upstream patch-embedding class in isolation: FP64 forward outputs, input gradients and weight/bias gradients agree with Conv3d at rtol/atol 1e-12 for 1 and 17 patches. State dict keys remain unchanged.
• Using the equivalent change with the released HF InternVideo3-8B-Instruct model (revision c4602918b65225650d152db2850fe34e01d21fcd), RTX 4090, BF16, SDPA, batch size 1: a single-frame request decreased from 73.79 s to 2.08 s, and a 64-frame request from 138.22 s to 3.03 s. These are individual measurements, not aggregate throughput guarantees.
• BF16 patch relative L2 differences were below 5e-5. Generated outputs are not guaranteed identical: the video answer matched, while the image answer changed one bbox coordinate from 499 to 490. These rounding differences must be considered when comparing evaluation results.
• The consuming evaluation tool's 139 tests passed; its corrected 14-sample smoke covering chain history and three OracleTrack variants, plus resume verification, completed successfully.
• Full XTuner imports/distributed training and float8 integration have not been tested. Draft pending maintainer review of those paths.

The same fix should also be synchronized into the published Hugging Face custom model code, which has the same patch-embedding forward.