Google researchers introduce Multimodal Bottleneck Transformer for audiovisual fusion
Google researchers have proposed a new transformer architecture (MBT) for audiovisual fusion and explored different fusion strategies using cross-attention between latent tokens in a new paper called, Attention Bottlenecks for Multimodal Fusion. Machine perception models are usually modality-specific and optimised for unimodal benchmarks, and hence the late-stage fusion of final representations or predictions from each modality (‘late-fusion’) is still a dominant paradigm for multimodal video classification. Multimodal Bottleneck Transformer uses ‘fusion bottlenecks’ for modality fusion at multiple layers. Compared to traditional pairwise self-attention, MBT forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the important information in each modality and only share what is necessary. The researchers showed restricting cross-modal attention via a small set of fusion bottlenecks achieved state-of-the-art results on a number of video classification benchmarks while also reducing computational costs compared to vanilla cross-attention models.
Machine perception models are usually modality-specific and optimised for unimodal benchmarks.
Machine perception models are usually modality-specific and optimised for unimodal benchmarks.
["Generative Pre-Trained Transformer"]
["Go", "programming_languages:R", "AI", "Modal", "programming_languages:Go", "Generative Pre-Trained Transformer", "transformer architecture", "R"]
["AI", "R", "Go", "transformer architecture", "Modal", "programming_languages:R", "programming_languages:Go"]
A writer by passion, Kartik strives to get a deep understanding of AI, Data analytics and its implementation on all walks of life. As a Senior Technology Journalist, Kartik looks forward to writing about the latest technological trends that transform the way of life!