Self-supervised Visual Grounding of Sound and Language (2024)

3.1 Multi-Headed Aggregation of Similarities

DenseAV’s key architectural distinction is its loss function that directly supervises the “local” tokens of the visual and audio featurizers. This is a significant departure from other works[54, 22, 50, 6, 43, 20] that pool modality specific information into “global” representations prior to the contrastive loss. Unlike prior works, our loss function aggregates the full pairwise similarities between the local tokens into an aggregate measure of similarity for a given pair of audio and visual signals. We show in Figure2 that this architectural choice enables DenseAV’s local features to align across modalities whereas other approaches such as average pooling, class tokens, and SimPool [48] do not.

We first describe our loss function informally and definite it more precisely in the next paragraph. Our loss function computes the (un-normalized) inner product between every pair of visual and audio features to form a “volume” of inner products. This volume represents how strongly each part of an audio signal “couples” to each part of a visual signal. We aim to find many large couplings between positive pairs of audio and visual signals. Ideally, these couplings should connect visual objects with their references in the audio signal. Conversely, we do not want to find couplings between negative pairs of signals. To compute a single global coupling strength for a pair of signals, we aggregate this volume of pairwise similarities into a single number. There are myriad ways to aggregate this volume ranging from “soft" average-pooling to “hard" max-pooling. Average pooling yields dense gradients and can improve convergence speed and stability. However, max-pooling allows the network to focus on the best couplings regardless the object’s size or a sound’s duration. Our aggregation function combines the benefits of average and max pooling by max-pooling visual dimensions and average pooling audio dimensions as proposed in[26]. Intuitively speaking, this averages the strongest image couplings over an audio signal. It allows small visual objects to have large effects yet provides a strong training gradient to many regions of the signals. Finally, we draw inspiration from multi-head self-attention [59] and generalize this operation to multiple “heads” that we max-pool before pooling the visual and audio dimensions. This allows DenseAV to discover multiple “ways” to associate objects across modalities.

More formally, let $\mathcal{S}(a,v)\in\mathbb{R}$ represent the similarity between a tensor of audio features $a\in\mathbb{R}^{CKFT}$ of size (Channel $\times$ K-heads $\times$ Frequency $\times$ Time) and a tensor of visual features $v\in\mathbb{R}^{CKHW}$ of size (Channel $\times$ K-heads $\times$ Height $\times$ Width). To define this scalar similarity score, we first create a local similarity volume, $s(a,v)\in\mathbb{R}^{kfthw}$. For simplicity, we consider the aggregated similarity between a single image and audio clip but note one can easily generalize this to max-pool over video-frames. We define the full pairwise volume of similarities as:

Where $a[c,k,f,t]$ represents the value of $a$ at location $[c,k,f,t]$ and $\cdot$ is scalar multiplication. We aggregate this similarity volume into a single score $\mathcal{S}(a,v)\in\mathbb{R}$:

We note that this operation can be viewed as a multi-head generalization of the MISA loss of[26], and a multi-head multi-time generalization of the MIL loss of[3].

3.2 Loss

We can use the similarity between audio and visual signals defined in Equation2 to construct a contrastive loss. We follow recent works[20, 18, 61] and use the temperature-weighted InfoNCE [40] to encourage similarity between positive pairs of signals and dissimilarity between negative pairs. In DenseAV, we form $B$ positive pairs by splitting the audio and visual components of a Batch of training data. We form $B^{2}-B$ negative pairs by comparing a signal to all of the other signals in the training batch. More formally let $(a_{b},v_{b})_{1}^{B}$ be $B$ pairs of audio and visual signals. The visual-retrieval term of our InfoNCE loss is then:

Where $\gamma\in\mathbb{R}^{+}$ is a trainable inverse temperature parameter. We symmetrize this loss by adding the analogous audio-retrieval term, $\mathcal{L}_{V\to A}$, which iterates over negative audio signals in the denominator.

3.3 Audio and Visual Featurizers

The core of DenseAV is two modality-specific backbone networks. We use the DINO vision transformer[6] with ImageNet pretrained weights (without labels) to provide a strong, yet fully unsupervised, vision backbone. Unlike other approaches that use CLIP[50] as a backbone, DINO does not require paired text captions and learns from unlabeled images only. Practically, we find that DINO outperforms CLIP because of its better-behaved local tokens[13], an effect we explore in the Supplement. We append an additional layer norm operation across the channel dimension[4] and a $1\times 1$ Convolution to DINO. The layer-norm and $1\times 1$ convolution ensure the architecture does not start with a saturated loss function. We use the HuBERT audio transformer[28] as DenseAV’s audio backbone. HuBERT operates on waveforms and is trained on the LibriSpeech[44] dataset using only self-supervision. Hubert outputs a single feature per time frame, corresponding to $F=1$ in Section3. Though HuBERT was only trained on speech, its audio features can be fine-tuned for more general sounds, much like how vision backbones can be fine-tuned for new datasets[63]. As in the visual branch, we append a channel-wise LayerNorm block and two $3\times 3$ convolutions to the audio branch. These layers help the network avoid saturation and speed convergence. Furthermore, the two convolutions help the model aggregate information, which reduces the cost of the pairwise feature comparison used in our loss function. We refer to these added layers after the pretrained backbones as the “aligners” in later sections.

3.4 Regularizers

Disentanglement Regularizer, $\mathcal{L}_{Dis}$: We add a small regularization term to encourage each head of Equation1 to specialize and learn independent types of audio-visual associations. Interestingly we find that our 2-head model naturally learns to distinguish the meaning of words with one head and capture the sounds objects produce with another head. To further encourage this unsupervised discovery of concepts, we penalize the network when multiple attention heads are simultaneously active. More precisely, let $(a_{b},v_{b})_{1}^{B}$ be a Batch of $B$ paired audio and visual signals. Our disentanglement loss for two heads is then:

Where $\circ$ is elementwise multiplication and $|\cdot|$ is the elementwise absolute value function. $[k]$ mirrors PyTorch slicing notation and refers to selecting the activations for only the $k$th attention head. Intuitively, this loss encourages one head to be silent if the other head is active and is a “cross-term” generalization of the $l^{2}$ regularizer[27] for encouraging activation shrinkage. When $K>2$ we average contributions from every combination of heads. We ablate this, and our decision to max-pool heads in Table3.

Stability Regularizers, $\mathcal{L}_{Stability}$: Finally, we add several other small regularization terms to encourage stable convergence. We detail and ablate these terms in the Supplement. Briefly, these terms include standard regularizers like Total Variation[52] smoothness over time and non-negative pressure to encourage the network to focus on similarity instead of dissimilarity. In addition, we add a regularizer to prevent the calibration temperature, $\gamma$, from drifting too quickly, and a regularizer to discourage activations during silence and noise. In the supplement we show that each regularizer alone does not have a dramatic effect on final metrics but together they can stop collapses during training.

Combining these losses into a single loss function yields:

In our experiments we use $\lambda_{Dis}=0.05$ and refer interested readers to the supplement for the details of our small stability regularizer, $\mathcal{L}_{Stability}$.

3.5 Training

In our experiments we train DenseAV and relevant baselines on the AudioSet[19] dataset for sound prompted segmentation and AudioSet retrieval. We train on the PlacesAudio[25] dataset for speech prompted segmentation, PlacesAudio retrieval, and the ablation studies of Table4. In our disentanglement experiments of Table3 and feature visualizations of Figures1 and 2 we train on both AudioSet and PlacesAudio so that DenseAV can be familiar with both language, the prominent audio signal in PlacesAudio, and more general sounds from AudioSet. In these experiments we sample training data from these two corpora, so each batch has an even split between AudioSet and PlacesAudio.

Warming up Aligners: We find that we can dramatically improve the stability by first training the added aligners (convolutions and layer norms) for $3000$ steps while keeping pretrained DINO and HuBERT backbones fixed. This allows the aligners to adapt to these intelligent backbones before modifying each backbone’s sensitive weights. We use random resize crops, color jitter, random flips, and random greyscaling as image augmentations. We randomly sample a single video frame to feed to our visual branch. Audio clips are converted to single-channel format and are trimmed or padded with silence to create uniform 10 second clips. We re-sample audio clips according to the requirements of the backbone models used. For HuBERT, we re-sample to 16KhZ. We train on 8 V100 GPUs with an effective batch size of 80, and aggregate negative samples on all GPUs prior to computing the loss to ensure efficient parallelization. We provide additional training information and hyperparameters in the supplement.

Full Training: After warming up the aligners, we train the full model for an additional 800,000 steps using the same loss, batch-size, and training logic. We train all aligner weights and fine-tune all HuBERT audio backbone weights. We use low rank adaptation (LoRA)[29] to fine-tune the “Q”, “K”, and “V” layers of the DINO visual backbone attention blocks. This allows us to efficiently adapt DINO and stabilize the training as it is quite easy to collapse the carefully trained DINO weights. We use a LoRA rank of 8.

4.1 Qualitative Comparison of Feature Maps

Our first experiment in Figure2 highlights the dramatic differences in quality between DenseAV’s features and other approaches in the literature. DenseAV is the only backbone whose local tokens are semantically meaningful and show cross-modal alignment for speech and sound. Though both CAVMAE and ImageBind show high-quality retrieval performance, neither shows high quality aligned local tokens. As a result, DenseAV can associate and localize both sound and language significantly better than other backbones. DAVENet shows coarse correspondences between language and visual objects but cannot associate sound with visual objects and does not match DenseAV’s high resolution maps. Furthermore, the right half of Figure1 demonstrates that DenseAV naturally discovers and separates word semantics from the sound of objects without labels to supervise this separation. In the supplement, we provide additional visualizations of all backbones considered across a wide range of words and sounds.

4.2 Speech Prompted Image Segmentation

Dataset: We introduce a speech prompted segmentation dataset using the ADE20K dataset, which is known for its comprehensive ontology and pixel-precise annotations[66]. From this dataset, we curate an evaluation subset of image-class pairs by sampling up to 10 images for each object class in ADE20K, excluding images where the selected class was tiny ($<5\%$ of pixels). We only consider classes with at least 2 images that pass the tiny object criterion. For each class and image, we formed a binary target mask by selecting the semantic segmentation mask for that class. This resulted in 3030 image-object pairs spanning 478 ADE20K classes.

We created paired speech signals by speaking the prompt “A picture of a(n) [object]” where [object] is the name of the ADE20K class. We create clear, controlled, and consistent audio prompts using Microsoft’s neural text to speech service[51]. This service also provides exact timing of the “[object]” utterance within the broader prompt and ensures each class is measured equally. Grammar was manually verified for the utterances to ensure proper singular/plural and a/an agreement with the class name. We release images, masks, and audio prompts for reproducibility.

Evaluation Measure: We evaluate methods based on how well their speech-prompted activations align with ground truth masks for the visual object’s class. We quantify this with the binary Average Precision (AP) and Intersection over Union (IoU) metrics. These quantify how close activations match with the binary label mask from the ADE20K dataset. To compute an aggregate score over all of the object classes considered, we compute the mean average precision (mAP) and mean intersection over union (mIoU) by averaging AP scores across all object categories considered.

The mAP is particularly well suited for evaluating feature similarities because it is unaffected by monotonic transformations of the similarity scores. This eliminates the need for arbitrary thresholding and calibration. This is particularly important because many networks’ inner products are not centered at zero, and the best thresholding strategy can be nontrivial, and dependent on the network and object class. Average Precision avoids these confounding factors and ensures a fair comparison across methods. Unfortunately, unlike the mAP, the mIoU metric requires selecting a threshold. To ensure our mIoU measurement is similarly invariant to monotonic transformations we evaluate 20 uniformly spaced thresholds between the smallest and largest activations of each model. For each baseline, we report results for the best threshold to ensure a fair comparison between all networks considered.

Implementation: We compute image heatmaps by evaluating each modality-specific network on the image-audio pairs from our dataset. We extract dense features from the final layer of each network and form their similarity volume according to Equation1. For DenseAV we max-pool the head dimension to properly compare with single-headed models. We average activations over the temporal extent of the “[object]” utterance using the word timing information from the ground truth audio clip. This creates a heatmap over the image features that can be bi-linearly resized to the original image’s size. We then compare these per-pixel activation scores to ground truth object masks from our dataset.

Results: In Speech mAP and mIoU columns of Table1 we show that DenseAV achieves a 51% (+16.5 mAP) relative increase in speech-prompted semantic segmentation over previous methods. Approaches that use global token based contrastive strategies such as CAVMAE and ImageBind perform particularly poorly in this task, and this observation aligns with the qualitative results of Figure2.

4.3 Sound Prompted Image Segmentation

Dataset: To evaluate how well deep features localize sound, we build on Section4.2 and create a dataset of sound prompts that align with ADE20K classes. We first select the same (large) image-object pairs from ADE20K. We then create a mapping between the ADE20K and VGGSound[7] ontologies. To compute a robust mapping, we first embed ADE20K class names and VGGSound class names with the GPT Ada 2 text embedding model[41]. For each ADE20K class, we create a list of at most three candidates from the VGGSound ontology that have a cosine similarity $(>.85)$. We then manually review these candidates to select the best VGGSound class for each ADE20K class and remove any spurious or mistaken matches. This produces a set of 95 ADE20K classes with strong matches in the VGGSound ontology. For each of our original 3030 image-object pairs we select a random VGGSound validation clip with a matching class according to our mapped ontology. This yields 106 image-object pairs across 20 ADE20K classes.

Evaluation Measure: We use the same mAP and mIoU evaluation metrics as Section4.2, but instead average over the 20 ADE20K classes considered.

Implementation: We compute sound prompted image activations as in section4.2 but with one key change: we average activations over the entire clip because we do not have ground-truth sound timing information.

Results: The “Sound mAP and mIoU” columns of Table1 show that DenseAV achieves a 25% (+6.4mAP) relative improvement in sound prompted segmentation compared to the prior art. Most notably, ImageBind’s features cannot localize sound despite their high cross-modal retrieval performance learned from millions of hours of sound.

4.4 Cross-Modal Retrieval

We show that DenseAV’s representations are not only better for localization, but significantly outperform other approaches on cross-modal retrieval. We adopt the evaluation setting of[26] and measure cross modal retrieval accuracy at 1, 5, and 10 in a thousand-way retrieval task. In particular, we use the same thousand images from the validation set of[26] and also replicate this analysis on one-thousand random clips from the AudioSet validation data. Table2 shows results for 1000-way retrieval tasks on both the Places Audio and AudioSet datasets. We show cross-modal accuracy at 10, but also show larger tables in the supplement that echo these results using accuracy at 1 and 5. DenseAV significantly outperforms all baselines across all metrics. Interestingly, DenseAV outperforms ImageBind with less than half of the trainable parameters and no reliance on text.

4.5 Measuring Disentanglement

We observe that DenseAV’s heads naturally learn to differentiate audio-visual couplings that capture the meaning of words (language) and those that capture the sounds of objects (sound). Furthermore this effect generalizes to novel clips, including those with both sound and language as shown in Figure1. We quantify this observation in two ways, the first measures if a head’s average activation strength predicts whether a clip contains mainly “language” or “sound”. The second method quantifies how often the “sound” head is incorrectly active when the “language” head should be active and vice versa. We leverage the fact that AudioSet dataset contains mostly clips with ambient sound and rarely contains language. In contrast, Places Audio is entirely language-based without external ambient sound. We note that these analyses are specifically for our architecture with two heads $K=2$ and trained on both AudioSet and PlacesAudio data.

For both measures of disentanglement, we first compute a clip’s aggregated similarity for each head. In particular, we remove the max-pooling over heads in Equation2 to create a single-head similarity, $\mathcal{S}(a,v)_{k}$. We then min-max scale the scores of each head across both datasets to lie in the $[0,1]$ interval, which we refer to as $\hat{\mathcal{S}}(a,v)_{k}$. Using these normalized scores, we can create metrics that capture how well a given head responds only to a specific dataset.

Our first metric measures how well a head’s scores predict whether a clip is from the “sound” or “language” dataset. Let $(a_{b},v_{b})_{1}^{B}$ be tuples of paired audio and visual signals. let $l[k^{\prime}]_{b}$ be an indicator variable of whether the signal $(a_{b},v_{b})$ arises from the sound dataset, AudioSet, $(k^{\prime}=1)$, or the language dataset Places Audio $(k^{\prime}=2)$.

Where $AP(\cdot,\cdot)$ is the binary average precision with prediction and label arguments respectively. Intuitively, this measures whether the scores of head $k$ are direct predictors of whether the data is from dataset $k^{\prime}$. We can find the best assignment between heads and datasets such that each head is maximally predictive of the given dataset:

The prediction disentanglement score, PredDis, is a percentage that ranges from $50\%$ for completely entangled signals to $100\%$ if one can perfectly classify the signals using the scores of either head. The maximum over the two possible assignments makes this metric invariant to permutations of the heads. We note that this metric is a Hungarian matching assignment[32] over two entries, a common technique to asses unsupervised classification performance[31, 23].

Our second measure quantifies “spurious activations” in the non-dominant head. A truly disentangled system should have a head that only fires on sound, and another head that only fires on language. We create another disentanglement measure, ActDis, by replacing $\delta_{pred}$ in Equation7 with:

Intuitively, this measures the “inactivity” of head $k$ on dataset $k^{\prime}$. If head $k$ is totally silent on dataset $k^{\prime}$ then $\delta_{act}(k,k^{\prime})=1$. Like PredDis, ActDis is a percentage ranging from $50\%$ to $100\%$ with $100\%$ representing perfect disentanglement where the sound head is completely silent during the language clips, and vice versa.

Table3 shows that DenseAV achieves near perfect predictive ($99\%$) and activation ($91\%$) disentanglement. It also shows that our disentanglement regularizer and max-pooling over heads improves DenseAV’s natural ability to distinguish sound from language without supervision.

Negative Audio Splicing

Though using 3 is enough to make a reasonable cross-modal retrieval system, the extreme flexibility of self-attention operator in modern transformers can lead to degenerate solutions. For example, we found that without regularizers that encourage local features to be meaningful, the network could develop its own “global” tokens by selecting a handful of local tokens to carry all of the information. This is similar to the observation of [13] and we observed this occasionally in our audio branch, which would collapse to only use the first tokens. To keep the network from collapsing the semantics of the audio clip into a single token, we introduce small negative sample clips into our audio samples. These small negative audio regions are randomly spliced into the larger audio clip, and we encourage the network to set the couplings in these regions to zero with a $l_{2}$ regularizer. We include further details of the DenseAV’s architecture, hyperparameters, and regularizers in the Supplement.

More formally, let $(a_{b},v_{b})_{1}^{B}$ be a Batch of $B$ paired audio and visual signals as before. Let $m_{b}\in[0,1]^{T}$ be a soft mask where that measures whether a given location in the audio signal is actually part of a spliced negative clip. For example, $m_{b}[t]=1$ when the clip at time $t$ is part of the negative clip, $m_{b}[t]=0$ in the positive part of the clip, and $0<m_{b}[t]<1$ in the small boundary regions when the true clip is being spliced into the negative clip and both sounds are present. Our negative audio splicing regularizer squares each entry of the similarity tensor and averages these according to the strength of the negative clip indicator $m_{b}$:

Where the mean assumes that the weighting strength $m_{b}$ has been broadcast to the shape of $s(a_{b},v_{b})^{2}$. We point interested readers to the supplement for explicit formulations of these regularizers which are too verbose for the double-column format here. Intuitively, this term penalizes the network for having activations during a period of spliced negative audio. We also note that we apply this regularizer to any padded silence at the ends of short audio clips.

Calibration Regularization

The calibration temperature provides the network with the crucial ability to increase or decrease its certainty by updating a single parameter. However, the network can also achieve this effect by increasing or decreasing the magnitudes of its features. We found that sometimes the temperature would accelerate downward, forcing the feature magnitudes to increase to compensate. As a result, the network would eventually saturate or become unstable. We hypothesize that this is due to optimizer momentum, and we prevent this “runaway calibration”, by adding a small regularizer to the temperature parameter $\gamma$

This term penalizes the calibrator when it drops below 1.0 and encourages the calibrator to stay at or above 1.0.

Nonnegative Pressure

The InfoNCE loss function is invariant to the addition of a scalar to every inner product. Thus, to the network can choose to either find evidence of “positive” couplings connecting similar objects or “negative” couplings connecting regions that definitely do not belong together. We found that by encouraging the network to look for “positive” evidence, as opposed counterfactual evidence, improved training stability and performance across the key metrics we investigate. To encourage this behavior, we add a small regularizer to encourage inner products between features to be $\geq 0$. More specifically, let $\Omega$ be a set of 250 randomly selected coordinates $(b,b^{\prime},k,f,t,h,w)$. We then form our non-negativity regularizer:

This regularizer penalizes the similarity tensor if it drops below zero, encouraging features to exhibit positive couplings. We note than other works [23], have noted the benefits of using only non-negative feature couplings.

Disentangement Regularization

DenseAV’s multi-head similarity aggregation allows the network to use its different heads to model different independent ways that the audio and video modalities could couple together. Interestingly we find that if we give DenseAV two heads, one naturally specializes to language and the other head to more generic sounds. In particular, we find that one head will rediscover the meaning of words by “grounding” them to visual objects and another head will localize which objects created a given sound. To purify this disentanglement of concepts without supervision, we encourage different attention heads of our algorithm to specialize. More specifically we penalize the network when multiple attention heads are simultaneously active. In our experiments we use two attention heads. As before, we let $(a_{b},v_{b})_{1}^{B}$ be a Batch of $B$ paired audio and visual signals. Our disentanglement loss for two heads is then:

Where $\circ$ represents elementwise multiplication and $|\cdot|$ is the elementwise absolute value function. $[k]$ mirrors PyTorch slicing notation and refers to selecting the activations for only the $k$th attention head. Intuitively, this loss will encourage one head to be silent if the other head is active and can be viewed as a “cross-term” generalization of the $l^{2}$ regularizer [27] for encouraging activation shrinkage.

Total Variation Smoothness

To improve the quality and temporal consistency of discovered audio-visual couplings we impose a smoothness regularizer, $\mathcal{L}_{TV}$, in the audio-time dimension.

Where the activations for a given time slice $[1,t-1]$ are given by:

Informally, this regularizer penalizes when the inner product strengths change quickly over time.

Full Stability Regularizer

Putting these terms together into a single equation we have:

Where $\lambda_{Splice}=0.01$, $\lambda_{Cal}=0.1$, $\lambda_{NonNeg}=0.01$, and$\lambda_{TV}=0.01$.