1. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

The field of Information Retrieval (IR) has seen remarkable progress with the advent of deep language models. These models, often based on the Transformer architecture, achieve state-of-the-art effectiveness in ranking tasks. However, their computational cost poses a significant challenge for real-world deployment, where query latency is a critical metric.

A typical neural ranker computes a relevance score $𝑆\left(𝑞,𝑑\right)$ for a given query $𝑞$ and a document $𝑑$. The core issue is that computing this score often requires a joint, deep encoding of the query-document pair, making it impossible to pre-compute document representations offline. Let ${𝐸}_{𝑞}$ and ${𝐸}_{𝑑}$ represent the sets of contextualized token embeddings for the query and document, respectively, where each embedding is a vector in ${𝑅}^{𝑑}$.

To address this efficiency bottleneck, ColBERT proposes a novel late interaction architecture. Instead of joint encoding, we independently encode the query and document and perform a lightweight, yet powerful, interaction step afterward. The final score is computed as a sum of maximum similarities:¹

The model is trained end-to-end by minimizing a pairwise ranking loss function.

2. FILIP: Fine-grained Interactive Language-Image Pre-Training

Vision-Language Pre-training (VLP) models like CLIP have demonstrated powerful capabilities by aligning global image and text features. However, this global alignment approach lacks the ability to capture finer-grained relationships, such as the correspondence between specific objects in an image and words in a text description.

To address this, the FILIP (Fine-grained Interactive Language-Image Pre-training) model was introduced. It enhances cross-modal alignment by adopting a late interaction mechanism inspired by ColBERT. This mechanism operates directly within the contrastive learning objective.

Instead of comparing single global feature vectors, FILIP computes a token-wise similarity matrix between all image patch embeddings ${𝐸}_{𝐼}$ and all text token embeddings ${𝐸}_{𝑇}$. The final image-to-text similarity score ${𝑠}^{𝐼}$ is calculated by taking the average of token-wise maximum similarities:

where ${𝑛}_1$ and ${𝑛}_2$ are the number of image and text tokens, respectively. A symmetric operation is performed to compute the text-to-image similarity ${𝑠}^{𝑇}$. This encourages a detailed alignment between image patches and textual words.

The model is trained using a standard contrastive loss $ℒ$ over batches of image-text pairs. The key advantages are:

• It maintains the efficiency of dual-stream models, allowing for offline pre-computation of image and text representations.
• It achieves superior performance in downstream tasks, outperforming CLIP on zero-shot ImageNet classification and various image-text retrieval benchmarks, even with less training data.
• Visualizations confirm that FILIP learns meaningful, fine-grained alignments, correctly mapping textual tokens to corresponding image regions.

References

1. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
2. FILIP: Fine-grained Interactive Language-Image Pre-Training