1. FlowTok: Flowing Seamlessly Across Text and Image Tokens

FlowTok presents a revolutionary approach to cross-modal generation by enabling direct flow matching between text and image modalities. Unlike conventional methods that treat text as a conditioning signal for image generation, FlowTok projects both modalities into a unified, compact 1D latent space.

1.1. Direct Flow Between Modalities

1.1.1. Unified Latent Space Design

FlowTok encodes both text and images into compact 1D tokens with shape $77\times 16$:

Text Processing:

• CLIP text encoder extracts initial embeddings ${𝑇}_{\text{init }}\in {𝑅}^{𝑁\times 𝐶}$
• Text projector maps to latent space: ${𝑍}^{𝑇}\in {𝑅}^{𝑁\times 𝐷}$
• Gaussian distribution modeling with KL regularization

Image Processing:

• Enhanced TA-TiTok with RoPE and SwiGLU FFN
• Direct encoding to ${𝑍}^{𝐼}\in {𝑅}^{𝐾\times 𝐷}$ where $𝐾=𝑁=77$
• Maintains semantic information in compact representation

1.1.2. Flow Matching Framework

The flow matching objective learns direct transformation:

$${𝑋}_{𝑡}=\left(1-𝑡\right)\cdot 𝑋+𝑡\cdot 𝑁$$

where the velocity field is:

$${𝑉}_{𝑡}=\frac{𝑑{𝑋}_{𝑡}}{𝑑𝑡}=𝑁-𝑋$$

Unlike standard flow matching that uses noise as source distribution, FlowTok treats text tokens ${𝑍}^{𝑇}$ and image tokens ${𝑍}^{𝐼}$ as both source and target distributions.

1.1.3. Semantic Preservation

To prevent information loss during dimensionality reduction, FlowTok introduces text alignment loss:

$${ℒ}_{\text{align }}=\frac{\text{CE}\left({\text{logits}}_{\text{TZ}},\text{ labels}\right)+\text{CE}\left({\text{logits}}_{\text{ZT}},\text{ labels}\right)}{2}$$

where:

$$\begin{array}{cc}{\text{logits }}_{\text{TZ }}& =\exp \left(𝜏\right)\times \left({𝑇}_{𝑃}\times {𝑍}_{𝑇}^{𝑇}\right)\\ {\text{ logits }}_{\text{ ZT }}& =\exp \left(𝜏\right)\times \left({𝑍}_{𝑇}\times {𝑇}_{𝑃}^{𝑇}\right)\end{array}$$

1.1.4. Training Objective

Complete loss function:

$$ℒ={ℒ}_{\text{ fm }}+{𝛾}_1\cdot {ℒ}_{\text{ kld }}+{𝛾}_2\cdot {ℒ}_{\text{ align}}$$

where:

• ${ℒ}_{\text{fm}}$: Flow matching loss
• ${ℒ}_{\text{kld}}$: KL divergence regularization
• ${ℒ}_{\text{align}}$: Text alignment preservation

References

1. FlowTok: Flowing Seamlessly Across Text and Image Tokens