1. FlowTok: Flowing Seamlessly Across Text and Image Tokens

FlowTok presents a revolutionary approach to cross-modal generation by enabling direct flow matching between text and image modalities. Unlike conventional methods that treat text as a conditioning signal for image generation, FlowTok projects both modalities into a unified, compact 1D latent space.

1.1. Direct Flow Between Modalities

1.1.1. Unified Latent Space Design

FlowTok encodes both text and images into compact 1D tokens with shape $77\times 16$:

Text Processing:

• CLIP text encoder extracts initial embeddings ${𝑇}_{\text{init}}\in {ℝ}^{𝑁\times 𝐶}$
• Text projector maps to latent space: ${𝑍}^{𝑇}\in {ℝ}^{𝑁\times 𝐷}$
• Gaussian distribution modeling with KL regularization

Image Processing:

• Enhanced TA-TiTok with RoPE and SwiGLU FFN
• Direct encoding to ${𝑍}^{𝐼}\in {ℝ}^{𝐾\times 𝐷}$ where $𝐾=𝑁=77$
• Maintains semantic information in compact representation

1.1.2. Flow Matching Framework

The flow matching objective learns direct transformation:

$${𝑋}_{𝑡}=\left(1-𝑡\right)\cdot 𝑋+𝑡\cdot 𝑁$$

where the velocity field is:

$${𝑉}_{𝑡}=\frac{𝑑{𝑋}_{𝑡}}{𝑑𝑡}=𝑁-𝑋$$

Unlike standard flow matching that uses noise as source distribution, FlowTok treats text tokens ${𝑍}^{𝑇}$ and image tokens ${𝑍}^{𝐼}$ as both source and target distributions.

1.1.3. Semantic Preservation

To prevent information loss during dimensionality reduction, FlowTok introduces text alignment loss:

$${\mathrm{ℒ︀}}_{\text{align}}=\frac{\text{CE}\left({\text{logits}}_{\text{TZ}},\text{labels}\right)+\text{CE}\left({\text{logits}}_{\text{ZT}},\text{labels}\right)}{2}$$

where:

$$\begin{array}{cc}{\text{logits}}_{\text{TZ}}& =exp\left(𝜏\right)\times \left({𝑇}_{𝑃}\times {𝑍}_{𝑇}^{𝑇}\right)\\ {\text{logits}}_{\text{ZT}}& =exp\left(𝜏\right)\times \left({𝑍}_{𝑇}\times {𝑇}_{𝑃}^{𝑇}\right)\end{array}$$

1.1.4. Training Objective

Complete loss function:

$$\mathrm{ℒ︀}={\mathrm{ℒ︀}}_{\text{fm}}+{𝛾}_1\cdot {\mathrm{ℒ︀}}_{\text{kld}}+{𝛾}_2\cdot {\mathrm{ℒ︀}}_{\text{align}}$$

where:

• ${\mathrm{ℒ︀}}_{\text{fm}}$: Flow matching loss
• ${\mathrm{ℒ︀}}_{\text{kld}}$: KL divergence regularization
• ${\mathrm{ℒ︀}}_{\text{align}}$: Text alignment preservation

References

1. FlowTok: Flowing Seamlessly Across Text and Image Tokens