A Language-Informed Loss for Visual Generation

Abstract

End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced “lily”), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 → 53.01) against a pixel-only baseline on the GPIC dataset.

Motivation

An image of three airplanes against a cloudy sky. Two patches receive additive noise with the same pixel distance of 1000: one in the background sky and one on an airplane. Pixel loss treats both errors equally.

Pixel loss treats every pixel equally, but pixels contribute unevenly to image semantics. The above example shows two errors of equal pixel distance. One alters background texture while the other degrades a subject, yet pixel loss treats them the same. We hypothesize that language may help address this gap. In this work, we learn a visual space from image-caption pairs (GPIC) with an invertible image transform.

Overview

LILIE is an invertible encoder built with the TARFlow architecture. We train the encoder by passing an image through the encoder, corrupting the encoded image with additive Gaussian noise, then captioning the noised features with a LoRA-finetuned VLM. The objective is a standard cross-entropy loss against ground-truth captions from the GPIC corpus.

Once we have LILIE, we use L² distance in LILIE space to supervise JiT training. LILIE loss works on its own and improves generation quality over a pixel-only baseline (66.84 → 55.78 gFD), and combining LILIE loss with pixel loss further improves gFD to 53.01.

Results

Horizontal bar chart of gFD (lower is better). JiT-T2I-700M: Pixel 80.24, LILIE 68.88, Pixel+LILIE 65.86. JiT-T2I-1.1B: Pixel 66.84, LILIE 55.78, Pixel+LILIE 53.01. Callouts: replacing pixel loss gives 66.84 to 55.78 (−11.06 gFD); combining with pixel loss gives 66.84 to 53.01 (−13.83 gFD).

LILIE loss alone improves gFD at both the 700M and 1.1B JiT-T2I sizes, and combining Pixel+LILIE loss improves gFD further. Models are trained for 390,600 steps (one epoch) on GPIC.

Does language content matter?

Horizontal bar chart of gFD for the 700M generator: Pixel 80.24; Pixel+LILIE with random captions 82.06; Pixel+LILIE with ground-truth captions 65.86.

We train a LILIE encoder by replacing captions from GPIC with random tokens. For JiT-T2I-700M, the resulting gFD worsens over the Pixel baseline, indicating that informative captions play a crucial role during LILIE training.

Samples

Click a pair to zoom in.

PixelPixel+LILIE

Pixel sample for: orange petals, brown stamens, close-up, flower, blurred background — tagorange petals, brown stamens, close-up, flower, blurred background

Pixel+LILIE sample for: orange petals, brown stamens, close-up, flower, blurred background — tagorange petals, brown stamens, close-up, flower, blurred background

Pixel sample for: ancient temple, carved reliefs, hieroglyphs, stone wall, sunlit — tagancient temple, carved reliefs, hieroglyphs, stone wall, sunlit

Pixel+LILIE sample for: ancient temple, carved reliefs, hieroglyphs, stone wall, sunlit — tagancient temple, carved reliefs, hieroglyphs, stone wall, sunlit

Pixel sample for: tabby cat, white chest, sunlight, close-up — tagtabby cat, white chest, sunlight, close-up

Pixel+LILIE sample for: tabby cat, white chest, sunlight, close-up — tagtabby cat, white chest, sunlight, close-up

Pixel sample for: red building, debris, construction, truck, cone — tagred building, debris, construction, truck, cone

Pixel+LILIE sample for: red building, debris, construction, truck, cone — tagred building, debris, construction, truck, cone

Pixel sample for: A man with gray hair and a white beard smiles while wearing glasses and a black — shortA man with gray hair and a white beard smiles while wearing glasses and a black shirt.

Pixel+LILIE sample for: A man with gray hair and a white beard smiles while wearing glasses and a black — shortA man with gray hair and a white beard smiles while wearing glasses and a black shirt.

Pixel sample for: A yellow crocheted moon and star sit on a dark blue, textured blanket. — shortA yellow crocheted moon and star sit on a dark blue, textured blanket.

Pixel+LILIE sample for: A yellow crocheted moon and star sit on a dark blue, textured blanket. — shortA yellow crocheted moon and star sit on a dark blue, textured blanket.

Pixel sample for: A mandrill monkey with bright red and blue facial markings looks directly at the — shortA mandrill monkey with bright red and blue facial markings looks directly at the camera.

Pixel+LILIE sample for: A mandrill monkey with bright red and blue facial markings looks directly at the — shortA mandrill monkey with bright red and blue facial markings looks directly at the camera.

Pixel sample for: Tall, tree-like structures with purple lights stand in a park at night. — shortTall, tree-like structures with purple lights stand in a park at night.

Pixel+LILIE sample for: Tall, tree-like structures with purple lights stand in a park at night. — shortTall, tree-like structures with purple lights stand in a park at night.

Pixel sample for: Two ducks swim side by side on calm water. The duck on the right has a green hea — mediumTwo ducks swim side by side on calm water. The duck on the right has a green head and gray body, while the one on the left is brown with mottled feathers. Their reflections ripple gently in the still surface.

Pixel+LILIE sample for: Two ducks swim side by side on calm water. The duck on the right has a green hea — mediumTwo ducks swim side by side on calm water. The duck on the right has a green head and gray body, while the one on the left is brown with mottled feathers. Their reflections ripple gently in the still surface.

Pixel sample for: Red and orange maple leaves hang against a clear blue sky. Their veins are visib — mediumRed and orange maple leaves hang against a clear blue sky. Their veins are visible, and sunlight highlights their edges as they sway slightly on thin branches.

Pixel+LILIE sample for: Red and orange maple leaves hang against a clear blue sky. Their veins are visib — mediumRed and orange maple leaves hang against a clear blue sky. Their veins are visible, and sunlight highlights their edges as they sway slightly on thin branches.

Pixel sample for: A colorful mural of a blue, cartoonish character with glowing eyes covers a conc — mediumA colorful mural of a blue, cartoonish character with glowing eyes covers a concrete wall. Snow melts into a puddle on the ground next to a black hexagonal pavement. In the background, a tree and buildings are visible under daylight.

Pixel+LILIE sample for: A colorful mural of a blue, cartoonish character with glowing eyes covers a conc — mediumA colorful mural of a blue, cartoonish character with glowing eyes covers a concrete wall. Snow melts into a puddle on the ground next to a black hexagonal pavement. In the background, a tree and buildings are visible under daylight.

Pixel sample for: A calm lake reflects the overcast sky and surrounding mountains. Green hills ris — mediumA calm lake reflects the overcast sky and surrounding mountains. Green hills rise on either side, with clouds drifting near their peaks. Sparse vegetation emerges from the water’s surface in the foreground.

Pixel+LILIE sample for: A calm lake reflects the overcast sky and surrounding mountains. Green hills ris — mediumA calm lake reflects the overcast sky and surrounding mountains. Green hills rise on either side, with clouds drifting near their peaks. Sparse vegetation emerges from the water’s surface in the foreground.

Pixel sample for: A spider with a dark, patterned body and long legs is centered on a delicate, ra — longA spider with a dark, patterned body and long legs is centered on a delicate, radiating web. The web stretches across the frame, with fine threads visible against a blurred background of green foliage and soft blue sky. Behind the spider, a faint, out-of-focus shape suggests a person wearing a light-colored garment. The background shows muted greens and blues, indicating an outdoor setting with natural lighting. The spider’s legs are spread outward, gripping the web as if it’s stationary or preparing to move. The web appears taut and intact, with no obvious damage or debris caught in it. The overall scene is shallow-focused, drawing attention to the spider’s detailed markings and the intricate geometry of its web.

Pixel+LILIE sample for: A spider with a dark, patterned body and long legs is centered on a delicate, ra — longA spider with a dark, patterned body and long legs is centered on a delicate, radiating web. The web stretches across the frame, with fine threads visible against a blurred background of green foliage and soft blue sky. Behind the spider, a faint, out-of-focus shape suggests a person wearing a light-colored garment. The background shows muted greens and blues, indicating an outdoor setting with natural lighting. The spider’s legs are spread outward, gripping the web as if it’s stationary or preparing to move. The web appears taut and intact, with no obvious damage or debris caught in it. The overall scene is shallow-focused, drawing attention to the spider’s detailed markings and the intricate geometry of its web.

Pixel sample for: A round, golden-brown pastry with visible spiral layers sits on a white plate, d — longA round, golden-brown pastry with visible spiral layers sits on a white plate, dusted lightly with powdered sugar. Beside it is a generous swirl of white whipped cream, topped with a small green leaf. To the right of the cream is a dark, glossy fruit compote with a single greenish fruit nestled inside. The plate has a smooth, slightly reflective surface and is illuminated by warm lighting. In the background, faint shadows and a blurred edge suggest the plate is placed on a table or counter. The dessert appears freshly served, with the textures of the flaky pastry and creamy topping clearly distinguishable.

Pixel+LILIE sample for: A round, golden-brown pastry with visible spiral layers sits on a white plate, d — longA round, golden-brown pastry with visible spiral layers sits on a white plate, dusted lightly with powdered sugar. Beside it is a generous swirl of white whipped cream, topped with a small green leaf. To the right of the cream is a dark, glossy fruit compote with a single greenish fruit nestled inside. The plate has a smooth, slightly reflective surface and is illuminated by warm lighting. In the background, faint shadows and a blurred edge suggest the plate is placed on a table or counter. The dessert appears freshly served, with the textures of the flaky pastry and creamy topping clearly distinguishable.

Pixel sample for: A large, rectangular swimming pool with clear blue water is centered in the scen — longA large, rectangular swimming pool with clear blue water is centered in the scene, reflecting the bright sky above. To the left stands a grand stone structure with tall columns and a triangular pediment topped by sculpted figures, while to the right is a matching colonnade with a balcony and a small statue on its steps. Two white marble statues of winged figures sit near the pool’s edge—one on the left beside a metal ladder, and another further down on the right side. The pool’s surface has dark geometric patterns etched into it, and the surrounding area includes low stone railings and green trees visible behind the structures. In the far background, distant hills rise under a clear blue sky, and the entire scene is bathed in bright sunlight. The architecture resembles ancient Greek or Roman design, with detailed carvings on the columns and pediment.

Pixel+LILIE sample for: A large, rectangular swimming pool with clear blue water is centered in the scen — longA large, rectangular swimming pool with clear blue water is centered in the scene, reflecting the bright sky above. To the left stands a grand stone structure with tall columns and a triangular pediment topped by sculpted figures, while to the right is a matching colonnade with a balcony and a small statue on its steps. Two white marble statues of winged figures sit near the pool’s edge—one on the left beside a metal ladder, and another further down on the right side. The pool’s surface has dark geometric patterns etched into it, and the surrounding area includes low stone railings and green trees visible behind the structures. In the far background, distant hills rise under a clear blue sky, and the entire scene is bathed in bright sunlight. The architecture resembles ancient Greek or Roman design, with detailed carvings on the columns and pediment.

Pixel sample for: A wide, calm lake stretches across the scene under a deep blue twilight sky with — longA wide, calm lake stretches across the scene under a deep blue twilight sky with soft white clouds. On the left side, a wooden dock extends into the water, lit by warm yellow lights that reflect on the surface. Across the lake, a dense line of bare trees borders the far shore, silhouetted against the darkening sky. The water is smooth and glassy, mirroring the sky’s hue with gentle ripples. In the distance, a small structure with a roof sits near the shoreline, partially obscured by trees. The horizon shows a faint glow from distant lights, suggesting a town or park area beyond the treeline. The overall atmosphere is quiet and still, with no visible people or moving objects.

Pixel+LILIE sample for: A wide, calm lake stretches across the scene under a deep blue twilight sky with — longA wide, calm lake stretches across the scene under a deep blue twilight sky with soft white clouds. On the left side, a wooden dock extends into the water, lit by warm yellow lights that reflect on the surface. Across the lake, a dense line of bare trees borders the far shore, silhouetted against the darkening sky. The water is smooth and glassy, mirroring the sky’s hue with gentle ripples. In the distance, a small structure with a roof sits near the shoreline, partially obscured by trees. The horizon shows a faint glow from distant lights, suggesting a town or park area beyond the treeline. The overall atmosphere is quiet and still, with no visible people or moving objects.

BibTeX

@misc{agarwal2026lilie,
  title={A Language-Informed Loss for Visual Generation},
  author={Suchir Agarwal and Keshigeyan Chandrasegaran and Ziyu Chen and Kyle Sargent and Juan Carlos Niebles and Li Fei-Fei and Jiajun Wu},
  year={2026},
  eprint={2610.23653},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.23653},
}