Watch someone fight a diffusion model over composition and you will see the same ritual. "Subject on the left third." Re-roll. "Subject on the LEFT." Re-roll. "Wide shot, subject small, left side, negative space on the right." Re-roll forty times, then settle for the least wrong one.
The model is not being stubborn. Text is simply a poor channel for geometry. Words describe what is in a picture far better than where it is. The fix has existed since 2023, and it is not a better prompt. It is a different input.
Text Was Never the Right Tool for Layout
A prompt is compressed into a text embedding that conditions the whole image. Nothing in it says "this pixel region". Asking for precise placement through words is like describing a floor plan over the phone: possible, slow, and never quite what you drew in your head.
Lvmin Zhang, Anyi Rao and Maneesh Agrawala attacked this directly in Adding Conditional Control to Text-to-Image Diffusion Models, the ControlNet paper. Instead of describing structure, you hand the model a structural image (edges, depth, segmentation or human pose) and it follows it. The authors report that training is robust with both small datasets under 50,000 images and large ones over a million, which is why ControlNets exist today for so many conditions and model families.
Reference Beats Description, Too
The same logic applies to style and identity. Describing a look in words is lossy. Showing the model an image is not.
IP-Adapter, from Hu Ye and colleagues, adds an image prompt to a frozen text-to-image model with only 22 million extra parameters. The authors report performance comparable to, or better than, a fully fine-tuned image prompt model. Because the base model stays frozen, the adapter works with other custom models built on the same base.
Which Control Does Which Job
| You want to fix | Use | Why |
|---|---|---|
| Outline and silhouette | Canny edges or scribble | Locks shapes without locking colour or texture |
| Foreground and background separation | Depth | Keeps the spatial layout while freeing the style |
| A figure's stance | OpenPose | Fixes the skeleton, not the person |
| Surface orientation | Normal map | Holds how light should fall on forms |
| A look, a palette, a character | Reference image (IP-Adapter) | Shows instead of describes |
| One wrong region | Inpaint mask | Regenerates only what is broken |
Control Is Also Authorship
There is a reason beyond speed to move from prompting to controlling, and it has a legal edge. In January 2025 the U.S. Copyright Office published Copyright and Artificial Intelligence, Part 2: Copyrightability. Two of its conclusions are worth quoting closely:
- "Based on the functioning of current generally available technology, prompts do not alone provide sufficient control."
- "Human authors are entitled to copyright in their works of authorship that are perceptible in AI-generated outputs, as well as the creative selection, coordination, or arrangement of material in the outputs, or creative modifications of the outputs."
The report also says copyrightability is decided case by case, and nothing here is legal advice. But the direction is clear. A prompt alone sits on the weak side of that line. A sketch you drew, a pose you set, a mask you painted and a composition you arranged are your expression, and they are the kind of input the report describes.
How This Works in Vision Studio
In Vision Studio, structure lives on the canvas as layers, and the next render reads them. Inpaint masks, ControlNet guides and reference images are all control layers, so composition is something you place, not something you plead for.
ControlNet support depends on the model family, and the exact list matters:
- SD 1.5: canny, depth, OpenPose, scribble and normal.
- SDXL: canny, depth and OpenPose, plus scribble and normal through the Union model.
- FLUX.1 [dev]: canny, depth and OpenPose through the Union model.
- SD 3.5 Large: canny and depth.
Two or more reference layers condition a render through IP-Adapter, region by region with masks on SD 1.5 and SDXL, and whole-image on FLUX.1 [dev]. A single reference layer runs as image-to-image. Inpainting regenerates a masked region with your selected checkpoint, with strength from 0.05 to 1.0 setting how far the pass moves from the original, and FLUX inpainting switches to FLUX.1 Fill.
A Worked Example
Say a client wants a product hero: a bottle on the right third, a long shadow falling left, empty space on the left for headline copy. Prompting for that is a lottery. Controlling for it is three steps.
- Sketch it. A rough scribble, placed on the canvas as a scribble guide (SD 1.5, or SDXL through the Union model): an oval for the bottle where it belongs, a wedge for the shadow, nothing on the left. Ugly is fine. The model reads structure, not polish.
- Describe it. The prompt now only carries what words are good at: material, lighting, mood. "Frosted glass bottle, warm studio light, soft long shadow, minimal backdrop."
- Fix what is wrong. If the cap renders badly, mask the cap and inpaint it at a moderate strength, leaving the rest of the frame exactly as it was.
The composition is decided once, by you, and every later render inherits it. That is the difference between steering and hoping.
A Better Loop
- Block the composition first. A rough sketch, a depth pass or a posed figure. Thirty seconds of drawing replaces thirty re-rolls.
- Prompt for content and style only. Let words do what words are good at.
- Add a reference for the look. Show the palette instead of naming it.
- Fix regions, not images. When one hand is wrong, mask the hand.
Put Down the Thesaurus
Prompt craft still matters, but it has a ceiling, and composition is where most people hit it. The tools to go past that ceiling are open, published and running on consumer hardware. Draw the layout, show the reference, mask the mistake. The model follows instructions far better than it reads minds.
Vision Studio is free, open-source (MIT), and runs ControlNet, IP-Adapter references and inpainting locally on your GPU. Every feature is listed with its source files at vision-studio-x.com/features. Download it at vision-studio-x.com/download.
-Rocky
#ControlNet #IPAdapter #AIArt #Composition #AICopyright #OpenSource #VisionStudio