Locking SDXL lighting consistency with ComfyUI demands depth maps over vague prompts

The most reliable way to eliminate "lighting drift" in SDXL character renders is by anchoring directional light sources to a ControlNet Depth map, rather than relying on generic prompt modifiers like rim lighting. Unlike simple text instructions—treated by SDXL as ambient "vibes"—spatial adjustments require a fixed 3D reference (e.g., a single canny or depth map) paired with IP-Adapter-FaceID to preserve facial continuity. This method forces the model to respect scene geometry, preventing shifts between frames where light positions fluctuate unpredictably.

Forcing lighting consistency in SDXL requires two parallel optimizations: workflow architecture and prompt phrasing. DeepSeek-V3 excels at streamlining JSON workflows for 16GB VRAM efficiency, reducing bottlenecks during batch processing. Meanwhile, Claude 3.5 Sonnet outperforms in distinguishing between global illumination and local light sources, a critical distinction that prevents the washed-out, plastic appearance common in other outputs. The model’s ability to weight modifiers like directional light from top-left:1.3 with precision ensures the light behaves as a fixed vector rather than an artistic suggestion.

The Prompt-Only Method fails entirely for professional use, as lighting shifts wildly between seeds with no consistency. A hybrid approach—IP-Adapter + Lighting Prompt—preserves likeness but inherits source-image lighting, clashing with intentional scene lighting. The only viable solution is ControlNet Depth + Fixed Light Prompt, which enforces geometric constraints. This trades render speed for stability, requiring a reference depth map to lock the light’s position in 3D space.

In ComfyUI, isolating lighting instructions from character descriptions is critical. Plugging everything into a single CLIP Text Encode causes the model to prioritize clothing details over light direction. Instead, use a Conditioning (Combine) node to split prompts: one string for character traits, another for lighting modifiers. Boost the weight of the lighting conditioning separately to override default behaviors. For latent noise variations, adjust the seed incrementally rather than resetting it entirely:

seed = 42069
variation_strength = 0.1
# Apply variation to latent noise without altering the light source

The ideal workflow treats DeepSeek-V3 as a structural optimizer—minimizing VRAM usage and accelerating workflows—while Claude 3.5 Sonnet refines the artistic language of lighting. Combining a depth map to anchor the light in 3D with Claude’s phrasing (e.g., high contrast, deep shadows, moody atmosphere) yields a cinematic, gritty consistency across multiple renders. Without this dual approach, even meticulous prompting yields unpredictable results.