Digital Marionette
An Object-Aware Generation Workflow
An Object-Aware Generation Workflow
Digital Marionette represents a groundbreaking R&D collaboration with Lightcraft Technology, the creators of Jetset Cine. This innovative workflow solves a fundamental challenge in AI-powered VFX: how to apply different AI styles to specific objects within a single scene without "concept bleeding" between elements.
The system uses 3D proxy geometry and Cryptomatte masks to control multiple specialized LoRAs simultaneously, creating precise, object-aware generation that maintains consistency across animated sequences. This approach enables professional VFX workflows to harness generative AI with the precision required for commercial production, bridging the gap between creative AI capabilities and industry-standard control requirements.
Develop a production-ready workflow for object-aware AI generation using 3D geometry to control multiple LoRAs within a single scene.
3 months intensive R&D (June - August 2025)
Lightcraft Technology
Lead AI/VFX Developer, AI Training Specialist, AI Application Specialist, Custom Tool Creator, Workflow Architect.
ComfyUI, Blender, Custom LoRA Trainer, Cryptomatte, PartPacker, Custom Nodes, Python Scripting, Docker.
Generative AI's lack of precise control creates major obstacles for professional VFX integration. The core challenges included:
Digital Marionette creates a revolutionary approach to AI generation control using "invisible strings" like a marionette:
Early challenge: AI generated unwanted elements (roof) despite specific prompts
Solution: Precise control achieved workflow refinement
This project required pioneering new techniques in AI-powered VFX, combining cutting-edge generative models with traditional production pipelines through systematic R&D and iterative refinement.
The project began with extensive research into the latest generative models and concept refinement. Initial attempts using standard prompting revealed the need for more sophisticated control mechanisms.
Model A: Concept variations
Model B: Refined outputs
Through systematic testing of leading-edge generative models, we identified the optimal foundation approaches and began developing historically accurate concepts for the target assets.
Instead of complex 3D texturing, we developed a LoRA pipeline that proved more robust and production-friendly.
3D-generated chariot geometry
3D-generated tent geometry
Using advanced 3D generation tools, we created the foundation assets needed for the workflow.
This phase involved creating a sophisticated automated system for LoRA training. A critical breakthrough was solving the "Knowledge Stitching" problem for consistent AI captioning.
Breakthrough: First successful turntable dataset generation
Data augmentation: Automated lighting variations for robust training
As the workflow became more sophisticated, GPU memory constraints threatened the project's viability. This led to a strategic decision to architect the system as multiple interconnected workflows rather than one monolithic process.
First successful LoRA test: Proof of concept demonstrating consistent Persian war chariot generation
Breakthrough Moment: The first successful LoRA training proved the core concept. The trained model consistently generated Persian war chariots that maintained the desired aesthetic while allowing for natural variation - exactly what professional VFX workflows require.
The final phase involved integrating Blender's Cryptomatte system for precise object isolation and developing the final compositing workflow. This stage proved the entire system could work with production 3D scenes and animated camera movements.
Production Integration: Rather than requiring video footage, the system uses animated Cryptomatte sequences from Blender to define exactly where each LoRA should be applied. This provides the precision needed for professional VFX while maintaining the creative flexibility of generative AI.
Siggraph Showcase: The project culminated in a showcase-ready demonstration delivered in time for Siggraph 2025, proving the workflow's readiness for industry presentation and potential commercial adoption.
Developed proprietary automation techniques for generating specialized AI models with consistent, production-quality results.
Created specialized tools that solved key technical challenges.
Engineered a complete workflow enabling professional VFX teams to achieve precise, reliable results with advanced AI capabilities.
Structured as a professional R&D engagement with clear licensing and IP terms.
Digital Marionette successfully delivered a production-ready workflow that bridges the gap between generative AI capabilities and professional VFX requirements. The final demonstration showcased a 100-frame animated sequence with perfect object-specific styling.
Drag the slider to compare the 3D proxy scene with the final AI-generated result
Beyond the technical metrics, this project achieved several groundbreaking outcomes:
"Fantastic! It looks like the process is working."
— Eliot Mack, CEO, Lightcraft Technology
Digital Marionette represented a fascinating intersection of R&D innovation and practical problem-solving, requiring both cutting-edge experimentation and disciplined engineering to deliver a functional system within tight deadlines.
This project reinforced the importance of balancing innovation with practicality. While pushing the boundaries of what's possible with AI-powered VFX, the real value came from creating systems that professionals could actually use in production environments. The experience of building custom tools, solving undocumented technical challenges, and delivering within client timelines strengthened my ability to translate experimental research into robust, commercial-ready solutions. Working with Lightcraft Technology also demonstrated how collaborative R&D can accelerate innovation when both parties bring complementary expertise to complex problems.