Whitepaper 1

AI Filmmaking Infrastructure

AI iteration is infinite but production time is not.

This draft is still being iterated on, but it already maps the practical journey so far.

---

Research Thesis and Origin

This research began as a practical attempt to build a real working AI filmmaking pipeline, not a theoretical exploration.

Early experimentation revealed a gap between what AI tools are capable of and how filmmakers actually work. Most generative tools are designed by machine learning engineers using terminology and workflows that assume engineering knowledge.

Filmmakers, however, operate through a different system:

• Scene planning
• Shot lists
• Character continuity
• Assistant editors
• Asset organization
• Production scheduling

The central thesis that emerged during testing is:

AI iteration is infinite. Production time is not.

Generative systems allow endless experimentation, but filmmaking still operates within real constraints:

• Schedules
• Budgets
• Creative decisions
• Team communication

Without structure, AI tools quickly produce chaos rather than usable footage.

What This Research Attempted

The goal of this work was to answer a practical question:

What infrastructure is required for AI tools to function inside a real film production workflow?

To explore this, an experimental offline stack was built that combined:

• ComfyUI for node-based generation pipelines
• Ollama for local language models and prompt support
• Kohya SS for LoRA character training
• AudioX and Ace-Step pipelines for music generation
• Miniconda and FFmpeg for audio and processing infrastructure

During testing, the team attempted to reproduce real filmmaking practices inside generative systems:

• Building character identities before production
• Maintaining prompt libraries similar to visual style guides
• Structuring outputs by scene, shot, and take
• Testing video generation using first and last frame logic

The process revealed both useful techniques and major workflow failures. These findings form the basis of the whitepaper series.

Key Discoveries from Early Experiments

Character identity must be engineered

Attempting to generate characters repeatedly without identity training produced inconsistent results across scenes. Creating LoRA identity models stabilized both images and video generations.

Video models interpret spatial changes as edits

If the first and last frames of a generation differ too much in framing, angle, or composition, models often interpret the transition as a scene cut rather than motion. This revealed the need for shot-based planning rather than simple prompting.

Prompting changes the psychology of collaboration

Prompt engineering relies heavily on reading and writing rather than verbal collaboration. In large creative rooms, this slows down communication because someone must constantly translate spoken ideas into prompts. Some teams may work better in smaller prompt groups or quieter environments.

File organization becomes critical

Generative workflows produce hundreds or thousands of files. Without scene-based file structures similar to traditional film pipelines, outputs quickly become difficult to manage.

Purpose of the Whitepaper Series

Rather than presenting a single tool or technique, this research documents a broader system. Each whitepaper explores one part of the pipeline:

• Infrastructure and system architecture
• Offline toolchain installation
• Prompt collaboration models
• Character engineering and LoRA training
• Video generation methodology
• Music generation experiments
• File organization and human workflow

Together they form a practical guide for filmmakers attempting to integrate AI tools into production environments.

The goal is not simply to generate images or videos. The goal is to build repeatable filmmaking workflows that can support real productions.

This paper explains the infrastructure required to run modern AI filmmaking pipelines locally. The goal is to help filmmakers understand both the creative system and the technical layers required to operate generative workflows reliably.

System Overview

Modern AI filmmaking workflows work best when they operate as modular systems rather than single tools. Instead of relying on one platform, the system combines specialized tools for different functions:

• Image generation
• Video generation
• LoRA training
• Local language models
• Music generation
• Audio processing

The goal of this stack is:

• Fully offline capable
• Modular across creative mediums
• Adaptable to different filmmaking pipelines
• Capable of local experimentation without cloud dependency

Primary orchestration hub: ComfyUI.

Prompt engineering, dataset preparation, and identity-locked characters form the foundation of consistent results.

System Components

• Primary orchestration layer: ComfyUI
• Language model layer: Ollama
• Model training: KOHYA SS
• Audio environment: Miniconda + FFmpeg
• Music generation: AudioX 1.5
• Internal workflow tools: Lyra Creator
• Version control: GitHub
• Development stack: Rust, Python, Node.js