Skip to content

Video Generation — the Director Workflow

The Director turns a short brief into a fully editable video document. Unlike a fire-and-forget pipeline, it is gated: the AI proposes a storyboard and content, and a human approves (or edits) at each gate before anything is committed. The engine lives in web/src/director/ and is driven through /api/director/*.

The state machine

INTAKE → ANALYZE → [GATE: storyboard] → WRITE → ASSEMBLE → [GATE: content] → HANDOFF

Orchestrated by web/src/director/orchestrator.ts:

Step Function Output
INTAKE startRun(brief) Captured Brief
ANALYZE Analyst role analyze(brief) Storyboard (product facts, narrative, scene plans, anchor blueprint)
GATE 1 resolveGate(runId, decision) Human approves / edits / rejects the storyboard
WRITE Copywriter role writeScene(plan, ctx) SceneCopy[] — props validated per template schema
ASSEMBLE assembleIR(storyboard, copies, brief) A concrete ResolvedIR
GATE 2 Review phase in the editor Human edits content/voiceover, then approves
HANDOFF document store + editor The document opens in the Studio editor

Run state is persisted to a file-backed runStore, so a generation survives a server restart.

Stage 1 — Intake (the brief)

The brief is captured in the DirectorPanel (web/src/components/DirectorPanel.tsx) with a minimal-first form:

  • Required: "What are you promoting?" (the product)
  • Optional (behind "Add details"): goal · audience · tone (multi-select) · length (15s / 30s / 60s / Auto) · CTA · aspect (16:9 / 9:16 / 1:1) · free-text notes
  • Brand (optional): brand name · accent color · logo upload
  • Voice: pick one narration voice (sample-playable)
  • Source documents (optional): upload .pdf, .docx, .txt, .md, .html, .csv → distilled into a summary + key points and folded into the brief

Document distillation

POST /api/director/distill (multipart) extracts text from uploaded marketing material — dependency-free (Node zlib/crypto; DOCX via ZIP, PDF via stream parsing). Per-file cap 40 KB, total 120 KB; text must be ≥85% printable. Returns { summary, points[], usedFiles[], skippedFiles[] }.

AI-driven clarify questions

Before generating, the Director can ask product-specific follow-ups via POST /api/director/clarify:

  • Round 1: up to 4 questions; Round 2: ≤ 1 adaptive follow-up if a gap remains
  • Each question arrives pre-filled with the AI's best guess — editable, and every question is skippable
  • Questions are tailored to the product (feature to spotlight, proof point, offer, core problem) and never re-ask anything already answered or provided
  • Falls back to a fixed heuristic set when the LLM is unconfigured

Stage 2 — Analyze (storyboard)

POST /api/director/start runs INTAKE + ANALYZE and returns a Storyboard plus a runId, suspended at the first gate. The Analyst role:

  1. Extracts structured product facts from the brief + distilled docs
  2. Shortlists blueprints (catalog.ts → shortlistBlueprints) by how well each matches the product facts
  3. Adopts the top blueprint as an anchor — its scene spine becomes the storyboard's beats
  4. Emits a narrative + per-scene ScenePlan ( beat · intent · templateId )

Gate 1 — Storyboard review (human-in-the-loop)

The run suspends and the storyboard is shown for approval. See Review & Export for the full gate UX. In short, the reviewer can:

  • Edit each scene's intent, swap its template, or remove the scene
  • Add notes and Regenerate (re-ANALYZE with feedback)
  • Approve to proceed to WRITE

POST /api/director/:runId/resolve carries the decision (approve | edit | reject):

Decision Effect
approve Proceed WRITE → ASSEMBLE → done, returns the ResolvedIR
edit Adopt the human-edited storyboard, then proceed as approve
reject Re-ANALYZE with the reviewer's notes, return a revised storyboard

Stage 3 — Write (copy)

The Copywriter role generates per-scene props for each ScenePlan, validated against that template module's Zod schema (completeStructured retries on validation failure). One scene at a time, with the brief + product facts as context, so the copy stays consistent across the video.

Stage 4 — Assemble (ResolvedIR)

assembleIR() (web/src/director/assemble.ts) maps the approved storyboard + generated copy into a concrete ResolvedIR:

  • Each ScenePlan → a ResolvedScene ( templateId + props + durationInFrames + transitionOut )
  • Applies brand theme tokens (accent, bg, font) from the brief
  • Attaches the logo as a document-level overlay (optional)
  • Sets one consistent voice for narration
  • Default transition: fade, 15 frames (overridable; last scene cuts)

Narration (voiceover)

POST /api/director/voiceover synthesizes narration via Azure TTS (web/src/director/voiceover.ts → llm/tts.ts):

  • Voices: coral, nova, shimmer, sage, echo, onyx
  • One MP3 per scene, written to storage/voiceover/ and served via /api/media
  • Each scene's voiceover carries the editable script text, base volume, and a VolumeEnvelope (fade in/out + keyframes)
  • "Auto" scene durations resolve from the measured narration length (@remotion/media-utils); fallback 150 frames

Gate 2 — Content review

After assembly the document opens in the Review phase, where each scene's template params and voiceover script can be edited before final approval. Edits update the ResolvedIR immediately. From here you Approve & open in editor for free-form editing, or Regenerate from brief for fresh content from the same intake.

Handoff

The finished ResolvedIR is saved to the document store (storage/projects/, via /api/director/docs) and loaded into the Studio editor for editing, then export.

Where it differs from the old pipeline

The retired v1 system ran a fixed, ungated script → voiceover → music → footage → render pipeline over per-run runs/<id>/ directories and chat sessions. The Director instead produces a single editable document with explicit approval gates, AI-authored on-screen copy (not just narration), and a 216-module template library — and there is no separate render stage in the generation flow; rendering is an on-demand action on the finished document. See Review & Export.