AI + Storytelling
I. Intro
a. AI
b. AI and Traditional Production
II. New Models of AI Production
a. AI in Virtual Production
b. Reality Capture and Virtual
c. AI Script-to-Screen Models
d. Script-to-Screen vs. Pre-Viz
e. Procedurally Generated Narratives
III. Towards the Future of Format
a. AI Worlds
b. New Asset Models
c. Studio Optionality
At the initial stage, AI algorithms can generate script outlines or story concepts based on predefined parameters or input from filmmakers. These algorithms can analyze existing scripts, genre conventions, and audience preferences to generate original and coherent story structures.
Once a script outline is generated, AI can assist in fleshing out the details, including character development, dialogue, and plot progression. It can suggest alternative scenes, rewrites, or adjustments to enhance the story. AI algorithms can analyze vast amounts of existing film data to generate dialogue that aligns with the chosen genre and audience intentions.
Let's first briefly cover the role of AI in bringing efficiencies to existing workflows. There is an iterative approach here, with //. In this model, AI is a tool that removes the repetitive work of tasks, enabling each individual within the production cycle to have more time to focus on the
In this model, AI is used as a tool to enhance and streamline various aspects of film production while working within the existing traditional workflows. It involves integrating AI technologies into different stages of the production process.
​
During pre-production, AI can assist in tasks like script analysis, casting suggestions, and location scouting. It can analyze large volumes of data from previous successful films to provide insights and recommendations for improving scripts or identifying potential cast members. Additionally, AI can help in optimizing the logistics of location scouting by analyzing parameters like weather, lighting conditions, and transportation requirements.
During production, AI can assist with tasks like camera tracking, scene composition, and real-time feedback. For instance, AI-powered camera tracking systems can automatically track and adjust camera movements, ensuring consistent framing and alignment. AI can also analyze scenes as they are being shot, providing feedback on lighting, color grading, and overall visual aesthetics.
In post-production, AI can aid in tasks like video editing, visual effects, and sound design. AI algorithms can analyze footage and automatically generate rough cuts or assemble scenes based on predefined parameters, reducing the time-consuming manual editing process. [Aiding with the first cut problem] Furthermore, AI-powered visual effects tools can generate realistic and high-quality effects, such as CGI elements or digital compositing. AI can also assist in audio post-production by automatically separating and cleaning audio tracks, enhancing sound effects, or generating music based on desired moods.
There are fundamentally three different models for content generation with AI, each with parameters of control that they enable for creators. Eventually, there will be more of an overlap between each of these streams, but for now some of these limitations are driven by where AI models are at the moment. However, given the rapid explosion of ____
As we start to consider virtual production techniques integrated as a part of the production process,
.png)
Currently, these are two independent processes, because they approach creation with AI from two different angles. Each offers unique affordances: script-to-screen models enable quick iteration by tapping into rapid generation and increase optionality for creation, while procedural models allow more control for the creator but with a tradeoff of needing to build out more cohesive top-down logic.
The benefits and the drawbacks of this approach center around the fact that this model is based on the frame as the final output. In a traditional production, behind each frame are an infinitude of decisions about the —each of which has effects on the story. The distributed nature of this decision makign process means that each division can bring creative innovation within their fields to the space.
​
The collapsing of this process into making decisions at the level of the frame on one hand has contributed to the notion of democratized access to creator tools. Rather than needing a whole crew, creators can make all of those decisions on their own. Of course, the flip side of this means that the intention that goes into making those decisions may often be lost.
Because the content arising out of popular image and video generators typically does so from a transformer model, there is no way to have consistency across separate outputs from these generators.
The benefits and the drawbacks of this approach center around the fact that this model is based on the frame as the final output. In a traditional production, behind each frame are an infinitude of decisions about the —each of which has effects on the story. The distributed nature of this decision makign process means that each division can bring creative innovation within their fields to the space.
​
The collapsing of this process into making decisions at the level of the frame on one hand has contributed to the notion of democratized access to creator tools. Rather than needing a whole crew, creators can make all of those decisions on their own. Of course, the flip side of this means that the intention that goes into making those decisions may often be lost.
Because the content arising out of popular image and video generators typically does so from a transformer model, there is no way to have consistency across separate outputs from these generators.
The benefits and the drawbacks of this approach center around the fact that this model is based on the frame as the final output. In a traditional production, behind each frame are an infinitude of decisions about the —each of which has effects on the story. The distributed nature of this decision makign process means that each division can bring creative innovation within their fields to the space.
​
The collapsing of this process into making decisions at the level of the frame on one hand has contributed to the notion of democratized access to creator tools. Rather than needing a whole crew, creators can make all of those decisions on their own. Of course, the flip side of this means that the intention that goes into making those decisions may often be lost.
.png)
Because the content arising out of popular image and video generators typically does so from a transformer model, there is no way to have consistency across separate outputs from these generators.
Top-down procedural narratives emerging from simulated volumetric storyworlds: This model of AI incorporation involves the creation of simulated volumetric storyworlds, where narratives emerge procedurally based on predefined rules and constraints. These storyworlds are complex virtual environments designed to simulate realistic and dynamic scenarios.
In this approach, AI algorithms are used to generate and evolve narratives within these storyworlds. The AI system is programmed with a set of rules, relationships, and parameters that govern the behavior of characters, objects, and the overall story progression. The AI can simulate the interactions and decisions of characters, taking into account factors like emotions, goals, and environmental influences.
As the story unfolds within the simulated storyworld, the AI system dynamically adapts and generates new events and plot developments. It can create branching narratives with multiple storylines, allowing for user interactivity or personalized experiences. The AI algorithms analyze the simulated environment, characters' behaviors, and user inputs to determine the most appropriate narrative path, leading to a unique and immersive storytelling experience.
This model of AI incorporation offers the potential for highly interactive and personalized narratives, where users can explore and shape the story within the simulated storyworld. It enables filmmakers to create complex and dynamic narratives that respond to user input or preferences, offering a more engaging and customized cinematic experience.
With the dynamic, responsive world as the base, the workflow decisions are determined by the format of the output.
Rather than pre-scripted interactions, these individual interactions could have a range of implementations or a category of interactions that allow for different forms of enactment. ie. an NPC character may be required to exhibit displeasure when confronted with a specific type of interaction from the player/user/participant, but the actual text of that intersection could be shaped by an AI language model that is unique to that NPC and that differs for each participant. (As we build more specific models for building emergent personality from the player side, these responses from the NPC can be shaped both by their own personality, and by the known facts about the player—either the facts that the in-world NPC is likely to know, or by the NPC as a vehicle for the broader knowledge that the systems designer has about the player).
Because the content arising out of popular image and video generators typically does so from a transformer model, there is no way to have consistency across separate outputs from these generators.
As we start to consider virtual production techniques integrated as a part of the production process,
