How Gen AI Tools Like Lore Machine Revisualize Storyboarding

Generative AI development is gaining momentum for professional previsualization tools for storyboards for film, TV and games. In the past few weeks alone, AI visualization software has seen the addition of the “consistent characters” feature to Midjourney, the beta launch of Lightrick’s LTX Studio and Katalist.ai, where they join several storyboard solutions already in the marketplace.

Among them is Lore Machine, which came out of private beta on March 4 and is now accessible to the public and a waitlist of over 25,000 users. It was initially created to help screenwriters turn their screenplays into storyboards, to “own the entire process,” founder Thobey Campion told Variety Intelligence Platform in an interview on March 3.

Generative AI promises to accelerate the process of creating a storyboard, translating a script to visuals. Systems like Lore Machine could also help writers or creators produce novel visual adaptations across different media formats. But for professional film and TV projects, gen AI storyboards might still fall short of expectations because the purpose of a storyboard is to make tightly controlled decisions about the composition of shots in a scene.

Discussions with various communities made it clear Lore Machine could have broader usefulness for film, TV, video games, comic books and even podcast studios; branded entertainment departments at agencies or brands; creators on social media or platforms including Substack or Wattpad; and even educators to develop visuals for curricula.

When a user provides the system with original text — whether written as a screenplay, novel, short story or otherwise — Lore Machine analyzes the story and outputs images of key characters, scenes and locations rendered in a specific visual art style, which can then be exported and saved locally as image files in up to 4K resolution.

“What comes out looks like a storyboard — the establishing shots for almost any visual format,” said Campion.

To render these images, Lore Machines offers a “gallery,” or marketplace, of art style presets from which users can choose, with a handful of styles inspired by traditional storyboarding illustration, manga and anime. In the coming weeks, Campion noted that marketplace will grow to 150 styles developed in partnership with artists.

Lore Machine users pay an additional fee to use an artist’s style, with 100% being paid directly to the artist, who sets the price for its use. This art style marketplace is analogous to some AI voice companies such as ElevenLabs that provide a “library” of voices and allow voice actors or individuals to make their voice clone available and be compensated when it’s used.

The platform is multimodal in that it can also animate or bring movement to scenes by running images through a diffusion model. Whereas tools including OpenAI’s Sora and Runway’s Gen-2 are text-to-video, Lore Machine offers image-to-video functionality that produces an “animated version of that image,” said Campion.

Speech, music and sound effects can also be overlaid and exported as audio tracks, where the system can assign fully synthetic voices (not voice clones) to characters and add AI-generated music primarily based on a sentiment analysis of the text to determine a story’s genre or mood.

In film and TV production, the prospect of writers or studios developing complete storyboards or animatics with an AI visualization tool like Lore Machine most obviously and immediately concerns storyboard artists and potentially concept artists, but it would further hold important implications for directors, editors and cinematographers who make or rely on storyboards.

Still, it’s worth considering the function of a storyboard in a film or TV production when considering generative AI’s usefulness. Fundamentally, a storyboard translates script to visuals and presents an ordered frame-by-frame or shot-by-shot sequence of panels. If a storyboard translates a script, a film translates a storyboard.

Panels can have varying degrees of conceptual or stylistic detail, but their substance matters more. Oftentimes, storyboards are hand-drawn sketches with descriptions or labels. Their visual simplicity is partly because their purpose is to convey important information about the construction and sequencing of shots in a scene.

A good storyboard must efficiently convey precise, intentional information about exactly how specific scenes will be shot or animated to tell the story, the key basic elements in a shot — including character blocking, framing, camera angles and movement, lighting and sound.

Generative AI systems used to develop a storyboard for a film or TV project would be subject to these same demands — namely, the artist or filmmaker’s need for precise control to independently make or refine these kind of creative and logistical decisions shot by shot.

That degree of control isn’t guaranteed with most of today’s generative AI tools, but it’s being built.

At the least, systems such as Lore Machine would demonstrably improve on a storyboarding workflow that only tried to involve other off-the-shelf AI tools. That’s chiefly because it can offer greater consistency and more specific composability of characters, objects, scenes and art style.

With Lore Machine, users are able to select, adapt and retain a specific artistic style consistently throughout. Improvements coming to the software will provide additional control levers for users to easily modify or customize system outputs to match their desired vision for a character, scene, location and even angles and lighting.

By contrast, attempting to create a storyboard with a text-to-image generator like DALL-E 3 would require prompt engineering and sequentially iterating imagery shot by shot, making it impossible to maintain consistency from one output to the next. For that reason, Midjourney earlier this week announced it was testing a new algorithm to allow users to have “consistent characters” across images. 

“Version 1 of the system was very stiff. What I mean by that is you enter your story and your characters, locations and scenes [would be] sort of it’s what you see is what you get,” said Campion. “The feedback we got from the community was we want more control. We [realized] we have to prioritize which parts of the system [will] become composable first. Within a month and a half, you’ll have full regenerative capabilities on your final outputs.”

Also, these systems promise significant efficiency gains that could prove useful where speed and volume are a premium, such as creative agencies or studios developing material for client approval.

In a similar vein, Lore Machine suggests opportunities to produce visual adaptations of specific story IP rapidly from one media format to another, such as a screenplay to a comic book or a podcast to a YouTube video.

In one early use case, screenwriter Philip Gelatt (“Love, Death and Robots”) provided Lore Machine with an original script that became the foundation of a manga, where the art was created by fine-tuning a manga image model with original art and sketches developed by Lore Machine’s art department and consented for use by the artists.

We’re still in a moment where previsualization is what these systems can most readily offer, but the final objective is to deliver “full-on production vis,” said Campion.

“Concept art and previs is firmly where we are. But I can see a scenario where within three or four months, we’re in full production vis zone. We’d love to get to the point where Lore Machine is building full-on films that are ready for the silver screen,” she added.

Ultimately, how a system like Lore Machine gets used — whether for rapid storyboards or any number of multimedia adaptations — will be up to the users.

“We’re going to be onboarding tens of thousands of people this week,” Campion said. “We’re watching it like, OK, well, what’s gonna happen here?”

Variety VIP+ Explores Generative AI From All Angles — Pick a Story