Since the beginning of movies and television, directors have needed to film their stories in particular settings. Sometimes they went on location, sometimes they filmed on studio back lots with fake New York street scenes and building facades, and sometimes they hired painters to create backdrops. More recently, they filmed in front of green screens on which images could be superimposed later.

Industrial Light & Magic made a big breakthrough in backdrops in 2016 when they pioneered the use of high-definition LED video walls. Used initially on the film Rogue One, the same video walls (also called “volumes”) have been used extensively on the Disney+ series The Mandelorian. Much of the content on volumes over the past several years has been generated by the video gaming environment system Unreal Engine from Epic Games.

Generating TV and Film Backdrops

But now there is another alternative for generating the images that go on volumes for film and TV production. A few weeks ago I visited Hollywood—specifically XR Studios, which has a couple of large volumes—to check out Cuebric (after Stanley, of course), the first generative AI system to produce and edit images for film and TV volumes. The founders of Seyhan Lee, the Massachusetts-based company that produced Cuebric, have partnered with XR Studios to demonstrate the tool’s capabilities. They were in LA giving a demo to a large group of producers and video effects people, and I squeezed in just before them. Pinar Seyhan Demirdag has an arts and design background, and her co-founder Gary Lee Koepke has lots of ad agency experience on the creative side.

We are now getting used to the idea that generative AI can generate almost anything, but seeing the generative images displayed on a huge semi-circular screen was rather amazing. Cuebric generates images from the Stable Diffusion model, using the normal prompt interface. However, the prompts themselves are anything but normal. The one I saw started with “Image from an award-winning movie” and went on to specify a number of other attributes. Another sample prompt, for example, goes:

a cinematic well-composed mountain landscape with a river, long exposure, moody, digital art,8K, high contrast, high detail, cinematic lighting, DLSS, smooth unreal engine render, clear octane render, volumetric lighting

These prompts are way more complex than my amateurish efforts, and some even include a set of non-desirable attributes to be excluded from the generated image in brackets, e.g.:

[blur, cg, lowres, aberration, text, deformation, drawing, noise, grain, amateur, cartoon, render, closeup, collage, mosaic, watermark, person]

I asked Demirdag and Koepke where the industry will find prompt engineers with the needed expertise, and Koepke said, “They will need to be a combination of VFX designer and AI whisperer.”

From Image to Screen

The Cuebric action doesn’t end with the creation of an image. There is a lot of work after that which is enabled by the tool. It includes:

AI-Based Segmentation—turning two-dimensional images into “2.5D” images with automated shape and depth-based image manipulation;

• AI Inpainting—automated fill-in of gaps in segmented images with other AI images;
• Editing—removing unwanted elements of the segmented images;
• AI Upscaling—converting the image to a higher resolution such as 4K;
• Exporting—sending the edited, scaled image to a media server for use on a volume stage.

Cuebric images can also be animated. The big image I saw had some clouds moving across the sky, for example. No doubt there is more animation to come.

Of course, these capabilities go well beyond my own VFX technical depth, but the resulting images look very realistic on a large screen. Compared to a green screen with backdrop images added later, the generative images on a volume are much easier for actors to deal with; they can see the environment in which they are acting.

The Future of Filmed Entertainment

That raises the question of whether actors themselves will eventually be replaced by animated generative images. Looking at a human actor standing in front of an AI-generated moving image brought to my mind the thought that the actor could also be AI-generated. When I asked Demirdag about this possibility, she said, “Ten years.” My guess is that it may be a bit sooner given the rapid progress of generative AI.

I would also expect that these generative technologies would soon be used to generate video entertainment directly (to memory or tape or whatever medium you prefer). That capability already exists, but in very primitive form—see this Seinfeld video, for example. Generative AI film—or videocameras—seem to me just an intermediate step in the transformation of entertainment media.

We will see how quickly the Hollywood establishment gravitates toward these new ways of producing content. According to the Seyhan Lee founders, it is very enthusiastic. Within the first two weeks after Cuebric was announced, there were more than 150 requests to demo the technology. Apparently the appeal of realistic backdrops at much lower cost than going on location, and much greater tangibility than green screens, is quite strong.