Virtual Production
Virtual Production
Casey Schatz • The Third Floor
Casey Schatz is a veteran of virtual production techniques for motion pictures and television. His credits include Avatar, Game of Thrones, Avengers: Age of Ultron, The Martian, and more. As Head of Virtual Production at The Third Floor, Schatz promotes the design and use of on-set simulcam.
Casey Schatz is a veteran expert in virtual production techniques for motion pictures and television. His credits include Avatar, Game of Thrones, Avengers: Age of Ultron, The Martian and more. As Head of Virtual Production at The Third Floor, Schatz promotes the design and use of on-set simulcam. As with many of our successful guests, his fascination with technology started at an early age.
Casey Schatz:Something clicked, very early on, I mean, maybe nine or 10 years old with a Commodor 64. And you could do these particular keystrokes where you could make a Siam magenta yellow, or a red green, or blue cube. And I would just sit in front of that thing for hours and at the same time, my dad did a lot of documentary work for the news station he was working at and, this fascination with optics and film and, traditional photography was kind of growing concurrently , with computer graphics and that theme. Really has never left and still hasn't frankly,
Noah Kadner:Schatz looked to gain education and experience initially as a cinematographer but found himself drawn to other areas of filmmaking.
Casey Schatz:I went to Cal arts and I got denied by the cinematography department, but got accepted into the still photography and the animation department, and Cal arts ended up being the perfect place because they let me sort of customize my own degree where I did theater lighting, traditional photography, cinematography. And then of course I was always in the F 1 0 5 lab, the computer graphics lab. And I had no idea what previz or techvis or virtual production was, at the time, but looking back, I can't think of a better education because it was so multidisciplinary and, fusing, modern, nifty computer graphic stuff with traditional old fashioned, filmmaking principles. And it's not that one is better than the other. It's where they shake hands that I think is so fascinating.
Noah Kadner:Like many other filmmakers in his peer group, Schatz was inspired by a string of blockbuster visual effects films that pushed the technology of the time forward.
Casey Schatz:were in high school when jurasic park and teach, who came out, which were okay, tick the box career done. Like I was just so enamored with the work and that work still holds up to this day, frankly., so as far as soul searching and career, like, that was almost, never an issue. So then the only question afterwards was which field within it, the first school I went to was the academy of art, San Francisco. And I was working on modeling, a human face and, you know, putting, specul reflections on the eyes. And we were trying to do what at the time was extremely primitive, hair and stuff. And I had this kind of moment of like, why am I trying so hard to recreate reality when reality is just right outside the door and so I kind of had this phase actually where I just stopped doing computer graphics and really thought. I'm gonna be a national geographic photographer or a traditional cinematographer, and I got a light meter and some spotlights and I really thought maybe I'm going down the career path and the computer graphics bug, bit again, you know, there was no way I was ever gonna give it up, but I think looking back, it was good because now what we see with virtual production is these worlds, shaking hands, , the principles of live action filmmaking that have existed since the Lumyer brothers. You have that on one side and then you have, unreal engine and VR headsets, but who cares at the end of the day? We're all making stories. We're putting images on the screen. So what medium, does it really matter? And there's so many great between these extremes that, to categorize them is really of secondary importance to just using the tool that's right. For the client and for the job.
Noah Kadner:Schatz embraced the multidisciplinary nature of his education and, as a result, found his skills in demand right away.
Casey Schatz:So every year, , the Kal arts animation department has the end of year producer show. And my short film showed at the producer's showcase.. And the hallway afterwards, , Tom bear who started image G, which is a motion control company, came up and said, can you start Monday basically? I was so excited. I hardly knew what MOS control was and I didn't even really know what previ was It was still a very young, art form at the time. And when I started there, I started to do previs for car commercial that were being used with this huge, like three ton motion control camera, crane. And the way that the offices were set up, , there was a window and then the rig was in the parking lot. So literally as I was. Animating the real thing that weighed three tons and had to move was right there next to me. And so that instilled this sort of discipline to not get too seduced by the tools that the computer had, like no gravity and, velocity was not a, a concern, , so I started prevising with real equipment in mind. And the two things have always been embedded to do previs for things that are actually suitable. And if if they're not. That's fine. I mean, let's have a superhero take off and fly to outer space. But let's then break it down into its constituent plates and think about how we're gonna photograph this the worst thing you could do in previz is to make pixels move on this green in a certain way. And the director says, Yippy, then you get to set and you realize there's no bearing and reality with this. We can't accomplish this. the, The computer a person didn't have a respect for the physics and the parameters of live action shooting.
Noah Kadner:Along with learning about the mechanics of motion control and how it should inform accurate previs, Schatz also took lessons from how to operate in front of clients professionally.
Casey Schatz:So I had, been the cinematographer on this, music video and we went to a Tein place in Hollywood. It was a really swanky, fancy place. what really changed me was seeing. The operator work. we were just sitting in these, really cushy chairs and talking about the color balances and, and let's add a point of, know, red and,, subtract a point of this and that. And the person was just doing it and you could see, the impact in real time. And I. Thought . why would 3d graphics be any different? So , all right, I just need to get fast enough at Maya so that the client can be in the cushy to chair behind me. And I'm just like that. Tell any guy I'm moving the mouse in the keyboard really fast. And the image is just forming sort of like developing in the dark room and if I failed to do something really quickly, While live with a client, I would say, well, that sucked. How can I automate that in, you know, Mel and Python, I would write my own little tools. Jump cut to now a lot of that's happening with virtual production, because rather than them saying. Okay. Let's boom, up let's counter tilt down. We could actually hand them an iPad or something that's, optically captured and say, well, go for it. But also, there are still folks that are shy about the equipment and want to just sit on that couch or whatever, and, sort of dictate what their, vision is. And everyone works differently and one's not better than the other.
Noah Kadner:Schatz excelled at his work with creatives and was ultimately offered a position on Avatar, a movie that would become known for its pioneering work in motion capture and virtual cinematography.
Casey Schatz:I was like, I haven't done that. Project yet where I worked with one of the master directors, you know, and I, wanted to see if I could do it, if I was up for it. Just to even be in the same building as Jim Cameron was gonna be an honor. So talk about trial by fire for virtual production, had Some experience editing mocap, but had never been on a mocap stage. So on day one to see horror horses in mocap suits and then the actors also in the mocap suits, riding the horses and they're, they're going full speed down. The length of the stage was, a pretty remarkable, education. I ended up being like, I think my, credit on the film was SEL cam supervisor. So simultaneous camera. This would be sort of the art and science of capturing the motion picture camera much like you would, , a person by putting funny dots on their body and , they get wronged and calibrated and everything let's do that same thing to the camera. So those, active markers are put onto the camera and that camera is being captured. So I gots super duper involved in thes cam and also was also doing a lot of the motion based motion control camera. A lot of the tricky, live action setups. So yeah, I was on avatar for three and a half or four years. I mean, that was a life changing experience, getting to learn from, Jim and all the other, folks on the crew. And, I still watched the movie, like, did we do that. How did we pull this off? You
Noah Kadner:The Avatar team invented many techniques which are now considered standard practice in the virtual production world.
Casey Schatz:I'll never forget The first really successful moment on thes cam, which is the shot where, net Teri goes up to the shack because the, real Jake is, breathing the Pandora and air and, suffer. And so she runs up to the shack, looks through the window, sees him. Goes to the stairs, jumps and holds what is now, the live action, same Worthington character. And it was very, very tough. our virtual production engineer and all around just brilliant dude. Brian Champney, he and others, made this, optical volume in a very claustrophobic situation, with optical volumes, you want a lot of space. You want the, field of view of the lenses to have a nice spread so that they're encompassing a lot and overlapping area, and that's very tough and a very claustrophobic cuz the shack was real. The art department made . A real shack. And, we got it working and the live action cameraman, I think it was Dave Ericks, was able to see net teary and compose the shot with a CG character. As good as an operator as Dave is. And, and many camera operators, are it's hard to perfectly imagine it and do it without seeing the digital element and that was kind of the moment that we really got it and it was, , the culmination of a lot of testing, a lot of hard work , from Brian, from Glen D from day on from just all these remarkable fun.
Noah Kadner:Schatz's focus on visualization techniques ultimately led him to join The Third Floor, one of Hollywood's most successful visualization companies.
Casey Schatz:Yeah. So third floor was, , formed by , the five, , founders. So it's, , Chris, Josh, Barry, , Hirosi and, Nick, and, , I had been Mr. Solo previz guy , for many years, , after image G and before digital domain, I was just sort of using that, , mentality from the Teles, like I, up with a laptop and set up my monitor and you just start talking and think, you know, I was, I was loving it, but previous was becoming more popular. And I had lost a couple of gigs because I was only one guy and it went to, a previous company of sorts. And then when I met all the folks from third floor, I felt like we think alike, we all come from some kind of traditional live action, medium and really love. Previs a sort of a hybrid of traditional live action and computer graphics. So why would I compete with these people? Let's just all become friendly. And I just joined the third floor
Noah Kadner:One of The Third Floor's most successful collaborations has been with the wildly popular movies of the Marvel Cinematic Universe. Marvel's a blast to work for. We've had a really good synergy with them for a while. Now. I think our first show with them was iron man, two a while back. And, they take their previous very seriously, which we love, of course. We're very lucky to just have these amazing previous soups Gerardo Ramirez, Shannon justice sins who have the ability to absorb the script and, break it down kind of into these component pieces. So. what assets need to be modeled? What animation cycles could we potentially reuse or what new ones might we need to make? What I've personally seen in terms of the approach, is starting soon. And progressing the complexity very carefully. So starting with storyboards and editing those into an animatic and those have temp music attached and right away you really get a quick sense of the pacing and the story, . Those might one by one become replaced with stills from Maya. This might have a . Version, one of the environment, the characters in the props are whatever state they're in but. rendering it through the legit focal length and the legit cameras. Then we might start doing. Posed a pose sort of animation where maybe they look like they're ice skating and they're going from point a to point B in the correct timing., but these are all these simple iterations. The point of all of it is, we want to increase the complexity slowly, such that if any one moment. You were told, we've made a big change to the scene or the scene's been replaced by another one. Like whatever the situation may be, your heart's not going to be broken . Because you've put all this energy into the scene. And, another thing is we try to animate in master scenes. So as opposed to just animating , one shot at a time, we might try to have one file that encompasses an action beat. So this has a number of benefits. It tends to be better for distributing to artists with different skillsets. So, , let's pick on me for example, I'm not a particularly strong character animator at all, but I might be able to inherit that scene and then start doing camera and layout kind of stuff. Or maybe I would break the scene down into a tech. And start to think about, well, when we get to live action, how will we really shoot this on set? The importance , of teamwork with this whole workflow, can't be understated, especially on projects of this size. Schatz continually refined his approach to previs and virtual production over many Marvel movies but found one project that opened his eyes to new areas of inclusivity with other artists.
Casey Schatz:The Avengers age of Tron, was a very, very, very. Rich experience an important part of the definition of virtual production for me is opening computer graphics to non-digital people., so actors cinematographers, whatever it is, we will record your talents in the computer so that we can fuse all the years of experience that you have and said artistic, traditional medium inside of the world of computer graphics. And so something that really, really demonstrated this to its fullest was the first time that we put James Spader in the mocap suit. And so James Spader played, the bad guy Tron. And we put him in the XN suit and we had the live feed going into motion builder, and we had, one of the sets. It was very fiery and dark and scary. And, we had a big, , flat screen TV that was showing the motion builder output. And he asked me to arrange the perspective camera so that when he looked at the television, he was looking at himself. So it was essentially like a mirror. So the camera was about where his eyes were in the CG, in our, , and he just started looking at the TV and pacing and you could just see him inhabiting. This character and studying it. And he's, , this encyclopedia of knowledge of acting and, Shakespearean references. I mean, he's an amazingly smart guy and there were Marvel . Execs in the room but everyone just sort of fell silent. You could just see James started pacing , and becoming ultra on. And he was studying the movements and how he walked and how he articulated his body. And I thought. This is so cool. And I thought that was a really amazing moment to describe the power of virtual production
Noah Kadner:Another project Schatz worked on was Game of Thrones, which became well-known for its seamless blend of gritty realism with fantasy elements like dragons.
Casey Schatz:Something really remarkable about that show was this almost obsessive dedication about getting as much in camera as we can and come coming from previz and this sort of marriage of traditional photography and computer graphics. I've never been a fan of the fix it and post mentality. I always say fix it in pre, let's do our homework ahead of time and then go to set, with a battle plan and get that shot. And then computer graphics helps it later on rather than rescuing something that wasn't shot from the correct perp active. And so even when it came to,, shooting, , fire, , We were actually shooting real fire elements. And so I think the first gig with the game of Thrones was I think it was season four where, one, one, , the big, , giant, , walked past the camera. And so this was, , an old fashioned scale trick where we had an actor that was six feet tall. I tracked the live action element. that was shot with a fella running with a tennis ball. So that's how the eye line was done. Fella running with a tennis ball, tracked that, and then scaled it down to 50% and then , aligned the perspective. So when that monster walked by, he looked like? a 12 foot giant again, all old fashioned stuff, just being used in, in a smart way. Every trick in the book, whether it's old school or new school, we would pull it out and it was such a blast to do. And then, season five te de ended up being used for a lot of stuff in G O T anytime., someone's looks like they're walking right into fire. Well, they were photographed and then they went away someplace safe, and then all the fire elements were there, but we kept the camera there and repeated the camera move,, Noah Kadner: The Mandalorian was another the interest in in-camera VFX and LED volume workflows in combination with other virtual production techniques. So on Mandalorian, third floor was working on the previ and I was sort of the, motion capture tech vis ambassador between, third floor and the, live action crew. is something else about virtual production, which I think is changing in an exciting way, is that. It's no longer this, separate discipline that you go down to this room and there's a grid on the floor and everything is gray and there's nothing wrong with mocap volumes. They're vital. But, , the idea that an animator in their office can say, I'm gonna throw the suit on and just jog back and forth and capture. This is also something that I think is really amazing. So we have a few of the, XN suits,, which is the inertial mocap system., so really my role , was, getting those suits working in the previous pipeline and doing a lot of the, , , tech fi for when they're on the blurbs., how that motion base. Then feeds into the corresponding background on the L E D screens because it has to move the other way to elicit that sense of, for travel.
Noah Kadner:Schatz has come full circle for his most recent project and is working with many of the original team on the highly anticipated Avatar sequels.
Casey Schatz:There really is a family vibe., , , a lot of us , are still very good friends, from, , working on avatar one together., and, , , it was probably inevitable that I would join it. It would be weird not to. What I think I love most about my job is being on set and not being , the know itall with the laptop and the fancy computer software. It's actually the exact opposite., the way I see it is everyone else are experts at their respective disciplines, construction, cinematography, the gaffer, , the art har. So what I love most is when everyone's over my shoulder and everyone's got a different little bit of feedback, like, well, okay, so for stunts, they need the pick point to be over there, but oh, no, that's gonna get in the way of the great off that lighting is gonna set up or whatever it is. And we solve these problems digitally right in there. Before we really, and truly execute it on set. And , that feeling is way more gratifying than being the show off that has all the answers.
Noah Kadner:Asked if he has advice for aspiring filmmakers and visual effects artists, Schatz suggests interacting with and observing the real world.
Casey Schatz:Turn the computer off, get outside with real optics, with a still camera or a motion picture, camera of sorts, not your phone, not a digital thing. Like get something where you have to think about the shutter speed, the F stop, the ISO, how did these three things, , react? How do these things overlap and why when you change one thing, , does another have to be changed to compensate,, why, when you're close to something with a wide angle lens, and you're far away with a telephoto lens, you actually have the same field view, but obviously all the objects that there are parent distance from each other, it's a different energy. What would be conveyed from a narrative standpoint? One might look like, a voyeuristic, like a spy. And the other is more of a documentary feel where you're really there in the action, because as amazing as computer graphics is nothing will place the sort of magic of being on a live action set, , working with a crew, collaborating and having ideas and bouncing ideas off of each other. That's something that no computer techy thing will ever replace. Being on a live action, set, you open computer graphics up to these sort of happy accidents, which you normally wouldn't I think that's something really special and vital about virtual bringing that sort of messiness, so to speak, , from live action into the digital arts realm.
Noah Kadner:Finally, as someone who's spent an entire career on the cutting edge of virtual production, Schatz is uniquely qualified to speculate about its future.
Casey Schatz:For me the best way to look into the future is to actually look back at the patterns and milestones we've seen in recent history and kind of extrapolate that curve. So for instance, after avatar, the most common question was, Hey, that's really great, but what if we don't have that kind of big budget and rightfully? So this was seemingly outer space stuff on a big production. But the important thing is that the crew. Tech footprint got smaller. And many of us went on to real steel, which was not a big VFX driven movie. And we were able to stand on the shoulders of what we learned during avatar shortly later, , and cam arrived on the scene and that enabled you to, , track without an optical volume. So the theme, I guess, is that tools that were seemingly inaccessible only a few years prior are now being utilized on more and more. So what we hopefully did for Simon cam on avatar. Clearly Mandalorian has done that for led volumes today. And so this also goes for, , like individual pieces of tech as well. So a big one for me personally, has been LIDAR and photogrammetry. Which only a few years ago were really prohibitively expensive, not just financially, but like in terms of time, but we got the Leica BLK and , I've just been obsessively, scanning everything ever since, you know, I can have a mesh in my scene half an hour, and we've been using that here a lot for like technical layout, which then in. The simulcasting lineup. So there's this really cool feedback loop between reality and digital and inform each other. And now even the iPad pros can do LIDAR scanning, which is just amazing. And I think also in fairness, I think the XN suits have done a similar thing for motion capture, where you can get, really good capture without an optical volume. There's always a need for good mo-cap volumes and we're using them here daily. But I think what excites me is being able to do this stuff from home or the office, , or even right there on set. And, just opening computer graphics up to more artists from various disciplines. So what we're seeing is kind of this blurring of the lines between virtual production, previous and tech. Such that maybe what category each fits into. It doesn't really matter anymore. We just have this amazing toolbox and we are just going to pick what's right for the job.
Noah Kadner:You've been listening to the virtual production podcast. Special thanks to my guest Casey Schatz for joining us.