François Fripiat: “The Essence of Spatial Sound in VR Storytelling”

François Fripiat has been a sound engineer for over 12 years. He worked in many different fields such as sound design, audio mix, and he's been a sound supervisor for many different animation films, feature films, documentaries, etc... Passionate about spatial sound for a decade, François Fripiat founded the company Demute, a creative sound studio dedicated to new media (XR / Video Games / Immersive experiences). Demute is focusing on tech by developing new technological tools to improve immersion through sound. Our VRTL podcast host Pieter Van Leugenhagen will dig deeper to learn the different aspects within spatial sound and how it contributes to the storytelling of a cinematic VR experience.

Transcript

Speaker 1: 0:06

Hi everyone, this is Peter from virtual and we are ready for a new podcast series with me here. Francois PR, uh, founder and CEO of the mute. Hello. Welcome to our show friends. So how are you and how's the muse doing?

Speaker 2: 0:20

Actually quite good. It's a, it's Friday, so I'm thinking about my weekend. I, we'll go to the, to the Belgian coast this weekend. So no, anyway, uh, we are doing pretty well. We are, uh, creating quite different, uh, exciting sound for different production and making also, uh, innovation. So yeah, three going well.

Speaker 1: 0:40

Uh, can you tell us a little bit more about a particular project that you're working on?

Speaker 2: 0:43

So, uh, for a moment we are working on, um, on project. We are finishing some Playmobile, uh, VR experiences that were, uh, VR companions to, um, to the[inaudible] build the movie. Uh, we are working on motion capture of VR animation. Uh, also we are working for video games, so some video games that appeals sought out because we are working for VR and, uh, video games. So our focus is immersion and interaction. And uh, yeah, that's pretty much that. We, there is also a, a great, great VR game. The real sold out[inaudible] is like a street fighter of fairytales to treat fighter in VR. It will be exciting in, uh, and the result out in six months or something like that. And uh, and yeah, that's about that.

Speaker 1: 1:30

Nice. Sounds good. So you're working now as a sound engineer or in the sound space for more than 12 years. Uh, so how did you end up in a spatial sound and where does the passion come from?

Speaker 2: 1:41

And it's a, that's a good question actually. Uh, I'm, uh, I worked in the cinema industry and uh, I made like all the difference, uh, rulings, samples, production, so they'll being in advertising in a, in a mixing and create studios or run the role, et cetera. And um, actually that was my, my, my thesis at the end of my school. It was about ambition examined and then I went to, uh, to the business, so in the sound business and nobody were talking about that. And then boom, VR just came out. So these people that, uh, were in the same building as the company I was working for, uh, people from, uh, no zones. So they created a system for us in prisons and anyway, and they just, uh, knocked the door and saying, okay, can you do me some special sound for VR? And I was like, what does VR, so, uh, and of course, uh, I learned about that and I, I can I check that, um, one of the technology was ambisonic sound so special sound and yeah, I was like, okay, let's, let's launch a business about that because it, that was something that was pretty exciting, uh, already for, for 12, 12 years.

Speaker 1: 2:48

Is there actually a difference between spacial and binaural sound?

Speaker 2: 2:51

Yeah, actually the, there's not a difference, but we have to, to compare that to image because image is really more clear than sound. So you to see[inaudible] it's like binocular. So binocular clear. It's just seeing that you are looking with two eyes. So my role is just saying that you are here with two hears. So that's it. So by the role as nothing about technology inside that so special sound is the fact too, to have sound all around you. And then thanks to decoding and things to um, to special computing thing to, to, to computer, uh, to computing. Uh, thanks to computing, you will be able to um, to decode the sound in binaural. So binaural is all we hear sound in real life actually. So by the role is the, the hand of the, of the system. Uh, if we have a barrel. So I compare that special sound in fixed by NREL. So by fixed by neurol is something that is orally pre-calculated. So it's like if you have an image, a treated image, and where did you turn around your every time seeing the same image. So buying fixed binaural sound is not interesting actually in what we do in VR. What's interesting is, is special sound and that will thanks to the position of the head, we decode that to bind role. But it is important for VR. Yeah, indeed. Because actually we have to think that sound is the, is the sense of danger because in the real life you hear all around us, but we see only in one direction. And so sound is what, uh, gives us the, the intention to turn the head and to look at something. So sound is every time open. You don't have any, uh, any, you cannot blink with, uh, with, uh, with years actually. So you have to, um, to be really, really smart about sound and this is what gives you also a very interesting emotional triggers.

Speaker 1: 4:39

Okay. So actually unique. Think a lot about, um, how to integrate sounds in VR experiences. It's not just stupid.

Speaker 2: 4:48

Yeah. And a, and not only in term of technical purposes because actually techniques behind the special Odo exists for already two to at least 20 years. It's how it's implemented and how, it's interesting to hear that sound more than another one. When I'm looking at something, do I hear this sound on the higher the, the sound, just uh, uh, next to him. Uh, and all of that. It's also interesting in term of creativity. So just don't think that you put a sound in unity attached to a graphic design and that's, we'll make the job. That's interesting to have to go further than that and saying, okay, what sounds give in terms emotional triggers or Oh, sound can help me to be immersed inside of your chill world.

Speaker 1: 5:32

Are you also exploring augmented reality projects? Three years ago

Speaker 2: 5:36

we had an ID that was okay that, that's not very usual to have a, have a headset in front of her or of our eyes because, uh, we are not used to, to be blind somehow, but we use Walkman for four years now. So every people is listening to music inside a sang transportation. So we were thinking, so why not thinking about seeing the reality, but hearing virtuality, and this is how we are building now a system that's called[inaudible], this augmented audio a Realty. So it's a device, an audio device. So[inaudible] phones, I distract on the room track depending the position of your head. So a twist, six Duff, uh, augmented reality experiences. So you see something real, but you hear something, a, something, a virtual. So one of the key, uh, the key use case that we can think about, it's a Bruegel, uh, painting. So you have big Bruegel painting and when you are, uh, looking at the regular painting, just walking around the, the painting, you are hearing every difference, uh, characters in there. And if you are looking at the painting from far, you are listening to the wall, to the wall picture itself. So that's, that's things that are possible with that.

Speaker 1: 6:50

So, uh, I think the last time we saw charter was at gun X, our sort of come film festival

Speaker 2: 6:56

quite, quite late February, remember?

Speaker 1: 6:58

Yeah, it was a very interesting time. Um, but I, I, I attended your talk about the opportunity for spatial sound and podcasting. Can you tell us a little bit more about that?

Speaker 2: 7:08

Yeah, actually it's a, it's how we can bring, um, also augmented reality to simple devices because no, we need, uh, complicated visuals. So forget the visuals. But even what we are building right now is still a net set with different device on it. How we can have a Naga mounted reality experience just with a smartphones. And so special, a special podcast. So actually it's quite simple to, to, to create somehow because you just need a real good sound that is special and things to do. Gyroscope of the telephone, you can be in another reality. So you can think about podcasts that are, uh, triggers depending on where you are in the map. So let's go now we go to a[inaudible] or we go to, uh, uh, regs, museum, whatever, and just go in front of the building. And then the post gas in there is sound is just triggers and that and where the sound is precisely located, uh, where the building are. So you can actually just traveling time, for example, just thanks to sound in things to your imagination. And there is podcasts. So, uh, triggering the sound exactly where they are in the real life. So that's, that's an opportunity. For example, the only thing with special podcasts, uh, you need at least one of these two things. Either you need to be able to work around because you need to turn the head. So it's something that you probably don't do inside your car or inside the Metro, uh, that is to place where you, uh, listen to podcast. So either you have to walk there, you have to, to be able to interface to, um, to, to be with. So let's say you can have an interactive story. Uh, you choose option B or option a inside your phone and you can be inside the virtual audio world thanks to an application. That's also something that could be possible in term of special podcast.

Speaker 1: 9:05

So, um, the mute is one of the leading sound studios or agencies in Europe, uh, are any projects that you're proud of and that our listeners can watch somewhere somehow?

Speaker 2: 9:16

Uh, yeah, there is actually a, for a moment we worked on, on very different projects. Somehow it's secret project or we can not communicate on it, but a project that I can talk about, it's a probably construct that is pretty exciting project, uh, cause it's a robot fighting all around in a six Duff's cinematic experience. So in term of a sound design, uh, that was pretty exciting. So sometimes for a sound designer it's exciting to make sound that was really noisy somehow. Uh, we worked also on video games, more regular ones, but it's actually the same techniques because we use, we can use special sound not only for VR but also for, uh, for video games, uh, regular ones. Uh, so we are working on a, a Flemish project. We finished that floats, Sam and the RA and the secret of seasons that will be a Zelda. Like that will be probably one of the biggest in the game, uh, of Belgium also. And uh, yeah, we, we'll have pretty exciting projects coming in, uh, in 2020.

Speaker 1: 10:13

Nice. So actually on a, on a, on a scale, uh, of one to 10, how important is spatial sound for immersive experience?

Speaker 2: 10:21

I have to ask me to do the kind of question, you know, whether what I will answer, actually I think, uh, I think it depends on the production in cinematic. It's, it's, it's crazy how it's not a understand yet, uh, or, or unnecessary. It is because actually this sound that is really projecting you inside another, uh, another world just in terms of emotion. Just think about, um, you are, you are, you're just writing, you are in the car and you're just listening to music. Depending on the music that you are listening, you have a complete different emotional uh, behavior inside the road. Socio. My Spotify playlists. Yeah. So that's completely crazy. You are thinking stuff differently. You are then you are in the real life, but you are immersed in the, in other emotional state and the that's that's how, I mean there's just a silly example to say whole sound is inter important in terms of immersion, but it's, it's when it's become, to be really being in another world, a sound can help you like lot. And so think about more about the sound. It's pretty exciting and something that is really important to say. Is that recording sound? A sound field recording for VR is kind of tricky. It's complicated because you don't have boom microphone. You don't have people that can have handle the microphone because you see all around you. So it's you have time, microphones. We have a small microphone that you can have on people, but that's not sounding very well. So the best, we have a real good experience in term of sound, it's rerecord people afterwards

Speaker 1: 11:52

how that work. So

Speaker 2: 11:54

actually it's like, um, it's funny because, uh, it's, it's really something that is used in mainly old, um, all the American movies it's used. So every Marvel movies use that kind of stuff. So they record onsets, the sound just to have the information of what the[inaudible] said and that the actors is rerecording we call that ADR in a, in English, uh, rerecording the, the, the, the voice that you was doing on the set to have the perfect voice with a really clean, uh, sound booth.

Speaker 1: 12:30

Can you tell us something more about the heart and software that you're using?

Speaker 2: 12:33

So actually, uh, it's um, there is a bunch of different software, uh, all around. So there is the Facebook three 60, uh, there is a Google resonances. So actually the big guys is positioning on that. We are more using stuff from, um, from people call the blue ripple sound because we really do like that for, uh, I mean for a cinematic experience. When it's more interactive, we go to a unity. Indeed and real are we wise. We are the only wise ambassadors in a, in Belgium. So it's really the sound interactive sound for VR and video games, uh, dedicated to soft software. So you connected that to your, um, to your a game engine. Uh, so yeah, that's pretty much that. And for creating the sound, we use a Reaper. Uh, so pro tools is the main, uh, the main digital G workstations. So the main software inside the, the, the, the industry. But we use Reaper. That is really much more interesting because you can parameter everything inside that. And so because we are in the innovation, we need to have a software that can, can, can be adapted on, on, on our needs.

Speaker 1: 13:45

Uh, so very exciting. What you're all working on. Um, and what, uh, has the future to bring, um, for the mute.

Speaker 2: 13:52

So a, I think for us, the future will be definitely to have more and more creative that can helps us to the different creation that we have. And we are a really exciting to push our creation or devices that we create. So we are creating hardware, software and we really would like to, uh, put that on the hands of creators so they can build amazing sound experiences that we can, the experiences in museum, in exhibitions, in, uh, in the, in music, in a aquariums, in zoos, uh, every time a all around. So having the sound that can bring you in a, in another world. Uh, and I'm pretty excited to see how the Republic will react to that.

Speaker 1: 14:34

Cool. Does, does that this mean that you are actually transforming from a service company to more product company?

Speaker 2: 14:39

Exactly. Actually relation, uh, it's a, it's exactly that. It's not easy to explain that to investors sometimes because this, yes, like a service company, but we are really innovative company and we a product company, a Maura and that's, that's, there's the drift that we are, we are doing for the moment. So really focusing on creating a product, uh, innovative products. And the exciting part of it, saying, okay, why a small SME in Belgium in Brussels can invent something new. And you have big guys in Korea in United States. It's by being lean in, by being really creative and by having amazing talent in the studio that are really focused on sound and innovation. Okay. So thank you very much, Francois, for this conversation. As much pleasure, you're more than welcome to come. Cheers.