Development in VR: VR Audio

It is odd to hear sounds behind you, but it is actually on the screen in front of you. But that’s precisely what happens with Dolby 5.1 and gaming. We have grown to accept it. But it is artificial.
In VR, you want to hear a character coming up behind you. If a character is 100 feet away talking, you want to feel and hear them as if they really are that far away IRL. That is why the concept of spatial audio in VR is huge.
Spatial Audio
Sound is incredibly important in gaming. Yet, some developers do not realize how imperative it is. They’ll work on a game for a year, then show it off without sound. “Audio design is extremely important,” Dan Shimmyo says. “And if you are not thinking about audio early, I feel like you are failing to develop your game.”
In VR, audio is vital because players need to hear where sounds are coming from. Shimmyo describes a perfect example of this using zombies. “You want to be able to hear a zombie coming so that you can protect yourself and deal with them as quickly as possible, expecially when they’re coming/sneaking up behind you. And what could be more terrifying that sitting in a dark room but you can hear these zombies coming from all sides?”
If a player is holding a lightsaber in their hand, they should be able to hear sound like it is really there. Spatial audio is a fantastic part of the immersion of VR. There is a significant difference between looking at something two-dimensional and hearing a sound over here on stereo. It may work, and we may accept it in regular gaming. But in VR, we want technology to advance to full immersion for the best possible experience.

Engaging the Player: Audio, Haptics, and Information: Accessibility

Adding accessibility features when you’re developing your game can help make it playable by a larger audience, even if they aren’t diagnosed with a specific condition. You may unknowingly be limiting your audience by not adding these features.
For instance, many high performing players will play with the colorblind mode on even if they are not fully diagnosed with color deficiency. Colorblind mode has a higher contrast which makes it easier for them to play.
Even for those that aren’t colorblind, when people get in a really stressed situation its harder for people to see color. It’s as if your brain goes into a high-octane mode and optimizes for moving quickly and efficiently. Color information is a lot to process, so you stop being able to see color clearly when you are pumped full of adrenaline. This is the reason why when people are in a situation where someone has a gun, they might not actually remember their shirt color, or if they were in a car accident, they might not remember the car color.
Another time when people can experience a lot of adrenaline is when they play video games, especially if they are in a stressful firefight or a large boss battle. Because of the adrenaline it might be harder for them to perceive color. Running colorblind mode can help in these situations.
Another area of accessibility that’s becoming more popular in recent years is around cognitive differences. People have differences in short-term memory. You might tell a player on one screen that you need them to do this thing, and then they move to another screen. There is a sizable percentage of players that might not be able to remember those exact words transitioning from screen to screen. This is important for game developers to be aware of so you don’t limit your audience from being able to play and enjoy your game.

Engaging the Player: Audio, Haptics, and Information: Music and Experience

Audio and haptics play a vital role in creating a full, sensory experience in games. Genevieve Johnson explains how these senses can define a game – and even how gamers move through virtual spaces and engage emotionally.
“Besides the sounds, you want to think about the button clicks and the sound they’re making. Think about the music in your games – music is a huge deal. I happened to play maybe six different zombie games for a minor research project just a couple of days ago.
“In one particular game, it had solid gameplay – most of them had solid gameplay – but its music was mostly sort of soothing. So you felt sort of soothing, calm, like, alright, I’m getting ready to go to battle these zombies. And it was a long, long battle.”
That juxtaposition removed tension that would have otherwise made the game unplayable while setting a tone that still worked with the horror.
“There’s no difference between music in a game and music in life,” says Dan Shefelman. “If you’re working out, you want to hear a workout mix. If you’re trying to relax, you want to hear calm music. It emotionally creates a feeling in you.”
Haptics – the understanding of the world around us through the sense of touch – works similarly. Shefelman details how haptics shapes so much of modern gaming, and how it can change depending on the medium.
“My first experience with haptics, or, I should say, with the lack of haptics, was in virtual reality . . . I couldn’t tell how far anything was because I’m used to sculpting. I’m used to feeling something that I hit. Until they have developed haptics, you’re in a three-dimensional space and you’re not feeling it. It feels vague.”
Now that games are becoming more of a sensory experience, audio and haptics have become just as important as optics. When you’re immersed in a game, you want to feel something at the same time.

Engaging the Player: Audio, Haptics, and Information: Touch and Feel

“We’re saying something about something here,” says David Jaffee, “We’re saying something about the human condition.” Fundamentally, it’s what feels good when you’re holding that controller. That’s what most gamers love. Rocket League, Fortnite, PUBG– it’s fundamentally about mechanics.

Audio design is tricky. You know when you want the sound to show up and do a job. You know when you want to sound to evoke an emotion in the player-a reward– a warning specifically. Usually, it’s the audio and heads-up display that says, “Hey, a missile’s coming in,” or “a bad guy’s behind you or off to your right.”

Designing a HUD is really hard for a game designer. Because the game designer wants to think that the player wants to know everything they know about the game. It’s like there’s so much stuff that the game is tracking.

And you want to throw all that at the player and say we have audio cues, we have visual cues, and we have heads-up display cues. Did you notice the player’s feet glow a little bit when he’s low on health because the eye tends to be looking lower than the center of the screen?

And you can get carried away with that. The average player might ultimately– when they get good at the game– want that data. But if you throw that at the player at the very beginning, they’re going to get overwhelmed. The game is going to end up looking like user interface porn. And you’re going to push people away.

It’s a really hard thing to do, especially when you’re talking mechanics-based games, to design something that’s robust enough to yield a play experience that the player can have a meaty, deep, intellectual, satisfying time with but without overloading what’s on the screen. Because the minute we went over the shoulder or the minute we went first-person, we lost that brilliant play mechanics space.

These are not movies. And they’ve become movies because they look like movies. But what we’ve lost in that is so much that is germane to great gameplay.

Introduction to Television: Episodic Television

To understand the process behind creating and filming episodic television, director Harry Winer had to see the “rhythm” behind it.

“What happens is that directors are hired on a week-to-week basis. You come in and direct one episode. You leave. Another director comes in and directs an episode.” Winer says. “For a television series that goes on for years, the actors themselves are the ones that are running the show because they know their characters.”

When Winer came into doing his first TV series, Hart to Hart, he had no idea how television worked, so he treated it as if he was making a movie: “I go in and start setting up these elaborate shots.”

“So I get called into Leonard Goldberg’s office [at 20th Century Fox], and I’m sort of intimidated,” Winer recalls. “And he says to me, thank you. You directed a beautiful movie, and maybe one day we’ll get a chance to actually make a movie together. But right now I’d like you to direct Hart to Hart.”

That was one of Winer’s biggest moments of education. “I had no idea what he was talking about,” Winer says, “but subsequently came to learn that in that era, episodic television was a product.”

There is a standardization of the product — of television — and there are certain rhythms and ways in which a series is filmed that is the emblematic style of that particular episode. And subsequently when I recognized that, I started playing ball and would make several more Hart to Harts.

After directing his seventh episode of Hart to Hart, Winer realized that it was indistinguishable from every other episode of Hart to Hart. “That was the goal that they had set for me to achieve.”

That led him to discover an important lesson in the TV industry. “I realized that in serious television, there was a choice of being a really good series television director and implementing the style that others had created versus sharpening my skills of original creativity that originating a television series would allow.”

Introduction to Television: Television’s Home Invasion

In the late 1940s to 1950s, this thing called television invaded people’s homes, Alrick Brown comments. Film, on the other hand, went from this visual thing that eventually got sound. He says, “people were in their homes listening to radios and getting sound. Imagine what happened when a box came into your home, where an image was broadcast, and you could actually see pictures and people.”
Alrick thinks that television has evolved a lot. But he wants you to understand why TV was built as it tells you a lot about what television is today. At its core, TV was built to sell coffee, knives, vacuum cleaners, and whatever the American housewife needed in the 1950s. If they could just have commercials, that’s what TV would be, Alrick says.
They created television shows as sponsorships, as a part of selling these products. “And so this magical, wonderful thing that we call television, that we consume so much and shapes our understanding of the world, was created to sell products.”
Harry Winer thinks that television had a glorious introduction. Initially, it was film theater. In the 1950s, they were trying to figure out ways to capture an audience once they had this wonderful new tool.
They would take vaudeville comedians and other stand-up comedians and put them up for variety shows, Harry says. They tried to create legitimate storytelling. They would take plays that had been successful on Broadway, or that were part of the American theatrical canon, and stage them for TV.
Harry notes that this is where many directors of a previous generation to his own cut their teeth. For example, John Frankenheimer, Sidney Lumet, and a host of others went on to become wonderful filmmakers. They took that understanding of the medium and found new ways of using imagery within a very tight budget and tight time frame in order to tell stories.
[Please embed: https://unsplash.com/photos/HWhR6lbn5xU]

Introduction to Television: The Networks

In the 1950s and ‘60s, there were three television networks: ABC, CBS, and NBC. In the 1970s, Fox became the fourth network and changed the way in which the first three networks approached their business. There was more competition.

During that whole period of time—the 1960s to the 1970s– there was a certain kind of programmatic approach to content production. The networks saw what content worked and wanted to replicate it in everything they created. That was their product.

For example, they had a certain story to tell. They would cast a couple of stars and add a few particulars. They wanted to see the exact same story week after week, so audiences knew exactly what to expect when they tune in the next week.

Moving into the 1970s, content production fell under the auspices of the financial and syndication rule, where individual studios and individual producers were able to own their own product. There was an array of product that was available. But there wasn’t a tremendous amount of competition. Only certain buyers from the four networks were seeking out content.

In 1993, the financial and syndication rule was withdrawn. Networks were able to create and own their own product. Over the next few years, the networks started acquiring or creating production companies where they could then produce their own product. The networks understood that if they were producing the content, they would then not have to pay an independent producer for their creative efforts.

It was far more financially lucrative for the networks to be able to create their own production companies. They’d be able to keep the all money in the same loop, robbing the right pocket and putting it in the left pocket.

Looking to the Future of Film: Multi-Platform Storytelling

The film industry evolved significantly over the last century, from more theatrical presentations to more avant-garde Russian montages, to the advent of color, wide-screen, and now VR. The only way to move forward is to communicate thoughts or interpret those thoughts by looking to the past. From there, sometimes inspiration for something new comes.
Immersive VR Films – Emotional Journeys
Alfonso Gomez-Rejon describes an experience in which he felt like he saw something brand new. “Few years ago, I saw Carne y Arena,” Gomez-Rejon says, “which was Iñárritu’s VR short that was played at the Los Angeles County Museum of Art. And for the first time in I don’t know how long, almost like you saw the future.” The highly immersive piece takes the viewer on an emotional journey far different from the movie and documentary experience. Immersive pieces like Carne y Arena and Edison’s Black Maria take viewers on a journey to feel what they’re feeling innovatively and emotionally.
For Gomez-Rejon, Carne y Asada was an eye-opener. The invigorating method of storytelling is a true testament to the future of the film industry.
Multi-Dimensional Storytelling
Many consider projecting movies as the purist way of expressing stories to the world. But then, there are also ways online or streaming where you could spider web into short films that explore other characters or subplots that didn’t make the final cut. As one can see, there are various ways to tell a story. It is this Mobius strip of infinite possibilities and shapes.
There’s something about staying current that is always humbling and invigorating instead of feeling that you’re stuck in the past. But hopefully, it makes you want to work a little harder and keep moving with it and the chance of expressing yourself and reaching more and more people.

Pre-production: Setting up the Set: Casting

“Casting a film is extremely important, as is casting the crew for a film,” explains David K. Irving. “The people you surround yourself with are the people that are going to help you make the film. So you’re only as good as the people that you surround yourself with.”
Casting a film has its good sides and its bad sides. The good sides are that if you have a good script, you’re going to attract a lot of wonderful actors. Actors love to act. They really want a good script. So finding a good script becomes your first job because that’s what will get you the best actors.
The downside is when you go through the casting process, you can really only cast people who’ve actually come into your office to audition for the film. So there are several ways you can get actors. One is, of course, if it’s a known actor. If you want Robert Duvall to be in your film, you don’t have to audition him. You know what he does. And you can cast him as a celebrity.
The audition process is a much more grueling process, where actors respond to an advertisement or to a casting call, or agents send them in for you to look at. You read these people. You give them sides, which is a piece of the script, and you try to understand in your gut, are these people going to work and be the best people for the film?
Once you start getting a sense of who your cast is, you’ll bring two cast members in together to see if there’s chemistry between those two people, which may indeed influence how you’re actually going to cast somebody in the role. One of the great taboos in our business is firing an actor on set. So you really want to make sure that the casting process goes well and that you feel very comfortable about the people that you’re putting in front of the lens.

Pre-production: Setting up the Set: Collaboration Among Departments

 “One reason that I love costume design in film, television, theater, opera, et cetera, is the collaboration,” Explains Durinda Wood. “I love to collaborate with the other department heads, and that’s what the medium is all about. You’re not doing your own work in your own room like an artist. You’re collaborating with other people constantly.”

The script supervisor is really important for the costume designer because she or he is the person that breaks down the script and tells you how many script days there are. That’s really important for a costume designer because then you know how many changes there are for a character. We know what the time passage is. How many years pass? Things like that are very important for the logistics that have to go into your design.

“Then there’s costume department. That would be my costume supervisor. The costume supervisor is the second person in the hierarchy of the costume design department, and that person is all-important. The next would be hair and makeup. Hari and makeup in film are separate departments. There’s a hair department, there is a makeup department, and then there’s the costume department. Good communication with hair and makeup is so important,” Says Wood.

“Then there is the production designer, and within production design would be set decoration, and I would throw in locations, although they have their own department. Locations, production design, set decoration is so important to costume design because that’s the background to your costume. You don’t have a costume and an actor in your costume floating in space. It’s in front of something. You need to know what that wall or what that space is because sometimes you want the costume to fade into the background. Sometimes you want it to be incredibly showing away from the background.”

In film, it’s much harder than theater because in film, at the last minute, they often change what the background is. They’ll come to the set, and they’ll realize they don’t want to do it on that wall. They’re going to do it in that corner instead. Sometimes you have to go to the trailer and see if you have something else that will be better with that background. It’s a little bit instinct that you have to rely on when you’re a costume designer.

“I’m always on the set when a new costume comes up because there might have to be a change at the last minute. Finally, there is the director of photography. I want to know how it’s being shot. I want to know the lens. I want to know HD. I want to know video. I want to know if it’s film.”