Engaging the Player: Audio, Haptics, and Information: Music and Experience

Audio and haptics play a vital role in creating a full, sensory experience in games. Genevieve Johnson explains how these senses can define a game – and even how gamers move through virtual spaces and engage emotionally.
“Besides the sounds, you want to think about the button clicks and the sound they’re making. Think about the music in your games – music is a huge deal. I happened to play maybe six different zombie games for a minor research project just a couple of days ago.
“In one particular game, it had solid gameplay – most of them had solid gameplay – but its music was mostly sort of soothing. So you felt sort of soothing, calm, like, alright, I’m getting ready to go to battle these zombies. And it was a long, long battle.”
That juxtaposition removed tension that would have otherwise made the game unplayable while setting a tone that still worked with the horror.
“There’s no difference between music in a game and music in life,” says Dan Shefelman. “If you’re working out, you want to hear a workout mix. If you’re trying to relax, you want to hear calm music. It emotionally creates a feeling in you.”
Haptics – the understanding of the world around us through the sense of touch – works similarly. Shefelman details how haptics shapes so much of modern gaming, and how it can change depending on the medium.
“My first experience with haptics, or, I should say, with the lack of haptics, was in virtual reality . . . I couldn’t tell how far anything was because I’m used to sculpting. I’m used to feeling something that I hit. Until they have developed haptics, you’re in a three-dimensional space and you’re not feeling it. It feels vague.”
Now that games are becoming more of a sensory experience, audio and haptics have become just as important as optics. When you’re immersed in a game, you want to feel something at the same time.

When Film Became the Real: “You make film in the world”

“After the 1920s and 1930s, the heyday of silent film and transforming into talkies, there was life going on,” Alrick Brown notes. He teaches his students, “you can’t make films outside of the world. You make films in the world.”
Two world wars happened in that time period. There was filmmaking that happened before World War I and after World War I. And then you have filmmaking that happened after World War II when the world changed a bit. A lot of lives were lost. This has a reflection on society and a reflection on storytelling in Alrick’s opinion.
As societies go through traumatic times, the art also changes and shifts. The Italians, Germans, and other people started looking at their stories and said, “let’s be a little bit more honest, a little bit more truthful, and not do this romantic storytelling that Hollywood is always doing.” Hollywood picked up on that. There were people in Hollywood who said, “yeah, let’s stop romanticizing. Let’s get a little bit darker and a little bit grittier.”
Alrick thinks that Snow White was one of the first color-popping films. Filmmakers had always tried to play around with color and different hues, even in the black and white era, to give a different feel of a film. But 1937 or 1938, when color started becoming this thing, another layer was added, he says.
Alrick thinks that no one can argue that you’re able to capture color now. You’re able to look at real life and think, “what is that real life that you’re going to capture?” But back then, filmmakers had this existential crisis that the world was getting a little darker, but the films were becoming more colorful.
[Please embed: https://unsplash.com/photos/05lgkS2eZRY]

Engaging the Player: Audio, Haptics, and Information: Touch and Feel

“We’re saying something about something here,” says David Jaffee, “We’re saying something about the human condition.” Fundamentally, it’s what feels good when you’re holding that controller. That’s what most gamers love. Rocket League, Fortnite, PUBG– it’s fundamentally about mechanics.

Audio design is tricky. You know when you want the sound to show up and do a job. You know when you want to sound to evoke an emotion in the player-a reward– a warning specifically. Usually, it’s the audio and heads-up display that says, “Hey, a missile’s coming in,” or “a bad guy’s behind you or off to your right.”

Designing a HUD is really hard for a game designer. Because the game designer wants to think that the player wants to know everything they know about the game. It’s like there’s so much stuff that the game is tracking.

And you want to throw all that at the player and say we have audio cues, we have visual cues, and we have heads-up display cues. Did you notice the player’s feet glow a little bit when he’s low on health because the eye tends to be looking lower than the center of the screen?

And you can get carried away with that. The average player might ultimately– when they get good at the game– want that data. But if you throw that at the player at the very beginning, they’re going to get overwhelmed. The game is going to end up looking like user interface porn. And you’re going to push people away.

It’s a really hard thing to do, especially when you’re talking mechanics-based games, to design something that’s robust enough to yield a play experience that the player can have a meaty, deep, intellectual, satisfying time with but without overloading what’s on the screen. Because the minute we went over the shoulder or the minute we went first-person, we lost that brilliant play mechanics space.

These are not movies. And they’ve become movies because they look like movies. But what we’ve lost in that is so much that is germane to great gameplay.

Closeup on TV Writing: Child Separation

When Yahlin Chang wrote about child separation in The Handmaid’s Tale…

she had no idea how much that subject would hit close to home. “Parents and children were being ripped away from each other at the Mexican border,” Yahlin recalls. “Refugees were being put in cages.”

Yahlin remembers writing similar scenes for the show. “There is a scene with rebellious women being put in these large cages, these holding pens,” she explains. “And they did look like the holding pens that we’re using to jail migrants and refugees.”

The show took the startling imagery even further in an episode called, “The Last Ceremony.” “I wrote the scene where June gets to see her daughter for 10 minutes,” explained Yahlin. “Her daughter has been kidnapped and is now living with new parents. This scene is both a hello and a goodbye.”

When Yahlin wrote this part, she couldn’t imagine that the same scene played out every day in the United States. “I talked to UN experts, psychologists, and international human rights activists,” says Yahlin. “We were talking about things that happen in Laos, Cambodia, the Congo, and Syria. We were talking about these incidents happening all over the world. But I never for one second thought that these scenes would be happening in America.”

The week that the episode aired, news broke that the United States was separating parents and children at the southern border.

“You’d see these scenes on TV of parents and children being ripped away from each other. This was happening in our own country,” Yahlin remembers.

When the uncanny episode aired, Yahlin got a lot of attention from reporters. “Suddenly, my phone was lighting up,” she recalls. “All these reporters wanted to ask me, ‘How did you know this was going to happen?’ And my answer was that we had no idea. We just spent a lot of time asking what would happen if you have the worst people in charge with the worst possible motives. What are the consequences of their horrible and cruel decisions? And so sometimes, our show interacts with the real world in extremely unfortunate ways.”

Learn from Yalin Chang in the online certificate course, Film and TV Industry Essentials. Grads get a certificate of completion from New York University’s acclaimed Tisch School of the Arts and learn from experts across the industry – including the pros at NYU, IndieWire, Rolling Stone.

Animation: 3D Modeling

According to Dan Shefelman, 3-D game design revolves around an important process: creating dimensional assets. This includes objects and environments but also characters.

“A 3-D CG character in gaming or animation is essentially a puppet,” says Shefelman. “Inside that puppet, you’ll find what’s called a rig. The rig is attached to the 3-D mesh defining the puppet’s geometry.”

The animator’s job involves manipulating the parts of a character and creating behaviors for different movements. Shefelman also points out some conceptual parallels between three-dimensional characters and real creatures. “The best way to think of a rig is as a human or animal skeleton. It has bones with joints that fit into sockets, and they can move.”

Shefelman further describes the existence of a hierarchy governing a rig’s joints. “One joint is parented to another. If a joint at the base of a finger moves, a child of that joint, like one at the tip of the same finger, might also move. The joints can also move independently, just like what happens in nature.”

According to Shefelman, modeling on the computer feels similar to working with clay. “You start with some kind of lump,” he says. “In a tool like ZBrush, this takes the form of a sphere. In Maya or 3D Studio Max, you generally begin with a cube. From there, you begin to extrude, or pull, the shapes outward one by one. After pulling out one shape, which we call pulling points, you’ll move on to the next. A typical workflow consists of pulling the points of a polygon and adding more vertices as you build the shape. Or you can focus on pulling complex forms, like a character’s limbs. With either approach, you’ll mold the starting shape into something that becomes your model.”

Animation: Introduction to Animation

Animation for film, TV, and linear storytelling is you controlling the character, creating keyframes for the posing, and carefully planning it out.

Animation in gaming is about creating a character or an object that has certain behaviors assigned to it so that it will behave a certain way when you control it.

Animation in the gaming industry takes a cold, dead environment and breathes life into it. Animation is not just how your player moves through the world; it’s also the world interacting back with the player.

People tend to think about the obvious, walking around or dancing, but animation is so much more powerful than that.

For instance, you’ve got an ax that you’re swinging. If you go up to a tree and swing the ax and nothing happens, you’re probably going to think that you can’t knock the tree down. If, instead, the tree wiggles and shakes a little bit, you’re going to swing that ax again.

You feel like you’re engaging with the environment, and the environment starts feeling real. You’re also cluing players into the game mechanics and which actions they can take.

You swing the ax again. This time the tree shakes harder – you’re going to swing it again. You have positive reinforcement that swinging the ax does something, and you should keep doing it until the payoff.

There’s a huge team that works on all these projects within the esports industry. And at every level of gaming education and planning, they have to consider the gaming experience.

A character designer and a modeler work together to create the characters, make them look appealing, scary, or whatever the game requires.

Next, the rigger puts in a rig, a skeleton that goes inside the mesh, the skin of the character, and creates movement. Then, the character receives its texture. Textures give the feel of leather, skin, or cloth, and they create the mood, the sense of danger, or lightness, of comedy.

Animation in gaming is different from animation in television and feature films. Each step in the process must take into account the materials of the objects or characters.

For example, as online gaming education teaches, characters made of metal will move a certain way based upon the characteristics of the metal. A program (coding) will determine how a metal character will interact with its environment.

The environment must be specified as well. Is it metal, wood, or ground? Gravity is also critical; it’s the most important part of physics regarding how a character moves.

If you’re on Jupiter, you’re going to move very slowly. If you’re on Earth, you’re going to move like we’re used to seeing people move. If you’re on the moon, you’re going to move lightly.

All of these details get programmed into the game. Whether it’s an esports, 2D, or 3D game, these things give the character the performance that makes the game exciting.

[Please embed: https://pixabay.com/images/id-5091194/ ]

Choices in Development: Engines, Languages, and Platforms: Game Engines and Environments

Game development has become a very welcoming environment. The game engine is the hub that runs the entire game and makes your game assets work. Anybody can develop a game as there are many different types of game engines and environments available.
If you are an aspiring game developer just starting out, you may want to check out Unreal Engine. This is a popular engine that is good for starters in that it is a bit more user-friendly. Unreal Engine involves less technical aspects to it as compared to other engines.
Another popular game engine that quite a few people know about is Unity. This game engine has a platform that allows you to create game characters. You can also add physics to the game assets using Rigidbody dynamics, and you can make your characters do different things by adding some code to your program.
If you don’t want to install a hefty software application on your computer, there are a whole host of game development engines that are web-based. You would simply go onto an internet browser, type in the URL, and then start coding. Once you’ve created your app or game, you simply download your file.
MIT App Inventory is a form of Scratch, which uses a block-based programming language for you to create apps and games. You drag and drop various blocks to build the code needed to run your creation.
Flowlab is another web-based application. Using Flowlab you can create game characters, add different movements for those characters, and create a full-fledged game. You can also access this application on your phone.
If you start working for first-party studios in the gaming industry, such as PlayStation, Naughty Dog, or Rockstar, you may work with proprietary game engines that these studios have created themselves.

Choices in Development: Engines, Languages, and Platforms: Unity or Unreal?

“Some game engines are better for creating great, giant worlds,” says Genevieve Johnson. Unreal was initially known for that, whereas Unity’s metier was smaller, boutique 2D games that could render smoothly on a tablet.
“But what we’re seeing today is that Unity’s excellent for creating large, 3D games. And Unreal can create wonderful games for your phone,” continues Johnson, “It then becomes a question of, how strong are you in coding?”
One thing that Johnson particularly likes about Unreal is its visual scripter.
”I tend towards using Unity as my game engine because I find it to be programmer-friendly. Whenever I have a unique mechanism that I want to work on, I can effortlessly write the code from scratch. That may be more difficult for some people. But to me, it’s very freeing,” says Dan Shimmyo. Unity enables him to try lots of things in a flash.
Malik Forte’s advice to anybody trying to decide between Unreal and Unity is that if you’re looking for accessibility and don’t want to get too deep into learning a code language, then Unreal is the way to go. It’s a lot more accessible and easier to use.
“Unity is for more veteran developers who understand coding language and are further along with working with game engines,” he says.
So, for someone who is just getting started and wants to get an idea of how a video game engine works, according to Forte, Unreal is the way to go to get the hang of things.
“If you want to move on to something bigger like Unity, you can do that later,” says Forte.

Community & the Design Process: But Make Your Game

Community building is essential when designing your own game. Some designers are great at it. They are going to love us. And they might have loved us as humans. But the minute the game comes out, if it does not engage them, it does not matter.
Drawing a Death – When Community-Building Becomes Excessive
David Jaffe recalls a game he worked on. “Drawing a Death is the biggest failure I’ve ever worked on,” he says. “It was a total crash and burn. I loved it-super proud of it, so proud of the team. Got a couple of good reviews. But mostly, the reviewers hated it. The audience didn’t show up for it-total disaster.” However, Jaffe and his team did great with the community.
So, what went wrong? Jaffe felt they spent too much time community building on drone death. While they were community building, their open and closed betas were not doing well.
But you can put such an emphasis on community that you give it way too much value than it really deserves. Then, it is more lip service. They will say, “oh, community is everything. We listen all the time. We’re here to serve you.” But that is not true. As David Jaffe learned, if you focus too much time on community-building, you can lose sight of how well the game is performing.
Community is Not Just a Two-Way Street
The relationship between customer and designer works when you make what you want, the team wants, and you take appropriate feedback from the community. Pleasing the customer is crucial, but building a successful game is also important. It is not really a two-way street. To keep the relationship healthy, it is more like a 1 1/2 way street.

Community & the Design Process: Case Study in Community-Based Design: Roblox

The way Roblox looks at building our developer and player communities comes from a place of respect built into the very foundation of what Roblox is, starting with the building tools.

For example, Roblox Studio uses LUA, one of the most friendly coding languages there is. That wasn’t an accident. As a company, it was essential to choose a coding language that was accessible for all, not just those with a gaming education.

Rule 1: Give Players Creative Freedom

[Please embed: https://pixabay.com/images/id-1077863/ ]

From the very beginning, we wanted players to have the option of creating giant immersive worlds to expressive themselves in. Each decision made about gaming tools focuses on these shared collaborative experiences.

We respect our community; it’s one of our pillar beliefs.

That foundational respect is considered each step of the way. Before making decisions, we ask internal questions such as, “Is this really respecting the community?”

We recognize that the company is nothing without our players and developers. It’s no different than in any other gaming company, but Roblox, in particular, leads with our need to take care of them.

Rule 2: Listen to Your Players

We realized long ago that the best people to talk to when determining what players want were the players themselves.

[Please embed: https://pixabay.com/images/id-2587571/ ]

Because of this, we want to give them the tools they need to create. And that’s what we do. That’s where we see our players become developers.

They create games that a traditional gaming company would probably never allow.

If you look at the top games on Roblox and would pitch them in a traditional green light fashion in the gaming industry, they probably would never happen. But Roblox doesn’t have that green light process.

Any kid can come on and create a game. It allows them to express themselves without adults getting in the way and assuming to know what kids like. Kids know what kids like.

Rule 3: Remove Roadblocks for Community Development

For developers coming into Roblox, there are tremendous opportunities for creativity and expression in a fast-to-market way. It’s possible to come up with a wacky idea and create it, no matter what your crazy idea is.

You might think, “I want to make the floor lava, but not just the floor, the sky too!” I just came up with that idea, but on Roblox, you could prototype it very fast and then share it with the Roblox community instantly.